Specific Priorities
Object-based verification approach
RAL will continue to expand its internationally-recognized Verification Program and will further develop the diagnostic “object-based” verification approach for spatial forecasts of convection and precipitation. See the figure below for an illustration of this approach. The object-based approach provides diagnostic information about forecast performance, which focuses on the sources of forecast errors rather than simply providing traditional scores and measures. The goal of the approach is to provide information that can be used to improve forecasts, as well as information that can be used by decision makers to make optimal use of the forecasts. In addition to basic enhancements to the approach (e.g., adding the time dimension), this work will be extended to additional variables and to ensemble and probabilistic forecasts. Initially, much of this effort will continue to focus on the short-term prediction problem, but we also intend to investigate applications to global models and longer time ranges, which have received relatively little attention.
The following two specific tasks are a high priority for verification research efforts:
- Develop and apply the new phenomenologically-based techniques for intelligently assessing the quality, utility, and value of forecasts from high-resolution NWP models
- Adapt these verification techniques to probabilistic and ensemble forecasts and regional climate forecasts.
Verification toolkit
RAL will maintain and update a toolkit of verification methods in the open-source R language for use by the forecasting and verification communities. New methods, including spatial verification approaches, will be incorporated as they become available from NCAR and the wider (international) verification communities. This toolkit will also be used in workshops and tutorials to help guide the meteorological community regarding application of appropriate verification methodologies.
Example of object-based verification approach applied to Weather Research and Forecasting (WRF) model precipitation forecast (left) and Stage IV precipitation observations (right). Middle diagrams [(c) and (d)] show the raw precipitation amounts (22-km scale); lower diagrams [(e) and (f)] show the regions selected for evaluation as objects; and top diagrams [(a) and (b)] show the matched forecast and observed objects (colors indicate which pairs are matched).
Uncertainty in verification measures
RAL will extend the development of approaches to measure the uncertainty in verification measures – including hypothesis tests and confidence intervals – which will enable appropriate intercomparisons of forecasting systems. These approaches will consider spatial variability as well as observational uncertainty, and will allow rational choices between forecasting systems.
User-focused verification approaches
RAL will develop additional diagnostic tools that will be more informative, in general, for most forecast users. In addition, the RAL Verification Group will work closely with the Societal Impacts Program to identify the specific needs of particular users regarding forecast uncertainty and the quality and performance of forecast products. These measures and approaches will be directly related to the utility and optimal application of the forecasts.
International collaboration on forecast verification
RAL will lead and participate in a variety of efforts around the world that involve forecasting and forecast verification, including THORPEX and other WMO-sponsored programs (e.g., the Beijing 2008 summer Olympics Forecast Demonstration Project). RAL will provide leadership to the international community through the WMO Working Group on Verification. In addition, RAL will help develop and lead international tutorials on forecast verification methods and will develop and participate in workshops on leading-edge research on forecast verification methods.