Background
Forecast verification and evaluation activities historically have focused on relatively simple metrics regarding the meteorological performance of forecasts and forecasting systems. Metrics such as the Probability of Detection (POD), Root Mean Squared Error (RMSE), and Equitable Threat Score (ETS) provide information that is useful for monitoring changes in performance with time. However, they generally do not provide information that can be used to improve the forecasts, or that can be used by end users (including forecasters) for decision making. Moreover, it is possible for forecasts that are quite useful to score very poorly on the basis of these standard metrics. For these reasons, members of the NCAR/RAL Verification Group work to develop improved verification approaches and tools that can overcome some of the limitations of standard approaches. The focus of this effort is on diagnostic, statistically valid approaches, including object-based evaluation of precipitation and convective forecasts and other approaches (e.g., distribution-based) that can provide more useful information about forecast performance.
Current Activities
A major effort in 2005 was further development and application of the object-based verification approach. This approach (collaboratively developed by B. Brown, R. Bullock, C. Davis, J. Halley Gotway, and others) objectively defines precipitation/convective objects in the forecast and observation fields, attempting to mimic the way a human analyst would select meaningful regions. The system then matches pairs/sets of forecast and observed objects and evaluates and compares a variety of attributes of the pairs. The results of an evaluation using this approach are distributions of attributes and errors that are meaningful for users and developers (e.g., location, size, and timing errors). In 2005, a display and analysis system was developed by R. Bullock to allow investigation of variations of parameters needed to define, merge, and match objects. This system allows the conduct of a wide variety of experiments related to the application of the system to both numerical model precipitation forecasts and mesoscale convective forecasts. An intensive effort was focused on application of the approach to convective forecasts produced by the NCAR Autonowcaster system. This system incorporates two different types of forecasts, including a probabilistic convective initiation forecast and a deterministic storm motion, growth, and decay field; the combination of these fields provide a complex application of the verification approach.
Other verification activities included development and application of confidence intervals for model intercomparisons. When NWP models are being evaluated or compared, the common practice is to select the model with the “best” performance; this approach has traditionally ignored sampling error associated with the verification statistics. Thus, statistical confidence intervals are needed to ensure that any identified improvements or changes have statistical validity. The use of confidence intervals in these comparisons is now being strongly encouraged and was demonstrated by T. Fowler in a study for the Developmental Testbed Center (DTC).
Recent Accomplishments
A number of accomplishments can be identified for the past year:
- The object-based verification approach was extended and tested and applied to new types of NWP output, as well as to convective nowcasts.
- An interface was developed to allow testing of fuzzy logic and other parameters needed for the object-based verification system. This system provides immediate feedback on the effects of variations in the parameter settings.
- Two journal articles on applications of the object-based approach were accepted for publication in Monthly Weather Review.
- An R language verification library, developed by M. Pocernich, was extended to include more facilities. This system is very widely used internationally.
- B. Brown is participating in a NAS/NRC study on estimating and communicating uncertainty in weather and climate forecasts.
Plans for 2006
With regard to the object-based verification approach, a number of extensions and improvements are planned, including incorporation of the time dimension in the definition of objects, and the establishment of a general approach for merging shapes in the same field. The approach will also be tested for evaluating hurricane forecasts, which will require the use of satellite observations. In addition, an initial investigation of the use and benefit of this approach for evaluation of ensemble forecasts will be pursued. In particular, the concept of ensemble objects will be investigated.
The Verification Group will continue to advocate for the use of statistically valid diagnostic tools in verification studies and will contribute to the international verification community through the WMO Joint Working Group on Verification, which B. Brown chairs. Among other activities, this group is developing a plan for verification studies associated with a forecast demonstration project in Beijing in 2008. In addition, plans will be undertaken for a verification workshop and tutorial, sponsored by this working group, to be held in 2007. The NCAR/RAL Verification Group will also continue to work with the DTC on development and application of appropriate verification approaches, and will investigate other application areas, such as through the THORPEX/TIGGE program.