Quality Assessment/Verification
An example application of the MODE (Method for Object-based Diagnostic Evaluation) approach to a National Convective Weather Forecast (NCWF2) prediction from 15 September 2005 at 2300 UTC. The one-hour forecast coverage field is shown in the left column and convective observations for the valid time are shown in the right column. The top panel shows the raw forecast and observation fields on a 4-km grid. The lower panel shows the objects defined by the evaluation approach, the merged sets of objects (“composite” objects) in each field (indicated by the lines surrounding the groups of objects; the “convex hulls”) and the matched pairs of composite objects in the forecast and observed fields (indicated by object color). For this case, 32 simple forecast objects and 22 simple observation objects were identified. Three simple objects in each field were unmatched (royal blue color) and 16 matched composite objects were identified.
The FAA’s Aviation Weather Research Program (AWRP) is engaged in research that is directed toward improving weather forecasts for the aviation community. Much of this research, in the form of automated algorithms to predict aviation weather phenomena such as icing and turbulence, is transferred from research laboratories to the National Weather Service (NWS) through the Aviation Weather Technology Transfer (AWTT) process. The Quality Assessment Product Development Team (QA PDT) is responsible for the development and application of appropriate methods to evaluate these products and to ensure their scientific capabilities as they transition to operations. The QA PDT identifies observational datasets that are appropriate for comparison to particular types of forecasts, and develops methodologies to (a) answer specific questions about the performance of the forecasts and (b) provide meaningful information about forecast quality in the context of the operational application of the forecasts. The PDT implements these methodologies in controlled evaluations and intercomparison studies; analyzes the results of the evaluation studies to assess the performance of new forecasting techniques relative to the performance of current operational forecasts; and reports on the results of the studies to the AWTT evaluation board.
Accomplishments in FY06
Due to the nature of aviation weather and the limited observations available that represent aviation weather phenomena (e.g., in-flight icing), much of the effort of the QA PDT and the RAL Verification Group focused on identification of new observation datasets that can be used for verification and testing new methodologies for use in specific forecast evaluation studies. Work focused on examining icing observations from the TAMDAR (Tropospheric Airborne Meteorological Data Reporting) program for use in developing a climatology of icing conditions and evaluating icing forecasts; investigating the use of in-situ turbulence observations available from certain commercial aircraft; and evaluating the use of observations from the Multi-angle Imaging Spectro-Radiometer (MISR) instrument on the Terra satellite for evaluations of the cloud-top height product developed by the Oceanic Weather Product Development Team. The QA PDT also continued development of the Method for Object-based Diagnostic Evaluation (MODE) and applied it to evaluations of the National Convective Weather Forecast (NCWF2) product (FIGURE A) and the NCAR Autonowcaster. Other products that were evaluated for the AWTT process included the Current Icing Product (CIP) and the Graphical Turbulence Guidance Product (GTG2).