Forecast Evaluation and Applied Statistics

Providing meaningful information to scientific developers and operational end users

Welcome to the homepage of the Forecast Evaluation and Applied Statistics group of RAL's Weather Systems and Assessment Program (WSAP), which was formed out of the earlier forecast verification group within RAL. The new name better reflects the wide variety of projects and activities undertaken by us; including extreme value statistical analysis applied to weather and climate data, weather modification, and the development of a state-of-the-art suite of software tools for performing forecast verification.

The page is organized into a tabbed front page giving information about: relevant software (including many software packages designed by our staff), activities of our group (as well as some related activities by other groups in the Colorado area), related links, and people. Links to pages about specific projects involving our group are given at the right along with upcoming workshops being largely organized by us.

The roots of applied statistics in RAL are in forecast verification, which is the process of determining the quality of forecasts. Below is some further information about this subject.

Statistical verification of forecasts is a critical component of their development. Improvements can be made by evaluating forecast products throughout the development process as deficiencies in the algorithms are discovered. Verification is also beneficial to forecasters and end users because verification findings supply them with objective data about the quality or accuracy of the forecasts, which can feed into decision processes (Brown, 1996).

MET -- Model Evaluation Tools

Forecast verification software initially developed for the WRF model. Developed by John Halley Gotway, Lacey Holland, Randy Bullock, David Ahijevych, Barbara Brown, Chris Davis, and Eric Gilleland

R software packages

Developed primarily by our staff include:

Extremes Toolkit (extRemes)
Graphical User Interface software for weather and climate applications of statistical extreme value analysis. Developed by Eric Gilleland, Rick Katz (ISSE), and Greg Young.

verification
Forecast verification analysis and graphical functions. Developed by Matt Pocernich.

Other statistical software developed at NCAR

IMAGe Software

• Reading Groups
• Meetings
• Selected Conferences
• Schedule of Seminars
• Visitor Schedule
• Outside Seminars

The following reading groups are organized within the Forecast Evaluation and Applied Statistics group.

• Statistical Analysis of Extremes in Geophysical Science
• Verification Reading Group

The following is a schedule of 2008 meetings organized in full or in part by the Forecast Evaluation and Applied Statistics group.

• World Meteorological Organization (WMO): Tropical Cyclones and Joint Working Group on Verification, NCAR's Mesa Lab 21-23 April 2008.
• ICP Spring Workshop, 14-15 April, 2008, Boulder, Colorado.
• DTC Verification Workshop, 16-18 April 2008, Boulder, CO.

Selected seminar presentations by our staff.

• 2007-08 Program on Risk Analysis, Extreme Events and Decision Theory, Opening workshop, 16-19 September, 2007, Statistical and Applied Mathematical Sciences Institute (SAMSI), Research Triangle Park, North Carolina, U.S.A.
  • Gilleland E., Extremes and Atmospheric Data (invited talk, pdf)

The following regularly scheduled seminars are offered by other similar institutions, but are not necessarily affiliated with the Forecast Evaluation and Applied Statistics group.

• CSU Statistics Department Seminars
• CU Applied Mathematics Events and Activities

Refereed Papers

Gilleland E, TCM Lee, J Halley Gotway, RG Bullock, and BG Brown, 2008. Computationally efficient spatial forecast verification using Baddeley's Δ image metric. Mon. Wea. Rev. 136(5):1747--1757.

Davis CA, BG Brown, and RG Bullock, 2006a. Object-based verification of precipitation forecasts, Part I: Methodology and application to mesoscale rain areas. Mon. Wea. Rev. 134:1772--1784.

Davis CA, BG Brown, and RG Bullock, 2006b. Object-based verification of precipitation forecasts, Part II: Application to convective rain systems. Mon. Wea. Rev. 134:1785--1795.

Gilleland E and TL Fowler, 2006. Network design for verification of ceiling and visibility forecasts, Environmetrics 17(6):575--589.

Stephenson A and E Gilleland, 2005. Software for the Analysis of Extreme Events: The Current State and Future Directions, Extremes 8:87--109.

Gilleland E and D Nychka, 2005. Statistical Models for Monitoring and Regulating Ground-level Ozone, Environmetrics 16: 535--546.

Non-refereed Papers

Ahijevych DA, E Gilleland, BG Brown, EE Ebert, L Holland, and C Davis, 2008. Intercomparison of spatial verification methods. 88th Annual American Meteorological Society (AMS) meeting, New Orleans, Louisiana. 9.1 Probability/Statistics conference.

Gilleland E and RW Katz. "Analyzing seasonal to interannual extreme weather and climate variability with the extremes toolkit (extRemes)", 18th Conference on Climate Variability and Change, 86th American Meteorological Society (AMS) Annual Meeting, 29 January - 2 February, 2006, Atlanta, Georgia. P2.15

Technical Reports

Books and Book Chapters

Gilleland E, D Nychka, and U Schneider, 2006. Spatial models for the distribution of extremes, Hierarchical modelling for the Environmental Sciences: statistical methods and applications, Edited by JS Clark and A Gelfand. Oxford University Press, New York pp.170--183. ISBN 0-19-8569671

Related Links

• Cooperative Institute for Research in Environmental Sciences (CIRES)
• Developmental Testbed Center (DTC)
• Weather Systems and Assessment Program (WSAP)
• Weather Research and Forecasting (WRF) Model Support
• Data Assimilation Testbed Center (DATC)
• U.S. Air Force Weather Agency (AFWA)
• American Meteorological Society -- Committee on Probability and Statistics
• National Oceanic and Atmospheric Administration (NOAA)
• NCAR's Geophysical Statistics Project (GSP)
• ADDS -- Aviation Digital Weather Service
• AWC -- Aviation Weather Center
• RTVS -- Real-Time Verification System
• Forecast Verification -- Issues, Methods and FAQ

• Staff and Primary Contacts
• Collaborators

Staff and Primary Contacts

• BROWN, Barbara | WSAP DEPUTY DIRECTOR | ph: 8468 | email: bgb
• BULLOCK, Randy: | SOFT ENG/PROG III | ph: 8444 | email: bullock
• CHAPMAN, Michael: | ASSOC SCIENTIST III | ph: 8395 | email: mchapman
• GILLELAND, Eric | PROJ SCIENTIST I | ph: 2849 | email: ericg
• HALLEYGOTWAY, John | SOFT ENG/PROG II | ph: 2861 | email: johnhg
• HOLLAND, Lacey | ASSOC SCIENTIST II | ph: 8396 | email: lholland
• KUCERA, Paul | PROJ SCIENTIST II | ph: 2807 | email: pkucera
• POCERNICH, Matt: | ASSOC SCIENTIST III | ph: 8312 | email: pocernic

• Ahijevych, David, Mesoscale and Microscale Meteorology, National Center for Atmospheric Research (NCAR), Boulder, Colorado.
• Casati, Barbara, Ouranos Consortium on Regional Climatology and Adaptation to Climate Change, Montreal, Canada
• Bernadet, Ligia, NOAA Research-Earth System Research Laboratory/Global Systems Division, Boulder, Colorado, U.S.A.
• Brooks, Harold, Head, Mesoscale Applications Group, NSSL/FRDD National Weather Center, NOAA, Norman, Oklahoma.
• Davis, Chris, Mesoscale and Microscale Meteorology, National Center for Atmospheric Research (NCAR), Boulder, Colorado.
• Ebert, Beth, Bureau of Meteorology Research Centre, Weather Forecasting Group, Melbourne, Victoria, Australia.
• Katz, Richard W., Institute for Study of Society and Environment (ISSE), National Center for Atmospheric Research (NCAR), Boulder, Colorado, U.S.A.
• Lindgren, Finn, Mathematical Statistics, Centre for Mathematical Sciences, Lund University, Lund, Sweden.
• Lindström, Johan, Mathematical Statistics, Centre for Mathematical Sciences, Lund University, Lund, Sweden.
• Madine, Sean, NOAA Research-Earth System Research Laboratory/Global Systems Division, Boulder, Colorado, U.S.A.
• Mahoney, Jennifer, NOAA Research-Earth System Research Laboratory/Global Systems Division, Boulder, Colorado, U.S.A.
• Mittermaier, Marion. Mesoscale Model Development and Diagnostics Group, NWP, Met Office, United Kingdom

Visitors