Forecast Verification Reading Group
- Suggested Papers
- Past Papers/Presentations
- Join/Leave the Group
The purpose of this reading group is to discuss papers relating to forecast verification. Currently, we will be discussing papers related to the issue of climate model evaluation. However, other topics that may come up either in conjunction with this overall topic, or in addition to this topic include:
• Ensemble Verification Methods
• Confidence Intervals and Hypothesis Testing -- Including such issues as spatial/temporal dependence, multiple testing, Non-Gaussian Random Variables, bootstrap intervals, Bayesian precision intervals, Observational Error.
• Verification of Extreme or Rare Events
• Spatial Forecast Verification
• User-specific verification
Group organizer: Eric Gilleland
Our next meeting is TBD
|Paper/Seminar||Time/Place||Presenter||Presentation slides |
If there are any other papers you would like us to discuss or add to the list, please email the organizer (Eric Gilleland).
A beer glass () by a paper indicates papers "on tap" for upcoming sessions.
Climate Model Evaluation
Not every paper in this list is specifically Climate Model Evaluation per se, but are listed here anyway.
Hueffer, K., M. A. Fonseca, A. Leiserowitz, and K. M. Taylor, 2013. The wisdom of crowds: Predicting a weather and climate-related event. Judgment and Decision Making, 8 (2), 91 - 105.
Livina, V. N., N. R. Edwards, S. Goswami, and T. M. Lenton, 2008. A wavelet-coefficient score for comparison of two-dimensional climatic-data fields. Q. J. R. Meteorol. Soc., 134, 941 - 955.
Moise, A. F. and F. P. Delage, 2011. New climate model metrics based on object-oriented pattern matching of rainfall. J. Geophys. Res., 116, D12108, 7 pp., doi:10.1029/2010JD015318
Reichler, T. and J. Kim, 2008. How well do coupled models simulate today's climate? Bull. Amer. Meteorol. Soc., 89, 303 - 311, http://dx.doi.org/10.1175/BAMS-89-3-303.
Woldemeski, F. M., A. Sharma, B. Sivakumar, and R. Mehrotra, 2012. An error estimation method for precipitation and temperature projections for future climates. J. Geophys. Res., 117, D22104, 13 pp., doi:10.1029/2012JD018062.
Tropical Cyclone (and similar) Forecast Verification
Aberson, S. D., 2008. An alternative tropical cyclone intensity forecast verification technique Wea. Forecasting, 23, 1304--1310.
Done, J., G. J. Holland, C. Bruyère, and A. Suzuki-Parker, 2011. Effects of climate variability and change on Gulf of Mexico tropical cyclone activity. Proc. Offshore Technology Conference, 22190 (http://www.onepetro.org/mslib/servlet/onepetropreview?id=OTC-22190-MS&soc=OTC)
Gneiting, T., Stanberry, L. I., Grimit, E. P., Held, L. and Johnson, N. A. (2008). Assessing probabilistic forecasts of multivariate quantities, with applications to ensemble predictions of surface winds. Test, 17, 211-235. (Also has comment papers and a rejoinder)
Holland, G. J., J. Done, C. Bruyère, C. Cooper, and A. Suzuki-Parker, 2010. Model investigations of the effects of climate variability and change on future Gulf of Mexico tropical cyclone activity. Proc. 2010 Offshore Technology Conference, 20690. (http://www.onepetro.org/mslib/servlet/onepetropreview?id=OTC-20690-MS&soc=OTC).
Moskaitis, J. R., 2008. A Case study of deterministic forecast verification: tropical cyclone intensity. Wea. Forecasting, 23, 1195--1220.
Extreme and Rare Events
Atger F, 2001. Verification of intense precipitation forecasts from single models and ensemble prediction systems. Nonlin. Proc. Geophys., 8:401--417
Brooks HE, M Kay and JA Hart, 1998. Objective limits on forecasting skill of rare events. 19th Conf. Severe Local Storms, Amer. Met. Soc., 552--555
Ferro, C.A.T., 2007. A Probability Model for Verifying Deterministic Forecasts of Extreme Events. Wea. Forecasting, 22(5):1089--1100, DOI: 10.1175/WAF1036.1
Ferro, C.A.T. and D.B. Stephenson, 2011: Extremal dependence indices: improved verification measures for deterministic forecasts of rare binary events. Accepted to Wea. Forecasting. Available at Early Online Release from: http://journals.ametsoc.org/doi/pdf/10.1175/WAF-D-10-05030.1
Friederichs, P., M. Göber, S. Bentzien, A. Lenz, and R. Krampitz, 2009. A probabilistic analysis of wind gusts using extreme value statistics. Meteorol. Z., 18 (6), 615--629, DOI 10.1127/0941-2948/2009/0413.
Stephenson DB, B Casati, CAT Ferro and CA Wilson, 2008. The extreme dependency score: a non-vanishing measure for forecasts of rare events. Meteorol. Appl., 15:41--50, DOI: 10.1002/met.53
Spatial Forecast Verification
Elmore, K. L., M. E. Baldwin, and D. M. Schultz, 2006. Field significance revisited: Spatial bias errors in forecasts as applied to the Eta model. Mon. Wea. Rev., 134, 519--531.
A list of spatial verification papers that is relatively up-to-date can be found at the ICP web site http://ww.ral.ucar.edu/projects/icp
|Paper/Seminar||Time/Place||Presenter||Presentation slides |
|2013 Series on Climate Model Evaluation|
|Taylor, K. E., 2001. Summarizing multiple aspects of model performance in a single diagram. J. Geophys. Res., 106, (D7), 7183 - 7192.||Wednesday, 12 June 2013 at 10:00am/FL3-1021||Barb Brown||-|
|Gilleland, E., 2013. Testing competing precipitation forecasts accurately and efficiently: The spatial prediction comparison test. Mon. Wea. Rev., 141, (1), 340 - 355.||Monday, 22 April 2013 at 10:00am/FL3-1029||Eric Gilleland||-|
|Seminar Presentation: Some insights into cloud forecast biases and skill: Putting data and methods under the spotlight (pdf announcement/abstract)||Tuesday, 23 August 2011 at 3:00pm/FL2-1001||Marion Mittermeier||-|
|2011 Series on Tropical Cyclone (and similar) Forecast Verification|
|Discussion of methods used officially by the National Hurricane Center||Thursday, 6 October 2011 at 10:00am/FL3-1029||Mrinal Biswas||-|
|Dupont, T., M. Plu, P. Caroff, and G. Faure, 2011. Verification of ensemble-based uncertainty circles around tropical cyclone track forecasts. Accepted to Wea. Forecasting, Available at Early Online Release||Thursday, 18 August 2011 at 3:00pm/FL3-1029||Eric Gilleland||-|
|Davis, C. A., W. Wang, J. Dudhia and R. Torn, 2010: Does increased horizontal resolution improve hurricane wind forecasts? Wea. Forecasting, 25, (6), 1826--1841.||Thursday, 7 July 2011 at 10:00am/FL3-1021||Ligia Bernadet||-|
|Colle, B.A. and Charles, M.E., 2011: Spatial Distribution and Evolution of Extratropical Cyclone Errors over North America and Adjacent Oceans in the NCEP Global Forecast System Model. Wea. Forecasting, 26, 129--149. Available at Early Online Release from: Early Online Release||Thursday, 26 May 2011 at 2:00pm/FL3-1029||Eric Gilleland||-|
|2008 Series on Spatial Forecast Verification|
|Keil C and GC Craig, 2007. A displacement-based error measure applied in a regional ensemble forecasting system. Mon. Wea. Rev., 135:3248--3259.||Tuesday, June 3 at 12:15/EOL-Atrium (FL1-2198)||Matt Pocernich||-|
|Seminar Presentation: Image Warping for Forecast Verification (abstract, txt)||Tuesday, June 3 at 10:45am/FL-MSR (FL2-1022)||Johan Lindström||(pdf)|
|Seminar Presentation: High-resolution time lagged ensembles: walking the resolution-predictability tightrope (abstract, txt)||Wednesday, April 23 at 3:00pm/FL-MSR (FL2-1022)||Marion Mittermaier||-|
|Seminar Presentation: A wavelet-based verification approach to account for the variation in scale representativeness of observation networks (abstracts, txt)||Tuesday, April 8 at 3:00pm/FL-MSR (FL2-1022)||Barbara Casati||-|
|Venugopal, V.; Basu, S.; Foufoula-Georgiou, E. 2005. A new metric for comparing precipitation patterns with an application to ensemble forecasts. J. Geophys. Res. 110: D8, D08111 10.1029/2004JD005395.||Friday, April 4 at 10:00am/FL1-2133||David Ahijevych||(ppt, pdf)|
|Roberts NM and HW Lean, 2008. Scale-selective verification of rainfall accumulations from high-resolution forecasts of convective events. Mon. Wea. Rev., 136:78--96.||Thursday, March 13 at 3:00pm/FL3-2072||Eric Gilleland|
You can subscribe or unsubscribe from our email list at http://mail.rap.ucar.edu/cgi-bin/mailman/listinfo/vx-reading.*
Please also send an email to Eric Gilleland to explain who you are and why you want to be on the list to help protect against spam.
*Email addresses are retained only for announcing pertinent information about the reading group (e.g., meeting times, papers to be read, etc.).