Spatial Forecast Verification Methods Inter-Comparison Project

About the ICP

The spatial forecast verification inter-comparison project (ICP) was set up to attempt to sift through the maze of newly proposed methods for verifying primarily high-resolution forecasts.

The intent of this project is to compare the various newly proposed methods to give the user information about which methods are appropriate for which types of data, forecasts and desired forecast utility. The point is not to determine which method is best. However, it is hoped that advantages and disadvantages of the methods will become clear.

Most new methods are concerned with gridded verification sets, and that has been (so far) the thrust of the ICP.

ICP2 Coming Soon!

If you are interested in participating in the second round of the ICP (ICP2), please click on the Email list tab to subscribe. You will then be kept informed of activities.

News

Check here for the latest ICP news.

We are in the very early stages of planning the continuation of the ICP.

Upcoming Relevant Meetings/Conferences/Workshops

13th EMS Annual Meeting and 11th European Conference on Applications of Meteorology (ECAM) 9 - 13 September 2013

ICP1

The test cases for the ICP1 included geometric idealized, perturbed real, and real cases. There were some additional cases as well, but they were not the primary focus. These cases are available and described below. Test cases for ICP2 are coming soon.

Information

We currently have three sets of test cases picked out for this project. Thanks to Mike Baldwin, Beth Ebert, Barbara Casati and others for helping to obtain them, and to David Ahijevych for creating the fake test cases. All of the cases are included for download by clicking on the link to this file: Cases20081023.tar.gz

Once extracted you will have the following folders inside the folder called Cases: 'sp2005,' 'nimrod,' 'MeteoSwiss,' and 'Fake.' Inside 'Fake' are further folders for the geometric and perturbed cases. Many of these test cases are also available with the R package SpatialVx.

Initial Test Cases from the Storm Prediction Center/National Severe Storms Laboratory (SPC/NSSL) Spring 2005 Program (sp2005)

We have cases from the WRF Spring 2005 program (see Kain et al., 2008 Wea. Forecasting, 23:931--952) including stage II analyses and three versions of WRF referred to as 'wrf2,' 'wrf4ncar' and 'wrf4ncep.' The fields are hourly accumulated precipitation in hundredths-of-an-inch. Please convert to mm by multiplying by 0.254 so that we all use the same units. All of these models have been interpolated onto the same 501 x 601 grid. Specifically, the NCEP g240 grid, which is a "4-km grid," but note that the grid is irregular (i.e., not always 4-km at each grid square). The longitude and latitude information is provided in the sp2005 folder as 'g240LatNorth.txt' and 'g240LonEast.txt.'

In the 'figures' folder you will find postscript and pdf graphs in the 'ps' and 'pdf' folders, respectively, of the first 9 cases chosen from this Spring program for this inter-comparison. Note that there are a couple extra cases in the 'st2,' 'wrf2' and 'wrf4ncar' folders. We did not have these dates for the NCEP model, so we will exclude them, but they are still provided as they may make for nice cases. All files are in ASCII format. See the README file in this folder for more information.

The cases we will start with are for verification times (00Z issue times): 4/26, 5/13, 5/14, 5/18, 5/19, 5/25, 6/1, 6/3 and 6/4.

The entire 32 Spring 2005 ARW and NMM cases studied in Davis et al (2009), Gilleland et al (2010) and Gilleland (2013) are available here (spc2005agg24hLead00UTC.tar.gz).

Nimrod Test Cases

Please acknowledge the UK Met Office (see the README file in the 'Nimrod' folder. These are the same cases used in the Casati et al (2004) paper (Meteorol. Appl. 11:141--154), and are described therein. We do not presently have lon/lat info for these cases (sorry).

MeteoSwiss Test Cases

Daniel Leuenberger at MeteoSwiss has offered two cases we could consider using. He asks that we acknowledge their source, and use them only for research purposes (see the 'README' file in the 'MeteoSwiss' folder). The cases give high-resolution aLMo radar data over Switzerland, along with hourly forecasts for a 24-hour period. One case also has results on a 7-km grid corresponding to a lower resolution version of the model. The files are in netCDF format, and IDL code to read them is included. One case has a postscript file that shows the hourly fields and basic verification statistics. The other case has a postscript file for the 24-hr totals, plus gif files for 6h totals.

Fake Test Cases

Fake test cases are based on the same grid as the sp2005 cases. There are both idealized geometric cases as well as perturbed real cases. See the README files for each case for more information.

---

Future ICP Meetings

TBD

Past ICP Meetings

Fall 2009 Workshop: 24-25 August 2009 at NCAR's Center Green Campus, Boulder, Colorado.

Spring 2008 workshop held in Boulder, Colorado 14-15 April 2008.

20 February 2007 Planning Meeting (includes slides from the various talks).

Other Upcoming Related Meetings/Conferences/Workshops

13th EMS Annual Meeting and 11th European Conference on Applications of Meteorology (ECAM) 9 - 13 September 2013

Preliminary Results

An old link to some early results that may still be useful (mostly from past meeting presentations).

We will provide links to available software for performing spatial forecast verification from this page as they become available. All references listed below can be found in-full by clicking on the references tab above.

SpatialVx

A new R package called SpatialVx is in the works. It already has functions to do many of the newly proposed methods, but there are many yet to do. Keep checking for new updates.

Traditional Verification

The following book has a nice appendix by Matt Pocernich on verification software (pp. 231--240). Also see Forecast Verification Issues, Methods and FAQ. From these, it is clear that several software tools for traditional verification are available. Listed here are only tools that were developed specifically at NCAR.

Jolliffe, I.T. and Stephenson, D.B., Forecast Verification: A practioner's guide in atmospheric sciences, Second Edition, Wiley-Balckwell, Chichester, West Sussex, U.K., 274 pp.

R package verification by Matt Pocernich. From your R session, you can install the package onto your machine with the lib.loc function. You can then load it into your R session using the library function.

Model Evaluation Tools (MET) user's home page

SpatialVx contains some functionality for doing the traditional forecast verification methods.

Features-based Approaches

Contiguous Rain Area (CRA) (Ebert and McBride, 2000). IDL code from Beth Ebert available at http://www.cawcr.gov.au/staff/eee/index.php

Method for Object-based Diagnostic Evaluation (MODE) tool (Davis et al., 2006). Available through MET user's home page.

SpatialVx has some functionality for doing the method proposed by Davis et al. (2006), as well as the SAL technique (Wernli et al., 2008). In particular, it is possible to do either convolution thresholding, or just thresholding to identify features (or objects) as connected grid points above the threshold. It also has functions to calculate properties of single features (major/minor axis angle/length, aspect ratio, area, centroid) and properties for matched features (e.g., intersection area, area ratio, centroid distance, angle difference, as well as numerous binary image metrics; see below). More is on tap for future releases.

Field Deformation Approaches

Baddeley's Delta Metric: R code for computing Baddeley's delta metric for binary images (see, e.g., Gilleland, 2011)

FQI (Venugopal et al, 2005; Basu et al, 2003)

Image Warping: an R package to do the image warping found in Gilleland, Lindström and Lindgren (2010) is coming soon!. The original MatLab code will be replaced by these R functions, and they will be available through the SpatialVx package.

SpatialVx has functions to calculate FQI, Baddeley's Delta Metric, the Hausdorff metric, partial Hausdorff measure, mean error distance, mean square error distance, Pratt's Figure of Merit (FOM), minimum separation distance, (see the help file for locperf for references).

Neighborhood-Based Approaches (summarized in Ebert, 2008)

IDL code from Beth Ebert available at http://www.cawcr.gov.au/staff/eee/index.php

The R package SpatialVx has functions for doing most of the neighborhood methods.

Scale Decomposition Approaches

Intensity-Scale (IS) (Casati et al, 2004). Available in the R package verification (above under traditional verification). Also available from the Model Evaluation Tools (MET) software.

SpatialVx has the wavelet denoising (more a neighborhood type approach), and wavelet decomposition approaches of Briggs and Levine (1997), as well as the intensity-scale techniques of Casati et al. (2004) and Casati (2009).

References

This is by no means an exhaustive list of references, but should provide a good background to the issues concerning the ICP, and all of the methods presently being compared; among others. If you know of a relevant paper that should be listed here, please contact Eric Gilleland.

We have a special collection of papers for the ICP in the Weather and Forecasting (WAF) journal of the American Meteorological Society (see list below).

Papers in the collection are marked by .

All of the references below are also available in BibTeX format, where the cite key follows the format: AuthorLastNameYear (one author; e.g., Casati2009), Author1LastNameAuthor2LastNameYear (two authors; e.g., BriggsLevine1997), Author1LastNameEtAlYear (more than two authors; e.g., AhijevychEtAl2009). If the previous does not identify a unique paper in the list (e.g., Davis et al., 2006), then a lower case letter a, b, c, ... is appended to the end (e.g., DavisEtAl2006a and DavisEtAl2006b).

---

A     B     C     D     E - F     G     H     I - K     L     M     N - Q     R     S - T     U - V     W     X - Z

---

A

AghaKouchak, A., N. Nasrollahi, J. Li, B. Imam, and S. Sorooshian, 2011: Geometrical characterization of precipitation patterns. J. Hydrometeorology, 12, 274 - 285, doi:10.1175/2010JHM1298.1

AghaKouchak, A., A. Behrangi, S. Sorooshian, K. Hsu, and E. Amitai, 2011: Evaluation of satellite-retrieved extreme precipitation rates across the central United States. J. Geophys. Res., 116, D02115, 11 pp., doi:10.1029/2010JD014741.

Ahijevych, D., E. Gilleland, B.G. Brown, and E.E. Ebert, 2009: Application of spatial verification methods to idealized and NWP-gridded precipitation forecasts. Wea. Forecasting, 24 (6), 1485 - 1497, DOI: 10.1175/2009WAF2222298.1.

Alexander, G. D. , J. A. Weinman, V. M. Karyampudi, W. S. Olson, and A. C. L. Lee, 1999: The effect of assimilating rain rates derived from satellites and lightning on forecasts of the 1993 superstorm. Mon. Wea. Rev., 127, 1433 - 1457.

Alexander, G. D., J. A. Weinman, and J. L. Schols, 1998: The use of digital image warping of microwave integrated water vapor imagery to improve forecasts of marine extratropical cyclones. Mon. Wea. Rev., 126, 1469 - 1496.

Argence, S., D. Lambert, E. Richard, J.-P. Chaboureau, N. Söhne, 2008: Impact of initial condition uncertainties on the predictability of heavy rainfall in the Mediterranean: a case study. Q.J.R. Meteorol. Soc., 134 (636), 1775 - 1788.

B

Atger, F., 2001: Verification of intense precipitation forecasts from single models and ensemble prediction systems. Nonlin. Proc. Geophys., 8, 401 - 417

Baldwin, M. E. and J. S. Kain, 2006: Sensitivity of several performance measures to displacement error, bias, and event frequency. Wea. Forecasting, 21, 636 - 648

Basu, S., B. Dodov, and E. Foufoula-Georgiou, 2003: A novel measure for QPF verification and its usefulness in multimodel ensemble forecasting, Geophys. Res. Abstracts, 5, 04323, European Geophysical Society.

Bauer, H.-S., T. Weusthoff, M. Dorninger, V. Wulfmeyer, T. Schwitalla, T. Gorgas, M. Arpagaus, and K. Warrach-Sagi, 2011: Predictive skill of a subset of models participating in D-PHASE in the COPS region. Q.J.R. Meteorol. Soc., 137, 287 - 305.

Biazeto, B. and M. A. F. Silva Dias, 2012: Analysis of the impact of rainfall assimilation during LBA atmospheric mesoscale missions in Southwest Amazon. Atmospheric Research, 107, 126--144.

Briggs, W. M. and R. A. Levine, 1997: Wavelets and field forecast verification. Mon. Wea. Rev., 125, 1329 - 1341.

Brill, K.F. and F. Mesinger, 2009: Applying a General Analytic Method for Assessing Bias Sensitivity to Bias Adjusted Threat and Equitable Threat Scores. Wea. Forecasting, 24 (6), 1748 - 1754, DOI: 10.1175/2009WAF2222272.1.

Brooks, H. E., M. Kay and J. A. Hart, 1998: Objective limits on forecasting skill of rare events. 19th Conf. Severe Local Storms, Amer. Met. Soc., 552 - 555

Brown, B. G., E. Gilleland, and E. E. Ebert, 2012: Chapter 6: Forecasts of spatial fields. pp. 95 - 117, In "Forecast Verification: A Practitioner's Guide in Atmospheric Science", 2nd edition. Edts. IT Jolliffee and DB Stephenson, Wiley, Chichester, West Sussex, UK, 274 pp.

C

Caine, S., T. P. Lane, P. T. May, C. Jakob, S. T. Siems, M. J. Manton, and J. Pinto: Statistical assessment of tropical convection-permitting model simulations using a cell-tracking algorithm. Mon. Wea. Rev., 141, 557 - 581, DOI: http://dx.doi.org/10.1175/MWR-D-11-00274.1.

Callado, A., P. Escribá, J. A. García-Moya, J. Montero, C. Santos, D. Santos-Muñoz, and J. Simarro, 2013. Ensemble Forecasting, In: Climate Change and Regional/Local Responses, Dr. Pallav Ray (Ed.), ISBN: 978-953-51-1132-0, InTech, DOI: 10.5772/55699. Available from: http://www.intechopen.com/books/climate-change-and-regional-local-responses/ensemble-forecasting

Casati, B., G. Ross, and D. B. Stephenson, 2004: A new intensity-scale approach for the verification of spatial precipitation forecasts. Meteorol. Appl. 11, 141 - 154.

Casati, B., L. J. Wilson, D. B. Stephenson, P. Nurmi, A. Ghelli, M. Pocernich, U. Damrath, E. E. Ebert, B. G. Brown, and S. Mason, 2008: Forecast verification: Current status and future directions. Meteorol. Appl., 15, 3 - 18.

Casati, B., 2010: New Developments of the Intensity-Scale Technique within the Spatial Verification Methods Inter-Comparison Project. Wea. Forecasting 25, (1), 113 - 143, DOI: 10.1175/2009WAF2222257.1.

Clark, A. J., W. A. Gallus, Jr. and M. L. Weisman, 2010: Neighborhood-Based Verification of Precipitation Forecasts from Convection-Allowing NCAR WRF Model Simulations and the Operational NAM. Wea. Forecasting, 25, 1495-1509. DOI: 10.1175/2010WAF2222404.1

Clark, A. J., J. S. Kain, P. T. Marsh, J. Correia, Jr., M. Xue, and F. Kong, 2012: Forecasting tornado pathlengths using a three-dimensional object identification algorithm applied to convection-allowing forecasts. Wea. Forecasting, 27, 1090--1113, DOI: 10.1175/WAF-D-11-00147.1.

Clark, Adam J., Jidong Gao, Patrick T. Marsh, Travis Smith, John S. Kain, James Correia, Ming Xue, Fanyou Kong, 2013: Tornado Pathlength Forecasts from 2010 to 2011 Using Ensemble Updraft Helicity. Wea. Forecasting, 28, 387 - 407.

Clark, R. Bullock, T. L. Jensen, M. Xue, and F. Kong, 2013: Application of object-based time-domain diagnostics for tracking precipitation systems in convection-allowing models. To Be Submitted to Wea. Forecasting.

D

Damrath, U., 2004: Verification against precipitation observations of a high density network--what did we learn? Intl. Verification Methods Workshop, 15-17 September 2004, Montreal, Canada.

Davis, C. A., B. G. Brown, and R. G. Bullock, 2006: Object-based verification of precipitation forecasts, Part I: Methodology and application to mesoscale rain areas. Mon. Wea. Rev. 134, 1772 - 1784.

Davis, C. A., B. G. Brown, and R. G. Bullock, 2006: Object-based verification of precipitation forecasts, Part II: Application to convective rain systems. Mon. Wea. Rev. 134, 1785 - 1795.

Davis, C.A., B.G. Brown, R.G. Bullock and J. Halley Gotway, 2009: The Method for Object-based Diagnostic Evaluation (MODE) Applied to Numerical Forecasts from the 2005 NSSL/SPC Spring Program. Wea. Forecasting, 24 (5), 1252 - 1267, DOI: 10.1175/2009WAF2222241.1.

De Sales, F. and Y. Xue, 2010: Assessing the dynamic-downscaling ability over South America using the intensity-scale verification technique. Int. J. Climatol., DOI: 10.1002/joc.2139

Demaria, E. M., D. A. Rodriguez, E. E. Ebert, P. Salino, F. Su, and J. B. Valdes, 2011: Evaluation of mesoscale convective systems in South America using multiple satellite products and an object-based approach. J. Geophys. Res., 116, D08103, 13 pp., DOI: 10.1029/2010JD015157

Demaria, E. M., 2010: Evaluating the impacts of input and parameter uncertainty on streamflow simulations in large under-instrumented basins. Ph.D. Dissertation, University of Arizona, Department of Hydrology and Water Resources. Available here.

Duc, L., K. Saito, and H. Seko, 2013: Spatial-temporal fractions verification for high-resolution ensemble forecasts. Tellus A, 65, 18171.

Duff, T. J., D. M. Chong, P. Taylor, and K. G. Tolhurst, 2012: Procrustes based metrics for spatial validation and calibration of two-dimensional perimeter spread models: A case study considering fire. Agricultural and Forest Meteorology, 160, 110 - 117, DOI: 10.1016/j.agrformet.2012.03.002.

E - F

Ebert, E. E., 2008: Fuzzy verification of high resolution gridded forecasts: A review and proposed framework. Meteorol. Appl., 15, 51 - 64. DOI: 10.1002/met.25 (Available at http://www.ecmwf.int/newsevents/meetings/workshops/2007/jwgv/METspecialissueemail.pdf)

Ebert, E.E., 2009: Neighborhood verification: A strategy for rewarding close forecasts. Wea. Forecasting, 24 (6), 1498 - 1510, DOI: 10.1175/2009WAF2222251.1.

Ebert, E.E. and W.A. Gallus, Jr., 2009: Toward better understanding of the contiguous rain area (CRA) method for spatial forecast verification. Wea. Forecasting., 24 (5), 1401 - 1415, DOI: 10.1175/2009WAF2222269.1.

Ebert, E. E. and J. L. McBride, 2000: Verification of precipitation in weather systems: determination of systematic errors. J. Hydrology, 239, 179 - 202.

Elmore, K. L., M. E. Baldwin, and D. M. Schultz, 2006: Field significance revisited: Spatial bias errors in forecasts as applied to the Eta model. Mon. Wea. Rev., 134, 519 - 531.

G

Gallus, W. A. Jr., 2010: Application of Object-Based Verification Techniques to Ensemble Precipitation Forecasts. Wea. Forecasting, 25, 144 - 158.

Germann U and I Zawadzki, 2004. Scale dependence of the predictability of precipitation from continental radar images. Part II: Probability forecasts. J. Appl. Meteorol., 43, 74 - 89

Gilleland, E., 2013: Testing competing precipitation forecasts accurately and efficiently: The spatial prediction comparison test. Mon. Wea. Rev., 141, (1), 340 - 355.

Gilleland, E., 2011: Spatial Forecast Verification: Baddeley's Delta Metric Applied to the ICP Test Cases. Wea. Forecasting, 26 (3), 409 - 415.

Gilleland, E., D. Ahijevych, B.G. Brown, B. Casati, and E.E. Ebert, 2009: Intercomparison of Spatial Forecast Verification Methods. Wea. Forecasting, 24 (5), 1416 - 1430, DOI: 10.1175/2009WAF2222269.1.

Gilleland, E., D.A. Ahijevych, B.G. Brown and E.E. Ebert, 2010: Verifying Forecasts Spatially. Bull. Amer. Meteor. Soc., 91 (10), 1365 - 1373.

Gilleland, E., J. Lindström, and F. Lindgren, 2010: Analyzing the image warp forecast verification method on precipitation fields from the ICP. Wea. Forecasting, 25 (4), 1249 - 1262.

Gilleland, E., L. Chen, M. DePersio, G. Do, K. Eilertson, Y. Jin, E.L. Kang, F. Lindgren, J. Lindström, R.L. Smith, and C. Xia, 2011: Spatial Forecast Verification: Image Warping. NCAR Technical Note, TN-482+STR, 23pp.

Gilleland, E., T. C. M. Lee, J. Halley Gotway, R. G. Bullock, and B. G. Brown, 2008: Computationally efficient spatial forecast verification using Baddeley's Δ image metric. Mon. Wea. Rev. 136 (5), 1747 - 1757.

Gorgas, T. and M. Dorninger, 2012: Quantifying verification uncertainty by reference data variation. Meteorologische Zeitschrift, 21, (3), 259 - 277.

Grams, J. S., W. A. Gallus Jr., S. E. Koch, L. S. Wharton, A. Loughe, and E. E. Ebert, 2006: The Use of a Modified Ebert-McBride Technique to Evaluate Mesoscale Model QPF as a Function of Convective System Morphology during IHOP 2002. Wea. Forecasting, 21, 288 - 306.

Gromenko, O., 2013: Spatially Indexed Functional Data. All Graduate Theses and Dissertations Paper 1526, Utah State University, http://digitalcommons.usu.edu/etd/1526

Gromenko, O. and P. Kokoszka, 2012: Testing the equality of mean functions of ionospheric critical frequency curves. Journal of the Royal Statistical Society: Series C (Applied Statistics), 61 (5), 715 - 731.

H

Haiden, T., A. Kann, C. Wittmann, G. Pistotnik, B. Bica, and C. Gruber, 2011: The Integrated Nowcasting through Comprehensive Analysis (INCA) system and its validation over the eastern alpine region. Wea. Forecasting, 26, 166 - 183, DOI: 10.1175/2010WAF2222451.1

Harris, D., E. Foufoula-Georgiou, K. K. Droegemeier, J. J. Levit, 2001: Multiscale statistical properties of a high-resolution precipitation forecast. J. Hydrometeorology 2, 406 - 418.

Hartung, D. C., J. A. Otkin, R. A. Petersen, D. D. Turner, and W. F. Feltz, 2011: Assimilation of surface-based boundary layer profiler observations during a cool-season weather event using an observing system simulation experiment. Part II: Forecast assessment. Mon. Wea. Rev., 139, 2327 - 2346. DOI: 10.1175/2011MWR3623.1

Herbort, F. and E. Etling, 2011: Post-frontal shower cells in the COSMO-DE model: A comparison with radar measurements. Meteorol. Z., 20 (2), 217 - 226.

Hering, A. S. and M. G. Genton, 2011: Comparing spatial predictions. Technometrics, 53 (4), 414 - 425, DOI 10.1198/TECH.2011.10136

Hoffman, R. N. and C. Grassotti, 1996: A technique for assimilating SSM/I observations of marine atmospheric storms: Tests with ECMWF analyses. J. Appl. Meteorol., 35, 1177 - 1188.

Hoffman, R. N., Z. Liu, J.-F. Louis, and C. Grassotti, 1995: Distortion representation of forecast errors. Mon. Wea. Rev., 123, 2758 - 2770.

Hyvärinen, O., 2011: Categorical meteorological products: Evaluation and analysis. Doctoral dissertation, University of Helsinki, Faculty of Science, Department of Physics, Division of Atmospheric Sciences. Available from here.

I - K

Johnson, A. and X. Wang, 2012: Object-based evaluation of a storm scale ensemble during the 2009 NOAA Hazardous Weather Testbed Spring Experiment. Mon. Wea. Rev., 141, 1079 - 1098, DOI: http://dx.doi.org/10.1175/MWR-D-12-00140.1.

Johnson, A. and X. Wang, 2012: Verification and calibration of neighborhood and object-based probabilistic precipitation forecasts from a multimodel convection-allowing ensemble. Mon. Wea. Rev., 140, 3054 - 3077, DOI: 10.1175/MWR-D-11-00356.1

Johnson, A., X. Wang, F. Kong, and M. Xue, 2011: Hierarchical cluster analysis of a convection-allowing ensemble during the Hazardous Weather Testbed 2009 Spring Experiment. Part I: Development of object-oriented cluster analysis method for precipitation fields. Mon. Wea. Rev., 139, (12), 3673 - 3693.

Jury, M. R., 2011: Representation of the Caribbean mean diurnal cycle in observation, reanalysis, and CMIP3 model datasets. Theor. Appl. Climatol., 12pp., DOI 10.1007/s00704-011-0462-4.

Kain, J. S., S. J. Weiss, M. E. Baldwin, G. W. Carbin, D. R. Bright, J. J. Levit, and J. A. Hart, 2005: Evaluating high-resolution configurations of the WRF model that are used to forecast severe convective weather: The 2005 SPC/NSSL Spring Program. Preprints, 21th Conference on Weather Analysis and Forecasting/17th Conference on Numerical Weather Prediction, Washington, D. C., Amer. Meteor. Soc., CD-ROM, 2A.5.

Kanamitsu, M. and L. DeHaan, 2011: The Added Value Index: A new metric to quantify the added value of regional models. J. Geophys. Res., 116, D11106, 10 pp., DOI: 10.1029/2011JD015597

Keil, C. and G. C. Craig, 2007: A displacement-based error measure applied in a regional ensemble forecasting system. Mon. Wea. Rev., 135, 3248 - 3259.

Keil, C. and G.C. Craig, 2009: A displacement and amplitude score employing an optical flow technique. Wea. Forecasting, 24 (5), 1297 - 1308, DOI: 10.1175/2009WAF2222247.1.

Kisel'nikova, V. Z., 2013: Object-based evaluation of precipitation forecast quality. Russian Meteorology and Hydrology, 38 (4), 217 - 221.

L

Lack, S., G.L. Limpert and N.I. Fox, 2010: An object-oriented multiscale verification scheme. Wea. Forecasting, 25, 79 - 92, DOI: 10.1175/2009WAF2222245.1

Lakshmanan, V. and J.S. Kain, 2010: A Gaussian Mixture Model Approach to Forecast Verification. Wea. Forecasting, 25 (3), 908 - 920.

Larsén, X. G., J. Badger, A. N. Hahmann, N. G. Mortensen, 2012: The selective dynamical downscaling method for extreme-wind atlases. Wind Energy, DOI: 10.1002/we.1544.

Leuenberger, D., F. Laudanna del Guerra, and A. M. Rossa, 2010: Application of an empirical quality function for radar QPE in an NWP model. ERAD 2010 - The Sixth European Conference on Radar in Meteorology and Hydrology, (Available at: http://www.erad2010.org/pdf/POSTER/Thursday/03_NWP/10_ERAD2010_0140_extended.pdf)

Lin, J., G. Cervone, and P. Franzese, 2010: Assessment of error in air quality models using dynamic time warping. DMG '10 Proceedings of the 1st ACM SIGSPATIAL International Workshop on Data Mining for Geoinformatics, 2 November 2010, San Jose, California, U.S.A., 38 - 44 (Available at: http://dl.acm.org/citation.cfm?id=1869895).

Lin, Y. and K. E. Mitchell, 2005: The NCEP Stage II/IV hourly precipitation analyses: Development and applications. 19th Conf on Hydrology, Amer. Meteorol. Soc. 1.2 (Available at http://www.emc.ncep.noaa.gov/mmb/ylin/pcpanl/refs/stage2-4.19hydro.pdf)

Liu, Y., J. Brown, J. Demargne, and D.-J. Seo, 2011: A wavelet-based approach to assessing timing errors in hydrologic predictions. J. Hydrol., 397 (3 - 4), 210 - 224.

Livina, V. N., N. R. Edwards, S. Goswami, and T. M. Lenton, 2008: A wavelet-coefficient score for comparison of two-dimensional climatic-data fields. Q. J. R. Meteorol. Soc., 134, 941 - 955.

M

Marzban, C., S. Sandgathe, 2006: Cluster analysis for verification of precipitation fields. Wea. Forecasting, 21 (5), 824 - 838.

Marzban, C., S. Sandgathe, 2008: Cluster Analysis for Object-Oriented Verification of Fields: A Variation. Mon. Wea. Rev., 136 (3), 1013 - 1025.

Marzban, C. and S. Sandgathe, 2009: Verification with variograms. Wea. Forecasting, 24 (4), 1102 - 1120.

Marzban, C. and S. Sandgathe, 2010: Optical flow for verification. Wea. Forecasting, 25, 1479 - 1494.

Marzban, C., S. Sandgathe, and H. Lyons, 2007: Assessment of an automatic, object-oriented approach to the verification of spatial fields. Paper presented at 7th Euopean Meteorological Society Annual Meeting, El Escorial, Spain, October. (Available at: http://faculty.washington.edu/marzban/)

Marzban, C., S. Sandgathe, and H. Lyons, 2008: An Object-oriented Verification of Three NWP Model Formulations via Cluster Analysis: An objective and a subjective analysis. Mon. Wea. Rev., 136, 3392 - 3407.

Marzban, C., S. Sandgathe, H. Lyons, and N. Lederer, 2009: Three Spatial Verification Techniques: Cluster Analysis, Variogram, and Optical Flow. Wea. Forecasting, 24 (6), 1457 - 1471, DOI: 10.1175/2009WAF2222261.1.

Marsigli, C., Boccanera, A. Montani, and T. Paccagnella, 2005: The COSMO-LEPS ensemble system: validation of the methodology and verification, Nonlinear Processes in Geophysics, 12, 527 - 536. (Available at http://www.nonlin-processes-geophys.net/12/issue4.html).

Marsigli, C., A. Montani, and T. Paccagnella, 2006: Verification of the COSMO-LEPS new suite in terms of precipitation distribution, COSMO Newsletter No. 6

Mass, C. F., D. Ovens, K. Westrick, and B. A. Colle, 2002: Does Increasing Horizontal Resolution Produce More Skillful Forecasts? Bulletin of the American Meteorological Society, 83, 407-430.

Melick, C. J., I. L. Jirak, A. R. Dean, J. Correia Jr., and S. J. Weiss, 2012: Real Time Objective Verification of Convective Forecasts: 2012 HWT Spring Forecast Experiment. Preprints, 37th Natl. Wea. Assoc. Annual Meeting, Madison, WI, Natl. Wea. Assoc., P1.52.

Mesinger, F., 2008: Bias adjusted precipitation threat scores. Adv. Geosciences 16, 137 - 142.

Micheas, A. C., N. I. Fox, S. A. Lack, and C. K. Wikle. 2007: Cell identification and verification of QPF ensembles using shape analysis techniques. J. of Hydrology, 343, 105 - 116.

Mittermaier, M.P., and R. Bullock, 2013: Using MODE to explore the spatial and temporal characteristics of cloud cover forecasts from high-resolution NWP models. Meteorol. Apps. (in press).

Mittermaier, M. P., 2012: A critical assessment of surface cloud observations and their use for verifying cloud forecasts. Quarterly Journal of the Royal Meteorological Society, DOI: 10.1002/qj.1918.

Mittermaier, M. P., 2008: Introducing uncertainty of radar-rainfall estimates to the verification of mesoscale model precipitation forecasts. Nat. Hazards Earth Syst. Sci. 8, 1 - 16

Mittermaier, M. P., 2007: Improving short-range high-resolution model precipitation forecast skill using time-lagged ensembles. Q. J. R. Meteorol. Soc. 133, 1487 - 1500

Mittermaier, M. P., 2006: Using an intensity-scale technique to assess the added benefit of high-resolution model precipitation forecasts. Atmos. Sci. Lett., 7(2), 35 - 42

Mittermaier, M.P. and N. Roberts, 2010: Intercomparison of Spatial Forecast Verification Methods: Identifying skillful spatial scales using the Fractions Skill Score. Wea. Forecasting 25 (1), 343 - 354, DOI: 10.1175/2009WAF2222260.1.

Mittermaier, M., N. Roberts, and S. A. Thompson, 2011: A long-term assessment of precipitation forecast skill using the Fractions Skill Score. Meteorol. Appl., DOI: 10.1002/met.296 (in press, available at Early View)

Moise, A. F. and F. P. Delage, 2011: New climate model metrics based on object-oriented pattern matching of rainfall. J. Geophys. Res., 116, D12108, 7 pp., DOI: 10.1029/2010JD015318

N - Q

Nachamkin, J.E., 2009: Application of the Composite Method to the Spatial Forecast Verification Methods Intercomparison Dataset. Wea. Forecasting, 24 (5), 1390 - 1400, DOI: 10.1175/2009WAF2222225.1.

Nachamkin, J. E., S. Chen, and J. S. Schmidt, 2005: Evaluation of heavy precipitation forecasts using composite-based methods: A distributions-oriented approach. Mon. Wea. Rev., 133, 2163 - 2177.

Nachamkin, J. E., 2004: Mesoscale verification using meteorological composites. Mon. Wea. Rev., 132, 941 - 955.

Nan, Z., S. Wang, X. Liang, T. E. Adams, W. Teng, Y. Liang, 2010: Analysis of Spatial Similarities Between NEXRAD and NLDAS Precipitation Data Products. IEEE J. Selected Topics in Applied Earth Observations and Remote Sensing, 3 (3), 371 - 385.

Nehrkorn, T., R. N. Hoffman, C. Grassotti, and J.-F. Louis. 2003: Feature calibration and alignment to represent forecast errors: Empirical regularization. Q.J.R. Meteorol. Soc., 129, 195 - 218. DOI: 10.1256/qj.02.18.

Prein, A. F. and A. Gobiet, 2011: NHCM-1: Non-hydrostatic climate modelling part I: Defining and detecting added value in cloud-resolving climate simulations. Scientific Report No. 39-2011, Wegener Center for Climate and Global Change, University of Graz, Graz, Austria, 69 pp. Available at: http://www.uni-graz.at/igam7www-wcv-scirep-no39-apreinagobiet-nhcm1-i-feb2011.pdf

R

Reilly, C., P. Price, A. Gelman, and S. A. Sandgathe, 2004: Using image and curve registration for measuring the goodness of fit of spatial and temporal predictions. Biometrics, 60, 954 - 964.

Rezacova, D., Z. Sokol, and P. Pesice, 2007: A radar-based verification of precipitation forecast for local convective storms. Atmos. Res., 83, 221 - 224.

Rezacova, D., P. Zacharov, and Z. Sokol, 2010: The use of radar data in the verification of high resolution prognostic rainfall fields. ERAD 2010 - The Sixth European Conference on Radar in Meteorology and Hydrology. (Available at http://www.erad2010.org/pdf/oral/friday/nowcasting/05_ERAD2010_0087.pdf)

Roberts, N. M. and H. W. Lean, 2008: Scale-selective verification of rainfall accumulations from high-resolution forecasts of convective events. Mon. Wea. Rev. , 136:78 - 96.

Roberts, N. M., 2005: An investigation of the ability of a storm-scale configuration of the Met Office NWP model to predict flood-producing rainfall. Forecasting Research Tech. Rept. 455, Met Office, 80 pp.

Roberts, R. D., A. R. S. Anderson, E. Nelson, B. G. Brown, J. W. Wilson, M. Pocernich, and T. Saxen, 2012: Impacts of forecaster involvement on convective storm initiation and evolution nowcasting. Wea. Forecasting, 27, 1061 - 1089, DOI: 10.1175/WAF-D-11-00087.1.

Rossa, A., G. Haase, C. Keil, P. Alberoni, S. Ballard, J. Bech, U. Germann, M. Pfeifer, and K. Salonen, 2010: Propagation of uncertainty from observing systems into NWP: COST-731 Working Group 1. Atmospheric Science Letters, 11 (2), 145 - 152.

Rossa, A., P. Nurmi and E. E. Ebert, 2008: Overview of methods for the verification of quantitative precipitation forecasts. Precipitation: Advances in Meas urement, Estimation and Prediction, Part III, 419 - 452, DOI: 10.1007/978-3-540-77655-0_16

S - T

Sampson, P. D. and P. Guttorp, 1992: Operational evaluation of air quality models. NRCSE Technical Report, NRCSE-TRS 018, National Research Center for Statistics and the Environment, Seattle, Washington, 22 pp., available at: http://www.nrcse.washington.edu/pdf/trs18_aqmodels.pdf

Sass, B. H. and Yang, X., 2012: A verification score for high resolution NWP: Idealized and preoperational tests. HIRLAM Technical Report, 69, 29 pp. (Available at: http://www.hirlam.org/index.php?option=com_content&view=article&id=114:hirlam-technical-report-no69&catid=60:announcements-for-publications&Itemid=65).

Schaffer, C. J., W. A. Gallus, Jr., and M. Segal, 2011: Improving Probabilistic Ensemble Forecasts of Convection through the Application of QPF-POP Relationships. Wea. Forecasting, 26, 319 - 336.

Schwedler, B.R.J. and M.E. Baldwin, 2011: Diagnosing the sensitivity of binary image measures to bias, location, and event frequency within a forecast verification framework. Wea. Forecasting, 26, 1032 - 1044.

Sobash, R. A., J. S. Kain, D. R. Bright, A. R. Dean, M. C. Coniglio, and S. J. Weiss, 2011: Probabilistic forecast guidance for severe thunderstorms based on the identification of extreme phenomena in convection-allowing model forecasts. Wea. Forecasting, 26, 714 - 728, DOI: 10.1175/WAF-D-10-05046.1.

Sokol, Z., D. Kitzmiller, P. Pesice, and J. Mejsnar, 2012: Comparison of precipitation nowcasting by extrapolation and statistical-advection methods. Atmos. Res., (in press; available at http://www.sciencedirect.com/science/article/pii/S0169809512003390#).

Tafferner, A., C. Forster, M. Hagen, C. Keil, T. Zinner, and H. Volkert, 2008: Development and propagation of severe thunderstorms in the Upper Danube catchment area: Towards an integrated nowcasting and forecasting system using real-time data and high-resolution simulations. Meteorol. Atmos. Phys., 101, 211 - 227, DOI 10.1007/s00703-008-0322-7

Theis, S. E., A. Hense, and U. Damrath, 2005: Probabilistic precipitation forecasts from a deterministic model: A pragmatic approach. Meteorol. Appl., 12, 257 - 268.

Trentmann, J., C. Keil, M. Salzmann, C. Barthlott, H.-S. Bauer, T. Schwitalla, M. G. Lawrence, D. Leuenberger, V. Wulfmeyer, U. Corsmeier, C. Kottmeier, and H. Wernli, 2009: Multi-model simulations of a convective situation in low-mountain terrain in central Europe. Meteorol. Atmos. Phys., 103, 95 - 103, DOI 10.1007/s00703-008-0323-6

Tustison, B., D. Harris and E. Foufoula-Georgiou, 2001: Scale issues in verification of precipitation forecasts. J. of Geophysical Res. 106 (D11), 11,775 - 11,784.

U - V

Venugopal, V., S. Basu, E. Foufoula-Georgiou, 2005: A new metric for comparing precipitation patterns with an application to ensemble forecasts. J. Geophys. Res. 110: D8, D08111 10.1029/2004JD005395.

Vich, M., R. Romero, and H. E. Brooks, 2011: Ensemble prediction of Mediterranean high-impact events using potential vorticity perturbations. Part I: Comparison against the multiphysics approach. Atmospheric Research, 102, (1 - 2), October, 227 - 241. DOI: 10.1016/j.atmosres.2011.07.017

W

Wang, C.-A., 2012: The impact of GPS radio occultation on the prediction of extreme rainfall associated with Typhoon Morakot (2009): Precipitation forecast verification. Significant Opportunities in Atmospheric Research and Science (SOARS) Program Report, 16pp. (pdf).

Wapler, K., M. Goeber and S. Trepte, 2012: Comparative verification of different nowcasting systems to support optimisation of thunderstorm warnings. Adv. Sci. Res., 8, 121 - 127.

Wernli, H., C. Hofmann, and M. Zimmer, 2009: Spatial forecast verification methods intercomparison project: Application of the SAL technique. Wea. Forecasting, 24 (6), 1472 - 1484, DOI: 10.1175/2009WAF2222271.1.

Wernli, H., M. Paulat, M. Hagen and C. Frei, 2008: SAL--a novel quality measure for the verification of quantitative precipitation forecasts. Mon. Wea. Rev., 136, 4470 - 4487, DOI: 10.1175/2008MWR2415.1.

Weusthoff, T., F. Ament, M. Arpagaus, M. W. Rotach, 2010: Assessing the Benefits of Convection-Permitting Models by Neighborhood Verification: Examples from MAP D-PHASE. Mon. Wea. Rev., 138, 3418 - 3433. DOI: 10.1175/2010MWR3380.1

Weygandt, S. S., A. F. Loughe, S. G. Benjamin, and J. L. Mahoney, 2004: Scale sensitivities in model precipitation skill scores during IHOP. 22nd Conf. Severe Local Storms, Amer. Met. Soc., 4-8 October 2004, Hyannis, M.A.

Willmott, C. J., K. Matsuura, and S. M. Robeson, 2009: Ambiguities inherent in sums-of-squares-based error statistics. Atmospheric Environment, 43, 749 - 752.

Willmott, C. J., S. M. Robeson and K. Matsuura, 2007: Geographic box plots. Physical Geography, 28 (4), 331 - 344.

Willmott, C. J. and K. Matsuura, 2006: On the use of dimensioned measures of error to evaluate the performance of spatial interpolators. International Journal of Geographical Information Science, 20 (1), 89 - 102

Willmott, C. J. and K. Matsuura, 2005: Advantages of the mean absolute error (MAE) over the root mean square error (RMSE) in assessing average model performance. Climate Research, 30, 79 - 82

X - Z

Yates, E. S., S. Anquetin, V. Ducrocq, J.-D. Creutin, D. Richard, and K. Chancibault, 2006: Point and areal validation of forecast precipitation fields. Meteorol. Appl., 13, 1 - 20

Zepeda-Arce, J., E. Foufoula-Georgiou, and K. K. Droegemeier, 2000: Space-time rainfall organization and its role in validating quantitative precipitation forecasts. J. Geophys. Res., 105 (D8), 10,129 - 10,146

Zhu, M., V. Lakshmanan, P. Zhang, Y. Hong, K. Cheng, and S. Chen, 2011: Spatial verification using a true metric. Atmos. Res., 102 (4), 408 - 419, DOI: 10.1016/j.atmosres.2011.09.004

Ziegeler, S. B., J. D. Dykes, J. F. Shriver, 2012: Spatial error metrics for oceanographic model verification. J. Atmos. Oceanic Technol., 29, 260 - 266. doi:http://dx.doi.org/10.1175/JTECH-D-11-00109.1

Zimmer, M. and H. Wernli, 2011: Verification of quantitative precipitation forecasts on short time-scales: A fuzzy approach to handle timing errors with SAL. Meteorol. Z., 20 (2), 095 - 105.

Contact Information

ICP2 Committee Members

Eric Gilleland (webmaster)

Marion Mittermaier (marion.mittermaier " at " metoffice.gov.uk)

Barb Brown

Manfred Dorninger

Beth Ebert

Laurie Wilson

The list of participants under ICP1 below was created post priori from the list of authors of papers in the ICP special collection of Weather and Forecasting. Please contact us with any errors.

Once the ICP2 gets up and rolling, we will create a new list for the second iteration of the project.

ICP1 Participants

David A. Ahijevych	Keith F. Brill	Barbara G. Brown
Randy G. Bullock	Barbara Casati	George C. Craig
Christopher A. Davis	Elizabeth E. Ebert	Neil I. Fox
Eric Gilleland	John Halley Gotway	Christiane Hofmann
John S. Kain	Christian Keil	Steven A. Lack
Valliappa Lakshmanan	Nicholas Lederer	George L. Limpert
Finn Lindgren	Johan Lindström	Hilary Lyons
Caren Marzban	Fedor Mesinger	Marion P. Mittermaier
Jason E. Nachamkin	Nigel Roberts	Scott Sandgathe
Heini Wernli	Matthias Zimmer

ICP2 Participants

Subscribe/Unsubscribe from the project email list

Subscribe

To subscribe to the email list, please do both of the following:

(1) Send an email to Eric Gilleland giving your name, institution, contact information, and whether or not you plan to participate in ICP2, and if not, why you want to be on the list.

(2) Using your same email address from step (1), sign up here.^*

Note: the reason for step (1) above is to verify who you are in order to protect against spam. If your identity cannot be easily verified, your subscription request will be ignored.

Unsubscribe

To unsubscribe, go to the same web page, and follow the instructions there to unsubscribe. No need to send Eric an email to unsubscribe.