Spatial Forecast Verification Methods Inter-Comparison Project
References
This is by no means an exhaustive list of references, but should provide a good background to the issues concerning the ICP, and all of the methods presently being compared; among others. See also the special collection of ICP papers for WAF, none of which are listed on the present page. If you know of a relevant paper that should be listed here, please contact Eric Gilleland.
All of the references below, and those for the special collection are also available in LaTeX from here spatialvx.bib.
AghaKouchak, A., N. Nasrollahi, J. Li, B. Imam, and S. Sorooshian, 2011: Geometrical characterization of precipitation patterns. J. Hydrometeorology, 12, 274--285, DOI: 10.1175/2010JHM1298.1
AghaKouchak, A., A. Behrangi, S. Sorooshian, K. Hsu, and E. Amitai, 2011: Evaluation of satellite-retrieved extreme precipitation rates across the central United States. J. Geophys. Res., 116, D02115, 11 pp., doi:10.1029/2010JD014741.
Alexander, G. D. , J. A. Weinman, V. M. Karyampudi, W. S. Olson, and A. C. L. Lee, 1999: The effect of assimilating rain rates derived from satellites and lightning on forecasts of the 1993 superstorm. Mon. Wea. Rev., 127, 1433--1457.
Alexander, G. D., J. A. Weinman, and J. L. Schols, 1998: The use of digital image warping of microwave integrated water vapor imagery to improve forecasts of marine extratropical cyclones. Mon. Wea. Rev., 126, 1469--1496.
Argence, S., D. Lambert, E. Richard, J.-P. Chaboureau, N. Söhne, 2008: Impact of initial condition uncertainties on the predictability of heavy rainfall in the Mediterranean: a case study. Q.J.R. Meteorol. Soc., 134 (636), 1775--1788.
Atger, F., 2001: Verification of intense precipitation forecasts from single models and ensemble prediction systems. Nonlin. Proc. Geophys., 8, 401--417
Baldwin, M. E. and J. S. Kain, 2006: Sensitivity of several performance measures to displacement error, bias, and event frequency. Wea. Forecasting, 21, 636--648
Basu, S., B. Dodov, and E. Foufoula-Georgiou, 2003: A novel measure for QPF verification and its usefulness in multimodel ensemble forecasting, Geophys. Res. Abstracts, 5, 04323, European Geophysical Society.
Bauer, H.-S., T. Weusthoff, M. Dorninger, V. Wulfmeyer, T. Schwitalla, T. Gorgas, M. Arpagaus, and K. Warrach-Sagi, 2011: Predictive skill of a subset of models participating in D-PHASE in the COPS region. Q.J.R. Meteorol. Soc., 137, 287--305.
Briggs, W. M. and R. A. Levine, 1997: Wavelets and field forecast verification. Mon. Wea. Rev., 125, 1329--1341.
Brooks, H. E., M. Kay and J. A. Hart, 1998: Objective limits on forecasting skill of rare events. 19th Conf. Severe Local Storms, Amer. Met. Soc., 552--555
Brown, B.G., Gilleland, E. and Ebert, E.E., 2012: Chapter 6: Forecasts of spatial fields. pp. 95--117, In "Forecast Verification: A Practitioner's Guide in Atmospheric Science", 2nd edition. Edts. IT Jolliffee and DB Stephenson, Wiley, Chichester, West Sussex, UK, 274 pp.
Casati, B., G. Ross, and D. B. Stephenson, 2004: A new intensity-scale approach for the verification of spatial precipitation forecasts. Meteorol. Appl. 11, 141--154.
Casati, B., L. J. Wilson, D. B. Stephenson, P. Nurmi, A. Ghelli, M. Pocernich, U. Damrath, E. E. Ebert, B. G. Brown, and S. Mason, 2008: Forecast verification: Current status and future directions. Meteorol. Appl., 15, 3--18.
Clark, A. J., W. A. Gallus, Jr. and M. L. Weisman, 2010: Neighborhood-Based Verification of Precipitation Forecasts from Convection-Allowing NCAR WRF Model Simulations and the Operational NAM. Wea. Forecasting, 25, 1495-1509. doi: 10.1175/2010WAF2222404.1
Clark, A. J., J. S. Kain, P. T. Marsh, J. Correia, Jr., M. Xue, and F. Kong, 2010: Forecasting tornado path lengths using a 3-dimensional object identification algorithm applied to convection-allowing forecasts. Submitted to Wea. Forecasting.
Damrath, U., 2004: Verification against precipitation observations of a high density network--what did we learn? Intl. Verification Methods Workshop, 15-17 September 2004, Montreal, Canada.
Davis, C. A., B. G. Brown, and R. G. Bullock, 2006a: Object-based verification of precipitation forecasts, Part I: Methodology and application to mesoscale rain areas. Mon. Wea. Rev. 134, 1772--1784.
Davis, C. A., B. G. Brown, and R. G. Bullock, 2006b: Object-based verification of precipitation forecasts, Part II: Application to convective rain systems. Mon. Wea. Rev. 134, 1785--1795.
De Sales, F. and Y. Xue, 2010: Assessing the dynamic-downscaling ability over South America using the intensity-scale verification technique. Int. J. Climatol., DOI: 10.1002/joc.2139
Demaria, E. M., D. A. Rodriguez, E. E. Ebert, P. Salino, F. Su, and J. B. Valdes, 2011: Evaluation of mesoscale convective systems in South America using multiple satellite products and an object-based approach. J. Geophys. Res., 116, D08103, 13 pp., doi:10.1029/2010JD015157
Ebert, E. E., 2008: Fuzzy verification of high resolution gridded forecasts: A review and proposed framework. Meteorol. Appl., 15, 51--64. DOI: 10.1002/met.25 (Available at http://www.ecmwf.int/newsevents/meetings/workshops/2007/jwgv/METspecialissueemail.pdf)
Ebert, E. E. and J. L. McBride, 2000: Verification of precipitation in weather systems: determination of systematic errors. J. Hydrology, 239, 179--202.
Elmore, K. L., M. E. Baldwin, and D. M. Schultz, 2006: Field significance revisited: Spatial bias errors in forecasts as applied to the Eta model. Mon. Wea. Rev., 134, 519--531.
Gallus, W. A. Jr., 2010: Application of Object-Based Verification Techniques to Ensemble Precipitation Forecasts. Wea. Forecasting, 25, 144--158.
Germann U and I Zawadzki, 2004. Scale dependence of the predictability of precipitation from continental radar images. Part II: Probability forecasts. J. Appl. Meteorol., 43, 74--89
Gilleland, E., 2011: Spatial Forecast Verification: Baddeley's Delta Metric Applied to the ICP Test Cases. Wea. Forecasting, 26 (3), 409--415.
Gilleland, E., L. Chen, M. DePersio, G. Do, K. Eilertson, Y. Jin, E.L. Kang, F. Lindgren, J. Lindström, R.L. Smith, and C. Xia, 2011: Spatial Forecast Verification: Image Warping. NCAR Technical Note, TN-482+STR, 23pp.
Gilleland, E., T. C. M. Lee, J. Halley Gotway, R. G. Bullock, and B. G. Brown, 2008: Computationally efficient spatial forecast verification using Baddeley's Δ image metric. Mon. Wea. Rev. 136 (5), 1747--1757.
Grams, J. S., W. A. Gallus Jr., S. E. Koch, L. S. Wharton, A. Loughe, and E. E. Ebert, 2006: The Use of a Modified Ebert-McBride Technique to Evaluate Mesoscale Model QPF as a Function of Convective System Morphology during IHOP 2002. Wea. Forecasting, 21, 288--306.
Haiden, T., A. Kann, C. Wittmann, G. Pistotnik, B. Bica, and C. Gruber, 2011: The Integrated Nowcasting through Comprehensive Analysis (INCA) system and its validation over the eastern alpine region. Wea. Forecasting, 26, 166--183, doi: 10.1175/2010WAF2222451.1
Harris, D., E. Foufoula-Georgiou, K. K. Droegemeier, J. J. Levit, 2001: Multiscale statistical properties of a high-resolution precipitation forecast. J. Hydrometeorology 2, 406--418.
Hartung, D. C., J. A. Otkin, R. A. Petersen, D. D. Turner, and W. F. Feltz, 2011: Assimilation of surface-based boundary layer profiler observations during a cool-season weather event using an observing system simulation experiment. Part II: Forecast assessment. Mon. Wea. Rev., 139, 2327--2346. DOI: 10.1175/2011MWR3623.1
Herbort, F. and E. Etling, 2011: Post-frontal shower cells in the COSMO-DE model: A comparison with radar measurements. Meteorol. Z., 20 (2), 217--226.
Hering, A. S. and M. G. Genton, 2011: Comparing spatial predictions. Accepted to Technometrics, DOI 10.1198/TECH.2011.10136
Hoffman, R. N. and C. Grassotti, 1996: A technique for assimilating SSM/I observations of marine atmospheric storms: Tests with ECMWF analyses. J. Appl. Meteorol., 35, 1177--1188.
Hoffman, R. N., Z. Liu, J.-F. Louis, and C. Grassotti, 1995: Distortion representation of forecast errors. Mon. Wea. Rev., 123, 2758--2770.
Johnson, A., X. Wang, F. Kong, and M. Xue, 2011: Hierarchical cluster analysis of a convection-allowing ensemble during the Hazardous Weather Testbed 2009 Spring Experiment. Part I: Development of object-oriented cluster analysis method for precipitation fields. Mon. Wea. Rev., 139, (12), 3673--3693.
Jury, M. R., 2011: Representation of the Caribbean mean diurnal cycle in observation, reanalysis, and CMIP3 model datasets. Theor. Appl. Climatol., 12pp., DOI 10.1007/s00704-011-0462-4.
Kain, J. S., S. J. Weiss, M. E. Baldwin, G. W. Carbin, D. R. Bright, J. J. Levit, and J. A. Hart, 2005: Evaluating high-resolution configurations of the WRF model that are used to forecast severe convective weather: The 2005 SPC/NSSL Spring Program. Preprints, 21th Conference on Weather Analysis and Forecasting/17th Conference on Numerical Weather Prediction, Washington, D. C., Amer. Meteor. Soc., CD-ROM, 2A.5.
Kanamitsu, M. and L. DeHaan, 2011: The Added Value Index: A new metric to quantify the added value of regional models. J. Geophys. Res., 116, D11106, 10 pp., doi:10.1029/2011JD015597
Keil, C. and G. C. Craig, 2007: A displacement-based error measure applied in a regional ensemble forecasting system. Mon. Wea. Rev., 135, 3248--3259.
Leuenberger, D., F. Laudanna del Guerra, and A. M. Rossa, 2010: Application of an empirical quality function for radar QPE in an NWP model. ERAD 2010 - The Sixth European Conference on Radar in Meteorology and Hydrology, (Available at: http://www.erad2010.org/pdf/POSTER/Thursday/03_NWP/10_ERAD2010_0140_extended.pdf)
Lin, J., G. Cervone, and P. Franzese, 2010: Assessment of error in air quality models using dynamic time warping. DMG '10 Proceedings of the 1st ACM SIGSPATIAL International Workshop on Data Mining for Geoinformatics, 2 November 2010, San Jose, California, U.S.A., 38--44 (Available at: http://dl.acm.org/citation.cfm?id=1869895).
Lin, Y. and K. E. Mitchell, 2005: The NCEP Stage II/IV hourly precipitation analyses: Development and applications. 19th Conf on Hydrology, Amer. Meteorol. Soc. 1.2 (Available at http://www.emc.ncep.noaa.gov/mmb/ylin/pcpanl/refs/stage2-4.19hydro.pdf)
Liu, Y., J. Brown, J. Demargne, and D.-J. Seo, 2011: A wavelet-based approach to assessing timing errors in hydrologic predictions. J. Hydrol., 397 (3--4), 210--224.
Marzban, C., S. Sandgathe, 2006: Cluster analysis for verification of precipitation fields. Wea. Forecasting, 21 (5), 824--838.
Marzban, C., S. Sandgathe, 2008: Cluster Analysis for Object-Oriented Verification of Fields: A Variation. Mon. Wea. Rev., 136 (3), 1013--1025.
Marzban, C. and S. Sandgathe, 2009: Verification with variograms. Wea. Forecasting, 24 (4), 1102--1120.
Marzban, C. and S. Sandgathe, 2010: Optical flow for verification. Wea. Forecasting, 25, 1479--1494.
Marzban, C., S. Sandgathe, and H. Lyons, 2007: Assessment of an automatic, object-oriented approach to the verification of spatial fields. Paper presented at 7th Euopean Meteorological Society Annual Meeting, El Escorial, Spain, October. (Available at: http://faculty.washington.edu/marzban/)
Marzban, C., S. Sandgathe, and H. Lyons, 2008: An Object-oriented Verification of Three NWP Model Formulations via Cluster Analysis: An objective and a subjective analysis. Mon. Wea. Rev., 136, 3392--3407.
Marsigli, C., Boccanera, A. Montani, and T. Paccagnella, 2005: The COSMO-LEPS ensemble system: validation of the methodology and verification, Nonlinear Processes in Geophysics, 12, 527--536. (Available at http://www.nonlin-processes-geophys.net/12/issue4.html).
Marsigli, C., A. Montani, and T. Paccagnella, 2006: Verification of the COSMO-LEPS new suite in terms of precipitation distribution, COSMO Newsletter No. 6
Mass, C.F., D. Ovens, K. Westrick, and B.A. Colle, 2002: Does Increasing Horizontal Resolution Produce More Skillful Forecasts? Bulletin of the American Meteorological Society, 83, 407-430.
Mesinger, F., 2007: Bias adjusted precipitation threat scores. Adv. Geosciences 16, 137--143.
Micheas, A. C., N. I. Fox, S. A. Lack, and C. K. Wikle. 2007: Cell identification and verification of QPF ensembles using shape analysis techniques. J. of Hydrology, 343, 105--116.
Mittermaier, M. P., 2008: Introducing uncertainty of radar-rainfall estimates to the verification of mesoscale model precipitation forecasts. Nat. Hazards Earth Syst. Sci. 8, 1--16
Mittermaier, M. P., 2007: Improving short-range high-resolution model precipitation forecast skill using time-lagged ensembles. Q. J. R. Meteorol. Soc. 133, 1487--1500
Mittermaier, M. P., 2006: Using an intensity-scale technique to assess the added benefit of high-resolution model precipitation forecasts. Atmos. Sci. Lett., 7(2), 35--42
Mittermaier, M., N. Roberts, and S. A. Thompson, 2011: A long-term assessment of precipitation forecast skill using the Fractions Skill Score. Meteorol. Appl., DOI: 10.1002/met.296 (in press, available at Early View)
Moise, A. F. and F. P. Delage, 2011: New climate model metrics based on object-oriented pattern matching of rainfall. J. Geophys. Res., 116, D12108, 7 pp., doi:10.1029/2010JD015318
Nachamkin, J. E., S. Chen, and J. S. Schmidt, 2005: Evaluation of heavy precipitation forecasts using composite-based methods: A distributions-oriented approach. Mon. Wea. Rev., 133, 2163--2177.
Nachamkin, J. E., 2004: Mesoscale verification using meteorological composites. Mon. Wea. Rev., 132, 941--955.
Nan, Z., S. Wang, X. Liang, T. E. Adams, W. Teng, Y. Liang, 2010: Analysis of Spatial Similarities Between NEXRAD and NLDAS Precipitation Data Products. IEEE J. Selected Topics in Applied Earth Observations and Remote Sensing, 3 (3), 371--385.
Nehrkorn, T., R. N. Hoffman, C. Grassotti, and J.-F. Louis. 2003: Feature calibration and alignment to represent forecast errors: Empirical regularization. Q.J.R. Meteorol. Soc., 129, 195--218. doi: 10.1256/qj.02.18.
Prein, A. F. and A. Gobiet, 2011: NHCM-1: Non-hydrostatic climate modelling part I: Defining and detecting added value in cloud-resolving climate simulations. Scientific Report No. 39-2011, Wegener Center for Climate and Global Change, University of Graz, Graz, Austria, 69 pp. Available at: http://www.uni-graz.at/igam7www-wcv-scirep-no39-apreinagobiet-nhcm1-i-feb2011.pdf
Rezacova, D., Z. Sokol, and P. Pesice, 2007: A radar-based verification of precipitation forecast for local convective storms. Atmos. Res., 83, 221--224.
Rezacova, D., P. Zacharov, and Z. Sokol, 2010: The use of radar data in the verification of high resolution prognostic rainfall fields. ERAD 2010 - The Sixth European Conference on Radar in Meteorology and Hydrology. (Available at http://www.erad2010.org/pdf/oral/friday/nowcasting/05_ERAD2010_0087.pdf)
Roberts, N. M. and H. W. Lean, 2008: Scale-selective verification of rainfall accumulations from high-resolution forecasts of convective events. Mon. Wea. Rev. , 136:78--96.
Roberts, N. M., 2005: An investigation of the ability of a storm-scale configuration of the Met Office NWP model to predict flood-producing rainfall. Forecasting Research Tech. Rept. 455, Met Office, 80 pp.
Rossa, A., G. Haase, C. Keil, P. Alberoni, S. Ballard, J. Bech, U. Germann, M. Pfeifer, and K. Salonen, 2010: Propagation of uncertainty from observing systems into NWP: COST-731 Working Group 1. Atmospheric Science Letters, 11 (2), 145--152.
Rossa, A., P. Nurmi and E. E. Ebert, 2008: Overview of methods for the verification of quantitative precipitation forecasts. Precipitation: Advances in Meas urement, Estimation and Prediction, Part III, 419--452, DOI: 10.1007/978-3-540-77655-0_16
Schaffer, C. J., W. A. Gallus, Jr., and M. Segal, 2011: Improving Probabilistic Ensemble Forecasts of Convection through the Application of QPF-POP Relationships. Wea. Forecasting, 26, 319--336.
Schwedler, B.R.J. and M.E. Baldwin, 2011: Diagnosing the sensitivity of binary image measures to bias, location, and event frequency within a forecast verification framework. Wea. Forecasting, 26, 1032--1044.
Sobash, R. A., J. S. Kain, D. R. Bright, A. R. Dean, M. C. Coniglio, and S. J. Weiss, 2011: Probabilistic forecast guidance for severe thunderstorms based on the identification of extreme phenomena in convection-allowing model forecasts. Wea. Forecasting, 26, 714--728, DOI: 10.1175/WAF-D-10-05046.1.
Tafferner, A., C. Forster, M. Hagen, C. Keil, T. Zinner, and H. Volkert, 2008: Development and propagation of severe thunderstorms in the Upper Danube catchment area: Towards an integrated nowcasting and forecasting system using real-time data and high-resolution simulations. Meteorol. Atmos. Phys., 101, 211--227, DOI 10.1007/s00703-008-0322-7
Theis, S. E., A. Hense, and U. Damrath, 2005: Probabilistic precipitation forecasts from a deterministic model: A pragmatic approach. Meteorol. Appl., 12, 257--268.
Trentmann, J., C. Keil, M. Salzmann, C. Barthlott, H.-S. Bauer, T. Schwitalla, M. G. Lawrence, D. Leuenberger, V. Wulfmeyer, U. Corsmeier, C. Kottmeier, and H. Wernli, 2009: Multi-model simulations of a convective situation in low-mountain terrain in central Europe. Meteorol. Atmos. Phys., 103, 95--103, DOI 10.1007/s00703-008-0323-6
Tustison, B., D. Harris and E. Foufoula-Georgiou, 2001: Scale issues in verification of precipitation forecasts. J. of Geophysical Res. 106 (D11), 11,775--11,784.
Venugopal, V., S. Basu, E. Foufoula-Georgiou, 2005: A new metric for comparing precipitation patterns with an application to ensemble forecasts. J. Geophys. Res. 110: D8, D08111 10.1029/2004JD005395.
Vich, M., R. Romero, and H. E. Brooks, 2011: Ensemble prediction of Mediterranean high-impact events using potential vorticity perturbations. Part I: Comparison against the multiphysics approach. Atmospheric Research, 102, (1--2), October, 227--241. doi:10.1016/j.atmosres.2011.07.017
Wernli, H., M. Paulat, M. Hagen and C. Frei, 2008: SAL--a novel quality measure for the verification of quantitative precipitation forecasts. Mon. Wea. Rev., 136, 4470--4487, doi: 10.1175/2008MWR2415.1.
Weusthoff, T., F. Ament, M. Arpagaus, M. W. Rotach, 2010: Assessing the Benefits of Convection-Permitting Models by Neighborhood Verification: Examples from MAP D-PHASE. Mon. Wea. Rev., 138, 3418--3433. doi: 10.1175/2010MWR3380.1
Weygandt, S. S., A. F. Loughe, S. G. Benjamin, and J. L. Mahoney, 2004: Scale sensitivities in model precipitation skill scores during IHOP. 22nd Conf. Severe Local Storms, Amer. Met. Soc., 4-8 October 2004, Hyannis, M.A.
Willmott, C. J., K. Matsuura, and S. M. Robeson, 2009: Ambiguities inherent in sums-of-squares-based error statistics. Atmospheric Environment, 43, 749--752.
Willmott, C. J., S. M. Robeson and K. Matsuura, 2007: Geographic box plots. Physical Geography, 28 (4), 331--344.
Willmott, C. J. and K. Matsuura, 2006: On the use of dimensioned measures of error to evaluate the performance of spatial interpolators. International Journal of Geographical Information Science, 20 (1), 89--102
Willmott, C. J. and K. Matsuura, 2005: Advantages of the mean absolute error (MAE) over the root mean square error (RMSE) in assessing average model performance. Climate Research, 30, 79--82
Yates, E. S., S. Anquetin, V. Ducrocq, J.-D. Creutin, D. Richard, and K. Chancibault, 2006: Point and areal validation of forecast precipitation fields. Meteorol. Appl., 13, 1--20
Zepeda-Arce, J., E. Foufoula-Georgiou, and K. K. Droegemeier, 2000: Space-time rainfall organization and its role in validating quantitative precipitation forecasts. J. Geophys. Res., 105 (D8), 10,129--10,146
Zhu, M., V. Lakshmanan, P. Zhang, Y. Hong, K. Cheng, and S. Chen, 2011: Spatial verification using a true metric. Atmos. Res., 102 (4), 408--419, doi:10.1016/j.atmosres.2011.09.004
Zimmer, M. and H. Wernli, 2011: Verification of quantitative precipitation forecasts on short time-scales: A fuzzy approach to handle timing errors with SAL. Meteorol. Z., 20 (2), 095--105.
The National Center for Atmospheric Research is sponsored by the National Science Foundation. Any opinions, findings, and conclusions or recommendations expressed in this publication are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.
