Spatial Forecast Verification Methods Inter-Comparison Project
We will provide links to available software for performing spatial forecast verification from this page as they become available. All references listed below can be found in-full by clicking on the references link above.
SpatialVx
A new R package called SpatialVx is in the works. It already has functions to do many of the newly proposed methods, but there are many yet to do. Keep checking for new updates.
Traditional Verification
The book:
Jolliffe, I.T. and Stephenson, D.B., Forecast Verification: A practioner's guide in atmospheric sciences, Second Edition, Wiley-Balckwell, Chichester, West Sussex, U.K., 274 pp.
has a nice appendix by Matt Pocernich on verification software (pp. 231--240). Also see Forecast Verification Issues, Methods and FAQ. From these, it is clear that several software tools are available. Listed here are only tools that were developed at NCAR.
R package verification by Matt Pocernich. From your R session, you can install the package onto your machine with the lib.loc function. You can then load it into your R session using the library function.
Model Evaluation Tools (MET) user's home page
SpatialVx contains some functionality for doing the traditional forecast verification methods.
Features-based Approaches
Contiguous Rain Area (CRA) (Ebert and McBride, 2000). IDL code from Beth Ebert available at http://www.cawcr.gov.au/staff/eee/index.php
Method for Object-based Diagnostic Evaluation (MODE) tool (Davis et al., 2006ab). Available through MET user's home page.
SpatialVx has some functionality for doing the method proposed by Davis et al. (2006a), as well as the SAL technique (Wernli et al., 2008). In particular, it is possible to do either convolution thresholding, or just thresholding to identify features (or objects) as connected grid points above the threshold. It also has functions to calculate properties of single features (major/minor axis angle/length, aspect ratio, area, centroid) and properties for matched features (e.g., intersection area, area ratio, centroid distance, angle difference, as well as numerous binary image metrics; see below). More is on tap for future releases.
Field Deformation Approaches
Baddeley's Delta Metric: R code for computing Baddeley's delta metric for binary images (see, e.g., Gilleland, 2011)
FQI (Venugopal et al, 2005; Basu et al, 2003)
Image Warping: an R package to do the image warping found in Gilleland, Lindström and Lindgren (2010) is coming soon!. The original MatLab code will be replaced by these R functions, and they will be available through the SpatialVx package.
SpatialVx has functions to calculate FQI, Baddeley's Delta Metric, the Hausdorff metric, partial Hausdorff measure, mean error distance, mean square error distance, Pratt's Figure of Merit (FOM), minimum separation distance, (see the help file for locperf for references).
Neighborhood-Based Approaches (summarized in Ebert, 2008)
IDL code from Beth Ebert available at http://www.cawcr.gov.au/staff/eee/index.php
The R package SpatialVx has functions for doing most of the neighborhood methods.
Scale Decomposition Approaches
Intensity-Scale (IS) (Casati et al, 2004). Available in the R package verification (above under traditional verification). Also available from the Model Evaluation Tools (MET) software.
SpatialVx has the wavelet denoising (more a neighborhood type approach), and wavelet decomposition approaches of Briggs and Levine (1997), as well as the intensity-scale technique of Casati et al. (2004) and Casati (2009).
The National Center for Atmospheric Research is sponsored by the National Science Foundation. Any opinions, findings, and conclusions or recommendations expressed in this publication are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.
