- Test Cases
- MesoVICT Papers
- Working Groups
MesoVICT: Mesoscale Verification Intercomparison over Complex Terrain (Phase 2 of the ICP)
About the ICP and MesoVICT
The spatial forecast verification inter-comparison project (ICP) was set up to attempt to sift through the maze of newly proposed methods for verifying primarily high-resolution forecasts. MesoVICT continues this aim with more meteorologically complex test cases including: more complex terrain, gridded as well as point observations, ensembles of forecasts, ensembles of observations, more variables in addition to precipitation (e.g. winds), as well as meteorological events that occur over a period of time (i.e., adding a short time dimension to the question).
The intent of this project is to compare the various newly proposed methods to give the user information about which methods are appropriate for which types of data, forecasts and desired forecast utility. The point is not to determine which method is best. However, it is hoped that advantages and disadvantages of the methods will become clear.
The MesoVICT project, like its predecessor (ICP), is a community project. It is not a model inter-comparison, but a verification methods (meta-verification) inter-comparison.
If you are interested in participating in the second phase of the ICP (to be called "Mesocale Verification Intercomparison in Complex Terrain", or MesoVICT for short), please click on the Email list tab to subscribe. You will then be kept informed of activities.
Registration is now open for the Workshop on Verification Inter-Comparison, which will be held 21 - 23 September 2016 in Bologna, Italy. Abstract submission deadline is 30 June 2016, and registration deadline is 1 September 2016.
There will also be a MesoVICT session (NWP7) at the 16th EMS Annual Meeting and 11th European Conference on Applied Climatology (ECAC) in Trieste, Italy, which takes place 12 - 16 September 2016.
The original ICP phase 1 test cases (as well as some extras) can be found further below.
For further model data that go beyond the MesoVICT project please visit the World Data Center for Climate in Hamburg, Germany, where all model output (and other data) from the DPHASE project are stored.
For groups wishing to provide model re-runs of the test cases below (i.e., test newer model versions on these dates), please contact the organizing committee. Note that our purpose is not to verify your models (this is a meta-verification project), but it may be possible to work with individual participants who might be interested in doing so. In such cases, it will not be possible to store the data on this website, but we will be happy to provide a link to the data.
Please refer to the Technical Note for the project to learn about the test cases for MesoVICT. Note that COSMO2 06 run is not available for the core case, so we will use the 00 run for the core case (for this model only) instead; the CMC GEMH is only available at 06, which is why we chose to use the 06 runs generally. For all other cases, we will use the 06 runs. See also Bauer et al. (2011) for more information about the data sets generally.
The figure below shows a schematic diagram of the tiered case structure for the project.
Vienna Enhanced Resolution Analysis (VERA)
The six VERA test case analyses are each available in a separate zip file linked below:
- VERA_case1.zip (126 MB)
- VERA_case2.zip (167 MB)
- VERA_case3.zip (169 MB)
- VERA_case4.zip (125 MB)
- VERA_case5.zip (43 MB)
- VERA_case6.zip (127 MB)
Point Observations: Joint D-PHASE - COPS (JDC) data set
Description of the JDC cases file format (pdf).
The six JDC test case point observation files are linked below:
- JDC_case1.zip (26 MB)
- JDC_case2.zip (35 MB)
- JDC_case3.zip (36 MB)
- JDC_case4.zip (26 MB)
- JDC_case5.zip (8.8 MB)
- JDC_case6.zip (25 MB)
- statlists.zip (3.3 MB)
Description of model output file format for all models (pdf).
Note that COSMO2 06 hour runs are not available for cases 1 and 6. Because case 1 is a core case, please use COSMO2 00 hour run for this case instead; as though it were the 06 hour run.
MeteoSwiss plans to recalculate these cases with the current version of their models (experimental COSMO-1, operational COSMO-2, and COSMO-7, which have grid meshes of 1.1, 2.2 and 6.6 km, respectively). These re-run cases should be available in the coming months.
Canadian high-resolution model: CMC GEMH (GEM-LAM) model output from Environment Canada
- CMH_06_case1.tar.gz (52 MB)
- CMH_06_case2.tar.gz (69 MB)
- CMH_06_case3.tar.gz (70 MB)
- CMH_06_case4.tar.gz (52 MB)
- CMH_06_case5.tar.gz (18 MB)
- CMH_06_case6.tar.gz (52 MB)
COSMO2 06 Hour Runs
- CO2_06_case2.tar.gz (108 MB)
- CO2_06_case3.tar.gz (131 MB)
- CO2_06_case4.tar.gz (87 MB)
- CO2_06_case5.tar.gz (44 MB)
COSMO2 00 Hour Runs
- CO2_00_case1.tar.gz (67 MB)
- CO2_00_case2.tar.gz (88 MB)
- CO2_00_case3.tar.gz (88 MB)
- CO2_00_case4.tar.gz (67 MB)
- CO2_00_case5.tar.gz (25 MB)
- CO2_00_case6.tar.gz (67 MB)
- VERA_ensemble_20070620.tar.gz (1.2 GB)
- VERA_ensemble_20070621.tar.gz (1.2 GB)
- VERA_ensemble_20070622.tar.gz (1.2 GB)
Description of the CLEPS file formats.
- cleE01.tar.gz (197 MB)
- cleE02.tar.gz (197 MB)
- cleE03.tar.gz (197 MB)
- cleE04.tar.gz (197 MB)
- cleE05.tar.gz (197 MB)
- cleE06.tar.gz (197 MB)
- cleE07.tar.gz (197 MB)
- cleE08.tar.gz (197 MB)
- cleE09.tar.gz (197 MB)
- cleE10.tar.gz (197 MB)
- cleE11.tar.gz (197 MB)
- cleE12.tar.gz (197 MB)
- cleE13.tar.gz (197 MB)
- cleE14.tar.gz (197 MB)
- cleE15.tar.gz (197 MB)
- cleE16.tar.gz (197 MB)
The test cases for the first phase of the ICP included geometric idealized, perturbed real, and real cases. There were some additional cases as well, but they were not the primary focus. These cases are available and described below.
Test cases for the second phase of the project, MesoVICT, are coming soon. See the ECAM talks under the "Meetings" tab for two short presentations describing the upcoming test cases.
Please refer to Ahijevych et al. (2009) for a thorough description of the ICP test cases. Some information is given below.
We had three sets of test cases picked out for phase 1 of this project. Thanks to Mike Baldwin, Beth Ebert, Barbara Casati and others for helping to obtain them, and to David Ahijevych for creating the fake test cases. All of the cases are included for download by clicking on the link to this file: Cases20081023.tar.gz
Once extracted you will have the following folders inside the folder called Cases: 'sp2005,' 'nimrod,' 'MeteoSwiss,' and 'Fake.' Inside 'Fake' are further folders for the geometric and perturbed cases. Many of these test cases are also available with the R package SpatialVx.
Initial Test Cases from the Storm Prediction Center/National Severe Storms Laboratory (SPC/NSSL) Spring 2005 Program (sp2005)
We have cases from the WRF Spring 2005 program (see Kain et al., 2008 Wea. Forecasting, 23:931--952) including stage II analyses and three versions of WRF referred to as 'wrf2,' 'wrf4ncar' and 'wrf4ncep.' The fields are hourly accumulated precipitation in hundredths-of-an-inch. Please convert to mm by multiplying by 0.254 so that we all use the same units. All of these models have been interpolated onto the same 501 x 601 grid. Specifically, the NCEP g240 grid, which is a "4-km grid," but note that the grid is irregular (i.e., not always 4-km at each grid square). The longitude and latitude information is provided in the sp2005 folder as 'g240LatNorth.txt' and 'g240LonEast.txt.'
In the 'figures' folder you will find postscript and pdf graphs in the 'ps' and 'pdf' folders, respectively, of the first 9 cases chosen from this Spring program for this inter-comparison. Note that there are a couple extra cases in the 'st2,' 'wrf2' and 'wrf4ncar' folders. We did not have these dates for the NCEP model, so we will exclude them, but they are still provided as they may make for nice cases. All files are in ASCII format. See the README file in this folder for more information.
The cases we will start with are for verification times (00Z issue times): 4/26, 5/13, 5/14, 5/18, 5/19, 5/25, 6/1, 6/3 and 6/4.
The entire 32 Spring 2005 ARW and NMM cases studied in Davis et al (2009), Gilleland et al (2010) and Gilleland (2013) are available here (spc2005agg24hLead00UTC.tar.gz).
Nimrod Test Cases
Please acknowledge the UK Met Office (see the README file in the 'Nimrod' folder. These are the same cases used in the Casati et al (2004) paper (Meteorol. Appl. 11:141--154), and are described therein. We do not presently have lon/lat info for these cases (sorry).
MeteoSwiss Test Cases
Daniel Leuenberger at MeteoSwiss has offered two cases we could consider using. He asks that we acknowledge their source, and use them only for research purposes (see the 'README' file in the 'MeteoSwiss' folder). The cases give high-resolution aLMo radar data over Switzerland, along with hourly forecasts for a 24-hour period. One case also has results on a 7-km grid corresponding to a lower resolution version of the model. The files are in netCDF format, and IDL code to read them is included. One case has a postscript file that shows the hourly fields and basic verification statistics. The other case has a postscript file for the 24-hr totals, plus gif files for 6h totals.
Fake Test Cases
Future MesoVICT Meetings
Our next meeting will be 21 - 23 September 2016 in Bologna, Italy. For more details, and to register, go to http://www.arpae.it/dettaglio_evento.asp?idLivello=32&id=2415. Abstract submission deadline is 30 June 2016 and registration deadline is 1 September 2016.
There will also be a MesoVICT session (NWP7) at the 16th EMS Annual Meeting and 11th European Conference on Applied Climatology (ECAC) in Trieste, Italy, which takes place 12 - 16 September 2016.
Past MesoVICT Meetings
The 2015 meeting was as a session at the 15th European Meteorological Society (EMS) annual meeting to be held 7 - 11 September 2015 in Sofia, Bulgaria. Our session was part of the Numerical Weather Prediction programme group (NWP6).
- Bundel et al. INtercomparison of Spatial vErification methods for COSMO Terrain (INSPECT): Preliminary results (pdf)
- Casati et al. An intensity-scale score to assess the added value of enhanced resolution (pdf)
- Dorninger. Towards a simple and easy to interpret wind verification scheme (pdf)
- Mittermaier, M. Exploring the impact of neighbourhood size and height stratification in neighbourhood-based methods using single-site observations (pdf, pptx)
- Weniger, M. and P. Friederichs, A wavelet approach to the verification of probabilistic spatial fields (pdf)
First Planning/information meeting for the second phase of the ICP (SM9) was held on Tuesday, 10 September 2013 at the 13th EMS and 11th ECAM Conference, Reading, UK. Relevant talks from the meeting: Overview of ICP and phase 2 by Marion Mittermaier and Introduction to the next set of test cases by Manfred Dorninger.
The MesoVICT kick-off meeting was held 2 - 3 October 2014 in Vienna, Austria.
Past ICP Meetings
Fall 2009 Workshop: 24-25 August 2009 at NCAR's Center Green Campus, Boulder, Colorado.
Spring 2008 workshop held in Boulder, Colorado 14 - 15 April 2008.
20 February 2007 Planning Meeting (includes slides from the various talks).
An old link to some early results that may still be useful (mostly from past meeting presentations).
We will provide links to available software for performing spatial forecast verification from this page as they become available. All references listed below can be found in-full by clicking on the references tab above.
The following book has a nice appendix by Matt Pocernich on verification software (pp. 231--240). Also see Forecast Verification Issues, Methods and FAQ. From these, it is clear that several software tools for traditional verification are available. Listed here are only tools that were developed specifically at NCAR.
Jolliffe, I.T. and Stephenson, D.B., Forecast Verification: A practioner's guide in atmospheric sciences, Second Edition, Wiley-Balckwell, Chichester, West Sussex, U.K., 274 pp.
R package verification by Matt Pocernich. From your R session, you can install the package onto your machine with the install.packages function. You can then load it into your R session using the library function.
SpatialVx contains some functionality for doing the traditional forecast verification methods.
Contiguous Rain Area (CRA) (Ebert and McBride, 2000). IDL code from Beth Ebert available at http://www.cawcr.gov.au/staff/eee/index.php
Method for Object-based Diagnostic Evaluation (MODE) tool (Davis et al., 2006). Available through MET user's home page.
SpatialVx has some functionality for doing the method proposed by Davis et al. (2006), as well as the SAL technique (Wernli et al., 2008). In particular, it is possible to do either convolution thresholding, or just thresholding to identify features (or objects) as connected grid points above the threshold. It also has functions to calculate properties of single features (major/minor axis angle/length, aspect ratio, area, centroid) and properties for matched features (e.g., intersection area, area ratio, centroid distance, angle difference, as well as numerous binary image metrics; see below). More is on tap for future releases.
Field Deformation Approaches
Baddeley's Delta Metric: R code for computing Baddeley's delta metric for binary images (see, e.g., Gilleland, 2011)
FQI (Venugopal et al, 2005; Basu et al, 2003)
Image Warping: an R package to do the image warping found in Gilleland, Lindström and Lindgren (2010) is coming soon!. The original MatLab code will be replaced by these R functions, and they will be available through the SpatialVx package.
SpatialVx has functions to calculate FQI, Baddeley's Delta Metric, the Hausdorff metric, partial Hausdorff measure, mean error distance, mean square error distance, Pratt's Figure of Merit (FOM), minimum separation distance, (see the help file for locperf for references), as well as optical flow as proposed by Marzban and Sandgathe (2010).
Neighborhood-Based Approaches (summarized in Ebert, 2008)
IDL code from Beth Ebert available at http://www.cawcr.gov.au/staff/eee/index.php
The R package SpatialVx has functions for doing most of the neighborhood methods.
Scale Decomposition Approaches
Intensity-Scale (IS) (Casati et al, 2004). Available in the R package verification (above under traditional verification). Also available from the Model Evaluation Tools (MET) software.
SpatialVx has the wavelet denoising (more a neighborhood type approach), and wavelet decomposition approaches of Briggs and Levine (1997), as well as the intensity-scale techniques of Casati et al. (2004) and Casati (2009).
Validation ToolboxSoftware from UC Irvine's Center for Hydrology and Remote Sensing (CHRS) Department of Civil and Environmental Engineering:
Includes: Performance Metrics for Evaluation of Remote Sensing Observations and Climate Model Simulations: A simple and easy to use Validation Toolbox (MATLAB source code) that can be used for validation of gridded data including satellite observations, reanalysis data, and weather and climate model simulations. In addition to the commonly used categorical indices, the toolbox includes the Volumetric Hit Index (VHI), Volumetric False Alarm Ration (VFAR), Volumetric Missed Index (VMI), and Volumetric Critical Success Index (VCSI).
The list of references was getting to be too long to list here. Please find them at this link.
All of the references are also available in BibTeX format, where the cite key follows the format: AuthorLastNameYear (one author; e.g., Casati2009), Author1LastNameAuthor2LastNameYear (two authors; e.g., BriggsLevine1997), Author1LastNameEtAlYear (more than two authors; e.g., AhijevychEtAl2009). If the previous does not identify a unique paper in the list (e.g., Davis et al., 2006), then a lower case letter a, b, c, ... is appended to the end (e.g., DavisEtAl2006a and DavisEtAl2006b).
Dorninger, M., M. P. Mittermaier, E. Gilleland, E. E. Ebert, B. G. Brown, and L. J. Wilson, 2013: MesoVICT: Mesoscale Verification Inter-Comparison over Complex Terrain. NCAR Technical Note NCAR/TN-505+STR, 23 pp, DOI: http://dx.doi.org/10.5065/D6416V21.
Geiß, S., 2015. Comparison of spatial verification methods. Bachelor thesis. Ludwig-Maximilians-University Munich Meteorological Institute Munich, Supervisor: Christian Keil, 55 pp. (pdf)
Gilleland, E., 2016. A new characterization in the spatial verification framework for false alarms, misses, and overall patterns. Submitted to Wea. Forecasting on 20 July 2016 (pdf).
We have divided the project into working groups to focus on specific tiers of MesoVICT (see image below). The groups and their leaders are listed below. Participants may work in more than one of these groups, if desired, but note that they all overlap with WG1 and WG3 with WG2.
Working Group 1 (Core Case)
Lead: Eric Gilleland
Brief description: Working Group 1 will focus on the core case (the minimal amount). For those participants merely looking to learn about the various existing methods, this working group would be the appropriate one to join.
Working Group 2 (Core Case + Tier 1)
Lead: Marion Mittermaier
Brief description: In addition to the core case (overlapping with WG1), Working Group 2 will focus on the wind variable and ensembles of forecast models.
Working Group 3 (Core Case + Tier 1 to Tier 3)
Lead: Manfred Dorninger
Brief description: In addition to the core case and tier 1, this group will fully investigate the use of ensembles of observations, as well as other models and variables.
Working Group 4 (NWP model re-runs)
Lead: Stefano Mariani (stefano "dot" mariani "at " isprambiente.it)
Brief description: Working Group 4 is for those planning to do NWP model re-runs in order to test newer models for the same cases.
The list of participants under ICP phase 1 below was created post priori from the list of authors of papers in the ICP special collection of Weather and Forecasting. Please contact us with any errors.
Organizing committee in bold.
|Amir AghaKouchak||Barbara G. Brown||Randy G. Bullock|
|Anastasia Bundel||Barbara Casati||George C. Craig|
|Alex Dekmyn||Manfred Dorninger||Izuchukwu Ebenebe|
|Elizabeth E. Ebert||Jian Fan||Eric Gilleland|
|Christian Keil||Chiara Marsigli||Caren Marzban|
|Marion P. Mittermaier||Andrea Montani||Jason E. Nachamkin|
|Phu Nguyen||Tiziana Paccagnella||Sabrina Radanovics|
|Nigel Roberts||Francis Schubiger||Scott Sellars|
|Gregor Skok||Geert Smet||Maria Stefania Tesini|
|Susanne Theis||Emiel van der Plas||Bert van Schaeybroeck|
|Volker Wulfmeyer||Laurence Wilson|
ICP phase 1 Participants
|David A. Ahijevych||Keith F. Brill||Barbara G. Brown|
|Randy G. Bullock||Barbara Casati||George C. Craig|
|Christopher A. Davis||Elizabeth E. Ebert||Neil I. Fox|
|Eric Gilleland||John Halley Gotway||Christiane Hofmann|
|John S. Kain||Christian Keil||Steven A. Lack|
|Valliappa Lakshmanan||Nicholas Lederer||George L. Limpert|
|Finn Lindgren||Johan Lindström||Hilary Lyons|
|Caren Marzban||Fedor Mesinger||Marion P. Mittermaier|
|Jason E. Nachamkin||Nigel Roberts||Scott Sandgathe|
|Heini Wernli||Matthias Zimmer|
MesoVICT Committee Members
Eric Gilleland (webmaster)
Marion Mittermaier (marion.mittermaier " at " metoffice.gov.uk)
Laurie Wilson (Lawrence.Wilson " at " ec.gc.ca)
Subscribe to or unsubscribe from the project email list
To subscribe to the email list, please do both of the following:
(1) Send an email to Eric Gilleland giving your name, institution, contact information, and whether or not you plan to participate in MesoVICT, and if not, why you want to be on the list.
(2) Using your same email address from step (1), sign up here.*
Note: the reason for step (1) above is to verify who you are in order to protect against spam. If your identity cannot be easily verified, your subscription request will be ignored.
To unsubscribe, go to the same web page, and follow the instructions there to unsubscribe. No need to send Eric an email to unsubscribe.
*Any information collected is used solely to determine the legitimacy of subscription requests (e.g., to protect against spam). Email addresses are added to a controlled list (only subscribers may send messages). The list is intended primarily for participants in the ICP, but others wishing to keep updated on the progress of the project may also subscribe. If you have any trouble subscribing (or unsubscribing) contact the webmaster (Eric Gilleland).
The National Center for Atmospheric Research is sponsored by the National Science Foundation. Any opinions, findings, and conclusions or recommendations expressed in this publication are those of the author(s) and do not necessarily reflect t he views of the National Science Foundation.