MesoVICT

  • About
  • News
  • Test Cases
  • Meetings
  • Software
  • References
  • MesoVICT Papers
  • Working Groups
  • Participants
  • Contact

MesoVICT: Mesoscale Verification Intercomparison over Complex Terrain (Phase 2 of the ICP)

About the ICP and MesoVICT

White paper describing the MesoVICT community project.

The spatial forecast verification inter-comparison project (ICP) was set up to attempt to sift through the maze of newly proposed methods for verifying primarily high-resolution forecasts. MesoVICT continues this aim with more meteorologically complex test cases including: more complex terrain, gridded as well as point observations, ensembles of forecasts, ensembles of observations, more variables in addition to precipitation (e.g. winds), as well as meteorological events that occur over a period of time (i.e., adding a short time dimension to the question).

The intent of this project is to compare the various newly proposed methods to give the user information about which methods are appropriate for which types of data, forecasts and desired forecast utility. The point is not to determine which method is best. However, it is hoped that advantages and disadvantages of the methods will become clear.

The MesoVICT project, like its predecessor (ICP), is a community project. It is not a model inter-comparison, but a verification methods (meta-verification) inter-comparison.

If you are interested in participating in the second phase of the ICP (to be called "Mesocale Verification Intercomparison in Complex Terrain", or MesoVICT for short), please click on the Email list tab to subscribe. You will then be kept informed of activities.

News

The new COSMO-1 model re-runs provided by MeteoSwiss are now available. Runs have been conducted for the core case and cases 2 and 6. Data can be accessed via the University of Vienna on request. Please contact Manfred.Dorninger@univie.ac.at

Workshop on Verification Inter-Comparison will be held 21 - 23 September 2016 in Bologna, Italy.

There will also be a MesoVICT session (NWP7) at the 16th EMS Annual Meeting and 11th European Conference on Applied Climatology (ECAC) in Trieste, Italy, which takes place 12 - 16 September 2016.

An example of how to read some of the test cases into R

The original ICP phase 1 test cases (as well as some extras) can be found further below.

For further model data that go beyond the MesoVICT project please visit the World Data Center for Climate in Hamburg, Germany, where all model output (and other data) from the DPHASE project are stored.

MesoVICT cases

For groups wishing to provide model re-runs of the test cases below (i.e., test newer model versions on these dates), please contact the organizing committee. Note that our purpose is not to verify your models (this is a meta-verification project), but it may be possible to work with individual participants who might be interested in doing so. In such cases, it will not be possible to store the data on this website, but we will be happy to provide a link to the data.

Information

Please refer to the Technical Note for the project to learn about the test cases for MesoVICT. Note that COSMO2 06 run is not available for the core case, so we will use the 00 run for the core case (for this model only) instead; the CMC GEMH is only available at 06, which is why we chose to use the 06 runs generally. For all other cases, we will use the 06 runs. See also Bauer et al. (2011) for more information about the data sets generally.

The figure below shows a schematic diagram of the tiered case structure for the project.

Vienna Enhanced Resolution Analysis (VERA)

Description of the VERA cases file format (pdf). The lon/lat coordinates are in this text file.

Topography information is given in VERA_topography.dat, with a description of this file in Description_of_VERA_topography_file_format.pdf

The six VERA test case analyses are each available in a separate zip file linked below:

Point Observations: Joint D-PHASE - COPS (JDC) data set

Description of the JDC cases file format (pdf).

The six JDC test case point observation files are linked below:

Model Output

Description of model output file format for all models (pdf).

Note that COSMO2 06 hour runs are not available for cases 1 and 6. Because case 1 is a core case, please use COSMO2 00 hour run for this case instead; as though it were the 06 hour run.

MeteoSwiss plans to recalculate these cases with the current version of their models (experimental COSMO-1, operational COSMO-2, and COSMO-7, which have grid meshes of 1.1, 2.2 and 6.6 km, respectively). These re-run cases should be available in the coming months.

Canadian high-resolution model: CMC GEMH (GEM-LAM) model output from Environment Canada

COSMO2 06 Hour Runs

COSMO2 00 Hour Runs

COSMO-1 model re-runs

The new COSMO-1 model re-runs provided by MeteoSwiss are available. Runs have been conducted for the core case and cases 2 and 6. Data can be accessed via the University of Vienna on request. Please contact Manfred.Dorninger@univie.ac.at

VERA ensembles

Description of VERA ensemble File Format

VERA_ensemble_8km_coordinates_lam_phi.txt

COSMO-LEPS (CLEPS)

Description of the CLEPS file formats.



ICP cases

The primary cases are also available via the SpatialVx package in R.

The test cases for the first phase of the ICP included geometric idealized, perturbed real, and real cases. There were some additional cases as well, but they were not the primary focus. These cases are available and described below.

Test cases for the second phase of the project, MesoVICT, are coming soon. See the ECAM talks under the "Meetings" tab for two short presentations describing the upcoming test cases.

Information

Please refer to Ahijevych et al. (2009) for a thorough description of the ICP test cases. Some information is given below.

We had three sets of test cases picked out for phase 1 of this project. Thanks to Mike Baldwin, Beth Ebert, Barbara Casati and others for helping to obtain them, and to David Ahijevych for creating the fake test cases. All of the cases are included for download by clicking on the link to this file: Cases20081023.tar.gz

Once extracted you will have the following folders inside the folder called Cases: 'sp2005,' 'nimrod,' 'MeteoSwiss,' and 'Fake.' Inside 'Fake' are further folders for the geometric and perturbed cases. Many of these test cases are also available with the R package SpatialVx.

Initial Test Cases from the Storm Prediction Center/National Severe Storms Laboratory (SPC/NSSL) Spring 2005 Program (sp2005)

We have cases from the WRF Spring 2005 program (see Kain et al., 2008 Wea. Forecasting, 23:931--952) including stage II analyses and three versions of WRF referred to as 'wrf2,' 'wrf4ncar' and 'wrf4ncep.' The fields are hourly accumulated precipitation in hundredths-of-an-inch. Please convert to mm by multiplying by 0.254 so that we all use the same units. All of these models have been interpolated onto the same 501 x 601 grid. Specifically, the NCEP g240 grid, which is a "4-km grid," but note that the grid is irregular (i.e., not always 4-km at each grid square). The longitude and latitude information is provided in the sp2005 folder as 'g240LatNorth.txt' and 'g240LonEast.txt.'

In the 'figures' folder you will find postscript and pdf graphs in the 'ps' and 'pdf' folders, respectively, of the first 9 cases chosen from this Spring program for this inter-comparison. Note that there are a couple extra cases in the 'st2,' 'wrf2' and 'wrf4ncar' folders. We did not have these dates for the NCEP model, so we will exclude them, but they are still provided as they may make for nice cases. All files are in ASCII format. See the README file in this folder for more information.

The cases we will start with are for verification times (00Z issue times): 4/26, 5/13, 5/14, 5/18, 5/19, 5/25, 6/1, 6/3 and 6/4.

The entire 32 Spring 2005 ARW and NMM cases studied in Davis et al (2009), Gilleland et al (2010) and Gilleland (2013) are available here (spc2005agg24hLead00UTC.tar.gz).

Nimrod Test Cases

Please acknowledge the UK Met Office (see the README file in the 'Nimrod' folder. These are the same cases used in the Casati et al (2004) paper (Meteorol. Appl. 11:141--154), and are described therein. We do not presently have lon/lat info for these cases (sorry).

MeteoSwiss Test Cases

Daniel Leuenberger at MeteoSwiss has offered two cases we could consider using. He asks that we acknowledge their source, and use them only for research purposes (see the 'README' file in the 'MeteoSwiss' folder). The cases give high-resolution aLMo radar data over Switzerland, along with hourly forecasts for a 24-hour period. One case also has results on a 7-km grid corresponding to a lower resolution version of the model. The files are in netCDF format, and IDL code to read them is included. One case has a postscript file that shows the hourly fields and basic verification statistics. The other case has a postscript file for the 24-hr totals, plus gif files for 6h totals.

Fake Test Cases

Fake test cases are based on the same grid as the sp2005 cases. There are both idealized geometric cases as well as perturbed real cases. See the README files for each case for more information.


Future MesoVICT Meetings

Our next meeting will be 21 - 23 September 2016 in Bologna, Italy. For more details, and to register, go to http://www.arpae.it/dettaglio_evento.asp?idLivello=32&id=2415. Abstract submission deadline is 30 June 2016 and registration deadline is 1 September 2016.

There will also be a MesoVICT session (NWP7) at the 16th EMS Annual Meeting and 11th European Conference on Applied Climatology (ECAC) in Trieste, Italy, which takes place 12 - 16 September 2016.

Past MesoVICT Meetings

The 2015 meeting was as a session at the 15th European Meteorological Society (EMS) annual meeting to be held 7 - 11 September 2015 in Sofia, Bulgaria. Our session was part of the Numerical Weather Prediction programme group (NWP6).

  • Bundel et al. INtercomparison of Spatial vErification methods for COSMO Terrain (INSPECT): Preliminary results (pdf)
  • Casati et al. An intensity-scale score to assess the added value of enhanced resolution (pdf)
  • Dorninger. Towards a simple and easy to interpret wind verification scheme (pdf)
  • Mittermaier, M. Exploring the impact of neighbourhood size and height stratification in neighbourhood-based methods using single-site observations (pdf, pptx)
  • Weniger, M. and P. Friederichs, A wavelet approach to the verification of probabilistic spatial fields (pdf)

First Planning/information meeting for the second phase of the ICP (SM9) was held on Tuesday, 10 September 2013 at the 13th EMS and 11th ECAM Conference, Reading, UK. Relevant talks from the meeting: Overview of ICP and phase 2 by Marion Mittermaier and Introduction to the next set of test cases by Manfred Dorninger.

The MesoVICT kick-off meeting was held 2 - 3 October 2014 in Vienna, Austria.

Past ICP Meetings

Fall 2009 Workshop: 24-25 August 2009 at NCAR's Center Green Campus, Boulder, Colorado.

Spring 2008 workshop held in Boulder, Colorado 14 - 15 April 2008.

20 February 2007 Planning Meeting (includes slides from the various talks).

Preliminary Results

An old link to some early results that may still be useful (mostly from past meeting presentations).

We will provide links to available software for performing spatial forecast verification from this page as they become available. All references listed below can be found in-full by clicking on the references tab above.

SpatialVx

A new R package called SpatialVx is in the works. It already has functions to do many of the newly proposed methods, but there are many yet to do. Keep checking for new updates.

Traditional Verification

The following book has a nice appendix by Matt Pocernich on verification software (pp. 231--240). Also see Forecast Verification Issues, Methods and FAQ. From these, it is clear that several software tools for traditional verification are available. Listed here are only tools that were developed specifically at NCAR.

Jolliffe, I.T. and Stephenson, D.B., Forecast Verification: A practioner's guide in atmospheric sciences, Second Edition, Wiley-Balckwell, Chichester, West Sussex, U.K., 274 pp.

R package verification by Matt Pocernich. From your R session, you can install the package onto your machine with the install.packages function. You can then load it into your R session using the library function.

Model Evaluation Tools (MET) user's home page

SpatialVx contains some functionality for doing the traditional forecast verification methods.

Features-based Approaches

Contiguous Rain Area (CRA) (Ebert and McBride, 2000). IDL code from Beth Ebert available at http://www.cawcr.gov.au/staff/eee/index.php

Method for Object-based Diagnostic Evaluation (MODE) tool (Davis et al., 2006). Available through MET user's home page.

SpatialVx has some functionality for doing the method proposed by Davis et al. (2006), as well as the SAL technique (Wernli et al., 2008). In particular, it is possible to do either convolution thresholding, or just thresholding to identify features (or objects) as connected grid points above the threshold. It also has functions to calculate properties of single features (major/minor axis angle/length, aspect ratio, area, centroid) and properties for matched features (e.g., intersection area, area ratio, centroid distance, angle difference, as well as numerous binary image metrics; see below). More is on tap for future releases.

Field Deformation Approaches

Baddeley's Delta Metric: R code for computing Baddeley's delta metric for binary images (see, e.g., Gilleland, 2011)

FQI (Venugopal et al, 2005; Basu et al, 2003)

Image Warping: an R package to do the image warping found in Gilleland, Lindström and Lindgren (2010) is coming soon!. The original MatLab code will be replaced by these R functions, and they will be available through the SpatialVx package.

SpatialVx has functions to calculate FQI, Baddeley's Delta Metric, the Hausdorff metric, partial Hausdorff measure, mean error distance, mean square error distance, Pratt's Figure of Merit (FOM), minimum separation distance, (see the help file for locperf for references), as well as optical flow as proposed by Marzban and Sandgathe (2010).

Neighborhood-Based Approaches (summarized in Ebert, 2008)

IDL code from Beth Ebert available at http://www.cawcr.gov.au/staff/eee/index.php

The R package SpatialVx has functions for doing most of the neighborhood methods.

Scale Decomposition Approaches

Intensity-Scale (IS) (Casati et al, 2004). Available in the R package verification (above under traditional verification). Also available from the Model Evaluation Tools (MET) software.

SpatialVx has the wavelet denoising (more a neighborhood type approach), and wavelet decomposition approaches of Briggs and Levine (1997), as well as the intensity-scale techniques of Casati et al. (2004) and Casati (2009).

Other

Validation Toolbox

Software from UC Irvine's Center for Hydrology and Remote Sensing (CHRS) Department of Civil and Environmental Engineering:

http://amir.eng.uci.edu/downloads/ValidationToolbox.zip

Includes: Performance Metrics for Evaluation of Remote Sensing Observations and Climate Model Simulations: A simple and easy to use Validation Toolbox (MATLAB source code) that can be used for validation of gridded data including satellite observations, reanalysis data, and weather and climate model simulations. In addition to the commonly used categorical indices, the toolbox includes the Volumetric Hit Index (VHI), Volumetric False Alarm Ration (VFAR), Volumetric Missed Index (VMI), and Volumetric Critical Success Index (VCSI).

References

The list of references was getting to be too long to list here. Please find them at this link.

All of the references are also available in BibTeX format, where the cite key follows the format: AuthorLastNameYear (one author; e.g., Casati2009), Author1LastNameAuthor2LastNameYear (two authors; e.g., BriggsLevine1997), Author1LastNameEtAlYear (more than two authors; e.g., AhijevychEtAl2009). If the previous does not identify a unique paper in the list (e.g., Davis et al., 2006), then a lower case letter a, b, c, ... is appended to the end (e.g., DavisEtAl2006a and DavisEtAl2006b).

Dorninger, M., M. P. Mittermaier, E. Gilleland, E. E. Ebert, B. G. Brown, and L. J. Wilson, 2013: MesoVICT: Mesoscale Verification Inter-Comparison over Complex Terrain. NCAR Technical Note NCAR/TN-505+STR, 23 pp, DOI: http://dx.doi.org/10.5065/D6416V21.

Geiß, S., 2015. Comparison of spatial verification methods. Bachelor thesis. Ludwig-Maximilians-University Munich Meteorological Institute Munich, Supervisor: Christian Keil, 55 pp. (pdf)

Gilleland, E., 2016. A new characterization in the spatial verification framework for false alarms, misses, and overall patterns. Submitted to Wea. Forecasting on 20 July 2016 (pdf).

We have divided the project into working groups to focus on specific tiers of MesoVICT (see image below). The groups and their leaders are listed below. Participants may work in more than one of these groups, if desired, but note that they all overlap with WG1 and WG3 with WG2.

Working Group 1 (Core Case)

Lead: Eric Gilleland

Brief description: Working Group 1 will focus on the core case (the minimal amount). For those participants merely looking to learn about the various existing methods, this working group would be the appropriate one to join.

Working Group 2 (Core Case + Tier 1)

Lead: Marion Mittermaier

Brief description: In addition to the core case (overlapping with WG1), Working Group 2 will focus on the wind variable and ensembles of forecast models.

Working Group 3 (Core Case + Tier 1 to Tier 3)

Lead: Manfred Dorninger

Brief description: In addition to the core case and tier 1, this group will fully investigate the use of ensembles of observations, as well as other models and variables.

Working Group 4 (NWP model re-runs)

Lead: Stefano Mariani (stefano "dot" mariani "at " isprambiente.it)

Brief description: Working Group 4 is for those planning to do NWP model re-runs in order to test newer models for the same cases.

The list of participants under ICP phase 1 below was created post priori from the list of authors of papers in the ICP special collection of Weather and Forecasting. Please contact us with any errors.

MesoVICT Participants*

Organizing committee in bold.

Amir AghaKouchak Barbara G. Brown Randy G. Bullock
Anastasia Bundel Barbara Casati George C. Craig
Alex Dekmyn Manfred Dorninger Izuchukwu Ebenebe
Elizabeth E. Ebert Jian Fan Eric Gilleland
Christian Keil Chiara Marsigli Caren Marzban
Marion P. Mittermaier Andrea Montani Jason E. Nachamkin
Phu Nguyen Tiziana Paccagnella Sabrina Radanovics
Nigel Roberts Francis Schubiger Scott Sellars
Gregor Skok Geert Smet Maria Stefania Tesini
Susanne Theis Emiel van der Plas Bert van Schaeybroeck
Volker Wulfmeyer Laurence Wilson
*The above list is an initial list of folks who have already expressed interest in participating and is subject to change. Please let Eric Gilleland know if your name is there, but should not be, or if your name is not on the list, but should be. Thank you!
ICP phase 1 Participants
David A. Ahijevych Keith F. Brill Barbara G. Brown
Randy G. Bullock Barbara Casati George C. Craig
Christopher A. Davis Elizabeth E. Ebert Neil I. Fox
Eric Gilleland John Halley Gotway Christiane Hofmann
John S. Kain Christian Keil Steven A. Lack
Valliappa Lakshmanan Nicholas Lederer George L. Limpert
Finn Lindgren Johan Lindström Hilary Lyons
Caren Marzban Fedor Mesinger Marion P. Mittermaier
Jason E. Nachamkin Nigel Roberts Scott Sandgathe
Heini Wernli Matthias Zimmer

Contact Information

MesoVICT Committee Members

Eric Gilleland (webmaster)

Marion Mittermaier (marion.mittermaier " at " metoffice.gov.uk)

Barb Brown

Manfred Dorninger

Beth Ebert

Laurie Wilson (Lawrence.Wilson " at " ec.gc.ca)

Subscribe to or unsubscribe from the project email list

Subscribe

To subscribe to the email list, please do both of the following:

(1) Send an email to Eric Gilleland giving your name, institution, contact information, and whether or not you plan to participate in MesoVICT, and if not, why you want to be on the list.

(2) Using your same email address from step (1), sign up here.*

Note: the reason for step (1) above is to verify who you are in order to protect against spam. If your identity cannot be easily verified, your subscription request will be ignored.

Unsubscribe

To unsubscribe, go to the same web page, and follow the instructions there to unsubscribe. No need to send Eric an email to unsubscribe.


MesoVICT is a project of the WMO Joint Working Group on Forecast Verification


*Any information collected is used solely to determine the legitimacy of subscription requests (e.g., to protect against spam). Email addresses are added to a controlled list (only subscribers may send messages). The list is intended primarily for participants in the ICP, but others wishing to keep updated on the progress of the project may also subscribe. If you have any trouble subscribing (or unsubscribing) contact the webmaster (Eric Gilleland).


The National Center for Atmospheric Research is sponsored by the National Science Foundation. Any opinions, findings, and conclusions or recommendations expressed in this publication are those of the author(s) and do not necessarily reflect t he views of the National Science Foundation.