Guidance Document on Model Quality Objectives and Benchmarking

4.3.Literature on the implementation and use of the Delta tool

Thunis et al., 2013: A tool to evaluate air quality model performances in regulatory applications

The article presents the DELTA Tool and Benchmarking service for air quality modelling applications, developed within FAIRMODE by the Joint Research Centre of the European Commission in Ispra (Italy). The DELTA tool addresses model applications for the AQD, 2008 and is mainly intended for use on assessments. The DELTA tool is an IDL-based evaluation software and is structured around four main modules for respectively the input, configuration, analysis and output. The user can run DELTA either in exploration mode for which flexibility is allowed in the selection of time periods, statistical indicators and stations, or in benchmarking mode for which the evaluation is performed on one full year of modelling data with pre-selected statistical indicators and diagrams. The Authors also present and discuss some examples of DELTA tool outputs.

Carnevale et al., 2014: 1. Applying the Delta tool to support AQD: The validation of the TCAM chemical transport model

This paper presents an application of the DELTA evaluation tool V3.2and test the skills of DELTA tool by looking at the results of a 1-year (2005) simulation performed using the chemical transport model TCAM at 6km × 6km resolution over the Po Valley. The modelled daily PM₁₀ concentrations at surface level are compared to observations provided by approximately 50 stations distributed across the domain. The main statistical parameters (i.e., bias, root mean square error, correlation coefficient, standard deviation) as well as different types of diagrams (scatter plots, time series plots, Taylor and Target plots) are produced by the Authors. A representation of the observation uncertainty in the Target plot, used to derive model performance criteria for the main statistical indicators, is presented and discussed.

Thunis et al., 2014: DELTA Version 4 User’s Guide

This is currently the most recent version of the user’s guide for the DELTA tool. The document consists of three main parts: the concepts, the actual user’s guide and an overview of the diagrams the tool can produce. The concepts part sets the application domain for the tool and lists the underlying ideas of the evaluation procedure highlighting that the tool can be used both for exploration and for benchmarking. The MQO and the MPCs that are applied are explained including a proposal for an alternative way to derive the linear expression relating uncertainty to observed concentrations. Examples of the model benchmarking report are presented for the cases model results are available hourly and as a yearly average. The actual user guide contains the information needed to install the tool, prepare input for the tool, and run the tool both in exploration and in benchmarking modes. Also details on how to customise certain settings (e.g. uncertainty) and how to use the included utility programs are given.

Carnevale et al., 2014: A methodology for the evaluation of re-analysed PM10 concentration fields: a case study over the Po valley

This study presents a general Monte Carlo based methodology for the validation of Chemical Transport Model (CTM) concentration re-analysed fields over a certain domain. A set of re-analyses is evaluated by applying the observation uncertainty (U) approach, developed in the frame of FAIRMODE. Modelled results from the Chemical Transport Model TCAM for the year 2005 are used as background values. The model simulation domain covers the Po valley with a 6 kmx6 km resolution. Measured data for both assimilation and evaluation are provided by approximately 50 monitoring stations distributed across the Po valley. The main statistical indicators (i.e. Bias, Root Mean Square Error, correlation coefficient, standard deviation) as well as different types of diagrams (scatter plots and Target plots) have been produced and visualized with the Delta evaluation Tool V3.6.

5.Model Quality Objective (MQO)

5.1.Statistical performance indicators

Models applied for regulatory air quality assessment are commonly evaluated on the basis of comparisons against observations. This element of the model evaluation process is also known as operational model evaluation or statistical performance analysis, since statistical indicators and graphical analysis are used to determine the capability of an air quality model to reproduce measured concentrations. It is generally recommended to apply multiple performance indicators regardless of the model application since each one has its advantages and disadvantages.

To cover all aspects of the model performance in terms of amplitude, phase and bias the following set of statistical indicators can be used for the statistical analysis of model performance with M_i and O_i respectively the modelled and observed values where i is a number (rank) between 1 and N and N the total number of modelled or observed values:

Root Mean Square Error ( RMSE)

(1)

correlation coefficient (R)

(2)

with the average observed value and the average modelled value.

Normalised Mean Bias (NMB)

where (3)

Normalised Mean Standard Deviation (NMSD)

(4)

with the standard deviation of the observed values and

the standard deviation of the modelled values.