Guidance Document on Model Quality Objectives and Benchmarking

6.2.Open issues

Based on user feedback the following improvements are proposed to the template:

• 
  In the Summary Report the name of the pollutant indicator for which the report was generated is missing.
  
• 
  A single symbol is used for the stations in the Summary Report: would it not be possible to reuse the symbols used in the Target Plot/Scatter Diagram to identify the different stations?
  
• 
  In the Target Plot/Scatter Diagram the colour coding by site type is useful, but a key to the colour coding that is used would be helpful.
  
• 
  In the summary plots for the observations, the green colour could be used to designate the area where the observations are within the limit values.
  
• 
  The definitions of the different indicators should be included in the report to make apparent that these are not the ‘standard’ definitions for bias, correlation and standard deviation but that these have been normalised with the measurement uncertainty.
  

7.Examples of good practice

In this section we present a number of examples provided to us by the following parties:

• 
  Regional Agency for Environmental Protection and Prevention (ARPA) Emilia Romagna, Italy
  
• 
  Cambridge Environmental Research Consultants (CERC), United Kingdom
  
• 
  University of Aveiro, Portugal
  
• 
  Belgian Interregional Environment Agency (IRCEL), Belgium
  
• 
  University of Brescia (UNIBS), Italy
  
• 
  Ricardo AEA, UK
  

7.1.CERC experience

Jenny Stocker, CERC

Background Information

1. 
   What is the context of your work:
   
   1. 
      Frame of the modelling exercise (Air Quality Plan, research project, …?)
      

Model verification exercise

2. 
   Scope of the exercise (pollutants, episodes…)
   

Pollutants: NO_x, NO₂, O₃, PM₁₀, PM_2.5, SO₂ interested in annual averages and model performance statistics, e.g. correlation, standard deviation. Also plots of results, such as scatter plots, bar-charts, Q-Q plots.

2. 
   Model
   
   1. 
      Model name
      

ADMS-Urban

2. 
   Main assumptions
   

Advanced three dimensional quasi-Gaussian model calculating concentrations hour by hour, nested within a straight line Lagrangian trajectory model which is used to calculate background concentrations approaching the area of interest. Road, industrial and residual sources can be modelled in detail using a variety of options such as terrain, buildings, street canyons and chemistry.

3. 
   I/O
   

Input: Emissions data, such as a grid of emissions over modelling domain, with detailed road and industrial source emissions; hourly meteorological data and measured/modelled background concentrations in text file format; text files containing the variation of terrain heights and roughness lengths over the domain; source parameters such as widths and street canyon geometry for roads, stack heights for industrial sources; and building dimensions.

Output: Concentrations may be output in an hourly average format over a 2D or 3D grid of receptor points and/or at specified receptor points.

4. 
   Reference to MDS if available
   

Reference to MDS: http://pandora.meng.auth.gr/mds/showshort.php?id=18

Figure Results of the CERC test case

1. 
   Test-case
   
   1. 
      Spatial resolution and spatial domain
      

Modelling covered Greater London (approx. 40 km x 50 km) with variable resolution, i.e. finer resolution near roadside areas, regular grid in background areas.

2. 
   Temporal resolution
   

Hourly average data for one year at multiple receptor points

3. 
   Pollutants considered
   

NO_x, NO₂, O₃, PM₁₀, PM_2.5, SO₂

4. 
   Data assimilation, if yes methodology used
   

Not used

Evaluation

1. 
   How did you select the stations used for evaluation?
   

All available monitoring stations with data for the modelled year have been included in the analysis.

2. 
   In case of data-assimilation, how are the evaluation results prepared?
   

Not applicable.

3. 
   Please comment the DELTA performance report templates
   

Looking at the summary statistics report given below both the scatter and target plots (Figure ), it is not clear which station performed well as all sites use the same symbol, so we cannot see which stations are performing well and which are underperforming from this summary plot alone. Even if the individual symbols are not used the colour coding by site type may be useful here.

Scatter plot

The scatter plot (Figure ) shows the bands well with the bold colouring and the individual symbols for each monitoring site is useful. The colour coding by site type is very useful, it would be good to have the key to this colour coding included in the plot.

Target plot

Most of the comments for the scatter plot apply to the Target Plot. It is useful to see the individual symbols for each site modelled, the colour coding by site type is useful, but a key to the colour coding used would be helpful.

Further it is stated the left and right hand side of the target plot distinguishes between points that have errors dominated by correlation and those dominated by standard deviation. However, in terms of ‘reading’ the plot, it is hard to know how close, in terms of accuracy, the points on either side of the plot are; could the plot be replaced by a semi-circle, and the information regarding correlation and standard deviation be presented separately? So that there is a smooth transition between the values, rather than a jump?

Feedback

1. 
   What is your overall experience with DELTA?
   

CERC argues that there is little justification for insisting that models and measurements are subject to the same degree of error as this would mean that models need to improve as measurement uncertainty becomes smaller. Model objective criteria need to be developed which ensure the model has a performance appropriate for the task for which it is being used, both in terms of application (for example compliance assessment, policy, local planning or research) and scale (for example regional, urban or roadside).

When performing validation, it is helpful to look at both NO_x as well as NO₂, as the former pollutant is less influenced by chemistry, and is therefore a better measure of the models’ ability to represent dispersion processes.

Furthermore CERC provides feedback on using the DELTA Tool implementation as provided by JRC. Points of improvement to the DELTA Tool implementation provided by CERC relate to the use of IDL and the different file formats that are used for observed and modelled data.

2. 
   How do you compare the benchmarking report of DELTA with the evaluation procedure you normally use? Please briefly describe the procedure you normally use for model evaluation?
   

CERC currently uses the benchmarking procedure described here but with the Myair toolkit developed during EU FP7 PASODOBLE project.

CERC currently uses the benchmarking procedure described here but with the Myair toolkit developed during EU FP7 PASODOBLE project. Some advantages and additional features of the Myair Toolkit compared to the DELTA tool are:

• 
  It is more flexible in terms of concentration data input so that for a typical project, much less time is spent re-formatting the data for input into the processing tool.
  
• 
  It includes some additional statistics: the number of valid observations and the observed and modelled maximum concentrations.
  
• 
  It allows statistics to be binned according to site type or pollutant while in the Delta Tool statistics are only given by site type.
  
• 
  It can process many pollutants and datasets together which is very useful for the inter-comparison between different modelled datasets in model validation.
  
• 
  It can produce Box and Whisker plot.
  

3. 
   What do you miss in the DELTA benchmarking report and/or which information do you find unnecessary
   

CERC would like to see statistics for each receptor point, and each pollutant in a numerical table. The statistics plot could use a different colour for each site type.