Guidance Document on Model Quality Objectives and Benchmarking

7.4.UAVR experience with DELTA

Alexandra Monteiro and Ana Miranda

Background Information

1. 
   What is the context of your work:
   
   1. 
      Frame of the modelling exercise (Air Quality Plan, research project, …?)
      

Air quality assessment; Air Quality Plans and also research work for publications

2. 
   Scope of the exercise (pollutants, episodes…)
   

PM₁₀, PM_2.5, NO₂ and O₃ have been considered in the scope of the annual air quality assessment delivered to the Portuguese Agency for Environment and PM₁₀ and NO₂ have been worked within AQP. Research activities include all PM₁₀, PM_2.5, NO₂ and O₃. If other pollutants will be included in DELTA Tool, we would consider them too.

2. 
   Model
   
   1. 
      Model name
      

Different models are used EURAD-IM, CHIMERE, CAMx, TAPM

2. 
   Reference to MDS if available
   

EURAD-IM: http://pandora.meng.auth.gr/mds/showshort.php?id=169

CHIMERE: http://pandora.meng.auth.gr/mds/showlong.php?id=144

CAMx: http://pandora.meng.auth.gr/mds/showshort.php?id=141

TAPM: http://pandora.meng.auth.gr/mds/showlong.php?id=120

3. 
   Test-case
   
   1. 
      Spatial resolution and spatial domain
      

Portugal (9 km x9 km; 3 km x3 km); Porto and Lisbon urban areas (1 km x 1 km)

2. 
   Temporal resolution
   

1 hour

3. 
   Pollutants considered
   

NO2, O3, PM10, PM2.5

4. 
   Data assimilation, if yes methodology used
   

Not used

Evaluation

1. 
   How did you select the stations used for evaluation?
   

Stations were select according to the data collection efficiency (> 75%) and the type of environment: traffic stations were only included in the urban scale model validation (Porto and Lisbon domains with 1x1 km2). For the other regional scale application we just use background stations (representative of the model grid).

2. 
   In case of data-assimilation, how are the evaluation results prepared?
   

Not applicable

3. 
   Please comment the DELTA performance report templates
   

Report templates are an excellent product of DELTA but they still need some improvements to be clearly understood, in particular by the air quality managers, more specifically with respect to the identification of stations and the inclusion of more pollutants to the analysis.

Feedback

1. 
   What is your overall experience with DELTA?
   

The UAVR experience with the DELTA Tool is based on several model validation exercises that we performed, together with some intercomparison modelling work. This experience involves several model types (EURAD, CHIMERE, CAMx, TAPM), besides all regional scale models, for different type of pollutants (O₃, PM₁₀, PM_2.5, NO₂) and different spatial domains (Portugal; Porto; Lisbon; Aveiro; …).

Our experience with DELTA is quite positive and we are using it more and more often. DELTA is well documented and relatively easy to apply. The chance to have a common evaluation framework is very well acknowledged and our national air quality management entities receive now model evaluation results based on DELTA and accept these with confidence.

About the things to be improved, we think DELTA should cover all the evaluation aspects included in the Directive:

• 
  Extend the tool to all pollutants of the Directive
  
• 
  Consider a section for AQ assessment prepared to work with all Directive thresholds;
  
• 
  Consider a section for AQP and its scenarios evaluation (incorporating the Planning Tool that is being developed in work group 4 (WG4) of FAIRMODE);
  
• 
  Consider a section for forecasting purposes with specific model skill/scores (which is already being prepared by INERIS).
  

2. 
   How do you compare the benchmarking report of DELTA with the evaluation procedure you normally use? Please briefly describe the procedure you normally use for model evaluation?
   

Before the DELTA Tool, UAVR performed their model validations using a group of three main statistical parameters (namely BIAS, correlation factor and RMSE) following the work of Borrego et al.⁵ produced in the scope of the AIR4EU project (http://www.air4eu.nl/).

3. 
   What do you miss in the DELTA benchmarking report and/or which information do you find unnecessary
   

The following are missing according to UAVR:

• Other pollutants, like CO, SO2, benzene, …

• Distinction of the monitoring sites (difficult to identify the different sites in some graphs/table summary report

• Easy to confuse the traditional parameters and the new ones, since the name is the same (BIAS, Standard Deviation and correlation)

7.5.TCAM evaluation with DELTA tool

Claudio Carnevale (UNIBS, Brescia, Italy)

Background Information

1. 
   What is the context of your work:
   
   1. 
      Frame of the modelling exercise (Air Quality Plan, research project, …?)
      

FAIRMODE Work Group 1 and an internal project at the UNIBS

2. 
   Scope of the exercise (pollutants, episodes…)
   

Application of the methodology to a “real” modelling case. A sensitivity analysis is also performed on the parameters used for the computation of the observation uncertainty.

2. 
   Model
   
   1. 
      Model name
      

TCAM (Transport Chemical Aerosol Model)

2. 
   Main assumptions
   

Horizontal Transport: Chapeau Function (+ Forester FIlter), Vertical Transport: Crack-Nicholson hybrid scheme, Deposition: Wet & Dry, Gas Chemistry: SAPRC mechanism, Aerosol: Condensation/Evaporation, Nucleation, Acqueous Chemistry

3. 
   I/O
   

Emission Inventory: POMI project, 2005

Meteorology: MM5 2005 output provided by JRC in the frame of POMI project,

Boundary Condition: Chimère 2005 BC provided in the frame of POMI project

4. 
   Reference to MDS if available
   

http://pandora.meng.auth.gr/mds/showlong.php?id=147

3. 
   Test-case
   
   1. 
      Spatial resolution and spatial domain
      

6 kmx6 km resolution over Northern Italy

2. 
   Temporal resolution
   

Daily

3. 
   Pollutants considered
   

PM10

4. 
   Data assimilation, if yes methodology used
   

Not used

Evaluation

1. 
   How did you select the stations used for evaluation?
   

Observations from approximately 50 monitoring sites located in the Po Valley have been used. The sites have been classified in terms of station type (suburban, urban, and rural). The orography (hilly, plane, valley) is also specified. Monitoring data are the same as those used in the model intercomparison exercise (POMI) performed for year 2005.

2. 
   In case of data-assimilation, how are the evaluation results prepared?
   

Not used

3. 
   Please comment the DELTA performance report templates
   

No feedback (“Target plot and MQO plot used only”)

Feedback

1. 
   What is your overall experience with DELTA?
   

Comments on tool not the procedure

“Useful for visualizing all main statistical indicators and for summarizing the results of the evaluation in specific statistic tables. It also provides a wide range of plots (scatter, time series, Taylor and target diagrams), which helps to tell whether the overall model response is actually acceptable for regulatory purposes according to the AQD (2008) guidelines.”

2. 
   How do you compare the benchmarking report of DELTA with the evaluation procedure you normally use? Please briefly describe the procedure you normally use for model evaluation?
   

Without the DELTA tool, the evaluation is usually performed in our cases on statistical indexes (correlation, RMSE, bias etc…) and on exceedance days modelling without considering the uncertainty in the measurements.

3. 
   What do you miss in the DELTA benchmarking report and/or which information do you find unnecessary
   

No feedback

7.6.UK feedback Ricardo AEA

Keith Vincent

The feedback is based on a comparison that was made between the DELTA 4.0 implementation by JRC and a spreadsheet calculation.

Background Information

1. 
   What is the context of your work?
   
   1. 
      Frame of the modelling exercise (Air Quality Plan, research project, …?)
      

Evaluation of the PCM modelled results produced as part of the annual AQ compliance for 2013 for the UK

2. 
   Scope of the exercise (pollutants, episodes…)
   

This evaluation is carried out for NO₂, PM₁₀ and PM_2.5 concentrations.

2. 
   Model
   
   1. 
      Model name
      

The Pollution Climate Mapping (PCM) model is a collection of models designed to fulfil part of the UK's EU Directive (2008/50/EC) requirements to report on the concentrations of particular pollutants in the atmosphere.

2. 
   Main assumptions
   

Not provided

3. 
   I/O
   

Not provided

4. 
   Reference to MDS if available
   

Not provided

3. 
   Test-case
   
   1. 
      Spatial resolution and spatial domain
      

The modelling is for the UK, the resolution is 1km x 1km.

2. 
   Temporal resolution
   

Annual average concentrations

3. 
   Pollutants considered
   

NO₂, PM₁₀ and PM_2.5

4. 
   Data assimilation, if yes methodology used
   

Not used

Evaluation

1. 
   How did you select the stations used for evaluation?
   

This evaluation is carried out for NO2, PM10 and PM2.5 concentrations predicted at both non-traffic (background + industrial) and traffic locations. This is because different models are used to predict concentrations for the respective locations.

2. 
   In case of data-assimilation, how are the evaluation results prepared?
   

Not applicable

3. 
   Please comment the DELTA performance report templates (10L per report)
   

No feedback

Feedback

1. 
   What is your overall experience with DELTA?
   

Ricardo-AEA has for a number of years played a supporting role in assessing and understanding the usefulness of the MQOs based on measurement uncertainty. A spreadsheet tool (spreadsheet_deltatool_v4.xls) has been developed by Ricardo-AEA and this replicates some of the functionality provided by the Delta tool. This has provided a degree of confidence in how the Delta tool has been applied. ( drawbacks/advantages of the method are not provided)

2. 
   How do you compare the benchmarking report of DELTA with the evaluation procedure you normally use? Please briefly describe the procedure you normally use for model evaluation?
   

No information is given on the normal procedure at Ricardo AEA. The results of the latest implementation of DELTA are compared to those of spreadsheet_deltatool_v4.xls. There seems to be a slight difference in how the fulfilment criteria is calculated between the two implementations. It is noticed that the N_p and N_np parameters seem to be treated as integers in DELTA 4.0. The parameters used for PM (u_r^LV , α) should also be changed depending on the measurement technique that is used.

3. 
   What do you miss in the DELTA benchmarking report and/or which information do you find unnecessary
   

No feedback

8.References

8.1.Peer reviewed articles:

1. 
   Applying the Delta tool to support AQD: The validation of the TCAM chemical transport model, C. Carnevale, G. Finzi, A. Pederzoli, E.Pisoni, P. Thunis, E.Turrini, M.Volta, Air Quality, Atmosphere and Health , 10.1007/s11869-014-0240-4, 2014
   
2. 
   Model quality objectives based on measurement uncertainty. Part I: Ozone, P. Thunis, D. Pernigotti and M. Gerboles, 2013. Atmospheric Environment, 79 (2013) 861-868.
   
3. 
   Model quality objectives based on measurement uncertainty. Part II: PM10 and NO2, D. Pernigotti, P. Thunis, C. Belis and M. Gerboles, 2013. Atmospheric Environment, 79 (2013) 869-878.
   
4. 
   Performance criteria to evaluate air quality modelling applications. P. Thunis, A. Pederzoli, D. Pernigotti. Atmospheric Environment, 59, 476-482, 2012.
   
5. 
   A tool to evaluate air quality model performances in regulatory applications, P. Thunis, E. Georgieva, A. Pederzoli, Environmental Modelling & Software, 38, 220-230, 2012
   
6. 
   A methodology for the evaluation of re-analysed PM10 concentration fields: a case study over the Po Valley, C. Carnevale, G. Finzi, A. Pederzoli, E. Pisoni, P. Thunis, E. Turrini, M. Volta. Air quality Atmosphere and Health, in press
   

8.2.Reports/ working documents / user manuals:

7. 
   FAIRMODE SG4 Report Model quality objectives Template performance report & DELTA updates P. Thunis, A. Pederzoli, D. Pernigotti March 2012. http://fairmode.jrc.ec.europa.eu/document/fairmode/WG1/FAIRMODE_SG4_Report_March2012.pdf
   
8. 
   Modeling quality objectives in the framework of the FAIRMODE project: working document. D. Pernigotti, P. Thunis and M. Gerboles, 2014.
   
9. 
   (http://fairmode.jrc.ec.europa.eu/document/fairmode/WG1/Working%20note_MQO.pdf)
   
10. 
    The DELTA tool and Benchmarking Report template Concepts and User guide P. Thunis, E. Georgieva, A. Pederzoli Joint Research Centre, Ispra Version 2 04 April 2011http://fairmode.jrc.ec.europa.eu/document/fairmode/WG1/FAIRMODE_SG4_Report_April2011.pdf
    
11. 
    A procedure for air quality models benchmarking P. Thunis, E. Georgieva, S. Galmarini Joint Research Centre, Ispra Version 2 16 February 2011 http://fairmode.jrc.ec.europa.eu/document/fairmode/WG1/WG2_SG4_benchmarking_V2.pdf
    
12. 
    DELTA Version 4.0 Concepts / User’s Guide / Diagrams P. Thunis, C. Cuvelier, A. Pederzoli, E. Georgieva, D. Pernigotti, B. Degraeuwe Joint Research Centre, Ispra, September 2014
    

8.3.Other documents/ e-mail:

13. 
    Feedback on Model Quality Objective formulation D.Brookes, J. Stedman, K. Vincent, B. Stacey, Ricardo-AEA, 18/06/14.
    
14. 
    Mail correspondence between RIVM – The Netherlands (J. Wesseling) and JRC (P.Thunis)
    

1^ Vlãsceanu, L., Grünberg, L., and Pârlea, D., 2004, /Quality Assurance and Accreditation: A Glossary of Basic Terms and Definitions /(Bucharest, UNESCO-CEPES) Papers on Higher Education, ISBN 92-9069-178-6. http://www.cepes.ro/publications/Default.htm

2^ The Forum for Air quality Modeling (FAIRMODE) is an initiative to bring together air quality modelers and users in order to promote and support the harmonized use of models by EU Member States, with emphasis on model application under the European Air Quality Directives. FAIRMODE is currently being chaired by JRC.

3^ JCGM, 2008. Evaluation of Measurement Data - Guide to the Expression of Uncertainty in Measurement.

4^ Confidence Interval

5^ BORREGO, C., MONTEIRO, A., FERREIRA, J., MIRANDA, A.I., COSTA, A.M., CARVALHO, A.C., LOPES, M. (2008). Procedures for estimation of modelling uncertainty in air quality assessment. Environment International 34, 613–620.