Guidance Document on Model Quality Objectives and Benchmarking

4.Overview of existing literature

4.1.Introduction

The development of the procedure for air quality model benchmarking in the context of the AQD has been an ongoing activity in the context of the FAIRMODE² community that has been led by JRC. The JRC has also developed the DELTA tool in which the Model Performance Criteria (MPC) and Model Quality Objective (MQO) are implemented. Other implementations of the MPC and MQO are found in the CERC Myair toolkit and the on-line ATMOSYS Model Evaluation tool developed by VITO.

In the following paragraphs a chronological overview is given of the different articles and documents that have led to the current form of the Model Performance Criteria and Model Quality Objective. Starting from a definition of the MPC and MQO in which the measurement uncertainty is assumed constant (Thunis et al., 2012) this is further refined with more realistic estimates of the uncertainty for O₃ (Thunis et al., 2013) and NO_x and PM₁₀ (Pernigotti et al., 2013). The DELTA tool itself and an application of this tool are respectively described in Thunis et al., 2013, Carnevale et al., 2013 and Carnevale et al., 2014. Full references to these articles can be found at the end of this document.

4.2.Literature on how these model performance criteria and model quality objectives are defined.

Thunis et al., 2011: A procedure for air quality model benchmarking

This document was produced in the context of the work done in the Subgroup 4 (SG4) of Working Group 2 (WG2) of FAIRMODE. The objective was to develop a procedure for the benchmarking of air quality models in order to evaluate their performances and indicate a way for improvements. The document first gives a global overview of the proposed approach by presenting the prerequisites, the four key elements envisioned (the DELTA tool, the ENSEMBLE tool, an online benchmarking service and an extraction facility) and a description of the procedure that focuses on how the different facilities could help in the model performance evaluation.

Some key concepts underlying the procedure are presented next: 1) the application domain which is the EU Air Quality Directive (AQD, 2008), 2) the need for input data consistency checks, 3) not only model to observation comparison but also model intercomparison and model response evaluation, 4) use of a limited set of model performance indicators that are assessed with respect to criteria and goals, 5) the aim of the procedure which is to provide model user with feedback and 6) the automatic reporting system of the benchmarking service.

The final section of the article is devoted to the methodology for the benchmarking service, the different testing levels and the goals, criteria and the observation uncertainty considered in the evaluation as well as a proposal for the automatic report. The document concludes with a number of annexes on the application domain (pollutants and scales), the statistics and charts, the different spatial and temporal aggregations for model results and performance criteria and goals.

Thunis et al., 2012: Performance criteria to evaluate air quality modelling applications

This article introduces the methodology in which the root mean square error (RMSE) is proposed as the key statistical indicator for air quality model evaluation. Model Performance Criteria (MPC) to investigate whether model results are ‘good enough’ for a given application are calculated based on the observation uncertainty (U). The basic concept is to allow the same margin of tolerance (in terms of uncertainty) for air quality model results as for observations. As the objective of the article is to present the methodology and not to focus on the actual values obtained for the MPC, U is assumed to be independent of the concentration level and is set according to the data quality objective (DQO) value of the Air Quality Directive (respectively 15, 15 and 25% for O₃, NO₂ and PM₁₀). Existing composite diagrams are then adapted to visualize model performance in terms of the proposed MPC. More specifically a normalized version of the Target diagram, the scatter plot for the bias and two new diagrams to represent the standard deviation and the correlation performance are considered. The proposed diagrams are finally applied and tested on a real case

Thunis et al., 2013: Model quality objectives based on measurement uncertainty.
Part I: Ozone

Whereas in Thunis et al., 2012 the measurement uncertainty was assumed to remain constant regardless of the concentration level and based on the DQO, this assumption is dropped in this article. Thunis et al., 2013 proposes a formulation to provide more realistic estimates of the measurement uncertainty for O₃ accounting for dependencies on pollutant concentration. The article starts from the assumption that the combined measurement uncertainty can be decomposed into non-proportional (i.e. independent from the measured concentration) and proportional fractions which can be used in a linear expression that relates the uncertainty to known quantities specific to the measured concentration time series. To determine the slope and intercept of this linear expression, the different quantities contributing to the uncertainty are analysed according to the direct approach or GUM³ methodology. This methodology considers the individual contributions to the measurement uncertainty for O₃ of the linear calibration, UV photometry, sampling losses and other sources. The standard uncertainty of all these input quantities is determined separately and these are subsequently combined according to the law of propagation of errors. Based on the new linear relationship for the uncertainty more accurate values for the MQO and MPC are calculated for O₃.

Pernigotti et al., 2013: Model quality objectives based on measurement uncertainty. Part II: PM10 and NO2

The approach presented for O₃ in Thunis et al., 2013 is in this paper applied to NO₂ and PM₁₀ but using different techniques for the uncertainty estimation. For NO₂ which is not measured directly but is obtained as the difference between NO_x and NO, the GUM methodology is applied to NO and NO_x separately and the uncertainty for NO₂ is obtained by combining the uncertainties for NO and NO_x. For PM which is operationally defined as the mass of the suspended material collected on a filter and determined by gravimetry there are limitations to estimate the uncertainty with the GUM approach. Moreover, most of the monitoring network data are collected with methods differing from the reference one (e.g. automatic analysers), so-called equivalent methods. For these reasons the approach based on the guide for demonstration of equivalence (GDE) using parallel measurements is adopted to estimate the uncertainties related to the various PM₁₀ measurements methods. These analyses result in the determination of linear expressions which can be used to derive the MQO and MPC. The Authors also generalise the methodology to provide uncertainty estimates for time-averaged concentrations (yearly NO₂ and PM₁₀ averages) taking into account the reduction of the uncertainty due to this time averaging.

Pernigotti et al., 2014: Modelling quality objectives in the framework of the FAIRMODE project: working document

This document corrects some errors found in the calculation of the NO₂ uncertainty in Pernigotti et al., 2013 and assesses the robustness of the corrected expression. In a second part, the validity of an assumption underlying the derivation of the yearly average NO₂ and PM₁₀ MQO in which a linear relationship is assumed between the averaged concentration and the standard deviation is investigated. Finally, the document also presents an extension of the methodology for PM_2.5 and NOx and a preliminary attempt to also extend the methodology for wind and temperature.