Guidance Document on Model Quality Objectives and Benchmarking

5.5.Open issues

In this section a few topics are introduced on which there currently is no consensus yet but which merit further consideration.

3. Data assimilation

The AQD suggests the integrated use of modelling techniques and measurements to provide suitable information about the spatial and temporal distribution of pollutant concentrations. When it comes to validating these integrated data, different approaches can be found in literature which are based on dividing the set of measurement data into two groups, one for the integration and one for the evaluation of the integrated fields. The challenge is how to select the set of validation stations. By repeating the procedure e.g. using a Monte Carlo approach until all stations have been included at least once in the evaluation group, validation is then possible for all stations. As a specific case the ‘leaving one out’ method can be mentioned in which all stations are included in the integration except for the single station that we want to validate. By repeating this procedure for each station in turn, all stations can be validated ‘Leaving one out’ therefore requires as many re-analyses as there are stations. It is currently investigated within FAIRMODEs Cross Cutting Activity Modelling & Measurements which of the methodologies is most robust and applicable in operational contexts.

4. Station representativeness

In the current approach only the uncertainty related to the measurement device is accounted for but another source of divergence between model results and measurements is linked to the lack of spatial representativeness of a given measurement station. Although objectives regarding the spatial representativeness of monitoring stations are set in the AQD these are not always fulfilled in real world conditions. The formulation proposed for the MQO and MPC could be extended to account for the lack of spatial representativeness if quantitative information on the effect of station (type) representativeness on measurement uncertainty becomes available.

5. Handling changes in observation data uncertainty

As defined in 5.2 the MQO depends on the observation data uncertainty. As measurement techniques improve this observation data uncertainty will likely reduce over time. A consequence of this could be that a model that produced results that complied to the MQO based on a set of measurements could have a problem fulfilling the MQO for a new set of measurements obtained using the improved technique. A clear procedure is thus needed on how to define and update the different parameters needed for quantifying the observation data uncertainty.

6. Performance criteria for high percentile values

The model quality objective described above provides insight on the quality of the model average performances but does not inform on the model capability to reproduce extreme events (e.g. exceedances). For this purpose, a specific indicator is proposed as:

(29)
where “perc” is a selected percentile value and M_perc and O_perc are the modelled and observed values corresponding to this selected percentile. The denominator, U(O_perc) is directly given as a function of the measurement uncertainty characterizing the O_perc value. For pollutants for which exceedance limit values exist in the legislation this percentile is chosen according to legislation.
For hourly NO₂ this is the 99.8% (19^th occurrence in 8760 hours), for the 8h daily maximum O₃ 92.9% (26^th occurrence in 365 days) and for daily PM₁₀ and PM_2.5 90.1% (36^th occurrence in 365 days). For general application, when e.g. there is no specific limit value for the number of exceedances defined in legislation, the 95% percentile is proposed. To calculate the percentile uncertainty used in the calculation of the equation 22 is used with .

7. Data availability

Currently a value of 75% is required in the benchmarking both for the period considered as a whole and when time averaging operations are performed for all pollutants.

The Data Quality Objectives in Annex I of the AQD require a minimum measurement data capture of 90% for sulphur and nitrogen oxides, particulate matter (PM), CO and ozone. For ozone this is relaxed to 75% in winter time. For benzene the Directive specifies a 90 % data capture (dc) and 35% time coverage (tc) for urban and traffic stations and 90% tc for industrial sites. The 2004 Directive in Annex IV requires 90% dc for As, Cd and Ni and 50% tc and for BaP 90 % dc of 33% tc.

As these requirements for minimum data capture and time coverage do not include losses of data due to the regular calibration or the normal maintenance of the instrumentation the minimum data capture requirements are in accordance with the Commission implementing decision of 12 December 2011 laying down rules for the AQD reduced by an additional 5%. In case of e.g. PM this further reduces the data capture to 85% instead of 90%.

In addition, in Annex XI the AQD provides criteria for checking validity when aggregating data and calculating statistical parameters. When calculating hourly averages, eight hourly averages and daily averages based on hourly values or eight hourly averages, the requested percentage of available data is set to 75%. For example a daily average will only be calculated if data for 18 hours are available. Similarly O₃ daily maximum eight hourly average can only be calculated if 18 eight hourly values are available each of which requires 6 hourly values to be available. This 75% availability is also required from the paired modelled and observed values. For yearly averages Annex XI of the AQD requires 90 % of the one hour values or - if these are not available - 24-hour values over the year to be available. As this requirement again does not account for data losses due to regular calibration or normal maintenance, the 90% should in line with the implementing decision above again further be reduced by 5% to 85%.

In the assessment work presented in the EEA air quality in Europe reports we can find other criteria. There, we find the criteria of 75% of valid data for PM10, PM2.5, NO2, SO2, O3, and CO, 50% for benzene and 14 % for BaP, Ni, As, Pb, and Cd. In these cases you also have to assure that the measurement data is evenly and randomly distributed across the year and week days.

8. 
   MPC fulfilment criteria: improved statistical basis for the MQO
   

By considering the requirement that the MPC should be fulfilled in at least 90% of the observation stations as a requirement for the confidence interval of the differences between observed and modelled values an alternative basis for the MQO can be derived.

The Model Quality Objective () is derived above with the simple principle of allowing a similar margin of tolerance to both model and observations. Assume a set of normal distributed data pairs consisting of observations and model calculations. The standard deviations of the observations and modelled concentrations are _O and _M. As the observations and model calculations are assumed to follow a normal distribution, their difference does too. For the MQO we define that 90% of the differences between observations and model results must be between -2 _ and +2 _. In statistical terms: the 90% CI⁴ is 4_. From the above it follows that the 95%CI of the concentrations differences, defined by 2_d, is given by (2.0/1.64) 4 _O. The factor 2.0/1.64 = 1.22 takes into account the difference between the 90%CI and 95%CI. The standard deviation (_d) of the distribution of the differences can be expressed as _d = _O, with  = 2.44.

It is furthermore evident that

_d)² = (_O)² + (_M)²

or: _O)² = (_O)² + (_M)²

or: ² – 1) _O² = _M²

and, finally: _M √² – 1) _O

Numerically:

_M √² – 1) _O ≈ 2.3 _O .

Conclusion: the present Model Quality Objective (MQO) implies that the uncertainty in the model result can be up to roughly twice as high as the measurement uncertainty.

9. Application of the procedure to other parameters

Currently only PM, O₃ and NO₂ have been considered but the methodology could be extended to other pollutants such as heavy metals and polyaromatic hydrocarbons which are considered in the Ambient Air Quality Directive 2004/107/EC.

The focus in this document is clearly on applications related to the AQD and thus those pollutants and temporal scales relevant to the AQD. However the procedure can off course be extended to other variables including meteorological data as proposed in Pernigotti et al. (2014)

In Table below values are proposed for the parameters in (23) and (29) for wind speed and temperature data.

Table List of the parameters used to calculate the uncertainty for the variables wind speed (WS) and temperature (TEMP)

				α
WS (test)	2.00	0.130	5 m/s	0.800	NA	NA
TEMP (test)	2.00	0.025	25 K	1.000	NA	NA

When performing validation using the Delta Tool, it is helpful to look at both NO_x as well as NO₂, as the former pollutant is less influenced by chemistry, and is therefore a better measure of the models’ ability to represent dispersion processes. The NOx uncertainty is not available but could be approximated by the NO₂ uncertainty for now. (Table ).