TL;DR—In February 2021, the United States Environmental Protection Agency (USEPA) released their Air Sensor Performance Targets and Testing Protocols for ozone and PM2.5. These protocols provide the most comprehensive guide to the quantitative evaluation of air sensor performance to date. The publication of these performance targets will allow air quality managers to proceed with greater confidence in the quality of data collected by various brands of air sensors, enabling the appropriate use of sensor technology for a wide range of applications. The result will be greater availability of real-time, high spatiotemporal resolution air quality data for air quality managers, researchers, and the general public. The availability of this data will help these parties better understand—and take action to reduce—air pollution around the world.
To learn more about the USEPA performance targets and how Clarity’s sensors perform against their recommended performance metrics, you can download our Guide to Accurate Particulate Matter Measurements: How to Meet USEPA, EU, and Other Performance Standards for Air Sensors.
Low-cost sensors are becoming the standard for non-regulatory supplemental and informational monitoring
For more than a decade, forward-looking researchers and air quality managers have recognized the potential of low-cost sensors (LCS) to democratize air pollution measurement. However, a lack of consensus regarding how to ensure data quality from LCS meant they have historically been used primarily by community groups, citizen scientists, and innovative air quality managers.
Until recently, it has been rare to see data from low-cost sensors used to support policy or operational decisions due to strict performance requirements for the accuracy, precision, and completeness of air quality data. Policymakers have also expressed concern that uncalibrated data from LCS could be misinterpreted by the public, potentially complicating the already complex debate around air quality policy.
The adoption of LCS at scale by cities like London demonstrates that a paradigm shift is underway. LCS are now seen as a valuable part of an air quality manager’s toolkit for a wide range of non-regulatory supplemental and informational monitoring (NSIM) applications. As technological advances in low-cost sensing technology and the calibration techniques used to ensure data accuracy progress, regulators have been working in tandem to establish protocols that would validate the use of LCS for official purposes and enable the reporting of high spatiotemporal resolution air quality data in near-real-time.
With the recent publication of performance targets and testing protocols for air sensors by the United States Environmental Protection Agency (USEPA), there is now a rubric by which air sensor technologies can be evaluated for “fitness for use” for a wide range of NSIM applications. The USEPA and other standardized testing protocols for LCS will serve to legitimize data coming from “hybrid networks” that leverage the strengths of both LCS and monitoring technologies in compliance with the Federal Reference or Federal Equivalent Methods (FRM/FEM).
Toward the adoption of performance protocols for low-cost air sensors
The USEPA performance protocols are the result of many years of expertise-building across academia and the public and private sectors. The timeline below highlights some of the significant milestones in the development of performance protocols for LCS in recent years.
(References: TD Environmental Services | USEPA)
While not as robust as the USEPA certification program for reference and equivalent method (FRM/FEM) applications*, the publication of standardized performance targets and testing protocols for LCS represents a huge step toward the broader understanding—and more effective use—of these technologies.
* It is important to recall that FRM and FEM designation require a rigorous testing process. Instruments with this designation are typically at least an order of magnitude more expensive than an LCS instrument.
How has LCS performance traditionally been evaluated?
Before the availability of the USEPA performance testing protocols and targets, the best resources available to most air quality managers were the results of third-party evaluations conducted by several research centers that conduct standardized testing on commercially available LCS. The results from these testing centers can be a valuable resource to those looking to arrive at a relatively neutral, quantitative understanding of sensor performance—some of the most notable sensor testing centers are included below here.
Who conducts third-party evaluations?
The efforts of these third-party evaluation centers have contributed significantly to the development of the USEPA and other official protocols, but these early testing protocols were not flawless. For example, here are Clarity’s uncalibrated accuracy results* from various testing centers.
Uncalibrated performance of Clarity sensors at testing centers*
These protocols are heterogeneous—note that AirLab uses an entirely different accuracy metric than SCAQMD and the USEPA—and not nearly as comprehensive as the USEPA’s.
While most testing centers report R², this statistic does not provide the full picture when considered in isolation without RMSE, bias, and offset (see our blog on evaluating sensor accuracy for more on the importance of considering factors other than R²). For this reason, while results from these testing centers can serve as useful references, we recommend using the USEPA’s method to arrive at a complete and objective understanding of LCS accuracy.
* Clarity’s Sensing-as-a-Service solution includes Remote Calibration, ensuring data quality superior to the uncalibrated results reported here.
What low-cost air sensor performance targets exist?
The USEPA’s are the first published protocols that specifically target the evaluation of LCS for NSIM applications. With that said, the EU and the UK have also established performance targets for air quality data that serve as requirements across all classes of sensors used for indicative monitoring. For illustrative purposes, we have included some of the existing targets for particulate matter in the table below.
Existing data quality objectives for low-cost particulate matter sensors
* USEPA deployment averages are based on 24-hour averages for PM2.5 concentrations over 30 days
** EU and UK averages are based on 24-hour averages for PM10 or PM2.5 concentrations over 40 days
*** Sensor PM2.5 measurements are the dependent variable (y) with FRM/FEM PM2.5 measurements as the independent variable (x)
**** USEPA notes that this metric is especially important when PM levels are much higher than general US levels (e.g., wildfires)
While the USEPA’s protocol provides more criteria for the evaluation of sensor performance than the EU or UK’s DQOs, both the EU and the UK employ an “Uncertainty” metric that takes many of these criteria into account. All three methods take a reference instrument as the “gold standard”, but importantly, while the USEPA method treats the reference instrument as error-free, the EU & UK methods allow for uncertainty in the reference instrument’s measurements. This helps to ensure that the LCS or other instrument being evaluated is not unfairly penalized for uncertainty in the reference instrument’s measurements.
Further, while the USEPA recommends issuing two separate reports for both raw and calibrated sensor data, the uncertainty metric is the residual uncertainty after any calibrations are applied. At Clarity, we believe that LCS data should always be calibrated to ensure the highest accuracy possible, and see this as an advantage of the approach taken by the EU and the UK.
While the USEPA’s performance targets represent the first comprehensive set of data quality objectives published specifically with LCS in mind, we expect that future standards will look to these three existing protocols for guidance on the most appropriate criteria and target values for sensor performance evaluation.
The USEPA performance protocols are a significant step toward the broader and more effective use of low-cost air sensors
Low-cost air sensors are still a relatively new technology that will continue to benefit from accuracy improvements as they see wider adoption by governments, communities, and businesses around the world. The USEPA’s protocols provide the first standardized, comprehensive benchmark for LCS performance, and represent a significant step toward the development of a standard for LCS data accuracy that will serve to further legitimize LCS for official uses and advance the availability of high spatiotemporal resolution air quality data around the world.
To learn more about the USEPA performance targets and how Clarity’s sensors perform against their recommended performance metrics, you can download our Guide to Accurate Particulate Matter Measurements: How to Meet USEPA, EU, and Other Performance Standards for Air Sensors. In this white paper, we examine the USEPA’s recommendations, compare them with other performance protocols for low-cost sensors, and explain the steps you can take to evaluate the performance of LCS against this rubric. Finally, we outline how to ensure the data from your LCS comply with USEPA and other performance targets for LCS, with a focus on particulate matter (PM).
The availability of performance targets in no way guarantees that all commercially available LCS will meet those targets, especially when looking at uncalibrated data. At Clarity, we provide the only air quality monitoring solution which comes complete with self-reliant hardware, scalable cloud software, and expert calibration services to help ensure the data from your network meet USEPA, EU, and MCERTS data quality objectives. For more on how to conduct a sensor performance evaluation and how Remote Calibration can help ensure data from your LCS meet data quality targets, stay tuned for our upcoming blogs!