Get faster access to real-time air quality data from around the world.
Get data pushed to you, rather than pulling it.
You may have seen a recent article here talking about how you can access all the older data collected by OpenAQ. As of today, that’s just over 300 million measurements from around the world, many of which are no longer available from their sources, that you can run large queries against. And of course, we have a RESTful API that allows you to query the latest data with a variety of parameters like location, city, and date as well as others.
But both of these methods require an action on your part, they’re a pull mechanism. What if you could get the data via a push mechanism? We’re happy to announce the availability of a real-time notification service for all the data flowing through the OpenAQ platform. With this mechanism, you can have just the data you want being sent directly to your applications. You no longer need to periodically check the API for updates, you’ll know within seconds of the data being available. And the updates are filterable, so if you only want a certain city or only want values above a threshold, that’s possible. Read on for more details!
This mechanism is based on the Amazon Simple Notification Service (SNS). With it, we can send notifications whenever new data is ingested into our system. This is currently around half a million notifications a day, but we’ll show you in a bit how to provide a filter to have your app listen to just the data you’re interested in.
Data Format
So what does this data look like? It’s just a JSON object, so easy to work with in your code. An example is below for one measurement, which corresponds to one notification.
To work with this data, you can use AWS Lambda or Amazon SQS to interact with the messages. If you’re already using AWS to take advantage of the OpenAQ data being on Amazon S3, you can easily use these services. If you want to have the notifications delivered to your own custom endpoints, you can create a simple Lambda function to listen for new notifications and then just send the data to your own endpoint. Or if you wanted to store all the data into your own system as it’s ingested, you can listen for notifications and insert into your own database, etc. For now, we’re only allowing SNS subscriptions from Lambda and SQS as it does not cost us money, which is something we need to be concerned about as a non-profit.
Getting only the data you want
Some OpenAQ users want to look at all the data we have, others are only interested in specific areas of the world or are only interested when levels rise above (or below certain levels). It’s possible to set a filter when you subscribe to the SNS topic.
Let’s say you only care about a subset of data coming from certain countries, when you subscribe to the topic, you can use a filtering policy like below.
{“country”: [“CN”, “IN”, “MN”]}
What if you only wanted data within a certain geographic bounding box?
{“latitude”: [{“numeric”:[“>”, 30, “<=”, 50]}], “longitude”: [{“numeric”:[“>”, 60, “<=”, 70]}]}
What if you wanted to only have data coming to your application that was above a certain threshold?
{“value”: [{“numeric”:[“>”, 100]}]}
You can see many more examples of how topic filtering works here.
Getting to work
There are two related SNS topics you can use for your needs.
arn:aws:sns:us-east-1:470049585876:OPENAQ_NEW_MEASUREMENT is the topic detailed above and at least one notification is sent out for each new or updated measurement.
arn:aws:sns:us-east-1:470049585876:NewFetchObject is sent a notification whenever a new object is added to the openaq-fetches/realtime S3 bucket. This may be more appropriate if you want to ingest all the data coming from the OpenAQ system as you get less notifications and can work with much more of the data at once.
We’re excited to see what you can do with these new methods of accessing the world’s air quality data! If you have any questions, please reach out to us at info@openaq.org.
