Down Boy: How to easily throttle requests to an API using Redis
Last week Eric and I needed to gather historic weather data from the Weather Underground API. We needed to retrieve large amounts of data and because we are pre-loading the data we do not mind spacing the collection out over a few days in order to minimize costs for our client.
Weather Underground's pricing is reasonable until you need the historical weather data add-on that bumps it to over $500 a month. With this in mind, we challenged ourselves to provide strategic value to our client by throttling our API usage.
The free plan allows you up to 500 requests per day and only 10 request per minute. If you exceed the 10 request per minute limit your API key gets suspended for the day. So we need to be able to keep track of both
Thankfully, Redis makes this rather easy with the INCR command.
redis> SET mykey "10" OK redis> INCR mykey (integer) 11 redis> GET mykey "11"
The example above comes from the Redis commands page. What it does not tell you, however, is that when you
INCR a key that does not exist it treats the nil as 0 and sets the key to 1.
redis> GET foo (nil) redis> INCR foo (integer) 1 redis> INCR foo (integer) 2
With this knowlege we can easily keep track of the number of calls to the API we make. The next step is to make sure they are reset appropriately, another place where Redis really makes this easy.
The Redis command
EXPIRE tells Redis to
DEL the key after a certain amount of seconds. So when we increment our counter we set it to expire after 60 seconds if we are at the first increment.
def increment_threshold redis.expire(:threshold, 60) if redis.incr(:threshold) == 1 end
We can add a check in our worker to idle until under our limits.
def perform sleep 1 while above_threshold? increment_threshold # Do work end def above_threshold? redis.get(:threshold) >= 10 end
Redis will automatically delete the key after 60 seconds and the next time we increment our threshold counter it will be set to 1 and expire in another 60 seconds.
We also needed to make sure that we did not exceed the allotted number of daily requests. This limit is reset by Weather Underground each day at midnight EST. We can easily achieve the same effect as the threshold with a set time using Redis'
EXPIREAT is the same as
EXPIRE except it takes a specific unix timestamp.
def increment_daily_count if redis.incr(:daily_count) == 1 redis.expireat(:daily_count, (Date.today + 1).to_time.to_i) end end
Now all we need to do is update the sleep conditional and add the
increment\_daily\_count to before we make the API call.
def perform sleep 1 while at_limit? increment_counters # Do API Calls end def at_limit? above_daily_limit? || above_threshold? end def above_daily_limit? redis.get(:daily_count) >= 500 end def above_threshold? redis.get(:threshold) >= 10 end def increment_counters increment_threshold increment_daily_count end def increment_daily_count if redis.incr(:daily_count) == 1 redis.expireat(:daily_count, (Date.today + 1).to_time.to_i) end end def increment_threshold redis.expire(:threshold, 60) if redis.incr(:threshold) == 1 end
With our API calls throttled, we can safely spin up workers without having to worry about going over our threshold or our daily limit. Of course it would be best to change all of the numbers in this code to constants that can be set through a setting file or initializers.