13

Few things in life give me pleasure like scraping structured and unstructured data from the Internet and making use of it in my models.

For instance, the Data Science Toolkit (or RDSTK for R programmers) allows me to pull lots of good location-based data using IP's or addresses and the tm.webmining.plugin for R's tm package makes scraping financial and news data straightfoward. When going beyond such (semi-) structured data I tend to use XPath.

However, I'm constantly getting throttled by limits on the number of queries you're allowed to make. I think Google limits me to about 50,000 requests per 24 hours, which is a problem for Big Data.

From a technical perspective getting around these limits is easy -- just switch IP addresses and purge other identifiers from your environment. However, this presents both ethical and financial concerns (I think?).

Is there a solution that I'm overlooking?

Dawny33
  • 8,226
  • 12
  • 47
  • 104
Hack-R
  • 1,919
  • 1
  • 21
  • 34

2 Answers2

6

For many APIs (most I've seen) ratelimiting is a function of your API Key or OAuth credentials. (Google, Twitter, NOAA, Yahoo, Facebook, etc.) The good news is you won't need to spoof your IP, you just need to swap out credentials as they hit there rate limit.

A bit of shameless self promotion here but I wrote a python package specifically for handling this problem.

https://github.com/rawkintrevo/angemilner

https://pypi.python.org/pypi/angemilner/0.2.0

It requires a mongodb daemon and basically you make a page for each one of your keys. So you have 4 email addresses each with a separate key assigned. When you load the key in you specify the maximum calls per day and minimum time between uses.

Load keys:

from angemilner import APIKeyLibrarian
l= APIKeyLibrarian()
l.new_api_key("your_assigned_key1", 'noaa', 1000, .2)
l.new_api_key("your_assigned_key2", 'noaa', 1000, .2)

Then when you run your scraper for instance the NOAA api:

url= 'http://www.ncdc.noaa.gov/cdo-web/api/v2/stations' 
payload= {  'limit': 1000,
        'datasetid':  'GHCND', 
        'startdate': '1999-01-01' }

r = requests.get(url, params=payload, headers= {'token': 'your_assigned_key'})

becomes:

url= 'http://www.ncdc.noaa.gov/cdo-web/api/v2/stations'
payload= {  'limit': 1000,
            'datasetid':  'GHCND',
            'startdate': '1999-01-01' }

r = requests.get(url, params=payload, headers= {'token': l.check_out_api_key('noaa')['key']})

so if you have 5 keys, l.check_out_api_key returns the key that has the least uses and waits until enough time has elapsed for it to be used again.

Finally to see how often your keys have been used / remaining useage available:

pprint(l.summary())

I didn't write this for R because most scraping is done in python (most of MY scraping). It could be easily ported.

Thats how you can technically get around rate limiting. Ethically ...

UPDATE The example uses Google Places API here

rawkintrevo
  • 268
  • 1
  • 7
1

I've found a fantastic way to get around IP address blocks is Amazon AWS (Or Azure or any other similar on-demand service). A t2.nano instance from AWS costs almost the same as a high quality proxy and is more than capable of handling rate-limited requests just fine.

For example, let's say you need to scrape 100,000 pages. Using the AWS CLI, your program can automatically start 1,000 instances. Even if you need to wait 2 seconds between requests, you'd still be done in 200 seconds. And how much do you pay for it?

Right now, the price for a t2.nano instance is \$0.0058 per Hour in the Ohio region of AWS. For a thousand instances, that's only \$5.8 per hour. But you don't need the entire hour. Your job of 100,000 pages was finished in less than 200 seconds. Add some extra time for setting up the script, installing required packages, zipping the results and downloading them to your server/pc and you still used at most 10 minutes of server time per instance.

Or about one dollar. 100,000 pages in 200 seconds for one dollar. Not bad.

Note: When scaling like this, you have to be very careful about not accidentally overloading the scraping target. If you unleash this much firepower on a single website, that's about 1,000 requests hitting the server every alternate second. Enough to kill most webservers. So while the 1,000 servers option may be a good idea if you have a diversified list of sites, you'll probably need to use 10-20 servers at maximum if you're hitting a single site.

Aayush Agrawal
  • 111
  • 2