Why your Google Trends scraper breaks after 40 keywords (and what fixed it for me)
Google Trends has no official public API, so most people scrape it, either with pytrends or with a hosted scraper. It works fine for the first few dozen keywords. Then everything starts coming back as 429 Too Many Requests, and it stays that way.
I hit exactly this while building a Google Trends tool, so here's what I measured.
The test
I ran the same job, 50 everyday keywords with 12 months of interest over time each, in a few different setups:
| Where the requests came from | Keywords that returned data |
|---|---|
| A cloud server, no proxy, one keyword at a time | 50 / 50 (2 needed a retry) |
| The same server, 3 keywords in parallel | blocked almost immediately |
| A shared datacenter proxy pool (5 IPs) | 44 / 50, then blocked |
| Residential proxies | 50 / 50, zero 429s |
The datacenter run is the interesting one. The first 44 keywords went through. After that, every request got a 429, including retries from fresh IPs in the same pool, and the pool was still blocked 13 minutes later. It isn't a per-request limit you can wait out. Google throttles the whole range of addresses.
What actually helped
- Residential IPs. This was the biggest single fix. Google treats residential traffic like normal users, and the run went from 88% to 100% of keywords returned.
- Reuse the session. The explore endpoint needs a cookie from the Trends page. Loading that page for every keyword almost triples your traffic. One cookie jar per proxy session cut my traffic by about 64%.
- Retry on a new IP, not the same one. A 429 on one IP rarely clears in seconds. Rotate the session, back off, and try again.
- Never return half a result. A keyword needs several requests: the timeline, regions and related queries. If one of them fails, retry the whole keyword instead of saving a row with a silently empty section.
The data you get
For each keyword:
- interest over time, 0 to 100 per week or day;
- interest by country, or by subregion, metro and city when you pick a country;
- related queries, top and rising, including "Breakout" ones.
One honest caveat: in my tests Google returned related topics empty for these automated requests.
If you just want the data
I packaged all of this as an Apify Actor: search "Google Trends Scraper & API (Reliable)" on the Apify Store, by cleanrows. It costs $2 per 1,000 keywords with the proxies included, and keywords that fail aren't charged. If you'd rather build your own, the four points above are what matter.
Happy to answer questions about the endpoints in the comments.