Skip to contentScraping Amazon: Proxies, Headers and Rate Limits
Amazon is one of the most-scraped sites on the web and one of the most defensive. Collecting prices, reviews or catalogue data at scale means distributing requests, sending realistic headers, and handling the CAPTCHAs and bot pages Amazon throws back. Here is a practical, honest walkthrough.
How Amazon fights scrapers
Amazon defends its catalogue with layered detection: per-IP rate-limits, the "Robot Check" / "Enter the characters you see" CAPTCHA page, dog error pages, TLS and header fingerprinting, and behavioural checks. Hammer product pages from one address and you will quickly see CAPTCHAs, then throttling or blocks. The site also varies pages by region and A/B tests layouts, so your parser must tolerate change.
Two honest caveats before you start. Amazon's terms prohibit scraping, and Amazon offers official routes — the Product Advertising API and Selling Partner API — for approved data access; weigh those first. And no setup scrapes Amazon reliably forever; the goal is a sustainable collection rate, not invisibility. For the common e-commerce use cases — price monitoring, competitive research — see our price monitoring and Amazon proxy pages.
A sustainable scraping setup
The winning approach mimics many ordinary shoppers:
- Rotate a proxy pool so no single IP exceeds Amazon's tolerance. This is the primary defence against per-IP throttling.
- Send full, realistic headers — User-Agent, Accept, Accept-Language, and keep them internally consistent. A bare requests default header is an instant tell.
- Pace with jitter — randomised delays beat fixed intervals. Prefer a steady trickle over bursts.
- Pin a region — use one exit country and matching Accept-Language and postcode so prices and availability are consistent; switching regions mid-run scrambles your data.
- Detect the bot page — check for the CAPTCHA / Robot Check markers and back off that IP rather than parsing garbage.
- Use a headless browser only when needed — much of a product page is in the initial HTML; reserve heavier tooling for JS-rendered sections.
Amazon scores datacenter IPs, so aggressive scraping is a case where residential proxies often survive longer. s4m sells datacenter proxies; a dedicated IP gives you a stable region for price-consistent collection at modest volumes, but be realistic about datacenter detection at high volume.
Fetch a product page through a proxy
Route through an s4m proxy with a full header set, and detect Amazon's bot page so you can rotate instead of parsing an error.
import requests
proxies = {
"http": "http://USER:[email protected]:3128",
"https": "http://USER:[email protected]:3128",
}
headers = {
"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) "
"AppleWebKit/537.36 (KHTML, like Gecko) "
"Chrome/122.0 Safari/537.36",
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
"Accept-Language": "en-US,en;q=0.9",
}
r = requests.get("https://www.amazon.com/dp/B0EXAMPLE",
headers=headers, proxies=proxies, timeout=20)
if "[email protected]" in r.text or "Robot Check" in r.text:
raise RuntimeError("Hit Amazon bot page - rotate IP and back off")
parse_product(r.text) # your parserQuestions, answered
Is scraping Amazon allowed?
Amazon's Conditions of Use prohibit scraping, and there can be legal risk depending on jurisdiction and data. Amazon provides official APIs (Product Advertising API, Selling Partner API) for approved access. Consider those first and get legal advice for anything at scale.
Why do my Amazon prices keep changing between requests?
Amazon localises prices and availability by region and runs A/B tests. If your proxy pool exits from different countries, you will collect inconsistent data. Pin one exit region and matching Accept-Language and postcode, and a dedicated IP helps keep it stable.
Do datacenter proxies work for Amazon?
For modest, well-paced collection, often yes. Amazon scores datacenter ranges, so at high volume you will see more CAPTCHAs than with residential IPs. s4m sells datacenter proxies; rotation, pacing and realistic headers matter as much as IP type, but no setup is block-proof.
Scrape e-commerce data cleanly
Route product and price scraping through authenticated s4m datacenter proxies, pin a region with a dedicated IP, and pace your jobs. Metered pay-as-you-go, anonymous accounts, crypto accepted.
People found this page by searching for
- scraping Amazon
- scraping Amazon for android
- how to test for DNS leaks
- how to configure scraping Amazon
- how to check scraping Amazon
- ip address
- datacenter vs residential vs mobile proxies
- best vpn kill switch 2026
- how long do free proxies live
- free proxy list
Real search phrases this page answers — the linked ones open the page that covers them in depth.