Skip to contentHow to fix 429 Too Many Requests when scraping
A 429 is a rate limit, not a ban — the server is asking you to slow down. The durable fix combines backoff, honouring Retry-After, and spreading load across a pool of IPs. Here is how, with working Python.
429 is a speed limit, not a wall
HTTP 429 Too Many Requests means you have sent too many requests in a given window and the server is throttling you. It is deliberately temporary — unlike a hard block, a 429 usually clears once you slow down. Many servers include a Retry-After header telling you exactly how many seconds to wait; honouring it is the single most effective fix, because it aligns you with the server's own limit instead of guessing.
There are two levers. The first is rate: send fewer requests per second per IP, add delays, cap concurrency, and back off exponentially when a 429 appears. The second is distribution: spread requests across multiple IPs so each stays under the per-IP limit. Rate control is where you should start — a proxy pool multiplies your throughput but does not excuse hammering a site. s4m provides authenticated datacenter proxies (SOCKS5 1080, HTTP 3128, USER:PASS) to spread load, and this pairs directly with the proxy rotation guide and scraping with requests.
Respect Retry-After, back off, then rotate
A fetch loop that honours Retry-After, backs off exponentially with jitter, and spreads across a proxy pool. Replace USER:PASS.
import random, time
import requests
USER, PASS, HOST = "USER", "PASS", "proxy.s4m.online"
POOL = [
f"socks5h://{USER}:{PASS}@{HOST}:1080",
f"http://{USER}:{PASS}@{HOST}:3128",
]
def polite_get(url, max_tries=6):
for attempt in range(1, max_tries + 1):
proxy = random.choice(POOL)
r = requests.get(url, proxies={"http": proxy, "https": proxy},
timeout=(5, 20))
if r.status_code != 429:
r.raise_for_status()
return r
# 429: prefer the server's Retry-After, else exponential backoff
retry_after = r.headers.get("Retry-After")
if retry_after and retry_after.isdigit():
wait = int(retry_after)
else:
wait = min(2 ** attempt, 60) + random.random() # cap + jitter
print(f"[{attempt}/{max_tries}] 429; waiting {wait:.1f}s")
time.sleep(wait)
raise SystemExit(f"still rate-limited after {max_tries} tries")
# Global throttle: cap how fast you fire, independent of retries.
MIN_INTERVAL = 0.5 # seconds between requests per worker
_last = 0.0
def throttled_get(url):
global _last
delta = time.time() - _last
if delta < MIN_INTERVAL:
time.sleep(MIN_INTERVAL - delta)
_last = time.time()
return polite_get(url)
print(throttled_get("https://httpbin.org/status/200").status_code)The order to fix a 429
Cheapest, most respectful fixes first.
Honour Retry-After
If the 429 response includes a Retry-After header, wait exactly that long before retrying. It is the server telling you its limit — following it clears most 429s immediately.
Add exponential backoff with jitter
When there is no Retry-After, wait 2, 4, 8… seconds (capped) plus a small random jitter between retries, so you stop hammering and avoid synchronised retries.
Throttle your request rate
Cap requests per second and concurrency so you stay under the limit in the first place. A steady, slower crawl finishes more reliably than a burst that keeps getting throttled.
Spread load across IPs
Once your per-request behaviour is polite, rotate through a proxy pool so each IP stays under the per-IP limit. This multiplies safe throughput without raising the rate on any single address.
Why rotation alone is not the answer
It is tempting to treat proxies as a way to ignore rate limits — just add more IPs and keep firing. That backfires. A site that returns 429 is watching request patterns, and a flood from a shifting set of addresses hitting the same endpoints in the same instant is itself a detectable signature that escalates to hard bans. Rotation multiplies polite throughput; it does not replace politeness. Always combine a proxy pool with backoff, Retry-After, realistic pacing, and sensible concurrency.
Honestly, some limits are meant to be respected — check whether an official API or a documented rate exists before scaling around a 429, and read each site's terms. When you do need distribution, s4m's authenticated datacenter proxies spread load and are metered so widening the pool is cheap; for the mechanics see the rotation guide. Remember datacenter IPs can still be flagged wholesale, and rotation does nothing about fingerprints. Prototype cheaply with the free proxy list and validate exits with the tools.
Questions, answered
Is a 429 a permanent ban?
No. HTTP 429 Too Many Requests is a temporary rate limit — it usually clears once you slow down. Honour any Retry-After header, add backoff, and reduce your request rate, and access typically resumes.
What is the single most effective fix for a 429?
Respecting the Retry-After header when the server sends one. It tells you exactly how long to wait, so you re-align with the server's limit instead of guessing and getting throttled again.
Will rotating proxies stop 429 errors?
It helps by keeping each IP under the per-IP limit, but only alongside backoff and throttling. Rotation without polite pacing produces a detectable flood that escalates to harder blocks, so combine the two.
Are s4m proxies enough to bypass any rate limit?
No honest tool can promise that. s4m datacenter proxies spread load across IPs, but you still need backoff and throttling, datacenter ranges can be flagged, and some limits should simply be respected via an official API.
Spread load the honest way
Combine backoff and throttling with authenticated datacenter proxies on proxy.s4m.online to keep each IP under the limit. Metered pay-as-you-go, dedicated IP optional.
People found this page by searching for
- how to fix 429 too many requests when scraping
- residential proxies
- fresh free proxy list
- free proxy list
- how openvpn works
- how to check http proxy
- how to configure wireguard
- proxy for scraping plans
- socks5 proxy step by step
- Tinyproxy vs 3proxy vs Squid
- is proxy authentication safe
- ip address settings
- UDP over SOCKS5
- is openvpn safe
- openvpn
- bypassing DPI
- http proxy
- ip address
Real search phrases this page answers — the linked ones open the page that covers them in depth.