How to rotate proxies in Python (requests and aiohttp)
Rotation is the difference between a scraper that gets one IP banned and one that spreads load across many. Here is a working rotator for both synchronous requests and async aiohttp, with health checks and automatic retirement of dead exits.
What rotation actually solves
Sites rate-limit and ban by IP. If every request leaves from one address, you hit the limit fast and a single ban stops the whole job. Rotation spreads requests across a pool of exits, so load per IP stays under the threshold and one dead or blocked exit only costs you a retry, not the run. Two designs cover almost everything: round-robin (cycle through the pool in order, predictable and even) and random (pick per request, simple and stateless).
The pool itself can come from two places. You can list several s4m endpoints and ports and rotate across them, or, for quick low-stakes tests, pull live entries from the free public proxy list — filtered by country, protocol and anonymity, though those third-party proxies are use-at-your-own-risk and die often. Production rotation should lean on authenticated datacenter proxies from s4m (SOCKS5 on 1080, HTTP on 3128, USER:PASS), which stay up and are metered so scaling the pool is cheap. This builds directly on scraping with requests.
A rotator for requests and aiohttp
A pool that round-robins, retries on failure, and retires exits that keep failing. Sync and async versions. Replace USER:PASS.
import asyncio, itertools, random
import requests, aiohttp
from aiohttp_socks import ProxyConnector # pip install aiohttp-socks
USER, PASS, HOST = "USER", "PASS", "proxy.s4m.online"
POOL = [
f"socks5h://{USER}:{PASS}@{HOST}:1080",
f"http://{USER}:{PASS}@{HOST}:3128",
# add more endpoints/ports here to widen the pool
]
# ---------- Synchronous: requests + round-robin + retirement ----------
class Rotator:
def __init__(self, pool):
self.alive = list(pool)
self.fails = {p: 0 for p in pool}
self._cycle = itertools.cycle(self.alive)
def pick(self):
return next(self._cycle)
def retire(self, proxy):
self.fails[proxy] += 1
if self.fails[proxy] >= 3 and proxy in self.alive:
self.alive.remove(proxy)
self._cycle = itertools.cycle(self.alive or POOL)
def get(url, rot, retries=4):
for _ in range(retries):
proxy = rot.pick()
try:
r = requests.get(url, proxies={"http": proxy, "https": proxy},
timeout=(5, 20))
r.raise_for_status()
return r
except requests.RequestException:
rot.retire(proxy)
raise SystemExit("pool exhausted")
rot = Rotator(POOL)
print(get("https://api.ipify.org", rot).text)
# ---------- Async: aiohttp, one connector per proxy ----------
async def fetch(url, proxy):
connector = ProxyConnector.from_url(proxy)
async with aiohttp.ClientSession(connector=connector) as s:
async with s.get(url, timeout=aiohttp.ClientTimeout(total=25)) as r:
return await r.text()
async def main():
tasks = [fetch("https://api.ipify.org", random.choice(POOL))
for _ in range(5)]
for ip in await asyncio.gather(*tasks, return_exceptions=True):
print(ip)
asyncio.run(main())Design choices that matter
The knobs that decide whether rotation helps or just adds churn.
Round-robin vs random
Round-robin spreads load evenly and is easy to reason about; random is stateless and simple. Both beat a single fixed exit — pick round-robin when even distribution matters.
Retire dead exits
Count consecutive failures per proxy and drop an exit after a few strikes, so the pool self-heals instead of retrying a corpse forever.
Health-check before use
Periodically hit an IP-echo endpoint through each proxy and keep only the ones that answer. Essential when the pool includes volatile free-list entries.
Back off, don't sprint
Rotation is not a licence to flood. Add per-host delays and exponential backoff so you stay under limits rather than fighting them across many IPs.
Sticky sessions and honest limits
Rotation is not always what you want. Some flows — a login, a cart, a multi-step form — need the same IP for the whole session, or the site sees a user teleporting between addresses and blocks it. That is a sticky session: rotate between jobs, stay fixed within one. When a workflow needs one guaranteed-stable address, add a dedicated personal IP instead of rotating.
Be honest about the ceiling. s4m proxies are datacenter, so rotating them helps with rate limits and even load but will not disguise datacenter IPs from an anti-bot system that blocks the whole range. Rotation also does nothing about browser fingerprints or request patterns. Use it as one layer: pair it with realistic headers, backoff, and respect for each site's rules. For the async pieces you will need aiohttp-socks for SOCKS5 support; validate exits with the tools and automate credential handling through the API.
Questions, answered
How do I rotate proxies with requests?
Keep a list of proxy URLs and pick one per request — round-robin or random — passing it as the proxies argument. Retire an exit after a few consecutive failures so the pool self-heals. The example above shows a complete rotator.
How do I rotate proxies with aiohttp?
aiohttp sets the proxy per request for HTTP, but for SOCKS5 you build a ProxyConnector (from aiohttp-socks) per proxy and create a session with it. Spread tasks across the pool with asyncio.gather.
When should I NOT rotate proxies?
When a workflow needs a consistent identity — logins, carts, multi-step forms. Switching IPs mid-session looks suspicious and gets blocked. Use a sticky session or a dedicated IP for those flows.
Will rotating datacenter proxies avoid all blocks?
No. Rotation spreads load and dodges per-IP rate limits, but datacenter ranges can still be flagged wholesale by aggressive anti-bot systems, and rotation does not change your browser fingerprint or behaviour.
Rotate across real exits
Widen your pool with authenticated datacenter SOCKS5 and HTTP proxies on proxy.s4m.online, metered so scaling is cheap. Add a dedicated IP for the sticky-session steps.
People found this page by searching for
- how to rotate proxies in Python (requests and aiohttp)
- best proxy for scraping 2026
- proxy for scraping price
- proxy for scraping
- what is http proxy
- what are ISP (static residential) proxies
- VPN kill switch on Windows
- socks5 proxy explained
- is free proxy list safe
- HTTP 407 proxy authentication required
- free proxy list without registration
- proxy authentication
Real search phrases this page answers — the linked ones open the page that covers them in depth.