Python Guide · 2 min read

How to rotate proxies in Python (requests and aiohttp)

Rotation is the difference between a scraper that gets one IP banned and one that spreads load across many. Here is a working rotator for both synchronous requests and async aiohttp, with health checks and automatic retirement of dead exits.

What rotation actually solves

Sites rate-limit and ban by IP. If every request leaves from one address, you hit the limit fast and a single ban stops the whole job. Rotation spreads requests across a pool of exits, so load per IP stays under the threshold and one dead or blocked exit only costs you a retry, not the run. Two designs cover almost everything: round-robin (cycle through the pool in order, predictable and even) and random (pick per request, simple and stateless).

The pool itself can come from two places. You can list several s4m endpoints and ports and rotate across them, or, for quick low-stakes tests, pull live entries from the free public proxy list — filtered by country, protocol and anonymity, though those third-party proxies are use-at-your-own-risk and die often. Production rotation should lean on authenticated datacenter proxies from s4m (SOCKS5 on 1080, HTTP on 3128, USER:PASS), which stay up and are metered so scaling the pool is cheap. This builds directly on scraping with requests.

A rotator for requests and aiohttp

A pool that round-robins, retries on failure, and retires exits that keep failing. Sync and async versions. Replace USER:PASS.

python
import asyncio, itertools, random
import requests, aiohttp
from aiohttp_socks import ProxyConnector  # pip install aiohttp-socks

USER, PASS, HOST = "USER", "PASS", "proxy.s4m.online"
POOL = [
    f"socks5h://{USER}:{PASS}@{HOST}:1080",
    f"http://{USER}:{PASS}@{HOST}:3128",
    # add more endpoints/ports here to widen the pool
]

# ---------- Synchronous: requests + round-robin + retirement ----------
class Rotator:
    def __init__(self, pool):
        self.alive = list(pool)
        self.fails = {p: 0 for p in pool}
        self._cycle = itertools.cycle(self.alive)

    def pick(self):
        return next(self._cycle)

    def retire(self, proxy):
        self.fails[proxy] += 1
        if self.fails[proxy] >= 3 and proxy in self.alive:
            self.alive.remove(proxy)
            self._cycle = itertools.cycle(self.alive or POOL)

def get(url, rot, retries=4):
    for _ in range(retries):
        proxy = rot.pick()
        try:
            r = requests.get(url, proxies={"http": proxy, "https": proxy},
                             timeout=(5, 20))
            r.raise_for_status()
            return r
        except requests.RequestException:
            rot.retire(proxy)
    raise SystemExit("pool exhausted")

rot = Rotator(POOL)
print(get("https://api.ipify.org", rot).text)

# ---------- Async: aiohttp, one connector per proxy ----------
async def fetch(url, proxy):
    connector = ProxyConnector.from_url(proxy)
    async with aiohttp.ClientSession(connector=connector) as s:
        async with s.get(url, timeout=aiohttp.ClientTimeout(total=25)) as r:
            return await r.text()

async def main():
    tasks = [fetch("https://api.ipify.org", random.choice(POOL))
             for _ in range(5)]
    for ip in await asyncio.gather(*tasks, return_exceptions=True):
        print(ip)

asyncio.run(main())

Design choices that matter

The knobs that decide whether rotation helps or just adds churn.

Round-robin vs random

Round-robin spreads load evenly and is easy to reason about; random is stateless and simple. Both beat a single fixed exit — pick round-robin when even distribution matters.

Retire dead exits

Count consecutive failures per proxy and drop an exit after a few strikes, so the pool self-heals instead of retrying a corpse forever.

Health-check before use

Periodically hit an IP-echo endpoint through each proxy and keep only the ones that answer. Essential when the pool includes volatile free-list entries.

Back off, don't sprint

Rotation is not a licence to flood. Add per-host delays and exponential backoff so you stay under limits rather than fighting them across many IPs.

Sticky sessions and honest limits

Rotation is not always what you want. Some flows — a login, a cart, a multi-step form — need the same IP for the whole session, or the site sees a user teleporting between addresses and blocks it. That is a sticky session: rotate between jobs, stay fixed within one. When a workflow needs one guaranteed-stable address, add a dedicated personal IP instead of rotating.

Be honest about the ceiling. s4m proxies are datacenter, so rotating them helps with rate limits and even load but will not disguise datacenter IPs from an anti-bot system that blocks the whole range. Rotation also does nothing about browser fingerprints or request patterns. Use it as one layer: pair it with realistic headers, backoff, and respect for each site's rules. For the async pieces you will need aiohttp-socks for SOCKS5 support; validate exits with the tools and automate credential handling through the API.

FAQ

Questions, answered

How do I rotate proxies with requests?

Keep a list of proxy URLs and pick one per request — round-robin or random — passing it as the proxies argument. Retire an exit after a few consecutive failures so the pool self-heals. The example above shows a complete rotator.

How do I rotate proxies with aiohttp?

aiohttp sets the proxy per request for HTTP, but for SOCKS5 you build a ProxyConnector (from aiohttp-socks) per proxy and create a session with it. Spread tasks across the pool with asyncio.gather.

When should I NOT rotate proxies?

When a workflow needs a consistent identity — logins, carts, multi-step forms. Switching IPs mid-session looks suspicious and gets blocked. Use a sticky session or a dedicated IP for those flows.

Will rotating datacenter proxies avoid all blocks?

No. Rotation spreads load and dodges per-IP rate limits, but datacenter ranges can still be flagged wholesale by aggressive anti-bot systems, and rotation does not change your browser fingerprint or behaviour.

Rotate across real exits

Widen your pool with authenticated datacenter SOCKS5 and HTTP proxies on proxy.s4m.online, metered so scaling is cheap. Add a dedicated IP for the sticky-session steps.

People found this page by searching for

Real search phrases this page answers — the linked ones open the page that covers them in depth.