Skip to contentHow to use proxies with Scrapy
Scrapy routes every request through a proxy in one of three ways: a per-request meta key, an environment variable, or a rotating downloader middleware. Here is each, with working code for authenticated s4m proxies.
Three ways Scrapy talks to a proxy
Scrapy ships an HttpProxyMiddleware that is enabled by default. It reads the proxy from one of two places: the proxy key on a request's meta dict, or the standard http_proxy / https_proxy environment variables. For a fixed proxy across a whole crawl, the environment variable is the quickest path; for per-request control and rotation, set request.meta["proxy"] yourself, usually from a custom downloader middleware.
Authentication is the part people miss. Scrapy does not read USER:PASS@ out of the proxy URL the way curl does — it expects a Proxy-Authorization header. The clean approach is to build that header once and attach it in your middleware. s4m proxies authenticate with USER:PASS against proxy.s4m.online, HTTP on port 3128 (the natural fit for Scrapy's HTTP proxy middleware). For SOCKS5 you would front the crawl with a local SOCKS-to-HTTP bridge, but for most scraping the HTTP endpoint is simpler. Learn more on the proxy page, and see other scraping guides.
A rotating proxy middleware
Drop this into middlewares.py, enable it in settings.py, and each request picks a proxy from your pool with the correct auth header.
# middlewares.py
import base64
import random
USER, PASS, HOST = "USER", "PASS", "proxy.s4m.online"
PROXIES = [f"http://{HOST}:3128"] # add more endpoints/ports to rotate
class S4mProxyMiddleware:
def _auth_header(self):
raw = f"{USER}:{PASS}".encode()
return b"Basic " + base64.b64encode(raw)
def process_request(self, request, spider):
request.meta["proxy"] = random.choice(PROXIES)
# Scrapy needs the Proxy-Authorization header explicitly:
request.headers[b"Proxy-Authorization"] = self._auth_header()
# settings.py
DOWNLOADER_MIDDLEWARES = {
# keep Scrapy's built-in proxy middleware after ours
"myproject.middlewares.S4mProxyMiddleware": 350,
"scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware": 400,
}
CONCURRENT_REQUESTS = 8
DOWNLOAD_DELAY = 0.5
RETRY_ENABLED = True
RETRY_TIMES = 3
RETRY_HTTP_CODES = [429, 500, 502, 503, 504]Wire it up
From a clean project to a proxied crawl.
Add the middleware
Paste the S4mProxyMiddleware class into your project's middlewares.py and replace USER, PASS and HOST with your s4m credentials and endpoint.
Enable it in settings
Register it in DOWNLOADER_MIDDLEWARES at a priority below Scrapy's HttpProxyMiddleware (a smaller number runs first), so your proxy and header are set before the built-in middleware runs.
Tune politeness
Set DOWNLOAD_DELAY and CONCURRENT_REQUESTS to stay under the target's rate limit, and enable RETRY_HTTP_CODES for 429 and 5xx so transient blocks are retried automatically.
Verify the exit IP
Add a first request to an IP-echo endpoint like https://api.ipify.org and log the response to confirm your crawl is really exiting through the proxy.
Rotation, retries and honesty
For real rotation, put several endpoints in PROXIES and let process_request pick one per request; combine that with Scrapy's built-in retry middleware so a 429 or 503 is retried on a fresh pick. Keep DOWNLOAD_DELAY non-zero and concurrency modest — a crawler that backs off is far harder to block than one that sprints.
Two honest caveats. First, s4m proxies are datacenter, not residential: excellent for APIs, testing and moderately defended sites, but aggressive anti-bot systems may flag datacenter ranges. Second, rotation is not a licence to ignore a site's rules — read robots.txt and its terms. When you need a single consistent exit for an allowlist, add a dedicated personal IP instead of rotating. Check exits and ports with the tools, and prototype against the free proxy list before committing traffic.
Questions, answered
Why doesn't Scrapy pick up my USER:PASS from the proxy URL?
Unlike curl, Scrapy's HttpProxyMiddleware does not parse credentials out of the URL. You must set a Proxy-Authorization header yourself — build a Basic auth header from USER:PASS as shown in the middleware example.
Can Scrapy use a SOCKS5 proxy?
Not natively through HttpProxyMiddleware, which is HTTP-oriented. Use the HTTP endpoint on port 3128 for Scrapy, or run a local SOCKS-to-HTTP bridge if you specifically need SOCKS5.
How do I rotate proxies per request?
Put several proxy endpoints in a list and choose one in process_request, setting request.meta["proxy"] each time. Pair it with Scrapy's retry middleware so a blocked request retries on a new pick.
Are these residential proxies?
No. s4m proxies are authenticated datacenter SOCKS5 and HTTP proxies, metered pay-as-you-go. They suit most scraping, though heavily defended targets may flag datacenter IP ranges.
Scale your Scrapy crawls
Authenticated datacenter proxies on proxy.s4m.online, metered pay-as-you-go, ready to rotate through a Scrapy middleware. Add a dedicated IP when a target expects one consistent exit.
People found this page by searching for
- how to use proxies with Scrapy
- how to configure vpn kill switch
- vpn kill switch for android
- vpn kill switch
- ip address for android
- proxy setup
- how to check if a proxy is anonymous
- best VPN protocols in 2026
- free proxy list
- socks5 proxy explained
- how to set up proxy for scraping
- how to fix 429 too many requests when scraping
- proxy authentication
- how to avoid IP bans when scraping
- socks5 proxy
Real search phrases this page answers — the linked ones open the page that covers them in depth.