Skip to contentHow to use proxies for web scraping
Proxies keep your scraper's origin server hidden and spread requests across IPs so a single address doesn't get rate-limited or blocked. This guide covers choosing proxies and wiring them into Python, Scrapy, and Playwright.
Why your scraper needs a proxy
Two problems drive almost every scraping setup: protecting your real IP, and staying under per-IP rate limits.
When a crawler sends thousands of requests from one address, the target site sees a single IP behaving like a bot. The usual result is a slowdown, a CAPTCHA wall, or an outright block — and worse, the blocked address is your origin server, the machine that runs your business logic. A proxy sits between your scraper and the target so the site only ever sees the proxy's exit IP, never your real one.
Two concerns shape almost every scraping setup:
- Protecting your origin. If your production server's IP gets blacklisted, the damage outlives the scrape. Routing through a proxy keeps that address clean and unlinked from the crawl.
- Distributing load. Spreading requests across several exit IPs keeps any one of them under the target's per-IP rate limit, so you collect data without tripping abuse defenses.
s4m provides authenticated datacenter SOCKS5 and HTTP proxies for exactly this. They are fast and inexpensive, which suits high-volume, tolerant targets — but because they are datacenter IPs, strict sites can fingerprint them more readily than residential addresses, so set expectations accordingly. For quick experiments you can pull throwaway endpoints from the free public proxy list, third-party proxies validated continuously but strictly use-at-your-own-risk. If you also want to hide a whole app or server rather than a single client, a VPN tunnel covers all its traffic. See pricing for metered proxy rates.
Choosing proxies for scraping
Match the tool to the job. Here's how s4m's options compare for a scraping workload.
Datacenter SOCKS5 / HTTP
Metered, pay-as-you-go proxies on proxy.s4m.online — port 1080 for SOCKS5, 3128 for HTTP, authenticated with USER:PASS. Fast and cost-effective for bulk crawling of targets that don't aggressively score IP reputation. This is the workhorse for production scraping.
Free public proxy list
Third-party proxies harvested from open sources and continuously re-validated, filterable by country, protocol, and anonymity. Ideal for prototyping a scraper before you commit budget — but unreliable and use-at-your-own-risk, never a fit for production jobs.
Dedicated personal IP
A dedicated IP is exclusive during its assignment under the service terms. Prior history, assignment duration, availability and supported bindings still matter. An exit IP can have a prior history. Accounts, cookies and browser signals may still identify a user; changing an address does not guarantee a clean reputation.
Integrate and control your crawl
From a first authenticated request to sessions, rotation, rate limits, and cost control.
Point your client at the proxy
In Python requests, set the proxies dict to socks5://USER:[email protected]:1080 (install requests[socks]) or the HTTP endpoint on 3128. In Scrapy, enable HttpProxyMiddleware and set the proxy per request or via the http_proxy environment variable. In Playwright, pass proxy={"server": ...} to browser.launch().
Choose sessions vs rotation
Use one persistent requests.Session — or a dedicated personal IP — when the target ties cookies and login state to an IP, since switching mid-session looks suspicious. Rotate across many exit IPs when crawling public listings, to stay under per-IP limits. Don't mix the two for the same logical identity.
Throttle and respect the target
Honor robots.txt, read the site's terms of service, and add delays: time.sleep in requests, DOWNLOAD_DELAY and AutoThrottle in Scrapy. Steady, human-scale request rates get blocked far less often than bursts — scraping responsibly is both the ethical choice and the effective one.
Cap your spend
Because datacenter proxies are metered, watch usage. s4m lets admins set limits on three axes — per account-type, per IP, and per account — so a runaway crawler can't quietly burn your budget. Set a per-account ceiling before launching a large job.
Python: scrape through the s4m proxy
A minimal, polite requests scraper that exits through your s4m datacenter proxy instead of your origin server.
import time import requests # s4m datacenter proxy endpoints: # SOCKS5 -> proxy.s4m.online:1080 (pip install "requests[socks]") # HTTP -> proxy.s4m.online:3128 PROXY = "socks5://USER:[email protected]:1080" # PROXY = "http://USER:[email protected]:3128" session = requests.Session() session.proxies.update({"http": PROXY, "https": PROXY}) session.headers.update( {"User-Agent": "my-scraper/1.0 ([email protected])"} ) # Confirm requests now exit through the proxy, not your real IP: print("exit IP:", session.get("https://api.ipify.org", timeout=15).text) for page in range(1, 6): r = session.get(f"https://example.com/list?page={page}", timeout=20) r.raise_for_status() # ... parse r.text here ... time.sleep(1.5) # throttle: stay within the target's rate limits
Questions, answered
Are s4m proxies residential?
No. s4m provides authenticated datacenter SOCKS5 and HTTP proxies. They are fast and cost-effective, but because they are datacenter IPs, sites that score IP reputation can detect them more readily than residential addresses. We do not sell or claim residential proxies.
Can I rotate IPs automatically?
You drive rotation from your side by cycling through requests or sessions. For a fixed, non-rotating exit, add a dedicated personal IP and bind it to your proxy. Pair either approach with s4m's per-IP and per-account usage limits to keep both behavior and cost predictable.
SOCKS5 or HTTP — which should I use for scraping?
Both work. SOCKS5 (port 1080) is protocol-agnostic and handles any TCP traffic, which is convenient for browser automation like Playwright. HTTP (port 3128) is the simplest fit for plain requests-based scripts. Pick whichever your client supports best.
Can I use the free proxy list for production scraping?
Keep it for testing. The free list aggregates third-party proxies from open sources; they are validated continuously but can disappear or slow down at any moment, and you use them at your own risk. For reliable jobs, use s4m's authenticated datacenter proxies.
Is web scraping through a proxy legal?
A proxy changes which IP a site sees; it does not grant permission. Always check the target's terms of service and robots.txt, avoid personal or copyrighted data you're not allowed to collect, and keep request rates reasonable. A proxy is not a substitute for scraping lawfully and ethically.
Start scraping without exposing your origin
Spin up authenticated datacenter SOCKS5 and HTTP proxies, or prototype with the free public list first. Anonymous accounts, no-logs by design, and you pay only for the data you meter.
People found this page by searching for
- how to use proxies for web scraping
- split tunneling
- ip address for scraping
- ip address
- vpn kill switch for scraping
- proxy for scraping
- free proxy list
- proxy setup
- double VPN / multi-hop
- how to use a proxy with wget
Real search phrases this page answers — the linked ones open the page that covers them in depth.