Skip to contentWeb scraping with Python requests and proxies
A practical scraping workflow built on requests and BeautifulSoup, routed through an authenticated proxy so your origin IP stays hidden and your crawler survives rate limits. Full runnable code included.
A scraper is a session, not a loop of gets
The difference between a scraper that works for ten minutes and one that runs for hours is almost never the parsing — it is how you make requests. A naive for url in urls: requests.get(url) opens a fresh TCP connection every time, sends no realistic headers, keeps no cookies, and hammers the target from one IP until it gets a 429 or a block.
The fix is a requests.Session plus a proxy. A session pools connections and persists cookies, so a burst of pages is faster and looks like one coherent client. An authenticated proxy replaces your real address with a datacenter exit, so the site never sees your origin IP and a single blocked address does not end the whole job. s4m proxies speak SOCKS5 on port 1080 and HTTP on port 3128, both authenticated with USER:PASS — use socks5h:// so DNS is resolved remotely and hostnames never leak. For throwaway experiments, the free public proxy list is a separate, use-at-your-own-risk pool.
A resilient requests scraper
Session, realistic headers, retries with backoff, proxy rotation and BeautifulSoup parsing in one file. Replace USER:PASS with your credentials.
import random
import time
import requests
from bs4 import BeautifulSoup
from requests.exceptions import RequestException
USER, PASS, HOST = "USER", "PASS", "proxy.s4m.online"
POOL = [
f"socks5h://{USER}:{PASS}@{HOST}:1080", # needs requests[socks]
f"http://{USER}:{PASS}@{HOST}:3128",
]
HEADERS = {
"User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36",
"Accept-Language": "en-US,en;q=0.9",
}
def make_session(proxy):
s = requests.Session()
s.headers.update(HEADERS)
s.proxies = {"http": proxy, "https": proxy}
return s
def fetch(url, retries=4):
for attempt in range(1, retries + 1):
proxy = random.choice(POOL)
try:
s = make_session(proxy)
r = s.get(url, timeout=(5, 20))
if r.status_code == 429:
raise RequestException("rate limited")
r.raise_for_status()
return r
except RequestException as e:
wait = 2 ** attempt + random.random() # exponential backoff
print(f"[{attempt}/{retries}] {proxy} -> {e}; sleeping {wait:.1f}s")
time.sleep(wait)
raise SystemExit(f"giving up on {url}")
html = fetch("https://quotes.toscrape.com/").text
soup = BeautifulSoup(html, "html.parser")
for q in soup.select(".quote"):
text = q.select_one(".text").get_text(strip=True)
who = q.select_one(".author").get_text(strip=True)
print(f"{who}: {text}")What keeps a crawler alive
Four habits that matter more than the parser you pick.
Send a real User-Agent
The default python-requests UA is an instant flag. Set a normal browser User-Agent and an Accept-Language header so requests blend in with ordinary traffic.
Back off, do not retry blindly
On a 429 or a timeout, sleep with exponential backoff and jitter before retrying. Tight retry loops just deepen the block and burn proxy traffic.
Rotate the exit, not just the retry
Pick a fresh proxy from a pool on each attempt so one flagged IP does not stop the job. Metered datacenter proxies make this cheap to scale.
Respect the target
Read robots.txt, throttle concurrency, and scrape only what you are allowed to. A polite crawler is also a harder-to-block one.
From one script to a real pipeline
Once the fetch loop is solid, the rest is orchestration. Keep a small pool of proxy URLs and rotate per request; when a target gets aggressive, grow the pool rather than the retry count. Persist results as you go so a mid-run failure never costs you the whole crawl, and add a polite delay between pages so you stay under the rate limit instead of fighting it.
When you outgrow ad-hoc scripts, the same proxy endpoint drops straight into Scrapy or an async client — see rotating proxies in Python for an aiohttp pool. Everything is built on the same s4m building blocks: authenticated datacenter SOCKS5 and HTTP proxies, metered pay-as-you-go, with the API and tools to check exit IPs and ports. If you only need to verify a country or protocol quickly, start from the free list.
Questions, answered
Do I need PySocks for the SOCKS5 examples?
Yes. SOCKS support in requests comes from PySocks — install it with pip install "requests[socks]". The HTTP endpoint on port 3128 works with a stock requests install and no extra dependency.
How do I avoid getting blocked while scraping?
Send a realistic User-Agent, throttle your request rate, back off on 429 responses, and rotate exit IPs through a proxy pool. Blocks are usually about behaviour and IP reputation, not the parser you use.
Are s4m proxies residential?
No. s4m sells authenticated datacenter SOCKS5 and HTTP proxies, metered pay-as-you-go. The separate free public proxy list is third-party and use-at-your-own-risk, useful for quick tests only.
Why use socks5h instead of socks5?
socks5h asks the proxy to resolve DNS, so the hostnames you scrape never leak from your machine. Plain socks5 resolves locally first, which can reveal your targets.
Proxies built for scraping
Point requests at proxy.s4m.online, rotate through authenticated datacenter SOCKS5 and HTTP exits, and pay only for the traffic you use. Add a dedicated IP whenever a target expects one.
People found this page by searching for
- web scraping with Python requests and proxies
- proxy for scraping tutorial
- port forwarding through VPNs and proxies
- proxy for scraping
- how to configure openvpn
- download proxy setup
- vpn kill switch
- http proxy
- how free proxy list works
- WireGuard vs OpenVPN
- ip address
- paying for a VPN or proxy with crypto
- buy http proxy
Real search phrases this page answers — the linked ones open the page that covers them in depth.