Skip to contentA proxy sits between your Python script and the target site. Your code sends the request to the proxy first, and the proxy forwards it onward. That extra hop can matter for testing, access control, or simple network routing. If you have ever asked how to use a proxy with Python requests, the answer starts with that middle step.
This is not magic. It is routing. A script on your laptop can behave as if it is coming from another network, and that changes what some services return. One request may pass, another may be blocked, and a third may look different because the proxy's IP is now the public face of the connection.
1. What a Proxy Does in Python Requests
In requests, the proxy does not live inside the website call itself. It is attached to the request as routing information, then the library sends traffic through that address. For HTTP calls, the proxy can see the full request path; for HTTPS, the proxy usually handles the tunnel and passes encrypted traffic onward, which is why the configuration must match the protocol.
Three common cases show up quickly. Testing from a different region is one. Checking whether a site behaves differently behind a corporate gateway is another. Network routing inside a lab or office can be the third. Small detail, big difference.
A proxy is not a cure-all. Some sites block known proxy IP ranges, and some proxy services log traffic in ways that may conflict with your policy. If privacy or compliance matters, read about proxy logs and privacy compliance before you point automation at a live system.
2. Prerequisites: Install Requests and Gather Proxy Details
Start with Python and the requests library. If requests is missing, install it first with pip install requests. Then collect the proxy details in a plain list: scheme, host, port, and, if required, username and password.
The scheme matters because http and https are not interchangeable in every setup. Host is the proxy address. Port is the listening number, such as 8080 or another value given by your provider. Credentials may be optional, but if they exist, write them down exactly as issued. One wrong character can break the whole chain.
If your team uses different proxy terms loosely, the VPN and proxy glossary is a useful reference. It saves time when someone says "forward proxy" and someone else means "authenticated gateway." Two people, one outage.
3. Set Up a Basic Proxy in requests
The simplest setup uses a proxies dictionary passed to requests.get(). You can also attach the dictionary to a Session if you want to reuse the same settings across multiple calls. Keep both http and https entries in the dictionary, even if you think only one protocol will be used.
Example:
import requests |
|---|
proxies = { |
"http": "http://proxy.example.com:8080", |
"https": "http://proxy.example.com:8080", |
} |
response = requests.get("https://example.com", proxies=proxies, timeout=10) |
That https key often surprises beginners. The value can still start with http:// because it tells requests how to talk to the proxy, not the final site. One line, two routes.
For repeated calls, a session keeps things tidy:
session = requests.Session() |
|---|
session.proxies.update(proxies) |
response = session.get("https://example.com", timeout=10) |
A session can help with cookie reuse and fewer repeated settings. It also makes the code easier to read after the tenth request, which is where messy scripts usually begin to wobble.
4. Add Authentication to a Proxy URL
Some proxies require credentials in the URL itself. The format is straightforward: http://username:password@host:port. If the password contains characters like @, :, or #, those characters must be encoded safely so the URL parser does not mistake them for separators.
Example with encoded credentials:
proxies = { |
|---|
"http": "http://user:pa%[email protected]:8080", |
"https": "http://user:pa%[email protected]:8080", |
} |
Never paste raw credentials into code you plan to commit. Put them in environment variables, a secret store, or another protected location. One leaked password can expose more than one script.
Encoding is safer than guessing. In Python, people often use urllib.parse.quote for usernames or passwords that contain reserved symbols. That keeps the proxy URL valid and avoids silent failures that are annoying because they look like network trouble when the real issue is a single character.
5. Use Environment Variables for Reusable Proxy Settings
For local development, environment variables are often cleaner than hardcoding values. Set HTTP_PROXY and HTTPS_PROXY in your shell, and requests can pick them up automatically. This is handy in scripts, notebooks, and CI jobs where the same proxy settings repeat across runs. If you are looking for Python requests proxy settings, this approach is often the simplest to maintain. If you need how to set HTTP_PROXY and HTTPS_PROXY, the shell examples below show the basic pattern.
Example in a Unix-like shell:
export HTTP_PROXY="http://user:[email protected]:8080" |
|---|
export HTTPS_PROXY="http://user:[email protected]:8080" |
On Windows, the variable names are the same, even if the command syntax differs. The point is the same: keep the proxy outside the source file. One file fewer to edit.
requests also respects proxy environment settings unless you override them in code. That gives you a clean default for a workstation and a different choice for a special script. If your automation setup includes more than one tool, see how to choose a VPN for the sort of environment planning that avoids surprises later.
6. Verify That the Proxy Is Working
Do not trust the config blindly. Check the outgoing IP with a simple request to a service that returns your public address. If the response shows the proxy's address instead of your local one, traffic is going through the proxy. That is the first confirmation.
You can also inspect response behavior. Some sites return a different language page, a different region banner, or a different consent page when traffic comes from another IP range. Those changes are not proof by themselves, but they add a clue. Two clues are better than one.
A quick test endpoint can help:
import requests |
|---|
proxies = { |
"http": "http://proxy.example.com:8080", |
"https": "http://proxy.example.com:8080", |
} |
r = requests.get("https://api.ipify.org?format=json", proxies=proxies, timeout=10) |
If the endpoint returns the proxy IP, the setup works. If it times out, the proxy may be unreachable, or the target site may reject the tunnel. A working test matters more than a neat config file.
7. Handle Common Proxy Errors
Connection failures usually mean the host, port, or scheme is wrong. Check the address first. If the proxy expects https:// and you send http://, the connection may fail before the request reaches the website. That mismatch is common.
Authentication errors point to the username, password, or encoding. If credentials include special characters, encode them and test again. A 407 response usually means the proxy wants authentication. Not the site. The proxy.
Timeouts can come from a slow proxy, a blocked destination, or a too-short timeout value in the script. Raise the timeout a little and test again. One second is often too little for a cross-network hop, especially if the proxy is under load or the target service is slow that day.
SSL problems often appear when the proxy intercepts HTTPS traffic or when certificate validation fails. In some controlled environments, the proxy presents its own certificate chain, so the client must trust that chain. Do not disable verification casually. That fix is fast and dangerous.
Here are practical fixes in one list:
- Confirm the proxy scheme matches the proxy service.
- Verify the host and port against the provider details.
- Encode usernames and passwords that contain reserved characters.
- Increase the timeout from a short default to a realistic value.
- Check whether the proxy requires a trusted certificate bundle.
- Test the proxy with a simple URL before using a larger workflow.
8. Best Practices for Reliable Proxy Use
Use timeouts on every request. Without one, a dead proxy can freeze a script longer than you expect. A value like timeout=10 is a common starting point, then you adjust it after a few real tests.
Retries help with temporary failures, but they should be limited. A loop that hammers a failing proxy can make a bad minute worse. Keep retry counts small, and stop when the error is clearly permanent. Three attempts are usually enough to expose a dead path.
Reuse sessions where you can. A session reduces repeated setup and can keep connections organized across multiple calls. That matters in a workflow that makes 20, 50, or 100 requests. It also keeps the code readable, which helps during debugging at 2 a.m.
Protect secrets as if they are sensitive, because they are. Do not write proxy credentials into logs, notebooks, or git history. If you need team guidance on authentication patterns, the Proxy Authentication Best Practices Guide is a sensible next read, especially if multiple people touch the same script.
Choose proxy sources that match the task and the rules you must follow. A personal test, a company network rule, and a public data collection job do not belong in the same bucket. If you work with regional infrastructure or changing provider behavior, the recent changes in proxy infrastructure can help frame what changed and why some old scripts now fail.
One last point. Keep the proxy setup boring. Boring is good. A proxy that is documented, tested, encoded, and monitored causes fewer surprises than a clever script that only works on one machine with one cached credential and one lucky route.