[Python] How to Access External Sites via Proxy with Requests

When using APIs from within a corporate network or distributing IP addresses for scraping, communication via a proxy server becomes necessary.

With Python’s requests library, you can easily implement communication via a proxy simply by passing the settings in a dictionary format. Here, I will explain the basic proxy settings and how to write them when authentication information (ID/password) is required.

Executable Sample Code

The following code is a script that defines proxy server settings and calls an API (httpbin.org/ip) to check the source IP address.

Note: The IP address 10.10.1.10 in the code is an example. When actually using it, please replace it with the address of your proxy server.

import requests
from requests.exceptions import RequestException

def fetch_via_proxy():
    # URL for connection check (Service that returns the source IP address)
    target_url = "https://httpbin.org/ip"

    # Definition of proxy settings
    # Specify proxy for http and https respectively
    # Format: 'protocol': 'proxy URL:port'
    proxy_settings = {
        "http": "http://10.10.1.10:3128",
        "https": "http://10.10.1.10:3128"
    }

    # [Important] How to write for a proxy requiring authentication
    # "http": "http://username:password@hostname:port"
    # Example: "http": "http://admin_user:secret_pass@10.10.1.10:3128"

    print(f"Connecting to {target_url} via proxy...")
    print(f"Proxy Config: {proxy_settings}")

    try:
        # Applied simply by passing a dictionary to the proxies argument
        # Strongly recommended to set a timeout
        response = requests.get(
            target_url, 
            proxies=proxy_settings, 
            timeout=10
        )
        
        response.raise_for_status()

        # Display results
        # If the proxy is working correctly, the proxy IP appears in 'origin'
        result = response.json()
        print("\n--- Response Data ---")
        print(f"Detected IP (Origin): {result.get('origin')}")

    except RequestException as e:
        print("\n[Connection Error]")
        print("Failed to connect to proxy. Please check settings and network.")
        print(f"Details: {e}")

if __name__ == "__main__":
    fetch_via_proxy()

Points of Proxy Settings

1. Structure of the proxies Argument

The requests.get() and post() methods have an argument called proxies. You pass a dictionary like the following to it:

proxies = {
    'http': 'http://proxy.example.com:8080',
    'https': 'http://proxy.example.com:8080',
}

• Key: The destination protocol (http or https).
• Value: The address of the proxy server to go through.

In many cases, even for HTTPS communication, the connection to the proxy server itself is made via HTTP, so the value typically starts with http://.

2. Authenticated Proxy (Basic Auth)

If an ID and password are required, such as in a corporate proxy, use the format that embeds them directly into the URL.

Format: http://user:password@host:port

3. Settings via Environment Variables (Supplement)

There is also a method using OS environment variables instead of hardcoding them in the code. If the environment variables HTTP_PROXY and HTTPS_PROXY are set, the requests library automatically reads and applies them. In this case, you do not need to specify the proxies argument in the code.

# Example setting for Linux/Mac
export HTTP_PROXY="http://10.10.1.10:3128"
export HTTPS_PROXY="http://10.10.1.10:3128"