REQUESTS-Work with HTTP checks

Introduction to the Requests Library

Requests is a simple and powerful library for working with HTTP requests in Python. It provides a convenient interface for performing operations with REST APIs, downloading data from web pages, as well as uploading files and handling authentication. Thanks to its concise syntax and extensive functionality, Requests has become the de‑facto standard among Python HTTP clients.

The Requests library was created by Kenneth Reitz in 2011 with the goal of making HTTP requests in Python as simple and user‑friendly as possible. It is built on top of urllib3 and offers a high‑level API for interacting with the HTTP protocol.

Advantages of Requests over urllib

• More readable syntax — intuitive methods and parameters
• Automatic cookie and redirect handling — no extra configuration required
• Convenient JSON, headers, and form handling — built‑in support for popular formats
• Support for sessions, proxies, and authentication — a complete toolkit for web services
• Support for all HTTP methods — GET, POST, PUT, DELETE, PATCH, HEAD, OPTIONS
• Automatic content decoding — intelligent handling of character encodings
• Built‑in SSL support — secure connections by default

Installation and Importing the Library

Install with pip:

pip install requests

Upgrade to the latest version:

pip install --upgrade requests

Import the library:

import requests

Main HTTP Request Methods

Sending GET Requests

GET requests are used to retrieve data from a server:

response = requests.get('https://api.example.com/data')
print(response.status_code)  # Response code
print(response.text)         # Response text

Fetching JSON data:

response = requests.get('https://api.example.com/users')
data = response.json()  # Automatic conversion to a Python dict
print(data)

Sending POST Requests

POST requests are used to send data to a server:

payload = {'username': 'admin', 'password': '123'}
response = requests.post('https://api.example.com/login', data=payload)

Sending JSON data:

user_data = {'name': 'John', 'email': 'john@example.com'}
response = requests.post('https://api.example.com/users', json=user_data)

Using PUT, PATCH, and DELETE

# PUT – full resource replacement
requests.put('https://api.example.com/user/1', json={'name': 'New Name', 'email': 'new@example.com'})

# PATCH – partial resource update
requests.patch('https://api.example.com/user/1', json={'active': True})

# DELETE – remove a resource
requests.delete('https://api.example.com/user/1')

# HEAD – fetch only headers
response = requests.head('https://api.example.com/user/1')
print(response.headers)

Working with Query Parameters

URL parameters can be passed via a dictionary:

params = {'search': 'python', 'limit': 10, 'page': 1}
response = requests.get('https://api.example.com/search', params=params)
print(response.url)  # Shows the full URL with parameters

You can also provide parameters as a list of tuples:

params = [('tag', 'python'), ('tag', 'web'), ('limit', 20)]
response = requests.get('https://api.example.com/posts', params=params)

Working with Headers and Cookies

Request Headers

headers = {
    'Authorization': 'Bearer TOKEN123',
    'User-Agent': 'MyApp/1.0',
    'Content-Type': 'application/json'
}
response = requests.get('https://api.example.com/data', headers=headers)

Response Headers

response = requests.get('https://api.example.com/data')
print(response.headers['Content-Type'])
print(response.headers.get('Server', 'Unknown'))

Working with Cookies

# Sending cookies
cookies = {'sessionid': 'abc123', 'csrftoken': 'xyz789'}
response = requests.get('https://example.com', cookies=cookies)

# Retrieving cookies from a response
response = requests.get('https://example.com')
print(response.cookies)
for cookie in response.cookies:
    print(f'{cookie.name}: {cookie.value}')

Uploading and Downloading Files

Uploading Files

# Upload a single file
files = {'file': open('report.pdf', 'rb')}
response = requests.post('https://api.example.com/upload', files=files)

# Upload multiple files
files = {
    'file1': open('document.pdf', 'rb'),
    'file2': open('image.png', 'rb')
}
response = requests.post('https://api.example.com/upload', files=files)

# Upload a file with additional data
files = {'file': ('report.pdf', open('report.pdf', 'rb'), 'application/pdf')}
data = {'description': 'Monthly report'}
response = requests.post('https://api.example.com/upload', files=files, data=data)

Downloading Files

# Download a file into memory
response = requests.get('https://example.com/file.zip')
with open('output.zip', 'wb') as f:
    f.write(response.content)

# Stream large file downloads
response = requests.get('https://example.com/large-file.zip', stream=True)
with open('large-file.zip', 'wb') as f:
    for chunk in response.iter_content(chunk_size=8192):
        f.write(chunk)

Handling JSON and Forms

Working with JSON

# Sending JSON
data = {'name': 'Alice', 'age': 30}
response = requests.post('https://api.example.com/users', json=data)

# Receiving JSON
response = requests.get('https://api.example.com/users/1')
try:
    user_data = response.json()
    print(user_data['name'])
except ValueError:
    print("Response does not contain valid JSON")

Submitting Forms

# Form data
form_data = {
    'username': 'user123',
    'password': 'secret',
    'remember': 'on'
}
response = requests.post('https://example.com/login', data=form_data)

# Form with a file
form_data = {'title': 'My Document'}
files = {'document': open('doc.pdf', 'rb')}
response = requests.post('https://example.com/upload', data=form_data, files=files)

Working with Sessions

Sessions allow you to persist state across requests:

session = requests.Session()

# Session configuration
session.headers.update({'User-Agent': 'MyApp/1.0'})
session.auth = ('username', 'password')

# Making requests via the session
session.get('https://example.com/login')
response = session.post('https://example.com/dashboard', data={'key': 'value'})
response = session.get('https://example.com/profile')

# Closing the session
session.close()

Using a session as a context manager:

with requests.Session() as session:
    session.get('https://example.com/login')
    response = session.get('https://example.com/data')

Exception and Error Handling

try:
    response = requests.get('https://api.example.com/data', timeout=5)
    response.raise_for_status()  # Raises for 4xx/5xx status codes
    data = response.json()
except requests.exceptions.HTTPError as e:
    print(f"HTTP error: {e}")
except requests.exceptions.ConnectionError as e:
    print(f"Connection error: {e}")
except requests.exceptions.Timeout as e:
    print(f"Timeout: {e}")
except requests.exceptions.RequestException as e:
    print(f"General request error: {e}")
except ValueError as e:
    print(f"JSON parsing error: {e}")

Timeouts and Retry Strategies

Timeouts

# Global timeout
response = requests.get('https://example.com', timeout=5)

# Separate connect and read timeouts
response = requests.get('https://example.com', timeout=(3, 10))

Retries

from requests.adapters import HTTPAdapter
from requests.packages.urllib3.util.retry import Retry

session = requests.Session()

# Retry configuration
retry_strategy = Retry(
    total=3,                 # Total attempts
    backoff_factor=1,        # Multiplier for exponential backoff
    status_forcelist=[429, 500, 502, 503, 504],  # Status codes to retry
    method_whitelist=["HEAD", "GET", "OPTIONS"]   # Methods to retry
)

adapter = HTTPAdapter(max_retries=retry_strategy)
session.mount("http://", adapter)
session.mount("https://", adapter)

response = session.get("https://example.com")

Authentication

Basic Authentication

from requests.auth import HTTPBasicAuth

# Method 1
response = requests.get('https://api.example.com/data', 
                       auth=HTTPBasicAuth('username', 'password'))

# Method 2 (shorthand)
response = requests.get('https://api.example.com/data', 
                       auth=('username', 'password'))

Token Authentication

# Bearer token
headers = {'Authorization': 'Bearer your_access_token'}
response = requests.get('https://api.example.com/secure', headers=headers)

# API key
headers = {'X-API-Key': 'your_api_key'}
response = requests.get('https://api.example.com/data', headers=headers)

Digest Authentication

from requests.auth import HTTPDigestAuth

response = requests.get('https://api.example.com/data',
                       auth=HTTPDigestAuth('username', 'password'))

Proxies and SSL Certificates

Using Proxies

proxies = {
    'http': 'http://10.10.1.10:3128',
    'https': 'http://10.10.1.10:1080',
}

response = requests.get('https://example.com', proxies=proxies)

# Proxy with authentication
proxies = {
    'http': 'http://user:pass@10.10.1.10:3128',
    'https': 'http://user:pass@10.10.1.10:1080',
}

SSL Certificates

# Disable SSL verification (not recommended in production)
response = requests.get('https://self-signed.badssl.com/', verify=False)

# Provide path to a CA bundle
response = requests.get('https://example.com', verify='/path/to/cert.pem')

# Client certificate
response = requests.get('https://example.com', 
                       cert=('/path/to/client.cert', '/path/to/client.key'))

Logging and Debugging Requests

Enabling Logging

import logging
import requests

# Enable detailed logging
logging.basicConfig(level=logging.DEBUG)

# Configure logging for requests only
requests_log = logging.getLogger("requests.packages.urllib3")
requests_log.setLevel(logging.DEBUG)
requests_log.propagate = True

response = requests.get('https://example.com')

Debugging with Hooks

def print_request_response(response, *args, **kwargs):
    print(f"Request: {response.request.method} {response.request.url}")
    print(f"Response: {response.status_code}")

response = requests.get('https://example.com', hooks={'response': print_request_response})

Working with Data Streams

Streaming Downloads

response = requests.get('https://example.com/large-file.zip', stream=True)

# Chunked download
with open('large-file.zip', 'wb') as f:
    for chunk in response.iter_content(chunk_size=1024):
        if chunk:
            f.write(chunk)

# Line‑by‑line reading of text files
response = requests.get('https://example.com/data.txt', stream=True)
for line in response.iter_lines():
    print(line.decode('utf-8'))

Custom Adapters

class TimeoutHTTPAdapter(HTTPAdapter):
    def __init__(self, *args, **kwargs):
        self.timeout = kwargs.pop("timeout", 5)
        super().__init__(*args, **kwargs)

    def send(self, request, **kwargs):
        kwargs["timeout"] = self.timeout
        return super().send(request, **kwargs)

session = requests.Session()
session.mount("https://", TimeoutHTTPAdapter(timeout=10))

Requests Library Methods and Functions Table

Core Method Parameters

Practical Usage Examples

Integrating with External APIs

# Working with the GitHub API
def get_user_repos(username):
    url = f'https://api.github.com/users/{username}/repos'
    response = requests.get(url)
    if response.status_code == 200:
        return response.json()
    return None

repos = get_user_repos('octocat')
for repo in repos:
    print(f"{repo['name']}: {repo['description']}")

Service Monitoring

def check_service_health(url):
    try:
        response = requests.get(url, timeout=5)
        return {
            'status': 'UP' if response.status_code == 200 else 'DOWN',
            'response_time': response.elapsed.total_seconds(),
            'status_code': response.status_code
        }
    except requests.exceptions.RequestException:
        return {'status': 'DOWN', 'response_time': None, 'status_code': None}

health = check_service_health('https://example.com/health')
print(f"Service status: {health['status']}")

Sending Telegram Notifications

def send_telegram_message(bot_token, chat_id, message):
    url = f'https://api.telegram.org/bot{bot_token}/sendMessage'
    data = {
        'chat_id': chat_id,
        'text': message,
        'parse_mode': 'HTML'
    }
    response = requests.post(url, data=data)
    return response.json()

# Example usage
send_telegram_message('YOUR_BOT_TOKEN', 'CHAT_ID', 'Hello from Python!')

Automation Scripts

def download_and_process_data():
    # Download data
    response = requests.get('https://api.example.com/data.json')
    data = response.json()
    
    # Process data
    processed_data = [item for item in data if item['active']]
    
    # Send processed data
    result = requests.post('https://api.example.com/processed', json=processed_data)
    return result.status_code == 200

success = download_and_process_data()
print(f"Processing {'succeeded' if success else 'failed'}")

Performance Optimization

Reusing Connections

# Inefficient – creates a new connection for each request
for i in range(10):
    response = requests.get(f'https://api.example.com/data/{i}')

# Efficient – uses a connection pool
with requests.Session() as session:
    for i in range(10):
        response = session.get(f'https://api.example.com/data/{i}')

Asynchronous Alternatives

For high‑performance applications consider using:

• aiohttp for asynchronous HTTP requests
• httpx as a modern async‑compatible alternative
• grequests for pseudo‑asynchronous requests

Security

SSL Certificate Validation

# Always verify SSL certificates in production
response = requests.get('https://api.example.com', verify=True)

# For self‑signed certificates, provide the CA bundle path
response = requests.get('https://internal-api.company.com', 
                       verify='/path/to/ca-bundle.crt')

Preventing Data Leakage

# Do not log sensitive data
import logging
logging.getLogger("requests.packages.urllib3").setLevel(logging.WARNING)

# Use environment variables for tokens
import os
api_key = os.getenv('API_KEY')
headers = {'Authorization': f'Bearer {api_key}'}

Frequently Asked Questions

What is Requests?

Requests is an HTTP library for Python that makes sending HTTP requests simple and intuitive. It provides a user‑friendly interface for interacting with web services and APIs.

Does Requests support SSL?

Yes, Requests supports SSL by default and automatically verifies certificates. Verification can be disabled with verify=False, but this is not recommended in production.

Is Requests asynchronous?

No, Requests is a synchronous library. For asynchronous work, it is recommended to use aiohttp, httpx, or grequests.

How do I upload a file via POST?

Use the files parameter with an open file object:

files = {'file': open('document.pdf', 'rb')}
response = requests.post('https://example.com/upload', files=files)

How can I reuse a connection?

Use requests.Session() to reuse connections and maintain state across requests.

How should I handle large files?

Use the stream=True parameter and the iter_content() method:

response = requests.get('https://example.com/large-file.zip', stream=True)
with open('file.zip', 'wb') as f:
    for chunk in response.iter_content(chunk_size=8192):
        f.write(chunk)

How do I set a custom User‑Agent?

Pass the header via the headers parameter:

headers = {'User-Agent': 'MyApp/1.0 (Contact: admin@example.com)'}
response = requests.get('https://example.com', headers=headers)

How do I work with a JSON API?

Use the json parameter for sending and the .json() method for receiving:

# Sending JSON
response = requests.post('https://api.example.com/users', json={'name': 'John'})

# Receiving JSON
data = response.json()

The Requests library is an indispensable tool for any Python developer working with HTTP. Its simplicity, power, and flexibility make it the ideal choice for integrating with web services, building API clients, and automating web tasks.