Requests Python: Installation and examples of HTTP checks

Installing and Configuring the Requests Library

The requests library is not included in the standard Python distribution, so it needs to be installed separately. Execute the following command in the terminal:

pip install requests

To verify successful installation, run:

import requests
print(requests.__version__)

What are HTTP Requests and Why Use the Requests Library?

HTTP (HyperText Transfer Protocol) is a protocol for transferring data between a client and a server. requests is an external Python library that greatly simplifies sending HTTP requests compared to built-in urllib modules.

Key Benefits of Requests

Ease of use and code readability
Support for all HTTP methods (GET, POST, PUT, DELETE, PATCH)
Automatic handling of JSON data
Built-in support for cookies and sessions
Flexible header and parameter configuration
Error and exception handling
Support for proxy servers and SSL certificates

Basic HTTP Methods with Practical Examples

GET Requests: Retrieving Data

A GET request is used to retrieve data from the server without modifying it.

import requests

# Simple GET request
response = requests.get('https://api.github.com')
print(response.status_code)  # 200
print(response.text)         # Response content

Passing Parameters in a GET Request

# Passing parameters through a dictionary
params = {
    'q': 'python',
    'sort': 'stars',
    'order': 'desc',
    'page': 1
}

response = requests.get('https://api.github.com/search/repositories', params=params)
print(response.url)  # Check the final URL with parameters

POST Requests: Sending Data

A POST request is used to send data to the server and create new resources.

# POST request with form data
data = {
    'username': 'test_user',
    'password': 'secure_password',
    'email': 'test@example.com'
}

response = requests.post('https://httpbin.org/post', data=data)
print(response.json())

POST Request with JSON Data

# Sending JSON data
json_data = {
    'title': 'New post',
    'body': 'Post content',
    'userId': 1
}

response = requests.post(
    'https://jsonplaceholder.typicode.com/posts',
    json=json_data
)
print(response.json())

PUT Requests: Updating Data

A PUT request is used to completely update existing resources.

update_data = {
    'id': 1,
    'title': 'Updated title',
    'body': 'New content',
    'userId': 1
}

response = requests.put(
    'https://jsonplaceholder.typicode.com/posts/1',
    json=update_data
)
print(f"Status: {response.status_code}")
print(response.json())

PATCH Requests: Partial Update

PATCH is used for partial updates of a resource.

patch_data = {'title': 'Partially updated title'}

response = requests.patch(
    'https://jsonplaceholder.typicode.com/posts/1',
    json=patch_data
)
print(response.json())

DELETE Requests: Deleting Data

A DELETE request is used to delete resources on the server.

response = requests.delete('https://jsonplaceholder.typicode.com/posts/1')
print(f"Deletion status: {response.status_code}")

Working with HTTP Headers

Headers contain metadata about the request and allow you to pass additional information to the server.

headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36',
    'Authorization': 'Bearer YOUR_API_TOKEN',
    'Content-Type': 'application/json',
    'Accept': 'application/json'
}

response = requests.get('https://api.example.com/data', headers=headers)
print(response.status_code)

# Viewing response headers
print(response.headers)

Managing Cookies

Cookies are used to maintain state between requests.

# Sending cookies
cookies = {'session_id': '123456', 'user_pref': 'dark_theme'}
response = requests.get('https://httpbin.org/cookies', cookies=cookies)

# Extracting cookies from the response
print(response.cookies)

# Using the RequestsCookieJar object
jar = requests.cookies.RequestsCookieJar()
jar.set('cookie_name', 'cookie_value', domain='httpbin.org', path='/cookies')
response = requests.get('https://httpbin.org/cookies', cookies=jar)

Processing JSON Data

JSON is the most popular data exchange format in modern APIs.

# Receiving JSON data
response = requests.get('https://jsonplaceholder.typicode.com/posts')

# Automatic JSON deserialization
try:
    data = response.json()
    print(f"Received {len(data)} posts")
    print(f"First post: {data[0]['title']}")
except requests.exceptions.JSONDecodeError:
    print("Response does not contain valid JSON")

Handling Errors and Exceptions

Proper error handling is critical to application reliability.

import requests
from requests.exceptions import RequestException, HTTPError, ConnectionError, Timeout

def safe_request(url, method='GET', **kwargs):
    try:
        response = requests.request(method, url, timeout=10, **kwargs)
        response.raise_for_status()  # Raises an exception for HTTP errors
        return response
    
    except HTTPError as e:
        print(f"HTTP error: {e}")
        print(f"Response code: {e.response.status_code}")
    
    except ConnectionError as e:
        print(f"Connection error: {e}")
    
    except Timeout as e:
        print(f"Timeout: {e}")
    
    except RequestException as e:
        print(f"General request error: {e}")
    
    return None

# Using the function
response = safe_request('https://api.github.com')
if response:
    print("Request successful")

Setting Timeouts

Timeouts prevent indefinite waiting for a server response.

# Unified timeout for connection and reading
response = requests.get('https://api.github.com', timeout=5)

# Separate timeouts (connection, reading)
response = requests.get('https://api.github.com', timeout=(3, 10))

# Disabling timeout (not recommended)
response = requests.get('https://api.github.com', timeout=None)

Working with Sessions

Sessions allow you to save cookies and settings between requests.

# Creating a session
session = requests.Session()

# Setting common headers
session.headers.update({
    'User-Agent': 'MyApp/1.0',
    'Accept': 'application/json'
})

# Executing requests through a session
response = session.get('https://httpbin.org/get')
print(response.json())

# The session automatically saves cookies
login_data = {'username': 'user', 'password': 'pass'}
session.post('https://httpbin.org/login', data=login_data)

# Subsequent requests will use saved cookies
protected_response = session.get('https://httpbin.org/protected')

# Closing the session
session.close()

Uploading Files

requests supports uploading files in various ways.

# Uploading a single file
with open('document.pdf', 'rb') as f:
    files = {'file': f}
    response = requests.post('https://httpbin.org/post', files=files)

# Uploading multiple files
files = {
    'file1': open('document1.pdf', 'rb'),
    'file2': open('document2.pdf', 'rb')
}
response = requests.post('https://httpbin.org/post', files=files)

# Don't forget to close the files
for file in files.values():
    file.close()

# Uploading a file with additional parameters
files = {
    'file': ('report.pdf', open('report.pdf', 'rb'), 'application/pdf')
}
response = requests.post('https://httpbin.org/post', files=files)

Downloading Files

# Downloading small files
response = requests.get('https://httpbin.org/image/png')
with open('downloaded_image.png', 'wb') as f:
    f.write(response.content)

# Downloading large files in chunks
url = 'https://httpbin.org/drip?duration=10&numbytes=1024'
response = requests.get(url, stream=True)

with open('large_file.bin', 'wb') as f:
    for chunk in response.iter_content(chunk_size=8192):
        f.write(chunk)

Using Proxy Servers

# Proxy configuration
proxies = {
    'http': 'http://proxy.example.com:8080',
    'https': 'https://proxy.example.com:8080'
}

response = requests.get('https://api.github.com', proxies=proxies)

# Proxy with authentication
proxies = {
    'http': 'http://user:password@proxy.example.com:8080',
    'https': 'https://user:password@proxy.example.com:8080'
}

Advanced Features

Custom Adapters

from requests.adapters import HTTPAdapter
from requests.packages.urllib3.util.retry import Retry

# Configuring retries
retry_strategy = Retry(
    total=3,
    status_forcelist=[429, 500, 502, 503, 504],
    method_whitelist=["HEAD", "GET", "OPTIONS"]
)

adapter = HTTPAdapter(max_retries=retry_strategy)
session = requests.Session()
session.mount("http://", adapter)
session.mount("https://", adapter)

Streaming Requests

# Streaming processing of large responses
response = requests.get('https://httpbin.org/stream/1000', stream=True)

for line in response.iter_lines():
    if line:
        print(line.decode('utf-8'))

Practical Use Cases

Working with REST APIs

class GitHubAPI:
    def __init__(self, token):
        self.base_url = 'https://api.github.com'
        self.session = requests.Session()
        self.session.headers.update({
            'Authorization': f'token {token}',
            'Accept': 'application/vnd.github.v3+json'
        })
    
    def get_user(self, username):
        response = self.session.get(f'{self.base_url}/users/{username}')
        response.raise_for_status()
        return response.json()
    
    def create_repo(self, name, description=None):
        data = {'name': name}
        if description:
            data['description'] = description
        
        response = self.session.post(f'{self.base_url}/user/repos', json=data)
        response.raise_for_status()
        return response.json()

# Usage
api = GitHubAPI('your_token_here')
user_info = api.get_user('octocat')

Web Scraping

def scrape_website(url):
    headers = {
        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36'
    }
    
    try:
        response = requests.get(url, headers=headers, timeout=10)
        response.raise_for_status()
        return response.text
    except RequestException as e:
        print(f"Error scraping {url}: {e}")
        return None

# Usage
html_content = scrape_website('https://example.com')

Frequently Asked Questions

How to Check if a Request is Successful?

response = requests.get('https://api.github.com')

# Method 1: Checking the ok attribute
if response.ok:
    print("Request successful")

# Method 2: Checking the status code
if response.status_code == 200:
    print("Request successful")

# Method 3: Using raise_for_status()
try:
    response.raise_for_status()
    print("Request successful")
except requests.exceptions.HTTPError:
    print("Error in the request")

How to Get the Final URL After Redirects?

response = requests.get('https://bit.ly/example')
print(f"Final URL: {response.url}")
print(f"Redirect history: {response.history}")

How to Disable SSL Verification?

# Disabling SSL verification (not recommended for production)
response = requests.get('https://self-signed.badssl.com/', verify=False)

# Specifying the path to the certificate
response = requests.get('https://api.example.com', verify='/path/to/cert.pem')

How to Limit the Number of Redirects?

# Disabling redirects
response = requests.get('https://example.com', allow_redirects=False)

# Limiting the number of redirects through an adapter
from requests.adapters import HTTPAdapter

adapter = HTTPAdapter(max_retries=3)
session = requests.Session()
session.mount('http://', adapter)
session.mount('https://', adapter)

How to Add Basic HTTP Authentication?

from requests.auth import HTTPBasicAuth

# Method 1: Using HTTPBasicAuth
auth = HTTPBasicAuth('username', 'password')
response = requests.get('https://api.example.com', auth=auth)

# Method 2: Passing a tuple
response = requests.get('https://api.example.com', auth=('username', 'password'))

Performance Optimization

Reusing Connections

# Use sessions for multiple requests
session = requests.Session()

# Executing multiple requests
urls = ['https://api.github.com/users/user1', 'https://api.github.com/users/user2']
results = []

for url in urls:
    response = session.get(url)
    results.append(response.json())

session.close()

Configuring the Connection Pool

from requests.adapters import HTTPAdapter

# Increasing the size of the connection pool
adapter = HTTPAdapter(pool_connections=100, pool_maxsize=100)
session = requests.Session()
session.mount('http://', adapter)
session.mount('https://', adapter)

Security When Working with Requests

Protection Against Attacks

# Always use HTTPS to transmit sensitive data
response = requests.get('https://api.example.com')

# Verify SSL certificates
response = requests.get('https://api.example.com', verify=True)

# Use environment variables for tokens
import os
token = os.environ.get('API_TOKEN')
headers = {'Authorization': f'Bearer {token}'}

Data Validation

def validate_response(response):
    """Validates the API response"""
    if not response.ok:
        raise requests.exceptions.HTTPError(f"HTTP {response.status_code}")
    
    content_type = response.headers.get('Content-Type', '')
    if 'application/json' not in content_type:
        raise ValueError("Response is not JSON")
    
    return response.json()

# Usage
response = requests.get('https://api.example.com')
data = validate_response(response)

Conclusion

The requests library is a powerful and flexible tool for working with HTTP requests in Python. It provides a simple interface for solving complex tasks, from simple data retrieval to creating full-fledged API clients.

Key principles for effective use of requests:

Always handle errors and exceptions
Use sessions for multiple requests
Set timeouts to prevent blocking
Apply correct authentication and security
Optimize performance through connection reuse

Mastering these skills will allow you to effectively integrate external APIs into your projects, create reliable web scrapers, and automate routine tasks of working with web resources.