C C++ CSS Full Stack HTML JavaScript Node PHP PYTHON React

PYTHON · LESSON 54

HTTP APIs and the requests Library

Consume web APIs with timeouts, status checks and validation.

Level: Beginner to Advanced

Duration: ~60 Mins Deep-Dive

Updated: 26 Jul 2026

Lesson 54: HTTP APIs and the requests Library

1. Introduction to Web Communication and RESTful Architecture

In Lesson 53, we mastered local database persistence using SQLite and SQL queries. However, modern enterprise software systems do not run in total isolation on a single local server. Applications must communicate across internal network microservices, integrate third-party payment processing gateways (like Stripe or PayPal), send transactional emails, or query real-time market data feeds.

The universal foundation of distributed web communications is the Hypertext Transfer Protocol (HTTP) and REST (Representational State Transfer) API Architecture. RESTful web APIs expose endpoint URLs that accept structured HTTP client requests and return standard JSON data payloads.

While Python contains low-level networking tools in its standard library (such as urllib.request), its syntax is verbose, complex, and un-pythonic. The global Python software community relies on the industry-standard third-party library: `requests` ("HTTP for Humans").

In this lesson, we will master the mechanics of HTTP Verbs and Status Codes, execute API requests using Python's requests package, handle query parameters, custom headers, and JSON payloads (`response.json()`), enforce network timeouts, implement API Authentication (Bearer Tokens and API Keys), leverage persistent HTTP sessions via requests.Session(), and route web API response streams using Structural Pattern Matching (`match-case`).

2. HTTP Protocol Fundamentals: Verbs, Status Codes & Headers

An HTTP transaction consists of a **Client Request** sent to a server URI, and a corresponding **Server Response** returned to the client.

1. HTTP Request Methods (Verbs) Matrix

GET
Read
Yes
Retrieves data from a server endpoint without modifying server state.
POST
Create
No
Submits payload data to create a new server resource.
PUT
Update / Replace
Yes
Completely overwrites/replaces an existing target resource on the server.
PATCH
Partial Update
No
Modifies specific modified attributes on an existing server resource.
DELETE
Delete
Yes
Removes the target resource permanently from the server.

HTTP Method Verb	CRUD Operation Mapping	Idempotent Status	Primary Purpose / Description

2. HTTP Response Status Code Classes Summary

2xx
Success
200 OK, 201 Created, 204 No Content
Request was received, understood, and processed successfully.
3xx
Redirection
301 Moved Permanently, 302 Found
Target resource has moved to a different URI location.
4xx
Client Error
400 Bad Request, 401 Unauthorized, 404 Not Found
The client request contains invalid parameters, missing credentials, or missing routes.
5xx
Server Error
500 Internal Error, 502 Bad Gateway, 503 Service Unavailable
The remote web server encountered an unhandled exception or crash while processing.

Code Range	Category Class	Common Examples	Meaning Description

3. Making Web Requests using the `requests` Library

To get started, install the third-party library using pip install requests.

1. Executing `GET` and `POST` Requests

Making a web request is as simple as invoking requests.get(url) or requests.post(url, json=data). The method returns a rich Response object containing HTTP status codes, headers, raw content, and automated JSON parsing methods!

REQUESTS_BASIC_GET_POST.PY

import requests

# 1. Executing HTTP GET Request with URL Query Parameters (?search=python&limit=5)
target_url = "https://httpbin.org/get"
query_params = {"search": "python_mastery", "limit": 5}
custom_headers = {"User-Agent": "EnterpriseVaultApp/1.0.0"}

print("=== 1. Executing HTTP GET Request ===")
response_get = requests.get(target_url, params=query_params, headers=custom_headers, timeout=5.0)

print("HTTP Status Code :", response_get.status_code)
print("Response Header  :", response_get.headers.get("Content-Type"))
# Extracting parsed JSON body directly using .json() method!
parsed_get_data = response_get.json()
print("Received Query Args from Server:", parsed_get_data.get("args"))

# 2. Executing HTTP POST Request with JSON Body Payload
post_url = "https://httpbin.org/post"
json_payload = {"account_id": "ACC_9901", "action": "DEPOSIT", "amount": 500.0}

print("\n=== 2. Executing HTTP POST Request ===")
response_post = requests.post(post_url, json=json_payload, timeout=5.0)

print("HTTP Status Code :", response_post.status_code)
parsed_post_data = response_post.json()
print("Echoed JSON Payload from Server:", parsed_post_data.get("json"))

OUTPUT

=== 1. Executing HTTP GET Request ===
HTTP Status Code : 200
Response Header  : application/json
Received Query Args from Server: {'limit': '5', 'search': 'python_mastery'}

=== 2. Executing HTTP POST Request ===
HTTP Status Code : 200
Echoed JSON Payload from Server: {'account_id': 'ACC_9901', 'action': 'DEPOSIT', 'amount': 500.0}

4. API Authentication & Session Optimization

Enterprise web APIs require authentication credentials to verify client permissions. Common authentication strategies include:

API Keys (Query Param or Custom Header): Passing key strings via headers: headers={"X-API-KEY": "secret_key_123"}.
Bearer Token Authorization (OAuth 2.0 / JWT): Passing encrypted JSON Web Tokens via HTTP headers: headers={"Authorization": "Bearer eyJhbGci..."}.
HTTP Basic Authentication: Passing username/password tuples via auth=("user", "pass").

1. Connection Pooling via `requests.Session()`

When executing dozens of sequential API calls to the same host domain, creating new TCP sockets and renegotiating TLS/SSL handshakes on every call is inefficient.

Performance Optimization with `requests.Session()`: Using a requests.Session() object enables TCP connection pooling (reusing the open network socket) and persists common request headers and authentication tokens across all subsequent API requests automatically!

REQUESTS_SESSION_AUTH_DEMO.PY

import requests

# Creating persistent Session object
session = requests.Session()

# Pre-configuring global headers and Bearer Token on session instance
session.headers.update({
    "Authorization": "Bearer ENTERPRISE_SECRET_TOKEN_XYZ",
    "Accept": "application/json",
    "User-Agent": "EnterpriseClientOS/2.0"
})

print("=== Executing Secure API Requests via Persistent Session ===")
# All calls made through session reuse network sockets and attach headers automatically!
response = session.get("https://httpbin.org/headers", timeout=5.0)

if response.status_code == 200:
    headers_echo = response.json().get("headers", {})
    print("Server Received Authorization Header:", headers_echo.get("Authorization"))
    print("Server Received User-Agent Header   :", headers_echo.get("User-Agent"))

session.close()  # Closing pooled connections cleanly

OUTPUT

=== Executing Secure API Requests via Persistent Session ===
Server Received Authorization Header: Bearer ENTERPRISE_SECRET_TOKEN_XYZ
Server Received User-Agent Header   : EnterpriseClientOS/2.0

5. Robust Error Handling & Network Timeouts

Distributed network communications are inherently unpredictable—remote DNS servers can fail, API gateways can time out, or networks can drop packets. Enterprise HTTP code MUST incorporate strict error handling.

Mandatory Timeout Rule: NEVER execute requests.get(url) without specifying an explicit timeout=seconds parameter! Omitting a timeout causes Python to block indefinitely if the target server hangs, freezing your entire application process permanently!

1. Raising HTTP Exception Pass via `response.raise_for_status()`

By default, requests does NOT raise an exception when a server returns a 404 Not Found or 500 Server Error status code. Calling response.raise_for_status() checks the status code and raises an HTTPError automatically if the status code indicates a 4xx or 5xx failure!

REQUESTS_FAULT_TOLERANCE_DEMO.PY

import requests

def fetch_external_api_safely(endpoint_url: str) -> dict | None:
    """
    Executes network requests with complete exception handling hierarchy.
    Handles Timeouts, Connection Errors, and HTTP status errors cleanly.
    """
    try:
        response = requests.get(endpoint_url, timeout=3.0)
        
        # Throws requests.exceptions.HTTPError if status is 4xx or 5xx!
        response.raise_for_status()
        
        return response.json()

    except requests.exceptions.Timeout:
        print(f"[NETWORK ERROR] Connection to '{endpoint_url}' timed out (Server unresponsive)!")
    except requests.exceptions.ConnectionError:
        print(f"[NETWORK ERROR] Failed to establish DNS / socket connection to '{endpoint_url}'!")
    except requests.exceptions.HTTPError as http_err:
        print(f"[HTTP ERROR] Server returned error status: {http_err}")
    except requests.exceptions.RequestException as req_err:
        print(f"[REQUEST ERROR] General networking exception caught: {req_err}")

    return None

print("=== Testing Fault Tolerant Network Calls ===")
# 1. Testing 404 Client Error Handling
fetch_external_api_safely("https://httpbin.org/status/404")

# 2. Testing 500 Server Error Handling
fetch_external_api_safely("https://httpbin.org/status/500")

OUTPUT

=== Testing Fault Tolerant Network Calls ===
[HTTP ERROR] Server returned error status: 404 Client Error: NOT FOUND for url: https://httpbin.org/status/404
[HTTP ERROR] Server returned error status: 500 Server Error: INTERNAL SERVER ERROR for url: https://httpbin.org/status/500

6. Combining API Responses with `match-case` Pattern Matching

Building web API consumer services involves querying external endpoints and routing returned JSON dictionary payloads using Python 3.10+ `match-case` Structural Pattern Matching.

MATCH_CASE_API_DISPATCHER.PY

import requests

def dispatch_api_response_payload(json_data: dict) -> str:
    """
    Routes JSON response dictionaries directly using Structural Pattern Matching.
    Demonstrates handling API response schema variations dynamically.
    """
    match json_data:
        case {"status": "SUCCESS", "data": {"user_id": str(uid), "balance": float(bal)}} if bal >= 10000.0:
            return f"MATCH_PRIME_VIP: User '{uid}' holds VIP balance of ${bal:,.2f}"

        case {"status": "SUCCESS", "data": {"user_id": str(uid), "balance": float(bal)}}:
            return f"MATCH_STANDARD: User '{uid}' account active (${bal:,.2f})"

        case {"status": "ERROR", "error_code": int(code), "message": str(msg)}:
            return f"MATCH_API_ERROR [{code}]: Remote API error -> {msg}"

        case {"url": str(endpoint_url), "args": dict() as query_args}:
            return f"MATCH_HTTPBIN_ECHO: Parsed response from '{endpoint_url}' with args {query_args}"

        case _:
            return f"UNRECOGNIZED_SCHEMA: Unable to parse API JSON payload: {json_data}"

# Testing Dispatcher with Simulated and Live API Objects
res_get = requests.get("https://httpbin.org/get", params={"type": "AUDIT"}, timeout=5.0)

print("=== Pattern Matched API Response Outputs ===")
print("1. Live Echo Query Result :", dispatch_api_response_payload(res_get.json()))
print("2. Mock VIP Success Result:", dispatch_api_response_payload({
    "status": "SUCCESS",
    "data": {"user_id": "USR_9901", "balance": 25000.0}
}))
print("3. Mock API Error Result  :", dispatch_api_response_payload({
    "status": "ERROR",
    "error_code": 403,
    "message": "Token quota limit exceeded!"
}))

OUTPUT

=== Pattern Matched API Response Outputs ===
1. Live Echo Query Result : MATCH_HTTPBIN_ECHO: Parsed response from 'https://httpbin.org/get?type=AUDIT' with args {'type': 'AUDIT'}
2. Mock VIP Success Result: MATCH_PRIME_VIP: User 'USR_9901' holds VIP balance of $25,000.00
3. Mock API Error Result  : MATCH_API_ERROR [403]: Remote API error -> Token quota limit exceeded!

7. Frequently Asked Interview Questions with Answers

Q1: What is the purpose of the `requests.Session()` object, and how does it improve network performance?

Answer: requests.Session() manages persistent HTTP connections using connection pooling (reusing open TCP sockets across multiple API calls). This eliminates the performance overhead of repeated DNS lookups, TCP three-way handshakes, and TLS/SSL certificate negotiations, while automatically sharing headers and cookies across calls.

Q2: Why is setting explicit timeouts mandatory when invoking network requests?

Answer: By default, HTTP network library calls without a timeout will wait indefinitely if a remote web server hangs or fails to send response packets. Setting an explicit timeout (e.g., timeout=5.0) ensures Python raises a requests.exceptions.Timeout exception, preventing process threads from freezing permanently.

Q3: How does `response.raise_for_status()` handle HTTP error response codes?

Answer: response.raise_for_status() checks the HTTP response status code. If the code falls within the 4xx Client Error or 5xx Server Error ranges, it raises a requests.exceptions.HTTPError exception automatically, allowing standard exception handling blocks to catch errors cleanly.

Q4: What is the difference between `params` and `json` parameters in `requests.get()` and `requests.post()`?

Answer: The params parameter formats a dictionary into URL query strings (e.g., ?key=val) appended to the URI line. The json parameter serializes a Python dictionary into a JSON formatted string, attaches the header Content-Type: application/json, and places the payload inside the HTTP Request Body.

Q5: How do you pass Bearer Authentication Tokens in `requests` headers?

Answer: By passing an HTTP Authorization header in the headers dictionary: headers={"Authorization": f"Bearer {access_token}"} or pre-configuring it directly on a requests.Session() instance.

8. Homework & Practical Assignments

Task 1: Weather API Fetcher with Timeout and Error Handling

Create a script named http_weather_task.py inside your lesson_54 folder:

Import requests and set up a request to a public mock API (or https://httpbin.org/get).
Define function fetch_endpoint_data(url: str, params: dict) -> dict | None.
Include explicit timeout=4.0 and call response.raise_for_status().
Wrap execution inside a try-except block catching Timeout, HTTPError, and RequestException.
Print formatted results using f-strings.

Task 2: Pattern-Matched API Status Code Dispatcher

Create a script named api_status_task.py:

Build function process_api_response(response_obj: requests.Response) -> str.
Write a match-case dispatcher analyzing response_obj.status_code:
- case 200 | 201 → Return "SUCCESS_OK: Payload retrieved cleanly."
- case 401 | 403 → Return "AUTH_ERROR: Invalid or expired API credentials!"
- case 404 → Return "NOT_FOUND: Target URI endpoint missing."
- case code if code >= 500 → Return "SERVER_CRASH: Remote service unavailable."
- case _ → Return "UNKNOWN_STATUS_CODE".
Execute test calls against https://httpbin.org/status/200 and 403 and print result strings.

Task 3: Master Review Capstone Project — Production Enterprise HTTP Gateway & REST API Audit OS (`api_gateway_os.py`)

Create a script named api_gateway_os.py inside your lesson_54 folder. This assignment tests and integrates **ALL concepts learned across Lessons 1 through 54**.

Project Architectural Requirements Specification:

HTTP APIs & requests Architecture (Lesson 54):
- Implement HTTP communication layers using requests.Session() connection pools with Bearer token headers.
- Enforce strict network timeouts, response.raise_for_status(), and complete exception handling hierarchies.
- Parse JSON response payloads directly into domain models.
SQLite Persistence Integration (Lesson 53):
- Persist fetched HTTP API audit records into an SQLite database table using parameterized SQL statements and sqlite3.Row.
Security, Performance & Memory Hardening (Lesson 52):
- Read API secrets dynamically from environment variables using os.environ.
- Use __slots__ across API response data classes for memory optimization.
Packaging & Distribution Integration (Lesson 51):
- Include a programmatically generated pyproject.toml package specification string and an argparse CLI entry point structure.
Clean Code, Documentation & Refactoring (Lesson 50):
- Apply SOLID principles, DRY, and Google Style Docstrings across all functions and classes.
Testing & Quality Assurance (Lessons 47-48):
- Include unit tests verified via pytest fixtures and unittest.mock (mocking requests.get).
Asyncio & Concurrency Integration (Lessons 45-46):
- Incorporate thread pool workers or async event loops for parallel multi-endpoint health checks.
Temporal & RegEx Text Processing (Lessons 43-44):
- Parse UTC-aware timestamps using datetime and zoneinfo.ZoneInfo.
- Pre-compile RegEx patterns with Named Groups for parsing URI paths.
Type Hints, Generics & Static Safety (Lesson 42):
- Apply explicit modern type annotations (X | Y, list[str], dict[str, Any]) throughout all functions and classes.
Metaprogramming, Closures & Decorators (Lessons 40-41):
- Build function factories and 3-level configurable decorators with @functools.wraps.
Iterators & Generators (Lesson 39):
- Implement generator functions streaming API response batches lazily with $O(1)$ memory.
Full OOP Architecture (Lessons 32-38):
- Define abstract base class BaseAPIService(ABC) and concrete Data Classes APIEndpointRecord and GatewayAudit.
- Build composite manager APIGatewayOS composing database drivers and network session handles.
Observability & Fault Tolerance (Lessons 29-31):
- Set up multi-handler logging to console and api_gateway.log file.
- Incorporate developer assertions (assert) verifying status code bounds.
- Define custom exception hierarchy (GatewayOSError, APITimeoutError).
Context Managers & Persistence Layer (Lessons 27-30):
- Use custom class-based context managers to manage network sessions cleanly.
- Persist JSON payload backups to api_db.json and export CSV reports to api_audit.csv using pathlib.Path.
Nested Collections & Comprehensions (Lessons 21-26):
- Maintain in-memory registry dictionaries mapping Endpoint IDs to Data Class Instances.
- Use List and Dictionary Comprehensions to clean and transform records.
Advanced Function Parameters & Scope (Lessons 15-17):
- Structure logic into pure modular functions with type hints, docstrings, early return Guard Clauses, and *args / **kwargs logging.
Indefinite & Definite Loops (Lessons 13-14):
- Run the interactive CLI interface inside a continuous while True menu loop.
- Iterate through output reports using for loops with enumerate() and .items().
Pattern Matching CLI Dispatcher (Lesson 12):
- Process user CLI command tokens inside a match-case block with HTTP match guards:
  - case ["FETCH", "GET", target_url] → Execute GET request via session pool.
  - case ["POST", "JSON", target_url, payload_str] → Execute POST request with JSON body.
  - case ["AUDIT", "RESPONSES"] → Route API response dictionaries using match-case pattern dispatcher.
  - case ["EXPORT", "CSV"] → Export completed API response log history to CSV.
  - case ["EXIT" | "QUIT"] → Terminate session safely using a sentinel flag.
  - case _ → Output command error message.
Conditionals, Formatting & Foundations (Lessons 1-11):
- Sanitize all inputs using .strip() and .upper().
- Format tabular reports using f-strings with field width alignment specifiers (:<15, :>10) and clear visual borders.

9. File & Workspace Directory Structure

Standard Course Workspace Layout:

Ensure all exercise files are stored inside their corresponding lesson directories:

python_mastery_course/
│
├── lesson_01/ ... lesson_53/
│
└── lesson_54/
    ├── requests_basic_get_post.py
    ├── requests_session_auth_demo.py
    ├── requests_fault_tolerance_demo.py
    ├── match_case_api_dispatcher.py
    ├── http_weather_task.py          <-- (Task 1)
    ├── api_status_task.py            <-- (Task 2)
    └── api_gateway_os.py             <-- (Task 3: Master Review Capstone)

10. What We Will Learn Next

Next Up: Lesson 55 — Web Scraping and Responsible Data Collection

Now that you master querying structured RESTful HTTP APIs using the requests library, we will explore extracting data from unstructured HTML websites!

In the next lesson, we will cover:

Understanding HTML Structure, DOM Trees, and CSS Selectors.
Parsing HTML documents using BeautifulSoup4 (`bs4`).
Extracting text, links (`href`), and attribute streams from HTML elements.
Ethical Web Scraping Guidelines: Respecting robots.txt, rate limiting, and User-Agent ethics.
Combining HTML scraped element streams with match-case structural pattern matchers.

📝 Live Lesson Practice

HTML/CSS JavaScript Python C++ C PHP

⌨️ Practice Inputs (लाइव इनपुट भरें) (खाली होने पर RED, भरने पर GREEN underline)

💻 Code Editor (Monaco VS Code Engine)

👀 Live Preview

SQLite and Relational Databases Web Scraping and Responsible Data Collection

PYTHON Tutorial

HTTP APIs and the requests Library

1. Introduction to Web Communication and RESTful Architecture

2. HTTP Protocol Fundamentals: Verbs, Status Codes & Headers

1. HTTP Request Methods (Verbs) Matrix

2. HTTP Response Status Code Classes Summary

3. Making Web Requests using the `requests` Library

1. Executing `GET` and `POST` Requests

4. API Authentication & Session Optimization

1. Connection Pooling via `requests.Session()`

5. Robust Error Handling & Network Timeouts

1. Raising HTTP Exception Pass via `response.raise_for_status()`

6. Combining API Responses with `match-case` Pattern Matching

7. Frequently Asked Interview Questions with Answers

8. Homework & Practical Assignments

Task 1: Weather API Fetcher with Timeout and Error Handling

Task 2: Pattern-Matched API Status Code Dispatcher

Task 3: Master Review Capstone Project — Production Enterprise HTTP Gateway & REST API Audit OS (`api_gateway_os.py`)

Project Architectural Requirements Specification:

9. File & Workspace Directory Structure

Standard Course Workspace Layout:

10. What We Will Learn Next

Next Up: Lesson 55 — Web Scraping and Responsible Data Collection

📝 Live Lesson Practice

OnlineCBT

Useful Link

Important Pages

Courses & Practice

Contact

Share