C C++ CSS Full Stack HTML JavaScript Node PHP PYTHON React

PYTHON · LESSON 39

Iterators and Generators

Process streams lazily with iterator protocol and yield.

Level: Beginner to Advanced

Duration: ~60 Mins Deep-Dive

Updated: 26 Jul 2026

Lesson 39: Iterators and Generators

1. Introduction to High-Efficiency Stream Processing

Welcome to Module 5 of our Python Mastery Series: Advanced Functional Programming and Memory Optimization! Up to this point, whenever we processed collections of data (such as reading lists, populating dictionaries, or parsing JSON files), our programs used Eager Evaluation. Eager evaluation loads the complete dataset directly into system RAM all at once.

While loading small arrays into memory works well, enterprise data engineering often deals with multi-gigabyte log files, real-time IoT metric streams, or infinite sequence calculations. Attempting to load millions of records into a single Python list exhausts RAM instantly, triggering an Operating System MemoryError and crashing the service.

To solve this challenge, Python provides the Iteration Protocol alongside Lazy Evaluation tools: Iterators and Generators. In this lesson, we will dissect the underlying mechanics of __iter__() and __next__(), build custom iterator classes, master generator functions powered by the yield statement, compare Generator Expressions with List Comprehensions, and route data streams using Structural Pattern Matching (`match-case`).

2. The Python Iteration Protocol: Iterables vs Iterators

To write memory-efficient code, developers must understand the technical distinction between an Iterable and an Iterator:

1. What is an Iterable?

An Iterable is any Python object capable of returning its members one at a time. Examples include list, tuple, dict, set, and str. An object is an iterable if it implements the __iter__() dunder method, which produces a fresh Iterator instance when called via iter(obj).

2. What is an Iterator?

An Iterator is a value-stream pointer object. It implements two mandatory dunder methods (known as the Iterator Protocol):

__iter__(self): Returns the iterator object itself.
__next__(self): Computes and returns the next item in the sequence. When no more items remain, it raises a StopIteration exception to signal the end of the stream!

Protocol Methods
Implements __iter__()
Implements __iter__() AND __next__()
State Maintenance
Holds full collection state in memory. Can be iterated over multiple times.
Stateful active cursor pointer. Consumed items are lost permanently! Single-use.
Creation Syntax
Literal collections: [1, 2, 3]
Calling iter(iterable) or Generator Functions

Protocol Dimension	Python Iterable (e.g., `list`)	Python Iterator (e.g., `iter(list)`)

ITERATION_PROTOCOL_DEMO.PY

raw_numbers = [10, 20, 30]

# 1. Obtaining an Iterator pointer from an Iterable list using iter()
num_iterator = iter(raw_numbers)

print("=== Demystifying the Low-Level Iteration Loop ===")
print("Object Type of raw_numbers :", type(raw_numbers))
print("Object Type of num_iterator:", type(num_iterator))

# 2. Stepping through the stream manually via next()
print("\nFirst next() call :", next(num_iterator))
print("Second next() call:", next(num_iterator))
print("Third next() call :", next(num_iterator))

# 3. Exhausted stream raises StopIteration!
try:
    print(next(num_iterator))
except StopIteration:
    print("\n[STREAM EXHAUSTED] StopIteration exception caught cleanly!")

OUTPUT

=== Demystifying the Low-Level Iteration Loop ===
Object Type of raw_numbers : 
Object Type of num_iterator: 

First next() call : 10
Second next() call: 20
Third next() call : 30

[STREAM EXHAUSTED] StopIteration exception caught cleanly!

How `for` Loops Work Behind the Scenes: When you write for x in my_list:, Python executes three hidden steps: (1) Calls it = iter(my_list) to obtain an iterator pointer, (2) Calls x = next(it) continuously inside an implicit loop, and (3) Catches StopIteration automatically to terminate the loop cleanly without throwing an error!

3. Creating Custom Iterator Classes

You can transform any custom class into a stateful stream iterator by implementing the __iter__() and __next__() methods explicitly.

CUSTOM_ITERATOR_CLASS.PY

class RangeStepIterator:
    """Custom iterator generating an arithmetic progression step stream."""
    def __init__(self, start: int, stop: int, step: int = 1):
        self.current = start
        self.stop = stop
        self.step = step

    def __iter__(self):
        return self  # The iterator returns itself

    def __next__(self) -> int:
        if (self.step > 0 and self.current >= self.stop) or (self.step < 0 and self.current <= self.stop):
            raise StopIteration
        
        value = self.current
        self.current += self.step
        return value

print("=== Iterating over Custom RangeStepIterator ===")
custom_range = RangeStepIterator(10, 30, 5)

for val in custom_range:
    print(f"Stream Step Value: {val}")

OUTPUT

=== Iterating over Custom RangeStepIterator ===
Stream Step Value: 10
Stream Step Value: 15
Stream Step Value: 20
Stream Step Value: 25

4. Lazy Evaluation with Generator Functions and `yield`

While writing custom iterator classes works well, managing manual class state variables inside __next__() can be complex and error-prone. Python provides a much simpler alternative: Generators.

A Generator Function is a standard Python function that contains the yield keyword instead of return. When called, a generator function does NOT execute its body immediately; instead, it returns a Generator Object that automatically implements the full Iterator Protocol!

1. The Mechanics of the `yield` Keyword

When next(gen) is called, execution starts (or resumes) and runs until it reaches a yield value statement.
Python evaluates value, returns it to the caller, and pauses function execution state (preserving local variables, stack pointer, and instruction position in memory).
On the subsequent next(gen) call, execution resumes instantly from the exact line following the yield statement!

GENERATOR_YIELD_DEMO.PY

def fibonacci_stream_generator(limit_count: int):
    """
    Generator function yielding a Fibonacci number sequence lazily.
    Uses constant O(1) memory space regardless of limit_count!
    """
    a, b = 0, 1
    count = 0
    while count < limit_count:
        yield a  # Yield value and pause execution state
        a, b = b, a + b
        count += 1

print("=== Consuming Fibonacci Generator Stream ===")
fib_gen = fibonacci_stream_generator(7)

print("Generated Object Type:", type(fib_gen))

for idx, num in enumerate(fib_gen, start=1):
    print(f"Fibonacci Item #{idx}: {num}")

OUTPUT

=== Consuming Fibonacci Generator Stream ===
Generated Object Type: 
Fibonacci Item #1: 0
Fibonacci Item #2: 1
Fibonacci Item #3: 1
Fibonacci Item #4: 2
Fibonacci Item #5: 3
Fibonacci Item #6: 5
Fibonacci Item #7: 8

5. Generator Expressions vs List Comprehensions: Memory Benchmark

Just as List Comprehensions construct lists with [expr for item in iterable], Generator Expressions construct lazy stream iterators using parentheses syntax: (expr for item in iterable).

Memory Performance Benchmark: A List Comprehension computes and stores ALL items in RAM simultaneously ($O(n)$ space). A Generator Expression computes items on demand one by one ($O(1)$ constant memory), making it vastly superior for large datasets!

GENERATOR_MEMORY_BENCHMARK.PY

import sys

# Testing scalability over 1,000,000 integers
N = 1_000_000

# 1. Eager List Comprehension (Allocates full array in RAM)
list_comp = [x * 2 for x in range(N)]

# 2. Lazy Generator Expression (Allocates only iterator cursor in RAM)
gen_expr = (x * 2 for x in range(N))

print("=== Memory Allocation Benchmark Comparison ===")
print(f"List Comprehension Size (1M items): {sys.getsizeof(list_comp):,} bytes")
print(f"Generator Expression Size (1M items): {sys.getsizeof(gen_expr):,} bytes")
print(f"Memory Efficiency Ratio             : List is ~{sys.getsizeof(list_comp) // sys.getsizeof(gen_expr):,}x LARGER in RAM!")

OUTPUT

=== Memory Allocation Benchmark Comparison ===
List Comprehension Size (1M items): 8,448,728 bytes
Generator Expression Size (1M items): 208 bytes
Memory Efficiency Ratio             : List is ~40,618x LARGER in RAM!

6. Combining Generator Streams with `match-case` Pattern Matching

Building scalable data transformation pipelines involves using generator functions to read raw data line-by-line, and passing individual stream items into match-case structural pattern dispatchers for real-time routing and filtering.

MATCH_CASE_STREAM_PIPELINE.PY

def log_stream_generator():
    """Simulates an infinite or continuous log line stream generator."""
    raw_logs = [
        ("INFO", "AuthService", "User logged in successfully"),
        ("WARN", "Database", "Query latency high: 450ms"),
        ("ERROR", "PaymentGateway", "Connection timeout to bank API"),
        ("CRITICAL", "SecurityVault", "Unauthorized token access attempt!")
    ]
    for log_item in raw_logs:
        yield log_item  # Stream item lazily

def process_log_pipeline(stream_generator):
    """
    Consumes a generator stream and dispatches records using match-case.
    Demonstrates low-memory pipeline processing.
    """
    print("=== Processing Lazy Log Stream via Pattern Matcher ===")
    for log_entry in stream_generator:
        match log_entry:
            case ("CRITICAL", service, msg):
                print(f"[SECURITY ALERT] Emergency in '{service}': {msg}")
                
            case ("ERROR" | "WARN", service, msg):
                print(f"[SYSTEM WARNING] Issues in '{service}': {msg}")
                
            case ("INFO", service, msg):
                print(f"[SYSTEM LOG] Routine info from '{service}': {msg}")
                
            case _:
                print(f"[UNKNOWN] Unrecognized log schema: {log_entry}")

# Executing pipeline
process_log_pipeline(log_stream_generator())

OUTPUT

=== Processing Lazy Log Stream via Pattern Matcher ===
[SYSTEM LOG] Routine info from 'AuthService': User logged in successfully
[SYSTEM WARNING] Issues in 'Database': Query latency high: 450ms
[SYSTEM WARNING] Issues in 'PaymentGateway': Connection timeout to bank API
[SECURITY ALERT] Emergency in 'SecurityVault': Unauthorized token access attempt!

7. Frequently Asked Interview Questions with Answers

Q1: What is the technical difference between an Iterable and an Iterator in Python?

Answer: An Iterable is an object that implements __iter__() to produce an iterator (e.g., lists, tuples). An Iterator is a stateful stream cursor object that implements both __iter__() and __next__(). Calling next(iterator) yields the next element until a StopIteration exception is raised.

Q2: How does the `yield` keyword differ from the standard `return` keyword in functions?

Answer: return terminates function execution completely and returns a value, destroying local stack variables. yield returns a value to the caller and **pauses function execution state**, preserving all local variables and instruction pointers so execution can resume from that exact position on the next next() invocation.

Q3: Why do Generator Expressions require significantly less memory than List Comprehensions?

Answer: List Comprehensions use **Eager Evaluation**, building and holding the entire dataset in RAM simultaneously ($O(n)$ space). Generator Expressions use **Lazy Evaluation**, computing and yielding one item at a time on demand ($O(1)$ constant space memory footprint).

Q4: What happens when an Iterator reaches the end of its sequence items?

Answer: Python raises a StopIteration exception. Constructs like for loops, list() constructors, and sum() catch this exception automatically to exit iteration gracefully.

Q5: Can an Iterator be reused once it has been fully consumed?

Answer: No. Iterators are single-use objects that maintain a forward-only stateful cursor. Once exhausted, calling next() continually raises StopIteration. To iterate again, a new iterator object must be created from the underlying iterable.

8. Homework & Practical Assignments

Task 1: Custom Batch Batching Generator Function

Create a script named batch_generator_task.py inside your lesson_39 folder:

Define a generator function chunk_data_stream(dataset: list, chunk_size: int).
Use a while loop or slicing to yield sub-lists (chunks) of size chunk_size lazily.
Test function with a list of 10 items using chunk_size=3 and iterate over generated chunks using a for loop.
Print each yielded chunk and its length using f-strings.

Task 2: Pattern-Matched Sensor Generator Pipeline

Create a script named sensor_pipeline_task.py:

Define generator read_sensor_metrics() yielding tuples: ("TEMP", 25.5), ("CRITICAL_TEMP", 105.0), ("PRESSURE", 1.2), ("UNKNOWN", 0).
Write a dispatcher function consuming the stream using for entry in read_sensor_metrics(): with match-case pattern matching:
- case ("CRITICAL_TEMP", val) if val > 100.0 → Output emergency alert string.
- case ("TEMP" | "PRESSURE", val) → Output normal metric string.
- case _ → Output unknown signal error.
Execute pipeline and print outputs.

Task 3: Master Review Capstone Project — High-Performance Memory-Efficient Log Pipeline OS (`log_pipeline_os.py`)

Create a script named log_pipeline_os.py inside your lesson_39 folder. This assignment tests and integrates **ALL concepts learned across Lessons 1 through 39**.

Project Architectural Requirements Specification:

Iterators & Generator Architecture (Lesson 39):
- Implement custom iterator class LogFileStreamIterator(filepath) streaming file lines lazily.
- Implement generator function parse_log_records(line_stream) yielding sanitized record dicts.
- Use Generator Expressions for filtering high-severity records without loading full files into RAM.
Full OOP Architecture (Lessons 32-38):
- Define abstract class BaseProcessor(ABC) and concrete Data Classes TelemetryMetric and LogEvent.
- Build composite manager PipelineOS containing storage and audit logger components.
Observability & Fault Tolerance (Lessons 29-31):
- Set up multi-handler logging to console and pipeline.log file.
- Incorporate developer assertions (assert) verifying batch count non-negativity.
- Define custom exceptions (PipelineError, CorruptedStreamError).
Context Managers & Persistence Layer (Lessons 27-30):
- Use custom class-based context manager FileStreamLock to manage stream locks.
- Persist metric summaries to metrics.json and export CSV audits to audit_report.csv using pathlib.Path.
Nested Collections, Comprehensions & Data Structures (Lessons 21-26):
- Maintain an in-memory dictionary mapping event IDs to Data Class Instances.
- Use List and Dictionary Comprehensions to aggregate streaming metrics.
Advanced Function Parameters & Scope (Lessons 15-17):
- Structure logic into pure modular functions with type hints, docstrings, early return Guard Clauses, and *args / **kwargs logging.
Indefinite & Definite Loops (Lessons 13-14):
- Run the interactive CLI interface inside a continuous while True menu loop.
- Iterate through record streams using for loops with enumerate() and .items().
Pattern Matching CLI Dispatcher (Lesson 12):
- Process user CLI command tokens inside a match-case block:
  - case ["STREAM", filename, *filters] → Process file lazily via LogFileStreamIterator and match severity.
  - case ["BENCHMARK", filename] → Compare RAM size of List Comprehension vs Generator Expression using sys.getsizeof().
  - case ["EXPORT", "CSV"] → Export active records to CSV.
  - case ["EXIT" | "QUIT"] → Terminate session safely using a sentinel flag.
  - case _ → Output command error message.
Conditionals, Formatting & Foundations (Lessons 1-11):
- Sanitize all inputs using .strip() and .upper().
- Format tabular reports using f-strings with field width alignment specifiers (:<15, :>10) and clear visual borders.

9. File & Workspace Directory Structure

Standard Course Workspace Layout:

Ensure all exercise files are stored inside their corresponding lesson directories:

python_mastery_course/
│
├── lesson_01/ ... lesson_38/
│
└── lesson_39/
    ├── iteration_protocol_demo.py
    ├── custom_iterator_class.py
    ├── generator_yield_demo.py
    ├── generator_memory_benchmark.py
    ├── match_case_stream_pipeline.py
    ├── batch_generator_task.py     <-- (Task 1)
    ├── sensor_pipeline_task.py    <-- (Task 2)
    └── log_pipeline_os.py         <-- (Task 3: Master Review Capstone)

10. What We Will Learn Next

Next Up: Lesson 40 — Decorators

Now that you master Iterators, Generators, and Lazy Evaluation, we will explore metaclass functional transformation using Decorators!

In the next lesson, we will cover:

First-Class Functions, Closures, and High-Order Functions in Python.
Understanding the Decorator Pattern (`@decorator` syntax).
Preserving function signatures and metadata using `functools.wraps`.
Creating Decorators that accept custom arguments.
Building Class-Based Decorators and combining decorated functions with match-case pattern dispatchers.

📝 Live Lesson Practice

HTML/CSS JavaScript Python C++ C PHP

⌨️ Practice Inputs (लाइव इनपुट भरें) (खाली होने पर RED, भरने पर GREEN underline)

💻 Code Editor (Monaco VS Code Engine)

👀 Live Preview

Project: Object-Oriented Inventory System Decorators

PYTHON Tutorial

Iterators and Generators

1. Introduction to High-Efficiency Stream Processing

2. The Python Iteration Protocol: Iterables vs Iterators

1. What is an Iterable?

2. What is an Iterator?

3. Creating Custom Iterator Classes

4. Lazy Evaluation with Generator Functions and `yield`

1. The Mechanics of the `yield` Keyword

5. Generator Expressions vs List Comprehensions: Memory Benchmark

6. Combining Generator Streams with `match-case` Pattern Matching

7. Frequently Asked Interview Questions with Answers

8. Homework & Practical Assignments

Task 1: Custom Batch Batching Generator Function

Task 2: Pattern-Matched Sensor Generator Pipeline

Task 3: Master Review Capstone Project — High-Performance Memory-Efficient Log Pipeline OS (`log_pipeline_os.py`)

Project Architectural Requirements Specification:

9. File & Workspace Directory Structure

Standard Course Workspace Layout:

10. What We Will Learn Next

Next Up: Lesson 40 — Decorators

📝 Live Lesson Practice

OnlineCBT

Useful Link

Important Pages

Courses & Practice

Contact

Share