ch7s3_GeneratorsAndIterators

Generators and iterators form the foundation of **lazy evaluation** in Python.

Chapter 7: Functional Programming

Sub-Chapter: Generators and Iterators — Lazy and Efficient Data Processing

Generators and iterators form the foundation of lazy evaluation in Python.
They allow you to handle sequences of data one element at a time, making your programs memory-efficient, stream-friendly, and ideal for large datasets or infinite streams.

🧩 1. Iterables vs. Iterators

Iterable

An iterable is any Python object that can return an iterator — for example, lists, tuples, strings, sets, and dictionaries.

nums = [1, 2, 3]
it = iter(nums)
print(next(it))  # 1
print(next(it))  # 2

Iterator

An iterator is an object with two methods:

• __iter__() — returns itself (the iterator object)
• __next__() — returns the next value or raises StopIteration when done

Example of a custom iterator:

class Counter:
    def __init__(self, start, end):
        self.current = start
        self.end = end

    def __iter__(self):
        return self

    def __next__(self):
        if self.current > self.end:
            raise StopIteration
        self.current += 1
        return self.current - 1

for num in Counter(1, 5):
    print(num)  # 1, 2, 3, 4, 5

✅ Every generator is an iterator, but not every iterator is a generator.

⚙️ 2. The Iterator Protocol

🧠 3. Generators — Functions that Remember State

A generator is a special kind of iterator created with a yield statement inside a function.

def countdown(n):
    while n > 0:
        yield n
        n -= 1

for num in countdown(3):
    print(num)
# Output: 3, 2, 1

Each time the generator yields, Python suspends its state until the next call.

⚡ 4. Lazy Evaluation in Action

Generators process data on demand — no values are computed until requested.

def infinite_sequence():
    num = 0
    while True:
        yield num
        num += 1

gen = infinite_sequence()
print(next(gen))  # 0
print(next(gen))  # 1
print(next(gen))  # 2

⚠️ Infinite generators must be consumed carefully to avoid endless loops.

🔄 5. Generator Expressions — Memory-Efficient Alternatives

Generator expressions are compact generators defined inline using parentheses.

squares = (x ** 2 for x in range(10))
print(sum(squares))  # 285

Comparison:

🧱 6. yield from — Delegating to Subgenerators

You can delegate iteration to another generator using yield from, simplifying nested loops.

def generator_a():
    yield from range(3)
    yield from ["a", "b", "c"]

for value in generator_a():
    print(value)

Output:

0
1
2
a
b
c

🧮 7. Advanced Generator Operations

Sending Values into a Generator

You can use .send() to send data back into a running generator.

def echo():
    while True:
        value = yield
        print("Received:", value)

g = echo()
next(g)         # Prime the generator
g.send("Hello") # Received: Hello
g.send("World") # Received: World

Closing a Generator

def endless():
    while True:
        yield "Running..."

g = endless()
print(next(g))
g.close()  # Gracefully stops the generator

📚 8. Itertools — Power Tools for Iteration

The itertools module enhances generators with combinatorial and infinite tools.

import itertools

count = itertools.count(10, 2)        # 10, 12, 14, ...
cycled = itertools.cycle(["A", "B"])  # A, B, A, B, ...
limited = itertools.islice(count, 5)  # Take first 5 values

print(list(limited))  # [10, 12, 14, 16, 18]

Other useful itertools functions:

• chain(): link multiple iterables
• combinations() and permutations()
• accumulate(): running totals

🧾 9. Real-World Examples

Example 1 — Fibonacci Sequence

def fibonacci():
    a, b = 0, 1
    while True:
        yield a
        a, b = b, a + b

fib = fibonacci()
print([next(fib) for _ in range(10)])

Example 2 — Lazy File Reader

def read_large_file(path):
    with open(path) as file:
        for line in file:
            yield line.strip()

for line in read_large_file("data.txt"):
    print(line)

Example 3 — Streaming Data Pipeline

def get_data():
    for i in range(1, 1000000):
        yield i

def process(data):
    for n in data:
        yield n * 2

def filter_even(data):
    for n in data:
        if n % 2 == 0:
            yield n

# Combine pipeline
stream = get_data()
stream = process(stream)
stream = filter_even(stream)

for val in stream:
    if val > 20:
        break
    print(val)

🧠 10. Memory and Performance

Generators let you process infinite or massive datasets without exhausting RAM.

🔄 11. Asynchronous Generators (Advanced)

Async generators allow asynchronous iteration in concurrent code:

import asyncio

async def async_counter():
    for i in range(3):
        yield i
        await asyncio.sleep(1)

async def main():
    async for value in async_counter():
        print(value)

asyncio.run(main())

🧭 12. Visualization — Lazy Data Flow

Iterable → Iterator → Generator → Consumer
  |           |           |          |
  v           v           v          v
[1,2,3]  →  __next__() → yield n → print()

🧾 13. Best Practices

✅ Use generators for large or infinite sequences.
✅ Prefer generator expressions for quick, one-liners.
✅ Combine with itertools for advanced iteration.
✅ Avoid converting large generators to lists.
✅ Use yield from to simplify nested loops.
✅ Always handle StopIteration when manually consuming.
✅ For concurrency, use async generators.

🧠 Summary

Mastering generators and iterators gives you full control over how data flows through your programs — efficiently, lazily, and elegantly.