Skip to content

Generators

Create lazy, memory-efficient sequences with generator functions, yield, and generator expressions.

What you'll learn

  • Write generator functions with yield
  • Understand lazy evaluation and memory efficiency
  • Use generator expressions for inline generators
  • Chain generators in pipelines
  • Handle infinite generators with itertools

Concept

What Is a Generator?

A generator is a function that produces a sequence of values lazily — one at a time, on demand — instead of computing them all upfront and storing them in memory. You create one with the yield keyword:

def count_up_to(n):
    i = 0
    while i < n:
        yield i
        i += 1

# Create the generator — nothing runs yet
counter = count_up_to(5)

# Values are produced one at a time
print(next(counter))  # 0
print(next(counter))  # 1
print(next(counter))  # 2

When you call a generator function, it returns a generator object without running any code. Each call to next() runs the function until it hits yield, produces that value, and pauses. The next next() resumes from where it left off. When the function returns (or falls off the end), StopIteration is raised.

yield vs return

  • return — returns a value and ends the function
  • yield — produces a value and pauses the function, to be resumed later
def squares(n):
    for x in range(n):
        yield x ** 2

for sq in squares(5):
    print(sq)  # 0, 1, 4, 9, 16

A generator function can have multiple yield statements. Each yield produces a value and suspends execution. The function resumes from the yield on the next next() call.

Why Generators? Memory Efficiency

The key advantage is memory. Compare:

# List — all 1 million values in memory at once
squares_list = [x ** 2 for x in range(1_000_000)]
# Uses ~8 MB of memory

# Generator — only one value in memory at a time
squares_gen = (x ** 2 for x in range(1_000_000))
# Uses ~100 bytes of memory

For large or infinite sequences, generators are essential. You can process a 10 GB file line by line with a generator using only kilobytes of memory.

Generator Expressions

The parenthesized version of a list comprehension creates a generator:

# List comprehension — eager, all values in memory
squares_list = [x ** 2 for x in range(10)]

# Generator expression — lazy, one value at a time
squares_gen = (x ** 2 for x in range(10))

print(sum(squares_gen))  # 285 — computed without storing all values

Generator expressions are perfect for feeding into sum(), max(), min(), or any function that consumes an iterable. They avoid building an intermediate list.

Sending Values with .send()

Generators can receive values from the caller using .send():

def accumulator():
    total = 0
    while True:
        value = yield total
        if value is None:
            break
        total += value

acc = accumulator()
next(acc)           # prime the generator — runs to first yield
print(acc.send(10)) # 10
print(acc.send(20)) # 30
print(acc.send(5))  # 35

This makes generators coroutines — they can both produce and consume values. This is an advanced feature used in async programming.

Chaining Generators

Generators compose beautifully — you can pipe them together:

def numbers():
    for n in range(1, 20):
        yield n

def squared(seq):
    for n in seq:
        yield n ** 2

def evens(seq):
    for n in seq:
        if n % 2 == 0:
            yield n

# Pipeline: numbers → squared → evens
pipeline = evens(squared(numbers()))
print(list(pipeline))  # [4, 16, 36, 64, 100, 144, 196, 256, 324]

Each stage processes values lazily — no intermediate lists are created. This is the foundation of streaming data pipelines.

Infinite Generators

Generators can be infinite — they never run out of values:

def fibonacci():
    a, b = 0, 1
    while True:
        yield a
        a, b = b, a + b

fib = fibonacci()
for _ in range(10):
    print(next(fib))  # 0, 1, 1, 2, 3, 5, 8, 13, 21, 34

An infinite generator is safe as long as you only consume a finite number of values (with next(), itertools.islice(), or a bounded loop). itertools provides many useful generators:

from itertools import islice, count, cycle

# count() — infinite counter: 0, 1, 2, 3, ...
for n in islice(count(), 5):
    print(n)  # 0, 1, 2, 3, 4

# cycle() — repeat a sequence forever
for item in islice(cycle(["A", "B", "C"]), 7):
    print(item)  # A, B, C, A, B, C, A

yield from — Delegating to Sub-generators

yield from delegates to another generator, flattening nested generators:

def inner():
    yield 1
    yield 2

def outer():
    yield 0
    yield from inner()  # yields all values from inner()
    yield 3

print(list(outer()))  # [0, 1, 2, 3]

This is cleaner than manually iterating and yielding each value from the sub-generator.

Example

              # Basic generator with yield
def count_up_to(n):
    i = 0
    while i < n:
        yield i
        i += 1

print(list(count_up_to(5)))  # [0, 1, 2, 3, 4]

# Generator vs list — memory efficiency
import sys

big_list = [x ** 2 for x in range(10000)]
big_gen = (x ** 2 for x in range(10000))
print(f"List size: {sys.getsizeof(big_list)} bytes")
print(f"Gen size:  {sys.getsizeof(big_gen)} bytes")

# Using next() manually
counter = count_up_to(3)
print(next(counter))  # 0
print(next(counter))  # 1
print(next(counter))  # 2

# Infinite generator — Fibonacci
def fibonacci():
    a, b = 0, 1
    while True:
        yield a
        a, b = b, a + b

fib = fibonacci()
fib_10 = [next(fib) for _ in range(10)]
print(f"Fibonacci: {fib_10}")

# Chaining generators in a pipeline
def numbers():
    for n in range(1, 10):
        yield n

def squared(seq):
    for n in seq:
        yield n ** 2

def evens(seq):
    for n in seq:
        if n % 2 == 0:
            yield n

pipeline = evens(squared(numbers()))
print(f"Pipeline: {list(pipeline)}")

# yield from — delegating
def inner():
    yield 1
    yield 2

def outer():
    yield 0
    yield from inner()
    yield 3

print(f"yield from: {list(outer())}")  # [0, 1, 2, 3]

# Reading a large file lazily
def read_lines(filename):
    with open(filename) as f:
        for line in f:
            yield line.strip()  # one line at a time, low memory
            

Shows a basic generator, memory comparison vs list, manual next() usage, an infinite Fibonacci generator, a chained pipeline (numbers → squared → evens), yield from delegation, and a lazy file reader pattern.

Try it

  • Text Counter

    Generators are perfect for streaming text processing line by line.

  • JSON Formatter

    Generators can stream large JSON arrays without loading everything into memory.

Common mistakes

The mistake

Trying to reuse a generator that has been exhausted

The fix

Generators are single-use — once consumed, they are empty and next() raises StopIteration. To reuse, recreate the generator: gen = my_generator() again, or convert to a list first if you need to iterate multiple times.

The mistake

Creating an infinite generator and iterating with a for loop without a break

The fix

An infinite generator (while True: yield) never stops. Always use a bounded consumer: next() a fixed number of times, itertools.islice(gen, n), or a for loop with a break condition. A bare 'for x in infinite_gen:' hangs forever.

Related Cheatsheets

Practice Generators

4 exercises

Practice Generators

Look up unfamiliar terms

A beginner-friendly glossary explains jargon in plain English — variables, functions, DOM, Promise, and more.

View glossary

Build real projects

Apply what you learned by building guided projects with starter code and solutions.

View projects

Decode error messages

Plain-English explanations of common errors — what they mean, why they happen, and how to fix them.

View errors