Chapter 50 of 54

Iterators & Generators in Python

You have been using for loops on lists, strings, dictionaries and range() since the loops chapters. But have you wondered how a for loop knows how to walk through so many different kinds of objects?

The answer is the iterator protocol. In this chapter you'll see what happens inside a for loop, build your own iterator, and then learn generators - a much easier way to create iterators that can even produce values lazily, one at a time, using very little memory.

Iterable vs Iterator

These two words sound similar but mean different things.

An iterable is any object you can loop over (list, tuple, string, dict, set, range, file). An iterator is the object that actually produces the values one by one and remembers where it is.

  • iter(iterable) gives you an iterator.

  • next(iterator) gives you the next value.

  • When there are no more values, next() raises StopIteration.

nums = [10, 20, 30]

it = iter(nums)
print(next(it))
print(next(it))
print(next(it))
print(next(it))

Output:

10
20
30
Traceback (most recent call last):
  ...
StopIteration

The fourth next() has nothing left to return, so it raises StopIteration (shown here as an error because we did not catch it).

A list is like a book, and an iterator is like a bookmark. The book doesn't change; the bookmark moves forward page by page and cannot go back.


What a for Loop Really Does

This loop:

for x in nums:
    print(x)

works roughly like this behind the scenes:

nums = [10, 20, 30]

it = iter(nums)
while True:
    try:
        x = next(it)
    except StopIteration:
        break
    print(x)

Output:

10
20
30

So any object that supports iter() and next() can be used in a for loop, as well as with list(), sum(), max(), in, unpacking and more.

Iterators Get Used Up

An iterator can be consumed only once. After it is exhausted, it stays empty.

it = iter([1, 2, 3])
print(list(it))
print(list(it))   # already exhausted

Output:

[1, 2, 3]
[]

The list itself is not used up - calling iter(nums) again gives a fresh iterator.


Building Your Own Iterator

To make a class an iterator, give it two special methods (you met dunder methods in the previous chapters):

  • __iter__() - returns the iterator object (usually self).

  • __next__() - returns the next value, or raises StopIteration when done.

class Countdown:
    def __init__(self, start):
        self.current = start

    def __iter__(self):
        return self

    def __next__(self):
        if self.current <= 0:
            raise StopIteration
        value = self.current
        self.current -= 1
        return value

for n in Countdown(3):
    print(n)

Output:

3
2
1

It works, but that's a lot of code just to count down. Generators make this much simpler.


Generators: Iterators the Easy Way

A generator function is a normal function that uses yield instead of return. Calling it does not run the body - it returns a generator object, which is an iterator.

def countdown(start):
    while start > 0:
        yield start
        start -= 1

for n in countdown(3):
    print(n)

Output:

3
2
1

Same result as the class, in four lines. Python creates __iter__ and __next__ for you automatically.

How yield Works Step by Step

yield gives a value back and pauses the function. Its local variables are remembered. The next call to next() resumes right after the yield.

def steps():
    print("Start")
    yield 1
    print("After first yield")
    yield 2
    print("End")

g = steps()
print("Generator created")
print(next(g))
print(next(g))
print(next(g, "No more values"))

Output:

Generator created
Start
1
After first yield
2
End
No more values

Notice that "Start" is printed only on the first next(), not when steps() is called. The optional second argument of next() is a default returned instead of raising StopIteration.

return

yield

Sends back one value and ends the function

Sends back a value and pauses the function

Local variables are lost

Local variables are kept for the next resume

Function called once, result returned

Can produce many values over time


Why Generators Save Memory

A list stores all its values in memory at once. A generator produces each value only when asked (this is called lazy evaluation), so it needs O(1) extra memory no matter how many values it produces.

import sys

squares_list = [x * x for x in range(1_000_000)]
squares_gen = (x * x for x in range(1_000_000))

print(sum(squares_list) == sum(squares_gen))
print(sys.getsizeof(squares_list) > 1_000_000)
print(sys.getsizeof(squares_gen) < 1_000)

Output:

True
True
True

The list takes several megabytes; the generator object takes only a couple of hundred bytes (exact sizes vary by Python version, which is why we compare instead of printing them).

The second line uses a generator expression: it looks like a list comprehension but uses round brackets. When passed directly to a function, you can even drop the extra brackets: sum(x * x for x in range(10)).

Infinite Generators

Because values are produced on demand, a generator can even be infinite. Just make sure the consumer stops.

def fibonacci():
    a, b = 0, 1
    while True:
        yield a
        a, b = b, a + b

for i, num in enumerate(fibonacci()):
    if i == 10:
        break
    print(num, end=" ")
print()

Output:

0 1 1 2 3 5 8 13 21 34 

Practical Example: A Data Pipeline

Generators can be chained so that each item flows through every step one at a time. This is how you process huge log files without loading them fully into memory.

lines = [
    "INFO server started",
    "ERROR disk full",
    "INFO user login",
    "ERROR timeout",
]

def errors(rows):
    for row in rows:
        if row.startswith("ERROR"):
            yield row

def messages(rows):
    for row in rows:
        yield row.split(" ", 1)[1]

for msg in messages(errors(lines)):
    print(msg)

Output:

disk full
timeout

With a real file you would pass the file object instead of lines - a file object is itself an iterator over its lines.

yield from

yield from iterable yields every value from another iterable. It's a shortcut for a for loop that yields each item.

def flatten(nested):
    for item in nested:
        if isinstance(item, list):
            yield from flatten(item)
        else:
            yield item

print(list(flatten([1, [2, [3, 4]], 5])))

Output:

[1, 2, 3, 4, 5]

Common Mistakes

  • Reusing an exhausted generator. A generator can be looped over only once. Call the generator function again to get a new one.

  • Expecting len() or indexing. len(gen) and gen[0] raise TypeError. Convert to a list if you need them (and accept the memory cost).

  • Forgetting to call the function. for x in countdown: fails; you need countdown(3).

  • Printing a generator. print(gen) shows something like <generator object countdown at 0x...>, not the values. Use list(gen) to see them.

  • Calling list() on an infinite generator. It never finishes and eventually runs out of memory.


Interview Tips

  • Iterable vs iterator: an iterable has __iter__() (or a sequence-style __getitem__()) and can give an iterator; an iterator has both __iter__() and __next__() and keeps track of position.

  • Generator vs list: a list stores everything (O(n) memory) and can be reused and indexed; a generator is lazy (O(1) extra memory) and single-use.

  • Is every generator an iterator? Yes. Is every iterator a generator? No - e.g. our Countdown class.

  • Practice: write a generator evens(n) that yields even numbers up to n; write a generator that reads chunks of a list of size k; implement my_range(start, stop, step) as a generator.

Summary

  • iter() turns an iterable into an iterator; next() fetches values until StopIteration.

  • A for loop is just iter() + repeated next() + catching StopIteration.

  • Custom iterators need __iter__ and __next__.

  • Generator functions use yield to pause and resume; generator expressions use ( ... ).

  • Generators are lazy, memory efficient and single-use.

Use a list when you need to keep and reuse the data. Use a generator when you just need to walk through it once.