You have been using for loops on lists, strings, dictionaries and range() since the loops chapters. But have you wondered how a for loop knows how to walk through so many different kinds of objects?
The answer is the iterator protocol. In this chapter you'll see what happens inside a for loop, build your own iterator, and then learn generators - a much easier way to create iterators that can even produce values lazily, one at a time, using very little memory.
Iterable vs Iterator
These two words sound similar but mean different things.
An iterable is any object you can loop over (list, tuple, string, dict, set, range, file). An iterator is the object that actually produces the values one by one and remembers where it is.
iter(iterable)gives you an iterator.next(iterator)gives you the next value.When there are no more values,
next()raisesStopIteration.
nums = [10, 20, 30]
it = iter(nums)
print(next(it))
print(next(it))
print(next(it))
print(next(it))Output:
10
20
30
Traceback (most recent call last):
...
StopIterationThe fourth next() has nothing left to return, so it raises StopIteration (shown here as an error because we did not catch it).
A list is like a book, and an iterator is like a bookmark. The book doesn't change; the bookmark moves forward page by page and cannot go back.
What a for Loop Really Does
This loop:
for x in nums:
print(x)works roughly like this behind the scenes:
nums = [10, 20, 30]
it = iter(nums)
while True:
try:
x = next(it)
except StopIteration:
break
print(x)Output:
10
20
30So any object that supports iter() and next() can be used in a for loop, as well as with list(), sum(), max(), in, unpacking and more.
Iterators Get Used Up
An iterator can be consumed only once. After it is exhausted, it stays empty.
it = iter([1, 2, 3])
print(list(it))
print(list(it)) # already exhaustedOutput:
[1, 2, 3]
[]The list itself is not used up - calling iter(nums) again gives a fresh iterator.
Building Your Own Iterator
To make a class an iterator, give it two special methods (you met dunder methods in the previous chapters):
__iter__()- returns the iterator object (usuallyself).__next__()- returns the next value, or raisesStopIterationwhen done.
class Countdown:
def __init__(self, start):
self.current = start
def __iter__(self):
return self
def __next__(self):
if self.current <= 0:
raise StopIteration
value = self.current
self.current -= 1
return value
for n in Countdown(3):
print(n)Output:
3
2
1It works, but that's a lot of code just to count down. Generators make this much simpler.
Generators: Iterators the Easy Way
A generator function is a normal function that uses
yieldinstead ofreturn. Calling it does not run the body - it returns a generator object, which is an iterator.
def countdown(start):
while start > 0:
yield start
start -= 1
for n in countdown(3):
print(n)Output:
3
2
1Same result as the class, in four lines. Python creates __iter__ and __next__ for you automatically.
How yield Works Step by Step
yield gives a value back and pauses the function. Its local variables are remembered. The next call to next() resumes right after the yield.
def steps():
print("Start")
yield 1
print("After first yield")
yield 2
print("End")
g = steps()
print("Generator created")
print(next(g))
print(next(g))
print(next(g, "No more values"))Output:
Generator created
Start
1
After first yield
2
End
No more valuesNotice that "Start" is printed only on the first next(), not when steps() is called. The optional second argument of next() is a default returned instead of raising StopIteration.
|
|
|---|---|
Sends back one value and ends the function | Sends back a value and pauses the function |
Local variables are lost | Local variables are kept for the next resume |
Function called once, result returned | Can produce many values over time |
Why Generators Save Memory
A list stores all its values in memory at once. A generator produces each value only when asked (this is called lazy evaluation), so it needs O(1) extra memory no matter how many values it produces.
import sys
squares_list = [x * x for x in range(1_000_000)]
squares_gen = (x * x for x in range(1_000_000))
print(sum(squares_list) == sum(squares_gen))
print(sys.getsizeof(squares_list) > 1_000_000)
print(sys.getsizeof(squares_gen) < 1_000)Output:
True
True
TrueThe list takes several megabytes; the generator object takes only a couple of hundred bytes (exact sizes vary by Python version, which is why we compare instead of printing them).
The second line uses a generator expression: it looks like a list comprehension but uses round brackets. When passed directly to a function, you can even drop the extra brackets: sum(x * x for x in range(10)).
Infinite Generators
Because values are produced on demand, a generator can even be infinite. Just make sure the consumer stops.
def fibonacci():
a, b = 0, 1
while True:
yield a
a, b = b, a + b
for i, num in enumerate(fibonacci()):
if i == 10:
break
print(num, end=" ")
print()Output:
0 1 1 2 3 5 8 13 21 34 Practical Example: A Data Pipeline
Generators can be chained so that each item flows through every step one at a time. This is how you process huge log files without loading them fully into memory.
lines = [
"INFO server started",
"ERROR disk full",
"INFO user login",
"ERROR timeout",
]
def errors(rows):
for row in rows:
if row.startswith("ERROR"):
yield row
def messages(rows):
for row in rows:
yield row.split(" ", 1)[1]
for msg in messages(errors(lines)):
print(msg)Output:
disk full
timeoutWith a real file you would pass the file object instead of lines - a file object is itself an iterator over its lines.
yield from
yield from iterable yields every value from another iterable. It's a shortcut for a for loop that yields each item.
def flatten(nested):
for item in nested:
if isinstance(item, list):
yield from flatten(item)
else:
yield item
print(list(flatten([1, [2, [3, 4]], 5])))Output:
[1, 2, 3, 4, 5]Common Mistakes
Reusing an exhausted generator. A generator can be looped over only once. Call the generator function again to get a new one.
Expecting
len()or indexing.len(gen)andgen[0]raiseTypeError. Convert to a list if you need them (and accept the memory cost).Forgetting to call the function.
for x in countdown:fails; you needcountdown(3).Printing a generator.
print(gen)shows something like<generator object countdown at 0x...>, not the values. Uselist(gen)to see them.Calling
list()on an infinite generator. It never finishes and eventually runs out of memory.
Interview Tips
Iterable vs iterator: an iterable has
__iter__()(or a sequence-style__getitem__()) and can give an iterator; an iterator has both__iter__()and__next__()and keeps track of position.Generator vs list: a list stores everything (O(n) memory) and can be reused and indexed; a generator is lazy (O(1) extra memory) and single-use.
Is every generator an iterator? Yes. Is every iterator a generator? No - e.g. our
Countdownclass.Practice: write a generator
evens(n)that yields even numbers up ton; write a generator that reads chunks of a list of sizek; implementmy_range(start, stop, step)as a generator.
Summary
iter()turns an iterable into an iterator;next()fetches values untilStopIteration.A
forloop is justiter()+ repeatednext()+ catchingStopIteration.Custom iterators need
__iter__and__next__.Generator functions use
yieldto pause and resume; generator expressions use( ... ).Generators are lazy, memory efficient and single-use.
Use a list when you need to keep and reuse the data. Use a generator when you just need to walk through it once.