The Iterator Protocol

"An object representing a stream of data. Repeated calls to the iterator's __next__() method (or passing it to the built-in function next()) return successive items in the stream. When no more data are available a StopIteration exception is raised instead." — The Python Glossary, "iterator"

The for loop is the first thing anyone learns in Python and one of the last things they understand. It appears to be a language construct that knows how to walk lists, strings, dicts, files, and ranges — five unrelated kinds of thing, each walked differently. In fact it knows how to walk none of them. It knows one protocol, and every one of those types implements it.

That protocol is two methods. __iter__ returns an iterator, and __next__ produces the next item or signals that there are none left. Everything else — for, comprehensions, sum, sorted, zip, tuple unpacking, in, the * operator at a call site — is built on those two methods and nothing more. Once you can see the protocol, a large amount of Python that looked like special-case syntax turns out to be one mechanism viewed from different angles.

This chapter establishes the protocol precisely, draws the distinction that causes the most confusion in practice — iterable versus iterator — and shows what it costs when you get that distinction wrong. It is the foundation for everything else in Part IV: generators are a way to write iterators, itertools is a library of them, and laziness is a property that only makes sense once you know when the next item is actually computed.

The protocol: two methods

An iterable is an object with an __iter__ method that returns an iterator. An iterator is an object with a __next__ method that returns the next item, and an __iter__ method that returns itself. That self-returning __iter__ is not a formality — it is what allows an iterator to be used anywhere an iterable is accepted, and it is the source of the whole iterable/iterator confusion.

iter(it) is it returning True is the defining property of an iterator. A list, by contrast, hands out a new iterator every time you ask.

There is the answer to the big question in one line: a list is an iterable but not an iterator; a generator is both. A list can be walked repeatedly because each walk gets a fresh iterator over unchanged data. A generator is the iterator — there is nothing behind it to make a second one from — so when it is exhausted, it stays exhausted.

That empty second pass is one of the most common bugs in Python code that handles data lazily, and its worst property is that it raises nothing. A function that receives an exhausted iterator sees an empty sequence and behaves as though there were no data — a wrong answer, silently.

What for actually does

The for statement is sugar over the protocol. Every for loop is equivalent to calling iter() once, then next() repeatedly inside a try, and breaking when StopIteration arrives.

StopIteration being an exception rather than a sentinel value is a deliberate design choice with real consequences. It means an iterator can yield any value at all, including None and False, without ambiguity — there is no in-band "end of data" marker that could collide with real data. The cost is that raising and catching an exception ends every loop, which is why next(it, default) exists for the cases where you would rather not write a try.

Because the protocol is all for needs, anything implementing it works — including your own classes.

Those two classes produce identical output on a first pass and differ completely on a second, and the difference is entirely in what __iter__ returns. Countdown.__iter__ is a generator function, so calling it builds a fresh generator each time — the class is a reusable iterable. Broken.__iter__ returns self, making the object its own iterator with one shared position.

The rule for your own classes is simple: if the object represents a collection, __iter__ should return a new iterator; if it represents a stream or cursor, returning self is correct. Most classes people write are collections, and returning self from __iter__ is almost always a bug in that case.

Iterators everywhere in the standard library

Once you know the protocol, a large part of the built-ins reveal themselves as iterators — and therefore as one-shot. This is a fact about Python 3 specifically: in Python 2 several of these returned lists.

Two details there are worth keeping. zip silently stops at the shortest input, which has quietly dropped data in more programs than anyone can count — strict=True, added in Python 3.10, turns that into a ValueError and should be your default when the inputs are supposed to be the same length. And reversed requires a sequence with a known length, because reversing a stream is not something a stream can do; the error message 'generator' object is not reversible is telling you that no amount of laziness makes backwards traversal possible.

There is also a lesser-known second form of iter() that turns any zero-argument callable into an iterator, stopping when it returns a sentinel value.

This two-argument iter is the cleanest way to wrap any "call this until it says stop" API — socket reads, queue gets, database cursors — in something a for loop can consume directly.

Consuming without looping

Many built-ins take an iterable and consume it, and knowing which ones consume fully and which stop early is the difference between a fast program and a hung one on an infinite stream.

any and all short-circuit — they stop at the first decisive item. sum, sorted, min, max, list, and tuple do not; they must see everything. On a finite iterable that is only a performance question. On an infinite one it is the difference between an answer and a program that never returns.

Membership testing with in also short-circuits, and on an iterator it consumes what it passed.

The 3 in gen case is worth staring at. The membership test returned True and, as a side effect, permanently removed four items from the stream. On a list this would be harmless; on an iterator it means a test changed the data. Any code that checks membership on something it later iterates is at risk of this, and it is invisible at the call site.

Common Mistakes

1. Treating "iterable" and "iterator" as synonyms. An iterable can produce many iterators; an iterator is a single position in a stream. Nearly every laziness bug traces back to this.

2. Iterating the same iterator twice. The second pass yields nothing and raises nothing, so the failure is a silently wrong answer rather than an error.

3. Returning self from __iter__ on a collection class. It makes the object its own iterator, so it can be walked exactly once for its entire lifetime. Return a fresh iterator — a generator function body is the easiest way.

4. Assuming zip complains about unequal lengths. It truncates to the shortest input without a word. Pass strict=True when the inputs are meant to match.

5. Forgetting that map, filter, zip, enumerate, and reversed return iterators in Python 3. Code ported from Python 2 that indexes or re-walks their results breaks in ways that look like data problems.

6. Calling len() on an iterator. There is no length — the items may not exist yet. TypeError: object of type 'generator' has no len() means "this is a stream, not a collection."

7. Testing membership on a stream you plan to iterate. x in gen consumes everything up to and including the match, and the items are gone.

8. Using a fully-consuming built-in on an infinite iterator. sum(itertools.count()) does not error; it hangs. Bound it with islice first.

9. Expecting reversed() to work on a generator. Reversal needs a known length and random access. Materialise with list() first, or restructure so you never need to go backwards.

10. Writing a function that iterates its argument twice without saying so. Callers passing a generator get a wrong answer. Either document the requirement or start with items = list(items).

Interview Corner

This topic is a favourite because the surface question is easy and the follow-up separates people fast. Anyone can say what for does; far fewer can state the iterable/iterator distinction precisely, and fewer still can explain why StopIteration is an exception rather than a returned sentinel. If you can name a real bug the distinction causes — a function that counts then loops, over a generator — you are describing production experience rather than trivia.

Summary

• The protocol is two methods. __iter__ returns an iterator; __next__ returns the next item or raises StopIteration. Everything iteration-related in Python is built on these.
• Iterable and iterator are different things. An iterable produces iterators; an iterator is one position in one stream. An iterator's __iter__ returns itself, which is what blurs the line.
• A list is iterable; a generator is an iterator. That is why a list can be looped over repeatedly and a generator cannot, and why iter(gen) is gen is True.
• Exhaustion is silent. A second pass over a spent iterator yields nothing and raises nothing, so the symptom is a wrong answer rather than an error.
• for is sugar. It calls iter() once, then next() in a loop, and breaks on StopIteration; next(it, default) does the same without the try.
• StopIteration is an exception so that any value can be yielded. An in-band sentinel would collide with real data such as None or 0.
• Your __iter__ should usually return a fresh iterator. Writing it as a generator function gives a reusable iterable; returning self makes a one-shot cursor.
• map, filter, zip, enumerate, and reversed are iterators in Python 3. They are lazy, one-shot, and have no length.
• zip truncates silently. Use strict=True when the inputs are supposed to be the same length.
• Know which built-ins short-circuit. any, all, and in stop early; sum, sorted, min, max, and list consume everything and will hang on an infinite stream.

The next chapter takes the most convenient way to build an iterator — the generator function — and looks at what yield actually does to a function's execution, why calling a generator function runs none of its body, and how a function can be suspended in the middle and resumed later.