Generators and Iterators: Lazy Evaluation

You already know this line of code: for i in range(1000000):. Before you read another word, answer honestly — when Python hits that line, does it build a list of one million numbers in memory and then start looping? It feels like it should. range(1000000) certainly behaves as if a million numbers are sitting there, ready to be counted through. But they are not. Not one of those numbers exists until the loop asks for it. This is the entire idea behind this chapter, and once you see it clearly, a whole category of "how does that even fit in memory?" questions about real software — from IRCTC processing lakhs of ticket bookings to a cricket app updating you ball by ball — stops being mysterious.

Try this yourself in Python:

import sys

r = range(1_000_000)
print(sys.getsizeof(r))

full_list = list(range(1_000_000))
print(sys.getsizeof(full_list))

Run it, and something strange happens. The size printed for r is a tiny, fixed number of bytes — and if you change 1_000_000 to 1_000_000_000, that number does not change at all. The range object never stores a million numbers; it only remembers three things: where to start, where to stop, and the step size. It computes each number the instant you ask for it, and forgets it the instant you move on. The full_list, on the other hand, really does contain a million separate integer objects sitting in memory, and its size grows every time you make the range bigger. range() is the first "lazy" object you ever met in Python — you just didn't have a name for it yet. This chapter gives you that name, and teaches you how to build your own lazy objects.

How a for Loop Actually Works

To understand laziness, you first need to see what a for loop is secretly doing, because Python hides two steps behind the scenes. When you write for x in something:, Python does not magically know how to walk through something. Instead, it performs a fixed two-step protocol every single time:

1. It calls iter(something) to get back a special helper object called an iterator.
2. It repeatedly calls next() on that iterator to pull out one value at a time, until the iterator signals "no more values" by raising a special error called StopIteration — at which point the loop quietly stops.

You can do this by hand, with no for loop at all:

nums = [10, 20, 30]
it = iter(nums)

print(next(it))
print(next(it))
print(next(it))
print(next(it))

Trace it exactly as Python would. iter(nums) creates an iterator object it that starts positioned before the first element. The first next(it) hands back 10 and moves the internal position forward. The second hands back 20. The third hands back 30. There is nothing left after that — the fourth call finds no more elements and raises StopIteration, which would crash your program if you ran it as plain code outside a loop (a real for loop catches this error for you automatically and simply exits). So the output is 10, then 20, then 30, and then a traceback ending in StopIteration. This is the exact machinery — no more, no less — that every for loop you have ever written has been running underneath.

Iterable vs Iterator — Two Different Words That Are Not the Same Thing

Students frequently use "iterable" and "iterator" as if they were interchangeable. They are related, but distinct, and the difference matters:

• An iterable is anything you can call iter() on to get an iterator. Lists, strings, tuples, dictionaries, and range objects are all iterables. Crucially, a list itself is not an iterator — try calling next() directly on a list and Python refuses, because a list has no memory of "where you currently are" in a loop. A list only knows how to hand out a fresh iterator whenever you ask for one, which is exactly why you can loop over the same list twice and get all the values again both times.
• An iterator is the object that actually does remember position, produced by calling iter() on an iterable. It supports next(), and it is inherently single-use: once it has given you every value, it stays "empty" forever — calling iter() on an iterator just returns itself, not a fresh restart.

Here is proof of that difference. A list can be looped over repeatedly:

nums = [1, 2, 3]
print(sum(nums))
print(sum(nums))

Both lines print 6, because each call to sum() silently asks the list for a brand-new iterator. Keep that distinction in your head — iterable means "can produce an iterator on demand," iterator means "is currently walking through values and remembers where it stopped." Every iterator is also an iterable (it can hand back itself when asked), but not every iterable is an iterator.

Building Your Own Iterator

Suppose your class teacher wants a tool that calls out roll numbers one at a time during attendance, for a class of any size, without ever holding "all the roll numbers" as a pre-built list. You can build exactly that by writing a class with two special methods: __iter__, which must return an iterator (here, the object itself), and __next__, which produces the next value or raises StopIteration when done.

class AttendanceRollCall:
    def __init__(self, total_students):
        self.total_students = total_students
        self.current = 0

    def __iter__(self):
        return self

    def __next__(self):
        if self.current >= self.total_students:
            raise StopIteration
        self.current += 1
        return self.current


roll_call = AttendanceRollCall(3)
for roll_number in roll_call:
    print("Present, roll number", roll_number)

Trace it precisely. The for loop calls iter(roll_call), which runs __iter__ and gets back roll_call itself, since it defines __next__ too. Then the loop repeatedly calls __next__. First call: current is 0, which is not >= 3, so it becomes 1 and is returned — prints "Present, roll number 1". Second call: current is 1, becomes 2, returned — prints "Present, roll number 2". Third call: current is 2, becomes 3, returned — prints "Present, roll number 3". Fourth call: current is now 3, which is >= 3, so StopIteration is raised, and the for loop ends silently. Total output is three lines, roll numbers 1 through 3, produced one at a time, on demand, with the object never holding more than a single integer (self.current) in memory at once. That is the iterator protocol, written by hand.

Generators: yield Writes the Iterator For You

Writing that __iter__/__next__ class was not difficult, but it was verbose for something conceptually this simple. Python gives you a shortcut: a generator function. It looks exactly like a normal function, except that somewhere in its body it uses the keyword yield instead of (or in addition to) return. The moment Python sees yield anywhere inside a function, that function stops being an ordinary function and becomes a generator function — calling it never runs its body immediately. Instead, calling it produces a special object, called a generator, which already satisfies the entire iterator protocol automatically: it already has working __iter__ and __next__ methods, built for you by Python, with no class required.

Here is the exact same attendance roll call, rewritten as a generator:

def attendance_roll_call(total_students):
    current = 0
    while current < total_students:
        current += 1
        yield current


for roll_number in attendance_roll_call(3):
    print("Present, roll number", roll_number)

This produces identical output to the class version — three lines, roll numbers 1, 2, 3 — but in five lines of code instead of eleven, with no manual StopIteration to raise (Python raises it for you the instant the function body finishes running, i.e. the moment the while loop condition goes false and there is no more code to execute).

Proving That Generators Are Lazy

Here is a common misconception worth naming directly. Misconception: calling a generator function runs its code immediately, the same way calling a normal function does. This is false, and you can prove it is false with a small experiment using print() statements planted inside the generator body:

def trace_demo():
    print("Generator body: about to yield 1")
    yield 1
    print("Generator body: about to yield 2")
    yield 2
    print("Generator body: finished")


g = trace_demo()
print("Step 1: generator object created")
print("Step 2: value =", next(g))
print("Step 3: value =", next(g))

If calling trace_demo() ran the function body immediately, you would expect to see "Generator body: about to yield 1" printed before "Step 1" ever appears. That is not what happens. The actual printed order is:

Step 1: generator object created
Generator body: about to yield 1
Step 2: value = 1
Generator body: about to yield 2
Step 3: value = 2

Walk through why. The line g = trace_demo() creates the generator object but executes zero lines of its body — that is why "Step 1" prints first, with nothing from inside the generator yet. Only when Python evaluates next(g) as part of building the "Step 2" print statement does the generator's body actually start running: it prints "about to yield 1," then hits yield 1, which pauses execution right there and hands the value 1 back to next(). That value flows into the outer print(), giving "Step 2: value = 1". The next call to next(g) does not restart the function from the top — it resumes exactly where it paused, immediately after yield 1, runs the "about to yield 2" print, hits yield 2, and pauses again. This pause-and-resume behaviour, keeping the function's local variables and exact execution point alive between calls, is what "lazy evaluation" means in concrete terms: work happens only at the moment a value is actually requested, never before, and never all at once.

A Fully Worked Example: Fibonacci Numbers

Lazy evaluation becomes genuinely useful once the sequence you are generating is expensive to compute in full, or you don't know in advance how many terms you'll need. Fibonacci numbers are a clean example — each number is the sum of the previous two, starting 0, 1, 1, 2, 3, 5, 8, 13, ...:

def fibonacci_upto(limit):
    a, b = 0, 1
    while a <= limit:
        yield a
        a, b = b, a + b


for num in fibonacci_upto(10):
    print(num, end=" ")

Trace it carefully, one iteration at a time, tracking a and b:

• Start: a=0, b=1. Check 0 <= 10 — true. Yield 0. Then a, b = b, a+b → a=1, b=1.
• Check 1 <= 10 — true. Yield 1. Then a, b = 1, 1+1 → a=1, b=2.
• Check 1 <= 10 — true. Yield 1. Then a, b = 2, 1+2 → a=2, b=3.
• Check 2 <= 10 — true. Yield 2. Then a, b = 3, 2+3 → a=3, b=5.
• Check 3 <= 10 — true. Yield 3. Then a, b = 5, 3+5 → a=5, b=8.
• Check 5 <= 10 — true. Yield 5. Then a, b = 8, 5+8 → a=8, b=13.
• Check 8 <= 10 — true. Yield 8. Then a, b = 13, 8+13 → a=13, b=21.
• Check 13 <= 10 — false. Loop ends, generator raises StopIteration.

So the printed output is exactly 0 1 1 2 3 5 8 — the correct Fibonacci sequence up to (and including) 10. Notice something important: at no point does this generator hold the whole sequence in a list. It only ever remembers two numbers, a and b. If you asked for Fibonacci numbers up to ten billion instead of ten, the memory used by this function would not change at all — only the number of times the loop runs would grow.

Generator Expressions — and the One-Shot Trap

You already know list comprehensions: [x * x for x in range(5)] builds [0, 1, 4, 9, 16] immediately, as a real list, in memory, all at once. Change the square brackets to round brackets, and you get a generator expression instead — the lazy equivalent:

squares = (x * x for x in range(5))
print(list(squares))
print(list(squares))

Read that as: build a generator, force it to produce every value by wrapping it in list(), print the result, then do the exact same thing again. You might expect both lines to print [0, 1, 4, 9, 16]. They do not. The output is:

[0, 1, 4, 9, 16]
[]

This is the second misconception worth naming directly. Misconception: a generator (or generator expression) can be looped over again and again, the way a list can. It cannot. Once a generator has yielded its final value and reached StopIteration, it stays exhausted permanently — it does not reset. The first list(squares) pulls every value out and leaves the generator empty; the second list(squares) finds nothing left to give and returns an empty list. Compare this to the eager version:

squares_list = [x * x for x in range(5)]
print(squares_list)
print(squares_list)

Both lines here print [0, 1, 4, 9, 16], because squares_list is an ordinary list — a stored, reusable collection, not a single-pass stream. This trade-off is the whole story of generators in one sentence: you give up the ability to re-use or randomly access values, in exchange for using a constant, tiny amount of memory no matter how large the sequence is.

Sequences With No End

Lazy evaluation is not just a memory optimization — it makes entirely new kinds of programs possible, specifically sequences that are infinite or genuinely unknown in length. A live cricket scorecard is a good mental model: nobody can hand you "the complete list of overs" for a match in progress, because future overs have not been bowled yet. What you can do is ask, "what happened in the next over?" — one request at a time, as the match unfolds. A generator can model exactly this kind of unbounded, on-demand sequence:

def natural_numbers():
    n = 1
    while True:
        yield n
        n += 1


counter = natural_numbers()
for _ in range(5):
    print(next(counter))

Notice the while True — an infinite loop. If this were an ordinary function trying to return a list, calling it would hang your program forever, since it could never finish building the full result. But natural_numbers() returns instantly, because calling a generator function never runs its body — it only creates the paused generator object, exactly as you proved earlier with the trace_demo() experiment. The five next(counter) calls each advance the infinite loop by exactly one step and then pause again, so the output is simply 1, 2, 3, 4, 5, one per line — five values pulled out of a sequence that, left running, would never stop on its own. This is precisely how real streaming systems behave: a UPI app's transaction feed, a live train-running-status update, or a sensor on an ISRO satellite beaming down telemetry all produce values that don't exist as a finished collection anywhere — they exist only as "the next one," produced the moment it's asked for.

Why the Distinction Actually Matters

It is tempting to treat this as a syntax curiosity, but the memory argument is real and it is the reason production systems use generators deliberately. Picture a school of 5,000 students, each with roughly 220 recorded attendance entries across a school year — about 1.1 million individual records. If a program's job is simply to compute one running statistic (say, overall attendance percentage) by looking at each record exactly once, building a Python list of all 1.1 million records first, and only then looping over it, wastes memory for no benefit: every record is visited once and then never touched again. A generator that reads and yields one record at a time — from a file, a database cursor, or a network response — does the identical computation while never holding more than one record's worth of data in memory at any instant. The eager version's memory use grows with the size of the dataset; the lazy version's memory use stays flat, regardless of whether the dataset has a thousand records or a hundred million.

Two Misconceptions, Side by Side

Both misconceptions from this chapter are worth restating together, because they are the two mistakes examiners and interviewers most often probe for:

• Wrong: "Defining a generator function and calling it runs the code inside, just like a normal function call." Correct: calling a generator function only builds a paused generator object; not a single line of the body executes until the first next() call, as the trace_demo() print-ordering experiment proved directly.
• Wrong: "A generator (or generator expression) can be reused the same way a list can — just loop over it again." Correct: a generator is single-pass. Once it reaches its final yield and raises StopIteration, it is permanently exhausted; getting the values again requires creating a brand-new generator by calling the function (or writing the expression) a second time.

Check Your Understanding

1. Predict the exact printed output, line by line, of this code before running it:

   def mystery():
       yield "A"
       yield "B"
       yield "C"
   
   g = mystery()
   print(next(g))
   print(next(g))
   for val in g:
       print(val)

   (Think carefully about what the for loop sees — g is a generator that has already given up two of its three values before the loop begins.)
2. Is range(5) itself an iterator, or only an iterable? Justify your answer using the fact that you can write for i in range(5): ... twice in a row and both loops complete correctly.
3. Rewrite fibonacci_upto(limit) so that it yields only the even Fibonacci numbers up to limit, without changing the underlying recurrence a, b = b, a + b. Trace your version by hand for limit = 10 and state the exact sequence it should print.
4. A classmate writes total = sum([x**2 for x in range(10_000_000)]) to compute a sum of squares. Explain, in terms of what this chapter covered, exactly one change that would make this computation use dramatically less memory, and why the final value of total would be unaffected by that change.

Summary

A for loop is not magic — it is a fixed two-step protocol, calling iter() once and then next() repeatedly until StopIteration appears. An iterable is anything that can hand out a fresh iterator on request (lists, strings, range); an iterator is the object doing the actual remembering of position, and it is inherently single-use. You can build an iterator by hand with a class defining __iter__ and __next__, or far more concisely with a generator function — any function containing yield — which Python automatically turns into a paused, resumable object. Calling a generator function never executes its body immediately; execution starts, and then pauses at each yield, only as next() is called, which is the precise mechanical meaning of "lazy evaluation." This laziness is what lets a generator represent sequences that would be wasteful or outright impossible to store as a full list — a stream of ball-by-ball cricket updates, a live transaction feed, or simply a dataset too large to fit comfortably in memory — using a constant, small amount of memory regardless of how many values the sequence ultimately produces.

Practice Exercises

Now it is time to practice! Complete these challenges to solidify your understanding:

• Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
• Exercise 2: Find a real-world example where generators and iterators: lazy evaluation is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
• Exercise 3: Create a mind-map connecting generators and iterators: lazy evaluation to at least 3 other topics you have studied.