Generators and Iterators: Memory-Efficient Processing

Suppose your Computer Science teacher gives you this task: "Find the first five prime numbers greater than 1 crore (1,00,00,000)." A prime number is a number greater than 1 that has no divisors other than 1 and itself — you already check this in Maths using trial division. Your classmate writes this program:

def is_prime(n):
    if n < 2:
        return False
    if n % 2 == 0:
        return n == 2
    i = 3
    while i * i <= n:
        if n % i == 0:
            return False
        i += 2
    return True

def primes_above_crore():
    all_candidates = []
    n = 10_000_000
    while n < 100_000_000:      # check every number up to 10 crore
        n += 1
        if is_prime(n):
            all_candidates.append(n)   # store EVERY prime found
    return all_candidates

first_five = primes_above_crore()[:5]
print(first_five)

This code is logically correct — it will eventually print [10000019, 10000079, 10000103, 10000121, 10000139] (you can verify these are the first five primes after 1,00,00,000). But watch what it actually does: it insists on finding every single prime between 1 crore and 10 crore, storing each one in a growing Python list, before it is allowed to hand you the first five. On a school laptop this could take minutes and consume a large, unnecessary chunk of RAM — to answer a question that only needed five numbers. The list keeps growing even after you have the answer you wanted, because nothing tells the function "stop, I have enough."

This chapter is about fixing exactly this kind of wastefulness. The tool for the job is called a generator, and to understand generators properly, you first need to understand the more general idea they are built on: iterators.

Two ways to hand out data: all at once, or one at a time

Think of two ways a canteen could serve 500 students during lunch break. Method A: the canteen cooks all 500 meals first, stacks them on trays across every table, and then lets students in. Method B: the canteen cooks one meal, serves the student at the counter, then cooks the next one, on demand, only when someone is ready for it.

Method A needs enough table space and storage for 500 meals simultaneously, even if only 3 students have arrived so far. Method B never needs space for more than one meal at a time, no matter how many students eventually show up. Both methods eventually serve everyone the same food — but their memory footprint (how much space is occupied at any given moment) is completely different.

A Python list behaves like Method A: [10000019, 10000079, ...] is computed completely and held in memory before you can use any of it. A Python generator behaves like Method B: it produces one value, pauses, and only computes the next value when you actually ask for it. This is called lazy evaluation — "lazy" because it refuses to do work until the work is actually demanded.

Iterables and iterators: two related but different ideas

Before writing generators, you need two precise vocabulary words, because Python code and error messages use them constantly.

• Iterable: any object you are allowed to loop over with for. Lists, strings, tuples, dictionaries, and range() objects are all iterables. Formally, an object is iterable if it has an __iter__() method that produces an iterator.
• Iterator: the object that actually does the walking — it remembers where you currently are in the sequence and knows how to produce the next value. Formally, an iterator has a __next__() method, which either returns the next value or raises a special exception, StopIteration, when there is nothing left.

The relationship: a list is iterable, but a list is not itself an iterator — it has no memory of "where you are." Every time you start a fresh for loop over the same list, Python asks the list to produce a brand-new iterator object via iter(), and that iterator is what actually tracks position and calls __next__() repeatedly. This is why you can loop over the same list twice and get all the values both times — each loop gets its own fresh iterator.

You never usually see this happening because for does it silently, but you can trigger it by hand:

nums = [10, 20, 30]
it = iter(nums)        # ask the list for an iterator
print(next(it))         # 10
print(next(it))         # 20
print(next(it))         # 30
print(next(it))         # raises StopIteration

Trace it step by step: iter(nums) creates an iterator object it whose internal position pointer starts at index 0. The first next(it) call reads index 0 (value 10), then quietly advances the pointer to index 1. The second call reads index 1 (20), advances to index 2. The third call reads index 2 (30), advances to index 3 — which is past the end. The fourth call finds nothing at index 3 and raises StopIteration instead of returning a value. This exception is not a bug; it is the standard, expected signal that means "the sequence is finished." A for loop catches this exception internally and simply stops looping — you never see it unless, like above, you call next() yourself.

A generator you already know: range()

You have used range(1000000) since Class 6 or 7 without asking how it stores a million numbers so instantly. The answer: it doesn't. range is a lazy object — it only remembers three numbers (start, stop, step) and computes each value algebraically the moment it is asked for one, using the formula start + step × position. That is why range(10_000_000) and range(3) are created in exactly the same, tiny amount of time and memory — neither one actually builds a list.

This is measurable. Running sys.getsizeof() (a function that reports how many bytes an object occupies) on these three objects shows the pattern clearly:

import sys

print(sys.getsizeof(list(range(1000))))     # 8056  (approx, grows with size)
print(sys.getsizeof(range(1000)))            # 48    (constant)
print(sys.getsizeof((x for x in range(1000))))  # 200  (constant)

print(sys.getsizeof(list(range(1_000_000))))  # 8000056 (~7.6 MB, grows linearly)
print(sys.getsizeof(range(1_000_000)))         # 48       (unchanged!)
print(sys.getsizeof((x for x in range(1_000_000))))  # 200 (unchanged!)

(Exact byte counts differ slightly between Python versions and operating systems, so do not memorise these numbers — the pattern is what matters: the list's size scales up with however many items it holds, while range and the generator stay essentially flat no matter how large the range is, because they never materialise the whole sequence at once.)

Writing your own generator with yield

A generator function looks exactly like a normal function, with one difference: instead of return, it uses the keyword yield. This single keyword changes everything about how the function behaves.

def fibonacci_upto(limit):
    a, b = 0, 1
    while a <= limit:
        yield a
        a, b = b, a + b

for value in fibonacci_upto(20):
    print(value, end=' ')
# Output: 0 1 1 2 3 5 8 13

Trace this precisely, because the mechanics of yield are the whole point of this chapter:

1. Calling fibonacci_upto(20) does not run any of the code inside the function. It only creates and returns a generator object — a paused, ready-to-run packet of code that remembers its own local variables (a, b, and the current line number).
2. The for loop calls next() on this generator. Only now does the function body actually start executing: a, b = 0, 1 runs, the while condition 0 <= 20 is checked (true), and execution hits yield a. At this point the function freezes mid-execution, hands the value 0 back to the for loop, and prints 0.
3. The next next() call does not restart the function from the top — it resumes exactly where it froze, right after yield a. It runs a, b = b, a + b (now a=1, b=1), loops back, checks 1 <= 20 (true), and yields 1.
4. This repeats — freeze, resume, update, yield — producing 1, 2, 3, 5, 8, 13. When a becomes 21, the while condition 21 <= 20 is false, the loop ends, the function falls off the end, and Python automatically raises StopIteration to tell the for loop to stop.

Crucially, at no point does the generator hold the full list [0, 1, 1, 2, 3, 5, 8, 13] in memory. It only ever remembers the current a and b, plus the line it froze on. You can watch this pause-and-resume behaviour directly by calling next() yourself instead of using a for loop:

gen = fibonacci_upto(20)
print(next(gen))   # 0  -- function runs to first yield, then freezes
print(next(gen))   # 1  -- resumes, computes, yields again, freezes
print(next(gen))   # 1

Fixing the original problem: primes, the lazy way

Now rewrite the opening prime-search program as a generator. Instead of stuffing results into a list, it yields each prime the instant it is found, and the calling code decides when to stop asking for more:

def is_prime(n):
    if n < 2:
        return False
    if n % 2 == 0:
        return n == 2
    i = 3
    while i * i <= n:
        if n % i == 0:
            return False
        i += 2
    return True

def primes_above(start):
    n = start
    while True:                 # keeps going forever -- that's fine!
        n += 1
        if is_prime(n):
            yield n

gen = primes_above(10_000_000)
first_five = [next(gen) for _ in range(5)]
print(first_five)
# Output: [10000019, 10000079, 10000103, 10000121, 10000139]

Notice while True — an infinite loop, which would be a serious bug in an ordinary function, because it would never return. Inside a generator it is not only safe but a common, powerful pattern: the generator does not run all the way through in one go. It runs only up to the next yield, freezes, and waits. The code outside decides how many times to call next(), so here it calls exactly five times and then simply stops asking — the generator obediently never computes a sixth value. Numbers from 1,00,00,006 upward are never even checked. Compare this to the very first version of this program in the chapter, which was forced to scan every number up to 10 crore before it could return anything at all.

Generator expressions: the compact cousin of list comprehensions

You already know list comprehensions, like [x*x for x in range(5)], which produce [0, 1, 4, 9, 16] as a full list, eagerly. Change the square brackets [ ] to round brackets ( ) and you get a generator expression — the same computation, but lazy:

squares_list = [x*x for x in range(5)]   # list  : [0, 1, 4, 9, 16]
squares_gen  = (x*x for x in range(5))   # generator object, nothing computed yet

print(type(squares_list))   # <class 'list'>
print(type(squares_gen))    # <class 'generator'>

print(sum(x*x for x in range(1, 6)))   # 55 -- parentheses of sum() double as the genexpr's own

The last line is worth pausing on: sum(x*x for x in range(1, 6)) never builds the list [1, 4, 9, 16, 25] anywhere. Each square is generated one at a time and immediately folded into the running total by sum(), then discarded. For five numbers this makes no visible difference; for a similar sum over ten crore numbers, the list version would need tens of megabytes it never actually needs, while the generator version needs almost none.

Common misconception: "a generator is just a list with a different name"

This is false in a way that causes real bugs. A generator is exhausted after one full pass — once every value has been pulled out via next() (whether you called next() yourself or a for loop / list() / sum() did it for you), the generator has no way to "rewind." Asking it for more simply raises StopIteration immediately, giving you nothing:

squares = (x*x for x in range(5))
print(list(squares))   # [0, 1, 4, 9, 16]  -- consumes the generator fully
print(list(squares))   # []                -- already empty, nothing left!

A list has no such problem — you can loop over [0, 1, 4, 9, 16] as many times as you like, because it is just data sitting in memory, not a one-shot production line. If your program genuinely needs to read the same sequence of values more than once, either store it as a list (accepting the memory cost) or create a fresh generator each time you need another pass by calling the generator function again, since each call produces an independent generator object with its own paused state.

A second, related misconception: students sometimes assume a generator function's code runs the moment you call it, "in the background," and just delivers results slowly. It does not run in the background at all — it runs zero lines until the first next() arrives, and then it runs synchronously (blocking) up to the next yield before handing control back. There is no parallelism or background thread involved; generators are purely about when computation happens, not making it happen on a separate track.

Building your own iterator class (what yield is doing for you automatically)

Generators are actually a shortcut. Under the hood, any generator function automatically builds an object with __iter__ and __next__ methods that remembers its paused state for you. You can build that machinery by hand to see exactly what yield is saving you from writing:

class Countdown:
    def __init__(self, start):
        self.current = start

    def __iter__(self):
        return self          # the object is its own iterator

    def __next__(self):
        if self.current < 0:
            raise StopIteration
        value = self.current
        self.current -= 1
        return value

for n in Countdown(3):
    print(n, end=' ')
# Output: 3 2 1 0

Trace it: for calls iter(Countdown(3)), which runs __iter__ and gets back the object itself (since it returned self). Each loop iteration calls __next__(): while self.current is 3, 2, 1, 0, it saves that value, decrements self.current, and returns the saved value. When self.current becomes -1, the guard condition self.current < 0 is true, so __next__ raises StopIteration, and the loop ends. This is precisely what the generator version below achieves in three lines instead of nine — which is exactly why yield exists: it is Python writing the __iter__/__next__/state-tracking boilerplate for you.

def countdown(start):
    current = start
    while current >= 0:
        yield current
        current -= 1

A realistic use: generating admission roll numbers on demand

Suppose your school needs roll numbers for new Class 8 admissions in the format 8SCI0001, 8SCI0002, and so on, and the admission office will keep adding students throughout the week — the exact final count isn't known yet. A generator fits this perfectly, because it produces roll numbers one at a time without needing to decide the total in advance:

def roll_numbers(prefix, start, count):
    for i in range(count):
        yield f'{prefix}{start + i:04d}'

for roll in roll_numbers('8SCI', 1, 5):
    print(roll)
# Output:
# 8SCI0001
# 8SCI0002
# 8SCI0003
# 8SCI0004
# 8SCI0005

Here {start + i:04d} is a format specification meaning "print this integer padded to 4 digits with leading zeros" — so 1 becomes 0001. If tomorrow the office wants 500 more roll numbers, you do not need to change how the function stores anything; you simply call roll_numbers('8SCI', 6, 500) and it lazily produces exactly as many as requested, never holding more than one in memory at once.

When should you actually reach for a generator?

Use a generator when any of these are true: the full sequence would be too large to comfortably fit in memory (crores of records, a huge log file read line by line, an infinite mathematical sequence); you only need to look at each value once, in order, and do not need to jump backward or index into the middle; or you might stop early and do not want to pay the cost of computing values you will never use, as in the prime-search example.

Prefer an ordinary list when you need to access elements by index (data[7]), need to know the length in advance with len(), need to loop over the same data multiple times, or the data is genuinely small enough that memory was never the concern to begin with. Generators trade away these conveniences (no indexing, no len(), one-shot only) in exchange for near-zero memory footprint and the ability to represent sequences that are enormous or even infinite, like countdown or primes_above above, which a list could never hold in the first place.

Check your understanding

1. What will list(range(3)) and range(3) print if you run print() on each directly, and why do they look different even though they represent the same three numbers?
2. Trace this generator by hand and write down exactly what gets printed:

   def mystery(n):
       total = 0
       for i in range(1, n + 1):
           total += i
           yield total
   
   for value in mystery(4):
       print(value, end=' ')

3. A classmate writes squares = (x*x for x in range(100)) then runs total1 = sum(squares) followed immediately by total2 = sum(squares). What will total2 be, and why is it not equal to total1?
4. Rewrite this list comprehension as a generator expression, and explain in one sentence what changes about when the squaring actually happens: values = [n * n for n in range(1, 1000001)].
5. Why is while True: inside the generator function primes_above() not an infinite-loop bug, when the same while True: inside an ordinary function (without yield) would freeze the program forever?

Answers: (1) list(range(3)) prints [0, 1, 2] because it is a fully built list; range(3) prints range(0, 3) because a range object only stores its start/stop/step and never materialises the actual numbers unless you loop over it or convert it. (2) 1 3 6 10 — running totals 1, 1+2=3, 3+3=6, 6+4=10. (3) total2 is 0, because the first sum() call fully consumed the generator by pulling every value out via repeated next() calls; the second call finds it already exhausted and immediately hits StopIteration, so it sums zero values. (4) values = (n * n for n in range(1, 1000001)); with square brackets every one of the ten lakh squares is computed immediately when the line runs, but with round brackets no squaring happens at all until something actually iterates over values and requests each one. (5) Because yield pauses execution and hands control back to whoever called next(); the loop only ever advances one step per next() call, so it never runs unattended to infinity — the calling code decides how many times to ask, exactly like [next(gen) for _ in range(5)] stopped after five.

Summary

• An iterable is anything you can loop over (lists, strings, ranges, dictionaries); it provides an __iter__() method that produces a fresh iterator each time.
• An iterator remembers its current position and produces the next value via __next__(), raising StopIteration once there is nothing left — this is the "iterator protocol" that every for loop relies on internally.
• range() is a lazy, constant-memory object: it stores only start/stop/step and computes each value algebraically on demand, never building a full list.
• A generator function is a normal function that uses yield instead of (or alongside) return. Calling it does not run the code — it returns a paused generator object that runs up to the next yield each time next() is called, then freezes, remembering only its local variables.
• A generator expression — (expr for item in iterable) — is the lazy counterpart of a list comprehension: same syntax, round brackets instead of square, values computed one at a time instead of all at once.
• Generators are memory-efficient because they never hold the whole sequence at once: a list's size grows in proportion to how many items it stores, while a generator's size stays essentially constant no matter how many values it will eventually produce, or even if that number is infinite.
• A generator can only be iterated over once. After it is exhausted, further calls to next() (or another for/list()/sum() pass) yield nothing — unlike a list, which can be reread as many times as you like.
• Choose generators for huge, one-pass, or potentially early-stopping sequences; choose lists when you need indexing, len(), or to read the data more than once.

Think About It

Think about this: How would you explain generators and iterators: memory-efficient processing to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Practice Exercises

Now it is time to practice! Complete these challenges to solidify your understanding:

• Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
• Exercise 2: Find a real-world example where generators and iterators: memory-efficient processing is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
• Exercise 3: Create a mind-map connecting generators and iterators: memory-efficient processing to at least 3 other topics you have studied.