Python Generator Functions and yield: Implementing Memory-Efficient Iteration

In Python, objects that allow you to “extract multiple elements in order,” like lists, are called iterables. Among them, “Generators” have a special and powerful feature. While lists expand all elements into memory at once, generators repeat the action of “generating one value and then pausing.” This makes it possible to minimize memory consumption even when handling large amounts of data.

This article explains how to define generator functions using the yield statement, their execution flow, and practical usages like generating infinite sequences.

---

Basic Structure of a Generator Function (yield)

Generator functions use the same syntax as normal function definitions (def), but they differ in using yield instead of return when returning a value.

• return: Returns a value and terminates the function.
• yield: Returns a value and pauses the function’s processing. The state is retained, and upon the next call, it resumes from where it left off.

Basic Code Example

Let’s create a generator that returns the progress of a process as a string.

def step_generator():
    """
    Generator that advances the process little by little each time it is called.
    """
    print("--- Step 1 Start ---")
    yield "Step 1 Complete"
    
    print("--- Step 2 Start ---")
    yield "Step 2 Complete"
    
    print("--- Step 3 Start ---")
    yield "Step 3 Complete"

# Create a generator object (The function body is NOT executed at this point)
gen_process = step_generator()

print(f"Object: {gen_process}")

Output:

Object: <generator object step_generator at 0x...>

---

Execution and Resumption with next() Function

To retrieve a value from the generated generator object, use the built-in function next(). When next() is called, the generator function proceeds to the next yield statement, returns the value there, and pauses.

# 1st next()
print(f"Value 1: {next(gen_process)}")

# 2nd next()
print(f"Value 2: {next(gen_process)}")

# 3rd next()
print(f"Value 3: {next(gen_process)}")

Output:

--- Step 1 Start ---
Value 1: Step 1 Complete
--- Step 2 Start ---
Value 2: Step 2 Complete
--- Step 3 Start ---
Value 3: Step 3 Complete

After the 3rd yield, if you call next() one more time, a StopIteration exception occurs because there are no more yields, notifying the end of the generator.

---

Using in for Loops

In practice, manually calling next() repeatedly is rare. Since generators are iterable, you can loop through them with a for statement just like lists. The for statement internally detects the StopIteration exception and automatically ends the loop, so you don’t need to write error handling.

# Create a new generator object
gen_loop = step_generator()

print("--- Loop Start ---")

for step in gen_loop:
    print(f"Received in loop: {step}")

print("--- Loop End ---")

Output:

--- Loop Start ---
--- Step 1 Start ---
Received in loop: Step 1 Complete
--- Step 2 Start ---
Received in loop: Step 2 Complete
--- Step 3 Start ---
Received in loop: Step 3 Complete
--- Loop End ---

---

Practical Example: Infinite Sequence (Fibonacci)

Utilizing the generator’s characteristic of “generating values only when needed,” you can represent sequences that theoretically continue infinitely without overwhelming memory. As an example, let’s create a generator that generates the Fibonacci sequence (a sequence where the sum of the previous two terms becomes the next term: 0, 1, 1, 2, 3, 5…).

def fibonacci_sequence():
    """
    Generator that generates the Fibonacci sequence infinitely.
    """
    num_a, num_b = 0, 1
    
    while True:
        # Return current num_a and pause
        yield num_a
        
        # Calculate next values
        num_a, num_b = num_b, num_a + num_b

# Create the generator
fib_gen = fibonacci_sequence()

print("Displaying the first 10 Fibonacci numbers:")

# Since it's an infinite loop, limit the count with range and call next()
for _ in range(10):
    value = next(fib_gen)
    print(value, end=", ")

Output:

Displaying the first 10 Fibonacci numbers:
0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 

Although this fibonacci_sequence function contains an infinite loop with while True, the program does not freeze because it pauses at yield every time. You can extract just the number needed with next().

---

Summary

• A generator is a type of iterable object.
• By using yield instead of return, you can pause and resume while retaining the function’s state.
• Use the next() function to retrieve the next value.
• Typically used in for loops.
• Because it does not expand all data into memory at once, it is very efficient when handling huge data or infinite sequences.