Understanding Python Iterables, Iterators and Generators

An Iterable is an object in Python which can return one of its member at a time, examples include list, str, tuple, dict, file etc. Iterables can be used with for loop and in some places with zip(), map() etc. 

We can pass an iterable object to built-in function iter() to get an iterator. An iterator is an object representing a stream of data and does have next() (Python 2.x) and __next__() (Python 3.x) methods which can be called to get the successive data. When no more data is available, a StopRaiseException is raised.

So what’s the difference between the Iterator and Iterables?

An Iterable can be considered as collection of objects which can be iterated over and Iterator lets you specify what it means to iterate over an object. Let’s consider some examples :- 

 

myList = [1, 2, 3, 4, 5, 6, 7, 8, 9] # Iterable (List)
myString = "SomeText" # Iterable (String)

Above two examples (list and str) are iterable objects and we can iterate over an individual element.

for x in myList:
    print x

But how does for loop work, we haven’t generated an iterator for the list and str? 

Whenever for loop is used, iter() method is called to get the iterator of the object and then in every iteration next() method is called.

If we want to achieve the same results as for loop using iter() method, we can do the following :- 

i = iter(myList)
i.next()  >> print 1
i.next()  >> print 2
...
i.next() >>  print 9
i.next()  >> StopIteration Exception Raised

So the bottom line is that while using itertables with for loop, there is no need to deal with the iter() method and it would be done automatically by for loop by creating a temporary variable to hold the value of iter() value for the duration of the loop.

Now the next part is how can I create an Iterable Object?

For creating an iterable object such that if it is called with for loop,  we need to add __iter__() and next() method to it.

class Counter:
    def __init__(self, low, high):
        self.current = low
        self.high = high

    def __iter__(self):
        return self

    def next(self):
        if self.current > self.high:
            raise StopIteration
        else:
            self.current += 1
            return self.current - 1

for c in Counter(1,5):
    print c

In general __iter__() should return the iterable object and is called implicitly at the start of the loops (e.g. for loop). The next() method returns the next value and is implicitly called at each loop increment. next() raises a StopIteration exception when there are no more value to return, which is implicitly captured by looping constructs to stop iterating.

Let's move to Generators now ...

Generators are also Iterators (not true vice-versa) which allows you to iterate over the values once i.e. the values are not stored in memory but are generated on the fly. Generators simplifies the creation of Iterators without having to write a class with __iter__() and __next__() methods.

Custom Iterator class for Counter can be written using Generators as below :-

def counter(low,high):
    while low <= high:
        yield low

        low = low + 1

for c in counter(1,5):
    print c

d = counter(1,5)

print d.next() => 1

print d.next() => 2

print d.next() => 3

print d.next() => 4

print d.next() => 5

print d.next() => StopIteration 

How it actually works?

- When the generator function is called, it returns a generator object without even beginning execution of any code written inside the function.

- When the next method is called for the first time, the function starts executing until it reaches yield statement. The yielded value is returned by the next call.