Iterators vs. iterables -- a comparison

I received a comment that I was confusing classes that implement iterators with classes that implement iterables. That may seem like a small, legalistic distinction, but Kent's comment also explained that this distinction had important consequences.

So, let's look at some of the differences between iterators and iterables.

• The __iter__ method:

  • An iterator is implemented by a class whose __iter__ method returns the object itself, and the __iter__ method has no side effects, e.g. it does not save something in the object or reset the iterator.
  • An iterable is implemented by a class whose __iter__ method returns a generator. It might do this by calling a method containing a yield statement.

• Resetting the iterator:

  • An iterator object can be "reset" only by creating a new object. You cannot iterator over the same object more than once.
  • An iterable object can create new iterators for the same object. This can be done in the __iter__ method (which can be called, for example, using the iter() built-in function or by referencing the object in a for statement) or by special purpose methods that return iterators.

• Multiple iterators from the same object:

  • An iterator object cannot produce multiple iterators. The __iter__ method always returns the object itself. And, once that object exhausts its sequence, it is required by the iterator protocol to stay exhausted.

  • An iterable object can produce multiple iterator (generator) objects, each of which can produce a sequence. In fact, an iteratible object can produce iterators that produce different sequences. Consider:

    • Dictionary objects provide different methods that return iterators over different sequences. iterkeys(), itervalues(), and iteritems(), return iterators that generate sequences of the keys, values, and (key, value) tuple sequences from the dictionary.

    • We can imagine a tree structure that creates iterators that return a sequence of all nodes in the tree that satisfy a predicate. Here is an example:

      class Node(object):
          o
          o
          o
          def recursive_walk_tree_with_predicate(self, predicate):
              if predicate(self):
                  yield self
              for child in self.children:
                  child_iter = child.recursive_walk_tree_with_predicate(predicate)
                  for child1 in child_iter:
                      yield child1
      
      def predicate1(node):
          val = node.get_value()
          val = ord(val[-1])
          if val % 2 == 0:
              return True
          else:
              return False
      
      def test(tree):
          for node in tree.recursive_walk_tree_with_predicate(predicate1):
              f.write('value: "%s"\n' % (node.value, ))
      

Consequences:

• An iterator object can be iteratored over only once. Another way of saying this this is that an iterator produces only one sequence. An iterable can produce multiple iterators and even iterators that produce different sequences.

• An iterable object (but not an iterator object) can produce multiple iterators whose use can be interleaved. These iterators are independent in the sense that calling one does not alter the sequence produce by the other. Here is an example:

  In [1]: class A(object):
     ...:     def __init__(self, collection):
     ...:         self.collection = collection
     ...:     def __iter__(self):
     ...:         return iter(self.collection)
     ...:
     ...:
  In [2]: a = A(range(5))
  In [3]: b = iter(a)
  In [4]: c = iter(a)
  In [5]: b.next()
  Out[5]: 0
  In [6]: b.next()
  Out[6]: 1
  In [7]: b.next()
  Out[7]: 2
  In [8]: c.next()
  Out[8]: 0
  In [9]: c.next()
  Out[9]: 1
  In [10]: b.next()
  Out[10]: 3
  

  To say that the multiple iterators produced by an iterable are independent is not to say that an object could not be implemented which produces iterators that do communicate. However, that will often be a bug, or would serve some special purpose.

The above are not "hard and fast" rules. You are certainly free to implement iterators and iterables any way that Python enables you to. However, understanding some of the possible distinctions between iterator object and iterable object explained above, should help you to implement the iterator or iterable that you do want and not something that you don't.

A few cautions

Here are a few pitfalls and corner cases that you will want to be aware of:

• An iterator (using the definition above) does not refresh or reset itself. Each time we ask an iterator object for its iterator/generator it returns itself, which may be (partially) exhausted. That might be what you want, but might not. Here is an example:

  In [24]: a = range(5)
  In [25]: iter1 = iter(a)
  In [26]: iter1.next()
  Out[26]: 0
  In [27]: iter1.next()
  Out[27]: 1
  In [28]: for x in iter1:
     ....:     print x
     ....:
  2
  3
  4
  

  An interable, in contrast with an iterator, might have behaved differently. In particular, depending on the implementation, an iterable might have begun from the beginning of the sequence when used in the for statement.

Non-recursive tree walk iterator version 1

This version has a next() method, as required by the iterator protocol.

class Node(object):
    def __init__(self, value='<no value>', children=None):
        self.value = chr(value + 97) * 3
        if children is None:
            children = []
        else:
            self.children = children
    def walk_tree(self):
        # stack to hold nodes as we walk through
        stack = []
        stack.append(self)
        while stack:
            node = stack.pop()
            # reverse children to get the right order.
            stack.extend(reversed(node.children))
            yield node
    def __iter__(self):
        self.iterator = self.walk_tree()
        return self
    def next(self):
        return self.iterator.next()

TREE = Node(0, [Node(1, [Node(2, [Node(3, []),
                      Node(4, []),
                      Node(5, [Node(6, []),
                           Node(7, []),
                           Node(8, []),
                           Node(9, []),
                           ]),
                      Node(10, [Node(11, []),
                            Node(12, []),
                            ]),
                      ]),
                 Node(13, [Node(14, []),
                       Node(15, []),
                       Node(16, [])]),
                 Node(17, []),
                 Node(18, []),
                 ]),
            Node(19, []),
            Node(20, []),
            ])

def test():
    for node in TREE:
        print 'value: %s' % (node.value, )

test()

Non-recursive tree walk iterator version 2

This version does not have a next() method, but the iterators returned by __iter__() do have a next().

class Node(object):
    def __init__(self, value='<no value>', children=None):
        self.value = chr(value + 97) * 3
        if children is None:
            children = []
        else:
            self.children = children
    def show(self, indent):
        self.showindent(indent)
        print 'value: %s' % (self.value, )
        for child in self.children:
            child.show(indent+1)
    def showindent(self, indent):
        sys.stdout.write(' ' * (indent * 4))
    def walk_tree(self):
        # stack to hold nodes as we walk through
        stack = []
        stack.append(self)
        while stack:
            node = stack.pop()
            # reverse children to get the right order.
            stack.extend(reversed(node.children))
            yield node
    def __iter__(self):
        return self.walk_tree()

TREE = Node(0, [Node(1, [Node(2, [Node(3, []),
                      Node(4, []),
                      Node(5, [Node(6, []),
                           Node(7, []),
                           Node(8, []),
                           Node(9, []),
                           ]),
                      Node(10, [Node(11, []),
                            Node(12, []),
                            ]),
                      ]),
                 Node(13, [Node(14, []),
                       Node(15, []),
                       Node(16, [])]),
                 Node(17, []),
                 Node(18, []),
                 ]),
            Node(19, []),
            Node(20, []),
            ])

def test():
    for node in TREE:
        print 'value: %s' % (node.value, )

test()

Concepts

Producers and consumers -- The separation of the producer of a stream or sequence of items from the consumers of that stream of items enables us to (1) reuse the producers in different contexts and applications, (2) maintain the producer implementation independently from the consumers, and (3) implement producers (iterators in Python talk) that can be used in different contexts (for example, in a for statement or passed to different parts of our program and used there.

Sequences -- An iterator produces a sequence of items. The iterator produces the elements of the sequence one by one. We can use the items in that stream (1) in a for statement, (2) in a list comprehension or a generator expression, or (3) in locations throughout our program (by calling the next() method on the iterator).

Concrete sequences and abstract sequences -- We can think of some sequences as being concrete, meaning that all the items in the sequence exist, at one time, in a data structure. Python lists, tuples, and strings would be concrete sequences in this sense. In contrast, we can think of an abstract sequence as one that we can define or at least can create an implementation that produces it, however the sequence might not exist until we produce it and even then, not all of the sequence may exist at any given time. Our implementation. when called, will produce one item at a time as and when requested. An abstract sequence (and the items in it) can be defined by connotation (rather than denotation) or by a calculation.

Consumers

Consuming an iterator at a single location -- the for statement.

Consuming an iterator at multiple locations -- An iterator object can be passed around in our program, can be passed into a function, can be returned from a function, and can be saved in a data structure. In other words an iterator is a first class object.

How to use an iterator object -- An iterator object obeys the iterator protocol. Specifically, it implements a next() method and raises the StopIteration exception when it is exhausted (there are no more items to be produced). So, here is an example of how we might get the next object from an iterator:

def use_one(aniter):
    try:
        obj = aniter.next()
        print obj
    except StopIteration, e:
        print "That's all"

def test():
    myiter = iter(range(3))
    use_one(myiter)
    use_one(myiter)
    use_one(myiter)
    use_one(myiter)

test()

Which, when run, produces the following output:

0
1
2
That's all

Iterators are "first class objects" -- (1) We can pass an iterator to a function. (2) We can return an iterator from a function. And (3), we can stuff an iterator into a data structure for use later. Therefore, the consumer of a sequence is not restricted to using all the items in the sequence in a single location.

Producers

An iterator is a producer.

An iterator produces a sequence, or perhaps more correctly, it produces the elements of a sequence one by one.

An iterator obeys (implements) the iterator protocol. See below.

An iterator can be used (1) in a for statement, (2) in a list comprehension or a generator expression, (3) by calling the next() method implemented by the iterator.

The iterator protocol

Iterator objects themselves are required to support the following two methods, which together form the iterator protocol:

• __iter__() -- This method should return the iterator object itself. This can be done with return self. If you wish to implement an iterable rather than an iterator (and object that can be asked to produce multiple sequences), then you might return the value returned by a generator method (i.e. a method that contains the yield statement.
• next() -- Return the next item in the sequence. Raise the StopIteration exception when there are no more items in the sequence. But, if you implement the iterator with a generator method (a method containing yield), then a next() method is not needed, because the object (the iterator) returned by the generator method will support the next() method.

You will (almost?) always want a constructor (the __init__ method). I'm not sure whether the protocol requires it, but it's usually needed.

For more on the iterator protocol, see 3.5 Iterator Types in the "Python Library Reference".

How to implement generators

Use the yield statement -- If a function definition contains a yield statement, then the value returned by the function is a generator/iterator, i.e. an object that obeys the iterator protocol.

Example:

>>> def test():
...     yield 'hi'
...     yield 'bye'
...
>>> testiter = test()
>>> testiter
<generator object at 0xb7d6b7cc>
>>> testiter.next()
'hi'
>>> testiter.next()
'bye'
>>> testiter.next()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration
>>>
>>> dir(testiter)
['__class__', '__delattr__', '__doc__', '__getattribute__',
'__hash__', '__init__', '__iter__', '__new__', '__reduce__',
'__reduce_ex__', '__repr__', '__setattr__', '__str__', 'close',
'gi_frame', 'gi_running', 'next', 'send', 'throw']

Notes:

• Since the function test contains yield, this function returns a generator/iterator object.
• The iterator raises the StopIteration exception when it is exhausted.
• Notice that the list of attributes for the iterator includes __iter__ and next, which are required for the iterator protocol.

Here is a more complicated example:

import sys
import getopt

def get_reply():
    while True:
        reply = raw_input("Entry ('quit' to quit): ")
        if reply == 'quit':
            raise StopIteration
        yield reply

def get_bundle(bundle_size):
    bundle = []
    count = 0
    for x in get_reply():
        count += 1
        bundle.append(x)
        if count >= bundle_size:
            yield bundle
            bundle = []
            count = 0
    yield bundle

def test():
    getter = get_bundle(3)
    for bundle in getter:
        print 'bundle:', bundle
        print

test()

When you run it, it might look like this:

$ python test_iterator.py
Entry ('quit' to quit): aaaa
Entry ('quit' to quit): bbbb
Entry ('quit' to quit): cccc
bundle: ['aaaa', 'bbbb', 'cccc']

Entry ('quit' to quit): dddd
Entry ('quit' to quit): eeee
Entry ('quit' to quit): quit
bundle: ['dddd', 'eeee']

Implement an iterator object (class) -- Two alternative methods:

1. Use yield -- In this technique, you will (1) implement a generator method in your class using the yield statement and (2) return the value returned by the generator method (an iterator) in your __iter__ method. Here is an example:

   class Tree(object):
       ...
       def walk_tree(self, level):
           yield (level, self, )
           for child in self.get_children():
               iterchildren = child.walk_tree(level+1)
               for level1, tree1 in iterchildren:
                   yield level1, tree1
       def __iter__(self):
           return self.walk_tree(self.initlevel)
   

   Notes:

   • This iterator is recursive -- walk_tree calls itself to produce an iterator that is used to walk a sub-tree.
   • The __iter__ method calls the walk_tree method to produce and return an iterator. Since the value returned by walk_tree() is an iterator, it obeys the iterator protocol.

2. Implement an explicit next() method -- In this technique, you will (1) implement a next method in your class and (2) return the object itself (i.e. self) in your __iter__ method. An example:

   class Doubler(object):
       def __init__(self, collection):
           self.collection = collection
           self.index = 0
       def next(self):
           if self.index < len(self.collection):
               value = self.collection[self.index]
               self.index +=1
               return value * 2
           else:
               raise StopIteration
       def __iter__(self):
           self.index = 0
           return self
   
   def test():
       collection = range(5)
       doubler = Doubler(collection)
       for value in doubler:
           print 'value: %d' % (value, )
       print '-' * 30
       for value in doubler:
           print 'value: %d' % (value, )
   
   test()
   

   Notes:

   • The Doubler class is not strictly an iterator. Note that its __iter__ method resets the index and, thereby resets the sequence to be produced. A careful reading of the the iterator protocol would disallow doing so. Therefore, the Doubler class as implemented above, may or may not behave the way we want.
   • The method next keeps track of what/where the next object to be returned is. Each call to next produces the next object in the sequence.
   • Method next raises the StopIteration exception when the sequence is exhausted.
   • __iter__ returns the object itself. It also initializes the iterator by setting the index to zero.

A few additional notes and suggestions on implementations:

• If a similar problem without generators could be solved with a (normal) recursive function, then consider using the yield statement in a function that recursively calls itself to produce "sub-" generators. See the walk_tree example, above.
• If there is an easy or straightforward way to keep track of what the next value should be or where the next value is in a data structure, then consider implementing a class and a next method in that class. See the Doubler example, above.

Using an iterator

Suppose we have an iterator myiter, that produces a sequence of words. Here are some ways that we can use that iterator:

In [1]: words = 'the baby bison romped in the meadow'.split()
In [2]: words
Out[2]: ['the', 'baby', 'bison', 'romped', 'in', 'the', 'meadow']
In [3]: myiter = iter(words)

Use it in a for statement:

>>> words = 'the baby bison romped in the meadow'.split()
>>> myiter = iter(words)
>>> for item in myiter:
    print item.capitalize()


The
Baby
Bison
Romped
In
The
Meadow

Use it in a list comprehension:

>>> words = 'the baby bison romped in the meadow'.split()
>>> myiter = iter(words)
>>> doublewords = [item * 2 for item in myiter]
>>> doublewords
['thethe', 'babybaby', 'bisonbison', 'rompedromped', 'inin', 'thethe', 'meadowmeadow']

Use it in a genrator expression:

>>> words = 'the baby bison romped in the meadow'.split()
>>> myiter = iter(words)
>>> doublewords = (item * 2 for item in myiter)
>>> doublewords.next()
'thethe'
>>> doublewords.next()
'babybaby'
>>> doublewords.next()
'bisonbison'
>>> doublewords.next()
'rompedromped'
>>> doublewords.next()
'inin'
>>> doublewords.next()
'thethe'
>>> doublewords.next()
'meadowmeadow'
>>> doublewords.next()

Traceback (most recent call last):
  File "<pyshell#25>", line 1, in <module>
    doublewords.next()
StopIteration

Pass it to a function:

>>> words = 'the baby bison romped in the meadow'.split()
>>> myiter = iter(words)
>>> def consumer(worditer):
    for item in worditer:
            print item.upper()

>>> consumer(myiter)
THE
BABY
BISON
ROMPED
IN
THE
MEADOW

Stuff it into a data structure:

>>> words = 'the baby bison romped in the meadow'.split()
>>> myiter = iter(words)
>>> def consumer(struct):
        print 'before:', struct[0]
        for item in struct[1]:
            print item.upper()
        print 'after:', struct[2]


>>> struct = ('start', myiter, 'finish')
>>> consumer(struct)
before: start
THE
BABY
BISON
ROMPED
IN
THE
MEADOW
after: finish

Some sources and references

See PEP 255 -- Simple Generators.

Lib/test/test_generators.py in the Python source distribution contains many examples.

There is more information, some of which is elementary and some advanced here: Functional Programming HOWTO.

The "What's new" section for Python 2.2 also discusses iterators. See: 3 PEP 234: Iterators

The process

Notes on the standard process can be found in your python2.x/site.py.

A qualification -- I believe, but am not sure, that if you have used easy_install on your system, then it might be that lib/python2.x/site.py, lib/python2.x/site-packages/site.py have been replaced.

An important thing to note is that, while .pth files are processed last, easy_install is able to patch the easy_install.pth file so that it shoves it's paths to the top of sys.path.

Complications -- easy_install and setuptools

If you have used easy_install on your system and you did not specify the -m or --multi-version, then easy_install creates and adds directories to a easy-install.pth file. But, it is even a bit more complex than that because easy_install also puts a bit of code in easy-install.pth that forces paths in which it has installed a package to the top of sys.path. Further more, this apparently happens after the PYTHONPATH environment variable is processed. So, you cannot

And, easy_install also appears to add it's own site.py, and tricks Python into using the easy_install version of site.py instead of the standard one. See these notes:

• http://mail.python.org/pipermail/distutils-sig/2006-February/006026.html
• http://mail.python.org/pipermail/distutils-sig/2006-February/006033.html

So perhaps a message to somebody would help, but I don't know who:

I've noticed some strange behavior on my machine since installing pylons. I believe that it is done by easy_install, but I'm not sure.

There is alse a site.py in lib/python2.5/site-packages (in addition to the one in lib/python2.5). Where did that come from? Did easy_install put it there? There are very few comments in it, as though someone does not want me to know who put it there.

Does anyone have a recommended method of overriding paths inserted into sys.path by easy_install? Basically, I want to force lib/python2.5/site-packages to the front/top. I've been using a sitecustomize.py file in my current directory, but that has the feel of a kludge to it.

Thanks for help.

Dave

Next step: Study the site.py replacement from easy_install.

What to do

Some options:

• Modify python2.x/site.py -- Wrong! This file is distributed with Python. It will be overwritten the next time you upgrade. That is probably not what you want.
• Add a sitecustomize.py to each directory where you need your custom path. The down-side here is remembering to add this file. On Linux/UNIX you can make it slightly more convenient by place a single copy in a common place, then creating a symbolic link to it from each needed directory. That will make it easier to update someday.
• Add your own .pth file. If you name your file something like zzz_mypaths.pth, your file will be processed last and will win.

Explanation

A variable is a name bound to a value in a namespace.

A namespace is a dictionary in which Python can look up a name (possible) to obtain its value. Note that the use of dictionaries to implement namespaces is an implementation specific to Python. Other programming languages implement namespaces differently.

Names refer to objects. Objects can be integers, tuples, list, dictionaries, strings, instances of classes, functions, classes (themselves), other Python built-in types, and instances of classes. And, don't forget, references to objects can also be held in data structures (lists, dictionaries, etc).

Determining which namespace a name is in is static. It can be determined by a lexical scan of the code. If a variable is assigned a value anywhere in a scope (specifically within a function or method body), then that variable is local to that scope. If Python does not find a variable in the local scope, then it looks next in the global scope (also sometimes called the module scope) and then in the built-ins scope. But, the global statement can be used to force Python to find and use a global variable (a variable defined at top level in a module) rather than create a local one.

Determining whether a name is bound (has a value) in a namespace is dynamic. You must follow the logic of the code in order to determine (1) when a variable has been bound to a value and (2) what value the variable is been bound to at any given point in the execution of the program. A variable is given a value at a certain time during the execution of the code in a scope. For example, in the following function, the variable count is not bound to a value until the end of the first iteration of the loop:

def test_dynamic():
    for idx in range(5):
        if idx > 1:
            x = count
            print 'x:', x
        count = idx

In Python, since the use of objects and references to them are so pervasive and consistent, we sometimes conflate a variable and the object it refers to. So, for example, if we have the following code:

total = 25
items = [11, 22, 33]
def func1():
    pass
class Class1:
    pass

we sometimes say:

• total is an integer.
• items is an list.
• func1 is a function.
• Class1 is a class.

But, if we were more careful, we might say:

• total is a variable that refers to an integer object.
• items is a variable that refers to a list object.
• func1 is a variable that refers to a function object.
• Class1 is a variable that refers to a class object.

Or, even:

• total is a name bound to an integer object in the current namespace.
• items is a name bound to a list object in the current namespace.
• func1 is a name bound to an function object in the current namespace.
• Class1 is a name bound to an class object in the current namespace.

Rules for determining in which scope a name is bound

Trying to keep this as simple as possible ... might have to ignore a few corner cases ...

Use these rules to determine the scope of a name/variable:

1. If the name or variable is assigned in a scope, then the variable is bound in that scope.
2. Else, the name/variable is bound in the global or module scope, or is a built-in.
3. Else, the name is undefined.

In order to force Python to use a global variable when it is assigned a value in a function, use the global statement.

A few additional notes:

• Modules and functions (and methods) create scopes.

• Python does not treat statement blocks as separate scopes. For example, the body or block in an if/else or for statement does not create a separate scope.

• A class does not create a separate scope (although the methods in it do). For example an instance of the following class:

  ClassGlobal1 = 'Class global data'
  
  class TestScope:
      ClassGlobal1 = 'Class local data'
      def show(self):
          print '(TestScope.show) ClassGlobal1: %s' % ClassGlobal1
  

  references the global (module) variable ClassGlobal1, not the one defined in the class, and will print out:

  (TestScope.show) ClassGlobal1: Class global data
  

  In order to access the class level variable, I must qualify it with the class, for example, as follows:

  TestScope.ClassGlobal1
  

• Although it may seem obvious, if from moduleA I import moduleB, then call function1 in moduleB, it is moduleB's module scope that is seen by function1, not moduleA's. In other words, it is lexical scope that matters, not dynamic scope. This is true by design, and so that I can understand moduleB and the code in it by reading that code and without knowing the code in moduleA or any other module from which function1 might be called.

Namespace binding statements etc

All the following Python statements bind a value to a name (a variable) in a namespace:

• The assignment statement -- Binds the computed value of an expression (on the right-hand side of the assignment operator) to a name.
• The def statement -- Creates a function object and binds that function object to a name.
• The class statement -- Creates a class object and binds that class object to a name.
• The import statement -- Evaluates a module (or a package) and binds that module object or an object within the module object to a name.
• The for statement -- Binds the next item(s) from a sequence or iterator to a name.
• The except: clause in a try: statement -- Binds an exception to a name.

Also, function or method parameters bind an actual parameter value from a call of the function to the formal parameter name in the local (function/method) scope. Or, if the actual value is omitted in the call and there is a default value, the default value is bound to the formal parameter name in the local scope. Additional notes: (1) The binding of actual values to formal parameters happens each time the function is called. (2) Default values, if the function definition has any, are evaluated only once, which is why you usually do not want to use mutable objects as default values. Specifically, do:

def f(values=None):
    if values is None:
        values = []
    ...

Do not do:

def f(values=[]):
    ...

The second of the above creates only one empty list which is shared by all invocations that omit the parameter.

Functions and classes and other objects

In Python, it is important to understand that functions and classes (and other types of objects too) are objects that can be referred to by a variable, stored in a data structure (e.g. a list or dictionary), passed to a function, returned by a function, etc.

It is also important to realize that variables in Python are simply names in a namespace that refer to objects of some kind.

Or, summarized in other words:

1. Names are references to objects and

2. Objects are first class, which means that we can:

   • Store them in variables.
   • Store them in data structures.
   • Pass them to functions and methods.
   • Return them from functions and methods.

   Although, perhaps we should qualify the above a bit by saying that variables and data structures hold references to objects, and that we can pass references to objects into functions, and so on.

Accessing namespace dictionaries

The built-in functions globals() and locals() return the dictionaries the represent the global and the local namespaces respectively. Caution: Although you can get a dictionary that represents the current global or local namespace, it is questionable that you can modify a namespace by modifying the dictionary returned by globals() or locals(). Actually, it seems to work with globals(), but definitely fails in some case with locals().

Nested scopes -- Note that for lexically/statically nested scopes (for example, a function defined inside a function), it seems that globals() and locals() still give access to all items in the accessible namespaces, but do not give dictionary style access to all visible scopes. In particular, variables in nested scopes are not included in locals() and globals() For more on this, see PEP 227: Statically Nested Scopes.

When you want to inspect Python's namespaces and symbol tables, also look at the built-in functions dir() and vars(). dir() returns a (not necessarily complete) list of names in the current local symbol table. vars() returns a dictionary corresponding to the current local symbol table. Note that dir() and vars() can take an optional object as an argument. See the Python documentation (below) for more on this.

For more information on the above built-in functions, see 2.1 Built-in Functions in the Python Library Reference.

Additional information

More help and explanation on names, namespaces, bindings, etc is here:

• An explanation of naming and binding in the Python Reference Manual: 4.1 Naming and binding.
• An explanation of closures at Wikipedia, for those of you who must use them: Closure (computer science).
• Alan Gauld's explanation of names, bindings, and namespaces: Namespaces. And, Alan's Learn to Program web site is here Learning to Program.
• For more information on statically nested scopes, see this PEP (Python enhancement proposal): PEP 227: Statically Nested Scopes.
• Here is another explanation of namespaces: Functions and Namespaces. It's from Building Skills in Python.

Introduction

Python is less rigid than some languages with respect to interfaces. And, while there are several implementations of some of the capabilities provided by interfaces (for example, in Zope), there is, as of this date, no implementation of interfaces in the Python standard library.

So, in Python, what do we mean by an interface? An interface, sometimes called a protocol, is a description of the methods and their signatures that must be implemented by a class in order to satisfy the protocol. If a class implements those methods, then we say that the class implements the interface/protocol and we say that the interface is provided by the class (that implements it).

Why would we care about defining and implementing interfaces? An interface can serve as a way for a service provider to tell a service consumer/user about the characteristics of a class or instance that the user must provide. An example is provided by the standard input, output, and error streams in the Python sys module:

stdin stdout stderr

File objects corresponding to the interpreter's standard input, output and error streams. stdin is used for all interpreter input except for scripts but including calls to input() and raw_input(). stdout is used for the output of print and expression statements and for the prompts of input() and raw_input(). The interpreter's own prompts and (almost all of) its error messages go to stderr. stdout and stderr needn't be built-in file objects: any object is acceptable as long as it has a write() method that takes a string argument. (Changing these objects doesn't affect the standard I/O streams of processes executed by os.popen(), os.system() or the exec*() family of functions in the os module.) [emphasis added]

See: 3.1 sys -- System-specific parameters and functions (and search for "stdin").

This suggests that we adopt the following point of view -- If I learn the required interface for a replacement for sys.stdout, then I have learned how to implement a class that redirects or filters output from my program. In a similar way, the developer of a framework can tell me how to implement a class, so that I can pass the class or an instance of the class to the framework, and by doing so can customize the behavior of the framework.

Let's categorize several approaches to interfaces in Python from strict to loose:

1. Strict -- Example, Zope interfaces. This implementation actually gives interfaces behaviors that I can use in my code. For example, I can ask whether a class or instance implements a given interface.
2. Python abstract classes -- Use a Python class with method headers and documentation but no implementation as a means of describing your interface.
3. Loose -- "Duck-typing" -- If it looks like a duck and quacks like a duck ... With this approach, I give a text description of the requirements for a class, and by doing so, inform a user of the minimum requirements that must be satisfied in order to use that class or an instance of it for a specific purpose.

Zope interfaces

Zope provides an implementation of interfaces for Python. Using Zope interfaces, if you are not already using Zope, might be more trouble than it is worth. And, it will require that all your users will need to install Zope also. Still, especially for someone who is interested in tutoring and teaching new Python programmers, possibly programmers who are familiar with Java, Zope interfaces may be worth thinking about.

Although they are implemented in the Zope distribution, it can also be installed separately and can be used outside of Zope applications.

Here are a few of the capabilities provided by Zope interfaces (copied from my_zope_install/lib/python/zope/interface/README.txt):

• We can ask an interface whether it is implemented by a class:

  >>> IFoo.implementedBy(Foo)
  True
  

• We can ask whether an interface is provided by an object:

  >>> foo = Foo()
  >>> IFoo.providedBy(foo)
  True
  

• We can ask what interfaces are implemented by an object:

  >>> list(zope.interface.implementedBy(Foo))
  [<InterfaceClass __main__.IFoo>]
  

For more information on the Zope implementation of interfaces, see:

• http://www.zope.org/Wikis/Interfaces/FrontPage
• http://svn.zope.org/Zope3/trunk/src/zope/interface/README.txt?view=markup

Here is a trivial example:

from zope.interface import Interface, implements

class IA(Interface):
    def show(self, level):
        """Show this object.
        """

class A:
    implements(IA)
    def show(self, msg):
        print '(A.show) msg: "%s"' % msg

def test():
    a = A()
    a.show('hello')
    print IA.implementedBy(A)

test()

Notes:

• In order for the above example to work, you will need to do one of the following:
  1. Add your_zope_install/lib/python to your PYTHONPATH environment variable. This requires installing Zope.
  2. Install the Zope interface package, which is available at Welcome to the Interfaces Wiki
• One advantage of using Zope interfaces (over the more informal methods described below) is that interfaces that you define and the zope.interface module itself have capabilities that help you with the use of interfaces.

Python abstract classes

We can document an interface by providing a Python class containing method headers and method doc-strings but no method implementations.

Here is an example:

class MyAbstract:
    def __init__(self, name):
        if self.__class__ == MyAbstract:
            raise NotImplementedError, 'class MyAbstract is abstract'
    def write(self, msg):
        """Write a message.
        """
    def show(self):
        """Display the name etc.
        """

class MyConcrete(MyAbstract):
    """This class implements interface MyAbstract.
    """
    def __init__(self, name):
        MyAbstract.__init__(self, name)
        self.name = name
    def write(self, msg):
        print '(MyConcrete:%s) msg: %s' % (self.name, msg, )
    def show(self):
        print '(MyConcrete) name: %s' % self.name

Notes:

• The check for the creation of an instance of the abstract class is only for the paranoid among us.
• There is actually no need for class MyConrete to inherit from class MyAbstract. Class MyAbstract is merely serving to document the interface.
• Our empty methods in the abstract class do not require a pass statement, even though their bodies are empty. The doc-string serves this purpose.

Using documentation and "duck-typing"

We can describe the interface or protocol in text, saying something like "any class that has a method named this with these arguments and a method named that with those arguments and ...". Needless to say, if there are more than a couple of methods, you are likely to want to switch to the use of an abstract class to document your interface.

Error checking for interfaces

The following example might provide suggestions for those who want to do a bit of error checking in code that requires a class or instance that must provide an interface:

import inspect

class A:
    def show(self, level):
        print '(A.show) level: %s' % level

class B:
    def show(self):
        print '(B.show) level: %s' % level

#
# This function requires an object that implements a method named 'show'
#   which takes one argument in addition to self.
#
def test_interface(obj):
    # Does it have a 'show' attribute?
    if not hasattr(obj, 'show'):
        raise RuntimeError, 'obj must support method show'
    meth = getattr(obj, 'show')
    argnames = inspect.getargspec(meth)[0]
    # Does it have at least one argument (in addition to self)?
    if len(argnames) != 2:
        raise RuntimeError, 'method show must take two args (self + level)'
    obj.show(25)

def test():
    a = A()
    # This call is OK.
    test_interface(a)
    b = B()
    # This call generates an exception.  Class B does not
    #   support the required protocol.
    test_interface(b)

test()

Running the above code produces the following output:

(A.show) level: 25
Traceback (most recent call last):
  File "tmp.py", line 33, in ?
    test()
  File "tmp.py", line 31, in test
    test_interface(b)
  File "tmp.py", line 24, in test_interface
    raise RuntimeError, 'method show must take two args (self + level)'
RuntimeError: method show must take two args (self + level)

Notes:

• Module inspect is from the Python standard library.

Summary

A few words in summary:

• Interfaces are sometimes called "protocols" in the Python community.
• Interfaces are "a good thing". They make our code more understandable. They give the developer of a framework to enable a user of the framework to customize the behavior of the framework. We should use more of them.
• In Python, interfaces (also known as protocols) are about as informal as you want to make them.
• Python (as of this date) has no built-in support for interfaces. But, see Python enhancement proposal PEP 245 -- Python Interface Syntax and the Zope Interfaces Wiki Welcome to the Interfaces Wiki
• Since there is no language support in Python for interfaces, when you specify an interface that a user must follow, you are likely to want to do some error checking in the code that uses a class or instance that should support an interface. For example, you might want to check to determine that a class or instance implements a particular method. See section Error checking for interfaces above for an example.

The following articles provide additional reading on interfaces:

• Python Interfaces are not Java Interfaces
• Duck Typing
• PEP 245 -- Python Interface Syntax
• The Zope Interfaces Wiki at Welcome to the Interfaces Wiki

Script writers and maintainers

Prerequisites -- None, but familiarity with some other progamming lanuage is helpful, and familiarity with an object-oriented language is a plus.

Content:

• Programming in Python: (1) Introduction to Python; (2) Python lexical matters; (3) built-in data types; (4) statements; (5) functions; (6) modules; (7) classes and OOP.
• Using Java class libraries from Jython.

Extending Jython in Java.

Advantages and limitations of adding special Jython support to a Java class.

Features:

• Doc strings
• Emmulating Python/Jython built-in types.

Jython embedders

Prerequisites -- (1) Knowledge of Python; (2) knowledge of Java.

Content:

• Creating the interpreter.
• Evaluating scripts.
• Passing values between Java and Jython.
• Catching exceptions -- Trapping errors and exceptions thrown in Jython scripts.

Questions

This is one of the aspects of Python/Jython training that gives me the most difficulty. There are classes where I stop and ask for questions, and cannot get a response. I try probing. I try to start a question session by asking questions of the class. But, sometimes there is nothing. I believe that some class members believe that they are being polite by not putting me on the hot spot with a difficult question. But, actually, responding to questions is the part I like the most.

If pulling questions out of the class is the part that gives you difficulty, too, here are a few things you might try:

• Threaten that if you do not get questions, you will ask questions.
• Refuse to move on to the next topic until you get at least one question.
• After finishing a topic, ask the class, "Now, how would you use that?" What you are hoping to do is to initiate a discussion.

Agenda

Having an agenda or schedule is important. It establishes goals for the class, which will be of help and guidance for you, and also informs class members on what they can expect. It also gives you something to keep yourself on track.

Prepare a one to two page agenda. I usually break it up into a description of morning and afternoon sessions. You can find an example here: Jython Agenda. Note that this, like most of my documents, was generated from reStructuredText (reST) using Docutils, and you will find a link to the source document at the bottom of the page.

Teaching Python vs. Teaching Jython

Teaching Jython breaks down into two major areas:

1. Learning the Python language, because Jython is Python. That's the beauty of Jython.
2. Learning how Jython connects with Java, because Jython gives us such powerful and easy access to Java. That's also the beauty of Jython.

In a typical class, you are likely to face two categories of students:

1. Those who do not know Python and want to learn it so that they can write Python/Jython code.
2. Those who already know Python, and want to learn the Jython-specific Java part.

The difficulty is that the first type of student will be lost when you get to the Java part, and the second type of student will be bored while you do the Python part.

An ideal solution would be to teach two separate classes for the two types of students. But, you will often not have that option.

My own solution so far has been to try to make the Python sessions as lively as possible and to make the Jython/Java sessions as basic as possible. That's not the best of solutions, but it's the best I have been able to do so far. One possible tweak on this approach is to try to save the last afternoon of the training for practical exercises and individual projects. This allows each student to pick the area where s/he has an interest and feels a need to learn. In my last training, this approach was reasonably successful.

Practical Exercises

If you intend to teach Jython/Python, planning for practical exercises is almost as important as planning the teaching, explaining, and lecturing that you will do. Exercises are especially problematic in classes I teach because these classes are relatively short. How do you teach programming in Python and Jython's connections with Java and include practical exercises that are reasonably realistic all in a three or four day training schedule?

Homework? That might help stretch the available time. But, I don't think assigning work for after-hours is realistic.