==================== Patterns for Twisted ==================== :author: Dave Kuhlman :address: dkuhlman@rexx.com http://www.rexx.com/~dkuhlman :revision: 1.0c :date: Sept. 22, 2003 :copyright: Copyright (c) 2003 Dave Kuhlman. This documentation is under The Artistic License: http://www.opensource.org/licenses/artistic-license. :abstract: This document describes and discusses several design patterns for the development of Twisted Web applications. .. sectnum:: :depth: 4 .. contents:: Introduction ============ This document contains notes and examples that describe several programming patterns for Twisted, including patterns for deferred operations, for database access, for a client of a text search engine, and for an XML-RPC client. I've called these "design patterns", although they are also "implementation patterns" in an attempt to suggest that they are a bit more abstract. My intent is to encourage you to use these patterns both in the design of your Twisted Web application as well as in implementing it (possibly by copying and editing the examples providing). These patterns may seem to have a `REST (REpresentational State Transfer)`_ cast, and that is intentional. My aims are to help those attempting to treat the Web as a resource, in contrast to dealing with the Web as an extended computer using some style of RPC (e.g. XML-RPC or SOAP). However, I'm also hoping that the patterns described in this document are generic enough so that they will be helpful in other styles of Web application as well. .. _`REST (REpresentational State Transfer)`: http://internet.conveyor.com/RESTwiki/moin.cgi/RestFaq I've read the Twisted documentation on deferreds as closely as I could, but I do not feel that I am experienced enough to have good intuitions here, so any suggestions and corrections would be appreciated. Application Goals ================= Here are some goals for applications developed with the use of these patterns: - REST-ful applications. - Business logic through finite state machines (FSM). - Support for the development of clients. - Support for down-loadable clients and for clients that can be updated automatically from the server, e.g. by down-loading and installing scripts. - Delivery on top of Twisted. - Tailored support for several specific application architectures, for example: + Database search, listings, and update (add, modify, delete). + Text repository search, display, update. Use of a search engine, e.g. Glimpse. + Use of XML-RPC as a client to access back-end resources exposed by an XML-RPC server. + Multi-step, interactive, processes with complex logic implemented on top of a finite state machine (FSM). The Design Patterns =================== These are the patterns that we describe in this document. We hope to also provide some support for implementing with these patterns. We are interested in the following development patterns, which are described in more detail below: - The dispatcher pattern - The database update pattern - The FSM multi-step pattern - The text repository search and update pattern - The XML-RPC pattern The Dispatcher Pattern ====================== This pattern takes the URL and translates it into a call to the Python method intended to implement the request. It does this by inspecting the path on the URL. Example ------- The brute-force pattern ....................... Here is an example of a class that dispatches requests:: class ResourceDispatcher(resource.Resource): isLeaf = True def render(self, request): done = False if len(request.postpath) == 1 and request.postpath[0]: if request.postpath[0] == 'show_plants': dbaccess = dbmod.DBAccess(registry) dbaccess.show_plants(request) done = True elif request.postpath[0] == 'show_plants_by_rating': dbaccess = dbmod.DBAccess(registry) dbaccess.show_plants_by_rating(request) done = True elif request.postpath[0] == 'get_add_form': dbaccess = dbmod.DBAccess(registry) dbaccess.show_add_form(request) done = True elif request.postpath[0] == 'get_delete_form': dbaccess = dbmod.DBAccess(registry) dbaccess.show_delete_form(request) done = True elif request.postpath[0] == 'add_plant': dbaccess = dbmod.DBAccess(registry) dbaccess.add_plant(request) done = True elif request.postpath[0] == 'delete_plant': dbaccess = dbmod.DBAccess(registry) dbaccess.delete_plant(request) done = True if done: return NOT_DONE_YET else: content = Content_Dispatch % ( request.postpath, request.args, time.ctime(), ) request.write(content) request.finish() Comments and explanation: - Setting the class variable ``isLeaf`` to ``True`` tells Twisted not to perform further processing on the URL path. - The list ``request.postpath`` contains un-digested parts of the URL path. The name-prefix pattern ....................... For some purposes, this pattern is a bit simpler, although it may not give you as much control over individual cases:: class ResourceDispatcher(resource.Resource): isLeaf = True def render(self, request): if len(request.postpath) == 1 and request.postpath[0]: name = request.postpath[0] dbaccess = dbmod.DBAccess(registry) try: method = getattr(dbaccess, 'webrequest_' + name) except AttributeError: pass else: method(request) return NOT_DONE_YET content = Content_Dispatch % ( request.postpath, request.args, time.ctime(), ) request.write(content) request.finish() Explanation: - This dispatcher retrieves an attribute (the method) from a class given a request name on the right end of the URL. - If your implementation is spread across several modules and/or classes, then you might still have to do a bit of if-else stuff. Or, you might consider using a dictionary that maps names to modules and classes. The Database Update Pattern =========================== This pattern uses Twisted deferred objects to request database operations. Here is a description: - A database class with one method for each database operation. Note that if the "slow" resource is something else (XML-RPC, for example), then this would be a class for that resource. I'm guessing that various slow resources are different enough so that no single interface (API, set of public methods) could be defined, but that would be nice. - A resource class with one operation for each request or resource type. A dispatcher (a Twisted .rpy page) parses the request URL and dispatches/calls the appropriate method in this class. This class would also contain application logic. - For each step in the sequence, a (private) method in the resource class that performs that step by calling the appropriate method in the database class and (for all but the last step) by providing as the callback function, the method that implements the next step in the process. - My example code uses an PostgreSQL database, but other databases that have Python support could also be used. Strategy -------- - Separate the deferring operations (for example, requests to the PostgreSQL database) from the code that calls them and that also contains application logic. - Execute operations in sequence. Operation n+1 must not begin operation n until is complete. Implement this by chaining callbacks. - Provide simple Web page and form definitions. Enable editing and delivery of simple HTML documents. .. Data fill-in via Quixote template language or other template and string interpolation language. Example ------- The application logic class ........................... And, here is the resource class that requests the slow operations and provides a callback to be called when the slow operation is complete. This class also provides application logic. For each operation that requires multiple sequential operations, it provides multiple methods that each calls a database operation and provides a callback (except, of course, for the last step):: class DBAccess: def __init__(self, registry): self.registry = registry get_globals(registry) o o o # # Delete a row. # def delete_plant(self, request): self.request = request self.db = PlantDatabase("pyPgSQL.PgSQL", CONNECTION_ARGS) self.content = Content_ShowPlants self.plant_name = request.args.get('plant_name') if self.plant_name and len(self.plant_name) > 0: self.plant_name = self.plant_name[0] self.plant_desc = request.args.get('plant_desc') if self.plant_desc and len(self.plant_desc) > 0: self.plant_desc = self.plant_desc[0] self.plant_rating = request.args.get('plant_rating') if self.plant_rating and len(self.plant_rating) > 0: self.plant_rating = self.plant_rating[0] self.delete_plant_step_1() return NOT_DONE_YET def delete_plant_step_1(self): self.db.checkPlant(self.plant_name).addCallbacks( self.delete_plant_step_2, self.db.operationError) def delete_plant_step_2(self, resultlist): if resultlist: deferred = self.db.deletePlant( self.plant_name) deferred.addCallbacks(self.delete_plant_step_3, self.db.operationError) else: self.content = Content_Missing self.content = self.content % self.plant_name self.request.write(self.content) self.request.finish() def delete_plant_step_3(self, resultlist): self.db.getPlants().addCallbacks(self.gotPlants, self.db.operationError) # # Show the plants in the database. # def gotPlants(self, resultlist): plantlist = [] plantlist.append('') plantlist.append('') plantlist.append('') for plant in resultlist: plantlist.append('') s1 = '' % \ (plant[0], plant[1], plant[2]) plantlist.append(s1) plantlist.append('') plantlist.append('
Plant DB
NameDescriptionRating
%s%s%s
') plantstr = '\n'.join(plantlist) content = self.content % (self.request.args, plantstr) self.request.write(content) self.request.finish() The following second pattern uses a single method implemented as a state machine, rather than using separate methods for each step (as in the previous example). Whereas the previous example, deleted a plant, this example adds a plant if it is not already in the database, or updates the plant if it is in the database:: STEP_1 = 1 STEP_2 = 2 STEP_3 = 3 class DBAccess: o o o # # Add or update a row. # This uses a state machine pattern. # def add_plant(self, request): self.request = request self.state = STEP_1 self.add_plant_machine() def add_plant_machine(self, *args): if self.state == STEP_1: self.db = PlantDatabase("pyPgSQL.PgSQL", CONNECTION_ARGS) self.content = Content_ShowPlants self.plant_name = self.request.args.get('plant_name') if self.plant_name and len(self.plant_name) > 0: self.plant_name = self.plant_name[0] self.plant_desc = self.request.args.get('plant_desc') if self.plant_desc and len(self.plant_desc) > 0: self.plant_desc = self.plant_desc[0] self.plant_rating = self.request.args.get('plant_rating') if self.plant_rating and len(self.plant_rating) > 0: self.plant_rating = self.plant_rating[0] self.state = STEP_2 self.db.checkPlant(self.plant_name).addCallbacks( self.add_plant_machine, self.db.operationError) return NOT_DONE_YET elif self.state == STEP_2: resultlist = args[0] # Is the plant already in the database? if resultlist: self.state = STEP_3 deferred = self.db.updatePlant( self.plant_name, self.plant_desc, self.plant_rating) deferred.addCallbacks(self.add_plant_machine, self.db.operationError) else: self.state = STEP_3 deferred = self.db.addPlant( self.plant_name, self.plant_desc, self.plant_rating) deferred.addCallbacks(self.add_plant_machine, self.db.operationError) return NOT_DONE_YET elif self.state == STEP_3: self.db.getPlants().addCallbacks(self.gotPlants, self.db.operationError) return NOT_DONE_YET Explanation: - This state machine is sequential: it executes step 1, 2, and 3 in order. - Each state is implemented by a clause in the if-elif statement. - Each state sets the variable ``self.state`` to indicate the next state to be executed. - Each state sets this method itself as the next callback. Comparison and analysis: - The difference between the above two patterns is that the first uses separate methods for each step whereas the second pattern uses a single method with separate states (each implemented by a clause in an if-elif statement. - Once you learn to recognize each pattern and its structure, either is understandable and usable. The choice between them may be a personal preference. - A good recommendation is to pick one of the above two patterns and to use it consistently. What you want is code that is recognizable and clear, and that you can understand and debug. And, here is yet one more pattern for the same task. This one attempts to simplify things by replacing the ``if-elif-...`` statement. It implements each state as a separate method, and provides a dispatcher to these methods:: class DBAccess: o o o # # Add or update a row. # This method uses a state machine pattern. # It also has a dispatcher that retrieves and calls the # method that implements the current state. # def add_plant(self, request): self.request = request self.state = 'step_1' self.add_plant_machine() # # This method is the dispatcher. # def add_plant_machine(self, *args): method = getattr(self, 'add_plant_machine_' + self.state) method(*args) def add_plant_machine_step_1(self, *args): self.db = PlantDatabase("pyPgSQL.PgSQL", CONNECTION_ARGS) self.content = Content_ShowPlants self.plant_name = self.request.args.get('plant_name') if self.plant_name and len(self.plant_name) > 0: self.plant_name = self.plant_name[0] self.plant_desc = self.request.args.get('plant_desc') if self.plant_desc and len(self.plant_desc) > 0: self.plant_desc = self.plant_desc[0] self.plant_rating = self.request.args.get('plant_rating') if self.plant_rating and len(self.plant_rating) > 0: self.plant_rating = self.plant_rating[0] self.state = 'step_2' self.db.checkPlant(self.plant_name).addCallbacks( self.add_plant_machine, self.db.operationError) return NOT_DONE_YET def add_plant_machine_step_2(self, *args): resultlist = args[0] if resultlist: self.state = 'step_3' deferred = self.db.updatePlant( self.plant_name, self.plant_desc, self.plant_rating) deferred.addCallbacks(self.add_plant_machine, self.db.operationError) else: self.state = 'step_3' deferred = self.db.addPlant( self.plant_name, self.plant_desc, self.plant_rating) deferred.addCallbacks(self.add_plant_machine, self.db.operationError) return NOT_DONE_YET def add_plant_machine_step_3(self, *args): self.db.getPlants().addCallbacks(self.gotPlants, self.db.operationError) return NOT_DONE_YET Explanation: - Note the method ``add_plant_machine``. It acts as a dispatcher for our state machine. It takes the current state (a string: "step_1", "step_2, "step_3"), concatenates it to "add_plant_machine\_", then retrieves and calls the method by that name. - Each method that implements a state (except the last), sets ``self.state`` to the next state, and adds ``add_plant_machine`` as a callback. Repository class ................ Here is an example of the *repository access* class or database class -- Each method implements a single operation that may cause a delay, then returns a deferred:: class PlantDatabase(adbapi.ConnectionPool): """Update and retrieve from the Plant_DB database. """ def checkPlant(self, name): deferred = self.runQuery("select * from Plant_DB where p_name = '%s'" % name) return deferred def updatePlant(self, plant_name, plant_desc, plant_rating): sql = "update Plant_DB set p_desc='%s', p_rating='%s' where p_name='%s'" % \ (plant_desc, plant_rating, plant_name) deferred = self.runOperation(sql) return deferred def addPlant(self, plant_name, plant_desc, plant_rating): sql = "insert into Plant_DB values ('%s', '%s', '%s')" % \ (plant_name, plant_desc, plant_rating) deferred = self.runOperation(sql) return deferred def deletePlant(self, plant_name): sql = "delete from Plant_DB where p_name='%s'" % \ (plant_name,) deferred = self.runOperation(sql) return deferred def getPlants(self): sql = "SELECT * FROM Plant_DB order by p_name" deferred = self.runQuery(sql) return deferred def operationError(self, error): log.msg("%s Operation Failed: %s" % (reflect.qual(self.__class__), error)) log.err(error) Comments and explanation: - There is one function in this class for each type of request. In our example these are check for existence, update, add, delete, and get all. - Each of these operations is a "slow" operation. To prevent blocking, it requests the operation and immediately returns a deferred. (Note: We will learn how to implement a slow operation without blocking the in the text repository pattern. - By inheriting from ``twisted.enterprise.adbapi.ConnectionPool``, we get get the ability to run database queries that use the connection pool. In order to do so, we pass to the constructor for class ``PlantDatabase`` the name of the database API and the connection args. See the application logic class. Connection pooling .................. This section provides an example that shows both: 1. How to share values across sessions and 2. How to create and uses a database connection pool. ``ConnectionPool`` saves the time needed to create a database connection for each session. Here is code that shares a single instance of the database connection pool across all sessions:: class Globals: def __init__(self): self.dbpool = None def getDbpool(self): return self.dbpool def setDbpool(self, dbpool): self.dbpool = dbpool def get_globals(registry): globalContainer = registry.getComponent(globalvalues.Globals) if not globalContainer: globalContainer = globalvalues.Globals() dbpool = PlantDatabase("pyPgSQL.PgSQL", "localhost:5432:test:postgres:flicker") globalContainer.setDbpool(dbpool) registry.setComponent(globalvalues.Globals, globalContainer) else: dbpool = globalContainer.getDbpool() return dbpool Explanation: - Class ``Globals`` is our container for global values. In this example, the only value stored in it is the database connection pool. - The function ``get_globals`` attempts to get the existing, shared globals container. If the globals container exists, it returns the DB connection pool (an instance of ``PlantDatabase``) from that container. If it does not exist, it creates a container and a DB connection pool, puts the DB connection pool in the container, saves it, and returns the new DB connection pool. - In order to use the database connection pool, we create an instance of a class that subclasses ``twisted.enterprise.adbapi.ConnectionPool`` (in our example code this class is ``PlantDatabase``) passing to the constructor the database connection parameters. The query functions provided by that superclass know how to use the database connection pool. See the "Repository class" example. Questions and discussion ------------------------ Here are a few questions: - We've presented two patterns for using deferreds, specifically (1) chaining separate methods and (2) a state machine in a single method. Which is better? Is there an approved or commonly accepted Twisted pattern for using deferreds? Putting multiple steps in a single method seems to avoid some code clutter at the expense of code density. - I've separated the slow operations into a separate class. Is that a good idea? In my example, the operations are very simple, in part because I've kept the logic in the resource class. Perhaps, instead of putting them in a separate class, I should just code them in-line in the resource class in the state machine method. One reason for separate methods is to make them reusable. - The state-machine-pattern returns multiple times. It returns NOT_DONE_YET each time. Is that the correct thing to do? - Twisted already has an API that creates and returns deferreds for database operations. What if I need to wrap some other long lasting operation up in the same way? How do I do that? Answer: See `The Text Repository Pattern`_. The Text Repository Pattern =========================== Assumptions: - There is a text search engine, e.g. Glimpse. If the repository is composed of XML documents, then sgrep/pysgrep or XPath might be appropriate. - The documents form some kind of accessible collection. For example, they are all in a single disk directory or set of sub-directories. Requirements: - Searches may be slow, so we consider doing the following: + In C code, release the Python global interpreter lock. + Wrap the slow operations with a Twisted deferred. - Define a common API that can be wrapped around and can be implemented for a variety search engines. - Provide patterns and code samples that that use the API. - Provide access to the (underlying) search engine from Python. For example: + Provide a Python extension for a search engine, e.g. Glimpse or Harvest (which uses Glimpse). + Provide a server-client connection to a search engine, e.g. Glimpse or Harvest. Definitions: - In REST terms, the *resource* is the text repository. For example, a resource might be: + The search capability + The ability to retrieve the file in which a search string has been found. - Again, in REST terms, the *representation* is the content which the Web application formats and sends to the client. The representation could be HTML, XML, PDF, etc. For example, a representation might be: + A list of search results along with URI's for accessing each result + A file that contains the string or target of the previous search Here is a description of the pattern: - The search and update operations are collected in a separate class. Call it the *repository access* class. This class exposes the following methods: + *search()* -- Return a list of repository locations that satisfy **. + *fetch_one()* -- Return the text chunk from the repository pointed to by **. + *replace_one(, )* -- Replace the chunk at ** with **. - Application logic is in a separate class. Call it the *application logic* class. This *application logic* class is a client of and uses the *repository access* class. - Filtering of search results can be done in separate wrappers that are called by the search engine. The logic class can also do some filtering, of course, but for some purposes it might be preferable to remove unwanted earlier. - Formatting of the representation of the resource can be done in the *application logic* class, or can be separated from application logic by placing templates and formatting code in a separate *representation formatting* class or even a separate module. The implementation -- Here is a conceptual view of the implementation pattern: - The implementation is composed of the following layers: + The dispatcher -- ``dispatch.rpy/ResourceDispatcher`` + The access class -- Class ``GlimpseTextRepositoryAccess`` + The repository class -- Class ``glimpsetextmod.GlimpseTextRepository`` + The service class -- Class ``glimpsemod.GlimpseService`` + The Glimpse library -- Class ``glimpselib.Query`` + The representation formatting class -- ``glimpsetextmod.Formatter`` - The dispatcher -- Translates the URI path into a call to a method in the access class. - The access class -- Contains application logic. - The repository class -- Calls the service class to create and return a Twisted deferred. - The service class -- (1) Creates a Twisted deferred; (2) uses it schedule the callbacks that request the "slow" operations; (3) returns the deferred instance. - The Glimpse library -- Contains functions that make requests Glimpse. - The representation formatting class -- Formats the results of queries for delivery to the client. You may want to compare the structure of the text repository pattern with that of the database update pattern. I'm trying to re-work these patterns so that they are as isomorphic (structurally similar) as possible. Hopefully, that will make knowledge and understanding of one pattern transferable to the other. Example ------- Application logic class ....................... Here is an example of the *application logic* class:: # # Application logic class # class GlimpseTextRepositoryAccess: # # Produce a form to be used to search the text repository. # def show_search_form(self, request): self.request = request self.content = Content_SearchForm self.request.write(self.content) self.request.finish() # # Produce a list of the search results. # def show_search_results(self, request): self.request = request self.repository = GlimpseTextRepository() self.content = Content_ShowSearchResults self.querystr = request.args.get('querystr') if self.querystr and len(self.querystr) > 0: self.querystr = self.querystr[0] self.casesensitive = request.args.get('casesensitive') if self.casesensitive and len(self.casesensitive) > 0: self.casesensitive = self.casesensitive[0] self.state = STEP_1 self.show_search_results_machine() return NOT_DONE_YET def show_search_results_machine(self, *args, **kw): if self.state == STEP_1: #dbglogmsg('*** (show_search_results Step_1)') self.repository.search_repository( self.show_search_results_machine, self.querystr, self.casesensitive) self.state = STEP_2 elif self.state == STEP_2: resultlist = args[0] formatter = Formatter() content = formatter.format_search_results(self.content, resultlist) self.request.write(content) self.request.finish() # # Fetch and show one file from the repository. # def get_text_file(self, request, path): self.request = request infile = file(path, 'r') filecontent = infile.read() infile.close() formatter = Formatter() content = formatter.format_file(filecontent, path) self.request.write(content) self.request.finish() Explanation: - Method ``show_search_form`` provides a form that can be used to request a query. - Method ``show_search_results_machine`` implements a small FSM (finite state machine). This FSM contains two steps (states) which (1) get the deferred that starts the query and (2) format the results for delivery to the client. The variable ``self.state`` is used to keep track of state and to transfer from one state to another. - In step (state) 1, method ``show_search_results_machine`` (1) calls the ``search_repository`` method in the *repository* class, (2) adds itself as a callback, and (3) sets the (next) state to ``STEP_2`` so that when the callback is called, step 2 will be executed. - In step (state) 2, which is executed when the method is called as a *callback*, method ``show_search_results_machine`` formats the results and delivers them to the client. - Method ``get_text_file`` responds to a request for one of the files listed in the search results. The right-end of the URL contains the path to the requested file. Note that the list of search results contains the URLs needed to make these requests. In effect, this enables the user to "drill down" into one of the results (a file) returned by the text search engine. Repository class ................ Here is an example of the *repository access* class:: # # Repository class # class GlimpseTextRepository: def search_repository(self, callback, querystr, casesensitive): flags = '' if not casesensitive: flags = '-i' port = 2001 queryobj = glimpselib.Query() deferred = queryobj.query_server(querystr, flags, port) deferred.addCallback(callback) return deferred Explanation: - Method ``search_repository`` calls the method in the Glimpse to initiate the search and create a deferred object. It then adds a callback to the deferred. - This class is intended a "insulation" from the Glimpse library that actually does the work. It would, hopefully, enable user to easily switch to another implementation of text searching. The Glimpse library ................... This example from the *Glimpse library* directs its requests to the Glimpse text search engine:: # # This version uses glimpseserver. # Doing so saves loading the index into memory for each query. # It uses twisted.utils.getProcessOutput() to run the query, # captures the results, then creates a list of tuples to return # to the callback. # class Query: def query_server(self, querystr, flags='-i', port=2001): executable = '/usr/local/bin/glimpse' args = ['-C', '-K', '%d' % port] if flags: args.append(flags) if querystr: args.append(querystr) self.deferred = utils.getProcessOutput(executable, args) self.deferred.addCallbacks(self._query_server_2, self.error_back) return self.deferred def _query_server_2(self, result): lines = result.split('\n') result = [] for line in lines: pos = line.find(':') if pos > -1: val = (line[:pos], line[pos+2:]) result.append(val) return result def error_back(self, result): return [('***error***', result.getErrorMessage()),] Explanation: - The method ``query_server`` uses ``twisted.utils.getProcessOutput()`` to run the Glimpse search engine (as if through the command line) and capture the results. (Note to self: We really should wrap Glimpse as a Python extension module.) - The ``-C`` flag tells Glimpse to use the Glimpse server to perform the search. Doing so saves time that would have been used to load the search engine's index for each search. - ``getProcessOutput()`` returns a deferred. We add ``_query_server_2`` as a callback to that deferred. ``getProcessOutput()`` returns the output of the executable (in our case ``glimpse``) as a string to the callback. - The callback ``_query_server_2`` receives the results of the query, then creates and returns a list of tuples, one tuple for each match. Each tuple has (1) the file name of the match and (2) the line containing the match. Representation formatting class ............................... And, finally, when the results have been returned, the *formatting class* is used to produce a representation of the results. Putting this formatting code in a separate class has at least two benefits: - In a project of any size, there may be specialists working on the user interface, the formatting of representations, etc. Separating formatting code would provide a way to provide a separate set of modules for those specialists to work on. - It might be useful to do formatting with the Quixote PTL (Python templating language), which is also available in Twisted. By putting formatting code in a separate class, we would be able to put formatting code in ``.ptl`` modules (for example) for processing with PTL, then instructing Twisted to process them as resource templates with something like the following:: $ mktap web --path=/var/www \ --processor=.rtl=twisted.web.script.ResourceTemplate Our example, however, uses simple Python code to perform formatting. Here is an example of the *representation formatting* class:: # # Representation Formatting class # class Formatter: def format_search_results(self, pagecontent, resultlist): contentlist = [] contentlist.append('') contentlist.append('') idx = 0 prevfilename = '' for result in resultlist: filename = result[0] line = result[1].replace('<', '<') if filename != prevfilename: idx = 0 target = '%s' % \ (idx, filename, filename) linecontent = '' % \ (target, line) contentlist.append(linecontent) idx += 1 prevfilename = filename contentlist.append('
FileMatch
%s%s
') contentstr = '\n'.join(contentlist) contentstr = pagecontent % contentstr return contentstr def format_file(self, incontent, path): stem, ext = os.path.splitext(path) if ext == '.html': return incontent else: content1 = incontent.replace('<', '<') content2 = TextWrapper % (path, path, content1) return content2 Explanation: - Method ``format_search_results`` formats a list of search results, one for each hit. Note that it generates URLs that can be used to request the file in which the pattern is found. - Method ``format_file`` formats one file requested through a URL generated by ``format_search_results``. If the file contains HTML, the file's contents are returned unchanged. If the file does *not* contain HTML, then left-corner-brackets are replaced and a bit of HTML header and footer is wrapped around the content. Questions and discussion ------------------------ And, a few questions: - How slow does a slow method have to be before it is worth putting it into a deferred? After all, the purpose of using a search engine such as Glimpse is to speed up searches. Perhaps using deferred methods is over-kill. Is there any down-side or cost or over-head to using deferreds? The XML-RPC Pattern =================== This simple pattern can be used to make remote calls to an XML-RPC server and then to deliver content that contains the results. The Twisted Web application is acting as an XML-RPC client . The Web application uses XML-RPC to request information that it will return in its responses. Requirements: - Provide access to some resource through XML-RPC. The Twisted Web application acts as an XML-RPC client. - Deliver content that is formatted from the results of one or more XML-RPC calls. Definitions: - In REST terms, the back-end XML-RPC server is the *resource*. - And, the content that is formatted from the results of one or more XML-RPC requests is the *representation*. Example ------- The application logic class ........................... This class contains any application logic. A typical example would be the need to make more than one XML-RPC request, but to include logic between the requests so that one request is dependent on the results of a previous request. Here is an example:: # # XML-RPC applicationLogic class # class TemperatureAccess: # # Produce a form to be used to search the text repository. # def show_temperature_form(self, request): self.request = request self.content = Content_ConvertTemperatureForm self.request.write(self.content) self.request.finish() return NOT_DONE_YET # # Convert the temperature and produce the result. # def convert_temperature(self, request, direction): self.request = request intemp = None arg = request.args.get('intemp') if arg and len(arg) > 0: intemp = arg[0] self.intemp = intemp self.converter = Conversion() self.direction = direction if direction == 'c2f': self.content = Content_ConvertC2FResults deferred = self.converter.c2f(intemp) deferred.addCallbacks(self.receiveTemperature, self.receiveError) return NOT_DONE_YET elif direction == 'f2c': self.content = Content_ConvertF2CResults deferred = self.converter.f2c(intemp) deferred.addCallbacks(self.receiveTemperature, self.receiveError) return NOT_DONE_YET def receiveTemperature(self, value): valuestr = '%.2f' % value self.outtemp = valuestr if self.direction == 'c2f': scale = 'F' else: scale = 'C' deferred = self.converter.category(scale, value) deferred.addCallbacks(self.receiveCategory, self.receiveError) return NOT_DONE_YET def receiveCategory(self, category): self.category = category deferred = self.converter.ranges() deferred.addCallbacks(self.receiveRanges, self.receiveError) return NOT_DONE_YET def receiveRanges(self, ranges): self.ranges = ranges rangelist = ['
    '] for range in ranges: rangelist.append('
  • Less than %0.1f C is %s
    • ' % \ (range[0], range[1])) rangelist.append('
') rangestr = '\n'.join(rangelist) content = self.content % \ (self.intemp, self.outtemp, self.category, rangestr) self.request.write(content) self.request.finish() return NOT_DONE_YET def receiveError(self, error): raise RuntimeError, error Explanation: - Method ``show_temperature_form`` delivers a form that enables the client to provide a temperature and request a temperature conversion. - Method ``convert_temperature`` uses the XML-RPC class. It (1) creates and instance of the XML-RPC class, (2) calls the conversion method in that class, which returns a deferred, then (3) adds a callback function (``receiveTemperature``) to that deferred. - Method ``receiveTemperature`` is called when the first XML-RPC request has been completed. It also uses the XML-RPC class. It (1) saves the converted temperature, (2) calls the ``category`` method in the XML-RPC class, which returns a deferred, then (3) adds a callback function (``receiveCategory``) to this (new) deferred. - Method ``receiveCategory`` (1) receives the next result (the category) and (2) saves that value in an instance variable (``self.category``). This method also (3) calls the ``ranges`` method in the XML-RPC class, which returns a deferred, then (4) adds a callback function (``receiveRanges``) to this (new) deferred. - Method ``receiveRanges`` (1) receives the final result (the category), (2) formats the content, and (3) returns it to the client. - The methods described above show how to chain methods together when each method in a sequence is dependent on the results returned by the previous method. We could have alternatively used a single method (in place of ``receiveTemperature``, ``receiveCategory``, and ``receiveRanges``) which implemented a finite state machine with separate states (for ``receiveTemperature``, ``receiveCategory``, and ``receiveRanges``). - Method ``receiveError`` reports any errors raised by the XML-RPC server. The XML-RPC requester class ........................... This class makes the XML-RPC requests. It acts as the XML-RPC client. It uses Twisted's XML-RPC support, which returns a deferred object. Here is a sample class:: from twisted.web.xmlrpc import Proxy # # XML-RPC back-end resource class. # This class makes the XML-RPC requests and returns deferreds. # class Conversion: def c2f(self, intempstr): intemp = -1 try: intemp = float(intempstr) except: pass proxy = Proxy('http://localhost:8081') deferred = proxy.callRemote('c2f', intemp) return deferred def f2c(self, intempstr): intemp = -1 try: intemp = float(intempstr) except: pass proxy = Proxy('http://localhost:8081') deferred = proxy.callRemote('f2c', intemp) return deferred def category(self, scale, intemp): proxy = Proxy('http://localhost:8081') deferred = proxy.callRemote('temperature_category', scale, intemp) return deferred def ranges(self): proxy = Proxy('http://localhost:8081') deferred = proxy.callRemote('temperature_ranges') return deferred Explanation: - Each method in this class makes an XML-RPC request and returns a deferred. - These methods use Twisted's XML-RPC support (as opposed to using module ``xmlrpclib`` in the Python standard library, for example). The XML-RPC server .................. This is the simple XML-RPC server written in Python that I used while testing my examples:: import SimpleXMLRPCServer def c2f(ctemp): #ftemp = ((212.0 - 32.0)/100.0 * ctemp) + 32.0 ftemp = ((9.0 / 5.0) * ctemp) + 32 return ftemp def f2c(ftemp): #ctemp = 100.0/(212.0 - 32.0) * (ftemp - 32.0) ctemp = (5.0 / 9.0) * (ftemp - 32) return ctemp Ranges = [ (0.0, 'cold'), (20.0, 'cool'), (25.0, 'moderate'), (30.0, 'warm'), (35.0, 'hot'), (40.0, 'very hot'), (1000.0, 'extremely hot'), ] def temperature_category(scale, intemp): if scale.upper() == 'F': temp = f2c(intemp) else: temp = intemp category = 'Unknown' for range in Ranges: if temp <= range[0]: category = range[1] break return category def temperature_ranges(): return Ranges def startup(): server = SimpleXMLRPCServer.SimpleXMLRPCServer(("localhost", 8081)) server.register_function(c2f) server.register_function(f2c) server.register_function(temperature_category) server.register_function(temperature_ranges) server.serve_forever() Explanation: - The functions ``c2f``, ``f2c``, ``temperature_category``, and ``temperature_ranges`` are exported and are available to be called through XML-RPC requests. - Function ``startup`` creates a server, registers the exported functions, and starts the server. An XML-RPC server implemented with Twisted .......................................... And here is the same XML-RPC server implemented as a Twisted applications:: from twisted.web import xmlrpc, server Ranges = [ (0.0, 'cold'), (20.0, 'cool'), (25.0, 'moderate'), (30.0, 'warm'), (35.0, 'hot'), (40.0, 'very hot'), (1000.0, 'extremely hot'), ] class Converter(xmlrpc.XMLRPC): def xmlrpc_c2f(self, ctemp): #ftemp = ((212.0 - 32.0)/100.0 * ctemp) + 32.0 ftemp = ((9.0 / 5.0) * ctemp) + 32 return ftemp def xmlrpc_f2c(self, ftemp): #ctemp = 100.0/(212.0 - 32.0) * (ftemp - 32.0) ctemp = (5.0 / 9.0) * (ftemp - 32) return ctemp def xmlrpc_temperature_category(self, scale, intemp): if scale.upper() == 'F': temp = self.xmlrpc_f2c(intemp) else: temp = intemp category = 'Unknown' for range in Ranges: if temp <= range[0]: category = range[1] break return category def xmlrpc_temperature_ranges(self): return Ranges def main(): from twisted.internet.app import Application app = Application("xmlrpc") r = Converter() app.listenTCP(8081, server.Site(r)) return app application = main() if __name__ == '__main__': application.run(save=0) Explanation: - This example is basically a copy of the previous example with the functions encapsulated in a class. - To learn how to use Twisted to expose the methods in a class as XML-RPC services, read `Creating XML-RPC Servers and Clients with Twisted`_. I followed the instructions and example in that document. It's simple enough so that I won't add much. .. _`Creating XML-RPC Servers and Clients with Twisted`: http://www.twistedmatrix.com/documents/howto/xmlrpc - By default, methods prefixed with "xmlrpc\_" are exposed to clients as XML-RPC services. See ``twisted.web.xmlrpc.py``. Miscellaneous Hints and Howto's =============================== Here are a few clues and examples that were not covered in the patterns themselves. Body data/content ----------------- In order to extract the data from the body of a request, use ``request.content``. If there is content, then ``request.content`` will contain an instance of ``cStringIO``. Here is an example that shows how to use it:: data = '' if request.content: data = request.content.getvalue() Debugging --------- It is often helpful to produce some debugging information. This example shows how to print messages to a log file:: from twisted.python import log # Set to 0 (debugging off) or 1 (debugging on). _DEBUG = 0 def dbglogmsg(mesg): if _DEBUG: log.msg(mesg) l Sharing values across sessions ------------------------------ Twisted provides a mechanism for sharing values across sessions. In order to do this, define a class, store the values you wish to share in an instance of the class, then request the shared values by asking Twisted for the shared value of that data type. The `Connection pooling`_ example above shows how to share values across sessions. A test harness -------------- You may want to be able to create a testing harness that acts as a client. It can be used to send requests to your Twisted Web application and to display or process the responses. If expanded, it can serve as a simple client that drives your application. You can use the following example as a starting point for the implementation of a Python client that talks to your Twisted Web application. Here is the example:: import httplib, urllib # # Send a request to my Twisted Web application. # Add a few arguments to the request and pack some content into # the body of the request. # Get the response, display the status of the response, and # display the content received with the response. # def test(): # Create some sample content to send as the body of the request. datalist = [] datalist.append('') datalist.append('Here is stuff for a test.') datalist.append('') data = '\n'.join(datalist) # Create a few dummy parameters. paramdict = {'aaa': 111, 'bbb': 222} params = urllib.urlencode(paramdict) # Here are headers for the request. Note the "Content-length". headers = {"Content-type": "application/x-www-form-urlencoded", "Accept": "text/html", "Content-length": len(data), } conn = httplib.HTTPConnection("localhost:8080") conn.set_debuglevel(1) selector = '/dispatch.rpy/?%s' % params # Pick one: GET or POST? conn.putrequest("GET", selector) ## conn.putrequest("POST", selector) for key, val in headers.iteritems(): conn.putheader(key, val) conn.endheaders() conn.send(data) response = conn.getresponse() print 'status/reason', response.status, response.reason data1 = response.read() print '=' * 50 print data1 Explanation: - Each call to function ``test`` makes one request to the Web application. - Local variables ``datalist/data`` are used to create content to be sent in the *body* of the request. - Note the commented lines that would enable you to choose between making a ``GET`` or a ``POST`` request. - This example prints out the response from the Web application. However, your own client implementation could process the response in other ways. Summary and Conclusions ======================= This document has tried to help Twisted developers of Web applications in two ways: - Help beginning Twisted users a set of examples (patterns) which enable them to quickly construct Twisted Web applications. - Help experienced Twisted users by giving them a set of patterns (examples) that improve their code through consistency and clarity. Thanks in advance for suggestions and guidance. Acknowledgements ================ Thanks to Andrew Bennetts for many suggestions and criticisms on an earlier draft. Thanks to the developers and supporters of Twisted for a powerful product. See Also ======== `Twisted`_: The Twisted network framework at TwistedMatrix .. _`Twisted`: http://www.twistedmatrix.com/ `Python home`_: The Python Web site. .. _`Python home`: http://www.python.org RestFaq_: The REST FAQ -- Information and links for REpresentational State Transfer. .. _RestFaq: http://internet.conveyor.com/RESTwiki/moin.cgi/RestFaq Glimpse_: The Glimpse search engine Web site. .. _Glimpse: http://webglimpse.net/ `Docutils`_: Python Documentation Utilities -- This document was formatted with Docutils. .. _`Docutils`: http://docutils.sourceforge.net/