As the title says it, each presentation was filmed, and has just been uploaded for everyone’s pleasure! Here is the two-part video presentation of Twisted, done by Michael SCHERER.

Oh, by the way, please take a minute to vote for your favourite Internet Engine!

I found a nicely written article about the problems with Ruby, written by a Ruby user, and why he found Python to be really good. There’s even a quote about Twisted!

And, there are a bunch of things available to a Python guy that Ruby just can’t compete with that are of particular interest to me. Two that come to mind immediately are Twisted and Stackless Python. The former was used by others at TurnTide for creating a really powerful SMTP testing tool and the latter was used by TurnTide’s competitor IronPort to build one of the industry’s best MTAs.

I didn’t knew that IronPort was done in Python, even in Stackless Python!

Henrik Thostrup Jensen announced on the Twisted Users mailing list he made two extensions for the Twisted AMP protocol.

Currently there are two types: A dictionary and a list. The types of the element must be specified (key and value can be different in the dictionary), otherwise they are free form, i.e., the keys in the dictionary can have any name, and the list can be of any size. The types can be nested, e.g., you can create a list of list of strings. I use (or will) the latter to return a query result, for which I do not know the row size. This is currently impossible (AFAICT), in the otherwise excellent AMP protocol.

You can get them on http://www.cs.aau.dk/~htj/code/amptypes-0.1.tar.gz

For people that live under a rock, AMP is a new communication protocol for Twisted (added in Twisted 2.5) much lighter/simpler than PB. It is just a request/response protocol over a persistent connection.

Making and deploying a Twisted project as a service under Windows

1. What is the objective?

The goal is to implement a program in Python+Twisted (using PB for network access) under Windows XP or 2000+, that can be run before a user logs on, so it has to be a windows service, launched automatically, at boot. Another goal is to show some developement patterns in Twisted. You will find a lot of ‘theoretical’ patterns about how to make singletons/borgs, proxy, etc. stuffs, but I never found patterns about ‘Twisted code’, except for the wonderful ‘finger tutorial’ by squishy moshez. This tutorial can be split in two parts: The first one is about writing a good skeleton for your Twisted development. The second part is about making a Windows service.

Note that these instructions (the service building part) will not work under Windows 95 or 98 or below.
2. Required packages

On the Windows machine, we can separate the required stuff in two:

• The required stuff for generating the files: To do so, we need ActiveState Python 2.3+ (because it includes the latest version of the modules pywin32), Twisted 1.3+, and py2exe 0.5+, and eventually gvim for editing the files.
• The required stuff when we have generated all the required files: nothing, as we’ll have a compiled program which includes all the required DLLs.

3. Installation of the required packages

Nothing special here, just click, except that I needed (found necessary) to add the path to the scripts (twistd) in Python2.3 (installation path to Python and path to scripts directory).
4. Writing the program

In our program, we don’t need anything specific that will require the use of the specially made reactor for win32, in fact, there’s not really any reason to use it anyway. There’s just the case if you want to use the utils.getProcessValue() method of twisted which doen’t work if you use the default select recator. That’s not really a problem, as you can use the win32pipe.popen() from the pywin32 modules which works perfectly in Twisted (at least for me).

Here, our program is a simple one that just listens on port 1234 using PB, making the method getFoo() available to the outside. This method just returns the current time.
4.1. Simple, Fast and short code

Here is the simplest code, without using tricks to shorten the code making it unreadable (Myprog1.py):

Myprog1.py

How to run this code: Simply type python Myprog1.py.

The method requestAvatar of the class SimpleRealm is called whenever a user (a pb client here) has been authenticated. This method associates him to a class (here Presence) which contains the allowed remotely callable methods (identified by perspective_*). Here, there is just one method that can be called: getFoo. A PB factory is created to respond to requests from outside, and map successful requests to the SimpleRealm class.

To test this code, you can write a simple PB client that will ask for the getFoo method and get the result. I’ve done it for you in the file miniclient.py:

Toggle line numbers

1 # -*- coding: latin-1 -*-
2 from twisted.spread import pb
3 from twisted.internet import reactor,stdio,defer
4 from twisted.python.util import println
5 from twisted.cred import credentials
6 from twisted.python import log,logfile
7 from twisted.protocols import basic
8
9 import sys,time,os,ConfigParser
10
11
12 class Client(pb.Referenceable, pb.Broker):
13 def __init__(self):
14 pb.Broker.__init__(self, isClient=1)
15 self.presence = None
16
17 # Low LEVEL methods
18
19 def run(self):
20 self.connect()
21
22 def connect(self):
23 user = ‘foo’
24 password = ‘bar’
25 log.msg(‘Attempting to connect to server as user %s…’ % user)
26
27 self.factory = pb.PBClientFactory()
28 reactor.connectTCP(“localhost”, 1234, self.factory)
29 def1 = self.factory.login(credentials.UsernamePassword(user, password),
30 client=self)
31 def1.addCallbacks(callback=self.connected, errback=self.noLogin)
32 def1.addCallback(callback=self.action)
33 def1.addErrback(errback=self.genericError)
34
35 def genericError(self, ref) :
36 log.msg( “Generic error: %s.” % ref.getErrorMessage() )
37
38 def noLogin(self, reason):
39 print “Got rejected, will try in 5 seconds”, reason
40 reactor.callLater(5, self.connect)
41 self.factory.disconnect()
42 return defer.fail(reason)
43
44 def connected(self, perspective):
45 print “Connected, got perspective ref:”, perspective
46 self.presence = perspective
47 perspective.notifyOnDisconnect(self.server_disconnected)
48
49 def server_disconnected(self, ref):
50 “”" “”"
51 #print “Server has disconnected, trying to reconnect in 5 seconds”
52 #reactor.callLater(5, self.connect)
53
54 def shutdown(self):
55 reactor.stop()
56
57 # High LEVEL methods
58 def action(self,data):
59 “”" “”"
60 log.debug(“Executing action”)
61 df1 = self.presence.callRemote(‘getFromCommon’,'hello’)
62 df1.addCallback(lambda x: println(“>>>>>>>>> Result: %s” % x))
63 df1.addCallback(lambda x: reactor.stop())
64
65 if __name__ == ‘__main__’:
66 log.startLogging(sys.stdout, 0)
67 c = Client()
68 c.run()
69
70 reactor.run()

miniclient.py

If you have problems understanding deferred, you can read the corresponding HOWTO on http://twistedmatrix.com/projects/core/documentation/howto/defer.html.
4.2. Cleaner, more open, but more verbose code

For a cleaner, more ‘Production grade’ source, you can look at this one: Myprog2.py

Toggle line numbers

1 # -*- coding: latin-1 -*-
2
3 from twisted.internet import reactor,defer,app
4 from twisted.application import service,internet
5 from twisted.internet import defer
6 from twisted.spread import pb
7 from twisted.python import log,logfile
8 from twisted.cred import portal,checkers,credentials
9 from twisted.manhole.telnet import ShellFactory
10
11 from twisted.cred.portal import Portal,IRealm
12 from twisted.cred.checkers import InMemoryUsernamePasswordDatabaseDontUse
13 import time,sys
14
15 class SimpleRealm:
16 __implements__ = IRealm
17
18 def requestAvatar(self, avatarId, mind, *interfaces):
19 if pb.IPerspective in interfaces:
20 p = Presence()
21 return pb.IPerspective, p, lambda : None
22 else:
23 raise NotImplementedError(“no interface”)
24
25 class Presence(pb.Avatar):
26 def perspective_getFoo(self):
27 log.msg(‘getFoo is called’)
28 return time.time()
29
30 class Agent(service.Service,pb.Referenceable):
31 def __init__(self, *args, **kw):
32 self.realm = None
33
34 def getRealmResource(self):
35 self.realm = SimpleRealm()
36
37 checker = InMemoryUsernamePasswordDatabaseDontUse()
38 checker.addUser(“foo”, “bar”)
39
40 p = Portal(self.realm)
41 p.registerChecker(checker)
42 return pb.PBServerFactory(p)
43
44 if __name__ == ‘__main__’:
45 log.startLogging(sys.stdout, 0)
46
47 application = service.Application(‘Agent’)
48 serviceCollection = service.IServiceCollection(application)
49
50 pSvc = Agent(‘Agent’, application)
51 pSvc.setName(‘Agent’)
52 pSvc.setServiceParent(serviceCollection)
53
54 shellF = ShellFactory()
55 shellF.setService(pSvc)
56 shellF.username = ‘admin’
57 shellF.password = ‘test’
58 reactor.listenTCP(8789,shellF)
59
60 reactor.listenTCP(1234, pSvc.getRealmResource())
61 reactor.run()

Myprog2.py

How to run this code: Simply type python Myprog2.py.

The difference with the previous Myprog1.py_ program is that a new class Agent is created, a subclass of service.Service. The reason for doing this is multiple:

*

It allows us to have a “central class” where you can store your common
o

methods to all your eventual different Presences and other methods. You could store them in SimpleRealm (and you’d be right if your program is really small) but I don’t think it’s really clean. For me, the class related to realms is just supposed to contain classes related to user management.
*

If you want to use twistd, .tac and other fun Twisted stuff, you have to get an application
o

defined. The first child of this application would be an instance of the Agent class.

This code is much cleaner, but longer, but allows to continue developping without barriers like communication between services, etc. This version is launched just by running python and the .py file.
4.3. Code shaped for usage with twistd

In Twisted you have the possibility to launch a .py file with a wrapper, named twistd (note the absence of ‘e’). But to do this, you need to initialize things differently from the method shown in Myprog2.py. Here’s the code: Myprog3.py

Toggle line numbers

1 # -*- coding: latin-1 -*-
2
3 from twisted.internet import reactor,defer,app
4 from twisted.application import service,internet
5 from twisted.internet import defer
6 from twisted.spread import pb
7 from twisted.python import log,logfile
8 from twisted.cred import portal,checkers,credentials
9 from twisted.manhole.telnet import ShellFactory
10
11 from twisted.cred.portal import Portal,IRealm
12 from twisted.cred.checkers import InMemoryUsernamePasswordDatabaseDontUse
13 import time, sys
14
15 class SimpleRealm:
16 __implements__ = IRealm
17
18 def requestAvatar(self, avatarId, mind, *interfaces):
19 if pb.IPerspective in interfaces:
20 p = self.getPerspectiveNamed(avatarId)
21 p.attached(mind)
22 return (pb.IPerspective, p, lambda p=p, mind=mind: p.detached(mind))
23 else:
24 raise NotImplementedError(“no interface”)
25
26 def getPerspectiveNamed(self, name):
27 log.msg(‘getPerspective() called for perspective named: %s’ % name)
28 presence = OnePresence(perspectiveName=name)
29 presence.setRealm(self)
30 return presence
31
32
33 class Presence(pb.Avatar):
34 def __init__(self, perspectiveName, identityName=’Nobody’):
35 self.identity = perspectiveName
36 self.remote = None
37 self.realm = None
38
39 def setRealm(self, realm): self.realm = realm
40
41 def success(self, message): log.msg(‘Success: %s’ % message)
42 def failure(self, error): log.msg(“Failure: error received: %s” % error)
43
44 def attached(self, mind):
45 “”" Called when a client connects to this Presence “”"
46 log.msg(‘Attached: mind: %s’ % (mind))
47 self.remote = mind
48 return self
49
50 def detached(self, remote):
51 “”" Called when a client disconnects from his Presence “”"
52 log.msg(‘Detached: mind: %s’ % (remote))
53 self.remote = None
54
55 def perspective_test(self, text):
56 log.msg(‘Presence received: %s’ % text)
57 if self.remote:
58 return text
59 else:
60 print ‘no remote?!’
61 return None
62
63
64 def OnePresence(Presence):
65 def perspective_getFoo(self):
66 log.msg(‘getFoo is called’)
67 return time.time()
68
69 def perspective_getFromCommon(self, data):
70 pass
71
72
73 class Agent(service.Service, pb.Referenceable):
74 def __init__(self, *args, **kw):
75 self.realm = None
76
77 def getRealmResource(self):
78 self.realm = SimpleRealm()
79
80 checker = InMemoryUsernamePasswordDatabaseDontUse()
81 checker.addUser(“foo”, “bar”)
82
83 p = Portal(self.realm)
84 p.registerChecker(checker)
85 return pb.PBServerFactory(p)
86
87 ###############################################
88 application = service.Application(‘Agent’)
89 serviceCollection = service.IServiceCollection(application)
90
91 pSvc = Agent(‘Agent’, application)
92 pSvc.setName(‘Agent’)
93 pSvc.setServiceParent(serviceCollection)
94
95 shellF = ShellFactory()
96 shellF.setService(pSvc)
97 shellF.username = ‘admin’
98 shellF.password = ‘test’
99
100 shell = internet.TCPServer(8789,shellF)
101 shell.setName(’shell’)
102 shell.setServiceParent(serviceCollection)
103
104 tcpServer = internet.TCPServer(1234, pSvc.getRealmResource())
105 tcpServer.setServiceParent(serviceCollection)

Myprog3.py

How to run this code: use twistd -noy Myprog3.py.

Here, what happens is that twistd looks for an object named application and uses it. So you ABSOLUTELY need to have that object named application. There are some modifications compared to Myprog2.py, where SimpleRealm is more verbose and allows to create different presences.
4.4. Dispatching code in different files/modules

The code is a little longer, making it more difficult to read. So I made another version, where code is split in two files: Myprog4.py and Myprog4Presences.py

Toggle line numbers

1 # -*- coding: latin-1 -*-
2
3 from twisted.internet import reactor,defer,app
4 from twisted.application import service,internet
5 from twisted.internet import defer
6 from twisted.spread import pb
7 from twisted.python import log,logfile
8 from twisted.cred import portal,checkers,credentials
9 from twisted.manhole.telnet import ShellFactory
10
11 from twisted.cred.portal import Portal,IRealm
12 from twisted.cred.checkers import InMemoryUsernamePasswordDatabaseDontUse
13 import time, sys
14
15 from Myprog4Presences import OnePresence
16
17 class SimpleRealm:
18 __implements__ = IRealm
19
20 def __init__(self, refServer):
21 self.refServer = refServer
22
23 def requestAvatar(self, avatarId, mind, *interfaces):
24 if pb.IPerspective in interfaces:
25 p = self.getPerspectiveNamed(avatarId)
26 p.attached(mind)
27 return (pb.IPerspective, p, lambda p=p, mind=mind: p.detached(mind))
28 else:
29 raise NotImplementedError(“no interface”)
30
31 def getPerspectiveNamed(self, name):
32 log.msg(‘getPerspective() called for perspective named: %s’ % name)
33 presence = OnePresence(perspectiveName=name)
34 presence.setRealm(self)
35 return presence
36
37 class Agent(service.Service, pb.Referenceable):
38 def __init__(self, *args, **kw):
39 self.realm = None
40
41 def getRealmResource(self):
42 self.realm = SimpleRealm(refServer=self)
43
44 checker = InMemoryUsernamePasswordDatabaseDontUse()
45 checker.addUser(“foo”, “bar”)
46
47 p = Portal(self.realm)
48 p.registerChecker(checker)
49 return pb.PBServerFactory(p)
50
51 ## Specific code ##########################
52 def doSomething(self, data):
53 return “Hello %s” % data
54
55
56 ###############################################
57 application = service.Application(‘Agent’)
58 serviceCollection = service.IServiceCollection(application)
59
60 pSvc = Agent(‘Agent’, application)
61 pSvc.setName(‘Agent’)
62 pSvc.setServiceParent(serviceCollection)
63
64 # Manhole installation for DEBUGGING only
65 # Faire “import __main__” puis “service”, “service.parent” pour voir ce qu’il y a.
66 shellF = ShellFactory()
67 shellF.setService(pSvc)
68 shellF.username = ‘admin’
69 shellF.password = ‘test’
70
71 shell = internet.TCPServer(8789,shellF)
72 shell.setName(’shell’)
73 shell.setServiceParent(serviceCollection)
74
75 tcpServer = internet.TCPServer(1234, pSvc.getRealmResource())
76 tcpServer.setServiceParent(serviceCollection)

Myprog4.py

And the file Myprog4Presences.py:

Toggle line numbers

1 # -*- coding: latin-1 -*-
2 from twisted.internet import reactor, defer
3 from twisted.spread import pb
4 from twisted.python import log
5
6 import time
7
8 class Presence(pb.Avatar):
9 def __init__(self, perspectiveName, identityName=’Nobody’):
10 self.identity = perspectiveName
11 self.remote = None
12 self.realm = None
13
14 def setRealm(self, realm): self.realm = realm
15
16 def success(self, message): log.msg(‘Success: %s’ % message)
17 def failure(self, error): log.msg(“Failure: error received: %s” % error)
18
19 def attached(self, mind):
20 “”" Called when a client connects to this Presence “”"
21 log.msg(‘Attached: mind: %s’ % (mind))
22 self.remote = mind
23 return self
24
25 def detached(self, remote):
26 “”" Called when a client disconnects from his Presence “”"
27 log.msg(‘Detached: mind: %s’ % (remote))
28 self.remote = None
29
30 def perspective_test(self, text):
31 log.msg(‘Presence received: %s’ % text)
32 if self.remote:
33 return text
34 else:
35 print ‘no remote?!’
36 return None
37
38
39 class OnePresence(Presence):
40 def perspective_getFoo(self):
41 log.msg(‘getFoo is called’)
42 return time.time()
43
44 def perspective_getFromCommon(self, data):
45 return self.realm.refServer.doSomething(data)

Myprog4Presences.py

How to run this code: use twistd -noy Myprog4.py.

Besides splitting the classes in different files, I added an example of perspective (perspective_getFromCommon) that goes to the “main application” (in fact the root of the tree where you have the app, then the simulation realm, and then the presences) to execute a ‘common’ method. Note: This code uses the ex-manhole stuff (now ShellFactory) to allow connection to the program while running. I’ll make a small tutorial on this later.
4.5. Using .tac file to finish the skeleton

Finally, we’re at the last step of achieving our ‘perfect pattern program’ in Twisted. The last thing to do, is to create a .tac file. Why? Well, it’s not useful, but it’s a requirement for building a service under Windows. A .tac is really simple: it’s just the ‘main’ part of the program put in a separate file with an extension ‘.tac’: Myprog5.py, Myprog5Presences.py and Myprog5.tac

from twisted.internet import reactor
from twisted.application import service,internet
from twisted.manhole.telnet import ShellFactory

from Myprog5 import Agent

application = service.Application(‘Agent’)
serviceCollection = service.IServiceCollection(application)

pSvc = Agent(‘Agent’, application)
pSvc.setName(‘Agent’)
pSvc.setServiceParent(serviceCollection)

# Manhole installation for DEBUGGING only
# Faire “import __main__” puis “service”, “service.parent” pour voir ce qu’il y a.
shellF = ShellFactory()
shellF.setService(pSvc)
shellF.username = ‘admin’
shellF.password = ‘test’

shell = internet.TCPServer(8789,shellF)
shell.setName(’shell’)
shell.setServiceParent(serviceCollection)

Myprog5.tac

How to run this code: use twistd -noy Myprog5.tac.
4.6. Final note and update

As you surely know, Twisted is a fast moving target, it seems it is now the usage not to use internet.TCP/UDP* anymore, but the method strports.service(), which will surely allows in the future to pass the information on the twistd command-line.

As an example, now you just need to replace the following code:
Toggle line numbers

1 tcpServer = internet.TCPServer(1234, pSvc.getRealmResource())
2 tcpServer.setServiceParent(serviceCollection)

With:
Toggle line numbers

1 from twisted.application import strports
2 [...]
3 tcpServer = strports.service(‘tcp:1234′,pSvc.getRealmResource())
4 tcpServer.setServiceParent(serviceCollection)

5. Making a service of the program

This section is now specific to Windows.

How to create a service, having a python program? The only way I know is to use the py2exe_ program whose initial goal is to find all the dependencies (modules and libraries), make an archive of it, and create an executable of the main program. The goal is to be able to easily distribute some python program without having requirements like ‘python must be installed, wxPython and wax modules have to be present on version >x.y’. Another advantage of py2exe is to allow the developer to choose how your program is compiled. It can be of three formats: console, window, com_server or service. For testing purposes, it is best to use ‘console’ mode, which, when your program will be launched, will open a Windows shell and run your program inside, making it easy to see any exception or other error.

To use py2exe, we’ll have to create a setup.py file because py2exe uses the nice distutils module which should always be installed with python by default on ANY distribution.

We’ll first try to compile our program as a console app. Normally, for a normal python program the setup.py file would be as simple as running python setup.py build:
Toggle line numbers

1 from distutils.core import setup
2 import py2exe
3
4 setup(console=["myscript.py"])

When this works correctly, you can continue (the hard part) to transform your program into a Windows service. Now that your program runs in a console, we’ll make it a service. What are the advantages of making of your program a service? Simple, you can launch your service before anyone logs on, at the machine boot, that means you’re not attached to the presence of a user logged on. That allows you to secure a little more you application by setting the owner/runner of the service a least privileged user. Another reason is that you can easily start/stop/restart your service graphically by clicking in the Windows Service Manager.

For a normal python program that’s not really a problem, here’s the corresponding setup.py file:
Toggle line numbers

1 from distutils.core import setup
2 import py2exe
3
4 setup(service=["MyService"])

Here, you’ll need to have an attribute _svc_name_ in a module named MyService.

OK. That’s easy, you say, and you’re right. The only problem is that **this doesn’t work if you’re using Twisted**. The main reason is that that class containing the _svc_name_ has to be a loop, where you have a start, a stop, etc. That means you’ll be stuck running this loop, and not the reactor :-(

Moreover, we want to be able to run our program which is in a .tac, and with twistd. So we’ll have to use a supplementary module, written by Moonfallen, available in his svn sandbox (svn co svn://svn.twistedmatrix.com/svn/Twisted/sandbox/moonfallen). You can view it using the web interface on http://svn.twistedmatrix.com/cvs/sandbox/moonfallen/ . You’ll get the following files::

lstep@nefesh moonfallen $ ll
total 92
drwxr-xr-x 4 lstep users 280 Jun 20 17:49 ./
drwxr-xr-x 4 lstep users 112 Jun 20 17:49 ../
drwxr-xr-x 7 lstep users 296 Jun 20 17:49 .svn/
-rw-r–r– 1 lstep users 3163 Jun 20 17:49 README.txt
-rwxr-xr-x 1 lstep users 772 Jun 20 17:49 TODO.txt*
-rw-r–r– 1 lstep users 49833 Jun 20 17:49 modulegraph.zip
drwxr-xr-x 3 lstep users 200 Jun 20 17:49 ntsvc/
-rwxr-xr-x 1 lstep users 766 Jun 20 17:49 pysvc.ico*
-rw-r–r– 1 lstep users 799 Jun 20 17:49 serviceinfo.ini
-rw-r–r– 1 lstep users 21751 Jun 20 17:49 tpusage.py

Install them in the directory where you put your setup.py file. and unzip modulegraph.zip in your site-packages directory (follow the instructions in the README).
6. Bug in Py2exe 0.5.4

If you just installed Py2exe without patching it, you will certainly end-up with an error when you’ll try to run your program (a can’t find module linecache or os). As written in moonfallen’s README, you have to patch the boot_service.py file which should be in C:\Python24\site-packages\py2exe::

— boot_service.py 2004-11-03 11:02:31.147398500 -0800
+++ boot_service.py.new 2004-11-03 11:02:19.522844900 -0800
@@ -9,7 +9,11 @@
service_klasses = []
try:
for name in service_module_names:
mod = __import__(name)
+ # Use the documented fact that when a fromlist is present,
+ # __import__ returns the innermost module in “name”.
+ # This makes it possible to have a dotted name work the
+ # way you would expect.
+ mod = __import__(name, globals(), locals(), ["DUMMY"])
for ob in mod.__dict__.values():
if hasattr(ob, “_svc_name_”):
service_klasses.append(ob)

7. How to write the setup.py file

Modify the setup.py to use ntsvc:
Toggle line numbers

1 from distutils.core import setup
2 import py2exe
3 import os,sys,glob
4 import ntsvc
5
6 data_files = [
7 ('', ['afile.conf']),
8 ]
9
10 setup(appconfig=’Myprog5.tac’, options =
11 {‘twistedservice’:
12 {‘dll_excludes’: ['tk84.dll','tcl84.dll'],
13 },
14 ‘py2exe’: {‘optimize’:2,},
15 },
16 data_files = data_files
17 )

And then run python setup.py twistedservice. You should have an output that looks like this:

running twistedservice
I: finding modules imported by agent.tac
*** searching for required modules ***
*** parsing results ***
creating python loader for extension ‘py2exe.py2exe_util’
creating python loader for extension ’servicemanager’
creating python loader for extension ‘win32pipe’
creating python loader for extension ‘_tkinter’
creating python loader for extension ‘win32service’
creating python loader for extension ‘win32api’
[...]
creating python loader for extension ‘perfmon’
*** finding dlls needed ***
*** create binaries ***
*** byte compile python files ***
skipping byte-compilation of C:\Python24\lib\ConfigParser.py to ConfigParser.pyc
skipping byte-compilation of C:\Python24\lib\Queue.py to Queue.pyc
[...]
*** copy extensions ***
*** copy dlls ***
*** copy data files ***
setting sys.winver for ‘C:\TAL\agent-windows\dist\python24.dll’ to ‘py2exe’
copying C:\Python24\lib\site-packages\py2exe\run.exe -> C:\TAL\agent-windows\dist\agentctl.exe
The following modules appear to be missing
['Crypto.Cipher', 'FCNTL', 'OpenSSL', '_ssl', 'hexdump', 'resource']

The missing libraries at the end of the generation is not a problem if you don’t use them (they’re mostly crypto related).
8. What you get

What python setup.py twistedservice does is:

*

Create a directory named dist (and build, but less important)
*

Analyzes all the dependencies of your python code
*

Copy all the dependent modules and dependent libraries to that dist directory
*

Generate a library.zip with all the libraries.
*

In our case (because we chose to use ’service’ in the setup.py file), we got an executable with the name of our .tac with ctl appended to it.

Here’s an example of the content of the dist directory:

Le volume dans le lecteur C n’a pas de nom.

Repertoire de C:\TAL\AGENT\dist

24/06/2005 11:25 .
24/06/2005 11:25 ..
07/06/2005 12:46 295 Agent.conf
07/06/2005 16:24 2461 agent.tac
24/06/2005 11:25 13312 agentctl.exe
24/06/2005 11:25 4053167 library.zip
07/06/2005 15:23 107 ntsvc.cfg
30/03/2005 09:51 12288 perfmon.pyd
22/10/2004 20:00 9728 py2exe_util.pyd
30/03/2005 09:51 135168 pyexpat.pyd
30/03/2005 09:51 1867776 python24.dll
30/03/2005 09:51 311296 pythoncom24.dll
30/03/2005 09:51 98304 pywintypes24.dll
30/03/2005 09:51 8192 select.pyd
30/03/2005 09:51 23552 servicemanager.pyd
24/06/2005 11:23 tcl
30/03/2005 09:51 405504 unicodedata.pyd
30/03/2005 09:51 4608 w9xpopen.exe
30/03/2005 09:51 69632 win32api.pyd
30/03/2005 09:51 15872 win32pipe.pyd
30/03/2005 09:51 26624 win32process.pyd
30/03/2005 09:51 29696 win32service.pyd
30/03/2005 09:51 651264 win32ui.pyd
30/03/2005 09:51 65536 zlib.pyd
30/03/2005 09:51 49152 _socket.pyd
30/03/2005 09:51 40960 _tkinter.pyd
27/05/2005 14:07 11264 _zope_interface_coptimizations.pyd

For example, if our .tac file was agent.tac, we would end up with an executable named agentctl.exe. if we run it, we see that it’s a “control program” to add/remove our newly made service:

Services are supposed to be run by the system after they have been installed.
These command line options are available for (de)installation:
-help
-install
-remove
-auto
-disabled
-interactive
-user:
-password:

Connecting to the Service Control Manager

If you run agentctl.exe -install, and then look in the Windows Service GUI, you should see a new line:

[attachment:service.png]

You’ll just have to right-click on it and click on Start Service.
9. Add-on: how to make an installation file

OK, now you have a directory that you can distribute, without having to require Python, Twisted and other dependencies. But it’s a directory. You can zip it, but it’s not very efficient, especially if you have to be able to remove it later, or deploy it on several machines.

What I recommend is to use the new installation format, used by the official Python package too, that is, MSI.
9.1. Generating an MSI package

Generating an MSI file is really easy, especially if you’re using Windows XP, as the kit is already included, otherwise you’ll have to download it from Microsoft’s website.

To summarize: to generate an MSI you’ll have to create **an xml file** where you have quite a lot of variables (program path, name of program, files to copy, etc.). This is quite long, and, without a GUI that does it for you, it’s nearly impossible. That’s why I recommend you Installer2Go_. It’s a really nice freeware (there’s a registered version if you want support) that allows you to build your .msi all by just clicking:

[attachment:i2g2.png]

You can even define the automatic registration and the removal of you service:

[attachment:i2g1.png]

That’s all for now. It’s the first release of this document, so there may be some typo errors. I checked all the Myprog*.py, they all work on my machine (Python2.3 or Python2.4, Twisted 1.3 or Twisted 2.0.1). Don’t hesitate to comment/correct/update by writing an email to luc stepniewski adelux.fr
10. References

*

http://starship.python.net/crew/theller/moin.cgi/Py2Exe

*

http://svn.twistedmatrix.com/cvs/sandbox/moonfallen/

*

http://twistedmatrix.com/~moonfallen/

*

http://svn.twistedmatrix.com/cvs/sandbox/moonfallen/README.txt?view=auto

*

http://www.installsite.org/pages/en/msi/authoring.htm

*

http://www.dev4pc.com/installer2go.html

*

http://www.qwerty-msi.com/

*

http://www.aksdb.org/downloads.php

*

http://msi2xml.sourceforge.net/

*

http://desktopengineer.com/index.php?topic=0070MSI

*

py2exe
*

Installer2Go

Twisted Weekly News #11
Author: Lior Gradstein
Contact: lior@gradstein.info
Revision: 1.0
Date: 2004/2/17 14:13:20
Copyright: © The Twisted Community

Contents

• The Twisted Weekly News
• Discussions
• Articles/Sites related to Twisted
• Cool Things of the Week
• New/Updated Cool Software
• Current releases

Continue reading “Twisted Weekly News #11” »

Twisted Weekly News #9
Author: Lior Gradstein
Contact: lior@gradstein.info
Revision: 1.0
Date: 2003-12-14
Copyright: © The Twisted Community

Contents

* The Twisted Weekly News
* Discussions
* Progress
* Cool Thing of the Week
* Current releases

If you have any new information, ideas, announcements for the next TWN, please write to me, . TWN is also available as HTML formated ReST on http://gradstein.info/articles/twisted_news/twisted_news/twn9
The Twisted Weekly News

9th Issue. Sunday, December 14, 2003

QOTW:

dash: being squishy is pleasant for everyone always
moshez: i demand proof
dash: it is an AXIOM!!
dash: squishing is part of the basic axiom set
like DeCartes said “I squish therefore I am”

Discussions

* http://twistedmatrix.com/pipermail/twisted-python/2003-December/006692.html RodrigoB suggested to use Twisted for the refactoring of The Circle, a P2P app.
* http://twistedmatrix.com/pipermail/twisted-python/2003-December/006705.html Last week, someone asked for an example of a Jabber client using Twisted libs, darryl posted a reply this week. darryl posted another example using PEAK too (http://randomthoughts.vandorp.ca/syncato/WK/blog/671?t=projects)
* http://twistedmatrix.com/pipermail/twisted-python/2003-December/006723.html Cinober explains his discoveries on debugging Twisted code within Emacs. Itamar gave a more generic explanation on how to debug general python code (http://twistedmatrix.com/pipermail/twisted-python/2003-December/006724.html)
* http://twistedmatrix.com/pipermail/twisted-python/2003-December/006745.html Richard Livsey asked why his IRC bot gets disconnected without any possibility of reconnection after sending too much data to the server. Timothee Besset replied he should read the IRC RFCs, that explains that his IRC bot was banned (flooding protection) and that he should use a rate limiting algorithm which doesn’t yet exist in Twisted IRCClient implementation. Andrew Bennetts answered that, actually, there is a rate limiting method, but it is more a delay than a rate limitation (http://twistedmatrix.com/pipermail/twisted-python/2003-December/006749.html) which solved Richard’s problem.

Progress

* Twisted 1.1.1: Itamar announced Twisted 1.1.1 as the official new stable release (http://twistedmatrix.com/pipermail/twisted-python/2003-December/006731.html). The changes are mainly bugfixes and minor improvements. MSN8P support in the MSN protocol has been updated and Conch SSH agent client now supports SSH auth agent.
* Quotient 0.8.8: A new version (0.8.8) of Quotient (http://www.divmod.org/) has been released. The pace of releases is getting faster each week. In this release (except from bugfixes), the database has been simplified and throttling tools have been added. To show that Quotient can be used as a framework not just for email, a project planning tool has been started (Anvil).

Cool Thing of the Week

* Twisted-web split A new mailing list (http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web and stored on NNTP gmane.org) as well as a new IRC channel (irc://irc.freenode.net/#twisted.web) who focus on twisted.web and woven/nevow part of Twisted.

Current releases

* Twisted: 1.1.1 (released 2003-12-10)
* Quotient: 0.8.8 (released 2003-12-12)
* Twisted Java: 0.6.1 (released 2003-12-02)

Ok, that’s it for this week. Thanks for reading!

Lior Gradstein

Twisted Weekly News #10
Author: Lior Gradstein
Contact: lior@gradstein.info
Revision: 1.0
Date: 2003-11-23
Copyright: © The Twisted Community

Contents

* The Twisted Weekly News
* Discussions
* Cool Thing of the Week
* Current releases

There wasn’t too much mail this week, so I merged the previous TWN with the current, as I was ill last week.

If you have any new information, ideas, announcements for the next TWN, please write to me, . TWN is also available as HTML formated ReST on http://gradstein.info/articles/twisted_news/twisted_news/twn10
The Twisted Weekly News

9&10th Issue. Sunday, December 28, 2003

QOTW::

http://twistedmatrix.com/pipermail/twisted-python/2003-December/006789.html From a mail by Alan McIntyre: “I just wanted to thank the Twisted developers for making such a cool piece of software. [...] Thanks to Twisted and Python, I was able to start from scratch and put together a server to present some of this data via XML-RPC in about 2 or 3 hours (and that includes looking up a lot of stuff in the documentation).”

Discussions

* http://twistedmatrix.com/pipermail/twisted-python/2003-December/006786.html Jonathan Simms asked for help on his new (new) ftp server code which is currently in sandbox. He asks for help to resolve some problems with authentication on the Windows side.
* http://twistedmatrix.com/pipermail/twisted-python/2003-December/006828.html More and more people are becoming interested in integrating Twisted stuff into their projects. Last week, RodrigB asked about The Circle. This week, Valentino Volonghi asks about the possibility (wish) to replace the pygame event loop with a Twisted reactor.
* http://twistedmatrix.com/pipermail/twisted-python/2003-December/006820.html Michal Pasternak made some tests on Twisted, and noticed some performance degradation after 40 simultaneous connections. Itamar replied there wasn’t anything hardcoded that would change Twisted behaviour above 40 simultaneous connections.

Cool Thing of the Week

* Treasure of code examples and tutorials for nevow (http://www.livejournal.com/users/fzzzy/) fzZzy updated his new blog on LiveJournal with a lot of code samples he wrote for people who asked questions on irc. The second lecture he gave on irc is also available (http://soundfarmer.com/content/logs/NevowLecture2-121503.txt). With these docs and the previous more “in-depth” documentation from justinj (http://stewstuff.com/doc/nevow.xhtml) you do not any excuse for not learning nevow!
* SFTP support in Zope (using Twisted!) (http://twistedmatrix.com/pipermail/twisted-python/2003-December/006839.html) Fred L. Drake, Jr. wrote some support (http://cvs.zope.org/Packages/SFTPGateway/)for the sftp protocol to be included in Zope. The result may be available as a separate product in a near future.
* Impacket (http://oss.corest.com/projects/impacket.html) Not related to Twisted, but interesting anyway.Itamar recommended us to take a look at this python library which provides low-level programmatic access to network packets (tcp, udp, etc.). There is also a python pcap implementation on their web site.
* Reminder about the web forum (http://twisted.banquise.org/) May I remind you that a web interface exists for those that prefer reading informations online? We’re barely 8 now, but nobody posts anything which can be discussed :-(

Current releases

* Twisted: 1.1.1 (released 2003-12-10)
* Quotient: 0.8.8 (released 2003-12-12)
* Twisted Java: 0.6.1

Ok, that’s it for this week. Thanks for reading!

Lior Gradstein

Twisted Weekly News #8
Author: Lior Gradstein
Contact: lior@gradstein.info
Revision: 1.0
Date: 2003-11-23
Copyright: © The Twisted Community

Contents

* The Twisted Weekly News
* Discussions
* Progress
* Cool Thing of the Week

If you have any new information, ideas, announcements for the next TWN, please write to me, .
The Twisted Weekly News

8th Issue. Sunday, December 7, 2003

QOTW:

dash: ok I wrote something
dash: do you want to review and comment?
moshez: i would but my mouse battery is dead!! so i cannot paste it
moshez: yes, but today is a traditional american holiday
on which we make lame excuses

Discussions

* Moshe Zadka wrote a article (http://www.livejournal.com/users/moshez/5536.html), about the usefulness to change the page-file paradigm, and use class-page instead.

Progress

* A new version (0.8.6) of Quotient (http://www.divmod.org/) has been released (Editing of preferences via FreeForm, Tests,Improved table sorting,People and contacts,Inbox manager,Trash Manager,Compose and send messages with attachments, Password recovery,Thumbnailing,STUN server,Numerous tweaks and fixes).

Cool Thing of the Week

* Draft of documentation for nevow (http://stewstuff.com/doc/nevow.xhtml): A first start at a documentation for the new web piece of Twisted.

Ok, that’s it for this week. Thanks for reading!

Lior Gradstein

Twisted Weekly News #7
Author: Lior Gradstein
Contact: lior@gradstein.info
Revision: 1.0
Date: 2003-11-23
Copyright: © The Twisted Community

Contents

* The Twisted Weekly News
* Discussions
* Progress
* Cool Thing of the Week

I’ll try to keep the pace with the TWN, continuing the work of Chris Armstrong. Thanks to Noa Harvey who is correcting my english as it is not my primary language.
The Twisted Weekly News

7th Issue. Sunday, November 23, 2003

QOTW:

scripts are just usually short programs that do a very specific thing
that’s why a lot of us people who use interpreted languages hate it
when someone calls our language a “Scripting language” ;)
(I mean, look at Twisted and call it a “script” with a straight face)

Discussions

XXXX
Progress

* A new version (0.8.0) of Quotient (http://www.divmod.org/) has been released (Editing of preferences via FreeForm, Tests,Improved table sorting,People and contacts,Inbox manager,Trash Manager,Compose and send messages with attachments, Password recovery,Thumbnailing,STUN server,Numerous tweaks and fixes).

Cool Thing of the Week

* TouRSSt, by Tv (http://toursst.sourceforge.net/): TouRSSt tours the web so you don’t have to. Run frequently, it pulls new RSS news items from a number of channels and puts them where you want. The project existed for a long time, but it was revived on November, 2.
* Informant, by Nafai (http://www.travishartwell.net/projects/informant/): First version of Informant, an graphical event logger that makes notifications (Gnome based, but a standalone version is possible).
* Twibbler (http://slarty.polito.it:8069/~sciasbat/wiki/moin.cgi/twibber): Twibber is a Python package aimed at handling the Jabber/XMMP protocol in a completely asynchronous way, with no need of threads.
* Web Forum: A web forum has been created to allow people that like clicking to discuss more about Twisted! Its url is http://twisted.banquise.org/

Ok, that’s it for this week. Thanks for reading!

Lior Gradstein