Contents |
Win32 users: download the Installer as a self-extracting executable.
Beta 3 can be downloaded here (updated Oct 13, 1999 - building packages into .pyz...). Note that Beta 3 is different enough to have its own documentation (although most of the general comments on this page still apply). Unix weenies: download the archive portion of the project. Please read the sections on Archives and Import Hooks. |
ZlibArchiveA ZlibArchive (figure 1) contains compressed .pyc (or .pyo) files. The Table of Contents is a marshalled dictionary, with the key (the module's name as given in an "import" statement) associated with a seek position and length.A ZlibArchive hooks in with Greg Stein's imputil.py so that, with a little setup, the archived modules can be imported transparently. Even with compression at level 9, this works out to being faster than the normal import. Instead of searching sys.path, there's a lookup in the dictionary. There's no stat-ing of the .py and .pyc and no file opens. There's just a seek, a read and a decompress. Of course, a traceback is not going to show you what you want to see, but you wouldn't use a .jar file if you were debugging Java, would you. |
![]() |
CArchiveA CArchive (figure 2) contains whatever you want to stuff into it. It's very much like a .zip file, but one that's easy to create in Python and unpack from C code. The whole reason these exist is to make it easy to create self-extracting executables. For this reason, they expect to be opened from the end, so the TOC for a CArchive is at the back, followed only by a cookie that tells you where the TOC starts and where the archive itself starts.The TOC for a CArchive also contains additional information that makes it easy to use them from C. Each TOC entry is variable length. The first field in the entry tells you the length of the entry. The last field is the name of the corresponding packed file. The name is null terminated. |
![]() |
Self-extracting executablesThe COFF executable format (Windows and Linux) allows arbitrary data to be concatenated to the end of the executable without disturbing it's functionality. For this reason, a CArchive's Table of Contents is at the end of the archive. The executable can then look at argv[0] to get it's file name, open that file, seek to the end and 'open' the CArchive (see figure 3).For Windows (only), I have created enhanced versions of python.exe and pythonw.exe that do precisely this (and a bit more). They unpack the CArchive into the exe's directory. They set PYTHONPATH to be the exe's directory and then dynamically load python. This means that python15.dll can be one of the things in the CArchive, and that even if another version of python is installed, it should be this python15.dll that is loaded. Command line arguments to the exe are passed on to Python. |
![]() |
Essentially, new importer's are chained in front of the buit-in import mechanism. When an import statement is encountered, it is handed to the importer at the head of the chain. If that importer fails, it hands the request to the next importer in the chain.
The best place to set this up is in site.py, which is the second automatic import that Python does (after exceptions.py). (Note: Beta 03 does not use site.py - it sets up the import hook directly from the C code.) In order to set it up, site.py will need to import imputil.py. In order to use a ZlibArchive, it will also need archive.py. Accordingly, these four modules (exceptions, site, imputil and archive) must go through the standard import mechanism and can't be served from a ZlibArchive. (Also, your main script cannot be served from a ZlibArchive, because it is run, not imported.)
The statement that makes a ZlibArchive available looks like this:
imputil.FuncImporter(archive.ZlibArchive("mylib.pyz").get_code).install()
That is, it instantiates a FuncImporter, passing it the get_code
method of a ZlibArchive (instantiated from a filename).
The install
method of the FuncImporter is run, and that's all there is to it.
All the modules and packages with 'mylib.pyz' are now available to Python code.
All of this is pure Python (and thus cross platform).
(Well, the compression uses zlib, which is pretty ubiquitous, but optional.)
It would be nice if more of this was built-in, (so it worked from just an archive and a line in a .pth file, or something like that), but I'm not complaining.
(logicalname, pathname)
, where "logicalname" is the name under which it will be filed, and pathname is the file to be packed.
For a ZlibArchive, that's all the information we need.
For a CArchive, there is also a flag to tell the archive whether the file should be stored compressed.
As a first choice, bindepends will use dumpbin.exe (comes with MSVC) to find these dependencies, but if it can't find dumpbin.exe, it will walk (er, stumble) through the PE header, and pick them out from Python. (I have not yet found a case where dumpbin finds something my Python code misses, but MS is liable to fool with the PE header format at any time.) This process returns a logical table of contents that can be used to build a CArchive with all the required binaries. Note that it will find *all* the dependencies it can. I have built in (to bindepends.py) a list of .dll's that should be excluded from the CArchive, but you should definitely make sure you are not redistributing some .dll that you do not have a license to redistribute, (see the sections on the config file and the log file below).
At the core is the ZlibArchive. You could use this to replace the standard library (many megs) with a 500K .pyz file, or to distribute your package in binary form, or to ship the support for your script.
Next is the CArchive which you can use like a .zip or .tar file. In fact, I use them that way to package up the .tcl files necessary to support Python/Tk apps.
Next comes the self-extracting executable. As explained above, these open the CArchive that has been appended to the .exe and extract all the files. They then load python and run a script named "launch.py" (which should be your main). Anything that was in the CArchive should be available in the current directory, which will also be the only entry on sys.path, (note: there are changes to this scheme in the current beta).
With these pieces, you can get as arcane and complex as you like. For convenience, I define two levels of packaging.
One is the "quasi-stand-alone" executable. This is an .exe with a CArchive appended. At a minimum, the CArchive contains a script. What else goes in the CArchive is up to you. The .exe at the front (Run.exe or Runw.exe) will clean up what it unpacked at end of run.
The other level is an "installer".
This is again an .exe (this time Launch.exe - a console app) and a CArchive.
In this case I use all the above dependency tracking to build the CArchive for you.
It expects your main script to be an install script, and some utility functions are included in installutils.py.
Typically, this (outer) CArchive would contain a "quasi-stand-alone" executable, and the .pyz and other files necessary for it to run.
Your install script would get an installation directory from the user, and then move the required files to this directory.
Builder also does some extra things for you - like write a "site.py" (while processing an INSTALL section) that contains the code to load any .pyzs included in that section.
It is also smart about Tk usage, and will archive the .tcl files.
The code to install these in a /lib sudirectory is included in installutils.py.
Please see Beta 3 for a substantial list of changes.
Please report bugs to gmcm@hypernet.com. Include information about your system, your version of Python (this won't work with 1.5.1 because zlib.pyd was built differently), the config file and Builder.log.
Please see Beta 3 for a substantial list of changes.
Send any requests for enhancements or (better yet) patches to gmcm@hypernet.com.
Using It
There are two higher levels for putting these components together.
The main workhorse is a script called Builder.py.
Builder.py works from a config file.
The config file
The config file must have at least one section, and can have any number more than that.
Sections come in four types.
One type (type= PYZ
) describes how to build a .pyz file.
The second, (type= STANDALONE
) describes how to build a "quasi-stand-alone" executable.
The third, (type= COLLECT
) is a way of gathering all the needed components into one directory.
You can then use whatever tool (WinZip, Wise, InstallShield...) to make your own installation package.
The last, (type= INSTALL
) describes how to build an "installer".
The first, third and fourth types are capable of tracking down dependencies automatically.
The second does not.
Running Builder
Pass Builder.py the name of your config file (which should be in the current directory).
Builder will figure out the inter-section dependencies (eg, your INSTALL section will almost undoubtedly reference all the other sections), and build them in the correct order (or yell at you if you've created circular dependencies).
Building a section consists of first determining the logical table of contents for that section (tracking down dependencies, merging, excluding...), then building the archive (or executable + archive).
All of this can be guided by options in the appropriate sections.
This is a fairly complex (and slow!) process, and Builder.py writes a fairly detailed log of what it is doing at each point.
Examining the Log
The log file will show you the "state" of each section as it is being built.
It starts by dumping a dictionary built from that section of the config file.
Then it shows how the logic table of contents changes with each option processed.
The "final lTOC" shows you exactly what went into the corresponding archive.
This is so you can debug / tweak your config file.
Note that you should always examine the "final lTOC" of an INSTALL section, to make sure you are not redistributing .dlls that you shouldn't.
Creating a .pyz
A type= PYZ
section uses the following options:
Creating a "standalone"
A type= RUN
section uses these options:
Creating a "collection"
A type=COLLECT
section uses these options:
Creating an installation package
A type= INSTALL
section uses these options:
Writing the install script
The job of your install script is simple: set up a directory for your app / package to live in, then clean up after yourself (not: changes in current beta).
Unfortunately, this isn't all that easy in practice.
The module installutils.py has some helper routines you can use.
You should hard code the names of all the files you are moving (and removing from the current directory) into your install script.
Note that this means creating an install package is an iterative process.
On your first attempt, you are unlikely to know exactly what goes into the package.
Also, the OS will not let you clean up files that are currently in use (the .exe hosting your install script, python15.dll and zlib.pyd at a minimum).
Using Simple
At the very top level is a script call Simple.py.
Put your main script in a directory, and run Simple.py on it.
This will create a config file and a sample install script.
This is really just to get you started.
It creates three sections: one for a .pyz containing all the Python module support, one a "quasi-standalone" for your main script, and one an installer for the .pyz, .exe and the required binary support.
Even if the config file turns out to do everything you need, you should probably rewrite the install script and point the INSTALL section to your install script.
Known Bugs
TypeError: Unsliceable object
is not a good way of telling the user they've specified an non-existant file name (Builder.py). (Fixed in beta3).
Planned Enhancements
Acknowledgements
Change Log
License / Copyright