Packaging Python Software¶
This is a how-to for developers with directions on packaging their software in ways that enable a painless installation experience for end-users. Installing Python Software is the related end-user guide.
See also these other resources on the web…
- The Python Packaging User Guide
- The Hitchhiker’s Guide to Python!
- The Sheer Joy of Packaging! – A Scipy 2018 tutorial, also covering Conda
The following figure gives a rough outline of what tools are involved in the development workflow. Regard services like GitLab and Artifactory as representatives, you can equally well use any git server solution and e.g. devpi as your artifact repository. There are also public services like GitHub, Travis, PyPI, and BinTray that can fill these roles for open-source projects.
Packaging PyPI Releases¶
This is a short summary of essentals, consult the above resources for all the details.
It covers the ‘classic’ tool-chain, there are more ‘modern’ tools like
flit that serve similar purposes.
Building with setuptools¶
Packaging with wheel¶
Uploading with twine¶
Once you have your deployment artifacts ready (typically in a
you can upload them to
pypi.org, or a local repository service.
There is a dedicated tool named twine for this. It supports using SSL for transfers, and also allows you to first build your artifacts, then test them as you see fit, and finally upload the tested artifacts.
Configuration is taken from
~/.pypirc, or the environment – especially useful for CI jobs.
A typical configuration might look like this:
[distutils] index-servers = local pypi [local] repository: https://artifactory.local/artifactory/api/pypi/pypi-releases-local username: «USER» password: «API_TOKEN» [pypi] # repository: https://pypi.org/pypi username: «PYPI_USER» password: «PYPI_PWD»
You can select from the list of index servers by using
twine upload -r «repo» …,
the default is
Building Zipapps (PEP 441)¶
Running Python code directly from ZIP archives is nothing new,
PEP 273 made its debut in 2001,
as part of Python 2.3 in the form of the
PEP 441 builds on this and
describes mechanisms to bundle full applications into a single ZIP file
that can be made executable.
It was approved in 2015 and a first implementation appeared in Python 3.5 via the
See the PEP for details on how making a ZIP into an executable file works,
but basically on POSIX systems the Python interpreter is called in a ‘bang path’
that is followed by the ZIP archive. The interpreter recognizes the ‘script’
is a whole application archive and acts accordingly.
On Windows, zipapps MUST carry the
.pyz extension which is bound to the
py wrapper command,
which in turn looks at the bang path and calls a matching Python interpreter from the installed set.
To display the bang path of a zipapp, use this command:
python3 -m zipapp --info foo.pyz
If you want to change the requested Python version to a newer one that is actually installed, change the bang path as part of the installation process:
python3 -m zipapp -p '/usr/bin/env python3.5' -o ~/bin/foo foo.pyz
This can also be done on an ad-hoc basis, by explicitly calling the desired interpreter:
python3.5 foo.pyz … # POSIX py -3.5 foo.pyz … # Windows
Well-known tools to build new zipapps, outside of the Python core, are pex (Twitter) and shiv (LinkedIn). See their documentation for details on bundling your own applications, and also the next section on PEX.
Packaging Python EXecutables (PEX)¶
PEX files are Python Executable ZIP files, a format that contains a full distribution of a Python application in a single archive (just like exectable JARs for Java). PEX files can be targeted at a specific platform and Python version, but might also support multiple runtime environments. See Installation With PEX for details on how to use them, and PEP 441 for a formal description of the underlying mechanics and all the details.
The Rituals task library for Invoke offers a
that performs all the necessary steps to create a PEX file.
If you want to do it ‘manually’ or integrate it into another task runner,
this is a concrete example:
pex -r requirements.txt . -c nanny \ -o bin/nanny-0.1.0.dev5-cp27-none-linux_x86_64.pex
At the time of this writing, you need to install
pex 1.0.dev directly from GitHub
for the above to work.