from IPython.display import HTML
HTML(open("../include/notes.css", "r").read())
# make sure the demos and applications directories exist
! [ -d demos ] || mkdir demos
# This allows us to import "dot" graph visualization files (recommended)
import graphviz
import os
def fig(fname, *args, **kwds):
dotSource = open(os.path.join("figures", fname)).read()
return graphviz.Source(dotSource, *args, **kwds)
# You also need to have the "graphviz" executables installed on your system.
# On Ubuntu, this shell command will install them:
# $ sudo apt install graphviz
How do you go about sharing Python code with the rest of the world? Desirable features:
There should be a single package you can email or put in a repository somewhere.
It should install and run anywhere there's a Python (3K) interpreter and possibly nothing else (e.g. make, cmake, apt-get etc.).
It should install to standard places where anybody can run (a script) or import (a module or package) it.
There should be a way to create "binary" installation packages for a variety of platforms.
It should be easy to submit your work to a shared repository.
The setuptools package was designed to permit these.
Suppose you have a Python module hello.py that you want to distribute:
! [ -d demos/d0_hello ] || mkdir demos/d0_hello
%%file demos/d0_hello/hello.py
print("Hello, world!")
For clarity, we put hello.py in an otherwise empty directory d0_hello. This name is arbitrary and is not used anywhere in the distribution process. This could in fact be the source code directory for your project.
fig("g00_module.gv")
In these diagrams, we'll use squares for directories and ellipses for files and red to indicate additions to the previous graph.
Aside: In this notebook, we're displaying graphs using the "graphviz" graph visualization format. It's pretty handy. Here's what produced the diagram above:
# %load figures/g00_module.gv
digraph setuptools {
node [ fontname="courier" ];
d0_hello -> hello_py;
d0_hello [ label="d0_hello", shape=box, color="red"];
hello_py [ label="hello.py", color="red" ];
}
RESUME ON 11/10
setup.py File¶You then create a new file setup.py in the same directory:
%%file demos/d0_hello/setup.py
from setuptools import setup
setup(name='hello', # the package/module name
version='1.0', # the version (an arbitrary string)
author='Bob Lewis', # someone to blame
author_email='bobl@wsu.edu', # where to flame
py_modules=[ 'hello' ], # modules in the package
)
so the directory now looks like this:
fig("g01_module_setup.gv")
All of the arguments to setup() are keyworded, and there are a lot of them. Most of them can be ignored. setup.py is the key to getting setuptools to work. You always invoke it as a Python module from the source ("d0_hello" here) directory.
The setup() invocation in setup.py does most of the work. What it actually does is determined by the command line (see below), which it accesses through sys.argv. Think of its role like that of a makefile. It even does timestamp dependency analyses.
setuptools provides extensive help:
!cd demos/d0_hello; python3 setup.py --help
and specific commands are listed like this:
!cd demos/d0_hello; python3 setup.py --help-commands
Two things to note:
You'll see the package distutils referred to in the help documentation for setuptools. setuptools is an enhanced version of the former packaging utility distutils and is referred to for backwards compatibility. (In fact, you can still use the distutils package if you like for much of what we present here.)
There are references to "eggs". An .egg file is a "logical structure embodying the release of a specific version of a Python project, comprising its code, resources, and metadata". They are typically zip-compressed and may be accessed from Python via the import statement just like a *.py file, a directory containing a __init__.py file, a *.so file, etc. We'll explore their contents below.
build Process¶The first step to distributing a Python package is the "build" step. You do it by cding to that directory and invoking setup.py with the build argument:
# this command avoids setuptool's optimization for demo purposes
!cd demos/d0_hello; touch hello.py
!cd demos/d0_hello; python3 setup.py build
The "running" messages are just informative: Any real actions taken are explicit (like the copy).
This will create the following:
fig("g02_after_module_build.gv")
All setuptools has had to do here as part of the build is copy the file. It's actually capable of doing much more (e.g. compiling C extensions -- in an upcoming unit), but that's all it needs to do here. The build subdirectory is a temporary holding area and can be deleted when no longer used.
If you're using git or subversion, you can ignore build entirely.
install Process¶Once the build directory is set up, the next step is to install the package. To be on the safe side, I recommend using the --dry-run option, which tells setuptools to show what it's going to do, not actually do:
# This will fail due to permissions:
!cd demos/d0_hello; python3 setup.py install --dry-run
As you can see, the install failed because we don't have install permission in the system-wide directory /usr/local/lib/python3.8/dist-packages. That means that to install there, we'd have to do so as root (possibly running the python3 command via sudo).
But there's an alternative if you're working on a system you can't run as root on:
The --user flag puts the module in a directory under your home directory:
!cd demos/d0_hello; python3 setup.py install --user
This installs all the files in directories below your home directory where you presumably have write access, including a site-packages directory that your import statements will automatically access, as seen below:
from pprint import pprint
import sys
pprint(sys.path)
This is also how your friend can install a package you send them without having sysadmin privileges, either.
Assuming we do have root access, the installation will do this:
fig("g03_after_module_install.gv")
The "EGG-INFO/PKG-INFO" file in the zip archive contains metainformation about the release. (we'll use the one installed in our user directory.)
!unzip -ca /home/bobl/.local/lib/python3.8/site-packages/hello-1.0-py3.8.egg EGG-INFO/PKG-INFO
This allows other installations to verify that your version is compatible with theirs. The "UNKNOWN"s are fields that we can fill in with arguments to setup().
Incidentally, if you forget to build, install will do it for you. The only problem with that is that if you need to install as root (using sudo), the build files in your directory will be owned by root and you'll have to remove them as root (again probably using sudo).
In short, when you install for the system (i.e. under /usr/local), build as you and install as root.
So we can build and install on our own system, but now we want to send the code somewhere else: a "source distribution". To do this, we again use setup.py:
!cd demos/d0_hello; python3 setup.py sdist
In order, the warnings tell us:
We should provide a "url=" argument to setup() to refer to a Web site (e.g. PyPI).
We can add a "MANIFEST.in" file in hello to list additional files (e.g. stand-alone documentation) to include in the distribution and also files to exclude.
We should add a "README", "README.txt", or "README.rst" file in the distibution.
Note the comment:
Remember: distutils is the predecessor of setuptools and is a subset. Don't worry if you see it in setuptools output.
If we had files listed in MANIFEST.in, their names would be in MANIFEST as well.
Here's the directory hierarchy now:
fig("g04_after_module_sdist.gv")
And the contents of "hello-1.0.tar.gz", which is just a gzipped tarball we all know and love.
!cd demos/d0_hello; tar -tzf dist/hello-1.0.tar.gz
The tarball includes the same setup.py we've been using all along, so anyone who gets the tarball and has setuptools installed can do anything we do in this demo, including create another source distribution. This is embodiment of open source.
The PKG-INFO file is the same as was included in the EGG-INFO directory within the *.egg zipfile we looked at above.
Binary Distributions¶A "binary" distribution is a bit of a misnomer: It includes the source, but it's designed to be used, not modified. In addition, it also installs in a single step, depending on the platform, and compiles Python files to bytecodes. (These, at least, are the defaults.)
There are four kinds of binary distribution, selected by the setup.py argument:
bdist_dumb
A tarball designed to be un-tarred in the right place.
bdist_rpm
A file in with the Red Hat Package Manager format.
bdist_wininst
An ".exe" file that installs the module or package when run on a Windows platform. (This setuptools command only works on a Windows machine.)
bdist_wheel
This is the new standard in use by PyPI (the Python Package Index) that is intended to replace "eggs"s.
If you don't specify any of these, it chooses a platform specific one: bdist_dumb on Linux (and maybe Mac) machines and bdist_wininst on Windows machines.
You'll notice that there's no way to build a Debian ("*.deb") package here. Actually, you can with some setuptools extensions, but its better to use pip3, which accesses PyPI (see below) than a distribution-specific package manager like apt-get or yum.
!cd demos/d0_hello; python3 setup.py bdist
Notice what's going on: setuptools is building a sparse directory hierarchy resembling the whole filesystem below build/bdist.{platform}/dumb/, byte compiling the Python source into it, and tarring everything up.
fig("g05_after_module_bdist.gv")
Here's what the tarball contains:
!tar -tzf demos/d0_hello/dist/hello-1.0.linux-x86_64.tar.gz
If you "cd /" and un-tar the file there (or pass "-C /" to tar), all the files (and, if needed, directories) will end up in the same places they would for "$ python3 setup.py install".
Most Python code worth distributing comes in a package: a directory containing multiple Python source files. Here's the initial layout:
fig("g06_package.gv")
Review: What makes this a package?
We add a setup.py that is slightly different from before:
%%file demos/d1_ancient_math/setup.py
from setuptools import setup
setup(name='ancient_math',
version='1.0',
author='Bob Lewis',
author_email='bobl@wsu.edu',
packages=['ancient_math'], # name change and "packages", not "py_modules"
)
!cd demos/d1_ancient_math; python3 setup.py build
setuptools copies all of the files in ancient_math to build/lib.
fig("g07_after_package_build.gv")
And everything else is much the same as for a module.
!cd demos/d1_ancient_math; python3 setup.py install --user
You need to restart the Jupyter kernel at this point to get the "import" to work after the install.
import ancient_math as am
am.euclid(234, 23400)
am.triangleArea(23, 19, 30)
Once you've installed a package, you can remove it using the pip (or pip3) command:
# The "-y" flag avoids the confirmation query that confuses Jupyter.
!python3 -m pip uninstall -y ancient_math
You need to restart the Jupyter kernel at this point to get the "import" to not work after the install.
import ancient_math
The Python Package Index (http://pypi.python.org) is a central repository for Python packages. Currently, it contains over 150,000 packages, all installable via the pip module:
python3 -m pip install {package_name}
On many systems, thie command does the same thing:
pip3 install {package_name}
pip is PyPI's equivalent to apt-get, yum, rpm, and homebrew.
setuptools has a built-in connection to PyPI:
$ python3 setup.py register
will register the distribution with PyPI.
$ python3 setup.py upload
will upload the latest distribution to PyPI.