Pip Requirements Standardization

Proposer

Mark Beierl (Canonical)

Type

Feature

Target MDG/TF

common, devops, IM, LCM, MON, N2VC, NBI, osmclient, PLA, POL, RO

Description

The history of OSM saw us installing the software as Debian (.deb) packages onto an Ubuntu based operating system. Over time, this had changed to being docker image based, with python packages still being provided as .deb installable components.

This presents us with a new problem: the mixing and matching of Python dependencies across pip, the python package installer, and apt, the Ubuntu OS level package installer. Python modules can be shipped using either of the two package installers, but they are at odds with each other as pip will not alter or update anything that was installed by apt, as it is considered to be a system level package manager and should be the authoritative manager of system packages. Having said that, not all versions of Python modules that are in pip are available in .deb format. Also, the apt repositories choose their own cadence for when versions are released, and might not carry versions that we require.

This change proposes moving away from using .deb+apt to manage dependency versions, and to rely solely on Python being able to express its own dependencies based on the Python Package Index (PyPI). The scope of this change involves all modules that produce Python packages, and can be broken down into the following areas.

  • Dependency expression (requirements.txt)
  • Debian package expression (apt requires)
  • Common format for building Python (tox.ini)
  • Installation of packages (Dockerfiles)

Sample Chain of Modules

This example uses the POL module as it is a fairly simple chain to follow:

  • POL requires osm_common
  • osm_common does not require anything from OSM, but has some upstream dependencies

Starting with common:

Define standard tox.ini

All projects will follow the same pattern:

  • have the following environments: flake8, cover, safety, dist
  • have different requirements files for final product and test modules
  • build will produce the .deb package, which must include the project's requirements.txt file

Expression of Requirements

No requirements may be expressed in setup.py. Anything that is in that file will be used when creating the .deb package and force the installation of the .deb version of the dependency. To avoid this, we must use requirements files external to setup.py until we move away from .deb packaging entirely.

requirements.in: this file is used to express the upstream dependencies from PyPi for the module. It is permissible to express the modules in the following ways:

  • Name only: aiokafka This means we are looking to use the latest version at this point in time
  • Range: aiokafka<=0.6.0 This one means we are looking for an older version, and it cannot be more recent than 0.6.0

Version ranges in requirements.in is how we control the dependency matrix when a version in upstream is known to cause problems. This should be avoided by spending the time to adjust our code to work with the particular version, but is acceptable as a measure to proceed until the time is available to resolve the issue.

requirements-test.in: this file is used to express additional modules that are used when unit testing the software only. For example, mocking libraries are often needed for a unit test, but are not part of the production code.

requirements-dist.in: this is used to install packages that are required while producing the final distributable package. For example, to create a .deb, we need the stdeb module. For uploading to PyPI, we would need twine.

All .in files must be compiled into their corresponding .txt equivalents, using pip-compile. This takes the expressed modules, with any possible version constraints and produces a final list of all the required modules at a specific version. For example, requirements.in containing aiokafka would produce a requirements.txt as follows:

aiokafka==0.5.2
kafka-python==1.4.6 # via aiokafka

Note that while only aiokafka was mentioned, the pip-compile tool looked further upstream and found that kafka-python was also required, found the best version, and noted why it was included in the requirements.txt

Promotion of Requirements to Downstream

Moving on to POL:

Now that the .deb package has been created without any dependencies, we face the next problem: how does the downstream software (in our example POL) know what osm_common requires in order to function? To support this, we will include common's requirements.txt file in its .deb package.

The Dockerfile (in devops, stage 3) for POL would then look something like this:

FROM ubuntu 18.04

RUN apt --yes update
RUN apt --yes install python3 python3-pip

...
RUN apt --yes install python3-osm-common${COMMON_VERSION} \
                      python3-osm-policy-module${POL_VERSION}

RUN pip3 install -r /usr/lib/python3/dist-packages/osm_common/requirements.txt \
                 -r /usr/lib/python3/dist-packages/osm_pol/requirements.txt

The packages that used to be automatically installed by apt now must be explicitly installed via the pip3 install command. This is the desired outcome: we now have once place to express requirements, which is in the source code of the module itself, and complete control over the final list of dependencies that gets installed.