From: beierlm Date: Thu, 11 Feb 2021 19:50:32 +0000 (-0500) Subject: Pip Requirements Standardization X-Git-Url: https://osm.etsi.org/gitweb/?a=commitdiff_plain;h=3e3b0ce78133b23ff18c7c648c3308325d5bef33;p=osm%2FFeatures.git Pip Requirements Standardization Change-Id: Id6371c971e8dda5934e296d952139dbde4ef6478 Signed-off-by: beierlm --- diff --git a/Release10/pip-requirements-standardization.md b/Release10/pip-requirements-standardization.md new file mode 100644 index 0000000..a7d0bce --- /dev/null +++ b/Release10/pip-requirements-standardization.md @@ -0,0 +1,126 @@ +# Pip Requirements Standardization # + +## Proposer ## +Mark Beierl (Canonical) + +## Type ## +**Feature** + +## Target MDG/TF ## +common, devops, IM, LCM, MON, N2VC, NBI, osmclient, PLA, POL, RO + +## Description ## +The history of OSM saw us installing the software as Debian (.deb) packages onto an +Ubuntu based operating system. Over time, this had changed to being docker image +based, with python packages still being provided as .deb installable components. + +This presents us with a new problem: the mixing and matching of Python dependencies +across pip, the python package installer, and apt, the Ubuntu OS level package +installer. Python modules can be shipped using either of the two package installers, +but they are at odds with each other as pip will not alter or update anything that was +installed by apt, as it is considered to be a system level package manager and should +be the authoritative manager of system packages. Having said that, not all versions +of Python modules that are in pip are available in .deb format. Also, the apt +repositories choose their own cadence for when versions are released, and might +not carry versions that we require. + +This change proposes moving away from using .deb+apt to manage dependency versions, +and to rely solely on Python being able to express its own dependencies based on the +Python Package Index (PyPI). The scope of this change involves all modules that +produce Python packages, and can be broken down into the following areas. + +* Dependency expression (requirements.txt) +* Debian package expression (apt requires) +* Common format for building Python (tox.ini) +* Installation of packages (Dockerfiles) + +### Sample Chain of Modules + +This example uses the `POL` module as it is a fairly simple chain to follow: + +* `POL` requires `osm_common` +* `osm_common` does not require anything from OSM, but has some upstream dependencies + +Starting with `common`: + +### Define standard tox.ini + +All projects will follow the same pattern: + +* have the following environments: `flake8`, `cover`, `safety`, `dist` +* have different requirements files for final product and test modules +* `build` will produce the .deb package, which must include the project's + requirements.txt file + +### Expression of Requirements + +No requirements may be expressed in `setup.py`. Anything that is in that file +will be used when creating the `.deb` package and force the installation of the +`.deb` version of the dependency. To avoid this, we must use requirements files +external to `setup.py` until we move away from `.deb` packaging entirely. + +`requirements.in`: this file is used to express the upstream dependencies from PyPi for +the module. It is permissible to express the modules in the following ways: + +* Name only: `aiokafka` This means we are looking to use the latest version at this + point in time +* Range: `aiokafka<=0.6.0` This one means we are looking for an older version, and it + cannot be more recent than 0.6.0 + +Version ranges in requirements.in is how we control the dependency matrix when a +version in upstream is known to cause problems. This should be avoided by spending +the time to adjust our code to work with the particular version, but is acceptable +as a measure to proceed until the time is available to resolve the issue. + +`requirements-test.in`: this file is used to express additional modules that are used +when unit testing the software only. For example, mocking libraries are often needed +for a unit test, but are not part of the production code. + +`requirements-dist.in`: this is used to install packages that are required while +producing the final distributable package. For example, to create a .deb, we +need the `stdeb` module. For uploading to PyPI, we would need `twine`. + +All `.in` files must be compiled into their corresponding `.txt` equivalents, using +`pip-compile`. This takes the expressed modules, with any possible version constraints +and produces a final list of all the required modules at a specific version. For +example, `requirements.in` containing `aiokafka` would produce a `requirements.txt` as +follows: + +``` +aiokafka==0.5.2 +kafka-python==1.4.6 # via aiokafka +``` + +Note that while only `aiokafka` was mentioned, the `pip-compile` tool looked further +upstream and found that `kafka-python` was also required, found the best version, +and noted why it was included in the `requirements.txt` + +### Promotion of Requirements to Downstream + +Moving on to `POL`: + +Now that the `.deb` package has been created without any dependencies, we face the +next problem: how does the downstream software (in our example `POL`) know what +`osm_common` requires in order to function? To support this, we will include common's +requirements.txt file in its `.deb` package. + +The Dockerfile (in devops, stage 3) for `POL` would then look something like this: + +``` +FROM ubuntu 18.04 + +RUN apt --yes update +RUN apt --yes install python3 python3-pip + +... +RUN apt --yes install python3-osm-common${COMMON_VERSION} \ + python3-osm-policy-module${POL_VERSION} + +RUN pip3 install -r /usr/lib/python3/dist-packages/osm_common/requirements.txt \ + -r /usr/lib/python3/dist-packages/osm_pol/requirements.txt +``` + +The packages that used to be automatically installed by apt now must be explicitly +installed via the `pip3` install command. This is the desired outcome: we now have +once place to express requirements, which is in the source code of the module itself, +and complete control over the final list of dependencies that gets installed.