Pip Requirements Standardization

Change-Id: Id6371c971e8dda5934e296d952139dbde4ef6478
Signed-off-by: beierlm <mark.beierl@canonical.com>

diff --git a/Release10/pip-requirements-standardization.md b/Release10/pip-requirements-standardization.md
new file mode 100644 (file)
index 0000000..a7d0bce
--- /dev/null
+++ b/Release10/pip-requirements-standardization.md
@@ -0,0 +1,126 @@
+# Pip Requirements Standardization #
+
+## Proposer ##
+Mark Beierl (Canonical)
+
+## Type ##
+**Feature**
+
+## Target MDG/TF ##
+common, devops, IM, LCM, MON, N2VC, NBI, osmclient, PLA, POL, RO
+
+## Description ##
+The history of OSM saw us installing the software as Debian (.deb) packages onto an
+Ubuntu based operating system.  Over time, this had changed to being docker image
+based, with python packages still being provided as .deb installable components.
+
+This presents us with a new problem: the mixing and matching of Python dependencies
+across pip, the python package installer, and apt, the Ubuntu OS level package
+installer.  Python modules can be shipped using either of the two package installers,
+but they are at odds with each other as pip will not alter or update anything that was
+installed by apt, as it is considered to be a system level package manager and should
+be the authoritative manager of system packages.  Having said that, not all versions
+of Python modules that are in pip are available in .deb format.  Also, the apt
+repositories choose their own cadence for when versions are released, and might
+not carry versions that we require.
+
+This change proposes moving away from using .deb+apt to manage dependency versions,
+and to rely solely on Python being able to express its own dependencies based on the
+Python Package Index (PyPI).  The scope of this change involves all modules that
+produce Python packages, and can be broken down into the following areas.
+
+* Dependency expression (requirements.txt)
+* Debian package expression (apt requires)
+* Common format for building Python (tox.ini)
+* Installation of packages (Dockerfiles)
+
+### Sample Chain of Modules
+
+This example uses the `POL` module as it is a fairly simple chain to follow:
+
+* `POL` requires `osm_common`
+* `osm_common` does not require anything from OSM, but has some upstream dependencies
+
+Starting with `common`:
+
+### Define standard tox.ini
+
+All projects will follow the same pattern:
+
+* have the following environments: `flake8`, `cover`, `safety`, `dist`
+* have different requirements files for final product and test modules
+* `build` will produce the .deb package, which must include the project's
+  requirements.txt file
+
+### Expression of Requirements
+
+No requirements may be expressed in `setup.py`.  Anything that is in that file
+will be used when creating the `.deb` package and force the installation of the
+`.deb` version of the dependency.  To avoid this, we must use requirements files
+external to `setup.py` until we move away from `.deb` packaging entirely.
+
+`requirements.in`: this file is used to express the upstream dependencies from PyPi for
+the module.  It is permissible to express the modules in the following ways:
+
+* Name only: `aiokafka`  This means we are looking to use the latest version at this
+  point in time
+* Range: `aiokafka<=0.6.0`  This one means we are looking for an older version, and it
+  cannot be more recent than 0.6.0
+
+Version ranges in requirements.in is how we control the dependency matrix when a
+version in upstream is known to cause problems.  This should be avoided by spending
+the time to adjust our code to work with the particular version, but is acceptable
+as a measure to proceed until the time is available to resolve the issue.
+
+`requirements-test.in`: this file is used to express additional modules that are used
+when unit testing the software only.  For example, mocking libraries are often needed
+for a unit test, but are not part of the production code.
+
+`requirements-dist.in`: this is used to install packages that are required while
+producing the final distributable package.  For example, to create a .deb, we
+need the `stdeb` module.  For uploading to PyPI, we would need `twine`.
+
+All `.in` files must be compiled into their corresponding `.txt` equivalents, using
+`pip-compile`.  This takes the expressed modules, with any possible version constraints
+and produces a final list of all the required modules at a specific version.  For
+example, `requirements.in` containing `aiokafka` would produce a `requirements.txt` as
+follows:
+
+```
+aiokafka==0.5.2
+kafka-python==1.4.6 # via aiokafka
+```
+
+Note that while only `aiokafka` was mentioned, the `pip-compile` tool looked further
+upstream and found that `kafka-python` was also required, found the best version,
+and noted why it was included in the `requirements.txt`
+
+### Promotion of Requirements to Downstream
+
+Moving on to `POL`:
+
+Now that the `.deb` package has been created without any dependencies, we face the
+next problem: how does the downstream software (in our example `POL`) know what
+`osm_common` requires in order to function?  To support this, we will include common's
+requirements.txt file in its `.deb` package.
+
+The Dockerfile (in devops, stage 3) for `POL` would then look something like this:
+
+```
+FROM ubuntu 18.04
+
+RUN apt --yes update
+RUN apt --yes install python3 python3-pip
+
+...
+RUN apt --yes install python3-osm-common${COMMON_VERSION} \
+                      python3-osm-policy-module${POL_VERSION}
+
+RUN pip3 install -r /usr/lib/python3/dist-packages/osm_common/requirements.txt \
+                 -r /usr/lib/python3/dist-packages/osm_pol/requirements.txt
+```
+
+The packages that used to be automatically installed by apt now must be explicitly
+installed via the `pip3` install command.  This is the desired outcome: we now have
+once place to express requirements, which is in the source code of the module itself,
+and complete control over the final list of dependencies that gets installed.