pip-and-virtualenv : deprecate source for CentOS 8, new variables

As described inline, deprecate the "source" install for CentOS 8.
Overwriting the packaged tools has long been a pain-point in our
images, and the best outcome is just not to play the game [1].

However, the landscape remains complicated.  For example, RHEL/CentOS
8 introduces the separate "platform-python" binary, which seems like
the right tool to install platform tools like "glean" (simple-init)
with.  However, platform-python doesn't have virtualenv (only the
inbuilt venv).

So that every element doesn't have to hard-code in workarounds for
these various layouts, create two new variables DIB_PYTHON_PIP and
DIB_PYTHON_VIRTUALENV to just "do the right thing".  If you need is
"install a pip package" or "create a virtualenv" this should work on
all the platforms we support.  If you know more specifically what you
want (e.g. must be a python3 virtualenv) then nothing stops elements
calling that directly (e.g. python3 -m virtualenv create); these are
just helper wrappers for base elements that need to be broadly
compatible.

[1] http://lists.openstack.org/pipermail/openstack-infra/2019-September/006483.html

Change-Id: Ia267a60eecfa8f4071dd477d86daebe07e9a7e38

diskimage_builder/elements/pip-and-virtualenv/README.rst

@@ -4,10 +4,6 @@ pip-and-virtualenv
 
 This element installs pip and virtualenv in the image.
 
-.. note:: This element setups and Python 2 and Python 3 environment.
-          This means it will bring in python2 packages, so isn't
-          appropriate if you want a python3 only environment.
-
 Package install
 ===============
 
@@ -16,10 +12,12 @@ from distribution packages.  In this case, ``pip`` and ``virtualenv``
 will be installed *only* for the python version identified by
 ``dib-python`` (i.e. the default python for the platform).
 
-Distribution packages have worked out name-spacing such that only
+Namespacing of the tools will be up to your distribution.  Some
+distribution packages have worked out name-spacing such that only
 python2 or python3 owns common scripts like ``/usr/bin/pip`` (on most
 platforms, ``pip`` refers to python2 pip, and ``pip3`` refers to
-python3 pip, although some may choose the reverse).
+python3 pip, although some may choose the reverse).  Other platforms
+have avoided making a decision and require explicit version suffixes.
 
 To install pip and virtualenv from package::
 
@@ -28,12 +26,28 @@ To install pip and virtualenv from package::
 Source install
 ==============
 
-Source install is the default.  If the source installtype is used,
+.. note:: For source installs this element setups and Python 2 and
-``pip`` and ``virtualenv`` are installed from the latest upstream
+          Python 3 environments.  This means it will bring in python2
-releases.
+          packages, so isn't appropriate if you want a python3 only
+          environment.
 
-Source installs from these tools are not name-spaced.  It is
+.. note:: Source install is considered deprecated for several reasons.
-inconsistent across platforms if the first or last install gets to own
+          Because it makes for a hetrogenous environment between
+          distro packaged tools and upstream it means the final images
+          create bespoke environments that make standarised testing
+          difficult.  The tricks used around holding packages to
+          overwrite them cause difficulty for users of images.  This
+          also brings in Python 2 unconditonally, something not wanted
+          on modern Python 3 only distributions.
+
+Source install is the default on most platforms for historical
+purposes.  The current exception(s) are RHEL8 and CentOS 8.
+
+If the source installtype is used, ``pip`` and ``virtualenv`` are
+installed from the latest upstream releases.
+
+Source installs from upstream releases are not name-spaced.  It is
+inconsistent across platforms if the first or last install will own
 common scripts like ``/usr/bin/pip`` and ``virtualenv``.
 
 To avoid inconsistency, we firstly install the packaged python 2
@@ -54,8 +68,25 @@ The system will be left in the following state:
 Source install is supported on limited platforms.  See the code, but
 this includes Ubuntu and RedHat platforms.
 
-Using the tools
+Environment Variables
-===============
+=====================
+
+To simplify the common-case of "install a package" or "create a
+virtualenv" with the default system Python, the following variables
+are exported by this element:
+
+* ``DIB_PYTHON_PIP``
+* ``DIB_PYTHON_VIRTUALENV``
+
+This will create/install using the ``dib-python`` version for the
+platform (i.e. python2 for older distros, python3 for modern distros).
+Note that on Python 3 platforms it will use the inbuilt ``venv``
+(rather than the ``virtualenv`` package -- if you absolutely need
+features only ``virtualenv`` provides you should call it directly in
+your element; see below).
+
+Explicit use of the tools
+=========================
 
 Due to the essentially unsolvable problem of "who owns the script", it
 is recommended to *not* call ``pip`` or ``virtualenv`` directly.  You

diskimage_builder/elements/pip-and-virtualenv/environment.d/51-pip-and-virutalenv-default.bash

@@ -0,0 +1,33 @@
+# Due to the many historical problems mixing python2/3 versions and
+# upgrading packaged system pip/setuptools/virtualenv binaries with
+# upstream non-packaged versions, we wish to avoid this completely on
+# modern distros.
+if [[ $DISTRO_NAME =~ (centos|rhel) && $DIB_RELEASE -ge 8 ]]; then
+    export DIB_INSTALLTYPE_pip_and_virtualenv=${DIB_INSTALLTYPE_pip_and_virtualenv:-package}
+
+    if [[ ${DIB_INSTALLTYPE_pip_and_virtualenv} == "source" ]]; then
+        echo "*** pip-and-virtualenv does not support 'source' install for $DISTRO_NAME/$DIB_RELEASE"
+        exit 1
+    fi
+fi
+
+# The default variables setup below are only useful during the phases
+# that dib-python exists
+if [[ ! -e /usr/local/bin/dib-python ]]; then
+    return 0
+fi
+
+# NOTE(ianw): you don't want to call "dib-python -m pip" because that
+# can leave behind interpreters #!/usr/local/bin/dib-python in
+# scripts.  De-reference the link
+_dib_python_path=$(readlink /usr/local/bin/dib-python)
+export DIB_PYTHON_PIP="$_dib_python_path -m pip"
+# We make an opinionated, but simplifying decision here that on
+# Python3 platforms, just use venv.  There are some corner cases that
+# the external "virtualenv" package still handles better, but for most
+# purposes "venv" should be fine.
+if [[ $DIB_PYTHON_VERSION == 3 ]]; then
+    export DIB_PYTHON_VIRTUALENV="$_dib_python_path -m venv"
+else
+    export DIB_PYTHON_VIRTUALENV="$_dib_python_path -m virtualenv"
+fi

releasenotes/notes/pip-and-virtualenv-args-5d3f2512edd7f3a3.yaml

@@ -0,0 +1,10 @@
+---
+features:
+  - |
+    The ``pip-and-virtualenv`` element has added variables ``DIB_PYTHON_PIP``
+    and ``DIB_PYTHON_VIRTUALENV`` to make it more straight forward for other
+    elements to install packages or create virtualenv environment
+deprecations:
+  - |
+    Source install for ``pip-and-virtualenv`` is deprecated for RHEL
+    8/CentOS 8