pip-and-virtualenv : install python2 & 3, and default to 2

Recent changes in project-config have shown that we leave the system
in an inconsistent state when installing from source.  On fedora, we
will have installed the python2 packages, but then used $DIB_PYTHON to
install python3 pip from source!

This tries to clarify the situation.  As described in the document,
with package installs, we just install the $DIB_PYTHON packaged
versions.

Source installs want to take over the global namespace.  This is the
price you pay for running the latest versions outside package managers
:) The only sane thing seems to be for us to normalise python2 &
python3 versions of pip, setuptools and virtualenv and then hacking
things such that "/usr/bin/pip" and "/usr/bin/virtalenv" remain
defaulted to python2 versions.

Documentation is added

Change-Id: Ibc6572b89e256d1f48b7fe7c672b8b9524dc704f

diskimage_builder/elements/pip-and-virtualenv/README.rst

@@ -2,14 +2,72 @@
 pip-and-virtualenv
 ==================
 
-This element installs pip and virtualenv in the image. If the package
-installtype is used then these programs are installed from distribution
-packages. If the source installtype is used these programs are installed
-from get-pip.py and pip (respectively).
+This element installs pip and virtualenv in the image.
 
-To install pip and virtualenv from package:
+Package install
+===============
+
+If the package installtype is used then these programs are installed
+from distribution packages.  In this case, ``pip`` and ``virtualenv``
+will be installed *only* for the python version identified by
+``dib-python`` (i.e. the default python for the platform).
+
+Distribution packages have worked out name-spacing such that only
+python2 or python3 owns common scripts like ``/usr/bin/pip`` (on most
+platforms, ``pip`` refers to python2 pip, and ``pip3`` refers to
+python3 pip, although some may choose the reverse).
+
+To install pip and virtualenv from package::
 
   export DIB_INSTALLTYPE_pip_and_virtualenv=package
 
+Source install
+==============
+
+Source install is the default.  If the source installtype is used,
+``pip`` and ``virtualenv`` are installed from the latest upstream
+releases.
+
+Source installs from these tools are not name-spaced.  It is
+inconsistent across platforms if the first or last install gets to own
+common scripts like ``/usr/bin/pip`` and ``virtualenv``.
+
+To avoid inconsistency, we firstly install the packaged python 2
+**and** 3 versions of ``pip`` and ``virtualenv``.  This prevents a
+later install of these distribution packages conflicting with the
+source install.  We then overwrite ``pip`` and ``virtualenv`` via
+``get-pip.py`` and ``pip`` respectively.
+
+The system will be left in the following state:
+
+* ``/usr/bin/pip`` : python2 pip
+* ``/usr/bin/pip2`` : python2 pip (same as prior)
+* ``/usr/bin/pip3`` : python3 pip
+* ``/usr/bin/virtualenv`` : python2 virtualenv
+
+(note python3 ``virtualenv`` script is *not* installed, see below)
+
+Using the tools
+===============
+
+Due to the essentially unsolvable problem of "who owns the script", it
+is recommended to *not* call ``pip`` or ``virtualenv`` directly.  You
+can directly call them with the ``-m`` argument to the python
+interpreter you wish to install with.
+
+For example, to create a python3 environment do::
+
+  # python3 -m virtualenv myenv
+  # myenv/bin/pip install mytool
+
+To install a python2 tool from pip::
+
+  # python2 -m pip install mytool
+
+In this way, you can always know which interpreter is being used (and
+affected by) the call.
+
+Ordering
+========
 Any element that uses these commands must be designated as
 05-* or higher to ensure that they are first installed.

diskimage_builder/elements/pip-and-virtualenv/install.d/pip-and-virtualenv-source-install/04-install-pip

@@ -7,6 +7,14 @@ set -eu
 set -o pipefail
 
 if [[ $DISTRO_NAME =~ (opensuse|fedora|centos|centos7|rhel|rhel7) ]]; then
+
+    _do_py3=0
+    packages="python-virtualenv python-pip python-setuptools"
+    if [[ $DISTRO_NAME =~ (fedora) ]]; then
+        _do_py3=1
+        packages+=" python3-virtualenv python3-pip python3-setuptools"
+    fi
+
     # GENERAL WARNING : mixing packaged python libraries with
     # pip-installed versions always creates issues.  Upstream
     # openstack-infra uses this a lot (especially devstack) but be
@@ -19,11 +27,11 @@ if [[ $DISTRO_NAME =~ (opensuse|fedora|centos|centos7|rhel|rhel7) ]]; then
     if [[ $DISTRO_NAME = opensuse ]]; then
         zypper -n install python-virtualenv python-pip python-setuptools
     else
-        ${YUM:-yum} install -y python-virtualenv python-pip python-setuptools
+        ${YUM:-yum} install -y $packages
     fi
 
-    # install the latest pip; this overwrites packaged pip
-    ${DIB_PYTHON} /tmp/get-pip.py
+    # install the latest python2 pip; this overwrites packaged pip
+    python /tmp/get-pip.py
 
     # pip and setuptools are closely related; we want to ensure the
     # latest for sanity.  Because distro packages don't include enough
@@ -38,10 +46,38 @@ if [[ $DISTRO_NAME =~ (opensuse|fedora|centos|centos7|rhel|rhel7) ]]; then
     rm -rf /usr/lib/python2.7/site-packages/setuptools*
     pip install -U setuptools
 
+    if [[ $_do_py3 -eq 1 ]]; then
+        # Repeat above for python3
+        # You would think that installing python3 bits first, then
+        # python2 would work -- alas get-pip.py doesn't seem to leave
+        # python3 alone:
+        #  https://github.com/pypa/pip/issues/4435
+        python3 /tmp/get-pip.py
+        rm -rf /usr/lib/python3.?/site-packages/setuptools*
+        pip3 install -U setuptools
+        # reclaim /usr/bin/pip back to pip2
+        ln -sf /usr/bin/pip2 /usr/bin/pip
+    fi
+
     # now install latest virtualenv.  it vendors stuff it needs so
     # doesn't have issues with other system packages.
+
+    # python[2|3]-virtualenv package has installed versioned scripts
+    # (/usr/bin/virtualenv-[2|3]) but upstream does not!  (see [2]).
+    # For consistency, clear them out and then reinstall so we're just
+    # left with python2's version
+    # [2] http://pkgs.fedoraproject.org/cgit/rpms/python-virtualenv.git/tree/python-virtualenv.spec#n116)
+    rm /usr/bin/virtualenv*
+    if [[ $_do_py3 -eq 1 ]]; then
+        pip3 install -U virtualenv
+    fi
     pip install -U virtualenv
 
+    # at this point, we should have the latest
+    # pip/setuptools/virtualenv packages for python2 & 3, and
+    # "/usr/bin/pip" and "/usr/bin/virtualenv" should be python2
+    # versions.
+
     if [[ $DISTRO_NAME = opensuse ]]; then
         for pkg in virtualenv pip setuptools; do
             cat - >> /etc/zypp/locks <<EOF
@@ -62,9 +98,18 @@ EOF
         else
             die "No conf to modify?"
         fi
-        echo "exclude=python-virtualenv,python-pip,python-setuptools" >> ${conf}
+        echo "exclude=$packages" >> ${conf}
     fi
 else
-    ${DIB_PYTHON} /tmp/get-pip.py
+    # pre-install packages, we will overwrite with latest below
+    apt-get -y install python-pip python3-pip \
+            python-virtualenv python3-virtualenv
+
+    # These install into /usr/local/bin so override any packages, even
+    # if installed later.
+    python3 /tmp/get-pip.py
+    python2 /tmp/get-pip.py
+
+    pip3 install virtualenv
     pip install virtualenv
 fi