pip-and-virtualenv : install python2 & 3, and default to 2

Recent changes in project-config have shown that we leave the system
in an inconsistent state when installing from source.  On fedora, we
will have installed the python2 packages, but then used $DIB_PYTHON to
install python3 pip from source!

This tries to clarify the situation.  As described in the document,
with package installs, we just install the $DIB_PYTHON packaged
versions.

Source installs want to take over the global namespace.  This is the
price you pay for running the latest versions outside package managers
:) The only sane thing seems to be for us to normalise python2 &
python3 versions of pip, setuptools and virtualenv and then hacking
things such that "/usr/bin/pip" and "/usr/bin/virtalenv" remain
defaulted to python2 versions.

Documentation is added

Change-Id: Ibc6572b89e256d1f48b7fe7c672b8b9524dc704f
This commit is contained in:
Ian Wienand 2017-04-11 12:05:35 +10:00
parent ffd4820d59
commit 79d4113cbe
2 changed files with 113 additions and 10 deletions

View File

@ -2,14 +2,72 @@
pip-and-virtualenv
==================
This element installs pip and virtualenv in the image. If the package
installtype is used then these programs are installed from distribution
packages. If the source installtype is used these programs are installed
from get-pip.py and pip (respectively).
This element installs pip and virtualenv in the image.
To install pip and virtualenv from package:
Package install
===============
If the package installtype is used then these programs are installed
from distribution packages. In this case, ``pip`` and ``virtualenv``
will be installed *only* for the python version identified by
``dib-python`` (i.e. the default python for the platform).
Distribution packages have worked out name-spacing such that only
python2 or python3 owns common scripts like ``/usr/bin/pip`` (on most
platforms, ``pip`` refers to python2 pip, and ``pip3`` refers to
python3 pip, although some may choose the reverse).
To install pip and virtualenv from package::
export DIB_INSTALLTYPE_pip_and_virtualenv=package
Source install
==============
Source install is the default. If the source installtype is used,
``pip`` and ``virtualenv`` are installed from the latest upstream
releases.
Source installs from these tools are not name-spaced. It is
inconsistent across platforms if the first or last install gets to own
common scripts like ``/usr/bin/pip`` and ``virtualenv``.
To avoid inconsistency, we firstly install the packaged python 2
**and** 3 versions of ``pip`` and ``virtualenv``. This prevents a
later install of these distribution packages conflicting with the
source install. We then overwrite ``pip`` and ``virtualenv`` via
``get-pip.py`` and ``pip`` respectively.
The system will be left in the following state:
* ``/usr/bin/pip`` : python2 pip
* ``/usr/bin/pip2`` : python2 pip (same as prior)
* ``/usr/bin/pip3`` : python3 pip
* ``/usr/bin/virtualenv`` : python2 virtualenv
(note python3 ``virtualenv`` script is *not* installed, see below)
Using the tools
===============
Due to the essentially unsolvable problem of "who owns the script", it
is recommended to *not* call ``pip`` or ``virtualenv`` directly. You
can directly call them with the ``-m`` argument to the python
interpreter you wish to install with.
For example, to create a python3 environment do::
# python3 -m virtualenv myenv
# myenv/bin/pip install mytool
To install a python2 tool from pip::
# python2 -m pip install mytool
In this way, you can always know which interpreter is being used (and
affected by) the call.
Ordering
========
Any element that uses these commands must be designated as
05-* or higher to ensure that they are first installed.

View File

@ -7,6 +7,14 @@ set -eu
set -o pipefail
if [[ $DISTRO_NAME =~ (opensuse|fedora|centos|centos7|rhel|rhel7) ]]; then
_do_py3=0
packages="python-virtualenv python-pip python-setuptools"
if [[ $DISTRO_NAME =~ (fedora) ]]; then
_do_py3=1
packages+=" python3-virtualenv python3-pip python3-setuptools"
fi
# GENERAL WARNING : mixing packaged python libraries with
# pip-installed versions always creates issues. Upstream
# openstack-infra uses this a lot (especially devstack) but be
@ -19,11 +27,11 @@ if [[ $DISTRO_NAME =~ (opensuse|fedora|centos|centos7|rhel|rhel7) ]]; then
if [[ $DISTRO_NAME = opensuse ]]; then
zypper -n install python-virtualenv python-pip python-setuptools
else
${YUM:-yum} install -y python-virtualenv python-pip python-setuptools
${YUM:-yum} install -y $packages
fi
# install the latest pip; this overwrites packaged pip
${DIB_PYTHON} /tmp/get-pip.py
# install the latest python2 pip; this overwrites packaged pip
python /tmp/get-pip.py
# pip and setuptools are closely related; we want to ensure the
# latest for sanity. Because distro packages don't include enough
@ -38,10 +46,38 @@ if [[ $DISTRO_NAME =~ (opensuse|fedora|centos|centos7|rhel|rhel7) ]]; then
rm -rf /usr/lib/python2.7/site-packages/setuptools*
pip install -U setuptools
if [[ $_do_py3 -eq 1 ]]; then
# Repeat above for python3
# You would think that installing python3 bits first, then
# python2 would work -- alas get-pip.py doesn't seem to leave
# python3 alone:
# https://github.com/pypa/pip/issues/4435
python3 /tmp/get-pip.py
rm -rf /usr/lib/python3.?/site-packages/setuptools*
pip3 install -U setuptools
# reclaim /usr/bin/pip back to pip2
ln -sf /usr/bin/pip2 /usr/bin/pip
fi
# now install latest virtualenv. it vendors stuff it needs so
# doesn't have issues with other system packages.
# python[2|3]-virtualenv package has installed versioned scripts
# (/usr/bin/virtualenv-[2|3]) but upstream does not! (see [2]).
# For consistency, clear them out and then reinstall so we're just
# left with python2's version
# [2] http://pkgs.fedoraproject.org/cgit/rpms/python-virtualenv.git/tree/python-virtualenv.spec#n116)
rm /usr/bin/virtualenv*
if [[ $_do_py3 -eq 1 ]]; then
pip3 install -U virtualenv
fi
pip install -U virtualenv
# at this point, we should have the latest
# pip/setuptools/virtualenv packages for python2 & 3, and
# "/usr/bin/pip" and "/usr/bin/virtualenv" should be python2
# versions.
if [[ $DISTRO_NAME = opensuse ]]; then
for pkg in virtualenv pip setuptools; do
cat - >> /etc/zypp/locks <<EOF
@ -62,9 +98,18 @@ EOF
else
die "No conf to modify?"
fi
echo "exclude=python-virtualenv,python-pip,python-setuptools" >> ${conf}
echo "exclude=$packages" >> ${conf}
fi
else
${DIB_PYTHON} /tmp/get-pip.py
# pre-install packages, we will overwrite with latest below
apt-get -y install python-pip python3-pip \
python-virtualenv python3-virtualenv
# These install into /usr/local/bin so override any packages, even
# if installed later.
python3 /tmp/get-pip.py
python2 /tmp/get-pip.py
pip3 install virtualenv
pip install virtualenv
fi