pip-and-virtualenv : install python2 & 3, and default to 2
Recent changes in project-config have shown that we leave the system in an inconsistent state when installing from source. On fedora, we will have installed the python2 packages, but then used $DIB_PYTHON to install python3 pip from source! This tries to clarify the situation. As described in the document, with package installs, we just install the $DIB_PYTHON packaged versions. Source installs want to take over the global namespace. This is the price you pay for running the latest versions outside package managers :) The only sane thing seems to be for us to normalise python2 & python3 versions of pip, setuptools and virtualenv and then hacking things such that "/usr/bin/pip" and "/usr/bin/virtalenv" remain defaulted to python2 versions. Documentation is added Change-Id: Ibc6572b89e256d1f48b7fe7c672b8b9524dc704f
This commit is contained in:
parent
ffd4820d59
commit
79d4113cbe
@ -2,14 +2,72 @@
|
|||||||
pip-and-virtualenv
|
pip-and-virtualenv
|
||||||
==================
|
==================
|
||||||
|
|
||||||
This element installs pip and virtualenv in the image. If the package
|
This element installs pip and virtualenv in the image.
|
||||||
installtype is used then these programs are installed from distribution
|
|
||||||
packages. If the source installtype is used these programs are installed
|
|
||||||
from get-pip.py and pip (respectively).
|
|
||||||
|
|
||||||
To install pip and virtualenv from package:
|
Package install
|
||||||
|
===============
|
||||||
|
|
||||||
|
If the package installtype is used then these programs are installed
|
||||||
|
from distribution packages. In this case, ``pip`` and ``virtualenv``
|
||||||
|
will be installed *only* for the python version identified by
|
||||||
|
``dib-python`` (i.e. the default python for the platform).
|
||||||
|
|
||||||
|
Distribution packages have worked out name-spacing such that only
|
||||||
|
python2 or python3 owns common scripts like ``/usr/bin/pip`` (on most
|
||||||
|
platforms, ``pip`` refers to python2 pip, and ``pip3`` refers to
|
||||||
|
python3 pip, although some may choose the reverse).
|
||||||
|
|
||||||
|
To install pip and virtualenv from package::
|
||||||
|
|
||||||
export DIB_INSTALLTYPE_pip_and_virtualenv=package
|
export DIB_INSTALLTYPE_pip_and_virtualenv=package
|
||||||
|
|
||||||
|
Source install
|
||||||
|
==============
|
||||||
|
|
||||||
|
Source install is the default. If the source installtype is used,
|
||||||
|
``pip`` and ``virtualenv`` are installed from the latest upstream
|
||||||
|
releases.
|
||||||
|
|
||||||
|
Source installs from these tools are not name-spaced. It is
|
||||||
|
inconsistent across platforms if the first or last install gets to own
|
||||||
|
common scripts like ``/usr/bin/pip`` and ``virtualenv``.
|
||||||
|
|
||||||
|
To avoid inconsistency, we firstly install the packaged python 2
|
||||||
|
**and** 3 versions of ``pip`` and ``virtualenv``. This prevents a
|
||||||
|
later install of these distribution packages conflicting with the
|
||||||
|
source install. We then overwrite ``pip`` and ``virtualenv`` via
|
||||||
|
``get-pip.py`` and ``pip`` respectively.
|
||||||
|
|
||||||
|
The system will be left in the following state:
|
||||||
|
|
||||||
|
* ``/usr/bin/pip`` : python2 pip
|
||||||
|
* ``/usr/bin/pip2`` : python2 pip (same as prior)
|
||||||
|
* ``/usr/bin/pip3`` : python3 pip
|
||||||
|
* ``/usr/bin/virtualenv`` : python2 virtualenv
|
||||||
|
|
||||||
|
(note python3 ``virtualenv`` script is *not* installed, see below)
|
||||||
|
|
||||||
|
Using the tools
|
||||||
|
===============
|
||||||
|
|
||||||
|
Due to the essentially unsolvable problem of "who owns the script", it
|
||||||
|
is recommended to *not* call ``pip`` or ``virtualenv`` directly. You
|
||||||
|
can directly call them with the ``-m`` argument to the python
|
||||||
|
interpreter you wish to install with.
|
||||||
|
|
||||||
|
For example, to create a python3 environment do::
|
||||||
|
|
||||||
|
# python3 -m virtualenv myenv
|
||||||
|
# myenv/bin/pip install mytool
|
||||||
|
|
||||||
|
To install a python2 tool from pip::
|
||||||
|
|
||||||
|
# python2 -m pip install mytool
|
||||||
|
|
||||||
|
In this way, you can always know which interpreter is being used (and
|
||||||
|
affected by) the call.
|
||||||
|
|
||||||
|
Ordering
|
||||||
|
========
|
||||||
Any element that uses these commands must be designated as
|
Any element that uses these commands must be designated as
|
||||||
05-* or higher to ensure that they are first installed.
|
05-* or higher to ensure that they are first installed.
|
||||||
|
@ -7,6 +7,14 @@ set -eu
|
|||||||
set -o pipefail
|
set -o pipefail
|
||||||
|
|
||||||
if [[ $DISTRO_NAME =~ (opensuse|fedora|centos|centos7|rhel|rhel7) ]]; then
|
if [[ $DISTRO_NAME =~ (opensuse|fedora|centos|centos7|rhel|rhel7) ]]; then
|
||||||
|
|
||||||
|
_do_py3=0
|
||||||
|
packages="python-virtualenv python-pip python-setuptools"
|
||||||
|
if [[ $DISTRO_NAME =~ (fedora) ]]; then
|
||||||
|
_do_py3=1
|
||||||
|
packages+=" python3-virtualenv python3-pip python3-setuptools"
|
||||||
|
fi
|
||||||
|
|
||||||
# GENERAL WARNING : mixing packaged python libraries with
|
# GENERAL WARNING : mixing packaged python libraries with
|
||||||
# pip-installed versions always creates issues. Upstream
|
# pip-installed versions always creates issues. Upstream
|
||||||
# openstack-infra uses this a lot (especially devstack) but be
|
# openstack-infra uses this a lot (especially devstack) but be
|
||||||
@ -19,11 +27,11 @@ if [[ $DISTRO_NAME =~ (opensuse|fedora|centos|centos7|rhel|rhel7) ]]; then
|
|||||||
if [[ $DISTRO_NAME = opensuse ]]; then
|
if [[ $DISTRO_NAME = opensuse ]]; then
|
||||||
zypper -n install python-virtualenv python-pip python-setuptools
|
zypper -n install python-virtualenv python-pip python-setuptools
|
||||||
else
|
else
|
||||||
${YUM:-yum} install -y python-virtualenv python-pip python-setuptools
|
${YUM:-yum} install -y $packages
|
||||||
fi
|
fi
|
||||||
|
|
||||||
# install the latest pip; this overwrites packaged pip
|
# install the latest python2 pip; this overwrites packaged pip
|
||||||
${DIB_PYTHON} /tmp/get-pip.py
|
python /tmp/get-pip.py
|
||||||
|
|
||||||
# pip and setuptools are closely related; we want to ensure the
|
# pip and setuptools are closely related; we want to ensure the
|
||||||
# latest for sanity. Because distro packages don't include enough
|
# latest for sanity. Because distro packages don't include enough
|
||||||
@ -38,10 +46,38 @@ if [[ $DISTRO_NAME =~ (opensuse|fedora|centos|centos7|rhel|rhel7) ]]; then
|
|||||||
rm -rf /usr/lib/python2.7/site-packages/setuptools*
|
rm -rf /usr/lib/python2.7/site-packages/setuptools*
|
||||||
pip install -U setuptools
|
pip install -U setuptools
|
||||||
|
|
||||||
|
if [[ $_do_py3 -eq 1 ]]; then
|
||||||
|
# Repeat above for python3
|
||||||
|
# You would think that installing python3 bits first, then
|
||||||
|
# python2 would work -- alas get-pip.py doesn't seem to leave
|
||||||
|
# python3 alone:
|
||||||
|
# https://github.com/pypa/pip/issues/4435
|
||||||
|
python3 /tmp/get-pip.py
|
||||||
|
rm -rf /usr/lib/python3.?/site-packages/setuptools*
|
||||||
|
pip3 install -U setuptools
|
||||||
|
# reclaim /usr/bin/pip back to pip2
|
||||||
|
ln -sf /usr/bin/pip2 /usr/bin/pip
|
||||||
|
fi
|
||||||
|
|
||||||
# now install latest virtualenv. it vendors stuff it needs so
|
# now install latest virtualenv. it vendors stuff it needs so
|
||||||
# doesn't have issues with other system packages.
|
# doesn't have issues with other system packages.
|
||||||
|
|
||||||
|
# python[2|3]-virtualenv package has installed versioned scripts
|
||||||
|
# (/usr/bin/virtualenv-[2|3]) but upstream does not! (see [2]).
|
||||||
|
# For consistency, clear them out and then reinstall so we're just
|
||||||
|
# left with python2's version
|
||||||
|
# [2] http://pkgs.fedoraproject.org/cgit/rpms/python-virtualenv.git/tree/python-virtualenv.spec#n116)
|
||||||
|
rm /usr/bin/virtualenv*
|
||||||
|
if [[ $_do_py3 -eq 1 ]]; then
|
||||||
|
pip3 install -U virtualenv
|
||||||
|
fi
|
||||||
pip install -U virtualenv
|
pip install -U virtualenv
|
||||||
|
|
||||||
|
# at this point, we should have the latest
|
||||||
|
# pip/setuptools/virtualenv packages for python2 & 3, and
|
||||||
|
# "/usr/bin/pip" and "/usr/bin/virtualenv" should be python2
|
||||||
|
# versions.
|
||||||
|
|
||||||
if [[ $DISTRO_NAME = opensuse ]]; then
|
if [[ $DISTRO_NAME = opensuse ]]; then
|
||||||
for pkg in virtualenv pip setuptools; do
|
for pkg in virtualenv pip setuptools; do
|
||||||
cat - >> /etc/zypp/locks <<EOF
|
cat - >> /etc/zypp/locks <<EOF
|
||||||
@ -62,9 +98,18 @@ EOF
|
|||||||
else
|
else
|
||||||
die "No conf to modify?"
|
die "No conf to modify?"
|
||||||
fi
|
fi
|
||||||
echo "exclude=python-virtualenv,python-pip,python-setuptools" >> ${conf}
|
echo "exclude=$packages" >> ${conf}
|
||||||
fi
|
fi
|
||||||
else
|
else
|
||||||
${DIB_PYTHON} /tmp/get-pip.py
|
# pre-install packages, we will overwrite with latest below
|
||||||
|
apt-get -y install python-pip python3-pip \
|
||||||
|
python-virtualenv python3-virtualenv
|
||||||
|
|
||||||
|
# These install into /usr/local/bin so override any packages, even
|
||||||
|
# if installed later.
|
||||||
|
python3 /tmp/get-pip.py
|
||||||
|
python2 /tmp/get-pip.py
|
||||||
|
|
||||||
|
pip3 install virtualenv
|
||||||
pip install virtualenv
|
pip install virtualenv
|
||||||
fi
|
fi
|
||||||
|
Loading…
Reference in New Issue
Block a user