From 18215274d895b4b6104c5d977c00b623568646b2 Mon Sep 17 00:00:00 2001 From: Ian Wienand Date: Mon, 30 Sep 2019 03:50:44 +0000 Subject: [PATCH] pip-and-virtualenv : deprecate source for CentOS 8, new variables As described inline, deprecate the "source" install for CentOS 8. Overwriting the packaged tools has long been a pain-point in our images, and the best outcome is just not to play the game [1]. However, the landscape remains complicated. For example, RHEL/CentOS 8 introduces the separate "platform-python" binary, which seems like the right tool to install platform tools like "glean" (simple-init) with. However, platform-python doesn't have virtualenv (only the inbuilt venv). So that every element doesn't have to hard-code in workarounds for these various layouts, create two new variables DIB_PYTHON_PIP and DIB_PYTHON_VIRTUALENV to just "do the right thing". If you need is "install a pip package" or "create a virtualenv" this should work on all the platforms we support. If you know more specifically what you want (e.g. must be a python3 virtualenv) then nothing stops elements calling that directly (e.g. python3 -m virtualenv create); these are just helper wrappers for base elements that need to be broadly compatible. [1] http://lists.openstack.org/pipermail/openstack-infra/2019-September/006483.html Change-Id: Ia267a60eecfa8f4071dd477d86daebe07e9a7e38 --- .../elements/pip-and-virtualenv/README.rst | 57 ++++++++++++++----- .../51-pip-and-virutalenv-default.bash | 33 +++++++++++ ...-and-virtualenv-args-5d3f2512edd7f3a3.yaml | 10 ++++ 3 files changed, 87 insertions(+), 13 deletions(-) create mode 100644 diskimage_builder/elements/pip-and-virtualenv/environment.d/51-pip-and-virutalenv-default.bash create mode 100644 releasenotes/notes/pip-and-virtualenv-args-5d3f2512edd7f3a3.yaml diff --git a/diskimage_builder/elements/pip-and-virtualenv/README.rst b/diskimage_builder/elements/pip-and-virtualenv/README.rst index e056a216..c983cb44 100644 --- a/diskimage_builder/elements/pip-and-virtualenv/README.rst +++ b/diskimage_builder/elements/pip-and-virtualenv/README.rst @@ -4,10 +4,6 @@ pip-and-virtualenv This element installs pip and virtualenv in the image. -.. note:: This element setups and Python 2 and Python 3 environment. - This means it will bring in python2 packages, so isn't - appropriate if you want a python3 only environment. - Package install =============== @@ -16,10 +12,12 @@ from distribution packages. In this case, ``pip`` and ``virtualenv`` will be installed *only* for the python version identified by ``dib-python`` (i.e. the default python for the platform). -Distribution packages have worked out name-spacing such that only +Namespacing of the tools will be up to your distribution. Some +distribution packages have worked out name-spacing such that only python2 or python3 owns common scripts like ``/usr/bin/pip`` (on most platforms, ``pip`` refers to python2 pip, and ``pip3`` refers to -python3 pip, although some may choose the reverse). +python3 pip, although some may choose the reverse). Other platforms +have avoided making a decision and require explicit version suffixes. To install pip and virtualenv from package:: @@ -28,12 +26,28 @@ To install pip and virtualenv from package:: Source install ============== -Source install is the default. If the source installtype is used, -``pip`` and ``virtualenv`` are installed from the latest upstream -releases. +.. note:: For source installs this element setups and Python 2 and + Python 3 environments. This means it will bring in python2 + packages, so isn't appropriate if you want a python3 only + environment. -Source installs from these tools are not name-spaced. It is -inconsistent across platforms if the first or last install gets to own +.. note:: Source install is considered deprecated for several reasons. + Because it makes for a hetrogenous environment between + distro packaged tools and upstream it means the final images + create bespoke environments that make standarised testing + difficult. The tricks used around holding packages to + overwrite them cause difficulty for users of images. This + also brings in Python 2 unconditonally, something not wanted + on modern Python 3 only distributions. + +Source install is the default on most platforms for historical +purposes. The current exception(s) are RHEL8 and CentOS 8. + +If the source installtype is used, ``pip`` and ``virtualenv`` are +installed from the latest upstream releases. + +Source installs from upstream releases are not name-spaced. It is +inconsistent across platforms if the first or last install will own common scripts like ``/usr/bin/pip`` and ``virtualenv``. To avoid inconsistency, we firstly install the packaged python 2 @@ -54,8 +68,25 @@ The system will be left in the following state: Source install is supported on limited platforms. See the code, but this includes Ubuntu and RedHat platforms. -Using the tools -=============== +Environment Variables +===================== + +To simplify the common-case of "install a package" or "create a +virtualenv" with the default system Python, the following variables +are exported by this element: + +* ``DIB_PYTHON_PIP`` +* ``DIB_PYTHON_VIRTUALENV`` + +This will create/install using the ``dib-python`` version for the +platform (i.e. python2 for older distros, python3 for modern distros). +Note that on Python 3 platforms it will use the inbuilt ``venv`` +(rather than the ``virtualenv`` package -- if you absolutely need +features only ``virtualenv`` provides you should call it directly in +your element; see below). + +Explicit use of the tools +========================= Due to the essentially unsolvable problem of "who owns the script", it is recommended to *not* call ``pip`` or ``virtualenv`` directly. You diff --git a/diskimage_builder/elements/pip-and-virtualenv/environment.d/51-pip-and-virutalenv-default.bash b/diskimage_builder/elements/pip-and-virtualenv/environment.d/51-pip-and-virutalenv-default.bash new file mode 100644 index 00000000..13c83902 --- /dev/null +++ b/diskimage_builder/elements/pip-and-virtualenv/environment.d/51-pip-and-virutalenv-default.bash @@ -0,0 +1,33 @@ +# Due to the many historical problems mixing python2/3 versions and +# upgrading packaged system pip/setuptools/virtualenv binaries with +# upstream non-packaged versions, we wish to avoid this completely on +# modern distros. +if [[ $DISTRO_NAME =~ (centos|rhel) && $DIB_RELEASE -ge 8 ]]; then + export DIB_INSTALLTYPE_pip_and_virtualenv=${DIB_INSTALLTYPE_pip_and_virtualenv:-package} + + if [[ ${DIB_INSTALLTYPE_pip_and_virtualenv} == "source" ]]; then + echo "*** pip-and-virtualenv does not support 'source' install for $DISTRO_NAME/$DIB_RELEASE" + exit 1 + fi +fi + +# The default variables setup below are only useful during the phases +# that dib-python exists +if [[ ! -e /usr/local/bin/dib-python ]]; then + return 0 +fi + +# NOTE(ianw): you don't want to call "dib-python -m pip" because that +# can leave behind interpreters #!/usr/local/bin/dib-python in +# scripts. De-reference the link +_dib_python_path=$(readlink /usr/local/bin/dib-python) +export DIB_PYTHON_PIP="$_dib_python_path -m pip" +# We make an opinionated, but simplifying decision here that on +# Python3 platforms, just use venv. There are some corner cases that +# the external "virtualenv" package still handles better, but for most +# purposes "venv" should be fine. +if [[ $DIB_PYTHON_VERSION == 3 ]]; then + export DIB_PYTHON_VIRTUALENV="$_dib_python_path -m venv" +else + export DIB_PYTHON_VIRTUALENV="$_dib_python_path -m virtualenv" +fi diff --git a/releasenotes/notes/pip-and-virtualenv-args-5d3f2512edd7f3a3.yaml b/releasenotes/notes/pip-and-virtualenv-args-5d3f2512edd7f3a3.yaml new file mode 100644 index 00000000..4f85fd88 --- /dev/null +++ b/releasenotes/notes/pip-and-virtualenv-args-5d3f2512edd7f3a3.yaml @@ -0,0 +1,10 @@ +--- +features: + - | + The ``pip-and-virtualenv`` element has added variables ``DIB_PYTHON_PIP`` + and ``DIB_PYTHON_VIRTUALENV`` to make it more straight forward for other + elements to install packages or create virtualenv environment +deprecations: + - | + Source install for ``pip-and-virtualenv`` is deprecated for RHEL + 8/CentOS 8 \ No newline at end of file