diskimage-builder/elements/pypi/extra-data.d/00-mount-pypi-mirror

#!/bin/bash

if [ ${DIB_DEBUG_TRACE:-0} -gt 0 ]; then
    set -x
fi
set -eu
set -o pipefail

MIRROR_SOURCE=$DIB_IMAGE_CACHE/pypi/mirror/
if [ -d "$MIRROR_SOURCE" ]; then
    MIRROR_TARGET=$TMP_MOUNT_PATH/tmp/pypi

    sudo mkdir -p $MIRROR_SOURCE $MIRROR_TARGET # dib-lint: safe_sudo
    sudo mount --bind $MIRROR_SOURCE $MIRROR_TARGET # dib-lint: safe_sudo
fi
Allow using a pypi mirror to install via pip. Using a custom pypi mirror can be very convenient, making image builds substantially faster - because we create multiple virtual environments we benefit more than single-virtualenv users would. Change-Id: I997daf1f9477c447e1fb30818aea9e80a49b31a6 2013-08-09 13:13:15 +00:00			`#!/bin/bash`

Standarise tracing for scripts There is a wide variety of tracing options through the various shell scripts. Some use "set -eux", others explicity set xtrace and others do nothing. There is a "-x" option to bin/disk-image-create but it doesn't flow down to the many scripts it calls. This adds a global integer variable set by disk-image-create DIB_DEBUG_TRACE. All scripts have a stanza added to detect this and turn on tracing. Any other tracing methods are rolled into this. So the standard header is --- if [ "${DIB_DEBUG_TRACE:-0}" -gt 0 ]; then set -x fi set -eu set -o pipefail --- Multiple -x options can be specified to dib-create-image, which increases the value of DIB_DEBUG_TRACE. If script authors feel their script should only trace at higher levels, they should modify the "-gt" value. If they feel it should trace by default, they can modify the default value also. Changes to pachset 16 : scripts which currently trace themselves by default have retained this behaviour with DIB_DEBUG_TRACE defaulting to "1". This was done by running [1] on patch set 15. See the thread beginning at [2] dib-lint is also updated to look for the variable being matched. [1] https://gist.github.com/ianw/71bbda9e6acc74ccd0fd [2] http://lists.openstack.org/pipermail/openstack-dev/2014-November/051575.html Change-Id: I6c5a962260741dcf6f89da9a33b96372a719b7b0 2014-09-04 04:56:29 +00:00			`if [ ${DIB_DEBUG_TRACE:-0} -gt 0 ]; then`
			`set -x`
			`fi`
Allow using a pypi mirror to install via pip. Using a custom pypi mirror can be very convenient, making image builds substantially faster - because we create multiple virtual environments we benefit more than single-virtualenv users would. Change-Id: I997daf1f9477c447e1fb30818aea9e80a49b31a6 2013-08-09 13:13:15 +00:00			`set -eu`
set -u and -o pipefail everywhere As with the previous similar changes, this is intended to catch problems as they happen instead of ignoring them and continuing on to potentially fail later. Setting this on all existing scripts will allow us to enforce use via Jenkins. Change-Id: Iad2d490c86dceab148ea9ab08f457c49a5d5352e 2014-04-03 02:24:15 +00:00			`set -o pipefail`
Allow using a pypi mirror to install via pip. Using a custom pypi mirror can be very convenient, making image builds substantially faster - because we create multiple virtual environments we benefit more than single-virtualenv users would. Change-Id: I997daf1f9477c447e1fb30818aea9e80a49b31a6 2013-08-09 13:13:15 +00:00
Use DIB_IMAGE_CACHE everywhere disk-image-create processes a DIB_IMAGE_CACHE variable and exports it, but there are several elements that ignore the value and wrote out the base location themselves. Use the variable everywhere so that it will get overridden everywhere. Change-Id: I00fff354d6c931ad67cf3052d055f0e4604dfdc8 2014-10-05 20:55:34 +00:00			`MIRROR_SOURCE=$DIB_IMAGE_CACHE/pypi/mirror/`
Always bind-mount pypi mirror if dir exists It may be desirable to specify file://tmp/pypi as one of the list of mirrors specified by PYPI_MIRROR_URL, so the local pypi mirror should be mounted as long as the directory exists. Even though the path is mounted, it is only used by pip if there is no specified PYPI_MIRROR_URL, or if one PYPI_MIRROR_URL entry specifies file://tmp/pypi. The pypi element previously assumed that a local mirror would be a complete offline mirror. This change allows for a mixture of local mirror plus remote mirrors. This means that the local mirror could (for example) contain a few locally built pip packages from arbitrary git checkouts for gate testing of those projects. In this scenario, most packages would be downloaded from the configured remote mirrors. This change is required for synchronous gating on the os-*-config tools. Change-Id: I0f6d153c06993ffd0d90bd8ca1717462e8b6b541 2014-05-26 02:20:50 +00:00			`if [ -d "$MIRROR_SOURCE" ]; then`
			`MIRROR_TARGET=$TMP_MOUNT_PATH/tmp/pypi`
Allow using a pypi mirror to install via pip. Using a custom pypi mirror can be very convenient, making image builds substantially faster - because we create multiple virtual environments we benefit more than single-virtualenv users would. Change-Id: I997daf1f9477c447e1fb30818aea9e80a49b31a6 2013-08-09 13:13:15 +00:00
Add a best-effort sudo safety check As motivation for this; we have had two breakouts of dib in recent memory. One was a failure to unmount through symlinks in the core code (I335316019ef948758392b03e91f9869102a472b9) and the other was removing host keys on the build-system (Ib01d71ff9415a0ae04d963f6e380aab9ac2260ce). For the most part, dib runs unprivileged. Bits of the core code are hopefully well tested (modulo bugs like the first one!). We give free reign inside the chroot (although there is still some potential there for adverse external affects via bind mounts). Where we could be a bit safer (and could have prevented at least the second of these breakouts) is with some better checking that the "sudo" calls outside the chroot at least looked sane. This adds a basic check that we're using chroot or image paths when calling sudo in those parts of elements that run outside the chroot. Various files are updated to accomodate this check; mostly by just ignoring it for existing code (I have not audited these calls). Nobody is pretending this type of checking makes dib magically safe, or removes the issues with it needing to do things as root during the build. But this can help find egregious errors like the key removal. Change-Id: I161a5aea1d29dcdc7236f70d372c53246ec73749 2016-04-08 05:46:21 +00:00			`sudo mkdir -p $MIRROR_SOURCE $MIRROR_TARGET # dib-lint: safe_sudo`
			`sudo mount --bind $MIRROR_SOURCE $MIRROR_TARGET # dib-lint: safe_sudo`
Always bind-mount pypi mirror if dir exists It may be desirable to specify file://tmp/pypi as one of the list of mirrors specified by PYPI_MIRROR_URL, so the local pypi mirror should be mounted as long as the directory exists. Even though the path is mounted, it is only used by pip if there is no specified PYPI_MIRROR_URL, or if one PYPI_MIRROR_URL entry specifies file://tmp/pypi. The pypi element previously assumed that a local mirror would be a complete offline mirror. This change allows for a mixture of local mirror plus remote mirrors. This means that the local mirror could (for example) contain a few locally built pip packages from arbitrary git checkouts for gate testing of those projects. In this scenario, most packages would be downloaded from the configured remote mirrors. This change is required for synchronous gating on the os-*-config tools. Change-Id: I0f6d153c06993ffd0d90bd8ca1717462e8b6b541 2014-05-26 02:20:50 +00:00			`fi`