Allow using a pypi mirror to install via pip.

Using a custom pypi mirror can be very convenient, making image builds
substantially faster - because we create multiple virtual
environments we benefit more than single-virtualenv users would.

Change-Id: I997daf1f9477c447e1fb30818aea9e80a49b31a6
This commit is contained in:
Robert Collins 2013-08-10 01:13:15 +12:00
parent d6d941a8f1
commit 97bc5d7853
5 changed files with 103 additions and 0 deletions

View File

@ -112,6 +112,12 @@ C and C++ compilation
Ccache is configured by the base element. Any compilation that honours ccache Ccache is configured by the base element. Any compilation that honours ccache
will be cached. will be cached.
PyPI
----
The pypi element will bind mount a PyPI mirror from the cache dir and configure
pip and easy-install to use it.
Design Design
====== ======

40
elements/pypi/README.md Normal file
View File

@ -0,0 +1,40 @@
Inject a PyPI mirror
====================
Bind mounts a PyPI mirror from ~/.cache/image-create/pypi/mirror into the build
environment and temporarily overwrites /root/.pip.conf and .pydistutils.cfg to
use it.
When online, the master pypi index is supplied as an extra-url, so uncached
dependencies will still be available. When offline, only the mirror is used - a
stale mirror will cause build failures.
[jeepyb](https://github.com/openstack-infra/jeepyb) can be useful in making a
partial PyPI mirror suitable for building images. For instance:
* sudo apt-get install libxml2-dev libxslt-dev libmysqlclient-dev libpq-dev \
libnspr4-dev pkg-config libsqlite3-dev libzmq-dev libffi-dev libldap2-dev \
libsasl2-dev
* pip install https://github.com/openstack-infra/jeepyb
* cat << EOF > mirror.yaml
cache-root: /home/USER/.cache/image-create/pypi/download
mirrors:
- name: openstack
projects:
- https://github.com/openstack/requirements
output: /home/USER/.cache/image-create/pypi/mirror
* mkdir -p /home/USER/.cache/image-create/pypi/{download,mirror}
* run-mirror -b remotes/origin/master --verbose -c mirror.yaml
# This creates and updates the mirror.
If you have additional packages that are not identified in the global openstack
requirements project, you can include them:
* pip install -d ~/.cache/image-create/pypi/download/pip/openstack \
heat-cfntools distribute os-apply-config
run-mirror -b remotes/origin/master --verbose -c mirror.yaml --no-download

View File

@ -0,0 +1,9 @@
#!/bin/bash
set -eu
MIRROR_SOURCE=~/.cache/image-create/pypi/mirror/
MIRROR_TARGET=$TMP_MOUNT_PATH/tmp/pypi
sudo mkdir -p $MIRROR_TARGET
sudo mount --bind $MIRROR_SOURCE $TMP_MOUNT_PATH/tmp/pypi

View File

@ -0,0 +1,16 @@
#!/bin/bash
set -eu
if [ -e ~/.pip/pip.conf.orig ]; then
mv ~/.pip/pip.conf{.orig,}
else
rm ~/.pip/pip.conf
fi
if [ -e ~/.pydistutils.cfg.orig ]; then
mv ~/.pydistutils.cfg{.orig,}
else
rm ~/.pydistutils.cfg
fi

View File

@ -0,0 +1,32 @@
#!/bin/bash
set -eu
PYPIURL=file:///tmp/pypi
mkdir -p ~/.pip
if [ -e ~/.pip/pip.conf ]; then
mv ~/.pip/pip.conf{,.orig}
fi
if [ -e ~/.pydistutils.cfg ]; then
mv ~/.pydistutils.cfg{,.orig}
fi
if [ -z $DIB_OFFLINE ]; then
ONLINE="extra-index-url = https://pypi.python.org/simple"
else
ONLINE=""
fi
cat <<EOF > ~/.pip/pip.conf
[global]
index-url = $PYPIURL
$ONLINE
log = $HOME/pip.log
EOF
cat <<EOF > ~/.pydistutils.cfg
[easy_install]
index_url = $PYPIURL
EOF