diskimage-builder/elements/pypi
Robert Collins 3b7d6694e9 Document a little the concerns for operators.
Change-Id: I2429371d3c5d931d46674c4e46d99599fda9f42e
2014-04-05 20:17:30 +13:00
..
extra-data.d Permit using arbitrary PyPI urls. 2014-02-19 14:05:31 +13:00
post-install.d Allow using a pypi mirror to install via pip. 2013-08-10 22:07:30 +12:00
pre-install.d Permit use of wheel mirrors in pypi element. 2014-03-25 16:48:18 +13:00
README.md Document a little the concerns for operators. 2014-04-05 20:17:30 +13:00

Inject a PyPI mirror

Use a custom PyPI mirror to build images. The default is to bind mount one from ~/.cache/image-create/pypi/mirror into the build environment. The element temporarily overwrites /root/.pip.conf and .pydistutils.cfg to use it.

When online, the official pypi.python.org pypi index is supplied as an extra-url, so uncached dependencies will still be available. When offline, only the mirror is used - be warned that a stale mirror will cause build failures. To disable the pypi.python.org index without using --offline (e.g. when working behind a corporate firewall that prohibits pypi.python.org) set DIB_NO_PYPI_PIP to any non-empty value.

To use an arbitrary mirror set PYPI_MIRROR_URL=http[s]://somevalue/

Additional mirrors can be added by exporting PYPI_MIRROR_URL_1=... etc. Only the one mirror can be used by easy-install, but since wheels need to be in the first mirror to be used, the last listed mirror is used as the pydistutils index. NB: The sort order for these variables is a simple string sort - if you have more than 9 additional mirrors, some care will be needed.

A typical use of this element is thus: export PYPI_MIRROR_URL=http://site/pypi/Ubuntu-13.10 export PYPI_MIRROR_URL_1=http://site/pypi/

pypi-mirror can be useful in making a partial PyPI mirror suitable for building images. For instance:

  • sudo apt-get install libxml2-dev libxslt-dev libmysqlclient-dev libpq-dev
    libnspr4-dev pkg-config libsqlite3-dev libzmq-dev libffi-dev libldap2-dev
    libsasl2-dev

  • pip install git+https://git.openstack.org/openstack-infra/pypi-mirror

  • cat << EOF > mirror.yaml cache-root: /home/USER/.cache/image-create/pypi/download

    mirrors:

  • mkdir -p /home/USER/.cache/image-create/pypi/{download,mirror}

  • run-mirror -b remotes/origin/master --verbose -c mirror.yaml

    This creates and updates the mirror.

If you have additional packages that are not identified in the global openstack requirements project, you can include them:

  • pip install -d ~/.cache/image-create/pypi/download/pip/openstack
    heat-cfntools distribute os-apply-config run-mirror -b remotes/origin/master --verbose -c mirror.yaml --no-download

A dedicated user with no privileges may be desirable when running this for unattended installs, since pypi-mirror will compile fairly arbitrary code from the internet.