sig_cloud/diskimage-builder
2
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
ca53af1184
diskimage-builder/elements/pypi/README.rst

62 lines
2.5 KiB
ReStructuredText
Raw Normal View History

Create docs site containing element READMEs We currently do not have the ability to create a docs site which outlines all the elements. Change-Id: I77ccf61e0c4b1509b3e7ce9b8f15ea5ccfd50d9b
2015-02-07 10:30:02 +00:00
====
pypi
====
Allow using a pypi mirror to install via pip. Using a custom pypi mirror can be very convenient, making image builds substantially faster - because we create multiple virtual environments we benefit more than single-virtualenv users would. Change-Id: I997daf1f9477c447e1fb30818aea9e80a49b31a6
2013-08-09 13:13:15 +00:00
Inject a PyPI mirror
====================
Permit using arbitrary PyPI urls. The pypi element is cool, but some folk have local network mirrors which we should permit them to use. Change-Id: Ie840ad1184e72b0e01966eee0298cfd6511b6c19
2014-02-19 01:03:56 +00:00
Use a custom PyPI mirror to build images. The default is to bind mount one from
Always bind-mount pypi mirror if dir exists It may be desirable to specify file://tmp/pypi as one of the list of mirrors specified by PYPI_MIRROR_URL, so the local pypi mirror should be mounted as long as the directory exists. Even though the path is mounted, it is only used by pip if there is no specified PYPI_MIRROR_URL, or if one PYPI_MIRROR_URL entry specifies file://tmp/pypi. The pypi element previously assumed that a local mirror would be a complete offline mirror. This change allows for a mixture of local mirror plus remote mirrors. This means that the local mirror could (for example) contain a few locally built pip packages from arbitrary git checkouts for gate testing of those projects. In this scenario, most packages would be downloaded from the configured remote mirrors. This change is required for synchronous gating on the os-*-config tools. Change-Id: I0f6d153c06993ffd0d90bd8ca1717462e8b6b541
2014-05-26 02:20:50 +00:00
~/.cache/image-create/pypi/mirror into the build environment as mirror URL
file:///tmp/pypi. The element temporarily overwrites /root/.pip.conf
and .pydistutils.cfg to use it.
Allow using a pypi mirror to install via pip. Using a custom pypi mirror can be very convenient, making image builds substantially faster - because we create multiple virtual environments we benefit more than single-virtualenv users would. Change-Id: I997daf1f9477c447e1fb30818aea9e80a49b31a6
2013-08-09 13:13:15 +00:00
Permit using arbitrary PyPI urls. The pypi element is cool, but some folk have local network mirrors which we should permit them to use. Change-Id: Ie840ad1184e72b0e01966eee0298cfd6511b6c19
2014-02-19 01:03:56 +00:00
When online, the official pypi.python.org pypi index is supplied as an
extra-url, so uncached dependencies will still be available. When offline, only
the mirror is used - be warned that a stale mirror will cause build failures.
Permit use of wheel mirrors in pypi element. pypi-mirror creates a separate mirror index for wheels (one per OS that mirrors are built on). To be able to use it one then needs to be able to export multiple mirrors for inclusion in pip.conf. As a drive by I made it possible to disable the use of the pypi.python.org index without using --offline (as --offline has larger impact). Change-Id: I3e85a8069b18cafd7eae4cd0591821acc3b5a739
2014-03-25 03:44:46 +00:00
To disable the pypi.python.org index without using --offline (e.g. when working
behind a corporate firewall that prohibits pypi.python.org) set
DIB\_NO\_PYPI\_PIP to any non-empty value.
Permit using arbitrary PyPI urls. The pypi element is cool, but some folk have local network mirrors which we should permit them to use. Change-Id: Ie840ad1184e72b0e01966eee0298cfd6511b6c19
2014-02-19 01:03:56 +00:00
Namespace PYPI_MIRROR_URL vars with DIB We have a good pattern of namespacing env vars with DIB_. Add support for DIB_PYPI_MIRROR_URL* and maintain backwards compat support. Change-Id: I434c9d1b14cd571b20754c9ad7cd64c936f8399a
2015-02-06 00:30:25 +00:00
To use an arbitrary mirror set DIB\_PYPI\_MIRROR\_URL=http[s]://somevalue/
Allow using a pypi mirror to install via pip. Using a custom pypi mirror can be very convenient, making image builds substantially faster - because we create multiple virtual environments we benefit more than single-virtualenv users would. Change-Id: I997daf1f9477c447e1fb30818aea9e80a49b31a6
2013-08-09 13:13:15 +00:00
Namespace PYPI_MIRROR_URL vars with DIB We have a good pattern of namespacing env vars with DIB_. Add support for DIB_PYPI_MIRROR_URL* and maintain backwards compat support. Change-Id: I434c9d1b14cd571b20754c9ad7cd64c936f8399a
2015-02-06 00:30:25 +00:00
Additional mirrors can be added by exporting DIB\_PYPI\_MIRROR\_URL\_1=... etc.
Only the one mirror can be used by easy-install, but since wheels need to be in
the first mirror to be used, the last listed mirror is used as the pydistutils
index. NB: The sort order for these variables is a simple string sort - if you
have more than 9 additional mirrors, some care will be needed.
Permit use of wheel mirrors in pypi element. pypi-mirror creates a separate mirror index for wheels (one per OS that mirrors are built on). To be able to use it one then needs to be able to export multiple mirrors for inclusion in pip.conf. As a drive by I made it possible to disable the use of the pypi.python.org index without using --offline (as --offline has larger impact). Change-Id: I3e85a8069b18cafd7eae4cd0591821acc3b5a739
2014-03-25 03:44:46 +00:00
Allow setting DIB_PIP_RETRIES Pip now supports retries on failure. If setting multiple pypi mirrors a fallback will not occure until the failures have finished for the first mirror. This can cause a substantial delay if mirror fallback occurs a lot. Change-Id: Iad37a9015a2d5c861a345a111bd1725b965a42d3
2015-02-19 23:11:52 +00:00
You can also set the number of retries that occur on failure by setting the
DIB\_PIP\_RETRIES environment variable. If setting fallback pip mirrors you
typically want to set this to 0 to prevent the need to fail multiple times
before falling back.
Permit use of wheel mirrors in pypi element. pypi-mirror creates a separate mirror index for wheels (one per OS that mirrors are built on). To be able to use it one then needs to be able to export multiple mirrors for inclusion in pip.conf. As a drive by I made it possible to disable the use of the pypi.python.org index without using --offline (as --offline has larger impact). Change-Id: I3e85a8069b18cafd7eae4cd0591821acc3b5a739
2014-03-25 03:44:46 +00:00
A typical use of this element is thus:
Namespace PYPI_MIRROR_URL vars with DIB We have a good pattern of namespacing env vars with DIB_. Add support for DIB_PYPI_MIRROR_URL* and maintain backwards compat support. Change-Id: I434c9d1b14cd571b20754c9ad7cd64c936f8399a
2015-02-06 00:30:25 +00:00
export DIB\_PYPI\_MIRROR\_URL=http://site/pypi/Ubuntu-13.10
export DIB\_PYPI\_MIRROR\_URL\_1=http://site/pypi/
export DIB\_PYPI\_MIRROR\_URL\_2=file:///tmp/pypi
Allow setting DIB_PIP_RETRIES Pip now supports retries on failure. If setting multiple pypi mirrors a fallback will not occure until the failures have finished for the first mirror. This can cause a substantial delay if mirror fallback occurs a lot. Change-Id: Iad37a9015a2d5c861a345a111bd1725b965a42d3
2015-02-19 23:11:52 +00:00
export DIB\_PIP\_RETRIES=0
Permit use of wheel mirrors in pypi element. pypi-mirror creates a separate mirror index for wheels (one per OS that mirrors are built on). To be able to use it one then needs to be able to export multiple mirrors for inclusion in pip.conf. As a drive by I made it possible to disable the use of the pypi.python.org index without using --offline (as --offline has larger impact). Change-Id: I3e85a8069b18cafd7eae4cd0591821acc3b5a739
2014-03-25 03:44:46 +00:00
Remove docs about deprecated pypi-mirror tool The pypi-mirror tool is no longer supported by the infra team, or reccomended for installs, and is known to have various bugs that will not be fixed. Remove mentions of it from the docs and point people at devpi-server instead. Change-Id: Ieedb8bf9bee2f5932edac6b9747e645d0f4eb08c
2014-08-28 01:32:07 +00:00
[devpi-server](https://git.openstack.org/cgit/openstack-infra/pypi-mirro://pypi.python.org/pypi/devpi-server)
can be useful in making a partial PyPI mirror suitable for building images. For
Permit use of wheel mirrors in pypi element. pypi-mirror creates a separate mirror index for wheels (one per OS that mirrors are built on). To be able to use it one then needs to be able to export multiple mirrors for inclusion in pip.conf. As a drive by I made it possible to disable the use of the pypi.python.org index without using --offline (as --offline has larger impact). Change-Id: I3e85a8069b18cafd7eae4cd0591821acc3b5a739
2014-03-25 03:44:46 +00:00
instance:
Allow using a pypi mirror to install via pip. Using a custom pypi mirror can be very convenient, making image builds substantially faster - because we create multiple virtual environments we benefit more than single-virtualenv users would. Change-Id: I997daf1f9477c447e1fb30818aea9e80a49b31a6
2013-08-09 13:13:15 +00:00
Remove docs about deprecated pypi-mirror tool The pypi-mirror tool is no longer supported by the infra team, or reccomended for installs, and is known to have various bugs that will not be fixed. Remove mentions of it from the docs and point people at devpi-server instead. Change-Id: Ieedb8bf9bee2f5932edac6b9747e645d0f4eb08c
2014-08-28 01:32:07 +00:00
* pip install -U devpi
Allow using a pypi mirror to install via pip. Using a custom pypi mirror can be very convenient, making image builds substantially faster - because we create multiple virtual environments we benefit more than single-virtualenv users would. Change-Id: I997daf1f9477c447e1fb30818aea9e80a49b31a6
2013-08-09 13:13:15 +00:00
Remove docs about deprecated pypi-mirror tool The pypi-mirror tool is no longer supported by the infra team, or reccomended for installs, and is known to have various bugs that will not be fixed. Remove mentions of it from the docs and point people at devpi-server instead. Change-Id: Ieedb8bf9bee2f5932edac6b9747e645d0f4eb08c
2014-08-28 01:32:07 +00:00
* devpi-server quickstart
Allow using a pypi mirror to install via pip. Using a custom pypi mirror can be very convenient, making image builds substantially faster - because we create multiple virtual environments we benefit more than single-virtualenv users would. Change-Id: I997daf1f9477c447e1fb30818aea9e80a49b31a6
2013-08-09 13:13:15 +00:00
Remove docs about deprecated pypi-mirror tool The pypi-mirror tool is no longer supported by the infra team, or reccomended for installs, and is known to have various bugs that will not be fixed. Remove mentions of it from the docs and point people at devpi-server instead. Change-Id: Ieedb8bf9bee2f5932edac6b9747e645d0f4eb08c
2014-08-28 01:32:07 +00:00
* devpi use http://machinename:3141
Allow using a pypi mirror to install via pip. Using a custom pypi mirror can be very convenient, making image builds substantially faster - because we create multiple virtual environments we benefit more than single-virtualenv users would. Change-Id: I997daf1f9477c447e1fb30818aea9e80a49b31a6
2013-08-09 13:13:15 +00:00
Remove docs about deprecated pypi-mirror tool The pypi-mirror tool is no longer supported by the infra team, or reccomended for installs, and is known to have various bugs that will not be fixed. Remove mentions of it from the docs and point people at devpi-server instead. Change-Id: Ieedb8bf9bee2f5932edac6b9747e645d0f4eb08c
2014-08-28 01:32:07 +00:00
* Re-export your variables to point at the new mirror:
Allow using a pypi mirror to install via pip. Using a custom pypi mirror can be very convenient, making image builds substantially faster - because we create multiple virtual environments we benefit more than single-virtualenv users would. Change-Id: I997daf1f9477c447e1fb30818aea9e80a49b31a6
2013-08-09 13:13:15 +00:00
Namespace PYPI_MIRROR_URL vars with DIB We have a good pattern of namespacing env vars with DIB_. Add support for DIB_PYPI_MIRROR_URL* and maintain backwards compat support. Change-Id: I434c9d1b14cd571b20754c9ad7cd64c936f8399a
2015-02-06 00:30:25 +00:00
export DIB\_PYPI\_MIRROR\_URL=http://machinename:3141/
unset DIB\_PYPI\__MIRROR\_URL\_1
unset DIB\_PYPI\__MIRROR\_URL\_2
Allow using a pypi mirror to install via pip. Using a custom pypi mirror can be very convenient, making image builds substantially faster - because we create multiple virtual environments we benefit more than single-virtualenv users would. Change-Id: I997daf1f9477c447e1fb30818aea9e80a49b31a6
2013-08-09 13:13:15 +00:00
Remove docs about deprecated pypi-mirror tool The pypi-mirror tool is no longer supported by the infra team, or reccomended for installs, and is known to have various bugs that will not be fixed. Remove mentions of it from the docs and point people at devpi-server instead. Change-Id: Ieedb8bf9bee2f5932edac6b9747e645d0f4eb08c
2014-08-28 01:32:07 +00:00
The next time packages are installed, they'll be cached on the local devpi
server; subsequent runs pointed at the same mirror will use the local cache if
the upstream can't be contacted.
Allow using a pypi mirror to install via pip. Using a custom pypi mirror can be very convenient, making image builds substantially faster - because we create multiple virtual environments we benefit more than single-virtualenv users would. Change-Id: I997daf1f9477c447e1fb30818aea9e80a49b31a6
2013-08-09 13:13:15 +00:00
Remove docs about deprecated pypi-mirror tool The pypi-mirror tool is no longer supported by the infra team, or reccomended for installs, and is known to have various bugs that will not be fixed. Remove mentions of it from the docs and point people at devpi-server instead. Change-Id: Ieedb8bf9bee2f5932edac6b9747e645d0f4eb08c
2014-08-28 01:32:07 +00:00
Note that this process only has the server running temporarily; see
[Quickstart: Permanent install on
server/laptop](http://doc.devpi.net/latest/quickstart-server.html) guide from
the devpi developers for more information on a more permanent setup.
Reference in New Issue Copy Permalink