diskimage-builder/diskimage_builder/elements/source-repositories/README.rst
Ian Wienand 97c01e48ed Move elements & lib relative to diskimage_builder package
Currently we have all our elements and library files in a top-level
directory and install them into
<root>/share/diskimage-builder/[elements|lib] (where root is either /
or the root of a virtualenv).

The problem with this is that editable/development installs (pip -e)
do *not* install data_files.  Thus we have no canonical location to
look for elements -- leading to the various odd things we do such as a
whole bunch of guessing at the top of disk-image-create and having a
special test-loader in tests/test_elements.py so we can run python
unit tests on those elements that have it.

data_files is really the wrong thing to use for what are essentially
assets of the program.  data_files install works well for things like
config-files, init.d files or dropping documentation files.

By moving the elements under the diskimage_builder package, we always
know where they are relative to where we import from.  In fact,
pkg_resources has an api for this which we wrap in the new
diskimage_builder/paths.py helper [1].

We use this helper to find the correct path in the couple of places we
need to find the base-elements dir, and for the paths to import the
library shell functions.

Elements such as svc-map and pkg-map include python unit-tests, which
we do not need tests/test_elements.py to special-case load any more.
They just get found automatically by the normal subunit loader.

I have a follow-on change (I69ca3d26fede0506a6353c077c69f735c8d84d28)
to move disk-image-create to a regular python entry-point.

Unfortunately, this has to move to work with setuptools.  You'd think
a symlink under diskimage_builder/[elements|lib] would work, but it
doesn't.

[1] this API handles stuff like getting files out of .zip archive
modules, which we don't do.  Essentially for us it's returning
__file__.

Change-Id: I5e3e3c97f385b1a4ff2031a161a55b231895df5b
2016-11-01 17:27:41 -07:00

166 lines
5.8 KiB
ReStructuredText

===================
source-repositories
===================
With this element other elements can register their installation source by
placing their details in the file ``source-repository-*``.
source-repository-* file format
-------------------------------
The plain text file format is space separated and has four mandatory fields
optionally followed by fields which are type dependent::
<name> <type> <destination> <location> [<ref>]
``name``
Identifier for the source repository. Should match the file suffix.
``type``
Format of the source. Either ``git``, ``tar``, ``package`` or ``file``.
``destination``
Base path to place sources.
``location``
Resource to fetch sources from. For ``git`` the location is cloned. For
``tar`` it is extracted.
``ref`` (optional). Meaning depends on the ``type``:
``file``: unused/ignored.
``git``: a git reference to fetch. A value of "``*``" prunes and fetches all
heads and tags. Defaults to ``master`` if not specified.
``tar``:
| "``.``" extracts the entire contents of the tarball.
| "``*``" extracts the contents within all its subdirectories.
| A subdirectory path may be used to extract only its contents.
| A specific file path within the archive is **not supported**.
The lines in the source-repository scripts are eval'd, so they may contain
environment variables.
The ``package`` type indicates the element should install from packages onto
the root filesystem of the image build during the ``install.d`` phase. If the
element provides an <element-name>-package-install directory, symlinks will be
created for those scripts instead.
``git`` and ``tar`` are treated as source installs. If the element provides an
<element-name>-source-install directory under it's ``install.d`` hook directory,
symlinks to the scripts in that directory will be created under ``install.d`` for
the image build.
For example, the nova element would provide::
nova/install.d/nova-package-install/74-nova
nova/install.d/nova-source-install/74-nova
source-repositories will create the following symlink for the package install
type::
install.d/74-nova -> nova-package-install/74-nova
Or, for the source install type::
install.d/74-nova -> nova-source-install/74-nova
All other scripts that exist under ``install.d`` for an element will be executed as
normal. This allows common install code to live in a script outside of
<element-name>-package-install or <element-name>-source-install.
If multiple elements register a source location with the same <destination>
then source-repositories will exit with an error. Care should therefore be taken
to only use elements together that download source to different locations.
The repository paths built into the image are stored in
etc/dib-source-repositories, one repository per line. This permits later review
of the repositories (by users or by other elements).
The repository names and types are written to an environment.d hook script at
01-source-repositories-environment. This allows later hook scripts during the
install.d phase to know which install type to use for the element.
An example of an element "custom-element" that wants to retrieve the ironic
source from git and pbr from a tarball would be:
*Element file: elements/custom-element/source-repository-ironic*::
ironic git /usr/local/ironic git://git.openstack.org/openstack/ironic.git
*File : elements/custom-element/source-repository-pbr*::
pbr tar /usr/local/pbr http://tarballs.openstack.org/pbr/pbr-master.tar.gz .
diskimage-builder will then retrieve the sources specified and place them
at the directory ``<destination>``.
Override per source
-------------------
A number of environment variables can be set by the process calling
diskimage-builder which can change the details registered by the element, these
are:
``DIB_REPOTYPE_<name>`` : change the registered type
``DIB_REPOLOCATION_<name>`` : change the registered location
``DIB_REPOREF_<name>`` : change the registered reference
For example if you would like diskimage-builder to get ironic from a local
mirror you would override the ``<location>`` field and could set:
.. sourcecode:: sh
DIB_REPOLOCATION_ironic=git://localgitserver/ironic.git
*As you can see above, the \<name\> of the repo is used in several bash
variables. In order to make this syntactically feasible, any characters not in
the set \[A-Za-z0-9_\] will be converted to \_*
*For instance, a repository named "diskimage-builder" would set a variable called
"DIB_REPOTYPE_diskimage_builder"*
Alternatively if you would like to use the keystone element and build an image with
keystone from a stable branch *stable/grizzly* then you would set:
.. sourcecode:: sh
DIB_REPOREF_keystone=stable/grizzly
If you wish to build an image using code from a Gerrit review, you can set
``DIB_REPOLOCATION_<name>`` and ``DIB_REPOREF_<name>`` to the values given by
Gerrit in the fetch/pull section of a review. For example, setting up Nova with
change 61972 at patchset 8:
.. sourcecode:: sh
DIB_REPOLOCATION_nova=https://review.openstack.org/openstack/nova
DIB_REPOREF_nova=refs/changes/72/61972/8
Alternate behaviors
-------------------
Override git remote
^^^^^^^^^^^^^^^^^^^
The base url for all git repositories can be set by use of:
``DIB_GITREPOBASE``
So setting ``DIB_GITREPOBASE=https://github.com/`` when the repo location is
set to http://git.openstack.org/openstack/nova.git will result in use of the
https://github.com/openstack/nova.git repository instead.
Disable external fetches
^^^^^^^^^^^^^^^^^^^^^^^^
When doing image builds in environments where external resources are not allowed,
it is possible to disable fetching of all source repositories by including an
element in the image that sets ``NO_SOURCE_REPOSITORIES=1`` in an
``environment.d`` script.