Problems were seen when building images in parallel with
git clones and fetches interfering with each other.
Stop it.
Co-Authored-By: Michael Kerrin <michael.kerrin@hp.com>
Change-Id: Iaa48f259755d8b22327459a5c6d224a342909aee
Add the ability to supply a gerrit ref in DIB_REPOREF_* and
have that reference fetched from the remote and checked out for
building within the image.
Closes-Bug: #1314021
Change-Id: I5e5742c4cbff98e14121c50dde5e512f192b5415
Co-Authored-By: Robert Collins <rbtcollins@hp.com>
As with the previous similar changes, this is intended to catch
problems as they happen instead of ignoring them and continuing on
to potentially fail later. Setting this on all existing scripts
will allow us to enforce use via Jenkins.
Change-Id: Iad2d490c86dceab148ea9ab08f457c49a5d5352e
* Create a standard element to copy manifests out of the image and
into the build area.
* Ensure all current manifests are copied into a standard manifests
directory
Change-Id: I37eff5a13a14564e1adc33eb4f0144d571267617
When the repository references are sha1s the fetch command used will
not work. This will ensure that all of the objects are still fetched,
and all of the local cache references for branches and tags will be
updated rather than just the specified reference in use.
Change-Id: Iad2554b3087a5da37dac6e9f6a72432d49fce83d
Closes-Bug: #1301435
Move whether the package or source install type is enabled out of the
source-repositories element and into the base element.
source-repositories wasn't a great fit for this functionality to begin
with.
Specify the install type for an element via the
DIB_INSTALLTYPE_<install_dir_prefix> environment variable.
Additionally with this approach, any new install type can be added
in an element, without having to update source-repositories or anything
in dib.
You could just write your install type implementation under
nova-foo-install, then define DIB_INSTALLTYPE_nova=foo in your
environment, and the scripts under nova-foo-install would get run during
the image build.
Source installs (any scripts under <install_dir_prefix>-source-install)
is the default install type for all elements.
Change-Id: I9414aca360c41e030e27d3d0c0a52d9d8e13d8b1
As git repositories are copied into the image make a note of their
details and add them to a manifest on the build system.
This allows the code used in building an image to be found after
the image build is completed without needing to open the image up
for inspection.
Change-Id: I7dbe9e163ad38a418cf2869a81e720de2c27dfb1
In 8b2325118f we added a line
so that 98-source-repositories tries to return from the top
level script (not from within a function).
Also, remove the return 1 within the function as well
because we just want to continue in these cases.
Change-Id: I66eddc12208b278594a0a8d8676c38d72045ca75
Looks like shells only support alphanumeric characters and "_"'s in
environment variables. We were substituting "-" characters but in
order to be able to set overrides in projects with names containing
other characters we need to substitute more, I'm looking at you
oslo.config.
Change-Id: I3e2b1b0bc5871e4ec4ffd8117906cd077aa2cb0d
Co-authored-by: James Polley <jp@jamezpolley.com>
This will allow us to add environment variables to source-repository
scripts. Useful for things like arch-specific download urls.
Change-Id: I4d046825a317dc88db3d7673ad261cedacd1a104
Testing showed that show-ref did not accurately determine whether a
git repository contained the given reference. Testing with name-rev
showed that it did.
Change the HAS_REF determination appropriately to avoid unnecessary
cache updates when using fixed references.
Change-Id: Iece213348516358af84417807f8dd36e959139eb
Add the DIB_GITREPOBASE variable to the source-repositories element
to enable global changing of git repository location.
Change-Id: Icfac5e103fce6e5a3f1a2db094144a0ed89fcc33
* Rename the cache directory for sources
Make the cache directory name match that of the element that creates
it. This makes it easier to find and more logically implies what
element created the cache directory.
* Add REPONAME to repository cache directory
For better usability add the reponame to the directory created
to cache it.
* Add migration code
Move pre-existing cache directories of the old name to the new name.
Change-Id: Ic62f19acb1746888ccbdf95eebf6e653f3e66357
The source-repositories element will now symlink the install scripts
corresponding to the set install type for an element into the install.d
directory.
Different install types are implemented by elements by writing scripts to do
the install type under install.d/<element-name>-<install-type>-install
directories.
For example, the nova element would provide:
nova/install.d/nova-package-install/74-nova
nova/install.d/nova-source-install/74-nova
source-repositories will create the following symlink for the package
install type:
$TMP_HOOKS_PATH/install.d/74-nova -> nova-package-install/74-nova
Or, for the source install type:
$TMP_HOOKS_PATH/install.d/74-nova -> nova-source-install/74-nova
Change-Id: I1bfaf39e5a98b2af904fbc6d674dbada30b27ed9
Adds a new "package" type to source-repositories. When the package type
is specified in an origin-repository-* file the package name and type
"package" need to be specified like so:
nova package
The existing map-packages mechanism can be used to map the specified
package name to the actual distribution package names. The
pre-configured package repositories on the image will be used to install
the packages during the install.d phase of the image build.
If the repo type is package, an environment.d hook script is updated
with an environment variable which is sourced by dib-run-parts. This is
so that other hooks, particularly in install.d, will know what repo type
was used and can make the correct decision about doing a source install
or not.
Change-Id: Ief4e524664cd32d2087aeb3b1766f05e017de91e
If we don't recognise git/tar/etc as source-repositories repotypes, we
now log the repotype, for debugging purposes.
Change-Id: I7b870807658441c928c7880291eb0284b6f97778
We now unconditionally cache the repositories being fetched with
source-repositories.
Additionally, by improving the way we handle repository refs, we are now
able to build images with code taken directly from gerrit reviews.
Change-Id: Ifd4c5691f6761eb5551663e6d0aa1c0e42afced3
Make the source-repositories element log which repo it's cloning from
so it's easier to tell what's going on.
Change-Id: I224f7bd98bc3d0ba94313dae9063d4ba4e25bdaa
Since we are using bash syntax in some of the element fragments,
we should make sure we use bash for all of them, so that things don't
break on systems where /bin/sh != /bin/bash.
Change-Id: If2f043c57aa4e1492b7f9839213ef6123f683612
OpenStack runs git.openstack.org which is more reliable and responsive
when projects operate within OpenStack Infra. Replace all of the
references to github with referneces to git.openstack.org.
Change-Id: Ib3ece85aba6451801487b0bdbd83147e39d9e155
Adds a file type to source-repositories to allow individual files
to make use of source-repositories caching.
REPO_DIRECTORY has been rename to REPO_DEST since it is now sometimes
a file.
This change also fixes the incorrect path to the cache-url in the
tar type.
Change-Id: Ib79e41969c982a02f0235318d9f254b39c3c6d93
We try to expose metadata to users about what dib included in the
image and source-repositories was not doing that, so this fixes it. It
also permits easy introspection from other elements (such as an
upcoming PyPI cache layer).
Change-Id: I1e48e65f69ebe1f4556cc3bf35ccf195e99bcc76
source-repositories uses find to discover things it should import. If
someone happens to use emacs, and happens to have had an autosave file
in a directory with a wildly broken version of a source-repositories-
file, then dib will attempt to execute the contents of the autosave file
and the user will be sad. Adding a filter to the find command for files
ending in ~ will prevent pain and suffering.
Change-Id: I27c8a4b53d3c9b913ca926db16b4271941f6fb5d
The search and replace that converted dashes in REPONAME only converted
the first dash. They should all be converted.
Change-Id: I76a48cf5bdfa8664f0a65d0238fa4da7d80305db
source-repositories does a git reset on the .git directory of cached
repositories. But doesn't specify the directory to reset. A working
directory needs to be specified so that the $PWD isn't used.
Without this change $PWD is polluted with the contents of repositories
being cached.
Change-Id: Ic37b702ac579bf766bb2204a988fa9468d308abf
Caching PyPI objects will require inspecting the content downloaded by
the source repositories, so needs to run after it.
Change-Id: I551c075e0bb6853b2ef79268b7f6455a4f06743d
Cloning large repositories over the internet can take considerable
time. Caching them locally makes repeated image builds significantly
faster, so lets do that.
When users override the element source they will often be using a
local repository, so in those cases I don't cache - but we could
easily change our minds on that in the future.
Change-Id: I9822362cf722b904d9806dbbb4bb07cfe2b33437
Git isn't a dependency of this element, but is a dependency of pbr
if you are doing pip install on a git cloned directoy, because this
looks to be a fairly common pattern it makes sense to have git on the
target image.
Change-Id: I49ee3fa4eac493b9dfa379a674ab9b3390cfc89a
Fixes bug 1184943
Adding an element that allows other elements to register
repositories it would like to be retrieved for it.
Doing this outside of the chroot allows locally cached
repositories to be used. It also gives the ability to d-i-b
to specify what revision to use if an alternative to the
most recent is required.
Effectively allowing a CI system to test d-i-b and elements
without being effected by unavailable git repositories or
breakages in actively developed source code being used by
the elements.
Change-Id: I1527facebaad256a357af680e017b34b1788575d
