Commit Graph

32 Commits

Author SHA1 Message Date
Jon-Paul Sullivan
b507cab2b5 Move instead of copy the temporary git manifest
This prevents a build-up of temporary files in the cache directory

Change-Id: Ice34b57ef4f24bbe69b43ee3b2e2e394894825e5
2014-04-15 11:24:43 +01:00
Jon-Paul Sullivan
4c467964f5 Standardise manifest creation and retrieval
* Create a standard element to copy manifests out of the image and
  into the build area.
* Ensure all current manifests are copied into a standard manifests
  directory

Change-Id: I37eff5a13a14564e1adc33eb4f0144d571267617
2014-04-10 12:25:13 +01:00
James Slagle
b4dfa6cb90 Move install type enablement into base element
Move whether the package or source install type is enabled out of the
source-repositories element and into the base element.
source-repositories wasn't a great fit for this functionality to begin
with.

Specify the install type for an element via the
DIB_INSTALLTYPE_<install_dir_prefix> environment variable.

Additionally with this approach, any new install type can be added
in an element, without having to update source-repositories or anything
in dib.

You could just write your install type implementation under
nova-foo-install, then define DIB_INSTALLTYPE_nova=foo in your
environment, and the scripts under nova-foo-install would get run during
the image build.

Source installs (any scripts under <install_dir_prefix>-source-install)
is the default install type for all elements.

Change-Id: I9414aca360c41e030e27d3d0c0a52d9d8e13d8b1
2014-03-14 12:25:11 -04:00
Jenkins
09ad2ae482 Merge "Extract move cache logic to a function" 2014-03-14 08:08:24 +00:00
James Polley
ca95cf9f70 Extract move cache logic to a function
We use the same logic twice - extract it to a function.

Change-Id: I32a257663de37d5f3588c91e4164abceb12ee80d
2014-03-14 09:36:52 +11:00
Jenkins
bae43c91c6 Merge "98-source-repositories tries to return from script" 2014-03-13 16:17:51 +00:00
Jenkins
7b0119808d Merge "Create a git manifest from source-repositories" 2014-03-13 15:11:40 +00:00
Jon-Paul Sullivan
179aa31d41 Fixup all occurrences of REPONAME for replacing '-'
There was a single REPONAME variable missed in change 76679.
Fix it up.

Change-Id: Ibc6cd59cb3f3d949fd7679ee18d85c7574d9acd5
2014-03-11 13:13:10 +00:00
Jenkins
ffdd57b8a5 Merge "Replace more then just "-" in REPONAME" 2014-03-11 12:51:31 +00:00
Jon-Paul Sullivan
e4c43e608c Create a git manifest from source-repositories
As git repositories are copied into the image make a note of their
details and add them to a manifest on the build system.

This allows the code used in building an image to be found after
the image build is completed without needing to open the image up
for inspection.

Change-Id: I7dbe9e163ad38a418cf2869a81e720de2c27dfb1
2014-03-11 12:38:52 +00:00
Dan Prince
ef379f05e4 98-source-repositories tries to return from script
In 8b2325118f we added a line
so that 98-source-repositories tries to return from the top
level script (not from within a function).

Also, remove the return 1 within the function as well
because we just want to continue in these cases.

Change-Id: I66eddc12208b278594a0a8d8676c38d72045ca75
2014-03-10 17:21:20 -04:00
Jenkins
e83d4dcbb2 Merge "Bash eval the lines in source-repository scripts" 2014-03-10 18:55:06 +00:00
Derek Higgins
0226cf2246 Replace more then just "-" in REPONAME
Looks like shells only support alphanumeric characters and "_"'s in
environment variables. We were substituting "-" characters but in
order to be able to set overrides in projects with names containing
other characters we need to substitute more, I'm looking at you
oslo.config.

Change-Id: I3e2b1b0bc5871e4ec4ffd8117906cd077aa2cb0d
Co-authored-by: James Polley <jp@jamezpolley.com>
2014-03-06 04:56:50 +11:00
Jenkins
e602863642 Merge "Replace use of show-ref with name-rev" 2014-03-04 18:05:46 +00:00
Gregory Haynes
6a6d913459 Bash eval the lines in source-repository scripts
This will allow us to add environment variables to source-repository
scripts. Useful for things like arch-specific download urls.

Change-Id: I4d046825a317dc88db3d7673ad261cedacd1a104
2014-03-03 16:37:43 -08:00
Jenkins
9595197474 Merge "Improve usability of the source-repositories cache" 2014-03-04 00:21:52 +00:00
Jon-Paul Sullivan
41f1ca2ad4 Replace use of show-ref with name-rev
Testing showed that show-ref did not accurately determine whether a
git repository contained the given reference.  Testing with name-rev
showed that it did.

Change the HAS_REF determination appropriately to avoid unnecessary
cache updates when using fixed references.

Change-Id: Iece213348516358af84417807f8dd36e959139eb
2014-02-28 14:35:07 +00:00
Jon-Paul Sullivan
0edfcf6587 Enable simple modification of git repo location
Add the DIB_GITREPOBASE variable to the source-repositories element
to enable global changing of git repository location.

Change-Id: Icfac5e103fce6e5a3f1a2db094144a0ed89fcc33
2014-02-26 13:52:03 +00:00
Jon-Paul Sullivan
8b2325118f Improve usability of the source-repositories cache
* Rename the cache directory for sources

Make the cache directory name match that of the element that creates
it.  This makes it easier to find and more logically implies what
element created the cache directory.

* Add REPONAME to repository cache directory

For better usability add the reponame to the directory created
to cache it.

* Add migration code

Move pre-existing cache directories of the old name to the new name.

Change-Id: Ic62f19acb1746888ccbdf95eebf6e653f3e66357
2014-02-24 20:42:48 +00:00
Shane Wang
4aeee175d8 Fix misspellings in diskimage-builder
Fix misspellings detected by:
* pip install misspellings
* git ls-files | grep -v locale | misspellings -f -

Change-Id: I42acbcce7611ff3e2e120168acccc518299c2c42
Closes-Bug: #1257295
2014-02-07 14:42:45 +08:00
James Slagle
57ba300be4 Symlink correct element install type
The source-repositories element will now symlink the install scripts
corresponding to the set install type for an element into the install.d
directory.

Different install types are implemented by elements by writing scripts to do
the install type under install.d/<element-name>-<install-type>-install
directories.

For example, the nova element would provide:

nova/install.d/nova-package-install/74-nova
nova/install.d/nova-source-install/74-nova

source-repositories will create the following symlink for the package
install type:

$TMP_HOOKS_PATH/install.d/74-nova -> nova-package-install/74-nova

Or, for the source install type:

$TMP_HOOKS_PATH/install.d/74-nova -> nova-source-install/74-nova

Change-Id: I1bfaf39e5a98b2af904fbc6d674dbada30b27ed9
2014-01-15 14:57:08 -05:00
James Slagle
d253ed16d8 Add package support to source-repositories
Adds a new "package" type to source-repositories. When the package type
is specified in an origin-repository-* file the package name and type
"package" need to be specified like so:

nova package

The existing map-packages mechanism can be used to map the specified
package name to the actual distribution package names.  The
pre-configured package repositories on the image will be used to install
the packages during the install.d phase of the image build.

If the repo type is package, an environment.d hook script is updated
with an environment variable which is sourced by dib-run-parts. This is
so that other hooks, particularly in install.d, will know what repo type
was used and can make the correct decision about doing a source install
or not.

Change-Id: Ief4e524664cd32d2087aeb3b1766f05e017de91e
2014-01-13 08:23:53 -05:00
Chris Jones
4794df890a Log unsupported source repository types.
If we don't recognise git/tar/etc as source-repositories repotypes, we
now log the repotype, for debugging purposes.

Change-Id: I7b870807658441c928c7880291eb0284b6f97778
2013-12-20 14:53:53 +00:00
Chris Jones
4c7a3bf79f Improve source-repositories git caching.
We now unconditionally cache the repositories being fetched with
source-repositories.

Additionally, by improving the way we handle repository refs, we are now
able to build images with code taken directly from gerrit reviews.

Change-Id: Ifd4c5691f6761eb5551663e6d0aa1c0e42afced3
2013-12-20 14:48:30 +00:00
Mark McLoughlin
10eac21043 Log the repository fetches in source-repositories
Make the source-repositories element log which repo it's cloning from
so it's easier to tell what's going on.

Change-Id: I224f7bd98bc3d0ba94313dae9063d4ba4e25bdaa
2013-12-05 06:28:13 +00:00
Steve Baker
9e5bc37265 Add support for file to source-repositories
Adds a file type to source-repositories to allow individual files
to make use of source-repositories caching.

REPO_DIRECTORY has been rename to REPO_DEST since it is now sometimes
a file.

This change also fixes the incorrect path to the cache-url in the
tar type.

Change-Id: Ib79e41969c982a02f0235318d9f254b39c3c6d93
2013-09-04 10:16:49 +12:00
Robert Collins
c7c4ec19a3 Capture the repositories from source-repositories.
We try to expose metadata to users about what dib included in the
image and source-repositories was not doing that, so this fixes it. It
also permits easy introspection from other elements (such as an
upcoming PyPI cache layer).

Change-Id: I1e48e65f69ebe1f4556cc3bf35ccf195e99bcc76
2013-08-05 09:52:28 +12:00
Monty Taylor
4535c5323a Ignore emacs autosave files in source-repositories
source-repositories uses find to discover things it should import. If
someone happens to use emacs, and happens to have had an autosave file
in a directory with a wildly broken version of a source-repositories-
file, then dib will attempt to execute the contents of the autosave file
and the user will be sad. Adding a filter to the find command for files
ending in ~ will prevent pain and suffering.

Change-Id: I27c8a4b53d3c9b913ca926db16b4271941f6fb5d
2013-07-27 14:53:29 -04:00
Jenkins
5f52562570 Merge "Support repo names with multiple '-'s" 2013-07-19 19:45:42 +00:00
Derek Higgins
a4a0523f2b Support repo names with multiple '-'s
The search and replace that converted dashes in REPONAME only converted
the first dash. They should all be converted.

Change-Id: I76a48cf5bdfa8664f0a65d0238fa4da7d80305db
2013-07-19 17:45:01 +01:00
Derek Higgins
51216dbd2d Set work-dir to cached repository
source-repositories does a git reset on the .git directory of cached
repositories. But doesn't specify the directory to reset. A working
directory needs to be specified so that the $PWD isn't used.

Without this change $PWD is polluted with the contents of repositories
being cached.

Change-Id: Ic37b702ac579bf766bb2204a988fa9468d308abf
2013-07-19 17:40:16 +01:00
Robert Collins
e187a402ae Move the getsources hook earlier.
Caching PyPI objects will require inspecting the content downloaded by
the source repositories, so needs to run after it.

Change-Id: I551c075e0bb6853b2ef79268b7f6455a4f06743d
2013-07-18 23:06:30 +12:00