Currently we have all our elements and library files in a top-level
directory and install them into
<root>/share/diskimage-builder/[elements|lib] (where root is either /
or the root of a virtualenv).
The problem with this is that editable/development installs (pip -e)
do *not* install data_files. Thus we have no canonical location to
look for elements -- leading to the various odd things we do such as a
whole bunch of guessing at the top of disk-image-create and having a
special test-loader in tests/test_elements.py so we can run python
unit tests on those elements that have it.
data_files is really the wrong thing to use for what are essentially
assets of the program. data_files install works well for things like
config-files, init.d files or dropping documentation files.
By moving the elements under the diskimage_builder package, we always
know where they are relative to where we import from. In fact,
pkg_resources has an api for this which we wrap in the new
diskimage_builder/paths.py helper [1].
We use this helper to find the correct path in the couple of places we
need to find the base-elements dir, and for the paths to import the
library shell functions.
Elements such as svc-map and pkg-map include python unit-tests, which
we do not need tests/test_elements.py to special-case load any more.
They just get found automatically by the normal subunit loader.
I have a follow-on change (I69ca3d26fede0506a6353c077c69f735c8d84d28)
to move disk-image-create to a regular python entry-point.
Unfortunately, this has to move to work with setuptools. You'd think
a symlink under diskimage_builder/[elements|lib] would work, but it
doesn't.
[1] this API handles stuff like getting files out of .zip archive
modules, which we don't do. Essentially for us it's returning
__file__.
Change-Id: I5e3e3c97f385b1a4ff2031a161a55b231895df5b
There is a wide variety of tracing options through the various shell
scripts. Some use "set -eux", others explicity set xtrace and others
do nothing. There is a "-x" option to bin/disk-image-create but it
doesn't flow down to the many scripts it calls.
This adds a global integer variable set by disk-image-create
DIB_DEBUG_TRACE. All scripts have a stanza added to detect this and
turn on tracing. Any other tracing methods are rolled into this. So
the standard header is
---
if [ "${DIB_DEBUG_TRACE:-0}" -gt 0 ]; then
set -x
fi
set -eu
set -o pipefail
---
Multiple -x options can be specified to dib-create-image, which
increases the value of DIB_DEBUG_TRACE. If script authors feel their
script should only trace at higher levels, they should modify the
"-gt" value. If they feel it should trace by default, they can modify
the default value also.
Changes to pachset 16 : scripts which currently trace themselves by
default have retained this behaviour with DIB_DEBUG_TRACE defaulting
to "1". This was done by running [1] on patch set 15. See the thread
beginning at [2]
dib-lint is also updated to look for the variable being matched.
[1] https://gist.github.com/ianw/71bbda9e6acc74ccd0fd
[2] http://lists.openstack.org/pipermail/openstack-dev/2014-November/051575.html
Change-Id: I6c5a962260741dcf6f89da9a33b96372a719b7b0
Just remove globally /lost+found after the root and install phases, so
distribution elements don't need to do that by themselves.
Change-Id: Ic783e613bd8794aefd3f40c9a7c308d14cd04b8d
The last patch to do this only handled the case of SHA256SUM being
stale, but the image is more likely to be stale in the default
configuration as the hash file is downloaded over HTTPS.
Change-Id: I6531fa684e560cad48c3696394d48a600680c875
Closes-Bug: #1336067
Some users on corp networks are finding that cloud images appear in
their proxies before SHA256SUMS files.
We now try to force any to upstream proxies to update their copy and
retry the checksum, rather than immediately failing.
We also now add proper argument parsing to cache-url, and associated
help functionality, since we have now grown our first argument.
Change-Id: I44d44b62db8d1df990606fdf087ec1b837f491f6
Closes-Bug: #1336067
The DIB_CLOUD_IMAGES variable in ubuntu's root.d is already used
everywhere with a /, so we can safely drop it from the default
value.
Change-Id: I39dbd3f4004052fcccb4131dc838759f4c82312a
Now that issues associated with this bump have been addressed
elsewhere, reinstate trusty as the default DIB_RELEASE for
the ubuntu element. Also adds a note about memory requirements
to the ramdisk element.
Change-Id: Ia89c83ba0ee4ad3bd89b833ba0db08135eac63ed
When the cache tarball gets fail the output doesn't indicate
what url is failing. Adding '-x' displays the urls attempted.
Change-Id: I71b2dbfd712e72c591410ca9b68edc76d493ad6e
As with the previous similar changes, this is intended to catch
problems as they happen instead of ignoring them and continuing on
to potentially fail later. Setting this on all existing scripts
will allow us to enforce use via Jenkins.
Change-Id: Iad2d490c86dceab148ea9ab08f457c49a5d5352e
Trusty ramdisks do not boot, failing to exec init. This reverts
the default DIB_RELEASE back to Saucy until Trusty ramdisks have
been properly tested.
Change-Id: I5ac81822b78805275268cecc021a74a62c22ef58
In tests, saucy has been able to match raring for the ability to bring
up clouds, so we can roll forwards and switch the default to saucy.
Change-Id: I4df35d40f902c0daa7b2761d7afea8db25526753
The command `sudo rmdir $TARGET_ROOT/lost+found` will fail
if `$TARGET_ROOT/lost+found` directory doesn't exist,
e.g. when you use non-default image.
Fixes bug #1245856
Change-Id: I48c8f2f201b29912a726249023ca7d20893cc958
When extracting the base image without --numeric-owner, user and group
names in the tarball are mapped to uid/gid by the host. This can cause
problems when building an image for some other distro than you're
running yourself. For example, building an Ubuntu image on openSUSE
ends up with /var/cache/man in the image owned by 'proxy' (uid 13)
instead of 'man' (uid 6), because the host (openSUSE) uses uid 13 for
the 'man' user. This particular man/proxy discrepancy results in
"fopen: Permission denied" errors when apt-get does its "Processing
triggers for man-db" thing in the Ubuntu system. I wouldn't be
surprised if there were other kinks caused by this uid/gid mapping
discrepancy too, but that's the one I found so far.
The same thing can also happen with Fedora, but seems to be less likely,
or at least less obvious to me when building Fedora images on openSUSE.
But, IMO, it's better to be safe and just use --numeric-owner on all
base image untarring outside the chroot.
Change-Id: I9da5ac66dd182e7278fe4fee932093f61d35673a
DIB_IMAGE_CACHE will be a user override for the location where images
are cached. Default location is ~/.cache/image-create
Change-Id: I3e9b9f970864d555c9ec9436344b53f6d3d66dfa
Ubuntu 13.04 has been released now for 3 months. The updated libvirt,
openvswitch and kernel are all beneficial to various OpenStack components,
and many other software is updated beyond the versions in Ubuntu 12.10.
Change-Id: I358aed8bf906c3ff5103f19b1f9e6ac689b5d5ee
When --offline is set elements should not revalidate cached data. The
ubuntu element had not been updated to match this. SHA checking is
also skipped as we only move a new cached file into place when the
hash matches, and we might download a new hash before updating the
image cache, which would cause persistent --offline failures.
Change-Id: If1a0366b51951a73b7a3ffe23a29a3d910b08938
The fedora element downloads images too, so we should re-use the caching
code from the ubuntu element.
There doesn't seem to be other examples of code shared between root.d
scripts. In the fedora and dpkg elements we copy install-packages into
the chroot, but that model doesn't apply when we're running scripts
outside of the chroot. Seems sane to just run it directly from the bin/
dir in the temporary hooks directory.
Change-Id: Iaa6aca660042fea323cab4271633a4bdbbc271b8
If a cached copy of the file doesn't exist, cache_url() passes a
non-existent path to -z/--time-cond and you see this warning:
Warning: Illegal date format for -z, --timecond (and not a file name).
Warning: Disabling time condition. See curl_getdate(3) for valid date syntax.
It works just fine, but the warning is ugly.
Change-Id: Ic6f13a2c596b988308d7fca9cd1745e5d48ae5fb
This also switches to using curl which some people may not have
installed. However, curl is far superior for this type of download.
Change-Id: I7ac5a84b30eb8daad320c082f976931c41a24669
Qemu-nbd does not perform well with older versions of qemu due to
the lack of writeback caching mode. It also only builds qcow2 images
and there is a desire for raw image support. Finally, qemu-nbd makes
it very difficult to build images concurrently due to the somewhat
opaque nature of how it selects a /dev/nbd# device. losetup, on
the other hand, makes this process very straight forward.
Change-Id: I309fad8af4fd1e8d1720c17b65e1897a76d5e897
Co-Author: Clint Byrum <clint@fewbar.com>
This switches $CLOUD_IMAGES and $RELEASE to the DIB_ namespace so
they will survive future changes to the sanitisation of the build
environment.
Change-Id: I7dc2aa82fb9ef452705b080cc404f41046014f20
Relies on https://cloud-images.ubuntu.com being served by a cert signed
by one of the CA's trusted by the build host.
Change-Id: I690b755acca54789110c2c8fa723c8b87b2485c9
This is a necessary but not complete step towards supporting Fedora and Suse
distributions. Further work is needed (e.g. to quiesce daemons on
installation).
Change-Id: If3ea6093d41a21de755db52328226b84b5a3ede6
