Currently we have all our elements and library files in a top-level
directory and install them into
<root>/share/diskimage-builder/[elements|lib] (where root is either /
or the root of a virtualenv).
The problem with this is that editable/development installs (pip -e)
do *not* install data_files. Thus we have no canonical location to
look for elements -- leading to the various odd things we do such as a
whole bunch of guessing at the top of disk-image-create and having a
special test-loader in tests/test_elements.py so we can run python
unit tests on those elements that have it.
data_files is really the wrong thing to use for what are essentially
assets of the program. data_files install works well for things like
config-files, init.d files or dropping documentation files.
By moving the elements under the diskimage_builder package, we always
know where they are relative to where we import from. In fact,
pkg_resources has an api for this which we wrap in the new
diskimage_builder/paths.py helper [1].
We use this helper to find the correct path in the couple of places we
need to find the base-elements dir, and for the paths to import the
library shell functions.
Elements such as svc-map and pkg-map include python unit-tests, which
we do not need tests/test_elements.py to special-case load any more.
They just get found automatically by the normal subunit loader.
I have a follow-on change (I69ca3d26fede0506a6353c077c69f735c8d84d28)
to move disk-image-create to a regular python entry-point.
Unfortunately, this has to move to work with setuptools. You'd think
a symlink under diskimage_builder/[elements|lib] would work, but it
doesn't.
[1] this API handles stuff like getting files out of .zip archive
modules, which we don't do. Essentially for us it's returning
__file__.
Change-Id: I5e3e3c97f385b1a4ff2031a161a55b231895df5b
This reverts commit 7a4c396948.
Note this time it sets it to F21, not F22, and gets rid of the
duplicate definition that doesn't get exported.
Change-Id: I240ad25d7a73c379559517a2a8399ae8c098314b
There is a wide variety of tracing options through the various shell
scripts. Some use "set -eux", others explicity set xtrace and others
do nothing. There is a "-x" option to bin/disk-image-create but it
doesn't flow down to the many scripts it calls.
This adds a global integer variable set by disk-image-create
DIB_DEBUG_TRACE. All scripts have a stanza added to detect this and
turn on tracing. Any other tracing methods are rolled into this. So
the standard header is
---
if [ "${DIB_DEBUG_TRACE:-0}" -gt 0 ]; then
set -x
fi
set -eu
set -o pipefail
---
Multiple -x options can be specified to dib-create-image, which
increases the value of DIB_DEBUG_TRACE. If script authors feel their
script should only trace at higher levels, they should modify the
"-gt" value. If they feel it should trace by default, they can modify
the default value also.
Changes to pachset 16 : scripts which currently trace themselves by
default have retained this behaviour with DIB_DEBUG_TRACE defaulting
to "1". This was done by running [1] on patch set 15. See the thread
beginning at [2]
dib-lint is also updated to look for the variable being matched.
[1] https://gist.github.com/ianw/71bbda9e6acc74ccd0fd
[2] http://lists.openstack.org/pipermail/openstack-dev/2014-November/051575.html
Change-Id: I6c5a962260741dcf6f89da9a33b96372a719b7b0
As with the previous similar changes, this is intended to catch
problems as they happen instead of ignoring them and continuing on
to potentially fail later. Setting this on all existing scripts
will allow us to enforce use via Jenkins.
Change-Id: Iad2d490c86dceab148ea9ab08f457c49a5d5352e
Adds the ability to set $DIB_LOCAL_IMAGE to use as the base cloud image
for a Fedora image build. There are many repetitve tasks that are done
every image build. With this change you can build an image with the
fedora and vm element and then reuse the resulting image as input to
future image builds. This greatly reduces future image build times.
For instance 99-up-to-date is already taking almost 2 minutes (even with
rpm downloads already cached) for Fedora 20, and is only going to keep
taking longer. By having a local up to date Fedora cloud image, this
time can be saved on each image build.
There is one minor change to support this in that /tmp/grub needs to get
cleaned up at the end of image builds so that the image can be reused.
Plus, there is no reason for it to stick around anyway. (didn't think
this was worth a seperate commit).
Change-Id: Ic74d138da922ecc99c38c27f105170d90009a84a
Update the default version of Fedora to 20 for the cloud image. A few of
us have run through devtest using Fedora 20 and it's working.
To continue to use 19, just specify DIB_RELEASE=19 for your image
builds.
Change-Id: I3d48272dedcd424747db5485ae0d8e9953f04191
If "kpartx -av" is called and there is no unused /dev/loop* device,
then kpartx fails. Calling "losetup -f" finds first unused device
and creates it if it doesn't exist.
Fixes bug #1195264
Change-Id: I5d59b519fd8e2a7380b71a788f9d3f2331b6567f
The command `sudo rmdir $TARGET_ROOT/lost+found` will fail
if `$TARGET_ROOT/lost+found` directory doesn't exist,
e.g. when you use non-default image.
Fixes bug #1245856
Change-Id: I48c8f2f201b29912a726249023ca7d20893cc958
I am getting a lot of http 404's from Fedora mirrors, in particular in
the US. Alternatively we could get cache-url to use previously existing
files if they exist and there is an error but this may not be the
correct thing todo in all cases.
So for now we do a retry if HTTP 404 is returned to the request for a
Fedora image.
Change-Id: I7634aeac3bdfe6930fafe6d276186c09fc8587b5
The fedora element downloads the latest available image so presumably
will jump to F20 once available. Probably causing several days (weeks?)
of busted stuff. Of course it will be impossible to know when all users
of the elements are ready to switch but the least we can do is allow a
little time as a buffer.
This commit ties it down to a specific version which can then be updated
when the consumers of this element are ready. This allso follows the
same pattern as the ubuntu element.
Change-Id: I15c8e15a66e8af1bd152c27144acbc55af9da88e
When extracting the base image without --numeric-owner, user and group
names in the tarball are mapped to uid/gid by the host. This can cause
problems when building an image for some other distro than you're
running yourself. For example, building an Ubuntu image on openSUSE
ends up with /var/cache/man in the image owned by 'proxy' (uid 13)
instead of 'man' (uid 6), because the host (openSUSE) uses uid 13 for
the 'man' user. This particular man/proxy discrepancy results in
"fopen: Permission denied" errors when apt-get does its "Processing
triggers for man-db" thing in the Ubuntu system. I wouldn't be
surprised if there were other kinks caused by this uid/gid mapping
discrepancy too, but that's the one I found so far.
The same thing can also happen with Fedora, but seems to be less likely,
or at least less obvious to me when building Fedora images on openSUSE.
But, IMO, it's better to be safe and just use --numeric-owner on all
base image untarring outside the chroot.
Change-Id: I9da5ac66dd182e7278fe4fee932093f61d35673a
DIB_IMAGE_CACHE will be a user override for the location where images
are cached. Default location is ~/.cache/image-create
Change-Id: I3e9b9f970864d555c9ec9436344b53f6d3d66dfa
This option does not exist on RHEL hosts and matches what is
currently present in elements/rhel/root.d/10-rhel-cloud-image.
Change-Id: I578233c1f37d035c67600fc60e7c4eb4ff75cbb3
Similar to the Ubuntu element the Fedora element had not been updated
to honour --offline. Also similarly we only check for the existence of
the final cached file rather than the image we directly download,
because they are generated separately.
The -u change is just hygiene to detect future buggy changes.
Change-Id: Ib92a9d419bb485c8861da041066827e8d6aac0b7
Latest Fedora cloud images are available here
http://cloud.fedoraproject.org/
With this change, when a Fedora version comes up,
d-i-b will point to it automatically.
Fixes 1199237
(Also, thanks to Lucas Gomes for testing.)
Change-Id: Ifcb8a48f3baf3ef841f00e6a20c6fd9f79fea129
On Fedora 19, there is a delay between the time loop0p2
is created to the time it is ready to be mounted.
This patches waits for 5 seconds for /dev/mapper/loop0p2
to be accessible. If it is not ready then the script
exits with status 1.
Fixes: bug #1197340
Change-Id: Iff0cfd0566673799a77500e0e24954f5f3743a17
The fedora element downloads images too, so we should re-use the caching
code from the ubuntu element.
There doesn't seem to be other examples of code shared between root.d
scripts. In the fedora and dpkg elements we copy install-packages into
the chroot, but that model doesn't apply when we're running scripts
outside of the chroot. Seems sane to just run it directly from the bin/
dir in the temporary hooks directory.
Change-Id: Iaa6aca660042fea323cab4271633a4bdbbc271b8
Fedora's partprobe does not create devices the same way it does on
Ubuntu. Instead of using partprobe, this patch uses kpartx, which
works on both Fedora and Ubuntu.
Change-Id: Iac5cbac083e423425693c8210cd5d058ce4bce72
Qemu-nbd does not perform well with older versions of qemu due to
the lack of writeback caching mode. It also only builds qcow2 images
and there is a desire for raw image support. Finally, qemu-nbd makes
it very difficult to build images concurrently due to the somewhat
opaque nature of how it selects a /dev/nbd# device. losetup, on
the other hand, makes this process very straight forward.
Change-Id: I309fad8af4fd1e8d1720c17b65e1897a76d5e897
Co-Author: Clint Byrum <clint@fewbar.com>
This switches $CLOUD_IMAGES and $RELEASE to the DIB_ namespace so
they will survive future changes to the sanitisation of the build
environment.
Change-Id: I7dc2aa82fb9ef452705b080cc404f41046014f20
