When uploading images to multiple clouds it is possible that the same
image will be needed in multiple formats to accomodate hypervisors
across clouds. Update disk-image-create's -t flag to take a list of
desired output image formats so that a single disk-image-create can
output all of the desired image formats.
Change-Id: If121b2342ae888855ba435aa3189f039e985b812
If we entered the cleanup trap due to exit with an error code we should
exit dib with an error code.
Change-Id: Iee1a05668b3239113fb91a2da0d9a66d7de4db6b
A user reported symptoms where the losetup line used to detach the
loopback device was failing in tar mode. We don't need to detach a
device that does not exist.
Change-Id: I807996e16199288927b49b4f300ae9b461cb8fe7
Closes-Bug: #1378033
Now that dib-run-parts has been moved to the dib-utils project, we
need to update diskimage-builder to use it instead of the version
directly in diskimage-builder.
This change removes the old copy of the dib-run-parts script in
the element, adds dib-utils as a dependency of diskimage-builder,
and updates the uses of dib-run-parts to correctly handle the fact
that it is now external to the project.
Requires I0be1f876d0e4a7d38e0d5c6010a552a8ebb158a4
Change-Id: Ia0a0df7784a14c49b5c47ac0b03e6c2602c84b3b
We need to be able to do install.d like things for ramdisks
themselves, but install.d runs outside the ramdisk context - and its
likely to break peoples brains if we mangle the two together - so this
adds a new hook point, ramdisk-install, specifically for installing
things into the ramdisk.
Change-Id: I37d1660309cda6e28bd0b316b08f61db4e080613
The two duplicated functions, save_image and finish_image will move
an existing image out of the way if it exists, but it isn't
configurable. Check an environment variable is 0 before doing so.
Switch save_image to just calling finish_image, rather than
duplicating its code exactly.
Change-Id: I26a5a8fa4b6e853c9440bffab195b0bc3728be40
If we have the old image file with same name as new image file
we shouldn't rewrite it, just need to rename old.
Uses date number in old image name.
Fixes TODO in lib/common-functions
Change-Id: Ica58a0f4435217912bddf950322299d2ed2889f3
Ensure the return code of element-info is actually checked by moving the
export IMAGE_ELEMENT to the subsequent line.
A recent change (Ic0a061995e2ae708c95a535cba6237bff58f7d93) exported the
IMAGE_ELEMENT variable. This caused the return code of element-info to not
actually be checked, since the export command is now checked instead and
almost always will return 0.
This caused a misleading error message when using diskimage-builder with an
unknown element since the script didn't exit when it should have:
$ bin/disk-image-create -a amd64 foobar vm
Building elements: base foobar vm
ERROR: Element 'foobar' not found in 'bin/../elements'
Expanded element dependencies to:
Building in /var/tmp/image.DuIDY1lW
Please include at least one distribution root element.
Change-Id: I3d9c02bf0b1a4f320280185dd3fab0697d2c92f2
Adds a new pkg-map element which can be used to install
packages based on an in element 'pkg-map' JSON config
file format.
As part of this change we expose DIB's IMAGE_ELEMENT variable
so that we can have access to the element names which are being
installed in a clean manner.
This change is intended to decouple elements from DIB
and allow new elements to support multiple distributions
(with various package naming schemes) without having to
constantly maintain DIB's various bin/map-packages files.
Change-Id: Ic0a061995e2ae708c95a535cba6237bff58f7d93
If the tmpfs being used to hold the image filesystem is close to being
full then housing the raw disk image in the same place can be
problematic as it tends to grow faster then the filesystem shrinks when
data is being moved into it.
Putting them both into separate tmpfs's will allow there to be an
overlap where they jointly use more then the size limit for a singe tmpfs.
Change-Id: Ia17ca357d3b865d70a2d3e13e0479d008ca5f924
Closes-Bug: #1289582
Fixes problems found by set -eu and pipefail, including:
-Many unset variables
-Commands that can fail under normal circumstances, which breaks
with set -e. This change swallows those expected errors to allow
our existing error code to handle them.
-The dkms element was not finding Fedora kernel versions correctly.
This may be an issue for other distros too, but since Fedora was
working fine without this functionality I only changed it to print
a warning message rather than failing the build when it happens.
-The ramdisk init script will not be set -eu because if it fails
the result is a kernel panic, which can be tricky to debug.
However, in testing with set -e a few failing commands were found
and have been fixed in this patch.
Change-Id: I44cf98dfc80cfcaec54b88cc83be80a3dbf2cec3
save_image is used to copy kernel and ramdisks out of the image, which
we will sometimes want to keep the source, and sometimes not. However
for the main image itself, the temp copy is never kept, so use mv
rather than cp and avoid the excess IO.
Change-Id: I5a9f0d69ffee3e6b872a8927537ac17f02f5aa4d
This will help to ensure images that would cause bug #1289582 would
still fail even on systems that have more RAM. Also this will ensure we
only reserve enough tmpfs space for this build.
Change-Id: Icece64e691d126a5fc89262897630fdbef1ffdf1
cleanup for ramdisk failed to umount TMP_BUILD_DIR with device or
resource busy error.
The patch unmounts all the mountpoints under TMP_BUILD_DIR and
detaches loop devices associated with TMP_IMAGE_PATH.
The unmounts are applied with both force(-f) and lazy(-l)
options. Force option is only for NFS mounts, it's kept here since no
harm for lazy option.
Change-Id: I84035e6a003d8135186b2fda3facbd2c37967529
disk-image-get-kernel is pretty noisy and you see e.g.:
$> load-image overcloud-compute.qcow2
XXX -d '/tmp/image.lWGCgPoj' -o 'tmp' -i '/home/stack/overcloud-compute.qcow2' --
Extracting kernel + ramdisk from /home/stack/overcloud-compute.qcow2 and writing them to /tmp/image.lWGCgPoj
nbd 17554 0
nbd 17554 0
basename: missing operand
Try 'basename --help' for more information.
/dev/nbd0 disconnected
tmp-vmlinuz,tmp-initrd
Clean all this up so we just get:
$> load-image overcloud-compute.qcow2
Extracting kernel + ramdisk from /home/stack/overcloud-compute.qcow2 to tmp-vmlinuz and tmp-initrd in /tmp/image.g6b0lG88
Change-Id: I8971ec0bbcd87157b07fc17254c56bb9f9f2a597
Converts our existing default root element code to be just a check
which exits with a failure message if no root/distribution element
is found.
Change-Id: I954a6abfd7871d5807b1a171a03fa98932410cff
Commit c7d80dd (Cleanup mount points automatically) removed the unmount
of $TMP_MOUNT_PATH/tmp/ccache in run_d_in_target() and moved the
"rm /tmp/ccache" to elements/base/finalise.d/02-remove-ccache. There
are two problems with this:
1) Not unmounting at the end of run_d_in_target() results in tmp/ccache
being bind mounted muliple times on top of itself (three times, if you
just run `disk-image-create base`). It is eventually unmounted, but
somehow the auto unmount code is confused, and tries to unmount it
one more time than it was mounted, which results in an error like
"umount: /tmp/image.THQkZxQa/mnt/tmp/ccache: not mounted".
This doesn't actually break anything, but it's a little messy.
2) "rm /tmp/ccache" in elements/base/finalise.d/02-remove-ccache never
succeeds in removing /tmp/ccache, because that hook is invoked by
run_d_in_target(), *while* /tmp/ccache is mounted.
This present commit solves the above by moving the ccache setup glue out
of img-functions and into the base element's root.d. This has the
following implications:
1) lib/img-functions is a little cleaner.
2) /tmp/ccache is available in the chroot during the root, extra-data,
pre-install, install and post-install stages. It is not available
during block-device, finalise and cleanup stages as it will have been
automatically unmounted by then.
3) /tmp/ccache won't be setup if you're building an image that doesn't
include the base element.
Change-Id: Ief4c0a6f4ec622db6c6f652776215684178d8943
Print a message and pause the build for 10 seconds to ensure interactive
users see the message.
Fixes bug #1212080
Change-Id: Ia388a54892c479e428b0ed7b8c70d64d65010e21
In some scenarios, the required space in the tmpfs partition can be
larger (or smaller) than the default one, producing errors due to
the lack of enough space (or performance penalties for not using
tmpfs).
Using --min-tmpfs <size>, we can hint the working set size we'll need
and let dib choose to avoid or use tmpfs.
Change-Id: I7d5fe498302a100c8555ae542268e14b21f3a0c5
Most of our build process runs as root and the overhead of explicitly
tracking sudo commands in the shipped sudoers.d file, plus the
operational overhead of keeping the sudoers.d file updated on build
hosts over time, is now considered to be unnecessary.
Instead, we now document that the build process will expect sudo
commands to work and the operator must make this happen (either by
running as root, via sudo -E, or allowing passwordless sudo for the
build user).
Change-Id: If2628e7d0c7efde0fb99c84a217523ba29b1b38d
Previously if mktemp failed in function mk_build_dir, die was not
called. The check for the exit code was actually checking that the
previous export statement completed successfully, not the call to
mktemp.
Change-Id: I477a2ce75c87c8167883ce7aa342e93c40770e29
- Ensures /sbin and friends are in $PATH when invoked (without this,
various sudo invocations fail in exciting ways).
- Use dib-run-parts in lib/common-functions instead of run-parts
(neither SLES nor openSUSE ship run-parts).
- Ensure dib-run-parts doesn't descend into subdirectories (same
behaviour as run-parts).
- Move dib-run-parts from root.d to bin (cleaner, consistent with
other elements with separate bin scripts).
- Tested by building Ubuntu image on openSUSE 12.3.
- Note: this doesn't add support for creating SUSE images, it just
lets you run disk-image-create on SUSE-based distros.
Change-Id: I906c6bc3cf51cdf2c4415adeae1ca250faac25e1
Ramdisks are now built inside a chroot which is built by the normal
image build process. Doing so improves our independence of the
precise state of the build host.
This fixes bug 1194055.
Change-Id: Ibc254fbb9e7b404b5f38c1b35bcde8a4136e8e28
RAMDISK_ELEMENT is a spurious difference, to merge the code we need
only semantic difference to exist.
Change-Id: I7d38b2457b6fc838d1a536406bb3c34c00f2b1f9
When partition tables are added to loopback devices, this can set off a
chain of udev hooks that may still be holding the loopback open. Failing
to detach loopback devices was the reason we were seeing leaked tmpfs
volumes.
Fixes bug #1178091
Change-Id: I836d6e2bbce824951dd4786e3ef28273ea18ee73
Ramdisk-image-create sources both img-functions and ramdisk-functions
causing duplicate functions. To correct this situation this patch
removes the import of img-functions from ramdisk-image-create.
It moves cleanup_dirs and run_d functions from img-functions to
common-functions to allow this.
fixes bug 1175427
Authored-by: Chris Krelle <nobodycam@gmail.com>
Change-Id: I971d019b0d92b06d9de661663b49c433e823ea42
For machines with low RAM (or no access to /proc/meminfo) the builder
will still run, but will just build inside the filesystem that is
hosting /tmp. This will result in a slower build (especially if there
are a lot of .deb packages installed).
Fixes bug #1175453
Change-Id: I79f2672058c11e377548820df0ab4fad8f47ffdc
A recent commit added a call to function cleanup_dirs to
the common-functions file, but this function is not in scope
for all code paths.
The problem can be reproduced like so:
bin/ramdisk-image-create deploy -o /tmp/foo
Which results in the error:
bin/../lib/common-functions: line 35: cleanup_dirs: command not found
This change causes cleanup_dirs to be in scope when calling ramdisk-image-create.
Change-Id: Iaf01beb9dd42a9d810d8aed1aa4ce19a5854c09b
Qemu-nbd does not perform well with older versions of qemu due to
the lack of writeback caching mode. It also only builds qcow2 images
and there is a desire for raw image support. Finally, qemu-nbd makes
it very difficult to build images concurrently due to the somewhat
opaque nature of how it selects a /dev/nbd# device. losetup, on
the other hand, makes this process very straight forward.
Change-Id: I309fad8af4fd1e8d1720c17b65e1897a76d5e897
Co-Author: Clint Byrum <clint@fewbar.com>
Using ELEMENTS_DIR env variable, you can specify multiple dirs containint your elements.
It must be a ":" separated elements list.
When an element is seen in one of those lists, it will be chosen and will continue with the next element.
Change-Id: I18eca27d943139cd6ca1ebd232b419e502d7b048
Move common openstack service installation operations
into a new script `os-svc-install`, which simplifies
the elements for openstack services.
Change-Id: Ied8ac3278e7fe8af76e24748ec4e598a84afa03c
