Due to a bug in how we were running some of our phases we were not
detecting some nonzero exit's. When this is fixed, dib fails early during
cleanup (leaving some resources attatched) due to not propertly
accounting for pipefail.
Change-Id: Icc0b35acbe035cac12a9291e2d07b6c690c3a6ad
With a slow file system, umount can return 0 and the immediately
following remove can fail with a "Device or resource busy" error.
This happened in DevStack in disk-image-create where unmount_image
is followed by an immediate cleanup_build_dir.
Solution is to apply same logic from bug 1332521 to allow the
remove to retry on failure (up to 5s) in case the umount has not
completed.
Change-Id: I3337e2b4ad0111e77f79dc179439cdfea8ebdeda
Closes-Bug: #1527721
Currently when these files are opened your editor doesn't know what to
do with them. Add #!/bin/bash to library functions so that editors,
diff-tools, etc can do syntax highlighting.
There are other ways to skin this cat, such as renaming to ".sh",
adding -* style editor flags, etc. We had this discussion in DevStack
too, and came to the conclusion the simplest thing that works for
everyone is to just put the #! at the top.
Change-Id: I4cf64321e14844696139f5d40e4d719436390b35
Temp dirs are created with mktemp and thus belong to the user. There
is no need to chown them unless we used `mount -t tmpfs`.
Move chown under the tmpfs_check conditional.
Change-Id: I37efe18ced3a06d461364dc5cb20600f1527e995
This reverts commit ea4a823810.
This function was actually still in use in lib/common-functions
and removing it causes the disk-image-get-kernel /bin
to fail entirely.
Change-Id: Icddb3ca369922a6ea915af8b1b62c434cb1bdf28
Closes-bug: 1464031
Split the cleanup_dirs function in two, i.e. cleanup of the build dir
and the image dir, and use the former to cleanup the temporary build
subdirs after their unmount, before the conversion to other disk
formats; they are not needed anyway at that point, and allows to save
disk space during the conversion phase.
Change-Id: Ie30d7e6033613d6979148423326ae7e17a7342e7
This allow custom elements to be added with symlink. Without -follow
a symlinked element is valid but scripts in *.d directory aren't used.
Change-Id: If50b7d9c3b1f6fe278c28488146709efe5cf065f
Closes-Bug: 1461124
By that point in the build it isn't generally useful, and it causes
confusion when builds fail because people think that's the error.
Change-Id: I26dee4ac0947b71a4a065ef6c5a18103e7df6667
Given this is often the final output, it can look like an error occured.
Changing the wording makes this clearer.
Change-Id: I70f157054e3120cffee6fa5241b1ffe0b7bfa650
I regularly see users report that their build fails because this unmount
line reports an error. Even though we dont bail here because of the ||
true, as a user it is hard to distinguish this from an error.
Change-Id: Ic43f4fb24c53c58329fdf501bba6ba14024ec2aa
Deprecated the `--expand-dependencies` flag from `element-info` usage.
The flag was required and not optional. We can rely on argparse to exit non-0
when the required positional argument is not provided.
Change-Id: Iaf8eb962eb600760974bc33c30b809a07a23278e
Closes-Bug: 1265649
When building the ramdisk we don't cleanup the temporary
directories after ourselves. This leaves /tmp/image.* directories
mounted and /tmp/image.* directories on the system.
Also the ramdisk-functions duplicate, from what I can see,
the cleanup function from common-functions. So when a job
is killed off it ends up leaving /tmp/image.* directories
on the system.
Change-Id: I2d73aabd0eb176027b4e7368580db08902e2b6ab
The element builds dracut from source on Ubuntu because the
Ubuntu dracut package is broken and very old, so it can't be
installed properly and causes a number of other issues that
are fixed by using a newer version of Dracut.
This initial version should work in virtualized environments.
Further validation of its suitability for real baremetal
deployments will need to be done in the future, but this should
be sufficient to enable that work.
Regarding Dracut specifically, in order to limit the changes
needed in the existing scripts this element continues to use a
cut down version of the /init script that we were building for the
existing ramdisk. However, instead of running it as pid 0 it is
run as a Dracut pre-mount hook. This allows Dracut to set up all
of the hardware and system bits, while falling early enough in the
Dracut sequence to complete the deployment before Dracut would try
to boot off the hard disk.
bp tripleo-juno-dracut-ramdisks
Change-Id: I144c8993fe040169f440bd4f7a428fdbe3d745cf
Until now there was a possibility for two elements to install hooks
with the same name, so one of them was overwritten. Change logic to
copy the hooks and fail in case one with the same name exists.
Change-Id: Ic2c46835b27c9319f7a889ffd0ccf3f5ccc1f0cd
Closes-Bug: 1251952
After being deprecated two releases ago, finally remove any reference
for the support of first-boot.d
Change-Id: I08d67404ef48cad61db3b18fb86e970abfa5d2b6
When uploading images to multiple clouds it is possible that the same
image will be needed in multiple formats to accomodate hypervisors
across clouds. Update disk-image-create's -t flag to take a list of
desired output image formats so that a single disk-image-create can
output all of the desired image formats.
Change-Id: If121b2342ae888855ba435aa3189f039e985b812
If we entered the cleanup trap due to exit with an error code we should
exit dib with an error code.
Change-Id: Iee1a05668b3239113fb91a2da0d9a66d7de4db6b
A user reported symptoms where the losetup line used to detach the
loopback device was failing in tar mode. We don't need to detach a
device that does not exist.
Change-Id: I807996e16199288927b49b4f300ae9b461cb8fe7
Closes-Bug: #1378033
When running inside a Docker container, we cannot rely on devices in
/dev/mapper to be automagically created by udev, because we probably
don't have a udev at all. To work around this, run dmsetup mknodes
after every kpartx run.
Change-Id: If7e30579224ce54c5ed26d08974d8293c144719a
Now that dib-run-parts has been moved to the dib-utils project, we
need to update diskimage-builder to use it instead of the version
directly in diskimage-builder.
This change removes the old copy of the dib-run-parts script in
the element, adds dib-utils as a dependency of diskimage-builder,
and updates the uses of dib-run-parts to correctly handle the fact
that it is now external to the project.
Requires I0be1f876d0e4a7d38e0d5c6010a552a8ebb158a4
Change-Id: Ia0a0df7784a14c49b5c47ac0b03e6c2602c84b3b
We need to be able to do install.d like things for ramdisks
themselves, but install.d runs outside the ramdisk context - and its
likely to break peoples brains if we mangle the two together - so this
adds a new hook point, ramdisk-install, specifically for installing
things into the ramdisk.
Change-Id: I37d1660309cda6e28bd0b316b08f61db4e080613
The two duplicated functions, save_image and finish_image will move
an existing image out of the way if it exists, but it isn't
configurable. Check an environment variable is 0 before doing so.
Switch save_image to just calling finish_image, rather than
duplicating its code exactly.
Change-Id: I26a5a8fa4b6e853c9440bffab195b0bc3728be40
If we have the old image file with same name as new image file
we shouldn't rewrite it, just need to rename old.
Uses date number in old image name.
Fixes TODO in lib/common-functions
Change-Id: Ica58a0f4435217912bddf950322299d2ed2889f3
Ensure the return code of element-info is actually checked by moving the
export IMAGE_ELEMENT to the subsequent line.
A recent change (Ic0a061995e2ae708c95a535cba6237bff58f7d93) exported the
IMAGE_ELEMENT variable. This caused the return code of element-info to not
actually be checked, since the export command is now checked instead and
almost always will return 0.
This caused a misleading error message when using diskimage-builder with an
unknown element since the script didn't exit when it should have:
$ bin/disk-image-create -a amd64 foobar vm
Building elements: base foobar vm
ERROR: Element 'foobar' not found in 'bin/../elements'
Expanded element dependencies to:
Building in /var/tmp/image.DuIDY1lW
Please include at least one distribution root element.
Change-Id: I3d9c02bf0b1a4f320280185dd3fab0697d2c92f2
Adds a new pkg-map element which can be used to install
packages based on an in element 'pkg-map' JSON config
file format.
As part of this change we expose DIB's IMAGE_ELEMENT variable
so that we can have access to the element names which are being
installed in a clean manner.
This change is intended to decouple elements from DIB
and allow new elements to support multiple distributions
(with various package naming schemes) without having to
constantly maintain DIB's various bin/map-packages files.
Change-Id: Ic0a061995e2ae708c95a535cba6237bff58f7d93
If the tmpfs being used to hold the image filesystem is close to being
full then housing the raw disk image in the same place can be
problematic as it tends to grow faster then the filesystem shrinks when
data is being moved into it.
Putting them both into separate tmpfs's will allow there to be an
overlap where they jointly use more then the size limit for a singe tmpfs.
Change-Id: Ia17ca357d3b865d70a2d3e13e0479d008ca5f924
Closes-Bug: #1289582
Fixes problems found by set -eu and pipefail, including:
-Many unset variables
-Commands that can fail under normal circumstances, which breaks
with set -e. This change swallows those expected errors to allow
our existing error code to handle them.
-The dkms element was not finding Fedora kernel versions correctly.
This may be an issue for other distros too, but since Fedora was
working fine without this functionality I only changed it to print
a warning message rather than failing the build when it happens.
-The ramdisk init script will not be set -eu because if it fails
the result is a kernel panic, which can be tricky to debug.
However, in testing with set -e a few failing commands were found
and have been fixed in this patch.
Change-Id: I44cf98dfc80cfcaec54b88cc83be80a3dbf2cec3
save_image is used to copy kernel and ramdisks out of the image, which
we will sometimes want to keep the source, and sometimes not. However
for the main image itself, the temp copy is never kept, so use mv
rather than cp and avoid the excess IO.
Change-Id: I5a9f0d69ffee3e6b872a8927537ac17f02f5aa4d
This will help to ensure images that would cause bug #1289582 would
still fail even on systems that have more RAM. Also this will ensure we
only reserve enough tmpfs space for this build.
Change-Id: Icece64e691d126a5fc89262897630fdbef1ffdf1
cleanup for ramdisk failed to umount TMP_BUILD_DIR with device or
resource busy error.
The patch unmounts all the mountpoints under TMP_BUILD_DIR and
detaches loop devices associated with TMP_IMAGE_PATH.
The unmounts are applied with both force(-f) and lazy(-l)
options. Force option is only for NFS mounts, it's kept here since no
harm for lazy option.
Change-Id: I84035e6a003d8135186b2fda3facbd2c37967529
disk-image-get-kernel is pretty noisy and you see e.g.:
$> load-image overcloud-compute.qcow2
XXX -d '/tmp/image.lWGCgPoj' -o 'tmp' -i '/home/stack/overcloud-compute.qcow2' --
Extracting kernel + ramdisk from /home/stack/overcloud-compute.qcow2 and writing them to /tmp/image.lWGCgPoj
nbd 17554 0
nbd 17554 0
basename: missing operand
Try 'basename --help' for more information.
/dev/nbd0 disconnected
tmp-vmlinuz,tmp-initrd
Clean all this up so we just get:
$> load-image overcloud-compute.qcow2
Extracting kernel + ramdisk from /home/stack/overcloud-compute.qcow2 to tmp-vmlinuz and tmp-initrd in /tmp/image.g6b0lG88
Change-Id: I8971ec0bbcd87157b07fc17254c56bb9f9f2a597
