Adding a test function which allows us to use elements to perform
element-specific tests. In order for this to work sanely, also adding
some configuration to our break system so we can assert on negative
tests.
Also adding a test for apt-sources to verify this code actually works.
Change-Id: I378a74255010eca192f5766b653f8a42404be5ea
Instead of executing `lsb_release` to know the current distro being
built, use the $DISTRO_NAME environment variable, already provided by
distribution elements.
Change-Id: I22b67afb481983cc40c198fd408ad5c7f4d68bec
In I084aff7e449f5de811a6169ec90e352ada7da439 we attemped to address a
bug for systems which dont have a path that works well for inside a
chroot. Turns out there were multiple issues:
* The PATH we were setting was after we attempted to call sh.
* PATH was being set in a sibling process to the command being run.
* PATH was not being exported so it was not effecting child processes.
Using env to set a sane path before we attempt to run our commands
addresses these issues.
Change-Id: I4285f8048465ee5c2490116447d32033007bd185
The build host might not necessarily have a complete set of dirs
required in its PATH. It will likely be better to statically code a
complete(ish) set of PATH entries inside the function.
Some distros (ex: archlinux) are lacking some dirs commonly found in a
PATH. If this is used as a host OS, it will transfer this incomplete
PATH into the chroot, where other guests (ubuntu, centos) will fail to
find basic binaries due to the lack of a /bin entry.
Change-Id: I084aff7e449f5de811a6169ec90e352ada7da439
ubuntu-signed element would install 'linux-signed-image-generic' that
provides signed kernel that can be used for deploy in UEFI secure boot mode.
Package 'linux-signed-image-generic' ships signed kernel with extension
'.efi.signed' (Ex. '/boot/vmlinuz-3.13.0-49-generic.efi.signed').
The kernel modules directory for signed kernel and unsigned kernel is same.
It is without 'efi.signed' extension to its name. This is different from normal
practice of directory naming in '/lib/modules' (Ex. For signed kernel
'vmlinuz-3.13.0-49-generic.efi.signed', modules directory is
'/lib/modules/3.13.0-49-generic').
This needed some changes in '/lib/ramdisk-functions' and 'ramdisk' element to
copy kernel modules.
The signed kernel package contains both signed and unsigned kernel. The
unsiged kernel is without extension '.efi.signed' (Ex.
'/boot/vmlinuz-3.13.0-49-generic'). This required change into
'/lib/img-functions' and 'baremetal' element to pick up signed kernel version
when this element is used.
Closes-Bug: 1443076
Change-Id: I60061cbea847b47fa752b9463cfd387e8e7f0635
Sometime, we will get "Device or resource busy" during the mount point
deleting, umount return 0, but the resouce is busy for a while, so need
to add sleep interval to wait resource free, then we can delete it.
Change-Id: Idaa219d12e847824960eec8907739add5d619d1a
Closes-Bug: 1332521
Some elements (such as the manifests element) want to use these
variables. We currently do not consistently export them (IMAGE_NAME is
only exported if you actually specify a name).
Change-Id: I43d17ddcdd7d0ff3cbb4c530caeebb8da915f4ef
Copy all of the necessary parts for a Fedora based dhclient to work. This
includes a number of network scripts. Also grab the ip command supplied by
the iproute package, the busybox "ip addr" command was missing the valid_lft
and preferred_lft options.
This will allow the dhcp to work in the ramdisk instead of getting passed the
PXE net config.
Related-Bug: #1417026
Change-Id: I8feee9a740855dab7b47162c5727bf91db77fcc6
We currently use qemu-img convert with a raw source and dest when
building raw images. We can just mv the file for increased speed.
Change-Id: I3da095cb9ecad7224a121a434a9fb204132bf6df
I regularly see users report that their build fails because this unmount
line reports an error. Even though we dont bail here because of the ||
true, as a user it is hard to distinguish this from an error.
Change-Id: Ic43f4fb24c53c58329fdf501bba6ba14024ec2aa
Deprecated the `--expand-dependencies` flag from `element-info` usage.
The flag was required and not optional. We can rely on argparse to exit non-0
when the required positional argument is not provided.
Change-Id: Iaf8eb962eb600760974bc33c30b809a07a23278e
Closes-Bug: 1265649
When building the ramdisk we don't cleanup the temporary
directories after ourselves. This leaves /tmp/image.* directories
mounted and /tmp/image.* directories on the system.
Also the ramdisk-functions duplicate, from what I can see,
the cleanup function from common-functions. So when a job
is killed off it ends up leaving /tmp/image.* directories
on the system.
Change-Id: I2d73aabd0eb176027b4e7368580db08902e2b6ab
ramdisk-image-create assumes the presence of vmlinuz file in OS image.
But ubuntu PPC LE images have only vmlinux file. This patch adds
support for using vmlinux file.
Change-Id: I5dd0b8ceb46b73be57d4c15b39b96f99b524fa3f
Closes-Bug: 1413362
I would like to recommend to use + instead of \; in the find
command. As this will ensure the removal of all selected
directories in a single invocation.
Hence improve the speed of deletion.
Change-Id: I409fe11aae217afb6f790491591005c679264ed4
The element builds dracut from source on Ubuntu because the
Ubuntu dracut package is broken and very old, so it can't be
installed properly and causes a number of other issues that
are fixed by using a newer version of Dracut.
This initial version should work in virtualized environments.
Further validation of its suitability for real baremetal
deployments will need to be done in the future, but this should
be sufficient to enable that work.
Regarding Dracut specifically, in order to limit the changes
needed in the existing scripts this element continues to use a
cut down version of the /init script that we were building for the
existing ramdisk. However, instead of running it as pid 0 it is
run as a Dracut pre-mount hook. This allows Dracut to set up all
of the hardware and system bits, while falling early enough in the
Dracut sequence to complete the deployment before Dracut would try
to boot off the hard disk.
bp tripleo-juno-dracut-ramdisks
Change-Id: I144c8993fe040169f440bd4f7a428fdbe3d745cf
Until now there was a possibility for two elements to install hooks
with the same name, so one of them was overwritten. Change logic to
copy the hooks and fail in case one with the same name exists.
Change-Id: Ic2c46835b27c9319f7a889ffd0ccf3f5ccc1f0cd
Closes-Bug: 1251952
After being deprecated two releases ago, finally remove any reference
for the support of first-boot.d
Change-Id: I08d67404ef48cad61db3b18fb86e970abfa5d2b6
There are some situations (use of libpam-tmpdir) where the $TMPDIR is
defined per user, so any sudo call will define an specific $TMPDIR for
the root user which doesn't exist inside the chroot.
Change-Id: I866651fed520007506c7be83837d3791ed9f2235
Closes-Bug: #1330290
When uploading images to multiple clouds it is possible that the same
image will be needed in multiple formats to accomodate hypervisors
across clouds. Update disk-image-create's -t flag to take a list of
desired output image formats so that a single disk-image-create can
output all of the desired image formats.
Change-Id: If121b2342ae888855ba435aa3189f039e985b812
qemu-img convert can take a few minutes to complete, or far longer
on heavily loaded systems. Notify the user that the image is
undergoing conversion to save them just seeing END PROFILING.
Change-Id: I1ad61ce5ed011b721d48e2d12fb42ef4aa5cd1f6
If we entered the cleanup trap due to exit with an error code we should
exit dib with an error code.
Change-Id: Iee1a05668b3239113fb91a2da0d9a66d7de4db6b
A user reported symptoms where the losetup line used to detach the
loopback device was failing in tar mode. We don't need to detach a
device that does not exist.
Change-Id: I807996e16199288927b49b4f300ae9b461cb8fe7
Closes-Bug: #1378033
Finding the Debian boot kernel and initrd is collapsed into a
single statement so that the script doesn't bail out when
'set -o pipefail' is set and the *generic kernel and initrd are
absent.
Change-Id: Ifd616818f7387e6f102636c7cf186be6097f1d5c
Closes-Bug: 1335016
When running inside a Docker container, we cannot rely on devices in
/dev/mapper to be automagically created by udev, because we probably
don't have a udev at all. To work around this, run dmsetup mknodes
after every kpartx run.
Change-Id: If7e30579224ce54c5ed26d08974d8293c144719a
For some of the scripts, the unversioned name is not
good enough. so determine the link target of "vmlinuz" and
"initrd" to get the full versioned path.
Change-Id: I52f9b5435fafaf7925e2332058a9e41b26478f1c
Now that dib-run-parts has been moved to the dib-utils project, we
need to update diskimage-builder to use it instead of the version
directly in diskimage-builder.
This change removes the old copy of the dib-run-parts script in
the element, adds dib-utils as a dependency of diskimage-builder,
and updates the uses of dib-run-parts to correctly handle the fact
that it is now external to the project.
Requires I0be1f876d0e4a7d38e0d5c6010a552a8ebb158a4
Change-Id: Ia0a0df7784a14c49b5c47ac0b03e6c2602c84b3b
We need to be able to do install.d like things for ramdisks
themselves, but install.d runs outside the ramdisk context - and its
likely to break peoples brains if we mangle the two together - so this
adds a new hook point, ramdisk-install, specifically for installing
things into the ramdisk.
Change-Id: I37d1660309cda6e28bd0b316b08f61db4e080613
Optimized the creation of directories by using the Brace expansion mechanism.
And corrected the dirname resolution to follow correct convention.
Change-Id: I048c04385d2706c4e38f8548555a59b0437e1fcd
The two duplicated functions, save_image and finish_image will move
an existing image out of the way if it exists, but it isn't
configurable. Check an environment variable is 0 before doing so.
Switch save_image to just calling finish_image, rather than
duplicating its code exactly.
Change-Id: I26a5a8fa4b6e853c9440bffab195b0bc3728be40
Add a --options flag that allows disk-image-create to send custom
options to qemu-img calls. It is useful for forcing compat into
qemu-img created images.
Usage:
disk-image-create --qemu-img-options compat=0.10 base ubuntu
Change-Id: I73ee2c86abc115220d51ffebbbe9ce3c4d92f188
Closes-Bug: #1329746
On Debian, /lib/firmware is provided by optional packages like
firmware-iwlwifi or firmware-linux-free. That's why this directory
may not exist.
This change ensure the directory exist before trying to copy it
content.
Change-Id: Icf9eedf4148141c92e9171f16ae744b88a8d8519
The script to find best kernel from image (used by vm
and baremetal element) is duplicated and is not in sync
with each other. Moving the code to img-functions as a
function will reduce duplicated code and make it reusable
in future.
Since img-functions is not accessible in chroot env,
kernel selection is being moved from finalise.d to
cleanup.d in the vm element.
Change-Id: I8fbccc13a2c61a5191ef9ea5d2a8302a3e43b000
If we have the old image file with same name as new image file
we shouldn't rewrite it, just need to rename old.
Uses date number in old image name.
Fixes TODO in lib/common-functions
Change-Id: Ica58a0f4435217912bddf950322299d2ed2889f3
Ensure the return code of element-info is actually checked by moving the
export IMAGE_ELEMENT to the subsequent line.
A recent change (Ic0a061995e2ae708c95a535cba6237bff58f7d93) exported the
IMAGE_ELEMENT variable. This caused the return code of element-info to not
actually be checked, since the export command is now checked instead and
almost always will return 0.
This caused a misleading error message when using diskimage-builder with an
unknown element since the script didn't exit when it should have:
$ bin/disk-image-create -a amd64 foobar vm
Building elements: base foobar vm
ERROR: Element 'foobar' not found in 'bin/../elements'
Expanded element dependencies to:
Building in /var/tmp/image.DuIDY1lW
Please include at least one distribution root element.
Change-Id: I3d9c02bf0b1a4f320280185dd3fab0697d2c92f2
Adds a new pkg-map element which can be used to install
packages based on an in element 'pkg-map' JSON config
file format.
As part of this change we expose DIB's IMAGE_ELEMENT variable
so that we can have access to the element names which are being
installed in a clean manner.
This change is intended to decouple elements from DIB
and allow new elements to support multiple distributions
(with various package naming schemes) without having to
constantly maintain DIB's various bin/map-packages files.
Change-Id: Ic0a061995e2ae708c95a535cba6237bff58f7d93
If the tmpfs being used to hold the image filesystem is close to being
full then housing the raw disk image in the same place can be
problematic as it tends to grow faster then the filesystem shrinks when
data is being moved into it.
Putting them both into separate tmpfs's will allow there to be an
overlap where they jointly use more then the size limit for a singe tmpfs.
Change-Id: Ia17ca357d3b865d70a2d3e13e0479d008ca5f924
Closes-Bug: #1289582
In some instances, the content of the image may want to convey specific
resolv.conf settings. For instance, if unbound is installed, it's also
desirable to set resolv.conf to point to it.
Short circuit diskimage-builder's friendliness around resolv.conf files
by not dying in a fire if it finds the file to be immutable.
Change-Id: I88632fb79289681dcf95d32f39a0ad658ba39a60
Fixes problems found by set -eu and pipefail, including:
-Many unset variables
-Commands that can fail under normal circumstances, which breaks
with set -e. This change swallows those expected errors to allow
our existing error code to handle them.
-The dkms element was not finding Fedora kernel versions correctly.
This may be an issue for other distros too, but since Fedora was
working fine without this functionality I only changed it to print
a warning message rather than failing the build when it happens.
-The ramdisk init script will not be set -eu because if it fails
the result is a kernel panic, which can be tricky to debug.
However, in testing with set -e a few failing commands were found
and have been fixed in this patch.
Change-Id: I44cf98dfc80cfcaec54b88cc83be80a3dbf2cec3
save_image is used to copy kernel and ramdisks out of the image, which
we will sometimes want to keep the source, and sometimes not. However
for the main image itself, the temp copy is never kept, so use mv
rather than cp and avoid the excess IO.
Change-Id: I5a9f0d69ffee3e6b872a8927537ac17f02f5aa4d
This will help to ensure images that would cause bug #1289582 would
still fail even on systems that have more RAM. Also this will ensure we
only reserve enough tmpfs space for this build.
Change-Id: Icece64e691d126a5fc89262897630fdbef1ffdf1
On some systems ldd gives a slightly different output for VDSOs. It doesn't
contain a '=>'. E.g.:
linux-vdso.so.1 (0x00007fff2f4a6000)
instead of:
linux-vdso.so.1 => (0x00007fff2f4a6000)
This patch simply skips all entries in the ldd output which don't expand to a
path name to workaround that.
Change-Id: Ie37637890b775b36bb31af4e586e61131bd80fa8
cleanup for ramdisk failed to umount TMP_BUILD_DIR with device or
resource busy error.
The patch unmounts all the mountpoints under TMP_BUILD_DIR and
detaches loop devices associated with TMP_IMAGE_PATH.
The unmounts are applied with both force(-f) and lazy(-l)
options. Force option is only for NFS mounts, it's kept here since no
harm for lazy option.
Change-Id: I84035e6a003d8135186b2fda3facbd2c37967529
The sbin paths (/sbin, /usr/sbin, /usr/local/sbin) need to be in $PATH.
Since $PATH is set in the build environment from the host, if the host
does not have these paths set, they need to be added to $PATH. In
particular, they must be set for apt-get which calls out to ldconfig,
start-stop-daemon, etc.
The previous check was insufficient because if you had /usr/sbin or
/usr/local/sbin in your $PATH, the addition to $PATH was not done since
the "if" statement evaluated to true.
Change-Id: I52b41d2eb9794e7ee0ae600215cb3ceca191582f
disk-image-get-kernel is pretty noisy and you see e.g.:
$> load-image overcloud-compute.qcow2
XXX -d '/tmp/image.lWGCgPoj' -o 'tmp' -i '/home/stack/overcloud-compute.qcow2' --
Extracting kernel + ramdisk from /home/stack/overcloud-compute.qcow2 and writing them to /tmp/image.lWGCgPoj
nbd 17554 0
nbd 17554 0
basename: missing operand
Try 'basename --help' for more information.
/dev/nbd0 disconnected
tmp-vmlinuz,tmp-initrd
Clean all this up so we just get:
$> load-image overcloud-compute.qcow2
Extracting kernel + ramdisk from /home/stack/overcloud-compute.qcow2 to tmp-vmlinuz and tmp-initrd in /tmp/image.g6b0lG88
Change-Id: I8971ec0bbcd87157b07fc17254c56bb9f9f2a597
busybox on rhel has no `--list' option, which cause populate_busybox
can not create busybox symlinks. So the ramdisk created will not work
totally.
Fixes bug #1257669
Change-Id: I4f905673bb577867c99e0ccc0756c56a292474f5
Converts our existing default root element code to be just a check
which exits with a failure message if no root/distribution element
is found.
Change-Id: I954a6abfd7871d5807b1a171a03fa98932410cff
ramdisk-defaults file is sourced before chroot to an image being created
is executed. Busybox path should be set inside chroot instead.
Change-Id: Ic0e1d3093ca9d489dd96b14157e93b336cf0baee
We now forcibly remove all framebuffer video drivers from ramdisks, as
part of our ongoing effort to keep every kernel we boot, in text mode,
so it can be more easily intercepted by LOM hardware, nova console
logging, etc.
Fixes-Bug: #1178409
Change-Id: Ia72211c97c03c3f60eb9a1483297fe587de51f0b
Commit c7d80dd (Cleanup mount points automatically) removed the unmount
of $TMP_MOUNT_PATH/tmp/ccache in run_d_in_target() and moved the
"rm /tmp/ccache" to elements/base/finalise.d/02-remove-ccache. There
are two problems with this:
1) Not unmounting at the end of run_d_in_target() results in tmp/ccache
being bind mounted muliple times on top of itself (three times, if you
just run `disk-image-create base`). It is eventually unmounted, but
somehow the auto unmount code is confused, and tries to unmount it
one more time than it was mounted, which results in an error like
"umount: /tmp/image.THQkZxQa/mnt/tmp/ccache: not mounted".
This doesn't actually break anything, but it's a little messy.
2) "rm /tmp/ccache" in elements/base/finalise.d/02-remove-ccache never
succeeds in removing /tmp/ccache, because that hook is invoked by
run_d_in_target(), *while* /tmp/ccache is mounted.
This present commit solves the above by moving the ccache setup glue out
of img-functions and into the base element's root.d. This has the
following implications:
1) lib/img-functions is a little cleaner.
2) /tmp/ccache is available in the chroot during the root, extra-data,
pre-install, install and post-install stages. It is not available
during block-device, finalise and cleanup stages as it will have been
automatically unmounted by then.
3) /tmp/ccache won't be setup if you're building an image that doesn't
include the base element.
Change-Id: Ief4c0a6f4ec622db6c6f652776215684178d8943
Print a message and pause the build for 10 seconds to ensure interactive
users see the message.
Fixes bug #1212080
Change-Id: Ia388a54892c479e428b0ed7b8c70d64d65010e21
A problem with unmounting the dev filesystem in Ubuntu images caused
the umount of the /dev bind mount to fail, which left it there to be
removed during the mv -t step, causing the build host's /dev to be
wiped out.
The lazy umount will detach it from the filesystem hierarchy and then
clean up the mount reference later.
Change-Id: I8f8cea857c445fb0b4fd02bc063722fb1553c947
In some scenarios, the required space in the tmpfs partition can be
larger (or smaller) than the default one, producing errors due to
the lack of enough space (or performance penalties for not using
tmpfs).
Using --min-tmpfs <size>, we can hint the working set size we'll need
and let dib choose to avoid or use tmpfs.
Change-Id: I7d5fe498302a100c8555ae542268e14b21f3a0c5
When compressing an image, this is done in the same dir where the raw
image resides, doubling the amount of space needed (scarce when
using tmpfs), and then it's moved to the .cache folder in disk.
Combining these two functions, we reduce the amount of space needed
in the tmpfs partition (when in use), and the compressed image is
created directly on the .cache folder disk, so there is no need to
move the compressed image after the process into disk.
Change-Id: I451d24bdd6fa0983414244135dff5e96c0549833
here is a bug associated with this patch because there is a
workaround included. Calling sync to enable the unmounting...
https://bugs.launchpad.net/diskimage-builder/+bug/1214388
Change-Id: I9d5a1f092fd8f3ae1c14de03dd516203baab4da3
Manually listing the mount points that need cleanup is not
maintainable and makes it hard to write elements that use bind
mounting to inject resources into the build environment.
Change-Id: I7c9ade444f4ebe42552f8e321f257a7ec0a674ef
Most of our build process runs as root and the overhead of explicitly
tracking sudo commands in the shipped sudoers.d file, plus the
operational overhead of keeping the sudoers.d file updated on build
hosts over time, is now considered to be unnecessary.
Instead, we now document that the build process will expect sudo
commands to work and the operator must make this happen (either by
running as root, via sudo -E, or allowing passwordless sudo for the
build user).
Change-Id: If2628e7d0c7efde0fb99c84a217523ba29b1b38d
A user running di-b several times while developing an element may not
want to drop to a shell in all cases but may only want to do so if one
of their in target hooks failed.
This patch gives them the ability to do so, If break=after-error is set
then a user will be provided a in target shell taking over from where the
last failed command left off.
Change-Id: Ia2f7ac4c21b64b971f87f4ae9cb867981b13eb5e
