Currently we have all our elements and library files in a top-level
directory and install them into
<root>/share/diskimage-builder/[elements|lib] (where root is either /
or the root of a virtualenv).
The problem with this is that editable/development installs (pip -e)
do *not* install data_files. Thus we have no canonical location to
look for elements -- leading to the various odd things we do such as a
whole bunch of guessing at the top of disk-image-create and having a
special test-loader in tests/test_elements.py so we can run python
unit tests on those elements that have it.
data_files is really the wrong thing to use for what are essentially
assets of the program. data_files install works well for things like
config-files, init.d files or dropping documentation files.
By moving the elements under the diskimage_builder package, we always
know where they are relative to where we import from. In fact,
pkg_resources has an api for this which we wrap in the new
diskimage_builder/paths.py helper [1].
We use this helper to find the correct path in the couple of places we
need to find the base-elements dir, and for the paths to import the
library shell functions.
Elements such as svc-map and pkg-map include python unit-tests, which
we do not need tests/test_elements.py to special-case load any more.
They just get found automatically by the normal subunit loader.
I have a follow-on change (I69ca3d26fede0506a6353c077c69f735c8d84d28)
to move disk-image-create to a regular python entry-point.
Unfortunately, this has to move to work with setuptools. You'd think
a symlink under diskimage_builder/[elements|lib] would work, but it
doesn't.
[1] this API handles stuff like getting files out of .zip archive
modules, which we don't do. Essentially for us it's returning
__file__.
Change-Id: I5e3e3c97f385b1a4ff2031a161a55b231895df5b
Block device handling can be somewhat complex - especially
when taking things like md, lvm or encryption into account.
This patch factors out the creation and deletion of the local
loop image device handling into a python library.
The main propose of this patch is to implement the needed
infrastructure. Based on this, more advanced functions can be added.
Example: (advanced) partitioning, LVM, handling different boot
scenarios (BIOS, UEFI, ...), possibility of handling multiple images
(local loop image, iSCSI, physical hard disk, ...), handling of
different filesystems for different partitions / LVs.
Change-Id: Ib626b36a00f8a5dc3dbde8df3e2619a2438eaaf1
Signed-off-by: Andreas Florath <andreas@florath.net>
Under some systems this leads to an error if the oder of parameters
does not comply exactly with the way it is specified.
Change-Id: I9561b85985e3f0917f1b4c7801c9048b4e73ae3b
In the function "run_in_target", it is failed that force an empty
TMPDIR inside the chroot. The TMPDIR is still the user defined
tmp dir. Due to the TMPDIR dir is not exist, using "mktemp" to
create tmp dir is failed.
Change-Id: I898f80099bc2a7c32e8676014d0f4263807f7039
Closes-Bug: #1597569
There were a couple of functions which were unused:
ensure_nbd, map_nbd, unmount_qcow_image, mount_qcow_image, ensure_sudo
Because some of them use 'trap ... EXIT' this hinders introducing a
separate exit phase - therefore they are removed.
(It would also be impossible to use them in the current setup, because
they overwrite the 'trap ... EXIT' of the disk-image-creates 'main'.)
Change-Id: If932a557dca9aea4864154ad6c4f286373d6dd7c
Signed-off-by: Andreas Florath <andreas@florath.net>
We have some test cases which attempt to build docker images, therefore
we need docker.
Fix a few bugs that showed up when we run docker tests - we need to
docker rm with sudo and docker images don't always have a /tmp so check
before unmounting it.
Change-Id: I147d0ef3f2ea83f35bac568214573a6bde0b1967
This cuts the image size down alot, esspecially if there were lots of
small file deletes.
The fstrim utility is in the util-linux package and should be on
most all systems. fstrim also works with XFS, ext4, btrfs, etc
prodiving the kernel is new enough.
A reduction of 25% or more in size is common.
Change-Id: I269b4416be450369616f9b8e030f84c30e329804
This reverts commit 5184d02a7c.
The decision was made to go with fstrim because it is faster and more
universal that zerofree. The related-id has the patchset that implements
fstrim.
Related-Id: I269b4416be450369616f9b8e030f84c30e329804
Change-Id: If40cf2fc0ecd8686768cbfeac9ecee90907674e7
If the image has an ext filesystem and the zerofree utility is present
on the build system then run zerofree. This should make images as
compressable as possible which is a nice feature when building
compressed qcow2 images.
Change-Id: Ia6062c291f7a3f58b85a4f408ecb3d0574c65d53
uses upstream's stage4 images, includes all the needed bells and
whistles for openstack on kvm.
Change-Id: Ibca43173c30c2a74a73a2e2d9dd6d6d832c62694
Closes-Bug: 1530911
Currently when these files are opened your editor doesn't know what to
do with them. Add #!/bin/bash to library functions so that editors,
diff-tools, etc can do syntax highlighting.
There are other ways to skin this cat, such as renaming to ".sh",
adding -* style editor flags, etc. We had this discussion in DevStack
too, and came to the conclusion the simplest thing that works for
everyone is to just put the #! at the top.
Change-Id: I4cf64321e14844696139f5d40e4d719436390b35
Split the cleanup_dirs function in two, i.e. cleanup of the build dir
and the image dir, and use the former to cleanup the temporary build
subdirs after their unmount, before the conversion to other disk
formats; they are not needed anyway at that point, and allows to save
disk space during the conversion phase.
Change-Id: Ie30d7e6033613d6979148423326ae7e17a7342e7
Clean all the content in the /tmp directory of the guest, leaving the
directory itself (usually has special attributes).
Modern distributions usually either setup a tmpfs on /tmp, or clean it
at every boot, so the leftovers will be just few bytes in the generated
image.
Regarding other distributions, a clean /tmp at their first boot will
surely not be a bad idea anyway.
Change-Id: I2b0f8864bc4909542d924f5bd9296dca5d0189f2
Adding a test function which allows us to use elements to perform
element-specific tests. In order for this to work sanely, also adding
some configuration to our break system so we can assert on negative
tests.
Also adding a test for apt-sources to verify this code actually works.
Change-Id: I378a74255010eca192f5766b653f8a42404be5ea
In I084aff7e449f5de811a6169ec90e352ada7da439 we attemped to address a
bug for systems which dont have a path that works well for inside a
chroot. Turns out there were multiple issues:
* The PATH we were setting was after we attempted to call sh.
* PATH was being set in a sibling process to the command being run.
* PATH was not being exported so it was not effecting child processes.
Using env to set a sane path before we attempt to run our commands
addresses these issues.
Change-Id: I4285f8048465ee5c2490116447d32033007bd185
The build host might not necessarily have a complete set of dirs
required in its PATH. It will likely be better to statically code a
complete(ish) set of PATH entries inside the function.
Some distros (ex: archlinux) are lacking some dirs commonly found in a
PATH. If this is used as a host OS, it will transfer this incomplete
PATH into the chroot, where other guests (ubuntu, centos) will fail to
find basic binaries due to the lack of a /bin entry.
Change-Id: I084aff7e449f5de811a6169ec90e352ada7da439
ubuntu-signed element would install 'linux-signed-image-generic' that
provides signed kernel that can be used for deploy in UEFI secure boot mode.
Package 'linux-signed-image-generic' ships signed kernel with extension
'.efi.signed' (Ex. '/boot/vmlinuz-3.13.0-49-generic.efi.signed').
The kernel modules directory for signed kernel and unsigned kernel is same.
It is without 'efi.signed' extension to its name. This is different from normal
practice of directory naming in '/lib/modules' (Ex. For signed kernel
'vmlinuz-3.13.0-49-generic.efi.signed', modules directory is
'/lib/modules/3.13.0-49-generic').
This needed some changes in '/lib/ramdisk-functions' and 'ramdisk' element to
copy kernel modules.
The signed kernel package contains both signed and unsigned kernel. The
unsiged kernel is without extension '.efi.signed' (Ex.
'/boot/vmlinuz-3.13.0-49-generic'). This required change into
'/lib/img-functions' and 'baremetal' element to pick up signed kernel version
when this element is used.
Closes-Bug: 1443076
Change-Id: I60061cbea847b47fa752b9463cfd387e8e7f0635
Sometime, we will get "Device or resource busy" during the mount point
deleting, umount return 0, but the resouce is busy for a while, so need
to add sleep interval to wait resource free, then we can delete it.
Change-Id: Idaa219d12e847824960eec8907739add5d619d1a
Closes-Bug: 1332521
We currently use qemu-img convert with a raw source and dest when
building raw images. We can just mv the file for increased speed.
Change-Id: I3da095cb9ecad7224a121a434a9fb204132bf6df
ramdisk-image-create assumes the presence of vmlinuz file in OS image.
But ubuntu PPC LE images have only vmlinux file. This patch adds
support for using vmlinux file.
Change-Id: I5dd0b8ceb46b73be57d4c15b39b96f99b524fa3f
Closes-Bug: 1413362
After being deprecated two releases ago, finally remove any reference
for the support of first-boot.d
Change-Id: I08d67404ef48cad61db3b18fb86e970abfa5d2b6
There are some situations (use of libpam-tmpdir) where the $TMPDIR is
defined per user, so any sudo call will define an specific $TMPDIR for
the root user which doesn't exist inside the chroot.
Change-Id: I866651fed520007506c7be83837d3791ed9f2235
Closes-Bug: #1330290
When uploading images to multiple clouds it is possible that the same
image will be needed in multiple formats to accomodate hypervisors
across clouds. Update disk-image-create's -t flag to take a list of
desired output image formats so that a single disk-image-create can
output all of the desired image formats.
Change-Id: If121b2342ae888855ba435aa3189f039e985b812
qemu-img convert can take a few minutes to complete, or far longer
on heavily loaded systems. Notify the user that the image is
undergoing conversion to save them just seeing END PROFILING.
Change-Id: I1ad61ce5ed011b721d48e2d12fb42ef4aa5cd1f6
If we entered the cleanup trap due to exit with an error code we should
exit dib with an error code.
Change-Id: Iee1a05668b3239113fb91a2da0d9a66d7de4db6b
Finding the Debian boot kernel and initrd is collapsed into a
single statement so that the script doesn't bail out when
'set -o pipefail' is set and the *generic kernel and initrd are
absent.
Change-Id: Ifd616818f7387e6f102636c7cf186be6097f1d5c
Closes-Bug: 1335016
For some of the scripts, the unversioned name is not
good enough. so determine the link target of "vmlinuz" and
"initrd" to get the full versioned path.
Change-Id: I52f9b5435fafaf7925e2332058a9e41b26478f1c
Add a --options flag that allows disk-image-create to send custom
options to qemu-img calls. It is useful for forcing compat into
qemu-img created images.
Usage:
disk-image-create --qemu-img-options compat=0.10 base ubuntu
Change-Id: I73ee2c86abc115220d51ffebbbe9ce3c4d92f188
Closes-Bug: #1329746
The script to find best kernel from image (used by vm
and baremetal element) is duplicated and is not in sync
with each other. Moving the code to img-functions as a
function will reduce duplicated code and make it reusable
in future.
Since img-functions is not accessible in chroot env,
kernel selection is being moved from finalise.d to
cleanup.d in the vm element.
Change-Id: I8fbccc13a2c61a5191ef9ea5d2a8302a3e43b000
In some instances, the content of the image may want to convey specific
resolv.conf settings. For instance, if unbound is installed, it's also
desirable to set resolv.conf to point to it.
Short circuit diskimage-builder's friendliness around resolv.conf files
by not dying in a fire if it finds the file to be immutable.
Change-Id: I88632fb79289681dcf95d32f39a0ad658ba39a60
save_image is used to copy kernel and ramdisks out of the image, which
we will sometimes want to keep the source, and sometimes not. However
for the main image itself, the temp copy is never kept, so use mv
rather than cp and avoid the excess IO.
Change-Id: I5a9f0d69ffee3e6b872a8927537ac17f02f5aa4d
disk-image-get-kernel is pretty noisy and you see e.g.:
$> load-image overcloud-compute.qcow2
XXX -d '/tmp/image.lWGCgPoj' -o 'tmp' -i '/home/stack/overcloud-compute.qcow2' --
Extracting kernel + ramdisk from /home/stack/overcloud-compute.qcow2 and writing them to /tmp/image.lWGCgPoj
nbd 17554 0
nbd 17554 0
basename: missing operand
Try 'basename --help' for more information.
/dev/nbd0 disconnected
tmp-vmlinuz,tmp-initrd
Clean all this up so we just get:
$> load-image overcloud-compute.qcow2
Extracting kernel + ramdisk from /home/stack/overcloud-compute.qcow2 to tmp-vmlinuz and tmp-initrd in /tmp/image.g6b0lG88
Change-Id: I8971ec0bbcd87157b07fc17254c56bb9f9f2a597
Commit c7d80dd (Cleanup mount points automatically) removed the unmount
of $TMP_MOUNT_PATH/tmp/ccache in run_d_in_target() and moved the
"rm /tmp/ccache" to elements/base/finalise.d/02-remove-ccache. There
are two problems with this:
1) Not unmounting at the end of run_d_in_target() results in tmp/ccache
being bind mounted muliple times on top of itself (three times, if you
just run `disk-image-create base`). It is eventually unmounted, but
somehow the auto unmount code is confused, and tries to unmount it
one more time than it was mounted, which results in an error like
"umount: /tmp/image.THQkZxQa/mnt/tmp/ccache: not mounted".
This doesn't actually break anything, but it's a little messy.
2) "rm /tmp/ccache" in elements/base/finalise.d/02-remove-ccache never
succeeds in removing /tmp/ccache, because that hook is invoked by
run_d_in_target(), *while* /tmp/ccache is mounted.
This present commit solves the above by moving the ccache setup glue out
of img-functions and into the base element's root.d. This has the
following implications:
1) lib/img-functions is a little cleaner.
2) /tmp/ccache is available in the chroot during the root, extra-data,
pre-install, install and post-install stages. It is not available
during block-device, finalise and cleanup stages as it will have been
automatically unmounted by then.
3) /tmp/ccache won't be setup if you're building an image that doesn't
include the base element.
Change-Id: Ief4c0a6f4ec622db6c6f652776215684178d8943
A problem with unmounting the dev filesystem in Ubuntu images caused
the umount of the /dev bind mount to fail, which left it there to be
removed during the mv -t step, causing the build host's /dev to be
wiped out.
The lazy umount will detach it from the filesystem hierarchy and then
clean up the mount reference later.
Change-Id: I8f8cea857c445fb0b4fd02bc063722fb1553c947
