Block device handling can be somewhat complex - especially
when taking things like md, lvm or encryption into account.
This patch factors out the creation and deletion of the local
loop image device handling into a python library.
The main propose of this patch is to implement the needed
infrastructure. Based on this, more advanced functions can be added.
Example: (advanced) partitioning, LVM, handling different boot
scenarios (BIOS, UEFI, ...), possibility of handling multiple images
(local loop image, iSCSI, physical hard disk, ...), handling of
different filesystems for different partitions / LVs.
Change-Id: Ib626b36a00f8a5dc3dbde8df3e2619a2438eaaf1
Signed-off-by: Andreas Florath <andreas@florath.net>
I realised I'd been using die() in a few places assuming it was
available, but it wasn't exported. I guess it didn't matter because
whatever was wrong, we were failing anyway :)
This exports the function to make it available to sub-processes, which
should remove the need to source it as done in several places.
Change-Id: I7b9a5a6db406e160099b6ed9fde80455ae227327
As motivation for this; we have had two breakouts of dib in recent
memory. One was a failure to unmount through symlinks in the core
code (I335316019ef948758392b03e91f9869102a472b9) and the other was
removing host keys on the build-system
(Ib01d71ff9415a0ae04d963f6e380aab9ac2260ce).
For the most part, dib runs unprivileged. Bits of the core code are
hopefully well tested (modulo bugs like the first one!). We give free
reign inside the chroot (although there is still some potential there
for adverse external affects via bind mounts). Where we could be a
bit safer (and could have prevented at least the second of these
breakouts) is with some better checking that the "sudo" calls
*outside* the chroot at least looked sane.
This adds a basic check that we're using chroot or image paths when
calling sudo in those parts of elements that run *outside* the chroot.
Various files are updated to accomodate this check; mostly by just
ignoring it for existing code (I have not audited these calls).
Nobody is pretending this type of checking makes dib magically safe,
or removes the issues with it needing to do things as root during the
build. But this can help find egregious errors like the key removal.
Change-Id: I161a5aea1d29dcdc7236f70d372c53246ec73749
A TODO was placed on the partitioning section of the vm element to
replace sfdisk with a saner (and less arcane) way of partitioning. It
suggested parted for replacement. This changeset should reproduce the
same disk label and partition layout as sfdisk, but with less ioctl
errors and version dependency. It will also ensure partition alignment.
Change-Id: I5d8d75131458b73bfb05f80f1bfa7e2970e004b3
In order to add more flexibility to the vm and bootloader
elements, split the functionality in two different ones, and
make vm depend on bootloader element.
This will allow to construct more elements that depend on
bootloader, and develop both elements independently.
Change-Id: Iad2503b7b8fe53b768a3bc79e4cb839700fbd747
Grub fails to install the bootloader due to it being on the root
partition of a block device. This is not actually a problem for us, so
we need to force it to succeed.
Change-Id: I335ef04ca8a8a8a5c242d3444b09bcce0a9f51e7
As described in the comments, sfdisk was rewritten for util-linux 2.26
(as shipped in F22) and now interprets arguments a sectors, rather
than cylinders.
The current partitioning line is "1 - - *" (start/size/type/bootable)
which means you start getting:
---
/usr/sbin/grub2-install: warning: this msdos-style partition label has
no post-MBR gap; embedding won't be possible.
/usr/sbin/grub2-install: warning: Embedding is not possible. GRUB can
only be installed in this setup by using blocklists. However,
blocklists are UNRELIABLE and their use is discoura ged..
/usr/sbin/grub2-install: error: will not proceed with blocklists.
---
when building images, because the start is interpreted by the new
sfdisk as sector 1 and it crams the partition right next to the MBR.
Specifying "-" for the size is undefined in the man page; even reading
the source it's not totally clear what "-" for the size does [2]. In
any case, the alignment is wrong in sectors or cylinders; we want to
be a multiple of 4KiB for best performance.
The intent here is to create one single, Linux, bootable, partition
taking up the whole disk starting at 1MiB, so "2048 + L *" makes this
clear.
We use the -uS argument to ensure both versions treat this start-value
as a sector offset (newer sfdisk essentially ignores the argument).
As described in the comments, bugs in the older sfdisk necessitate
usage of "--force".
Although we could choose more or less, it seems most common to align
to a 1MiB boundary (i.e. starting at sector 2048). libguestfs has
some disucssion around --alignment and where it sets it's default to
this [3]. The 2.26-era sfdisk also defaults to putting partitions
here. 1MiB should be enough for GPT schemes in the future as well.
[1] https://github.com/karelzak/util-linux/blob/master/libfdisk/src/script.c#L1050
[2] https://bugzilla.redhat.com/show_bug.cgi?id=1249893
[3] http://libguestfs.org/virt-resize.1.html
Change-Id: I2c2966f98d1d5ad4ebb433ea148b3b26c65dc1b5
Now that we have a generic yum-minimal element, just use it in centos
instead of rinse. Adding base as an element-provides of yum-minimal
because this element conflicts with the base element.
Co-Authored-By: Gregory Haynes <greg@greghaynes.net>
Change-Id: I15275d821781171c118f21aa0c0bca55f65a65b3
We don't export DIB_RELEASE in the base distro scripts, so many
times it will be an unset variable in 51-bootloader from the vm
element. Currently that script doesn't handle the situation
properly and we get errors like:
line 160: [: =: unary operator expected
This change quotes the references to DIB_RELEASE so that won't
happen anymore.
Change-Id: Ic8f91804334f8862c217f465be80e96d3116f5c8
There is a wide variety of tracing options through the various shell
scripts. Some use "set -eux", others explicity set xtrace and others
do nothing. There is a "-x" option to bin/disk-image-create but it
doesn't flow down to the many scripts it calls.
This adds a global integer variable set by disk-image-create
DIB_DEBUG_TRACE. All scripts have a stanza added to detect this and
turn on tracing. Any other tracing methods are rolled into this. So
the standard header is
---
if [ "${DIB_DEBUG_TRACE:-0}" -gt 0 ]; then
set -x
fi
set -eu
set -o pipefail
---
Multiple -x options can be specified to dib-create-image, which
increases the value of DIB_DEBUG_TRACE. If script authors feel their
script should only trace at higher levels, they should modify the
"-gt" value. If they feel it should trace by default, they can modify
the default value also.
Changes to pachset 16 : scripts which currently trace themselves by
default have retained this behaviour with DIB_DEBUG_TRACE defaulting
to "1". This was done by running [1] on patch set 15. See the thread
beginning at [2]
dib-lint is also updated to look for the variable being matched.
[1] https://gist.github.com/ianw/71bbda9e6acc74ccd0fd
[2] http://lists.openstack.org/pipermail/openstack-dev/2014-November/051575.html
Change-Id: I6c5a962260741dcf6f89da9a33b96372a719b7b0
Patch adds support to create PowerPC image with vm element.
It creates 2 partitions, one for PReP boot and other for root and
installs grub-ieee1275.
Change-Id: I4675ef2b82aa69b63e63a1cc7db01b0c0e6f9fff
Closes-Bug: 1418739
Using lsb_release to figure out what distro and release an image is
is redundant, as the same information is available in DIB_ vars.
Further, it causes errors when creating new rhel7/centos7 images
like this: http://paste.openstack.org/show/144729/
Change-Id: I213ee581b8abfe119ffe3b783e6a3236e58e3e47
Per [1], our current root label of cloudimg-root does not work with
XFS because XFS only allows 12 character labels. This change
addresses that by allowing a custom rootfs label to be specified
in the call to disk-image-create. There should be no backwards
compatibility concerns as the default label is unchanged. Any
external elements dealing with the label would need to be updated
to support this new feature, but should continue to work as before
as long as a custom label is not specified.
[1]: https://bugzilla.redhat.com/show_bug.cgi?id=1139584
Change-Id: I596104d1a63b5dc6549e8460a1ae3da00165ef04
When running inside a Docker container, we cannot rely on devices in
/dev/mapper to be automagically created by udev, because we probably
don't have a udev at all. To work around this, run dmsetup mknodes
after every kpartx run.
Change-Id: If7e30579224ce54c5ed26d08974d8293c144719a
While building images in a dual boot host,
/dev/sda is leaking into build's grub config.
It is visible in the logs that the commands
GRUB-INSTALL and GRUB-MKCONFIG probes Host OS environment.
This is avoided by setting grub_disable_os_prober
to true in /default/grub config file.
Change-Id: I7408de7b3bc8a62f8bf103f00a88a596f988353c
Closes-Bug: 1166491
The previous method only configured serial console for Ubuntu or
Fedora, I noticed this wasn't working when I built RedHat 7.
After reading about how grub-mkconfig works, I realized that we
shouldn't be using sed to modify grub.cfg, and set up the proper
variables in /etc/default/grub
Change-Id: I02b73dc5074cb48a716849cb8a8a9b2f054bea31
Some extlinux packages install mbr.bin under /usr/lib/extlinux or
/usr/lib/EXTLINUX so tese directories need to be checked as well.
Also, there are packages that create a /boot/extlinux directory
which needs to be taken into account. Finally, commit
bccffc8bfd dropped the /boot prefix
from the kernel and initrd references in the ext/syslinux config
file which is simply a bug that needs fixing.
Change-Id: Idb071c9b18ff557b0f0f76d0d754536f2deca904
Closes-Bug: 1335042
Initial support for a centos7 image.
This is separate to rhel7 because the major differences are things
like repo and image locations, which are always going to be different.
We should merge any real changes into the redhat-common layers.
Apart from the added support files in centos7/*, the other change is
mostly modifications to redhat-common's extract-image to handle
different partition layouts of the centos7 image.
Change-Id: I943abe5ff0a803f36eda266a79af0d9220edcae7
Debian based distros do not enable the serial console by default
in grub.cfg, instead sending all boot messages to the display
Change-Id: I0e80859198e77862ab0301ab27ebe11702fba09d
The script to find best kernel from image (used by vm
and baremetal element) is duplicated and is not in sync
with each other. Moving the code to img-functions as a
function will reduce duplicated code and make it reusable
in future.
Since img-functions is not accessible in chroot env,
kernel selection is being moved from finalise.d to
cleanup.d in the vm element.
Change-Id: I8fbccc13a2c61a5191ef9ea5d2a8302a3e43b000
As with the previous similar changes, this is intended to catch
problems as they happen instead of ignoring them and continuing on
to potentially fail later. Setting this on all existing scripts
will allow us to enforce use via Jenkins.
Change-Id: Iad2d490c86dceab148ea9ab08f457c49a5d5352e
Fixes problems found by set -eu and pipefail, including:
-Many unset variables
-Commands that can fail under normal circumstances, which breaks
with set -e. This change swallows those expected errors to allow
our existing error code to handle them.
-The dkms element was not finding Fedora kernel versions correctly.
This may be an issue for other distros too, but since Fedora was
working fine without this functionality I only changed it to print
a warning message rather than failing the build when it happens.
-The ramdisk init script will not be set -eu because if it fails
the result is a kernel panic, which can be tricky to debug.
However, in testing with set -e a few failing commands were found
and have been fixed in this patch.
Change-Id: I44cf98dfc80cfcaec54b88cc83be80a3dbf2cec3
--target has been introduced in grub2. This patch ensure we use
the minimal version before adding the --target XXX flag.
Debian Wheezy and Precise share the same limitation regarding the
root device.
Change-Id: I5b3ad42b9fb536d13fb65d84339bc15bc23d99a3
The grub install scripts on Fedora have hardcoded checks for
/sys/firmware/efi and if the directory exists, it uses efi
specific instruction like initrdefi, linuxefi etc. vm element
should fix it once the configuration is generated.
Closes-bug: #1231946
Change-Id: I888304cb9167b8e074b04ca6c5b2f9143b70d37d
Those parameters allow to access the console for VMs using
extlinux (CentOS 6 for example). They are already added to
Fedora images.
Change-Id: I035170bd101062e004fb3782b5f8e5a7944a4f03
Closes-Bug: 1296899
Extlinux bootloader configuration in "vm" element requires vmlinuz and
initrd.img files in /boot/ with "generic" suffix. When these don't exist,
bootloader configuration should fall back to using whatever does exist.
Change-Id: I79d426e8f008e9f23db899f0b8f8cf2f23643e28
Reference to ${GRUB_OPTS:""} in 'vm' element causes syntax error
if GRUB_OPTS is set. Fix by changing to ${GRUB_OPTS:-""}.
Change-Id: Iae3ee8b7c4064c92d0cfc1b5a8caa22e53d6d112
If creating an image of a disto without grub2 extlinux is instead used,
extlinux was being configured to mount the root filesystem as
read/write which in turn causes e2fsck to fail to test the filesystem,
on RHEL 6.5 this causes the boot process to fail.
Change-Id: I07859e0df9a2bc1ddd8c5f2ed4509ba14312e140
Delete stale symlinks created by current openSUSE 13.1
for kpartx loopback partitions. Those break bootloader
installation otherwise.
Change-Id: I13a7387e341b4426b459610fd1d3c49ce5730f46
52-force-text-mode-console, running after 51-bootloader, calls
grub2-mkconfig (on Ubuntu that will happen via the update-grub call).
This call will overwrite all the changes that 51-bootloader did to the
grub.cfg. By merging the 52-force-text-mode-console code into
51-bootloader we avoid that. This also means that 51-bootloader will now
call grub2-mkconfig regardless of whether there is already and existing
grub2 configuration or not (52-force-text-mode-console did that in the
past anyway).
Additionally this commit enables the force to textmode code for openSUSE
(was previously only working correctly for Ubuntu and Fedora).
Closes-Bug: #1260323
Change-Id: Ida8e1a69df22d41ad70eb7154707c50613b536a8
This is currently experimental/incomplete. See
elements/opensuse/README.md for details.
Change-Id: Ie5bcab15c9703cf15ceb642fb986a8afc34f96fb
Co-Authored-By: Ralf Haferkamp <rhafer@suse.de>
Since we are using bash syntax in some of the element fragments,
we should make sure we use bash for all of them, so that things don't
break on systems where /bin/sh != /bin/bash.
Change-Id: If2f043c57aa4e1492b7f9839213ef6123f683612
Changing the grub config makes no sense in a build not heading for a vm
and may fail because grub is removed from images not including the vm
element. Forcing textmode for those images would be better done in nova.
Change-Id: I1c5b89e551e62df2463200b1889cb2342498c7dd
When uninstalling grub2, leave all its dependencies
including grub2-tools installed to minimise the number of packages
which need to be installed in the finalise stage.
Since the yum cache is unmounted during finalise, installing
grub2 in finalise is slowed by re-populating the yum cache.
This change copies the grub2 rpm out of the yum cache so it can be installed
from file during finalise.
This should prevent disk becoming full during finalise on Fedora.
Closes-Bug: #1217185
Change-Id: If095adc4abb52a19a3aa0b1caebfb3e4d8f605ef
