The bootloader element now has variable
DIB_BOOTLOADER_VIRTUAL_TERMINAL to customize or suppress the
console=tty0 kernel argument.
This is proposed to allow console=tty0 to be removed entirely as it is
causing significant performance degredation in DPDK environments.
Change-Id: Iba2ee5b8a6b4acdd236a770550dffd29c784ce11
Related: rhbz#2179366
Adds an element whose purpose is to set the stage
in the resulting image so that a user can generate an
image utilizing DIB which can be used in a FIPS
configuration without doing so with the input image
or after the fact.
Change-Id: Ia8a45584a56f6e06856fc2920c333351935dcd9d
When your booting a Linux system using dracut, i.e. with any
redhat style distribution, dracut's internal code looks to validate
the kernel hmac signature in before proceeding to userspace.
It does this by looking at the /boot/ folder file for the kernel
hmac file.
And it normally does this with the root filesystem. Except if the
kernel is not on the root filesystem and is instead on a /boot
filesystem, this breaks horribly. This is compounded because
DIB enables the operator to restructure the OS image/layout
to fit their needs. In order for this to be navigated, as dracut
is written, we need to pass a "boot=" argument to the kernel.
So now we attempt to purge any prior boot entry in the disk image
content, which is good because any filesystem operations invalidate
it, and then we attempt to identify the boot filesystem, and save a
boot kernel command line parameter so the resulting image can
boot properly if FIPS was enabled in the prior image.
Regex developed with https://sed.js.org utilizing stdin:
VAR="quiet boot=UUID=173c759f-1302-48a3-9d51-a17784c21e03 text"
VAR="quiet boot=PARTUUID=173c759f-1302-48a3-9d51-a17784c21e03"
VAR="quiet boot=PARTUUID=173c759f-1302-48a3-9d51-a17784c21e03 reboot=meow"
VAR="quiet boot=UUID=/dev/sda1 text"
VAR="quiet boot=/dev/sda1"
VAR="quiet boot=/dev/sda1 reboot=meow"
VAR="quiet after_boot=1 reboot=meow boot=/dev/sda1"
VAR="quiet after_boot=1 reboot=meow"
Which resulted in stdout:
VAR="quiet text"
VAR="quiet"
VAR="quiet reboot=meow"
VAR="quiet text"
VAR="quiet"
VAR="quiet reboot=meow"
VAR="quiet after_boot=1 reboot=meow"
VAR="quiet after_boot=1 reboot=meow"
Change-Id: I9034c21e84deda2ba2c0ec0d1d6d6595ed10bed4
Creating a separate /boot partition is desirable in some cases[1].
This change detects if /boot is a partition, and ensures that the
kernel/ramdisk paths are correct in either case. This is applied to
all BLS entries files, whether they were generated by the previous
grub2-mkconfig call or in the source image.
This means the rhel9 specific workaround can be removed since all
paths are now normalised at this stage.
[1] https://review.opendev.org/c/openstack/tripleo-image-elements/+/846807
Change-Id: I62120ec8c65876e451532d2654d37435eb3606a6
Resolves: rhbz#2101514
After some recent reordering[0], the /boot/grub directory isn't created
early enough on Gentoo any more, let us just ensure ourselves that it is
in place when we create the grub config.
[0] I8cb34914bbbfa05521bbb71cc6637368b980358f
Change-Id: I8a84d08c3090e46b00d1d626fb984f66ea33f256
The bootloader element installs the grub bootloader for whole-disk
images, but it also correctly sets values in /etc/default/grub and BLS
entries.
This value setting is useful even if the bootloader isn't installed.
For example, the overcloud-full partition image benefits from a
correct /etc/default/grub and BLS entries which ironic-python-agent
will use when it installs grub on the disk during baremetal deploy.
This change moves the actual grub install to the end of the script,
and if there is no $DIB_BLOCK_DEVICE set then install is skipped.
This allows overcloud-full to use the bootloader element instead of
the grub2 element, so the correct grub defaults are set on centos9,
including the correct root device on centos9.
Change-Id: I8cb34914bbbfa05521bbb71cc6637368b980358f
The check removed here came in with
I4481b43e4a8fe4144be9c7eb9d9c618bbb2df21e a long time ago. At that
time we were not building EFI images, and were building i386 images;
both of which are now untrue.
We can simplify this now by merging it into the gpt/mbr path. If we
are in there we know that we should set --target=i386-pc for BIOS
boot. For sanity check that we are x86 in this path -- PPC is handled
separately (although it's probably bit-rotted) and ARM64 is EFI.
Change-Id: Ie9839c9adc642b0dd688bced3faa46e9314e9799
Co-Authored-By: Clark Boylan <clark.boylan@gmail.com>
This fixes a regression introduced by
Ia99687815667c3cf5e82cf21d841d3b1008b8fa9
It turns out that [[ -d /usr/lib/grub/*-efi ]] is not a good check,
because [[ doesn't split that and try to glob match ( [ would ). This
has resulted in us triggering this path on ARM64.
This is an x86-64 only check, because on other platforms we either
don't support EFI or are EFI only. Restrict this check to get arm64
working again.
Change-Id: I6a75f8504826bcb0ac122d53dfb9faff975077f4
GRUB_OPTS has never been documented as externally available, and is
not used. Assume it's value to simplify the code.
Move the grub version check separately, as we only support grub2
Remove references to buliding i386 images. I don't image it works in
any way.
Remove ci.md, which is no longer relevant.
Refactor the test for "building BIOS image on EFI system" consiberably
after these changes.
Change-Id: Ia99687815667c3cf5e82cf21d841d3b1008b8fa9
This change removes the uninstall grub2-efi which was required for
prerelease rhel-9 images but now breaks current centos-9-stream
images. A different approach may be required for rhel-9 if the base
image remains different to centos-9-stream (such as populating the
empty /boot/efi partition from the base image)
This change also fixes the detection of whether this is an efi build
to check the block device instead of checking for whether a grub efi
package is installed. This fixes building a centos-9-stream whole-disk
image when package grub2-efi-x64 is installed but a legacy fallback
grub also needs to be installed.
Change-Id: I24baf553e1acd15a66737fc0b2a79d5335e28aa5
Partial-Bug: #1957789
This reverts I2701260d54cf6bc79f1ac765b512d99d799e8c43,
Idf2a471453c5490d927979fb97aa916418172153 and part of
Iecf7f7e4c992bb23437b6461cdd04cdca96aafa6 which added special flags to
update kernels via grubby.
These changes actually ended up reverting the behaviour on Fedora 35,
which is what led me to investigate what was going on more fully.
All distros still support setting GRUB_DEVICE in /etc/default/grub;
even the BLS based ones (i.e. everything !centos7).
The implementation *is* confusing -- in earlier distros each BLS entry
would refer to the variable $kernelopts; which grub2-mkconfig would
write into /boot/grub2/grubenv. After commit [1] this was reverted,
and the kernel options are directly written into the BLS entry.
But the real problem is this bit from [2]
get_sorted_bls()
{
if ! [ -d "${blsdir}" ] || ! [ -e /etc/machine-id ]; then
return
fi
...
files=($(for bls in ${blsdir}/${machine_id}-*.conf; do
...
}
i.e., to avoid overwriting BLS entries for other OS-boots (?),
grub2-mkconfig will only update those BLS entries that match the
current machine-id.
The problem for DIB is that we are clearing the machine-id early in
finalise.d/01-clear-machine-id, but then running the bootloader update
later in finalise.d/50-bootloader.
The result is that the bootloader entry generated when we installed
the kernel (which guessed at the root= device, etc.) is *not* updated.
Even more annoyingly, the gate doesn't pick this up -- because the
gate tests run on a DIB image that was booted with
"root=LABEL=cloudimg-rootfs" the kernel initially installed with
"install-kernel" (that we never updated) is actually correct. But
this fails when built on a production host.
Thus we don't need any of the explicit grubby updates; these are
reverted here. This moves the machine-id clearing to after the
bootloader setup, which allows grub2-mkconfig to setup the BLS entries
correctly.
[1] 4a742183a3
[2] https://src.fedoraproject.org/rpms/grub2/blob/rawhide/f/0062-Add-BLS-support-to-grub-mkconfig.patch
Depends-On: https://review.opendev.org/c/zuul/nodepool/+/818705
Change-Id: Ia0e49980eb50eae29a5377d24ef0b31e4d78d346
CentOS Stream 9 is close to be released, and official mirrors are
already poplated. This patch is adding support to centos-minimal in CS9.
Also enable centos-minimal/[8,9]-stream-build-succeeds tests.
This patch is being tested together with [1] to apply following list of elements:
vm centos-minimal simple-init growroot nodepool-base openstack-repos infra-package-needs
[1] https://review.opendev.org/c/openstack/project-config/+/811442
Change-Id: Iecf7f7e4c992bb23437b6461cdd04cdca96aafa6
The if/elif block added in [0] doesn't work for gentoo, let's hope
that we can get along with an easy fix.
[0] https://review.opendev.org/c/openstack/diskimage-builder/+/804000
Signed-off-by: Dr. Jens Harbott <harbott@osism.tech>
Change-Id: I543e04d2d7efea3e718bae31aa1cc4767bd359f8
Fedora 30 and RHEL-8.2 onwards support the Bootloader Spec and use grubby
to manage kernel menu entries and kernel arguments.
https://fedoraproject.org/wiki/Changes/BootLoaderSpecByDefault
This change detects if this is a BLS enabled environment, and uses
grubby to set kernel arguments on all kernel entries if it is.
Change-Id: I2701260d54cf6bc79f1ac765b512d99d799e8c43
If the grubenv is regenerated, its changes won't be available to UEFI
boot systems unless the changed grubenv is copied to the EFI
directory.
This change copies the grubenv to the EFI directory when the grub.cfg
is copied.
Change-Id: I512502117a6bf1e6122fdfd8965ca488b4a5bae4
This is a first pass through the bootloader, that removes the extlinux
and syslinux install/cleanup path.
Change-Id: Ifb107796cdb6748430a124bf13ced93db9689bff
As noted inline, the switch to "boot loader spec" grub entries breaks
our setting of the root device. This happened some time ago, and it's
not 100% clear to me why our existing Fedora builds haven't broken on
this. However, the new containerfile based builds do seem to be
hitting this.
Disable it for now.
Change-Id: Ia3472947799bb35ffccfa92937cdd0d68b12a25c
The rhel-8.4 qcow2 base image already has the grub2-efi-x64 package
installed on its single partition which has files installed to
/boot/efi..., however a partitioned image will have an empty /boot/efi
partition when running 50-bootloader. This means dnf will not install
grub2-efi-x64 when requested and /boot/efi will remain empty.
This commit makes the following changes:
- Refactors redhat bootloader pkg-map for the following:
- Make x86_64/amd64, arm64/aarch64 adjancent so they don't diverge
- Map grub-efi to packages installed to /usr
- Map grub-efi-{arch} to packages installed to /boot/efi
- Removes packages grub-efi-{arch} before installing grub-efi and
grub-efi-{arch}
Change-Id: Ia197feea34f43bd870fed30829b740596e6b2f48
We've noticed that centos8 arm64 images have a root devices of
/dev/mapper/loop7p3 which make sense within a dib image build context
but not at boot time. Dib intends to use labels to set the root device
but when efi is used we end up running grub2-mkconfig against the efi
grub config path before we configure grub to use labels.
Fix this by running grub2-mkconfig after its configuration is set.
This should avoid confusion and complicated paths through the scripts
that configure this for us. We then copy the resulting config to the efi
specific grub.cfg location for platforms that have it.
There is also a small refactoring that is done to try and make the ~3
boot variants more clear:
1) Booting with legacy bios
2) Booting with uefi without a signed shim that directly calls grub
3) Booting with uefi and a signed shim that calls grub
Options 1 and 2 share the /boot/grub*/grub.cfg file. Option 3 needs its
grub.cfg to live alongside distro specific efi target.
Change-Id: Ie9790da9d1bbea58197b37b15a48e77f8a93c1ac
As of grub2 >= 2.02-95 on redhat family distros, calling grub2-install
on an EFI partition will fail with: "this utility cannot be used for
EFI platforms because it does not support UEFI Secure Boot."
This version of grub is now in centos8-stream and non-eus repos of
RHEL-8. It is not currently possible to build whole-disk UEFI images
on these distros, and when this package is promoted this will also
affect centos8 and RHEL-8 eus. The grub maintainers made this change
because the grub2-install generated /boot/efi/EFI/BOOT/BOOTX64.EFI
will never be capable of booting with Secure Boot.
This change defines a $EFI_BOOT_DIR for every distro element. When
directory /boot/efi/$EFI_BOOT_DIR exists a grub.cfg file in will be
generated there. This change also installs the shim package on redhat
family distros, which installs a copy of the shim bootloader to
/boot/efi/EFI/BOOT/BOOTX64.EFI. Using centos as an example, this
allows UEFI to boot the shim /boot/efi/EFI/BOOT/BOOTX64.EFI which
then chains to /boot/efi/EFI/centos/grubx64.efi.
If /boot/efi/$EFI_BOOT_DIR doesn't exist (such as for Ubuntu,
/boot/efi/EFI/ubuntu) the current behaviour of running grub-install to
generate /boot/efi/EFI/BOOT/BOOTX64.EFI will continue. For distros
such as Ubutnu where packaging does not populate /boot/efi/EFI/ubuntu
with .efi files, secure boot can be added in the future by copying
.efi files to /boot/efi/EFI/ubuntu and copying the shim file to
/boot/efi/EFI/BOOT/BOOTX64.EFI.
Change-Id: I90925218ff2aa4c4daffcf86e686b6d98d6b0f21
The rhel8.2 .qcow2 images have moved from a single xfs partition into
an EFI style layout with separate /boot/efi partition.
This means the root partition has /boot/grub2/grubenv as a symlink to
../efi/EFI/redhat/grubenv, which is dangling when we are running the
bootloader element as /boot/efi is blank. grub-install tries to call
rename() in the process of re-writing this file, which fails and bails
out dib.
Remove this symlink if it exists
Change-Id: I5591144a3617dbae148b0c5d2a6a404942ffcd4e
Parital-Bug: #1893029
Redhat-Bug: https://bugzilla.redhat.com/show_bug.cgi?id=1871669
For Bios and EFI compatibility, grub must be installed twice.
This patch adds the bios version when EFI is selected. The GPT EFI block partitioning
already adds the bios partition, but the bootloader only called grub once.
Change-Id: Iee6c8b3b97b3cfff4562bcb30a50800f5ade894a
Closes-Bug: #1889089
Currently, the serial console is hardcoded to ttyS0 in the bootloader
element. This is a challenge for users that want to build images for
some baremetal servers. Supermicro servers, for example, use ttyS1 for
the serial over lan interface.
This patch adds a new environment variable DIB_BOOTLOADER_SERIAL_CONSOLE
that can be set to override the default.
Change-Id: Ie8173be8690ac0b7164ce9e5b66d3c1c18f844d6
The grub.cfg has two variables [1]
GRUB_CMDLINE_LINUX : used on all boots
GRUB_CMDLINE_LINUX_DEFAULT : additionally used on all "normal" boots
The problem with I2298675dda1f699c572b3423e7274bc8bd7c1c9d is that it
appened the values in DIB_BOOTLOADER_DEFAULT_CMDLINE to both of these,
resulting in duplicated arguments. I don't think we considered that
GRUB_CMDLINE_LINUX_DEFAULT actually already appends to the
GRUB_CMDLINE_LINUX values.
Make DIB_BOOTLOADER_DEFAULT_CMDLINE only append itself to
GRUB_CMDLINE_LINUX_DEFAULT. That seems to line up sensibly with the
name of the variable.
Documentation is enhanced around this, and a releasenote added.
[1] https://help.ubuntu.com/community/Grub2/Setup
Change-Id: I76b5442a9090c19a6540ed2d4ab324546f241ebf
Closes: #1791736
When building the image on a non-efi environment, it generates
linux16/initrd16 entries. But to boot from UEFI they need to have
linuxefi/initrdefi entries.
Use sed to replace those entries, in case we have an EFI image.
Change-Id: I47c96450e10f34b91bcc32888532bd7ab87cf316
Without this change DIB appends a second command line entry to the GRUB
config. This causes the original command line entry to be ignored
when Linux is booted.
The expected behaviour is that DIB appends to the existing entry as
it does for Ubuntu and SUSE.
Following discussion on the review, this also removes the distro specific
switch statement, as update-grub just calls grub-mkconfig, meaning that
there was nothing distro specific in the first place.
Change-Id: I2298675dda1f699c572b3423e7274bc8bd7c1c9d
Closes-Bug: #1771366
Install hwe kernel for ubuntu-minimal. As noted this is currently
Xenial specific; we need this for initial bring-up so let's tackle
future releases as things progress.
Ensure we use ttyAMA0 for arm64 console too.
Change-Id: Ic607cf8369666dc24929aff6f2ef8a72e7980599
In the prior change we added block-device-[mbr|gpt|efi] elements to
create appropriate disk-layouts.
This adds an environment flag to each so the bootloader can install
the right thing. The EFI install path is updated to work with this
(this part a copy of I572937945adbb5adaa5cb09200752e323c2c9531)
We do some basic sanity checking in the block-device elements;
e.g. mbr is not suitable for aarch64, and efi is not suitable for
power.
This updates the bootloader to install EFI where appropriate
Co-Authored-By: Marcin Juszkiewicz <marcin.juszkiewicz@linaro.org>
Change-Id: Ib80acbfd9a12efd976c3fa15a5d1081eb0799305
Avoid incorrect use of [ with =~ matching
I guess this doesn't trip "-e" because it's in an if-conditional. I'm
looking at making bashate detect this; maybe we can run bashate over
things we know are scripts
Change-Id: Ia3fe2b978fae5bdaadbb1789058180d3ad950d00
On Power systems console should be added the kernel command line
in the following order: 'console=tty0 console=hvc0'.
The first one is the graphical console. The last one is the serial
console. The kernel enables all the consoles pointed through the
kernel command line. However, only the last one will receive
input/output during kernel boot. All the other consoles will be
enabled after the boot.
Change-Id: I0069f608e0ab104d3778954e033fb82ed5ea7693
In order to support {CentOS,RHEL}7 for building cloud images we need to
handle the differences in grub packaging from Ubuntu. We also need to
populate the defualt location for cloud images for CentOS builds.
Change-Id: Ie0d82ff21a42b08c4cb94b7a5635f80bfabf684e
Using the newly exposed variables from the prior change, install the
ppc bootloader to the boot partition, not the underlying loopback
device.
Change-Id: I0918e8df8797d6dbabf7af618989ab7f79ee9580
Currently we only export "image-block-device" which is the loopback
device (/dev/loopX) for the underlying image. This is the device we
install grub to (from inside the chroot ...)
This is ok for x86, but is insufficient for some platforms like PPC
which have a separate boot partition. They do not want to install to
the loop device, but do things like dd special ELF files into special
boot partitions.
The first problem seems to be that in level1/partitioning.py we have a
whole bunch of different paths that either call partprobe on the loop
device, or kpartx. We have _all_part_devices_exist() that gates the
kpartx for unknown reasons. We have detach_loopback() that does not
seem to remove losetup created devices. I don't think this does
cleanup if it uses kpartx correctly. It is extremley unclear what's
going to be mapped where.
This moves to us *only* using kpartx to map the partitions of the loop
device. We will *not* call partprobe and create the /dev/loopXpN
devices and will only have the devicemapper nodes kpartx creates.
This seems to be best. Cleanup happens inside partitioning.py.
practice. Deeper thinking about this, and more cleanup of the
variables will be welcome.
This adds "image-block-devices" (note the extra "s") which exports all
the block devices with name and path. This is in a string format that
can be eval'd to an array (you can't export arrays).
This is then used in a follow-on
(I0918e8df8797d6dbabf7af618989ab7f79ee9580) to pick the right
partition on PPC.
Change-Id: If8e33106b4104da2d56d7941ce96ffcb014907bc
Ubuntu 12.04 LTS reached its regular End of Life on April 28, 2017.
Depends-On: I5e145095a10db112bb27516bfe652d2cdc052a61
Change-Id: I64af4c5183d77a75dcd062895d19b0a1330c8da8
Instead, either use the bash built-in of type to ensure it exists. Since
which is an external dep, things can fail oddly in a constrained
environment.
Also add a dib-lint test for this.
Change-Id: I645029f5b5bfe1198c89ce10fd3246be8636e8af
Signed-off-by: Jesse Keating <omgjlk@us.ibm.com>
Depending on the types of deployment (security, nfv...) some extra
kernel flags are needed on the images. This change exposes the
DIB_BOOTLOADER_DEFAULT_CMDLINE parameter, defaulting to the
existing 'nofb nomodeset vga=normal', that will allow to modify
these boot params.
Change-Id: I67d191fa5ca44a57f776cb9739a02dd71212969c
Closes-Bug: #1668890
During the creation of a disk image (e.g. for a VM), there is the need
to create, setup, configure and afterwards detach some kind of storage
where the newly installed OS can be copied to or directly installed
in.
This patch implements partitioning handling.
Change-Id: I0ca6a4ae3a2684d473b44e5f332ee4225ee30f8c
Signed-off-by: Andreas Florath <andreas@florath.net>
Currently we have all our elements and library files in a top-level
directory and install them into
<root>/share/diskimage-builder/[elements|lib] (where root is either /
or the root of a virtualenv).
The problem with this is that editable/development installs (pip -e)
do *not* install data_files. Thus we have no canonical location to
look for elements -- leading to the various odd things we do such as a
whole bunch of guessing at the top of disk-image-create and having a
special test-loader in tests/test_elements.py so we can run python
unit tests on those elements that have it.
data_files is really the wrong thing to use for what are essentially
assets of the program. data_files install works well for things like
config-files, init.d files or dropping documentation files.
By moving the elements under the diskimage_builder package, we always
know where they are relative to where we import from. In fact,
pkg_resources has an api for this which we wrap in the new
diskimage_builder/paths.py helper [1].
We use this helper to find the correct path in the couple of places we
need to find the base-elements dir, and for the paths to import the
library shell functions.
Elements such as svc-map and pkg-map include python unit-tests, which
we do not need tests/test_elements.py to special-case load any more.
They just get found automatically by the normal subunit loader.
I have a follow-on change (I69ca3d26fede0506a6353c077c69f735c8d84d28)
to move disk-image-create to a regular python entry-point.
Unfortunately, this has to move to work with setuptools. You'd think
a symlink under diskimage_builder/[elements|lib] would work, but it
doesn't.
[1] this API handles stuff like getting files out of .zip archive
modules, which we don't do. Essentially for us it's returning
__file__.
Change-Id: I5e3e3c97f385b1a4ff2031a161a55b231895df5b
