Commit Graph

33 Commits

Author SHA1 Message Date
Ian Wienand
2d47d4157c Fix BLS based bootloader installation
This reverts I2701260d54cf6bc79f1ac765b512d99d799e8c43,
Idf2a471453c5490d927979fb97aa916418172153 and part of
Iecf7f7e4c992bb23437b6461cdd04cdca96aafa6 which added special flags to
update kernels via grubby.

These changes actually ended up reverting the behaviour on Fedora 35,
which is what led me to investigate what was going on more fully.

All distros still support setting GRUB_DEVICE in /etc/default/grub;
even the BLS based ones (i.e. everything !centos7).

The implementation *is* confusing -- in earlier distros each BLS entry
would refer to the variable $kernelopts; which grub2-mkconfig would
write into /boot/grub2/grubenv.  After commit [1] this was reverted,
and the kernel options are directly written into the BLS entry.

But the real problem is this bit from [2]

 get_sorted_bls()
 {
     if ! [ -d "${blsdir}" ] || ! [ -e /etc/machine-id ]; then
        return
     fi
     ...
     files=($(for bls in ${blsdir}/${machine_id}-*.conf; do
     ...
 }

i.e., to avoid overwriting BLS entries for other OS-boots (?),
grub2-mkconfig will only update those BLS entries that match the
current machine-id.

The problem for DIB is that we are clearing the machine-id early in
finalise.d/01-clear-machine-id, but then running the bootloader update
later in finalise.d/50-bootloader.

The result is that the bootloader entry generated when we installed
the kernel (which guessed at the root= device, etc.) is *not* updated.
Even more annoyingly, the gate doesn't pick this up -- because the
gate tests run on a DIB image that was booted with
"root=LABEL=cloudimg-rootfs" the kernel initially installed with
"install-kernel" (that we never updated) is actually correct.  But
this fails when built on a production host.

Thus we don't need any of the explicit grubby updates; these are
reverted here.  This moves the machine-id clearing to after the
bootloader setup, which allows grub2-mkconfig to setup the BLS entries
correctly.

[1] 4a742183a3
[2] https://src.fedoraproject.org/rpms/grub2/blob/rawhide/f/0062-Add-BLS-support-to-grub-mkconfig.patch

Depends-On: https://review.opendev.org/c/zuul/nodepool/+/818705
Change-Id: Ia0e49980eb50eae29a5377d24ef0b31e4d78d346
2021-11-25 14:26:23 +11:00
Jakub Darmach
5a4e8a2c9e
Added missing grubby arg DIB_BOOTLOADER_DEFAULT_CMDLINE
DIB_BOOTLOADER_DEFAULT_CMDLINE was missing from grubby args.
Closes-Bug: 1951084

Change-Id: Idf2a471453c5490d927979fb97aa916418172153
2021-11-16 11:43:49 +01:00
Zuul
311621a094 Merge "Add support for CentOS Stream 9 in DIB" 2021-10-29 09:55:30 +00:00
Alfredo Moralejo
cdff9045c0 Add support for CentOS Stream 9 in DIB
CentOS Stream 9 is close to be released, and official mirrors are
already poplated. This patch is adding support to centos-minimal in CS9.

Also enable centos-minimal/[8,9]-stream-build-succeeds tests.

This patch is being tested together with [1] to apply following list of elements:

 vm centos-minimal simple-init growroot nodepool-base openstack-repos infra-package-needs

[1] https://review.opendev.org/c/openstack/project-config/+/811442

Change-Id: Iecf7f7e4c992bb23437b6461cdd04cdca96aafa6
2021-10-27 13:38:14 +02:00
Dr. Jens Harbott
4bd8258fbf Fix bootloader installation for gentoo
The if/elif block added in [0] doesn't work for gentoo, let's hope
that we can get along with an easy fix.

[0] https://review.opendev.org/c/openstack/diskimage-builder/+/804000
Signed-off-by: Dr. Jens Harbott <harbott@osism.tech>
Change-Id: I543e04d2d7efea3e718bae31aa1cc4767bd359f8
2021-10-21 10:20:58 +02:00
Steve Baker
97f940ace5 Support grubby and the Bootloader Spec
Fedora 30 and RHEL-8.2 onwards support the Bootloader Spec and use grubby
to manage kernel menu entries and kernel arguments.
https://fedoraproject.org/wiki/Changes/BootLoaderSpecByDefault

This change detects if this is a BLS enabled environment, and uses
grubby to set kernel arguments on all kernel entries if it is.

Change-Id: I2701260d54cf6bc79f1ac765b512d99d799e8c43
2021-09-13 09:32:53 +12:00
Steve Baker
b01aac9715 Move grubenv to EFI dir
If the grubenv is regenerated, its changes won't be available to UEFI
boot systems unless the changed grubenv is copied to the EFI
directory.

This change copies the grubenv to the EFI directory when the grub.cfg
is copied.

Change-Id: I512502117a6bf1e6122fdfd8965ca488b4a5bae4
2021-09-13 09:32:29 +12:00
Ian Wienand
f6748a4cd4 bootloader: remove extlinux/syslinux path
This is a first pass through the bootloader, that removes the extlinux
and syslinux install/cleanup path.

Change-Id: Ifb107796cdb6748430a124bf13ced93db9689bff
2021-05-13 10:33:06 +10:00
Ian Wienand
4d16ccdbef bootloader: disable BLS for Fedora
As noted inline, the switch to "boot loader spec" grub entries breaks
our setting of the root device.  This happened some time ago, and it's
not 100% clear to me why our existing Fedora builds haven't broken on
this.  However, the new containerfile based builds do seem to be
hitting this.

Disable it for now.

Change-Id: Ia3472947799bb35ffccfa92937cdd0d68b12a25c
2021-05-11 16:48:58 +10:00
Steve Baker
5caeba0c68 Ensure redhat efi packages are reinstalled during finalise
The rhel-8.4 qcow2 base image already has the grub2-efi-x64 package
installed on its single partition which has files installed to
/boot/efi..., however a partitioned image will have an empty /boot/efi
partition when running 50-bootloader. This means dnf will not install
grub2-efi-x64 when requested and /boot/efi will remain empty.

This commit makes the following changes:
- Refactors redhat bootloader pkg-map for the following:
  - Make x86_64/amd64, arm64/aarch64 adjancent so they don't diverge
  - Map grub-efi to packages installed to /usr
  - Map grub-efi-{arch} to packages installed to /boot/efi
- Removes packages grub-efi-{arch} before installing grub-efi and
  grub-efi-{arch}

Change-Id: Ia197feea34f43bd870fed30829b740596e6b2f48
2021-04-21 10:56:37 +12:00
Clark Boylan
3294aecca2 Properly set grub2 root device when using efi
We've noticed that centos8 arm64 images have a root devices of
/dev/mapper/loop7p3 which make sense within a dib image build context
but not at boot time. Dib intends to use labels to set the root device
but when efi is used we end up running grub2-mkconfig against the efi
grub config path before we configure grub to use labels.

Fix this by running grub2-mkconfig after its configuration is set.
This should avoid confusion and complicated paths through the scripts
that configure this for us. We then copy the resulting config to the efi
specific grub.cfg location for platforms that have it.

There is also a small refactoring that is done to try and make the ~3
boot variants more clear:

 1) Booting with legacy bios
 2) Booting with uefi without a signed shim that directly calls grub
 3) Booting with uefi and a signed shim that calls grub

Options 1 and 2 share the /boot/grub*/grub.cfg file. Option 3 needs its
grub.cfg to live alongside distro specific efi target.

Change-Id: Ie9790da9d1bbea58197b37b15a48e77f8a93c1ac
2021-04-07 15:46:10 -07:00
Steve Baker
27a326dafb Support secure-boot bootloader where possible
As of grub2 >= 2.02-95 on redhat family distros, calling grub2-install
on an EFI partition will fail with: "this utility cannot be used for
EFI platforms because it does not support UEFI Secure Boot."

This version of grub is now in centos8-stream and non-eus repos of
RHEL-8. It is not currently possible to build whole-disk UEFI images
on these distros, and when this package is promoted this will also
affect centos8 and RHEL-8 eus. The grub maintainers made this change
because the grub2-install generated /boot/efi/EFI/BOOT/BOOTX64.EFI
will never be capable of booting with Secure Boot.

This change defines a $EFI_BOOT_DIR for every distro element. When
directory /boot/efi/$EFI_BOOT_DIR exists a grub.cfg file in will be
generated there. This change also installs the shim package on redhat
family distros, which installs a copy of the shim bootloader to
/boot/efi/EFI/BOOT/BOOTX64.EFI. Using centos as an example, this
allows UEFI to boot the shim /boot/efi/EFI/BOOT/BOOTX64.EFI which
then chains to /boot/efi/EFI/centos/grubx64.efi.

If /boot/efi/$EFI_BOOT_DIR doesn't exist (such as for Ubuntu,
/boot/efi/EFI/ubuntu) the current behaviour of running grub-install to
generate /boot/efi/EFI/BOOT/BOOTX64.EFI will continue. For distros
such as Ubutnu where packaging does not populate /boot/efi/EFI/ubuntu
with .efi files, secure boot can be added in the future by copying
.efi files to /boot/efi/EFI/ubuntu and copying the shim file to
/boot/efi/EFI/BOOT/BOOTX64.EFI.

Change-Id: I90925218ff2aa4c4daffcf86e686b6d98d6b0f21
2021-03-11 10:27:59 +13:00
Ian Wienand
8f12d9530e bootloader: remove dangling grubenv links
The rhel8.2 .qcow2 images have moved from a single xfs partition into
an EFI style layout with separate /boot/efi partition.

This means the root partition has /boot/grub2/grubenv as a symlink to
../efi/EFI/redhat/grubenv, which is dangling when we are running the
bootloader element as /boot/efi is blank.  grub-install tries to call
rename() in the process of re-writing this file, which fails and bails
out dib.

Remove this symlink if it exists

Change-Id: I5591144a3617dbae148b0c5d2a6a404942ffcd4e
Parital-Bug: #1893029
Redhat-Bug: https://bugzilla.redhat.com/show_bug.cgi?id=1871669
2020-09-08 18:04:42 +10:00
Pierre Crégut
6e71bd4239 Makes EFI images bootable by bios
For Bios and EFI compatibility, grub must be installed twice.
This patch adds the bios version when EFI is selected. The GPT EFI  block partitioning
already adds the bios partition, but the bootloader only called grub once.

Change-Id: Iee6c8b3b97b3cfff4562bcb30a50800f5ade894a
Closes-Bug: #1889089
2020-08-18 14:41:21 +10:00
Andrei Nistor
0e5e358063 bootloader: make serial console configurable
Currently, the serial console is hardcoded to ttyS0 in the bootloader
element.  This is a challenge for users that want to build images for
some baremetal servers. Supermicro servers, for example, use ttyS1 for
the serial over lan interface.

This patch adds a new environment variable DIB_BOOTLOADER_SERIAL_CONSOLE
that can be set to override the default.

Change-Id: Ie8173be8690ac0b7164ce9e5b66d3c1c18f844d6
2019-10-03 21:49:53 +00:00
Ian Wienand
f6a2452d4c Only append DIB_BOOTLOADER_DEFAULT_CMDLINE to default grub entry
The grub.cfg has two variables [1]

 GRUB_CMDLINE_LINUX : used on all boots
 GRUB_CMDLINE_LINUX_DEFAULT : additionally used on all "normal" boots

The problem with I2298675dda1f699c572b3423e7274bc8bd7c1c9d is that it
appened the values in DIB_BOOTLOADER_DEFAULT_CMDLINE to both of these,
resulting in duplicated arguments.  I don't think we considered that
GRUB_CMDLINE_LINUX_DEFAULT actually already appends to the
GRUB_CMDLINE_LINUX values.

Make DIB_BOOTLOADER_DEFAULT_CMDLINE only append itself to
GRUB_CMDLINE_LINUX_DEFAULT.  That seems to line up sensibly with the
name of the variable.

Documentation is enhanced around this, and a releasenote added.

[1] https://help.ubuntu.com/community/Grub2/Setup

Change-Id: I76b5442a9090c19a6540ed2d4ab324546f241ebf
Closes: #1791736
2018-09-13 09:51:50 +10:00
Yolanda Robla
61e6566c48 Fix bootloader for efi on rhel systems
When building the image on a non-efi environment, it generates
linux16/initrd16 entries. But to boot from UEFI they need to have
linuxefi/initrdefi entries.
Use sed to replace those entries, in case we have an EFI image.

Change-Id: I47c96450e10f34b91bcc32888532bd7ab87cf316
2018-06-19 08:37:30 +02:00
Doug Szumski
c1b1534c87 Remove duplicate GRUB command line entry
Without this change DIB appends a second command line entry to the GRUB
config. This causes the original command line entry to be ignored
when Linux is booted.

The expected behaviour is that DIB appends to the existing entry as
it does for Ubuntu and SUSE.

Following discussion on the review, this also removes the distro specific
switch statement, as update-grub just calls grub-mkconfig, meaning that
there was nothing distro specific in the first place.

Change-Id: I2298675dda1f699c572b3423e7274bc8bd7c1c9d
Closes-Bug: #1771366
2018-05-16 09:25:59 +01:00
Ian Wienand
e9ed983324 arm64: use HWE kernel and fix console
Install hwe kernel for ubuntu-minimal.  As noted this is currently
Xenial specific; we need this for initial bring-up so let's tackle
future releases as things progress.

Ensure we use ttyAMA0 for arm64 console too.

Change-Id: Ic607cf8369666dc24929aff6f2ef8a72e7980599
2018-02-23 10:04:48 +11:00
Ian Wienand
7b4c8abce3 Choose appropriate bootloader for block-device
In the prior change we added block-device-[mbr|gpt|efi] elements to
create appropriate disk-layouts.

This adds an environment flag to each so the bootloader can install
the right thing.  The EFI install path is updated to work with this
(this part a copy of I572937945adbb5adaa5cb09200752e323c2c9531)

We do some basic sanity checking in the block-device elements;
e.g. mbr is not suitable for aarch64, and efi is not suitable for
power.

This updates the bootloader to install EFI where appropriate

Co-Authored-By: Marcin Juszkiewicz <marcin.juszkiewicz@linaro.org>
Change-Id: Ib80acbfd9a12efd976c3fa15a5d1081eb0799305
2018-02-23 10:04:44 +11:00
Ian Wienand
c448864901 Use [[ for =~ matches
Avoid incorrect use of [ with =~ matching

I guess this doesn't trip "-e" because it's in an if-conditional.  I'm
looking at making bashate detect this; maybe we can run bashate over
things we know are scripts

Change-Id: Ia3fe2b978fae5bdaadbb1789058180d3ad950d00
2017-08-28 17:01:03 +10:00
Rafael Folco
bfdf7dc0f6 Enable console during kernel boot on Power
On Power systems console should be added the kernel command line
in the following order: 'console=tty0 console=hvc0'.
The first one is the graphical console. The last one is the serial
console. The kernel enables all the consoles pointed through the
kernel command line. However, only the last one will receive
input/output during kernel boot. All the other consoles will be
enabled after the boot.

Change-Id: I0069f608e0ab104d3778954e033fb82ed5ea7693
2017-07-07 17:55:56 +00:00
Chhavi Agarwal
6d69d7909d Support for Cloud Images on ppc64le for rhel7 and centos7
In order to support {CentOS,RHEL}7 for building cloud images we need to
handle the differences in grub packaging from Ubuntu.  We also need to
populate the defualt location for cloud images for CentOS builds.

Change-Id: Ie0d82ff21a42b08c4cb94b7a5635f80bfabf684e
2017-06-29 15:44:26 +10:00
Jenkins
1324f5b7db Merge "Remove use of 'which'." 2017-06-11 09:30:34 +00:00
Ian Wienand
5ac8a98e9a PPC bootloader; install to boot partition
Using the newly exposed variables from the prior change, install the
ppc bootloader to the boot partition, not the underlying loopback
device.

Change-Id: I0918e8df8797d6dbabf7af618989ab7f79ee9580
2017-06-08 17:14:22 +10:00
Ian Wienand
6c394f5746 Pass all blockdevices to bootloader
Currently we only export "image-block-device" which is the loopback
device (/dev/loopX) for the underlying image.  This is the device we
install grub to (from inside the chroot ...)

This is ok for x86, but is insufficient for some platforms like PPC
which have a separate boot partition.  They do not want to install to
the loop device, but do things like dd special ELF files into special
boot partitions.

The first problem seems to be that in level1/partitioning.py we have a
whole bunch of different paths that either call partprobe on the loop
device, or kpartx.  We have _all_part_devices_exist() that gates the
kpartx for unknown reasons.  We have detach_loopback() that does not
seem to remove losetup created devices.  I don't think this does
cleanup if it uses kpartx correctly.  It is extremley unclear what's
going to be mapped where.

This moves to us *only* using kpartx to map the partitions of the loop
device.  We will *not* call partprobe and create the /dev/loopXpN
devices and will only have the devicemapper nodes kpartx creates.
This seems to be best.  Cleanup happens inside partitioning.py.
practice.  Deeper thinking about this, and more cleanup of the
variables will be welcome.

This adds "image-block-devices" (note the extra "s") which exports all
the block devices with name and path.  This is in a string format that
can be eval'd to an array (you can't export arrays).

This is then used in a follow-on
(I0918e8df8797d6dbabf7af618989ab7f79ee9580) to pick the right
partition on PPC.

Change-Id: If8e33106b4104da2d56d7941ce96ffcb014907bc
2017-06-08 17:14:22 +10:00
Dirk Mueller
f58bf252de Drop support for Ubuntu precise
Ubuntu 12.04 LTS reached its regular End of Life on April 28, 2017.

Depends-On: I5e145095a10db112bb27516bfe652d2cdc052a61
Change-Id: I64af4c5183d77a75dcd062895d19b0a1330c8da8
2017-05-31 14:36:30 +02:00
Jesse Keating
84d10dce57 Remove use of 'which'.
Instead, either use the bash built-in of type to ensure it exists. Since
which is an external dep, things can fail oddly in a constrained
environment.

Also add a dib-lint test for this.

Change-Id: I645029f5b5bfe1198c89ce10fd3246be8636e8af
Signed-off-by: Jesse Keating <omgjlk@us.ibm.com>
2017-05-19 12:43:36 -07:00
Yolanda Robla
11275c212b Send custom parameters in bootloader to GRUB_CMDLINE
Depending on the types of deployment (security, nfv...) some extra
kernel flags are needed on the images. This change exposes the
DIB_BOOTLOADER_DEFAULT_CMDLINE parameter, defaulting to the
existing 'nofb nomodeset vga=normal', that will allow to modify
these boot params.

Change-Id: I67d191fa5ca44a57f776cb9739a02dd71212969c
Closes-Bug: #1668890
2017-03-08 15:43:46 +01:00
Ian Wienand
3f8800832a Release 1.27.0
-----BEGIN PGP SIGNATURE-----
 
 iQEcBAABAgAGBQJYk8+9AAoJEBty/58O8cX8LdIH+wU/VrEVs0XYohiL6DUgabzs
 112U3UUihH5xMc/ca9Tarx+XwEvfMZkwYN2Qr0JoRJjmSt2AL6AezUhGSV+98vaY
 iQEccaFDFYlyDHm4V2r7N1xwS0B3mx87FPqVQQSUKlc3JsQxCy4o9RtD9aM8Gvqy
 +gAxMxL3p3O131K0Rvb0U5lC1FLgft9SuljCV8i5nU4/HdoryD6hedz2/ss8a9KG
 KKEdBKvPBKn73+nb8peQD/VXpej9C31r87q5VEjUsZkJ7gduY/qYLlGGgoBQqAXN
 WQ/ef1RkQKW5ba2jsjnk7fdOrA0+wYENxorR2WecuZbe2ieXw6fP3lYiD6VeWsM=
 =IUuh
 -----END PGP SIGNATURE-----

Merge tag '1.27.0' into merge-branch

Release 1.27.0

Change-Id: I9f6948636cae6d375d1d8315976504021f5a3bbb
2017-02-03 11:49:45 +11:00
Ian Wienand
adf39c52cf Release 1.21.1
-----BEGIN PGP SIGNATURE-----
 
 iQEcBAABAgAGBQJYW2GoAAoJEBty/58O8cX8uSAH/15dJsglP6Zie7jSSJcR6k+e
 PJembHn9qrqrCjmJ5EwakojySaaLhwEJKvlP54OU9v7pmUXL9gJtK2OzW54LQ41g
 xBHIu0Pg4z7juyHm9+1P2Sr7Mzs1pVSEbsIYpDYUU19eghI1EAeIj3I1woKgajN7
 JlI61j3r67G6EAVtPOnmD1jvXS8CrtjiJ9wtWTH20pWfmksovg/GuXUCZrLkAAhO
 NcK35CdMii1Hkr7XOH424La/Ar+3qfUX18ZkbJY6yHzkq/ityTzzKOFjAaDl2Jg9
 WNc+SLCVYpPhPwgt7miTywamUNj3ZviA5/Hd8fuLXmtHSLQ23WOtBiaQMLtwXHs=
 =8dIl
 -----END PGP SIGNATURE-----

Merge tag '1.26.1' into merge-branch

Release 1.21.1

Change-Id: Ib9eb3dd1d384fc5b6a9846608216e056c57a173a
2017-02-02 20:36:23 +11:00
Andreas Florath
ec7f56c1b2 Refactor: block-device handling (partitioning)
During the creation of a disk image (e.g. for a VM), there is the need
to create, setup, configure and afterwards detach some kind of storage
where the newly installed OS can be copied to or directly installed
in.

This patch implements partitioning handling.

Change-Id: I0ca6a4ae3a2684d473b44e5f332ee4225ee30f8c
Signed-off-by: Andreas Florath <andreas@florath.net>
2017-01-24 19:59:10 +00:00
Ian Wienand
97c01e48ed Move elements & lib relative to diskimage_builder package
Currently we have all our elements and library files in a top-level
directory and install them into
<root>/share/diskimage-builder/[elements|lib] (where root is either /
or the root of a virtualenv).

The problem with this is that editable/development installs (pip -e)
do *not* install data_files.  Thus we have no canonical location to
look for elements -- leading to the various odd things we do such as a
whole bunch of guessing at the top of disk-image-create and having a
special test-loader in tests/test_elements.py so we can run python
unit tests on those elements that have it.

data_files is really the wrong thing to use for what are essentially
assets of the program.  data_files install works well for things like
config-files, init.d files or dropping documentation files.

By moving the elements under the diskimage_builder package, we always
know where they are relative to where we import from.  In fact,
pkg_resources has an api for this which we wrap in the new
diskimage_builder/paths.py helper [1].

We use this helper to find the correct path in the couple of places we
need to find the base-elements dir, and for the paths to import the
library shell functions.

Elements such as svc-map and pkg-map include python unit-tests, which
we do not need tests/test_elements.py to special-case load any more.
They just get found automatically by the normal subunit loader.

I have a follow-on change (I69ca3d26fede0506a6353c077c69f735c8d84d28)
to move disk-image-create to a regular python entry-point.

Unfortunately, this has to move to work with setuptools.  You'd think
a symlink under diskimage_builder/[elements|lib] would work, but it
doesn't.

[1] this API handles stuff like getting files out of .zip archive
modules, which we don't do.  Essentially for us it's returning
__file__.

Change-Id: I5e3e3c97f385b1a4ff2031a161a55b231895df5b
2016-11-01 17:27:41 -07:00