Commit Graph

4298 Commits

Author SHA1 Message Date
Zuul
9c1ee6dcd8 Merge "Correct boot path to cover FIPS usage cases" 2023-03-21 06:39:00 +00:00
Zuul
950ad3324d Merge "Add swap support" 2023-03-21 06:38:57 +00:00
Zuul
923ae116ae Merge "Document diskimage-builder command" 2023-03-21 04:33:42 +00:00
Zuul
1f8f770eb7 Merge "Switch run_functests.sh from disk-image-create to diskimage-builder" 2023-03-21 04:33:39 +00:00
Zuul
c214704614 Merge "A new diskimage-builder command for yaml image builds" 2023-03-21 04:30:19 +00:00
Zuul
a00a68ac7a Merge "Update Fedora to 37" 2023-03-20 00:52:48 +00:00
Zuul
63dd2f1edd Merge "Removal focal pins for testing" 2023-03-20 00:52:46 +00:00
Zuul
2986dd039c Merge "Repeat to umount filesystem when exception occurs" 2023-03-16 21:39:16 +00:00
Zuul
ad87cc76b8 Merge "Fix double-keyed json" 2023-03-16 21:39:14 +00:00
Julia Kreger
4633da7750 Correct boot path to cover FIPS usage cases
When your booting a Linux system using dracut, i.e. with any
redhat style distribution, dracut's internal code looks to validate
the kernel hmac signature in before proceeding to userspace.

It does this by looking at the /boot/ folder file for the kernel
hmac file.

And it normally does this with the root filesystem. Except if the
kernel is not on the root filesystem and is instead on a /boot
filesystem, this breaks horribly. This is compounded because
DIB enables the operator to restructure the OS image/layout
to fit their needs. In order for this to be navigated, as dracut
is written, we need to pass a "boot=" argument to the kernel.

So now we attempt to purge any prior boot entry in the disk image
content, which is good because any filesystem operations invalidate
it, and then we attempt to identify the boot filesystem, and save a
boot kernel command line parameter so the resulting image can
boot properly if FIPS was enabled in the prior image.

Regex developed with https://sed.js.org utilizing stdin:

VAR="quiet boot=UUID=173c759f-1302-48a3-9d51-a17784c21e03 text"
VAR="quiet boot=PARTUUID=173c759f-1302-48a3-9d51-a17784c21e03"
VAR="quiet boot=PARTUUID=173c759f-1302-48a3-9d51-a17784c21e03 reboot=meow"
VAR="quiet boot=UUID=/dev/sda1 text"
VAR="quiet boot=/dev/sda1"
VAR="quiet boot=/dev/sda1 reboot=meow"
VAR="quiet after_boot=1 reboot=meow boot=/dev/sda1"
VAR="quiet after_boot=1 reboot=meow"

Which resulted in stdout:

VAR="quiet text"
VAR="quiet"
VAR="quiet reboot=meow"
VAR="quiet text"
VAR="quiet"
VAR="quiet reboot=meow"
VAR="quiet after_boot=1 reboot=meow"
VAR="quiet after_boot=1 reboot=meow"

Change-Id: I9034c21e84deda2ba2c0ec0d1d6d6595ed10bed4
2023-03-15 11:25:21 -07:00
Ian Wienand
b6cdda4ab0
Update Fedora to 37
This seems like a relatively simple update with nothing breaking
(famous last words).

Change-Id: I5498371ad5af3f0f053af633eb26121ea9f780e6
2023-03-15 16:18:22 +11:00
Ian Wienand
89e571a52f
Removal focal pins for testing
This is a partial revert of I1af9f5599168aadc1e7fcdfae281935e6211a597.

I believe we worked around this issue with podman and cgroups with
Ie663d01d77e17f560a92887cba1e2c86b421b24d in the nodepool-builder
container.  So we can unpin this.

Change-Id: I6a818999006c539e84aae8b59d5055c2f3aa25ca
2023-03-15 14:41:02 +11:00
Steve Baker
cb395e672b Document diskimage-builder command
Change-Id: I607ecb4adee9cf2e2db98142fe5274f4c12f5ab0
2023-03-13 09:01:49 +13:00
Steve Baker
9860bcf447 Switch run_functests.sh from disk-image-create to diskimage-builder
This provides some test coverage on the new diskimage-builder.

It also makes the logfile argument handling a bit simpler.

Change-Id: Iecba581a00ba26131248566cb3088a1566dde00d
2023-03-13 09:01:49 +13:00
Steve Baker
7e38f85724 A new diskimage-builder command for yaml image builds
The `diskimage-builder` command provides a yaml file based interface
to `disk-image-create` and `ramdisk-image-create`. Every argument to
these scripts has a YAML equivalent. The command has the following
features:
- Environment values can be provided from the calling environment as
  well as YAML
- All arguments are validated with jsonschema in the most appropriate
  YAML type
- Schema is self-documenting and printed when running with --help
- Multiple YAML files can be specified and each file can have multiple
  images defined
- Entries with duplicate image names will be merged into a single
  image build, with attributes overwritten, elements appended, and
  environment values updated/overwritten. A missing image name implies
  the same image name as the previous entry.
- --dry-run and --stop-on-failure flags

A simple YAML defintion would resemble:

- imagename: centos-minimal
  checksum: true
  install-type: package
  elements: [centos, vm]
- imagename: ironic-python-agent
  elements:
  - ironic-python-agent-ramdisk
  - extra-hardware

The TripleO project has managed image build options with YAML files
and it has proved useful having git history and a diff friendly
format, specifically for the following situations:
- Managing differences between distros (centos, rhel)
- Managing changes in major distro releases (centos-8, centos-9-stream)
- Managing the python2 to python3 transition, within and across major
  distro releases

Now that the TripleO toolchain is being retired this tool is being
proposed to be used for the image builds of TripleO's successor, as
well as the rest of the community.

Subsequent commits will add documentation and switch some tests to
using `diskimage-builder`.

Change-Id: I95cba3530d1b1c6c52cf547338762e33738f7225
2023-03-13 09:01:49 +13:00
John L. Villalovos
47dc5a9834 chore: support building Fedora on arm64 AKA aarch64
By default [1] the `aarch64` ARCH value is converted to `arm64`. But
Fedora uses `aarch64` to refer to the architecture.

Convert incoming ARCH values of `arm64` into `aarch64` as is already
done for `amd64` -> `x86_64`

[1] 174089a6a5/diskimage_builder/lib/common-defaults (L29-L30)

Change-Id: I6d9698e45b1183007bac49544da196ec78a7ac6a
2023-03-10 09:32:08 -08:00
Maksim Malchuk
601dc0387f Add swap support
Adds swap as a valid "filesystem"

Closes-Bug: #1816136
Change-Id: Ie50834a9834815b1dfacafd283f505f3323d35c8
Co-Authored-By: luke.odom <luke.odom@dreamhost.com>
Signed-off-by: Maksim Malchuk <maksim.malchuk@gmail.com>
2023-03-06 14:54:35 +03:00
Erik Berg
b78afe4c62
Fix double-keyed json
The second release masks the first release, which is probably a
mistake.

Order them from most significant to least;
release > distro > family > default

And fix up the indentation.

Change-Id: I54a6a49d4fe001b1a16ab38637cb55542ce96cdb
2023-03-03 12:04:30 +01:00
Maksim Malchuk
84d6af7de8 Repeat to umount filesystem when exception occurs
Sometimes umount doesn't have much time to finish and failed with
error 'target is busy', but this is not an actual error in some cases
and the operation should be repeated again with some timeout.

This solves the issue and raise actual exception only after several
tries with timeout.

Closes-Bug: #2004492
Change-Id: I069af85b52e20e9fd688f9ae07e66beb2179f3e1
Signed-off-by: Maksim Malchuk <maksim.malchuk@gmail.com>
2023-02-01 20:09:47 +03:00
Zuul
174089a6a5 Merge "Reduce thin pool by one more extent" 2023-01-27 02:34:56 +00:00
Ian Wienand
2c4d230d7a
cache-url: Give up on curl install for Redhat platforms
The RedHat platforms vary if they come pre-installed with curl or
curl-minimal.  For example, Fedora 37 container images have curl, and
centos 9-stream and Rocky images have curl-minimal.  If you try and
install curl when curl-minimal is installed, you get an error, and
vice-versa.

Unfortunately package-installs can't really sort this out; we're just
passing a package list to the system package manager.  We don't have a
way to say "the curl OR curl-minimal package is fine".  As this breaks
builds and is such a common dependency that it's already there, let's
just add a note that curl is required and blank out the package-map.

Change-Id: I9ccebe2dbf3a8682dab60c2070c5f78849e01446
2023-01-20 17:29:13 +11:00
Clark Boylan
8bb9a37a0d Clean up tox.ini for tox v4
skipsdist now basically means don't install the project at all
(regardless of the usedevelop setting) which creates problems for dib's
entrypoints. Remove skipsdist so that entrypoints can be found. Also, we
remove basepython because this confuses tox v4 on whether or not the
python it wants is present.

Change-Id: I16388a8ad50483228d0b71745f11563f891249c0
2023-01-09 08:18:01 -08:00
Steve Baker
7e2a2aa027 Reduce thin pool by one more extent
The previous commit was tested on 2TB without issue, but testing on a
very small volume (80GB) resulted in the thin pool lvextend failing
for being one extent too large.

This change reduces the pool size by one extent.

Change-Id: I7ca002783f8f15946bc84af95eecaa097e70aaf1
Related: rhbz#2149586
2022-12-22 17:24:03 +13:00
Zuul
4a161d0cde Merge "Add variable for check installing python3 in yum element" 2022-12-21 05:16:35 +00:00
Zuul
9d0ba50494 Merge "Grow thin pool metadata by 1GiB" 2022-12-21 04:20:44 +00:00
Steve Baker
00ca126287 Grow thin pool metadata by 1GiB
An LVM thin pool has an associated metadata volume, and it can be
assumed that the size of this volume on the image is minimized for
distribution.

This change grows the metadata volume by 1GiB, which is recommended[1] as
a reasonable default. This fixes a specific issue with the metadata
volume being exausted when growing into a 2TB drive.

Other minor changes include:
- Human readable printed values have switched to GiB, MiB, KiB, B
- Growth percentage volumes are adjusted down to not over-provision
  the thin volume

[1] https://access.redhat.com/solutions/6318131

Change-Id: I1dd6dd932bb5f5d9adac9b78a026569165bd4ea9
Resolves: rhbz#2149586
2022-12-19 13:40:04 +13:00
Frikin Evgenii
a9edef6548 Add variable for check installing python3 in yum element
If client have not internet or have some limitation, such as firewall/proxy/etc. this step will stop build image with error. Client must have possible override of URL for pass this step.

Change-Id: Iafe3283665a437d0a9cf83a93ff66c0613310b69
2022-12-16 03:51:10 +00:00
Zuul
b706802a1c Merge "Fix issue in extract image" 2022-12-16 03:43:14 +00:00
Ian Wienand
41aa936fa2
tox jobs: pin to correct nodesets; use host networking for containerfile
These must have broken when we switched the base nodes to Jammy.
Update to use compatible versions of distros.

We need to squish another gate-breaking change in here to update the
containerfile "podman build" calls to use "--network host".  We added
this with Ia885237406bf4c7b9d49b349f374558ae746401f and the only
external user I can find is kayobe, which is setting this anyway.

I honestly haven't 100% root-caused what changed to require this; the
last time our containerfile jobs ran and worked has unfortunately been
purged so I can't compare versions to try and pinpoint something;
i.e. this may be a podman bug or feature.  At first I thought it
related to the networking plugin package from the Depends-On (which is
still useful for the right packages) but that didn't help get the
bridge networking working.

Depends-On: https://review.opendev.org/c/zuul/nodepool/+/867590
Change-Id: I23f091654cb212e8bdd908664b262de9bfe98cef
2022-12-16 09:52:46 +11:00
jgupta
f2e7cd1307 Fix issue in extract image
The problem lays with the 'extract-image' script as
it is using lsblk commands to extract image's partition
(find out root/efi/boot, lines:100-102) but the output
is empty inside a container.

lsblk gives empty output for FSTYPE, LABEL, GUID..
the fix is to use blkid.

Closes-Bug: 1974350
Change-Id: I3b460c6dd9caa519c55327c5bd4b7e4585a8bd22
2022-11-02 14:21:16 +00:00
Rafal Lewandowski
4ff8721a66 Added example configuration
Change-Id: Icebe0e2f43f4a96d3c1fa464c6f2dc3da3be65c5
2022-10-20 10:00:18 +02:00
Zuul
ea7de37ccb Merge "Added cloud-init growpart element" 2022-10-13 20:41:55 +00:00
Rafal Lewandowski
b73048f98d Added cloud-init growpart element
Added growpart element. It allows for growing specific partitions
during the deployment, which will result in less post deploy actions
needed for the server to be ready for use.

Change-Id: I6519fba3e8f1d078b99d3c03f2ac85f7b6e37d8a
2022-10-11 16:58:26 +02:00
Zuul
3e72fdbf77 Merge "Fix wrong yum.conf name of CentOS 9 Stream" 2022-10-05 14:50:07 +00:00
Ian Wienand
d321b08f4d
Add Rocky 9 ARM64 functional test
This will test if we can get a Rocky 9 ARM64 image

Needed-By: https://review.opendev.org/c/openstack/project-config/+/858554
Change-Id: I3e589e4838cea489e11719b5f7fce82bf0d978e8
2022-09-21 15:08:20 +10:00
Ian Wienand
2a25db9ee6
Start running dib-lint again
A recent change that didn't fail with hard-tabs made me realise we're
not running tox -e pep8 ... which means we're not running dib-lint
which should find this (and other things).

I couldn't pinpoint when this happened; maybe job config was never in
this repo.

Anyway, move the pylint and dib-lint/flake8 testing to the now
standard "linters" and update the linting job to
openstack-tox-linters.

It looks like pylint is very lightly used (came in with
I7e24d8348db3aef79e1395d12692199a1f80161a and we've never expanded any
testing).  Leave this alone for now, but probably it is not important
any more.

This revealed some issues; updated flake8
(Iaa19c36f8cab8482a01f764c588375db8e7d8be3) found some spacing issues
with keywords and an update to elrepo to match our standard bash
flags.

Change-Id: I45bf108c467f7c8190ca252e6c48450c2622aaf8
2022-09-21 07:56:05 +10:00
Zuul
cbc0f5db28 Merge "Allow flake8 version 5" 2022-09-20 21:28:17 +00:00
Clark Boylan
793e16b76c Install Fedora ifcfg NM compat package
Starting with Fedora 36 the NetworkManager package no longer includes
ifcfg support by default. You need an additional package
"NetworkManager-initscripts-ifcfg-rh" to pull in the compatibility
plugin. Glean's support for Fedora relies on this compatibility system
so we install this package via the simple-init element package deps.

Change-Id: I76ac39b8dedcb1c5bc4595aedc0a732c99c8721e
2022-09-20 08:58:37 -07:00
Zuul
9ce3358fde Merge "changed release check logic in lvm element" 2022-09-16 12:30:03 +00:00
Zuul
0c323755bc Merge "added elrepo element" 2022-09-16 09:50:21 +00:00
Rafal Lewandowski
35c0c2c6db added elrepo element
Change-Id: I7bc144afa96f45122857ff634c8c19c1fd759450
2022-09-02 15:09:08 +02:00
Rafal Lewandowski
f9287fe2ae changed release check logic in lvm element
Change-Id: I83b3c2dad3b0d6006ae23307ae7a8a83b12806e7
2022-09-02 12:37:15 +02:00
Zuul
4a1f684e5f Merge "Add subscription-manager repo names for RHEL-9" 2022-09-02 07:35:17 +00:00
Tristan Cacqueray
2c5eb04ba9 Allow flake8 version 5
This change enables using the latest flake8.

Change-Id: Iaa19c36f8cab8482a01f764c588375db8e7d8be3
2022-08-25 18:31:41 +00:00
Steve Baker
833c5b8ceb Support LVM thin provisioning
This change extends the block device lvs attributes to allow creating
a volume which represents a thin pool, and to create volumes which are
allocated from this pool.

Change-Id: Ic58f55c36236cc8c6279fbcb708e27dc2982f2d5
2022-08-24 10:34:42 +12:00
Steve Baker
f61548d863 Add thin provisioning support to growvols
This change enhances the growvols script to support all volumes being
backed by one thin provisioning pool.

If a pool is detected, the following occurs:
- validation to confirm every volume is backed by the pool
- only the pool is extended into the new partition
- volumes are extended by the same amount as the non thin-provisioned
  case

This results in no volumes being over-provisioned, so
out-of-space behaviour will be the same as the non thin-provisioned
case.

This change also switches to using /dev/mapper device mapper paths for
volume block devices, since that is the only path the thin pool is
mapped to.

Change-Id: I96085fc889e72c942cfef7e3acb6f6cd73f606dd
2022-08-24 10:14:26 +12:00
Zuul
15430098a0 Merge "Do dmsetup remove device in rollback" 2022-08-23 16:11:30 +00:00
Ian Wienand
467bbf36dd rocky : create machine-id in 9
It turns out we do need to create the machine-id for the same reason
as on 8.  This was being hidden by the bootloader choosing the root
disk label from the host (see the dependent change).

Change I3b518802d681b888916a5cc6a3dcf7e1b537da1e has modified the
testing to use a different root-disk label, which should help catch
this in the fututure.

Depends-On: https://review.opendev.org/c/zuul/nodepool/+/853574
Change-Id: I64de66cac25fd2e051780fb4812e075c647eb76e
2022-08-19 18:54:43 +10:00
Ian Wienand
3425624686 Allow setting ROOT_LABEL from environment
--root-label was added with I596104d1a63b5dc6549e8460a1ae3da00165ef04

This sets the ROOT_LABEL environment variable.

Over the years how this deploys has become more complex; now this
value gets written into DIB_BLOCK_DEVICE_PARAMS_YAML default values,
which is then loaded into DIB_ROOT_LABEL.

To override this from the environment you need to specify a full
DIB_BLOCK_DEVICE_CONFIG -- we don't have a way to just merge in the
root label setting.

Using the command-line argument is difficult with tools like nodepool
where the command-line is baked into something else.  However we
already have methods for overriding environment variables on dib
calls.

Several of the other variables here accept default values from the
environment, so this is not an outlier.  Making ROOT_LABEL also do
this allows us to test with non-default root devices in the gate (see
the linked change).

Change-Id: Ia1ef48c24841a86f387ff9603c64fd23d8670193
Needed-By: https://review.opendev.org/c/zuul/nodepool/+/853574
2022-08-18 11:21:19 +10:00
Steve Baker
1a4fb0b89b Do dmsetup remove device in rollback
Without this change, the final unmount will timeout after the
rollbacks are called when the partitioning fails due to a user error.

dmsetup remove is called both for partition and LVM volume devices.

Change-Id: I99679ea00338d4018a95d4da9b21685161cd5049
2022-08-18 10:23:41 +12:00