Commit Graph

1288 Commits

Author SHA1 Message Date
Rafal Lewandowski
4ff8721a66 Added example configuration
Change-Id: Icebe0e2f43f4a96d3c1fa464c6f2dc3da3be65c5
2022-10-20 10:00:18 +02:00
Zuul
ea7de37ccb Merge "Added cloud-init growpart element" 2022-10-13 20:41:55 +00:00
Rafal Lewandowski
b73048f98d Added cloud-init growpart element
Added growpart element. It allows for growing specific partitions
during the deployment, which will result in less post deploy actions
needed for the server to be ready for use.

Change-Id: I6519fba3e8f1d078b99d3c03f2ac85f7b6e37d8a
2022-10-11 16:58:26 +02:00
Zuul
3e72fdbf77 Merge "Fix wrong yum.conf name of CentOS 9 Stream" 2022-10-05 14:50:07 +00:00
Ian Wienand
d321b08f4d
Add Rocky 9 ARM64 functional test
This will test if we can get a Rocky 9 ARM64 image

Needed-By: https://review.opendev.org/c/openstack/project-config/+/858554
Change-Id: I3e589e4838cea489e11719b5f7fce82bf0d978e8
2022-09-21 15:08:20 +10:00
Ian Wienand
2a25db9ee6
Start running dib-lint again
A recent change that didn't fail with hard-tabs made me realise we're
not running tox -e pep8 ... which means we're not running dib-lint
which should find this (and other things).

I couldn't pinpoint when this happened; maybe job config was never in
this repo.

Anyway, move the pylint and dib-lint/flake8 testing to the now
standard "linters" and update the linting job to
openstack-tox-linters.

It looks like pylint is very lightly used (came in with
I7e24d8348db3aef79e1395d12692199a1f80161a and we've never expanded any
testing).  Leave this alone for now, but probably it is not important
any more.

This revealed some issues; updated flake8
(Iaa19c36f8cab8482a01f764c588375db8e7d8be3) found some spacing issues
with keywords and an update to elrepo to match our standard bash
flags.

Change-Id: I45bf108c467f7c8190ca252e6c48450c2622aaf8
2022-09-21 07:56:05 +10:00
Clark Boylan
793e16b76c Install Fedora ifcfg NM compat package
Starting with Fedora 36 the NetworkManager package no longer includes
ifcfg support by default. You need an additional package
"NetworkManager-initscripts-ifcfg-rh" to pull in the compatibility
plugin. Glean's support for Fedora relies on this compatibility system
so we install this package via the simple-init element package deps.

Change-Id: I76ac39b8dedcb1c5bc4595aedc0a732c99c8721e
2022-09-20 08:58:37 -07:00
Zuul
9ce3358fde Merge "changed release check logic in lvm element" 2022-09-16 12:30:03 +00:00
Zuul
0c323755bc Merge "added elrepo element" 2022-09-16 09:50:21 +00:00
Rafal Lewandowski
35c0c2c6db added elrepo element
Change-Id: I7bc144afa96f45122857ff634c8c19c1fd759450
2022-09-02 15:09:08 +02:00
Rafal Lewandowski
f9287fe2ae changed release check logic in lvm element
Change-Id: I83b3c2dad3b0d6006ae23307ae7a8a83b12806e7
2022-09-02 12:37:15 +02:00
Zuul
4a1f684e5f Merge "Add subscription-manager repo names for RHEL-9" 2022-09-02 07:35:17 +00:00
Will Szumski
01def0fb1a Adds passwd to rocky-container os packages
The utility `passwd` is currenly missing from the images built
with the rocky-container image due to its container lineage.

Change-Id: If80c202c8adab6c5b750c54da5784b5afcd6bf19
2022-09-01 16:25:38 +01:00
Steve Baker
833c5b8ceb Support LVM thin provisioning
This change extends the block device lvs attributes to allow creating
a volume which represents a thin pool, and to create volumes which are
allocated from this pool.

Change-Id: Ic58f55c36236cc8c6279fbcb708e27dc2982f2d5
2022-08-24 10:34:42 +12:00
Steve Baker
f61548d863 Add thin provisioning support to growvols
This change enhances the growvols script to support all volumes being
backed by one thin provisioning pool.

If a pool is detected, the following occurs:
- validation to confirm every volume is backed by the pool
- only the pool is extended into the new partition
- volumes are extended by the same amount as the non thin-provisioned
  case

This results in no volumes being over-provisioned, so
out-of-space behaviour will be the same as the non thin-provisioned
case.

This change also switches to using /dev/mapper device mapper paths for
volume block devices, since that is the only path the thin pool is
mapped to.

Change-Id: I96085fc889e72c942cfef7e3acb6f6cd73f606dd
2022-08-24 10:14:26 +12:00
Zuul
15430098a0 Merge "Do dmsetup remove device in rollback" 2022-08-23 16:11:30 +00:00
Ian Wienand
467bbf36dd rocky : create machine-id in 9
It turns out we do need to create the machine-id for the same reason
as on 8.  This was being hidden by the bootloader choosing the root
disk label from the host (see the dependent change).

Change I3b518802d681b888916a5cc6a3dcf7e1b537da1e has modified the
testing to use a different root-disk label, which should help catch
this in the fututure.

Depends-On: https://review.opendev.org/c/zuul/nodepool/+/853574
Change-Id: I64de66cac25fd2e051780fb4812e075c647eb76e
2022-08-19 18:54:43 +10:00
Ian Wienand
3425624686 Allow setting ROOT_LABEL from environment
--root-label was added with I596104d1a63b5dc6549e8460a1ae3da00165ef04

This sets the ROOT_LABEL environment variable.

Over the years how this deploys has become more complex; now this
value gets written into DIB_BLOCK_DEVICE_PARAMS_YAML default values,
which is then loaded into DIB_ROOT_LABEL.

To override this from the environment you need to specify a full
DIB_BLOCK_DEVICE_CONFIG -- we don't have a way to just merge in the
root label setting.

Using the command-line argument is difficult with tools like nodepool
where the command-line is baked into something else.  However we
already have methods for overriding environment variables on dib
calls.

Several of the other variables here accept default values from the
environment, so this is not an outlier.  Making ROOT_LABEL also do
this allows us to test with non-default root devices in the gate (see
the linked change).

Change-Id: Ia1ef48c24841a86f387ff9603c64fd23d8670193
Needed-By: https://review.opendev.org/c/zuul/nodepool/+/853574
2022-08-18 11:21:19 +10:00
Steve Baker
1a4fb0b89b Do dmsetup remove device in rollback
Without this change, the final unmount will timeout after the
rollbacks are called when the partitioning fails due to a user error.

dmsetup remove is called both for partition and LVM volume devices.

Change-Id: I99679ea00338d4018a95d4da9b21685161cd5049
2022-08-18 10:23:41 +12:00
Vanou Ishii
e8c7787ec8 Fix wrong yum.conf name of CentOS 9 Stream
This commit fixes wrong name of yum.conf files
on CentOS 9 Stream in centos element to correct ones.

Change-Id: I25f0661fa79b7bc8ac1b8e3b2831a413c4161d1d
2022-08-08 22:53:21 +09:00
Neil Hanlon
6b42cb23ee
Add Rockylinux 9 build configuration and update jobs for 8 and 9
* Add note in Rocky Linux 9 containerfile to explain not needing to run
  systemd-machine-setup-id

Change-Id: I01233ce27cc4558270b2f41fae36b46a17f56e52
2022-08-02 14:07:05 -04:00
Zuul
6745279243 Merge "Upgrade openEuler to 22.03 LTS" 2022-08-02 11:21:59 +00:00
Zuul
b918e7188b Merge "rockylinux : create machine-id early" 2022-08-02 09:27:37 +00:00
annumishra_17
2424943e11 Add subscription-manager repo names for RHEL-9
This commit adds subscription-manager repo names for RHEL-9 qcow2 images

Change-Id: I2f6418a70b6afe543c225d5b2d383f8809842393
2022-08-02 08:53:13 +00:00
Zuul
ac5823cab1 Merge "opensuse: better report checksum errors" 2022-08-01 14:44:36 +00:00
wangxiyuan
934a65bc34 Upgrade openEuler to 22.03 LTS
openEuler 20.03-LTS-SP2 was out of date in May 2022. 22.03 LTS
is the newest LTS version. It was release in March 2022 and
will be maintained for 2 years. This patch upgrades the LTS
version. It'll be used in Devstack, Kolla-ansible and so on
in CI jobs.

This patch also enables the YUM mirror to speed up the package
download.

Change-Id: Iba38570d96374226b924db3aca305f7571643823
2022-08-01 19:22:35 +08:00
Ian Wienand
99f10f9380 rockylinux : create machine-id early
Somewhere between the upstream container
rockylinux/rockylinux:8.6.20220515 and the latest release, systemd
started to be pre-installed in the container.

With <= 20220515 installing the kernel-core package would end up
pulling in systemd.  As part of the systemd package installation, the
/etc/machine-id file is created and populated.

The kernel package post-install steps install the kernel with
/bin/kernel-install; this is responsible for copying the kernel
binaries into /boot.  It does this based on the machine-id, and it
seems its failure case with a blank machine-id is to simply skip
copying the kernels into /boot.  To compound this problem, it seems
our bootloader installation doesn't notice that we don't have a kernel
installed, so we end up building an unbootable image.

Testing is/was showing us this; but as rocky is non-voting and this
occured at a random time (rather than in response to a dib change) I
think it slipped by us.

To work around this, create the machine-id early in the container.  We
already have paths that remove the machine-id from final images.

Change-Id: I07e8262102d4e76c861667a98ded9fc3f4f4b82d
2022-08-01 15:55:11 +10:00
Zuul
50390d047f Merge "Removing old grub removal step" 2022-07-25 04:32:11 +00:00
Zuul
e304ff7656 Merge "Parse block device lvm lvs size attributes" 2022-07-25 04:23:10 +00:00
Zuul
fc0699bec7 Merge "Allow Gentoo to manage python versions by itself" 2022-07-25 02:15:56 +00:00
Zuul
a282d2a2bf Merge "cache-url: turn off -x by default" 2022-07-25 02:04:58 +00:00
Zuul
ba88a12456 Merge "ubuntu: more exact match on squashfs file, containerfile: use focal" 2022-07-12 16:29:16 +00:00
Ian Wienand
4ba0c588dd cache-url: turn off -x by default
I think that generally this is a lot of noise in the logs, as the
internals of cache-url is well tested, so we don't need to trace log
by default.

Change-Id: I25b5a1ec0d8f99691b2b4b62b9fdd537e5a773e4
2022-07-12 17:09:24 +10:00
Ian Wienand
0abf65eaf7 opensuse: better report checksum errors
Report the checksums and size to make it clearer what is failing.

Change-Id: I7c662e13cdbc19b3935f66ed422f15c2de53e525
2022-07-12 14:03:10 +10:00
Ian Wienand
78d389526c ubuntu: more exact match on squashfs file, containerfile: use focal
This is a squash of two changes that have unfortunately simultaneously
broken the gate.

The functests are failing with

 sha256sum: bionic-server-cloudimg-amd64.squashfs.manifest: No such file or directory

I think what has happened here is that the SHA256 sums file being used
has got a new entry "bionic-server-cloudimg-amd64.squashfs.manifest"
which is showing up in a grep for
"bionic-server-cloudimg-amd64.squashfs".  sha256 then tries to also
check this hash, and has started failing.

To avoid this, add an EOL marker to the grep so it only matches the
exact filename.

Change I7fb585bc5ccc52803eea107e76dddf5e9fde8646 updated the
containerfile tests to Jammy and it seems that cgroups v2 prevents
podman running inside docker [1].  While we investigate, move this
testing back to focal.

[1] https://github.com/containers/podman/issues/14884
Change-Id: I1af9f5599168aadc1e7fcdfae281935e6211a597
2022-07-11 19:56:36 +10:00
Matthew Thode
eb162ec6ed
Allow Gentoo to manage python versions by itself
Gentoo can manage python versions itself.  Before this commit users were
forced to set python versions themselves.  Now they have the option to
set it if they wish.

The workaround needed for git is also no longer needed, so it's been
removed.

Change-Id: I06b259ef73a40df6b8ab92a5b424bffcf4ef764d
Signed-off-by: Matthew Thode <mthode@mthode.org>
2022-07-10 22:12:03 -05:00
Steve Baker
d090126c66 Parse block device lvm lvs size attributes
The block device lvm lvs `size` attribute was passed directly to
lvcreate, so using units M, G means base 2. All other block device
size values are parsed with accepted conventions of M, B being base 10
and MiB, GiB being base 2.

lvm lvs `size` attributes are now parsed the same as other size
attributes. This improves consistency and makes it practical to
calculate volume sizes to fill the partition size. This means existing
size values will now create slightly smaller volumes. Previous sizes
can be restored by changing the unit to MiB, GiB, or increasing the
value for a base 10 unit.

The impact on this change should be minimal, the only known uses of lvm
volumes (TripleO, and element block-device-efi-lvm) uses extents
percentage instead of size. The smaller sizes can always be increased
after deployment.

Requested sizes will also be rounded down to align with physical
extents (4MiB). Previously specifying a value which did not align on
4MiB would consume an extra extent which could unexpectedly consume
more than the partition size.

Change-Id: Ia109cc5105071d82cc895d8d9cb85bc47da20a7a
2022-07-06 11:27:42 +12:00
Zuul
527e75a6de Merge "update default python for gentoo to 3.10" 2022-07-01 00:13:39 +00:00
Zuul
e43597affe Merge "Use internal dhcp client for centos 9-stream and beyond" 2022-06-30 18:20:50 +00:00
Matthew Thode
d85a3421a7
update default python for gentoo to 3.10
Change-Id: Iaf1788342e7418af92dd35714415a979b0ba11db
Signed-off-by: Matthew Thode <mthode@mthode.org>
2022-06-30 09:50:35 -05:00
Christopher Stone
d0a1909e62 Removing old grub removal step
Closes-bug: #1950701
Change-Id: I01f7190db9924baafac4fbc0b45a4c0e158355cc
2022-06-29 15:30:39 -07:00
Julia Kreger
5f01bd5d61 Use internal dhcp client for centos 9-stream and beyond
All indication in CI is that Centos Stream9's use of dhclient
appears to point to compatability issues when interacting with
dnsmasq. However, this doesn't appear to be the issue with the
internal dhcp client. As such, lets constraint the RH default
so that it no longer applies to Centos 9-stream.

I've also added a documentation entry for DIB_DHCP_CLIENT which
was previously undocumented.

As an aside, I've already reached out to RH's NetworkManager team
regarding this, but root cause is not entirely understood at this
point.

Change-Id: I235f75b385a8b0348c8fe064038c51409f8722c4
Story: 2010109
Task: 45677
2022-06-28 11:02:03 -07:00
Steve Baker
f118649738 Fix BLS entries for /boot partitions
Creating a separate /boot partition is desirable in some cases[1].
This change detects if /boot is a partition, and ensures that the
kernel/ramdisk paths are correct in either case. This is applied to
all BLS entries files, whether they were generated by the previous
grub2-mkconfig call or in the source image.

This means the rhel9 specific workaround can be removed since all
paths are now normalised at this stage.

[1] https://review.opendev.org/c/openstack/tripleo-image-elements/+/846807

Change-Id: I62120ec8c65876e451532d2654d37435eb3606a6
Resolves: rhbz#2101514
2022-06-28 02:41:21 +00:00
Zuul
cdf931d190 Merge "Add Fedora 36 support" 2022-06-21 23:42:21 +00:00
Zuul
8bb14b9195 Merge "containerfile: warn if we don't have a Dockerfile" 2022-06-21 23:42:18 +00:00
Ian Wienand
0385857cd9 Add Fedora 36 support
Change-Id: I89ff95604aabf59217382bc784783759191c59a3
2022-06-21 14:47:12 +10:00
Zuul
80c1a519c2 Merge "Add a warning in satellite configuration" 2022-06-21 03:25:58 +00:00
Ian Wienand
3e51709307 containerfile: warn if we don't have a Dockerfile
Currently if no Dockerfile is specified or found, we exit later with
an obscure error.  Check this after the element search; if we still
don't have something to build then we can't continue.

Change-Id: Ifb17a0995fab0ccfe7ee08363676c1fa57e37592
2022-06-21 12:03:46 +10:00
Ian Wienand
0cebad3d06 Revert "CentOS 9-stream : work around selinux permissions issue"
This reverts commit 975522472a.

Thankfully the fix is released upstream now.  This spawned a thread
for disucssion at

  https://lists.centos.org/pipermail/centos-devel/2022-June/120414.html

Change-Id: I5b53e2b7e751ceaf74da8914875203d631a6137c
2022-06-16 11:24:24 +10:00
Lon Hohberger
887546d927 Fix backward regex match
'9-stream' was being matched against the regex '9',
causing builds on RHEL9 to try to install C9S RPMs.
We want this the other way so that DIB_RELEASE=9
will not match the regex '9-stream'.

Resolves: rhbz#2097443

Signed-off-by: Lon Hohberger <lhh@redhat.com>
Change-Id: Iefd7c23512c460e33117d12bbc33606134daa9e2
2022-06-15 13:58:24 -04:00
David Hill
49285db4d3 Add a warning in satellite configuration
Add a warning in satellite configuration as when no activation_key
is provided and no environment is configured, subscription-manager
might hang as it's prompting the user to provide the missing
parameter.

Change-Id: I9564841ca845eafc2bd39be6b05bef62e8062f28
2022-06-14 08:45:28 -04:00
Ian Wienand
975522472a CentOS 9-stream : work around selinux permissions issue
Due to the referenced inline issue, 9-stream currently fails running
setfiles in a chroot without /proc.  Since we want to actually label
/proc, we don't want it mounted.  This pulls in the fixed packages to
get things going until the fix is rolled out.

Change-Id: Id41c16130e975779cb70e2ab19807a689450d026
2022-06-09 17:39:50 +00:00
Zuul
70e90e1755 Merge "Fix grub setup on Gentoo" 2022-05-27 17:13:34 +00:00
Zuul
8e6418c0bb Merge "Ensure passwd is installed on RH and derivatives" 2022-05-26 18:03:01 +00:00
Zuul
37edd33044 Merge "Check and mount boot volume for data extraction with nouuid" 2022-05-26 03:55:27 +00:00
Julia Kreger
57149d9eb1 Check and mount boot volume for data extraction with nouuid
When building an image, say RHEL9, on a host installed with that
same image, you will be blocked from mounting the filesystems to
extract contents, as the host OS kernel will identify the duplicate
UUIDs and error accordingly.

This was previously fixed for the root filesystem, but not the boot
filesystem.

Change-Id: I63a34fba033ed1c459aeb9c201c8821fa38a36e9
2022-05-25 12:39:57 -07:00
Zuul
25f15f8965 Merge "Make centos reset-bls-entries behave the same as rhel" 2022-05-25 16:04:56 +00:00
Pavlo Shchelokovskyy
62626521ad Fix openssl example command in dynamic-login
the command had one error in it (missing one backslash)
and was rendered wrong, w/o any backslashes at all.

Change-Id: If187f645b818f47d10b602ccee12c29892a8d88d
2022-05-24 14:10:01 +03:00
Dr. Jens Harbott
b774869c3b Fix grub setup on Gentoo
After some recent reordering[0], the /boot/grub directory isn't created
early enough on Gentoo any more, let us just ensure ourselves that it is
in place when we create the grub config.

[0] I8cb34914bbbfa05521bbb71cc6637368b980358f

Change-Id: I8a84d08c3090e46b00d1d626fb984f66ea33f256
2022-05-22 14:18:20 +02:00
Zuul
2f06cbc4eb Merge "Add Jammy functesting to dib" 2022-05-13 06:31:17 +00:00
Denys Mischenko
abc2524a5c Adopted dkms element to work on Ubuntu Jammy and nvidia drivers
Previously a module version was splitted from the module name:

  nvidia, 510.47.03, 5.4.0-109-generic, x86_64: installed

In Jammy it is now a part of the name:

  nvidia/510.47.03, 5.15.0-27-generic, x86_64: installed

Assuming the fact that it would be threatted as a path this change
doesn't brake anything which was working before. But at the same
time it allows to pass last step where dkms is requested to build all
modules.

Change-Id: Ic1bb2b45f9db906b64ca03ae5c4e05b2114f2a74
2022-05-12 10:20:25 +00:00
Cédric Jeanneret
6645f47db4 Ensure cloud-init is configured to generated host keys
It may happen a base image has an edited version of cloud-init
"cloud.cfg" that prevents the host keys to be generated.

While it didn't represent an issue with older releases of cloud-init,
starting cloud-init-22 this isn't true anymore.

Before that release, an sshd-keygen@.service was present and called by
sshd-keygen.target (which was called by sshd.service), and we ended up
with ssh host keys in any cases - either generated from cloud-init, or
generated by sshd-keygen.service.

But cloud-init-22 introduced an edition to the sshd-keygen.service,
making it check for the presence of cloud-init service, and preventing
this sshd-keygen to kick in this case.

So we'd better ensure cloud-init is able to generate the keys, else
we'll be in a bad state, since it's instructed to remove the ones
present.

Closes-Bug: #1971751
Change-Id: I37b2f3e9d57a86544ef14e74a4a927309c18bbf0
2022-05-06 11:10:54 +02:00
Clark Boylan
7b6247554e Add Jammy functesting to dib
This adds arm64 ubuntu-minimal Jammy functests and x86 ubuntu image
based Jammy functests. To make this happen we have to install
debootstrap from debian unstable on the functest nodes in order to get
access to a debootstrap that knows what jammy is.

As we ramp up Jammy support in our tools having good testing will be
helpful.

Change-Id: I1d1dc752ce176457d0656cbd50e27a2721ca9856
2022-05-04 13:03:50 -07:00
Zuul
d3ce32016a Merge "containerfile: update test to jammy" 2022-05-03 22:56:35 +00:00
Zuul
a055b59f8d Merge "centos: avoid head pipe failure" 2022-05-03 21:04:39 +00:00
Neil Hanlon
7025d891c7
Ensure passwd is installed on RH and derivatives
Images built from containers may not have this package, so just ensure
its presence.

Change-Id: Iee77daccc61e284b675d63eeb922eafd48ae8611
2022-05-03 12:05:36 -04:00
Ian Wienand
999da87243 yum-minimal: workaround missing $releasedir variable
As described inline, work around missing $releasedir variables seen in
production builds.

Change-Id: Ia114f41d7a2fa86957235996f10a74adeecf767c
2022-05-03 15:47:55 +00:00
Zuul
45abd8ef95 Merge "Revert "Fallback to persistent netifs names with systemd"" 2022-05-03 00:26:20 +00:00
Steve Baker
8059a59583 Make centos reset-bls-entries behave the same as rhel
This makes 03-reset-bls-entries consistent with rhel so that the glob
match is *.conf, and a check is added to ensure that a rename is
actually required.

Change-Id: I4adff43cf7d4f31d939e6ddf37ac8d162ccd0db7
2022-04-29 13:12:24 +12:00
Zuul
555cecb670 Merge "Set machine-id to uninitialized to trigger first boot" 2022-04-28 01:35:54 +00:00
Zuul
737c9a8ff3 Merge "Fix dhcp-all-interfaces on debuntu systems" 2022-04-28 00:05:10 +00:00
Maksim Malchuk
b97dfb8fbd Revert "Fallback to persistent netifs names with systemd"
This reverts commit 8401290976.

We are reverting this because some users may want to use predictable
device names and may not even use Debian. However, after some
investigation we have found a couple of bugs in dhcp-all-interfaces on
Debuntu distros. The parent change corrects those bugs. Additionally new
Linux kernels emit "move" events to udev when interfaces are renamed to
their predictable name. Support this "move" in the dhcp-all-interfaces
udev rules. Making these changes appaers to produce functional images
for Debian users using predictable device names. If predictable device
names are not desired turning them off is straightforward and release
notes are updated to give users the info they need to do that outside of
this element.

Change-Id: I125f1a0c78a103b51bda961528c3e66c345bf604
Co-Authored-By: Clark Boylan <clark.boylan@gmail.com>
Signed-off-by: Maksim Malchuk <maksim.malchuk@gmail.com>
2022-04-27 16:29:58 +00:00
Clark Boylan
7b41f8160b Fix dhcp-all-interfaces on debuntu systems
There are two issues with dhcp-all-interfaces on debuntu interfaces
addressed here. First is the path to dhclient lease files is
/var/lib/dhcp not /var/lib/dhclient. Second there is a missing newline
in the ENI interface file which causes a parse error.

Change-Id: Ice83e0d49a4234301dc12daf828ba80fef414cdb
2022-04-26 13:37:13 -07:00
Zuul
638a94d8c9 Merge "Move reset-bls-entries to post-install" 2022-04-26 10:02:48 +00:00
Zuul
fb7db1b736 Merge "yum-minimal: clean up release package installs" 2022-04-26 09:54:34 +00:00
Ian Wienand
5bc9e87da1 centos: avoid head pipe failure
I just saw in the trace output of a failure

  > grep -o 'CentOS-.[^>]*GenericCloud-.[^>]*.qcow2'
  > sort -r
  > head -1
  sort: fflush failed: 'standard output': Broken pipe
  sort: write error

i.e. the "head -1" has exited after reading one line, but "sort -r"
still wants to write and thus has hit a pipe failure, and because we
run with "-o pipefail" this has halted the script.

This seems like it has been there more or less forever, maybe we just
got lucky hitting it now?  Anyway, we can work around this by using a
process substitution and passing the output of this into head, this
way we won't hit a pipe failure.

I also updated the fedora path as it does the same thing.

Change-Id: I44d97e5bb31702aacf396e0229329a2ef9c64f2f
2022-04-26 17:28:00 +10:00
Ian Wienand
28821fd283 containerfile: update test to jammy
We've not really been using the Focal containerfile, as we move
forward jammy is a better choice for keeping stable as we might find
some new users for it.

Also add binutils to bindep for native bullseye builds (see
Icb0e40827c9f8ac583fa143545e6bed9641bf613)

Change-Id: I22ebe2bbccaec34180e58996b21e47bfc4f36055
2022-04-22 14:48:18 +10:00
Zuul
adc40db9e9 Merge "Update gentoo python version to 3.9" 2022-04-21 13:56:25 +00:00
Steve Baker
e129fad7f8 Move reset-bls-entries to post-install
03-reset-bls-entries was previously a pre-install script to run after
the machine-id was set, but a new kernel may be installed during the
install phase, which will install another bls entry file with a
filename which differs from the machine-id.

This means this package installed bls file won't be updated when
grub2-mkconfig is called, resulting in incorrect kernel args and boot
device in the entry file that will get booted by default.

By fixing the filenames after the new kernel is installed,
grub2-mkconfig will update the bls file that actually gets used on
boot.

Change-Id: I653bef9638e38ded68458fd40d90e30e5206caad
2022-04-21 10:13:10 +12:00
Steve Baker
147641fc3e Set machine-id to uninitialized to trigger first boot
According to the systemd documentation[1], if /etc/machine-id is empty
it will be populated with a unique value, but not in a way which
triggers an actual first boot event (running units with
ConditionFirstBoot=yes set)

This change writes "uninitialized" to /etc/machine-id to ensure that
systemd-firstboot.service actually runs, and other units can use
first-boot-complete.target as a dependency to trigger on first boot.

Since /var/lib/dbus/machine-id is sometimes a symlink to
/etc/machine-id, it is truncated before writing to /etc/machine-id.

On older versions of systemd before first boot semantics were
formalised, any non-uuid value will trigger a new machine-id to be
generated, so "uninitialized" also works.

[1] https://www.freedesktop.org/software/systemd/man/machine-id.html#First%20Boot%20Semantics

Change-Id: I77c35e51a3da2e8a6b5a2c80d033a159b303c9af
2022-04-21 09:39:42 +12:00
Zuul
65569921bf Merge "Move grub-install to the end, and skip for partition images" 2022-04-20 15:14:05 +00:00
Zuul
ab75f1d8a2 Merge "Add interpolation note for dynamic-login password" 2022-04-20 06:04:38 +00:00
Ian Wienand
27903f90e2 yum-minimal: clean up release package installs
This started a long way from here, when I noticed that "top" on centos
9-stream images wasn't working because ncurses-base wasn't installed.

This led me to the extant install of bash/glibc/ncurses-libs from
Iecf7f7e4c992bb23437b6461cdd04cdca96aafa6.  However it didn't really
explain why these are brought in here.

Reading further it became clearer that over the years of distribution
additions, Fedora updates, etc. this has grown into a bit of a mess.

Refactor the release package installs into a more logical flow,
pulling out checks/comments for Fedora's of ancient history, etc.
Remove the 9-stream package installs; this isn't the place for them,
and the should be brought in by the base packages.

Ultimately, this is intendend to a be a no-op refactor.

Change-Id: Ie7d9a6497d0d20a3303ec0da3d0668c74efa2c3d
2022-04-20 09:11:16 +10:00
Ian Wienand
4cb3346fec source-repositories : use explicit sudo/-C args when in REPO_DEST
The recent git ownership-checking changes (see related bug for full
details) mean we can not run git in non-owned directories.

We have a couple of cases here where we have done a "pushd" to work in
the REPO_DEST context; this is the destination directory that is
inside the chroot so needs to be operated on as "root" (via sudo
calls).  This certainly makes sense -- but given the new way of things
it can hide what context each call is working in, which is now very
important.  Previously this worked because you could read it; now it's
doing the UID check too, calls in here without sudo now fail.

Remvoe the pushd's and make every call that works in REPO_DEST
explicit with -C, and add sudo calls around it.

Change-Id: Id1f6bd94c9c77ef6ab2b562a7e0bc48f749c58ac
Related-Bug: https://bugs.launchpad.net/devstack/+bug/1968798
2022-04-14 16:53:37 +00:00
Dr. Jens Harbott
4b565f1ddb Update gentoo python version to 3.9
Seems gentoo has updated their defaults so builds are failing with
py3.8.

Change-Id: I01747e96215cfb2e8648e168f823344bcdbcb4e7
2022-04-12 09:26:28 +02:00
Zuul
709d18fd42 Merge "Update fedora element testing to F35" 2022-04-11 05:15:52 +00:00
Marcin Juszkiewicz
64bdd24e4e CentOS Stream 9 has EPEL now
Change-Id: I13131f8f4b668d01661a57d918da1bfdb69a0584
2022-04-07 09:01:13 +02:00
Pawel Konczalski
7f5d6538c1 Add interpolation note for dynamic-login password
Change-Id: I416c60347679dc648914e7c1b02761882bae8d4a
2022-03-25 11:59:44 +01:00
Steve Baker
9987d0911a Move grub-install to the end, and skip for partition images
The bootloader element installs the grub bootloader for whole-disk
images, but it also correctly sets values in /etc/default/grub and BLS
entries.

This value setting is useful even if the bootloader isn't installed.
For example, the overcloud-full partition image benefits from a
correct /etc/default/grub and BLS entries which ironic-python-agent
will use when it installs grub on the disk during baremetal deploy.

This change moves the actual grub install to the end of the script,
and if there is no $DIB_BLOCK_DEVICE set then install is skipped.

This allows overcloud-full to use the bootloader element instead of
the grub2 element, so the correct grub defaults are set on centos9,
including the correct root device on centos9.

Change-Id: I8cb34914bbbfa05521bbb71cc6637368b980358f
2022-03-21 08:43:48 +13:00
Benedikt Loeffler
94fab70e58 Use https for downloading ubuntu images
Change-Id: I8d317582d454df2c418a7d3a37aea38dc7570391
2022-03-16 13:43:30 +01:00
Michal Nasiadka
7d782ae1c9 containerfile: Add support for setting network driver
Change-Id: Ia885237406bf4c7b9d49b349f374558ae746401f
2022-03-15 13:18:11 +00:00
Zuul
bce7af6a29 Merge "Handle btrfs root subvolume for fedora extract-image" 2022-03-14 18:54:12 +00:00
Ian Wienand
4f089b33c5 Update fedora element testing to F35
This reverts the mirror removal in
I817b412b7f06523df635e8b16111bc1081b40f66 and updates the test to F35,
which is mirrored.

Change-Id: I00d24690f57bedd3fc5ebbc18de0ed874ad1e4ef
2022-03-11 14:53:00 +11:00
Michal Nasiadka
a02cb9ff74 containerfile: add support for Docker
In some build environments Docker is already installed - and adding
podman is not an option.  Add a new variable to toggle this, and
rename the now incorrectly titled DIB_CONTAINERFILE_PODMAN_ROOT to
just ...RUNTIME_ROOT to match.

Change-Id: I677e4f491b40360dceabdf4f2a9e64c7cb493dc7
2022-03-11 14:04:52 +11:00
Steve Baker
7de5bc6fa3 Handle btrfs root subvolume for fedora extract-image
This adds a check for the root device having filesystem type btrfs,
and when it is assume there is a subvolume called "root". This fixes
extract-image when using Fedora-Cloud-Base btrfs images.

This should be sufficient until there is another btrfs base image with
a different subvolume layout.

Change-Id: Ib18979090585ba92566e523951b521b9d902fcb7
2022-03-11 15:48:03 +13:00
Zuul
e78159cd44 Merge "Correctly create DIB_ENV variable and dib_environment file" 2022-03-04 15:55:40 +00:00
Maksim Malchuk
f5e4060533 Correctly create DIB_ENV variable and dib_environment file
The DIB_ENV variable can contain multiline content, for example at
least DIB_BLOCK_DEVICE_CONFIG is a multiline YAML data [1], so for
this type of content the variable created with incomplete data, also
this incomple data echoed [2] when creating the dib_environment file
which creates an unusable file. This change fixes the issue.

1. 79ea63f525/doc/source/user_guide/building_an_image.rst (disk-image-layout)
2. 79ea63f525/diskimage_builder/elements/manifests/cleanup.d/01-copy-manifests-dir (L32)

Change-Id: I3b74ede69eb064ad813a9108ec68a228e549e8bb
Signed-off-by: Maksim Malchuk <maksim.malchuk@gmail.com>
2022-02-28 16:46:16 +03:00
Steve Baker
41c21e91db Revert "Revert "Detect boot and EFI partitions in extract-image""
This change is proposed again, avoiding lsblk features missing from
older distros:
- lsblk is avoided entirely for a whole-disk image with a single
  partition, which would be the majority of old image building jobs
- Field PARTTYPENAME not available on the lsblk in CentOS-8, instead
  rely on the GUID being correct for EFI partitions
- Argument --output-all not available on the lsblk in CentOS-7, this
  is just for logging debug, so can be removed

This reverts commit b06bac734c.

Change-Id: Ib0d4e7751fd968511fc7f672d524e58d1488ae11
2022-02-25 14:52:45 +13:00
Zuul
0e5986e9fb Merge "Revert "Detect boot and EFI partitions in extract-image"" 2022-02-24 16:45:31 +00:00
Riccardo Pittau
b06bac734c Revert "Detect boot and EFI partitions in extract-image"
This reverts commit 0630b3cb69.

Reason for revert: breaks compatibility with CentOS Stream 8, lsblk does not have PARTTYPENAME until version 2.35 and CS8 has version 2.32.1 installed

Change-Id: I7fc0e76f0eeb8594d8a0d57629b2c67526b961ad
2022-02-24 13:42:50 +00:00