Commit Graph

4078 Commits

Author SHA1 Message Date
Riccardo Pittau
4f151aeeb5 Install only python3 pip in debian bullseye
Debian bullseye has removed python-pip and python-virtualenv
from its repos, let's install only pip and virtualenv python3 modules.

Also split pip installation based on python2 and python3 for
debian-based distributions.

Change-Id: I2b75afd310f009ae8614f6ca75bb984b56d25c45
2021-12-10 15:11:17 +01:00
Zuul
a2b8d4d846 Merge "Update keylime-agent and tpm-emulator elements" 2021-12-07 04:29:36 +00:00
Zuul
b22161c116 Merge "Disable all repositories after attaching a pool" 2021-12-06 20:19:59 +00:00
Zuul
86e3d632fa Merge "Document EFI elements requirements" 2021-12-06 19:58:16 +00:00
Piotr Parczewski
f49dba8f62 Document EFI elements requirements
Change-Id: I6b47d44c86a11dfc71b7adf0d9bb27f5177717cb
2021-11-26 08:46:42 +01:00
Ian Wienand
2d47d4157c Fix BLS based bootloader installation
This reverts I2701260d54cf6bc79f1ac765b512d99d799e8c43,
Idf2a471453c5490d927979fb97aa916418172153 and part of
Iecf7f7e4c992bb23437b6461cdd04cdca96aafa6 which added special flags to
update kernels via grubby.

These changes actually ended up reverting the behaviour on Fedora 35,
which is what led me to investigate what was going on more fully.

All distros still support setting GRUB_DEVICE in /etc/default/grub;
even the BLS based ones (i.e. everything !centos7).

The implementation *is* confusing -- in earlier distros each BLS entry
would refer to the variable $kernelopts; which grub2-mkconfig would
write into /boot/grub2/grubenv.  After commit [1] this was reverted,
and the kernel options are directly written into the BLS entry.

But the real problem is this bit from [2]

 get_sorted_bls()
 {
     if ! [ -d "${blsdir}" ] || ! [ -e /etc/machine-id ]; then
        return
     fi
     ...
     files=($(for bls in ${blsdir}/${machine_id}-*.conf; do
     ...
 }

i.e., to avoid overwriting BLS entries for other OS-boots (?),
grub2-mkconfig will only update those BLS entries that match the
current machine-id.

The problem for DIB is that we are clearing the machine-id early in
finalise.d/01-clear-machine-id, but then running the bootloader update
later in finalise.d/50-bootloader.

The result is that the bootloader entry generated when we installed
the kernel (which guessed at the root= device, etc.) is *not* updated.
Even more annoyingly, the gate doesn't pick this up -- because the
gate tests run on a DIB image that was booted with
"root=LABEL=cloudimg-rootfs" the kernel initially installed with
"install-kernel" (that we never updated) is actually correct.  But
this fails when built on a production host.

Thus we don't need any of the explicit grubby updates; these are
reverted here.  This moves the machine-id clearing to after the
bootloader setup, which allows grub2-mkconfig to setup the BLS entries
correctly.

[1] 4a742183a3
[2] https://src.fedoraproject.org/rpms/grub2/blob/rawhide/f/0062-Add-BLS-support-to-grub-mkconfig.patch

Depends-On: https://review.opendev.org/c/zuul/nodepool/+/818705
Change-Id: Ia0e49980eb50eae29a5377d24ef0b31e4d78d346
2021-11-25 14:26:23 +11:00
Zuul
11d25f76d2 Merge "Added missing grubby arg DIB_BOOTLOADER_DEFAULT_CMDLINE" 2021-11-24 22:30:43 +00:00
Zuul
583b594ad4 Merge "Trivial: fix whitespace in ubuntu element rst" 2021-11-23 18:37:15 +00:00
Zuul
b2a37f85f3 Merge "Allowing ubuntu element use local image" 2021-11-23 16:59:28 +00:00
Julia Kreger
a40261326f Trivial: fix whitespace in ubuntu element rst
Change-Id: If054f1fca0c884c2b66225126efad8f15d58cc70
2021-11-23 07:01:36 -08:00
Matthew Thode
86682071a8
update gentoo source suffix (where it finds the file to download)
gentoo releng updated how they name the files.

Change-Id: Ib2cbc007e9690fdaaf1d6cb6f5fbcb634c4a4bbe
2021-11-20 20:18:55 -06:00
Zuul
94b5b77d55 Merge "Revert "centos 9-stream: make non-voting for mirror issues"" 2021-11-18 07:11:01 +00:00
Zuul
acb35df1dc Merge "Add openEuler jobs back" 2021-11-18 07:10:59 +00:00
Marco Vaschetto
443305ab61 Allowing ubuntu element use local image
Patch allow to set path for local image source,
instead download latest or use the cached image.
This permit to build image also in environment without internet access.

re-propose of patch: https://review.opendev.org/c/openstack/diskimage-builder/+/809009

Change-Id: I54395b09af339caee040326b809e8fbf8b0e7d6a
2021-11-17 20:34:48 +01:00
Zuul
2c10f65599 Merge "Replace deprecated assertEquals" 2021-11-17 01:39:28 +00:00
Jakub Darmach
5a4e8a2c9e
Added missing grubby arg DIB_BOOTLOADER_DEFAULT_CMDLINE
DIB_BOOTLOADER_DEFAULT_CMDLINE was missing from grubby args.
Closes-Bug: 1951084

Change-Id: Idf2a471453c5490d927979fb97aa916418172153
2021-11-16 11:43:49 +01:00
Ian Wienand
ddb06b6657 source-repositories: don't use --git-dir
A recent(-ish) change in git [1] has exposed a bug in caching that
appears in one very specific circumstance -- updating the
openstack/openstack super-repo [2].

This repo gets a submodule update every time something is pushed.  By
using "--git-dir" while the cwd is one-level above the actual repo we
are confusing [1] which is not finding the submodule directories
correctly and giving us an error:

 Could not access submodule 'foo'

for every submodule that has updated between now and the last time we
updated the cache. [3]

The git manual does warn about this

 If you just want to run git as if it was started in <path> then use
 git -C <path>.

Indeed, that is what we want to do in this path.  Modify the calls to
use -C.

[1] 505a276596
[2] https://opendev.org/openstack/openstack/
[3] The result for opendev production is that image builds fail every
    time an openstack/* project is checked in; we then race to retry
    the build before another commit lands and updates the submodules
    again.

Change-Id: Iadb23454e29d8869e11407e1592007b0f0963e17
2021-11-16 19:16:03 +11:00
likui
7a57da8cee Replace deprecated assertEquals
The assertEquals method has been deprecated since it was renamed
to assertEqual in Python 3.2.

https://docs.python.org/3/library/unittest.html#deprecated-aliases

Change-Id: I559ad57c2d96a3bb51080c72816f39d3baa4c9dd
2021-11-12 09:46:14 +08:00
Ian Wienand
aaf8e3eed8 Revert "centos 9-stream: make non-voting for mirror issues"
This reverts commit a1debc726d

Change-Id: I3cd31e0de53948fd5073b493389c93debc8b5d08
2021-11-10 11:43:33 +11:00
Ian Wienand
85e20d9852 containerfile: handle errors better
Refactor things to use explicit names, and put in a trap to cleanup
after any errors.

Currently, if the build/run/export steps fail, it leaves behind images
which eventually clog things to the point podman won't run any more
(see also https://github.com/containers/podman/pull/12233 about errors
seen due to this)

Change-Id: Ib328a07ad67e3f71f379fbf34ae7ef74e212ef1c
2021-11-10 11:42:49 +11:00
Ian Wienand
3833c2e59c containerfile: fix tar extraction
Ic68e8c5b839cbc2852326747c68ef89f630f26a3 removed the sudo from the
tar extraction here, meaning that production is failing to create the
chroot.  This is hidden in testing because
DIB_CONTAINERFILE_PODMAN_ROOT is set.  Make the sudo here
unconditional.

Change-Id: I6e36e3fc65981f85fad12ea2cd10780fde9c37da
2021-11-10 11:42:49 +11:00
Ian Wienand
a1debc726d centos 9-stream: make non-voting for mirror issues
Change-Id: I9dc5015e4f3f6a710329e024913abe822cac3ee1
2021-11-10 11:42:40 +11:00
Zuul
eb7782b78f Merge "tests: remove debootstrap install" 2021-11-08 04:59:04 +00:00
Ian Wienand
f75fed076e fedora-container: update to Fedora 35
Support Fedora 35 in this element, and update the default

Depends-On: https://review.opendev.org/c/zuul/nodepool/+/815766
Change-Id: I2a9a8648158e1e4b1a071e20266fb8d125da67ac
2021-11-03 06:26:56 +11:00
Zuul
311621a094 Merge "Add support for CentOS Stream 9 in DIB" 2021-10-29 09:55:30 +00:00
Alfredo Moralejo
cdff9045c0 Add support for CentOS Stream 9 in DIB
CentOS Stream 9 is close to be released, and official mirrors are
already poplated. This patch is adding support to centos-minimal in CS9.

Also enable centos-minimal/[8,9]-stream-build-succeeds tests.

This patch is being tested together with [1] to apply following list of elements:

 vm centos-minimal simple-init growroot nodepool-base openstack-repos infra-package-needs

[1] https://review.opendev.org/c/openstack/project-config/+/811442

Change-Id: Iecf7f7e4c992bb23437b6461cdd04cdca96aafa6
2021-10-27 13:38:14 +02:00
Ian Wienand
57b7d2bceb dracut-regenerate: drop Python 2 packages
We don't support any Python 2 guest systems, drop the Python 2 bits
from package-installs.

Change-Id: Ica5c71c2926968d438ac1d0b0f16981cadf0bab1
2021-10-27 11:11:50 +11:00
Ian Wienand
0f987ca4b6 tests: remove debootstrap install
This is not needed on the host any more; it's part of the
nodepool-builder container.

Change-Id: I56564d58fe6193c21294aacbf9f769bba2142ec7
2021-10-27 11:01:55 +11:00
Zuul
6a42cad8d5 Merge "Switch ARM64 testing to bullseye" 2021-10-26 08:54:55 +00:00
Ian Wienand
660bbaebb5 Add openEuler jobs back
We have resolved the issues causing dnf to segfault

Change-Id: Icfa2827b71de663a60c82515e19ea0100e1392c5
Closes-Bug: #1947960
2021-10-22 16:31:36 +11:00
Dr. Jens Harbott
4bd8258fbf Fix bootloader installation for gentoo
The if/elif block added in [0] doesn't work for gentoo, let's hope
that we can get along with an easy fix.

[0] https://review.opendev.org/c/openstack/diskimage-builder/+/804000
Signed-off-by: Dr. Jens Harbott <harbott@osism.tech>
Change-Id: I543e04d2d7efea3e718bae31aa1cc4767bd359f8
2021-10-21 10:20:58 +02:00
Ian Wienand
edbf1fa45b Switch ARM64 testing to bullseye
This more closely matches the nodepool-builder container, which is
Bullseye based.

Refactor to remove unnecessary abstract job.

Change-Id: I34822608f19e1ce9ef781034ff831d6359ed8e15
2021-10-21 16:00:05 +11:00
Sagi Shnaidman
d5a01519c6 Update centos element for 9-stream
This adds 9-stream support to the centos element.

See https://review.opendev.org/q/topic:cs9 for related patches.

Change-Id: Ib80fbd21edb77c25764eff2c0d66e55bde7a90af
2021-10-20 09:39:27 +11:00
Ian Wienand
1a5c563920 Run functional tests on Debian Bullseye
We need to update the base reference platform we perform the
functional tests on.  Debian bullseye seems like the best choice -- it
is recent enough to last for a while, and will match the
nodepool-builder container environment.

Depends-On: https://review.opendev.org/c/zuul/zuul-jobs/+/814088
Change-Id: Ic68e8c5b839cbc2852326747c68ef89f630f26a3
2021-10-20 09:39:27 +11:00
Ian Wienand
c41a2b7f84 Simplify functests job
At one point we had an array of functest jobs; we were testing
building on trusty, bionic, maybe centos, with python2 and python3.
The only thing left of all these combos these days is "bionic-python3"
(that's where this naming comes from).

We can remove the levels of abstraction now and just have the one job
to avoid confusion.

Change-Id: Id37d62a17f9b7f6dc6dc35585c29eddd435ce913
2021-10-20 09:39:27 +11:00
Ian Wienand
1ea71b3d69 Remove extras job, put gentoo job in gate
The only job left in the "extras" functional test is for gentoo.  We
might as well just drop this and put the gentoo devstack test in the
gate -- in terms of resources it's a 1:1 swap now.

Cleanup the ordering of the check queue list

Change-Id: I2af8f9235131cd0cce33e67c8d0d05c3b357320d
2021-10-20 09:39:27 +11:00
Ian Wienand
a6da91f753 functests: drop minimal tests in the gate
We test the minimal builds in the devstack based end-to-end tests.
There is no need to also run these functional tests; they test *less*
than the full build/upload/boot testing done in the devstack test.

We consolidate the separate "image" based job

Change-Id: I73ca83bedb4b0c40af5209f9c93a0e657c152591
2021-10-20 09:39:27 +11:00
Ian Wienand
55efeb575e centos7 : drop functional testing
This element has been deprecated since Sep 2019.  At this point we
don't need to keep gate testing it.

Change-Id: I2851b9be93effac30ca7c97ef8fd17e8641eeffa
2021-10-20 09:39:27 +11:00
Ian Wienand
f1f717ed07 functests: drop apt-sources
This builds ubuntu-minimal to test apt-sources, an element that
appears unused.  We don't need this as a gate test.

Change-Id: I5310c0a38a19a7c00764c1057b0a26aafdbacccc
2021-10-20 09:39:27 +11:00
Ian Wienand
f3a3153390 ubuntu: add Focal test
Switch this to testing bionic and focal, dropping xenial

Change-Id: Ifba6800ad84e504d248de8389e49a31f22c4554c
2021-10-20 09:39:27 +11:00
Ian Wienand
2f1e9320bd ubuntu-systemd-container: deprecate and remove jobs
I'm not aware this element is used/was ever used.  It hasn't ever been
updated to Focal.  To reduce our testing footprint remove this test,
and note in the element its probably broken.

Change-Id: I17cd3b13948287fe78990cfbe16a22919a329ba9
2021-10-20 09:39:27 +11:00
Ian Wienand
d5fa64325f Revert "Allowing ubuntu element use local image"
This reverts commit 1f4fb1d7a5.

This unfortunately wasn't actually tested.  Because the image-based
tests run sequentially, a prior failure in the centos-8 job meant the
ubuntu job never ran.

This is failing with

 10-cache-ubuntu-tarball: line 28: DIB_LOCAL_IMAGE: unbound variable

There is also a seemingly unused variable DIB_IMAGE_LOCAL_FILE; I'm
not sure what this is doing.

For now revert, and it can be re-proposed with appropriate testing.

Change-Id: I0f3897c90dc863ee04c3295b9cb094f02d8658e3
2021-10-20 09:39:27 +11:00
Ian Wienand
2d5981d673 epel: match replacement better
It looks like upstream have changed this line to "download.example",
breaking our subsitution.  Let's do a generic match.

Change-Id: I8e443022a5f239b98ccefe73a9abf8cf259dc8e9
2021-10-20 09:39:27 +11:00
Ian Wienand
2b85ddc669 Remove py35 tox jobs
At this point py35 practically means Xenial, and we are not interested
in having that as a build platform.  Drop the py35 jobs.

Update python setup.cfg metadata; we are building on 3.9 so add that.

Change-Id: I981f0f67a6fd809af1ab70934358dc3404890f35
2021-10-20 09:38:47 +11:00
Danni Shi
18df5a59b5 Update keylime-agent and tpm-emulator elements
Story: #2002713
Task: #41304

Change-Id: I735db46a62edecc85457d4163963f558c9fe461d
2021-10-06 15:05:28 -04:00
Zuul
6e83df4d40 Merge "Fix cron not installed in debian" 2021-10-06 13:23:00 +00:00
yatinkarel
8b4ad6dae6 Drop lower version requirement for networkx
With [1] to fix lower constraints job, networkx
requirement was bumped to 2.4.0, but since diskimage-builder
is branchless and to get it work with stable/train upper-constraints
for networkx i.e 2.3.0 can't bump to 2.4.0.
Also networkx-2.3.0 has dep on decorator>=4.3.0
so actual decorator issue with 3.4.0 shouldn't be there.

[1] https://review.opendev.org/c/openstack/diskimage-builder/+/811048
[2] https://github.com/networkx/networkx/blob/networkx-2.3/requirements/default.txt

Closes-Bug: #1945926
Change-Id: I93a2826019d9bb0d004f7daa661fdd1ca4269331
2021-10-05 12:55:43 +05:30
Zuul
82aa8c516d Merge "Add DIB_YUM_REPO_PACKAGE as an alternative to DIB_YUM_REPO_CONF" 2021-10-04 06:51:31 +00:00
Zuul
70378de688 Merge "Add policycoreutils package mappings for RHEL/Centos 9" 2021-10-04 06:51:29 +00:00
Zuul
0e93b853bf Merge "RHEL/Centos 9 does not have package grub2-efi-x64-modules" 2021-10-04 06:51:27 +00:00