This is a squash of two changes that have unfortunately simultaneously
broken the gate.
The functests are failing with
sha256sum: bionic-server-cloudimg-amd64.squashfs.manifest: No such file or directory
I think what has happened here is that the SHA256 sums file being used
has got a new entry "bionic-server-cloudimg-amd64.squashfs.manifest"
which is showing up in a grep for
"bionic-server-cloudimg-amd64.squashfs". sha256 then tries to also
check this hash, and has started failing.
To avoid this, add an EOL marker to the grep so it only matches the
exact filename.
Change I7fb585bc5ccc52803eea107e76dddf5e9fde8646 updated the
containerfile tests to Jammy and it seems that cgroups v2 prevents
podman running inside docker [1]. While we investigate, move this
testing back to focal.
[1] https://github.com/containers/podman/issues/14884
Change-Id: I1af9f5599168aadc1e7fcdfae281935e6211a597
This adds arm64 ubuntu-minimal Jammy functests and x86 ubuntu image
based Jammy functests. To make this happen we have to install
debootstrap from debian unstable on the functest nodes in order to get
access to a debootstrap that knows what jammy is.
As we ramp up Jammy support in our tools having good testing will be
helpful.
Change-Id: I1d1dc752ce176457d0656cbd50e27a2721ca9856
Patch allow to set path for local image source,
instead download latest or use the cached image.
This permit to build image also in environment without internet access.
re-propose of patch: https://review.opendev.org/c/openstack/diskimage-builder/+/809009
Change-Id: I54395b09af339caee040326b809e8fbf8b0e7d6a
This reverts commit 1f4fb1d7a5.
This unfortunately wasn't actually tested. Because the image-based
tests run sequentially, a prior failure in the centos-8 job meant the
ubuntu job never ran.
This is failing with
10-cache-ubuntu-tarball: line 28: DIB_LOCAL_IMAGE: unbound variable
There is also a seemingly unused variable DIB_IMAGE_LOCAL_FILE; I'm
not sure what this is doing.
For now revert, and it can be re-proposed with appropriate testing.
Change-Id: I0f3897c90dc863ee04c3295b9cb094f02d8658e3
Patch allow to set path for local image source,
instead download latest or use the cached image.
This permit to build image also in environment without internet access.
Change-Id: I9422e21c5d0445e31d5a7258aa7310b20e39b929
Autoremove can touch packages, that are assumed to be present
for post-install step.
A good example is dkms, which is dropped in
install.d, but dkms element try to use it in post-install step.
This results in the following log [1]
[1] http://paste.openstack.org/show/797809/
Change-Id: I635af230c6b250fee273039935cf19506e83b3d1
- DIB_DISTRIBUTION_MIRROR_UBUNTU_IGNORE matches when it is empty or not
set and DIB_DISTRIBUTION_MIRROR is being used. Checking for it being
set and not empty solves this.
- Normalizing bash conditionals for readability
Closes-Bug: #1808359
Change-Id: I87853fcda4c8b29a3f1720a2778debeb3acc3a53
Signed-off-by: Manuel Torrinha <manuel.torrinha@tecnico.ulisboa.pt>
When using the upstream cloud images with the "ubuntu" element, they
have universe and multiverse enabled which we don't mirror.
To use the infra mirrors as a DIB_DISTRIBUTION_MIRROR with this
element, we need to be able to skip redirecting to universe and
multiverse, and additionally enable insecure repos (as we don't gpg
sign our mirrors).
Add and document two new variables with the ubuntu element to do this.
This is then setup by the openstack-ci-mirrors element so that we use
local mirrors duing dib functional testing for the "ubuntu" element.
Change-Id: I6ffbde07fa0e103641ee5c5f9d9e854e5b2168dc
The ubuntu, and ubuntu-minimal elements both make use of a common set
of environment settings to determine the distribution name.
The ubuntu-minimal element also does a few extra things which would
appear to apply to both sets and bring in extra architecture support.
As such, these are included in the common element.
This intends to be part of a series of patches which will eventually
create a new element to build a minimal ubuntu-systemd-container
element which can be used for lxc/nspawn containers.
Change-Id: Ia4e620f7d3fa6215484a8d218cea2f28bd1ffaee
Patch allows to rebuild arbitrary images, which location, filename and
sha256sum are specified in variables, not only hardcoded $DIB_RELEASE/current.
Change-Id: I05418932a0c40d885fe00a49f1f49d7e86c67518
This moves the block-device default out of the "vm" element and into a
selection of other elements. There's "mbr" which retains the status
quo. There's an EFI version that has the boot/grub partitions as
required. In between there's the GPT only version, which is useful
for architectures like power without EFI, but still want possible
larger disks using GPT.
Change-Id: I4a566a97d073fc0dda0ab2494ac988fe015800a9
In a couple of places we use flock for critical sections, but we leave
lockfiles around in various locations which can be confusing.
Introduce DIB_LOCKFILES global (under ~/.cache/dib/lockfiles) and
write lockfiles in there.
Fix up removal of the lockfile in the yum path; we just want to make
sure we cleanup the .rpmmacros file, but we don't need to remove the
lockfile as well.
Co-Authored-By: Andreas Florath <andreas@florath.net>
Change-Id: Ie810b2836be521325afe923708d046112e1e1e20
The image download tests have long been too unreliable for the gate.
We need to cache the base images similar to how devstack caches it's
testing images. Let's move them to non-voting jobs for the time
being.
This means that the gate jobs are now all based on "-minimal" and are
using infra mirrors. Unfortunately, there is still some unreliability
because we currently have issues with infra mirrors being very slow
after AFS updates, leading to job timeouts. But we're on the right
path...
Also, I noticed we don't have tests of the "ubuntu" image-download
based tests, which were tacitly being tested by apt-sources before we
moved that to -minimal. Add simple tests for these.
Change-Id: Ie33ee49656872467ef68d753210032156bb6b2cb
This patch removes three nearly-copies of debootstrap documentation
and fixes some documentation aspects.
Change-Id: Ief7794f5c1abad73788c063af6c862472cd34744
Signed-off-by: Andreas Florath <andreas@florath.net>
The dib-run-parts element was copying our internal version of
dib-run-parts into /usr/local/bin to be used running scripts inside
the target chroot. However, it never cleaned up after itself. This
means all images were left with an unmanaged local install of
dib-run-parts.
This copies dib-run-parts into the hooks directory of the chroot and
runs it from there. It is cleaned up automatically on the exit path.
The dib-run-parts element is no longer required and it has been
removed from all dependencies. It is left with a deprecation notice
in the README. For compatability we convert it to simply install
dib-utils.
Codesearch shows no users depending on this unintentional implicit
install. Note os-refresh-config depends on dib-utils and thus will
have an explicitly installed version.
Partial-Bug: #1673144
Change-Id: Ia2e96c00a4246c04beb96c17f83b8aefb69219ca
Currently there is no description of dependencies in the generated
documentation of the elements: therefore a user of an element does not
know which other elements are automatically included and e.g. which
configuration options are available. In addition there are some
copy&pastes of parts of the README.rst scattered thought different
Ubuntu and Debian specific elements.
This patch adds a semi-automatic generation of dependency information
of all elements. Nevertheless these are not automatically included.
The author of the element's README.rst can decide if and where the
dependency information should appear and can use the descriptor
.. element_deps::
for this.
This patch adds the dependency information for some Debian and
Ubuntu patches - and creates the base for later removing the
duplicated parts.
A call is added to element_dependencies._find_all_elements() to
populate reverse dependencies for Element objects.
(This is a reworking of I31d2b6050b6c46fefe37378698e9a330025db430 for
the feature/v2 branch)
Change-Id: Iebb83916fed71565071246baa550849eef40560b
Currently we have all our elements and library files in a top-level
directory and install them into
<root>/share/diskimage-builder/[elements|lib] (where root is either /
or the root of a virtualenv).
The problem with this is that editable/development installs (pip -e)
do *not* install data_files. Thus we have no canonical location to
look for elements -- leading to the various odd things we do such as a
whole bunch of guessing at the top of disk-image-create and having a
special test-loader in tests/test_elements.py so we can run python
unit tests on those elements that have it.
data_files is really the wrong thing to use for what are essentially
assets of the program. data_files install works well for things like
config-files, init.d files or dropping documentation files.
By moving the elements under the diskimage_builder package, we always
know where they are relative to where we import from. In fact,
pkg_resources has an api for this which we wrap in the new
diskimage_builder/paths.py helper [1].
We use this helper to find the correct path in the couple of places we
need to find the base-elements dir, and for the paths to import the
library shell functions.
Elements such as svc-map and pkg-map include python unit-tests, which
we do not need tests/test_elements.py to special-case load any more.
They just get found automatically by the normal subunit loader.
I have a follow-on change (I69ca3d26fede0506a6353c077c69f735c8d84d28)
to move disk-image-create to a regular python entry-point.
Unfortunately, this has to move to work with setuptools. You'd think
a symlink under diskimage_builder/[elements|lib] would work, but it
doesn't.
[1] this API handles stuff like getting files out of .zip archive
modules, which we don't do. Essentially for us it's returning
__file__.
Change-Id: I5e3e3c97f385b1a4ff2031a161a55b231895df5b
