The vhdutil utility is completely dead; the whole subsystem it relies
on was removed with [1] so it's not even vaguely possible to keep it
up-to-date.
I took the .raw images on a nb and used the qemu-img there (so Xenial)
and generated some VPC images; uploaded them to rackspace and the all
seemed to boot fine. If there was a problem, maybe it's been fixed on
either the qemu or RAX side in the previous few years.
Thus swith to qemu-img to generate the vhd images too.
[1] https://xenbits.xen.org/gitweb/?p=xen.git;a=commit;h=5c883cf036cf5ab8b1b79390549e2475f7a568dd
Change-Id: I3099d2ebb958370fcec623087a093b2c8dbdc6c4
Add option to set the suite subpath after the release name for the
security mirror URL independently in the debian-minimal element,
since this can differ between mirrors.
Change-Id: I4cc8f54fba012986423e30e19bff276208b8ad62
It turns out that this breaks ipv6 config with NM. Instead what we want
is for glean to not up interfaces on boot (see the depends-on).
Change-Id: I6c5bc76c433e29f02d3266ab8f669015125ec954
Depends-On: https://review.opendev.org/#/c/688031
This adds CentOS 8 into functional and boot tests.
This completes centos-minimal support, documentation is updated and a
release note is added.
Change-Id: I435c2967b4f49faeb6d6edf189907b9f96e80357
NetworkManager with simple-init has proven to be stable in OpenStack
infra, switch to it by default for CentOS and Fedora. For CentOS 8
and Fedora, add a check to make it the only option. Thus only CenOS 7
remains optionally using the legacy scripts; this is likely not used
anywhere (infra is really the primary user, where NetworkManager is
already used); we can likely remove this variable (and hence path) in
a future cleanup.
In the setup, remove rhel7 element which was never really tested.
Reorganise the fallthrough to call out the default paths as doing
nothing.
Change-Id: Ic996956da4b85f7d95179b8df9881d5f52c091af
Add option to set the security mirror URL independently in the
debian-minimal element, since this can not be overriden by the
standard DIB_DISTRIBUTION_MIRROR variable.
Change-Id: I145844a410d06a479e68db1bf6d5d0159389305c
As described inline, deprecate the "source" install for CentOS 8.
Overwriting the packaged tools has long been a pain-point in our
images, and the best outcome is just not to play the game [1].
However, the landscape remains complicated. For example, RHEL/CentOS
8 introduces the separate "platform-python" binary, which seems like
the right tool to install platform tools like "glean" (simple-init)
with. However, platform-python doesn't have virtualenv (only the
inbuilt venv).
So that every element doesn't have to hard-code in workarounds for
these various layouts, create two new variables DIB_PYTHON_PIP and
DIB_PYTHON_VIRTUALENV to just "do the right thing". If you need is
"install a pip package" or "create a virtualenv" this should work on
all the platforms we support. If you know more specifically what you
want (e.g. must be a python3 virtualenv) then nothing stops elements
calling that directly (e.g. python3 -m virtualenv create); these are
just helper wrappers for base elements that need to be broadly
compatible.
[1] http://lists.openstack.org/pipermail/openstack-infra/2019-September/006483.html
Change-Id: Ia267a60eecfa8f4071dd477d86daebe07e9a7e38
Two bugs are addressed.
1) The sysprep element was broken in that it only truncates
/etc/machine-id, but not /var/lib/dbus/machine-id. systemd will
not generate a new machine-id if /var/lib/dbus/machine-id is
present[1], it will simply copy it to /etc/machine-id.
We observed machine-ids being packaged in /var/lib/dbus/machine-id
on several distros: Ubuntu Bionic, Fedora 29, Debian Stretch.
CentOS 7 and Ubuntu Xenial do not contain packaged machine-id as
far as I can tell.
All test builds were performed using -minimal elements.
2) A second bug existed where debian-minimal did not run the sysprep
element at all, so a stretch image I tested contained a populated
/etc/machine-id AND a populated /var/lib/dbus/machine-id.
[1] https://www.freedesktop.org/software/systemd/man/machine-id.html#Initialization
Change-Id: Ibb28b6e90d966a845de38a2cd5a1e8babd2604bc
linux-firmware and linux-firmware-whence (meta package for mostly iwl
firmwares) packages account for approx. 289 M install size on a F30
system, and linux-firmware for approx. 176 M on CentOS 7. Users needing
these firmwares are eventually baremetal users and are not looking for a
very minimal operating system base install like virtual image users are.
Thus, a non-minimal OS element is better suited for them. Alternatively,
it could be later considered a dedicated firmware element.
This is inline with I8ce65e1d357d15e8ed8995ad1dcaea02bbd1986f.
Change-Id: If104fc3c1e9349b8d501a2351fff1ab4c0dbc6a4
Add a new environment variable $DIB_GZIP_BIN allowing builders to
specify a different gzip (such as pigz) to be used when compressing
tgz images.
Change-Id: Ifb617568140a149e2fda241e07ff8a59429e6697
This is a follow-on to I475a253091cbaf63687b91c748c31a6753bb0f57 as we
are still seeing issues on some clouds with unconfigured networking.
We increase the timeout, but also make it configurable so we can
fiddle it without a dib release in the gate.
To follow-on from the experimentation done by clarkb, I can confirm by
emperical testing on a Centos 7 image (from today, today being this
change's date) that setting
net.ipv6.conf.all.autoconf=0
by itself is "fatal" and the interfaces do not come up; i.e. nm does
not by default seem to re-enable ipv6 for the interface. However,
explicitly adding:
IPV6INIT=yes
IPV6_AUTOCONF=yes
to the interface file *does* seem to make it work, even if
"all.autoconf=0" is set (then again, there's also bugs about the
effect of this [1]). However, no extant distribution (I can currently
find) does anything like this by default.
If this continues, this may be an option. Another might be to avoid
the use of the nm-settings-ifcfg-rh profiles and move directly to nm
ini files with glean.
[1] https://bugzilla.kernel.org/show_bug.cgi?id=11655
Change-Id: I869ebffc8cde3bbff573f6583fd9dd02a5598590
As noted in the change, 7fd52ba841
increased the size of the EFI partition considerably. This has meant
that our padding upwards of the disk size is insufficient and EFI
builds (arm64 in particular) is failing due to out-of-disk errors
during final image operations like installing kernels.
Similar to the discussion we had in
I65fa13a088eecdfe61636678578577ea2cfb3c0c, this feels a bit ugly
because we're mixing logic here with sizes specified in block-device
config files. But it boils down to the same problem; we are
calculating the disk size here and passing it to the block-layer, so
unless we want to make large changes to the status quo about where
these sizes are calculated, small adjustments here are the most KISS
solution.
Thus we check if we have selected the EFI bootloader element, and thus
assume there will be a large system EFI partition and expand the disk
size accordingly.
Change-Id: Ifa05366c2f2b95259f3312e4dde8c85347075ba1
This element configures systemd to send its journal to the console,
which can then be retreived by server commands. In the case of
nodepool, if the image failed to boot the console will be dumped into
the logs when nodepool decides the node is not responding. Having
this can be very helpful diagnosing early boot errors.
Needed-By: https://review.opendev.org/#/c/669787/
Change-Id: I6b6df7023acb6b2f967b84840bc4b542ebc03727
This patch adds a new environment variable to the ubuntu-minimal
element called DIB_UBUNTU_KERNEL that allows you to specify the kernel
meta package that will be using to install the kernel inside the image.
It supports "linux-image-generic" (The default), "linux-image-kvm", and
"linux-image-virtual".
This allows building images that are smaller in size (~200MB smaller
qcow2) that have only the kernel modules necessary for virtual
machines.
Change-Id: I8ce65e1d357d15e8ed8995ad1dcaea02bbd1986f
Use openSUSE 15.1 as default, which is the latest released stable
openSUSE release.
Remove leftovers for unmaintained openSUSE 42.2 images.
Depends-On: https://review.opendev.org/#/c/660126/
Change-Id: I0b204b7b3d7ae74b6749320b3bfe1ca89d154ebb
When I said in I8594d1fe05242f246a5809740a115ab2f84ac5a3 that 12 MiB
ought to be enough, I should have expected that I would be proven wrong.
While 12 MiB is enough to fit shim-x64 and grub2-efi-x64, yum fails to
update these packages to newer versions:
Transaction check error:
installing package shim-x64-15-2.el7.centos.x86_64 needs 7MB on the /boot/efi filesystem
installing package grub2-efi-x64-1:2.02-0.76.el7.centos.1.x86_64 needs 3MB on the /boot/efi filesystem
Error Summary
-------------
Disk Requirements:
At least 7MB more space needed on the /boot/efi filesystem.
It is recommended that the ESP partition be much bigger. This commit
bumps its size to 550MiB, following guidelines from Rod Smith to avoid
incompatibilities with some EFIs [1].
[1] https://www.rodsbooks.com/efi-bootloaders/principles.html
Change-Id: If9515234f1a803cda32b2482f8abe10ddf0e6d26
The rhel7 element is deprecated and is left only for backward
compatibility.
The rhel element should be used instead. Users should set DIB_RELEASE to
'7' to indicate which release you are using.
The new element is a version-less RHEL element to handle both '7'
and '8' DIB_RELEASE, which aligns with other elements which operate in
the same way such as the Fedora element.
Change-Id: Ic39ed85cacae9942448eb18ad685763f9369c2ed
Make a version-less RHEL element to handle both '7' and '8' DIB_RELEASE.
The element usage should align with other elements which operate in the
same way such as the Fedora element.
Additionally, this patch adds support for RHEL8 that operates with
Python 3.
As of now, users of diskimage-builder will still be able to use the
'rhel7' element, or migrate to 'rhel' and specify their respective
DIB_RELEASE value.
* mount the xfs file-system for extraction as read-only. vaguely
based on explaination in [1] and the fact we only read the image
data into a tar, so can ignore this.
XFS (dm-1): Superblock has unknown read-only compatible features (0x4) enabled.
* Use the redhat system python as the dib-python version. dib was
ahead of it's time making an abstracted python interpreter for
system work ;) the system python should work for running the various
dib element scripts.
[1] https://unix.stackexchange.com/questions/247550/unmountable-xfs-filesystem
Redhat-Bug: https://bugzilla.redhat.com/show_bug.cgi?id=1700253
Co-Authored-By: Ian Wienand <iwienand@redhat.com>
Change-Id: I90540675c70bb475d9db2ae24f81c648a31f3f95
I want to use the new --image-extra-size flag[1] but my use-case
calls for megabyte granularity of this value. Rather than adding
60% to an 800MB image, maybe I only want to add 100 or 200MB, etc.
[1] https://review.opendev.org/#/c/655127/
Change-Id: I8fb9685d60ebb1260d5efcf03c5c23c561c24384
Use openSUSE 15.0 as default, which is the latest released stable
openSUSE release. Switch to https for accessing download.o.org
as encrypted transfers should be used by default.
Remove leftovers for definitely unmaintained openSUSE 13.x images
and split into old/new leap style versioning scheme for clarity.
Change-Id: Iab129eeee2b1a2563f0f0d2cb17bbad57c068e38
Harden sshd configuration by adding KexAlgorithms, Ciphers and MACs for sshd,
following good pratices on https://infosec.mozilla.org/guidelines/openssh
Change-Id: I3051320d867a5033e82deef10c5e723ca9829884
Co-Authored-By: Nicolas Hicher <nhicher@redhat.com>
Currently diskimage-builder supports two ways to specify the image
size. One is defining a fixed image size using DIB_IMAGE_SIZE, the
other one is auto-detection while adding a security margin of 60% as
free space. This means when building larger images (e.g. >100GB) with
unknown size upfront we end up with much wasted space, IO and network
traffic when uploading the images to several cloud providers. This can
be optimized by adding a third way by defining DIB_IMAGE_EXTRA_SIZE to
specify the free space in GB. This makes it possible to easily build
images of varying sizes while still minimizing the overhead by keeping
the free space constant to e.g. 1GB.
Change-Id: I114c739d11d0cfe3b8d8abc6df5ff989edfb67f2
In many cases, the statically sized 64MB journal is far below the
e2fstools default calculation[0] which calls for a 64MB journal only
on filesystems smaller than 16GB. On bare metal in particular, the
correct default journal size will often be in the 512MB-1GB range.
Since we cannot know what the target system is, this should be a
tunable parameter that the user can set depending on the intended
image usage.
Add a DIB_JOURNAL_SIZE envvar and --mkfs-journal-size parameter
to the image creation so users can override the default journal
size.
[0] https://git.kernel.org/pub/scm/fs/ext2/e2fsprogs.git/tree/lib/ext2fs/mkjournal.c#n333
Change-Id: I65fa13a088eecdfe61636678578577ea2cfb3c0c
This plumbs through an "--use-nm" flag to glean which instructs it to
setup interface bringup with NetworkManager rather than legacy network
enablement scripts.
In this case, install the NetworkManager package. In the non-nm case,
also install the network-scripts for Fedora 29 -- this has stopped
being installed by default (it's been deprecated since forever).
As noted in the docs, this is currently really only relevant on the
supported rpm distros which are using the ifcfg-rh NetworkManager
plugin to effectively re-use old config files. However,
NetworkManager has similar plugins for other platforms, so support can
be expanded if changes are proposed.
Depends-On: https://review.openstack.org/618964
Change-Id: I4d76e88ce25e5675fd5ef48924acd09915a62a4b
Provide a "when" option that provides for not installing packages
based on a = or != match on an environment variable.
Unit tests are added.
Change-Id: Ifa824dccaff69fd447f45d54cb4a3083bcabdd86
This allows nodes with remote devices configured via iBFT to be
correctly used during Ironic introspection and deployment,
at least for non-multipath configurations.
The new element is added as a dependency for ironic-agent.
Change-Id: If3dac6504d26535593f12e851092065b688ef696
install-packages is running before install.d phase, there is a chance
that installing a package like "container-selinux" will failed the
build, moving "selinux-permissive" to run at pre-install stage make
more sense.
Change-Id: I32f988be725d4b385c3765c47a00cd57c53d7d71
I'm not really sure why I originally had --logfile also log to stdout
in I202e1cb200bde17f6d7770cf1e2710bbf4cca64c, but it seem
counter-intuitive (indeed, I just tripped myself up thinking that in a
devstack job "--logfile" would put the logs into a separate file and
avoid the stdout logging, and I wrote it!).
Make it so specifying a --logfile puts dib into quiet mode for stdout.
Explicitly overriding DIB_QUIET will allow both if someone wants that.
Change-Id: I3279c9253eee1c9db69c958b87a0ce73efc0be9b
While trying to get docker image pre-caching to work we couldn't get a
docker daeomon to run within the chrooted environment. However we got
docker running with the help of bwrap outside of the chrooted
environment. The only option so far for this is the block-device.d
phase. But this has the problem that it runs after the image size has
been calculated. This leads to broken builds if the docker images
being pulled are big.
This can be solved by adding a post-root.d phase that runs outside the
chroot but before the image size calculation.
Change-Id: I36c2a81e2d9f5069f18ce5b0d52c5f1c7212c3ae
In exploring Gentoo caching, it was realised that we have no way to
bind mount the cache into the finalised image for the finalise.d
phases.
By adding a pre-finalise.d phase that runs outside the chroot, we can
mount outside things into the hierarchy at $TMP_BUILD_DIR/mnt which
are then seen by the in-chroot finalise.d phase.
This is similar to the pre-install phase
Change-Id: I9d782994843383ddf90f62c40498af9925fd9558
The grub.cfg has two variables [1]
GRUB_CMDLINE_LINUX : used on all boots
GRUB_CMDLINE_LINUX_DEFAULT : additionally used on all "normal" boots
The problem with I2298675dda1f699c572b3423e7274bc8bd7c1c9d is that it
appened the values in DIB_BOOTLOADER_DEFAULT_CMDLINE to both of these,
resulting in duplicated arguments. I don't think we considered that
GRUB_CMDLINE_LINUX_DEFAULT actually already appends to the
GRUB_CMDLINE_LINUX values.
Make DIB_BOOTLOADER_DEFAULT_CMDLINE only append itself to
GRUB_CMDLINE_LINUX_DEFAULT. That seems to line up sensibly with the
name of the variable.
Documentation is enhanced around this, and a releasenote added.
[1] https://help.ubuntu.com/community/Grub2/Setup
Change-Id: I76b5442a9090c19a6540ed2d4ab324546f241ebf
Closes: #1791736
This element will replace modprobe-blacklist element. It wil
still have the blacklist functionality, but it also adds
the feature of passing a complete file with settings to the
modprobe.d directory. Adding this functionality, that will
allow elements that depends on this module, to just copy the
specified files to the final directory.
Change-Id: I9a44f7d11520b8b1e604956d3c1db2fc7e2bf457
