This is a follow-on to I475a253091cbaf63687b91c748c31a6753bb0f57 as we
are still seeing issues on some clouds with unconfigured networking.
We increase the timeout, but also make it configurable so we can
fiddle it without a dib release in the gate.
To follow-on from the experimentation done by clarkb, I can confirm by
emperical testing on a Centos 7 image (from today, today being this
change's date) that setting
net.ipv6.conf.all.autoconf=0
by itself is "fatal" and the interfaces do not come up; i.e. nm does
not by default seem to re-enable ipv6 for the interface. However,
explicitly adding:
IPV6INIT=yes
IPV6_AUTOCONF=yes
to the interface file *does* seem to make it work, even if
"all.autoconf=0" is set (then again, there's also bugs about the
effect of this [1]). However, no extant distribution (I can currently
find) does anything like this by default.
If this continues, this may be an option. Another might be to avoid
the use of the nm-settings-ifcfg-rh profiles and move directly to nm
ini files with glean.
[1] https://bugzilla.kernel.org/show_bug.cgi?id=11655
Change-Id: I869ebffc8cde3bbff573f6583fd9dd02a5598590
Upstream is now publishing 17.1 profile systemd stages
Also updates the docs that were forgotten in the last patch
Change-Id: I0f2e7976845b1d3c55ffe8869eec0bc04a191252
As described inline, we need to ensure the underlying directories in
the image are correctly labeled, or we get all manner of services
failing during boot with selinux in enforcing mode. Although the
problem is generic, this first shows up in Fedora 30 as systemd has
become more strict about namespace failures (I think) [1].
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1663040#c22
Change-Id: I52c1cc719884879169b606b00651aa26f5b783f1
This patch adds option DIB_YUM_MINIMAL_EXTRA_REPOS to yum-minimal to
allow DIB users to include extra repositories to their final image.
Change-Id: I89549f4b0f4c9470143b5064817acab5043e31c5
Something (possibly [1], but that change is at best cryptic) has
changed such that we don't get correct /etc/os-release files
installed. This flows on to grub half-installing itself, enough to
not fail the build but not enough to make something bootable.
Installing the -cloud release package gets it back, and seems like a
sane choice for dib.
[1] 617b1bed34
Change-Id: Iff0413887fad798273b2bfcb140cc07f36d54a04
This seems to miss the exit code of the dracut process, which actualy
caused some issues in I8511669e188717494daf2bc1384a6dd346f942a4 where
it would have been much clearer to stop after the initramfs generation
failed.
Add some debug messages, and catch any errors from the final call.
Change-Id: I6f89441ec4709f5199535e15a7cc53a3a8af273d
Should install "grub2-efi-aa64 grub2-efi-aa64-modules" instead of
"grub2-efi grub2-efi-modules" for arm64
Change-Id: Iee3191b0944b3b862890d166a9d36bd592fe8f7e
Closes-bug: #1839816
- DIB_DISTRIBUTION_MIRROR_UBUNTU_IGNORE matches when it is empty or not
set and DIB_DISTRIBUTION_MIRROR is being used. Checking for it being
set and not empty solves this.
- Normalizing bash conditionals for readability
Closes-Bug: #1808359
Change-Id: I87853fcda4c8b29a3f1720a2778debeb3acc3a53
Signed-off-by: Manuel Torrinha <manuel.torrinha@tecnico.ulisboa.pt>
The 'pypi' mirror element is generating invalid pip.conf files
when more than one "DIB_PYPI_MIRROR_URL" is specified.
This patch fixes the pip.conf file rendering to use a proper form
when there are multiple "extra_index_url" provided.
Closes-Bug: #1839558
Change-Id: Ibda0e7390955683560e09b486f636775643ff57c
Per the inline comment, a machine-id is required for kernels to
install correctly (this may well be a bug, but the linked issue
remained inconclusive).
Add a call to make the machine-id before install packages.
Change-Id: If75d04376e62bfdfe14ee3ca4d0bd5c8b383c1b0
Redhat-Bug: https://bugzilla.redhat.com/show_bug.cgi?id=1737355
The 17.1 profile changed the defaults used in portage for where we store
our repo, distfiles and binpkgs. Some portage related variables need to
be set deterministically. 17.1 is no enabled for Systemd's profile.
Change-Id: Ib55f6875c5cb461c3c530b51d7420ce3dc8da360
This element configures systemd to send its journal to the console,
which can then be retreived by server commands. In the case of
nodepool, if the image failed to boot the console will be dumped into
the logs when nodepool decides the node is not responding. Having
this can be very helpful diagnosing early boot errors.
Needed-By: https://review.opendev.org/#/c/669787/
Change-Id: I6b6df7023acb6b2f967b84840bc4b542ebc03727
Newer versions of open-iscsi seem to compile on Gentoo / musl. Use them
if we can. This also removes the cap on open-iscsi.
Change-Id: I596cb61494e459a419bce6a63deff89f9e78fe23
Previously we were trying to enable dbus-daemon service on all prior to
fedora 30. Unfortunately 28 and older don't have this service so this
broke those releases and only worked for 29. Fix this by only enabling
this service on fedora 29.
Change-Id: I1bd15dcf0bbe270afccb0c0c3ea6ad08862a53f1
The linux kernel and NetworkManager fight each other over control for
interface management when router advertisements are in use. Long story
short if the linux kernel configures a network interface for ipv6
before NetworkManager attempts to manage that interface then NM will
ignore the interface and not configure ipv4 on it.
This can happen because the kernel is configured to send router
advertisements solicitations which result in router advertisements which
the kernel uses to configure the interface(s). There is a default of a 1
second delay before sending the solicitation which in many cases is long
enough that NM has started before then. However, in slower environments
like those used for testing with qemu this isn't long enough.
Some testing by hand indicates that 15 seconds is about right so
increase the delay to 15 seconds via sysctl.conf.
Note this may increase boot times in ipv6 only environments (though it
is hard to be sure due to how systemd starts everything at once and does
socket activation and the like).
Change-Id: I475a253091cbaf63687b91c748c31a6753bb0f57
When virtualenv and setuptools gots installed from source and rpm
then their installation path lives at different places but when
the python script got called then that time it choses either of
rpm or source based path on system wide installation and leads to
different failure as their methods are not implemeted.
So by setting _clear_old_files to 0 will install
python3-virtualenv python3-pip python3-setuptools from rpms only
and avoid these failures.
Change-Id: I0c162f1fe8168513e352546ab8dd2b68fa65b88c
Signed-off-by: Chandan Kumar (raukadah) <chkumar@redhat.com>
autounmask=y (default) changes portage depsolving, causing errors
(mostly often seen in perl and binpkg related issues).
Disabling this functionality for DIB builds is OK as the enviroment is
not passed on post build and the build process is not interactive
anyway.
Change-Id: Ife9ace246bec16864ee4982bc456763af5dff2e8
Signed-off-by: Matthew Thode <mthode@mthode.org>
debian-minimal depends on debootstrap which depends on dpkg
This needs to be installed early as dpkg installs the apt keys early via
02-add-apt-keys in pre-install.d
Change-Id: I8580849ceaa7a5152c94f29afa890ac6d6983fb1
This change removes useless statement accidentally added in the
06576a02f0
Change-Id: I7ea4a24d8c72c9e72f5f87247403af0f9bf69b40
Signed-off-by: Maksim Malchuk <maksim.malchuk@gmail.com>
There are several jobs depending on working opensuse 15.1
images in nodepool, so it makes sense to ensure its working.
Also upgrade the previously marked experimental opensuse-15.0
job as it tests the xenial->opensuse combination, which is
particularly difficult to keep working and we'll need it in
the CI.
Change-Id: Icb6d998756ce5221e017959dcb59b21f0f023454
This patch adds a new environment variable to the ubuntu-minimal
element called DIB_UBUNTU_KERNEL that allows you to specify the kernel
meta package that will be using to install the kernel inside the image.
It supports "linux-image-generic" (The default), "linux-image-kvm", and
"linux-image-virtual".
This allows building images that are smaller in size (~200MB smaller
qcow2) that have only the kernel modules necessary for virtual
machines.
Change-Id: I8ce65e1d357d15e8ed8995ad1dcaea02bbd1986f
1. Sync sphinx dependency with global requirements. It caps python 2 since
sphinx 2.0 no longer supports Python 2.7.
2. Update some URLs to latest
3. Remove the unnecessary space
Change-Id: I5464be9e055feecd80918f691448acf5f100e701
Use openSUSE 15.1 as default, which is the latest released stable
openSUSE release.
Remove leftovers for unmaintained openSUSE 42.2 images.
Depends-On: https://review.opendev.org/#/c/660126/
Change-Id: I0b204b7b3d7ae74b6749320b3bfe1ca89d154ebb
This patch removes the check and default for rhel 8 requiring
xfs filesystem as rhel 8 images can successfully be built with
ext4 filesystems.
Change-Id: I1a6bfa26324fd43ae0c77c2c977dda0dd56e26e5
Nowadays, in the time of Predictable Network Interface Names, the
network interface names 'ethX' are not used that often any more.
Depending on the virtualization layer and the guest OS names like
'ens3', 'enp1s0' or 'enp0s31f6' are used.
This patch enables the user to set DIB_NETWORK_INTERFACE_NAMES to a
list of network interfaces which are brought up using DHCP during
(first) boot.
Change-Id: I04cc2ee710f0389a8085b1c91d9329784cb28048
Signed-off-by: Andreas Florath <Andreas.Florath@telekom.de>
The latest Fedora/Ubuntu images don't ship python2 by default, so we
need to use our dib-python wrapper for this so we work in python3 only
environments.
This change also correctly creates the pip.conf and .pydistutils.cfg
files with trusted host extracted from the index-url.
Related-bug: 1577105
Change-Id: Ibb5348af3e3bbe46b19affe90a8930a4b4ad4cad
Signed-off-by: Maksim Malchuk <maksim.malchuk@gmail.com>
RHEL8 ships a bunch of grub2-efi-X-modules in its main
repository, each of which provides grub2-efi-modules,
potentially causing nondeterminism when building images.
This changes the DIB elements to always use architecture-
specific RPMs when RHEL8 is selected.
Change-Id: If94f3721195d5ecd80036e4234a3ca223a19c349
Related: https://bugzilla.redhat.com/show_bug.cgi?id=1716672
When I said in I8594d1fe05242f246a5809740a115ab2f84ac5a3 that 12 MiB
ought to be enough, I should have expected that I would be proven wrong.
While 12 MiB is enough to fit shim-x64 and grub2-efi-x64, yum fails to
update these packages to newer versions:
Transaction check error:
installing package shim-x64-15-2.el7.centos.x86_64 needs 7MB on the /boot/efi filesystem
installing package grub2-efi-x64-1:2.02-0.76.el7.centos.1.x86_64 needs 3MB on the /boot/efi filesystem
Error Summary
-------------
Disk Requirements:
At least 7MB more space needed on the /boot/efi filesystem.
It is recommended that the ESP partition be much bigger. This commit
bumps its size to 550MiB, following guidelines from Rod Smith to avoid
incompatibilities with some EFIs [1].
[1] https://www.rodsbooks.com/efi-bootloaders/principles.html
Change-Id: If9515234f1a803cda32b2482f8abe10ddf0e6d26
Avoids failing on the first attempt to download the image to cache as
mirrors hosting them can randomly go down, usually with a connection
refused.
Change-Id: I9de9f33c2cc16596d04b35c4eb92621e6a2c7511
When the mirror returns a error, it was trying to interpret the error
message (e.g. <html><title>Internal server error..) as a download link.
By using -f on curl we get an empty reply and an exit code, which, as
we run in set -e mode, aborts.
Change-Id: Ibaa39aedb7db286f859c4b090114c6a233b150c7
The rhel7 element is deprecated and is left only for backward
compatibility.
The rhel element should be used instead. Users should set DIB_RELEASE to
'7' to indicate which release you are using.
The new element is a version-less RHEL element to handle both '7'
and '8' DIB_RELEASE, which aligns with other elements which operate in
the same way such as the Fedora element.
Change-Id: Ic39ed85cacae9942448eb18ad685763f9369c2ed
Make a version-less RHEL element to handle both '7' and '8' DIB_RELEASE.
The element usage should align with other elements which operate in the
same way such as the Fedora element.
Additionally, this patch adds support for RHEL8 that operates with
Python 3.
As of now, users of diskimage-builder will still be able to use the
'rhel7' element, or migrate to 'rhel' and specify their respective
DIB_RELEASE value.
* mount the xfs file-system for extraction as read-only. vaguely
based on explaination in [1] and the fact we only read the image
data into a tar, so can ignore this.
XFS (dm-1): Superblock has unknown read-only compatible features (0x4) enabled.
* Use the redhat system python as the dib-python version. dib was
ahead of it's time making an abstracted python interpreter for
system work ;) the system python should work for running the various
dib element scripts.
[1] https://unix.stackexchange.com/questions/247550/unmountable-xfs-filesystem
Redhat-Bug: https://bugzilla.redhat.com/show_bug.cgi?id=1700253
Co-Authored-By: Ian Wienand <iwienand@redhat.com>
Change-Id: I90540675c70bb475d9db2ae24f81c648a31f3f95
Use openSUSE 15.0 as default, which is the latest released stable
openSUSE release. Switch to https for accessing download.o.org
as encrypted transfers should be used by default.
Remove leftovers for definitely unmaintained openSUSE 13.x images
and split into old/new leap style versioning scheme for clarity.
Change-Id: Iab129eeee2b1a2563f0f0d2cb17bbad57c068e38
It looks like fedora-release on fedora 30+ has been split into sub
packages. Use fedora-release-common to avoid package conflicts.
Change-Id: I8f8711044fc4074b91939e0a6dfdac4d7a14a35b
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
In fedora-30 is when we migrate to dbus-broker, fedora-29 is still using
dbus-daemon.
Change-Id: I1e1d3a3826157b8b22386c211eaa58b6439b5f3c
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
Depending on the version of $DIB_PYTHON_VERSION, we can either use pip /
pip3 to install glean. This is helpful for newer OSes that might not
want to ship python2 (pip).
Change-Id: I25c5927a1eb55ee16b919dd64403184f335839b6
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
Harden sshd configuration by adding KexAlgorithms, Ciphers and MACs for sshd,
following good pratices on https://infosec.mozilla.org/guidelines/openssh
Change-Id: I3051320d867a5033e82deef10c5e723ca9829884
Co-Authored-By: Nicolas Hicher <nhicher@redhat.com>
Due to the referenced bug, many versions of debootstrap can't bring up
a buster environment. Unfortunately, these include versions we use to
do this on Xenial/Bionic nodes.
Also, there isn't backports or security updates, so elide these for
now.
I did get a working build (I haven't gone so far as a full boot+glean)
with this, at least.
Change-Id: If2420e92cb728ab6e91b0d70547da4483679b391
Paritial-Bug: #1822927
Currently, the cleanup script is using the existence of the folder
/sys/fs/selinux to check if SELinux is enabled. This, however, is
misleading in case disk-image-builder is used inside a Docker
container on a selinux-enabled host. In this case, the folder exists
in the container but SELinux is disabled.
This patch addresses the problem by checking, in addition to the
check already in place, the output of the command selinuxenabled.
Change-Id: I83e58f2467e60df9f0f00f7b7a58d0e2ce357a9a
Closes-Bug: #1820077
This is a mechanically generated change to replace openstack.org
git:// URLs with https:// equivalents.
This is in aid of a planned future move of the git hosting
infrastructure to a self-hosted instance of gitea (https://gitea.io),
which does not support the git wire protocol at this stage.
This update should result in no functional change.
For more information see the thread at
http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003825.html
Change-Id: Id26bec14c3d94e2f81b2148fc85d17f07866398c
Add a DIB_APT_MINIMAL_CREATE_INTERFACES boolean to the debootstrap
element which functions identically to
DIB_YUM_MINIMAL_CREATE_INTERFACES in the yum-minimal element.
This can be used to disable the creation of the
/etc/network/interfaces.d/eth[01] dhcp configuration files, which
are not needed on systems where cloud-init or other means are used
to configure networking.
The flag is enabled by default to keep creating the dhcp interface
files, maintaining backwards compatibility.
Change-Id: I1fdaca8350a5ceefd9e437af4fd000ce6a3ee7f3
in same cases it is required to avoid update all existing packages,
doing so can result in release update which is currently not possible
unless you not include "base" element.
"base" element used for most distribution (rhel, debain), and is
necessary for most cloud operations, this patch add
"DIB_AVOID_PACKAGES_UPDATE" parameter to skip updating all packages.
usecases for this patch can be:
* Avoid release update when building old release ex. RHEL7.5.
* build on network-less environment.
usage:
DIB_AVOID_PACKAGES_UPDATE=1
or
DIB_AVOID_PACKAGES_UPDATE=0
Change-Id: I71192b23c8f0bc48b348fe7377bf8a2399b53792
Related to I041a141366099093805e6052b1bbf64efd277e1e, we also need to
remove this on opensuse. The files for gate testing are added, but
the test is not added to any jobs at this point in the interests of
gate time.
Change-Id: I1af9e84d76bedcb2607717edc6d2abe2920b0584
This fixes a regression in I041a141366099093805e6052b1bbf64efd277e1e
where we starting skipping the removal of old files for image-based
builds (confusingly named centos7 rather than centos for historical
reasons). Fix the check
Change-Id: I74688a9e91d833b5d654056431729bed0585616c
As described inline, we only want to remove the system package files
on centos; it causes problems on Fedora where some system tools expect
these to be there.
But there is an additional bug -- pip actually removes the system
package files anyway. To work around this, reinstall the system
package.
Closes-Bug: #1813232
Change-Id: I041a141366099093805e6052b1bbf64efd277e1e
As described in the comments, it seems the transition between
dbus-daemon -> dbus-broker in Fedora 29 has made it so the packages
can get into a state where neither service is enabled.
Explicitly install and enable dbus-broker for F29
Change-Id: I06753043a75be2f635653899c6c251b9fbdd7c67
