autounmask=y (default) changes portage depsolving, causing errors
(mostly often seen in perl and binpkg related issues).
Disabling this functionality for DIB builds is OK as the enviroment is
not passed on post build and the build process is not interactive
anyway.
Change-Id: Ife9ace246bec16864ee4982bc456763af5dff2e8
Signed-off-by: Matthew Thode <mthode@mthode.org>
debian-minimal depends on debootstrap which depends on dpkg
This needs to be installed early as dpkg installs the apt keys early via
02-add-apt-keys in pre-install.d
Change-Id: I8580849ceaa7a5152c94f29afa890ac6d6983fb1
This change removes useless statement accidentally added in the
06576a02f0
Change-Id: I7ea4a24d8c72c9e72f5f87247403af0f9bf69b40
Signed-off-by: Maksim Malchuk <maksim.malchuk@gmail.com>
There are several jobs depending on working opensuse 15.1
images in nodepool, so it makes sense to ensure its working.
Also upgrade the previously marked experimental opensuse-15.0
job as it tests the xenial->opensuse combination, which is
particularly difficult to keep working and we'll need it in
the CI.
Change-Id: Icb6d998756ce5221e017959dcb59b21f0f023454
This patch adds a new environment variable to the ubuntu-minimal
element called DIB_UBUNTU_KERNEL that allows you to specify the kernel
meta package that will be using to install the kernel inside the image.
It supports "linux-image-generic" (The default), "linux-image-kvm", and
"linux-image-virtual".
This allows building images that are smaller in size (~200MB smaller
qcow2) that have only the kernel modules necessary for virtual
machines.
Change-Id: I8ce65e1d357d15e8ed8995ad1dcaea02bbd1986f
1. Sync sphinx dependency with global requirements. It caps python 2 since
sphinx 2.0 no longer supports Python 2.7.
2. Update some URLs to latest
3. Remove the unnecessary space
Change-Id: I5464be9e055feecd80918f691448acf5f100e701
Use openSUSE 15.1 as default, which is the latest released stable
openSUSE release.
Remove leftovers for unmaintained openSUSE 42.2 images.
Depends-On: https://review.opendev.org/#/c/660126/
Change-Id: I0b204b7b3d7ae74b6749320b3bfe1ca89d154ebb
This patch removes the check and default for rhel 8 requiring
xfs filesystem as rhel 8 images can successfully be built with
ext4 filesystems.
Change-Id: I1a6bfa26324fd43ae0c77c2c977dda0dd56e26e5
Nowadays, in the time of Predictable Network Interface Names, the
network interface names 'ethX' are not used that often any more.
Depending on the virtualization layer and the guest OS names like
'ens3', 'enp1s0' or 'enp0s31f6' are used.
This patch enables the user to set DIB_NETWORK_INTERFACE_NAMES to a
list of network interfaces which are brought up using DHCP during
(first) boot.
Change-Id: I04cc2ee710f0389a8085b1c91d9329784cb28048
Signed-off-by: Andreas Florath <Andreas.Florath@telekom.de>
The latest Fedora/Ubuntu images don't ship python2 by default, so we
need to use our dib-python wrapper for this so we work in python3 only
environments.
This change also correctly creates the pip.conf and .pydistutils.cfg
files with trusted host extracted from the index-url.
Related-bug: 1577105
Change-Id: Ibb5348af3e3bbe46b19affe90a8930a4b4ad4cad
Signed-off-by: Maksim Malchuk <maksim.malchuk@gmail.com>
RHEL8 ships a bunch of grub2-efi-X-modules in its main
repository, each of which provides grub2-efi-modules,
potentially causing nondeterminism when building images.
This changes the DIB elements to always use architecture-
specific RPMs when RHEL8 is selected.
Change-Id: If94f3721195d5ecd80036e4234a3ca223a19c349
Related: https://bugzilla.redhat.com/show_bug.cgi?id=1716672
When I said in I8594d1fe05242f246a5809740a115ab2f84ac5a3 that 12 MiB
ought to be enough, I should have expected that I would be proven wrong.
While 12 MiB is enough to fit shim-x64 and grub2-efi-x64, yum fails to
update these packages to newer versions:
Transaction check error:
installing package shim-x64-15-2.el7.centos.x86_64 needs 7MB on the /boot/efi filesystem
installing package grub2-efi-x64-1:2.02-0.76.el7.centos.1.x86_64 needs 3MB on the /boot/efi filesystem
Error Summary
-------------
Disk Requirements:
At least 7MB more space needed on the /boot/efi filesystem.
It is recommended that the ESP partition be much bigger. This commit
bumps its size to 550MiB, following guidelines from Rod Smith to avoid
incompatibilities with some EFIs [1].
[1] https://www.rodsbooks.com/efi-bootloaders/principles.html
Change-Id: If9515234f1a803cda32b2482f8abe10ddf0e6d26
Avoids failing on the first attempt to download the image to cache as
mirrors hosting them can randomly go down, usually with a connection
refused.
Change-Id: I9de9f33c2cc16596d04b35c4eb92621e6a2c7511
When the mirror returns a error, it was trying to interpret the error
message (e.g. <html><title>Internal server error..) as a download link.
By using -f on curl we get an empty reply and an exit code, which, as
we run in set -e mode, aborts.
Change-Id: Ibaa39aedb7db286f859c4b090114c6a233b150c7
The rhel7 element is deprecated and is left only for backward
compatibility.
The rhel element should be used instead. Users should set DIB_RELEASE to
'7' to indicate which release you are using.
The new element is a version-less RHEL element to handle both '7'
and '8' DIB_RELEASE, which aligns with other elements which operate in
the same way such as the Fedora element.
Change-Id: Ic39ed85cacae9942448eb18ad685763f9369c2ed
Make a version-less RHEL element to handle both '7' and '8' DIB_RELEASE.
The element usage should align with other elements which operate in the
same way such as the Fedora element.
Additionally, this patch adds support for RHEL8 that operates with
Python 3.
As of now, users of diskimage-builder will still be able to use the
'rhel7' element, or migrate to 'rhel' and specify their respective
DIB_RELEASE value.
* mount the xfs file-system for extraction as read-only. vaguely
based on explaination in [1] and the fact we only read the image
data into a tar, so can ignore this.
XFS (dm-1): Superblock has unknown read-only compatible features (0x4) enabled.
* Use the redhat system python as the dib-python version. dib was
ahead of it's time making an abstracted python interpreter for
system work ;) the system python should work for running the various
dib element scripts.
[1] https://unix.stackexchange.com/questions/247550/unmountable-xfs-filesystem
Redhat-Bug: https://bugzilla.redhat.com/show_bug.cgi?id=1700253
Co-Authored-By: Ian Wienand <iwienand@redhat.com>
Change-Id: I90540675c70bb475d9db2ae24f81c648a31f3f95
I want to use the new --image-extra-size flag[1] but my use-case
calls for megabyte granularity of this value. Rather than adding
60% to an 800MB image, maybe I only want to add 100 or 200MB, etc.
[1] https://review.opendev.org/#/c/655127/
Change-Id: I8fb9685d60ebb1260d5efcf03c5c23c561c24384
Use openSUSE 15.0 as default, which is the latest released stable
openSUSE release. Switch to https for accessing download.o.org
as encrypted transfers should be used by default.
Remove leftovers for definitely unmaintained openSUSE 13.x images
and split into old/new leap style versioning scheme for clarity.
Change-Id: Iab129eeee2b1a2563f0f0d2cb17bbad57c068e38
It looks like fedora-release on fedora 30+ has been split into sub
packages. Use fedora-release-common to avoid package conflicts.
Change-Id: I8f8711044fc4074b91939e0a6dfdac4d7a14a35b
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
In fedora-30 is when we migrate to dbus-broker, fedora-29 is still using
dbus-daemon.
Change-Id: I1e1d3a3826157b8b22386c211eaa58b6439b5f3c
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
Depending on the version of $DIB_PYTHON_VERSION, we can either use pip /
pip3 to install glean. This is helpful for newer OSes that might not
want to ship python2 (pip).
Change-Id: I25c5927a1eb55ee16b919dd64403184f335839b6
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
Harden sshd configuration by adding KexAlgorithms, Ciphers and MACs for sshd,
following good pratices on https://infosec.mozilla.org/guidelines/openssh
Change-Id: I3051320d867a5033e82deef10c5e723ca9829884
Co-Authored-By: Nicolas Hicher <nhicher@redhat.com>
Currently diskimage-builder supports two ways to specify the image
size. One is defining a fixed image size using DIB_IMAGE_SIZE, the
other one is auto-detection while adding a security margin of 60% as
free space. This means when building larger images (e.g. >100GB) with
unknown size upfront we end up with much wasted space, IO and network
traffic when uploading the images to several cloud providers. This can
be optimized by adding a third way by defining DIB_IMAGE_EXTRA_SIZE to
specify the free space in GB. This makes it possible to easily build
images of varying sizes while still minimizing the overhead by keeping
the free space constant to e.g. 1GB.
Change-Id: I114c739d11d0cfe3b8d8abc6df5ff989edfb67f2
In many cases, the statically sized 64MB journal is far below the
e2fstools default calculation[0] which calls for a 64MB journal only
on filesystems smaller than 16GB. On bare metal in particular, the
correct default journal size will often be in the 512MB-1GB range.
Since we cannot know what the target system is, this should be a
tunable parameter that the user can set depending on the intended
image usage.
Add a DIB_JOURNAL_SIZE envvar and --mkfs-journal-size parameter
to the image creation so users can override the default journal
size.
[0] https://git.kernel.org/pub/scm/fs/ext2/e2fsprogs.git/tree/lib/ext2fs/mkjournal.c#n333
Change-Id: I65fa13a088eecdfe61636678578577ea2cfb3c0c
Due to the referenced bug, many versions of debootstrap can't bring up
a buster environment. Unfortunately, these include versions we use to
do this on Xenial/Bionic nodes.
Also, there isn't backports or security updates, so elide these for
now.
I did get a working build (I haven't gone so far as a full boot+glean)
with this, at least.
Change-Id: If2420e92cb728ab6e91b0d70547da4483679b391
Paritial-Bug: #1822927
Currently, the cleanup script is using the existence of the folder
/sys/fs/selinux to check if SELinux is enabled. This, however, is
misleading in case disk-image-builder is used inside a Docker
container on a selinux-enabled host. In this case, the folder exists
in the container but SELinux is disabled.
This patch addresses the problem by checking, in addition to the
check already in place, the output of the command selinuxenabled.
Change-Id: I83e58f2467e60df9f0f00f7b7a58d0e2ce357a9a
Closes-Bug: #1820077
This is a mechanically generated change to replace openstack.org
git:// URLs with https:// equivalents.
This is in aid of a planned future move of the git hosting
infrastructure to a self-hosted instance of gitea (https://gitea.io),
which does not support the git wire protocol at this stage.
This update should result in no functional change.
For more information see the thread at
http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003825.html
Change-Id: Id26bec14c3d94e2f81b2148fc85d17f07866398c
This is only one line, but it takes a lot to untangle ... basically
the current "correct" path is:
---
mk_build_dir()
-> sets trap trap_cleanup EXIT
... stuff ..
mount_proc_dev_sys
-> mounts $TMP_MOUNT_PATH/<proc,dev.sysfs>
pre-finalise.d
finalise.d
unmount_image $TMP_BUILD_DIR/mnt # nb == $TMP_MOUNT_PATH
-> unmount_dir()
-> recursive unmount everything inside TMP_MOUNT_PATH
TMP_IMAGE_PATH=$(dib-block-device getval image-path)
export TMP_IMAGE_PATH
dib-block-device umount
dib-block-device cleanup
... actually cleanup directories ...
---
Our current failure exit trap does:
---
dib-block-device umount
unmount_image
...
---
Note this is the *opposite* of what is done in the correct exit path.
In the failure case, if a script fails in the finalise stages it leads
to /proc, /sys, /dev etc. still being mounted inside the image; the
"dib-block-device umount" call doesn't know anything about these
mounts and tries to unmount the parent directory, and we get a hard
failure with a busy mount, and all the mounts are subsequently leaked.
Note that "unmount_dir", which is ultimately called by
"unmount_image", already knows to skip those mounts that
"dib-block-device umount" manages (this is the DIB_MOUNTPOINTS list).
This is further evidence it should be called *before* the
dib-block-device umount.
Change-Id: Ibef3ce9d1167b9c4ff3d5717b113cd3ed374f5e3
Add a DIB_APT_MINIMAL_CREATE_INTERFACES boolean to the debootstrap
element which functions identically to
DIB_YUM_MINIMAL_CREATE_INTERFACES in the yum-minimal element.
This can be used to disable the creation of the
/etc/network/interfaces.d/eth[01] dhcp configuration files, which
are not needed on systems where cloud-init or other means are used
to configure networking.
The flag is enabled by default to keep creating the dhcp interface
files, maintaining backwards compatibility.
Change-Id: I1fdaca8350a5ceefd9e437af4fd000ce6a3ee7f3
in same cases it is required to avoid update all existing packages,
doing so can result in release update which is currently not possible
unless you not include "base" element.
"base" element used for most distribution (rhel, debain), and is
necessary for most cloud operations, this patch add
"DIB_AVOID_PACKAGES_UPDATE" parameter to skip updating all packages.
usecases for this patch can be:
* Avoid release update when building old release ex. RHEL7.5.
* build on network-less environment.
usage:
DIB_AVOID_PACKAGES_UPDATE=1
or
DIB_AVOID_PACKAGES_UPDATE=0
Change-Id: I71192b23c8f0bc48b348fe7377bf8a2399b53792
Related to I041a141366099093805e6052b1bbf64efd277e1e, we also need to
remove this on opensuse. The files for gate testing are added, but
the test is not added to any jobs at this point in the interests of
gate time.
Change-Id: I1af9e84d76bedcb2607717edc6d2abe2920b0584
This fixes a regression in I041a141366099093805e6052b1bbf64efd277e1e
where we starting skipping the removal of old files for image-based
builds (confusingly named centos7 rather than centos for historical
reasons). Fix the check
Change-Id: I74688a9e91d833b5d654056431729bed0585616c
As described inline, we only want to remove the system package files
on centos; it causes problems on Fedora where some system tools expect
these to be there.
But there is an additional bug -- pip actually removes the system
package files anyway. To work around this, reinstall the system
package.
Closes-Bug: #1813232
Change-Id: I041a141366099093805e6052b1bbf64efd277e1e
As described in the comments, it seems the transition between
dbus-daemon -> dbus-broker in Fedora 29 has made it so the packages
can get into a state where neither service is enabled.
Explicitly install and enable dbus-broker for F29
Change-Id: I06753043a75be2f635653899c6c251b9fbdd7c67
The path $TMP_BUILD_DIR/mnt becomes the / inside the chroot during
the chroot phases of diskimage-builder. Previously this path was being
created using the account running diskimage-builder. This account may
not be valid inside the chroot. This causes path validation, when running
on a Ubuntu bionic host, to fail.
This patch chown's the $TMP_BUILD_DIR/mnt to root.root to make sure
that / is owned by a valid account inside the chroot.
Change-Id: Ifedc136baa67c7952942aed2c8cb1041902fef91
Closes-Bug: 1811113
There is an use of get_image_element_array on the environment.d
phase, for the iscsi-boot element.
This function is not available on that step. So moving the check
at next step, extra-data-d, where it is available.
Change-Id: I89cfe565492142c2f7962109360fcbcebadfd469
This plumbs through an "--use-nm" flag to glean which instructs it to
setup interface bringup with NetworkManager rather than legacy network
enablement scripts.
In this case, install the NetworkManager package. In the non-nm case,
also install the network-scripts for Fedora 29 -- this has stopped
being installed by default (it's been deprecated since forever).
As noted in the docs, this is currently really only relevant on the
supported rpm distros which are using the ifcfg-rh NetworkManager
plugin to effectively re-use old config files. However,
NetworkManager has similar plugins for other platforms, so support can
be expanded if changes are proposed.
Depends-On: https://review.openstack.org/618964
Change-Id: I4d76e88ce25e5675fd5ef48924acd09915a62a4b
Provide a "when" option that provides for not installing packages
based on a = or != match on an environment variable.
Unit tests are added.
Change-Id: Ifa824dccaff69fd447f45d54cb4a3083bcabdd86
It looks like we dropped running these probably when we moved the
elements around. For testtools to find the test scripts we need to
add the __init__.py files to make the directories look like modules.
Also prevent copying any .pyc or cache files in as hooks.
Change-Id: I66d5f6ee62cc4d9ee14c64e819b4db57d035d09f
This allows nodes with remote devices configured via iBFT to be
correctly used during Ironic introspection and deployment,
at least for non-multipath configurations.
The new element is added as a dependency for ironic-agent.
Change-Id: If3dac6504d26535593f12e851092065b688ef696
install-packages is running before install.d phase, there is a chance
that installing a package like "container-selinux" will failed the
build, moving "selinux-permissive" to run at pre-install stage make
more sense.
Change-Id: I32f988be725d4b385c3765c47a00cd57c53d7d71
Update builds to Fedora 29. Remove the openstack gate CI mirror
workaround for pre-28 versions as they're not building in the gate any
more.
Change-Id: Ia6a8ae8d66d69f6add39e571043328e7274ba26c
8 MiB is not enough when using the grub2 element with centos7 images,
which installs binaries from the shim-x64 and grub2-efi-x64 packages
under /boot/efi. 12 MiB ought to be enough for anybody.
Change-Id: I8594d1fe05242f246a5809740a115ab2f84ac5a3
In order to allow the simple preparation of base images which
can be used for LXC/nspawn machine containers, we add this
element.
Containers inherit a kernel from the host, so there is no need
to build a kernel into the image. All the element needs is a
base init system which, in this case, is systemd.
Change-Id: I45651de2aa1b19bdeee301094f0bdffdd0a3b45c
This finalises the ports of the legacy jobs to zuul native jobs.
The dib-setup-gate-mirrors role preconfigures the repo templates,
etc. for the openstack-ci-mirrors element.
The dib-functests role runs the tests as specified by dib_functests,
and can run under python2 or 3.
Change-Id: Ied67a31f0d31503d13eccad8662c29740c93f33e
I'm not really sure why I originally had --logfile also log to stdout
in I202e1cb200bde17f6d7770cf1e2710bbf4cca64c, but it seem
counter-intuitive (indeed, I just tripped myself up thinking that in a
devstack job "--logfile" would put the logs into a separate file and
avoid the stdout logging, and I wrote it!).
Make it so specifying a --logfile puts dib into quiet mode for stdout.
Explicitly overriding DIB_QUIET will allow both if someone wants that.
Change-Id: I3279c9253eee1c9db69c958b87a0ce73efc0be9b
It looks like epel-release switch from "mirrorlist" to "metalink"
(around release 7-10 Jun 2017 according to [1]). Update our rewrite
matching to handle this "metalink" as well.
Add epel element to the centos7 (image-based) build for testing too
[1] https://koji.fedoraproject.org/koji/buildinfo?buildID=978473
Add epel element so it's tested during the centos7 functional test.
Change-Id: I2d6d4c2ec47bc69d2f16c96b5045b05c435a1af9
While trying to get docker image pre-caching to work we couldn't get a
docker daeomon to run within the chrooted environment. However we got
docker running with the help of bwrap outside of the chrooted
environment. The only option so far for this is the block-device.d
phase. But this has the problem that it runs after the image size has
been calculated. This leads to broken builds if the docker images
being pulled are big.
This can be solved by adding a post-root.d phase that runs outside the
chroot but before the image size calculation.
Change-Id: I36c2a81e2d9f5069f18ce5b0d52c5f1c7212c3ae
This updates diskimage-builder to support current Fedora releases (27
and 28) and removes support for Fedora 26 which is EOL as of June
2018.
Change-Id: I602b22ed4d5397b39dc1eef67964f6fbdcd93060
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
We want to set DIB_EPEL_MIRROR for the epel element, which means we
need access to the CI mirror info script in the chroot. Copy it into
the temp directory with extra-data.d and update the environment file
to find it.
Change-Id: Ia12f0cbdeb194eef3155497ceb5ffc4a452aad76
When using the upstream cloud images with the "ubuntu" element, they
have universe and multiverse enabled which we don't mirror.
To use the infra mirrors as a DIB_DISTRIBUTION_MIRROR with this
element, we need to be able to skip redirecting to universe and
multiverse, and additionally enable insecure repos (as we don't gpg
sign our mirrors).
Add and document two new variables with the ubuntu element to do this.
This is then setup by the openstack-ci-mirrors element so that we use
local mirrors duing dib functional testing for the "ubuntu" element.
Change-Id: I6ffbde07fa0e103641ee5c5f9d9e854e5b2168dc
openssl/cryptography versions are updated/stable
musl profiles need newer versions of open-iscsi masked as upstream
doesn't want to work with multiple libcs
Change-Id: If5baf339516390ae332015928557c6bb734486c2
This is a lot of very low value noise in the logs as these iterate
through all the elements (often doing nothing). Turn it down and add
an echo so we just see what elements it is working on.
Change-Id: I0687de4722766189db9d4a7bd7d3cfb45d387b62
To facilitate this I've created two new environment variables to set
the environment and default options for package actions.
eval is needed for the export as it preserves quotes.
Change-Id: Ib03651ee8dacd48cd1c135afd57cd31101356056
Signed-off-by: Matthew Thode <mthode@mthode.org>
In exploring Gentoo caching, it was realised that we have no way to
bind mount the cache into the finalised image for the finalise.d
phases.
By adding a pre-finalise.d phase that runs outside the chroot, we can
mount outside things into the hierarchy at $TMP_BUILD_DIR/mnt which
are then seen by the in-chroot finalise.d phase.
This is similar to the pre-install phase
Change-Id: I9d782994843383ddf90f62c40498af9925fd9558
Some minor things after looking at these parts.
The dib-run-parts element doesn't do any of the copying any more, so
these comments are wrong.
The reason for the multiple mounts in the bind mount was non-obvious
to modern eyes (as util-linux has handled this for some time).
Formatting fix for the rst
Change-Id: Idb4c9ff32c49aced2c68a5c905bf7a8b2832a5a2
Currently there's more-or-less an assumption that a kernel is
installed, so module blacklists are simply echoed into the modprobe
blacklist. This may not be the case with some ongoing container work.
Although we don't need to blacklist modules for containers, it also
doesn't hurt. Move the debootstrap element to the new modprobe
element, and allow it to create the blacklist directory.
Change-Id: I0f057caf473951df56a2af9633e3a5b53e0809b1
With the check added in commit 7566819139,
diskimage-builder fails to build RPM-based images if kauditd is not
running. However, this is only valid for environments where SELinux is
enabled. If SELinux is disabled (which is identified by an empty _runcon
variable), proceed with running setfiles.
Change-Id: I1b056f20a3a55f7333391207d9e1049d25ece041
Closes-Bug: #1779273
Diskimage-builder fails to build ubuntu-minimal images when run on
a Ubuntu bionic-beaver (18.04) instance.
The user gets "Couldn't create tempfiles for splitting up" when
apt-get update is run in the ubuntu-minimal element root.d.
The issue is that the /tmp inside the chroot is not getting the
proper permissions applied from the base-files package. This is likely
because the pip-cache element has already created the directory before
the base-files package is installed.
This patch changes the order of pip-cache to root.d/11-pip-cache so that
it runs after teh base OS root.d elements run.
Change-Id: I6fd1cb2a23422206884165eb502b260f0c1e52f7
The ubuntu-minimal README states that the latest Ubuntu LTS
is the default, but currently that is not true. This patch
changes the default to the current LTS.
Change-Id: I10f28314d1a5969c20094194637cfe31219d228c
