Allow the "when:" statements to be a list of values, which are
effectively anded together to filter the package install.
Change-Id: Ia6f10741fa6be24b11d6991c8a6b6e07951ff68d
Opendev has dropped trusty from the mirrors. With no testing the
distribution is effectively unsupported, add a release note. Update a
few other random doc bits (that are not really that up to date
anyway).
Change-Id: I5bd0d0a94477cf8d84cef72f5d4b2e9e15ab9fd2
This is an alternative approach to commit
68bb43535e. I think this proposes a
better overall solution that the prior change which had the Python 3
packages being installed, but did not specify the _do_py3 flag to do
the installation steps that redirect the various tool installations.
Fedora 31+ doesn't have python2, and Tumbleweed does have some Python
2 support but there seems to be no reason to bother updating this
element for either with infra very close to removing this completely
[1]. Error out on these platforms, and add a release note.
The 15 path should include the python2 and python3 packages, along
with the flags to do the "cleanup"; i.e. forced removal of distutils
packages that pip 10+ won't touch. As mentioned in the original
change, the six package causes problems here, but we can clear that
too by explicitly listing it instead of letting it come in via
dependencies. Again, this element will be removed from the infra 15
builds ASAP; but we can release with this to provide a roll-back point
if we need to revert the removal to fix things temporarily.
Add it to the testing path as well.
[1] https://docs.opendev.org/opendev/infra-specs/latest/specs/cleanup-test-node-python.html
Change-Id: I7a6a342461d6001c25e55638ba9b7438c28f2519
I don't see anywhere we bring this in, especially on a minimal build.
In 2020 it seems like a base dependency, put it alongside
software-properties-common that installs the other apt helper bits.
Change-Id: I5b079eac4912cb4a164e9aa6158ed106a28f576c
If the running kernel of the system building the image
matches the kernel that is to be removed dnf will fail.
Repalce use use of dnf with rpm -e.
Closes-Bug: #1623409
Change-Id: Ie2481ea8a02b7b0720e46fa179f24badf4aa25c5
All the platforms we care about now have python3 with venv (even
centos7 now) packaged somehow. Add an ensure-venv element to make
sure that "python3 -m venv" works. Any other elements that wish to
install non-distribution-packaged Python utilities can use this to
keep them separate from the main system installs.
Port glean to use this, and drop its dependency on pip-and-virtualenv.
Change-Id: Ic16f134fe34293bb68e7c632dd320f523366320d
* Add "centos" element, a CentOS version-independent element. This is in
line with the same work done for RHEL in Stein cycle.
* Deprecate the centos7 element. CentOS 7 support itself it not
deprecated though. The new "centos" element provides the same support
level as the "centos7" element.
* Add functional testing
The default CentOS version is 8. You can adjust it using the DIB_RELEASE
environment variable.
Change-Id: I373ba2296c4613765676e59aabd9c651345298d1
On IPA we are using efivar and efibootmgr, we already added the
packages on ipa-builder.
Adding the pacakges on diskimage-builder, so that people who use
it to build the images won't get into trouble.
Change-Id: I9ab6588f20302b4808b09dc060aced5fd267a3d2
If rdisc6 is available, a node using this element will loops until
DIB_DHCP_TIMEOUT is reached because of a missing 'break' when rdisc6
return code is 0.
This will mark the dhcp-interface@.service unit as failed (because it
has the same timeout) and not bring any network interface online.
Change-Id: I034dcda94d765f236950ebcbee36789f5bdc515f
Closes-Bug: #1854717
Signed-off-by: Hervé Rousseau <hroussea@cern.ch>
The current implementation evauates the dib-init-system
script too early. Also it looks that there is no simple
way of getting the info about the init system automatically:
another element can install (later on) a different
init system. Therefore the only reliable way of setting
this is manual.
Change-Id: I6e9ffa1bdb3154f488f4fd335b197699b86aacd4
Signed-off-by: Andreas Florath <andreas@florath.net>
This reverts commit a3e9e7f89e.
We still have some issues with vhd creation on RAX
In short, it appears that images fail to resize unless they have a
specific "creator" field. Revert this while we consider the options.
[1] https://bugs.launchpad.net/nova/+bug/862653
Change-Id: I2b6a3bfbfe28432fbb6a2ce4a0211939d224b8d5
The "ironic-agent" is copied to ironic-python-agent-builder and
hence it is deprecated from DIB.
Remove from functional testing
Change-Id: Ibc4f75b9d7e2a31994fc86d05bd57975f00fb74f
Task: 36198
Story: 2005114
This package is not installed by default on Debuntu, but is on RH
platforms. This is causing a build breakage as DIB_PYTHON_VIRTUALENV
tries to use this (I3414fb9e503f94ff744b560eff9ec0f4afdbb50e).
Add the package.
Change-Id: I9a551c57dd128bbb4b095c847f634c777b2cb553
The vhdutil utility is completely dead; the whole subsystem it relies
on was removed with [1] so it's not even vaguely possible to keep it
up-to-date.
I took the .raw images on a nb and used the qemu-img there (so Xenial)
and generated some VPC images; uploaded them to rackspace and the all
seemed to boot fine. If there was a problem, maybe it's been fixed on
either the qemu or RAX side in the previous few years.
Thus swith to qemu-img to generate the vhd images too.
[1] https://xenbits.xen.org/gitweb/?p=xen.git;a=commit;h=5c883cf036cf5ab8b1b79390549e2475f7a568dd
Change-Id: I3099d2ebb958370fcec623087a093b2c8dbdc6c4
Add option to set the suite subpath after the release name for the
security mirror URL independently in the debian-minimal element,
since this can differ between mirrors.
Change-Id: I4cc8f54fba012986423e30e19bff276208b8ad62
It turns out that this breaks ipv6 config with NM. Instead what we want
is for glean to not up interfaces on boot (see the depends-on).
Change-Id: I6c5bc76c433e29f02d3266ab8f669015125ec954
Depends-On: https://review.opendev.org/#/c/688031
This adds CentOS 8 into functional and boot tests.
This completes centos-minimal support, documentation is updated and a
release note is added.
Change-Id: I435c2967b4f49faeb6d6edf189907b9f96e80357
NetworkManager with simple-init has proven to be stable in OpenStack
infra, switch to it by default for CentOS and Fedora. For CentOS 8
and Fedora, add a check to make it the only option. Thus only CenOS 7
remains optionally using the legacy scripts; this is likely not used
anywhere (infra is really the primary user, where NetworkManager is
already used); we can likely remove this variable (and hence path) in
a future cleanup.
In the setup, remove rhel7 element which was never really tested.
Reorganise the fallthrough to call out the default paths as doing
nothing.
Change-Id: Ic996956da4b85f7d95179b8df9881d5f52c091af
Add option to set the security mirror URL independently in the
debian-minimal element, since this can not be overriden by the
standard DIB_DISTRIBUTION_MIRROR variable.
Change-Id: I145844a410d06a479e68db1bf6d5d0159389305c
As described inline, deprecate the "source" install for CentOS 8.
Overwriting the packaged tools has long been a pain-point in our
images, and the best outcome is just not to play the game [1].
However, the landscape remains complicated. For example, RHEL/CentOS
8 introduces the separate "platform-python" binary, which seems like
the right tool to install platform tools like "glean" (simple-init)
with. However, platform-python doesn't have virtualenv (only the
inbuilt venv).
So that every element doesn't have to hard-code in workarounds for
these various layouts, create two new variables DIB_PYTHON_PIP and
DIB_PYTHON_VIRTUALENV to just "do the right thing". If you need is
"install a pip package" or "create a virtualenv" this should work on
all the platforms we support. If you know more specifically what you
want (e.g. must be a python3 virtualenv) then nothing stops elements
calling that directly (e.g. python3 -m virtualenv create); these are
just helper wrappers for base elements that need to be broadly
compatible.
[1] http://lists.openstack.org/pipermail/openstack-infra/2019-September/006483.html
Change-Id: Ia267a60eecfa8f4071dd477d86daebe07e9a7e38
Two bugs are addressed.
1) The sysprep element was broken in that it only truncates
/etc/machine-id, but not /var/lib/dbus/machine-id. systemd will
not generate a new machine-id if /var/lib/dbus/machine-id is
present[1], it will simply copy it to /etc/machine-id.
We observed machine-ids being packaged in /var/lib/dbus/machine-id
on several distros: Ubuntu Bionic, Fedora 29, Debian Stretch.
CentOS 7 and Ubuntu Xenial do not contain packaged machine-id as
far as I can tell.
All test builds were performed using -minimal elements.
2) A second bug existed where debian-minimal did not run the sysprep
element at all, so a stretch image I tested contained a populated
/etc/machine-id AND a populated /var/lib/dbus/machine-id.
[1] https://www.freedesktop.org/software/systemd/man/machine-id.html#Initialization
Change-Id: Ibb28b6e90d966a845de38a2cd5a1e8babd2604bc
linux-firmware and linux-firmware-whence (meta package for mostly iwl
firmwares) packages account for approx. 289 M install size on a F30
system, and linux-firmware for approx. 176 M on CentOS 7. Users needing
these firmwares are eventually baremetal users and are not looking for a
very minimal operating system base install like virtual image users are.
Thus, a non-minimal OS element is better suited for them. Alternatively,
it could be later considered a dedicated firmware element.
This is inline with I8ce65e1d357d15e8ed8995ad1dcaea02bbd1986f.
Change-Id: If104fc3c1e9349b8d501a2351fff1ab4c0dbc6a4
Add a new environment variable $DIB_GZIP_BIN allowing builders to
specify a different gzip (such as pigz) to be used when compressing
tgz images.
Change-Id: Ifb617568140a149e2fda241e07ff8a59429e6697
This is a follow-on to I475a253091cbaf63687b91c748c31a6753bb0f57 as we
are still seeing issues on some clouds with unconfigured networking.
We increase the timeout, but also make it configurable so we can
fiddle it without a dib release in the gate.
To follow-on from the experimentation done by clarkb, I can confirm by
emperical testing on a Centos 7 image (from today, today being this
change's date) that setting
net.ipv6.conf.all.autoconf=0
by itself is "fatal" and the interfaces do not come up; i.e. nm does
not by default seem to re-enable ipv6 for the interface. However,
explicitly adding:
IPV6INIT=yes
IPV6_AUTOCONF=yes
to the interface file *does* seem to make it work, even if
"all.autoconf=0" is set (then again, there's also bugs about the
effect of this [1]). However, no extant distribution (I can currently
find) does anything like this by default.
If this continues, this may be an option. Another might be to avoid
the use of the nm-settings-ifcfg-rh profiles and move directly to nm
ini files with glean.
[1] https://bugzilla.kernel.org/show_bug.cgi?id=11655
Change-Id: I869ebffc8cde3bbff573f6583fd9dd02a5598590
As noted in the change, 7fd52ba841
increased the size of the EFI partition considerably. This has meant
that our padding upwards of the disk size is insufficient and EFI
builds (arm64 in particular) is failing due to out-of-disk errors
during final image operations like installing kernels.
Similar to the discussion we had in
I65fa13a088eecdfe61636678578577ea2cfb3c0c, this feels a bit ugly
because we're mixing logic here with sizes specified in block-device
config files. But it boils down to the same problem; we are
calculating the disk size here and passing it to the block-layer, so
unless we want to make large changes to the status quo about where
these sizes are calculated, small adjustments here are the most KISS
solution.
Thus we check if we have selected the EFI bootloader element, and thus
assume there will be a large system EFI partition and expand the disk
size accordingly.
Change-Id: Ifa05366c2f2b95259f3312e4dde8c85347075ba1
This element configures systemd to send its journal to the console,
which can then be retreived by server commands. In the case of
nodepool, if the image failed to boot the console will be dumped into
the logs when nodepool decides the node is not responding. Having
this can be very helpful diagnosing early boot errors.
Needed-By: https://review.opendev.org/#/c/669787/
Change-Id: I6b6df7023acb6b2f967b84840bc4b542ebc03727
This patch adds a new environment variable to the ubuntu-minimal
element called DIB_UBUNTU_KERNEL that allows you to specify the kernel
meta package that will be using to install the kernel inside the image.
It supports "linux-image-generic" (The default), "linux-image-kvm", and
"linux-image-virtual".
This allows building images that are smaller in size (~200MB smaller
qcow2) that have only the kernel modules necessary for virtual
machines.
Change-Id: I8ce65e1d357d15e8ed8995ad1dcaea02bbd1986f
Use openSUSE 15.1 as default, which is the latest released stable
openSUSE release.
Remove leftovers for unmaintained openSUSE 42.2 images.
Depends-On: https://review.opendev.org/#/c/660126/
Change-Id: I0b204b7b3d7ae74b6749320b3bfe1ca89d154ebb
When I said in I8594d1fe05242f246a5809740a115ab2f84ac5a3 that 12 MiB
ought to be enough, I should have expected that I would be proven wrong.
While 12 MiB is enough to fit shim-x64 and grub2-efi-x64, yum fails to
update these packages to newer versions:
Transaction check error:
installing package shim-x64-15-2.el7.centos.x86_64 needs 7MB on the /boot/efi filesystem
installing package grub2-efi-x64-1:2.02-0.76.el7.centos.1.x86_64 needs 3MB on the /boot/efi filesystem
Error Summary
-------------
Disk Requirements:
At least 7MB more space needed on the /boot/efi filesystem.
It is recommended that the ESP partition be much bigger. This commit
bumps its size to 550MiB, following guidelines from Rod Smith to avoid
incompatibilities with some EFIs [1].
[1] https://www.rodsbooks.com/efi-bootloaders/principles.html
Change-Id: If9515234f1a803cda32b2482f8abe10ddf0e6d26
The rhel7 element is deprecated and is left only for backward
compatibility.
The rhel element should be used instead. Users should set DIB_RELEASE to
'7' to indicate which release you are using.
The new element is a version-less RHEL element to handle both '7'
and '8' DIB_RELEASE, which aligns with other elements which operate in
the same way such as the Fedora element.
Change-Id: Ic39ed85cacae9942448eb18ad685763f9369c2ed
Make a version-less RHEL element to handle both '7' and '8' DIB_RELEASE.
The element usage should align with other elements which operate in the
same way such as the Fedora element.
Additionally, this patch adds support for RHEL8 that operates with
Python 3.
As of now, users of diskimage-builder will still be able to use the
'rhel7' element, or migrate to 'rhel' and specify their respective
DIB_RELEASE value.
* mount the xfs file-system for extraction as read-only. vaguely
based on explaination in [1] and the fact we only read the image
data into a tar, so can ignore this.
XFS (dm-1): Superblock has unknown read-only compatible features (0x4) enabled.
* Use the redhat system python as the dib-python version. dib was
ahead of it's time making an abstracted python interpreter for
system work ;) the system python should work for running the various
dib element scripts.
[1] https://unix.stackexchange.com/questions/247550/unmountable-xfs-filesystem
Redhat-Bug: https://bugzilla.redhat.com/show_bug.cgi?id=1700253
Co-Authored-By: Ian Wienand <iwienand@redhat.com>
Change-Id: I90540675c70bb475d9db2ae24f81c648a31f3f95
I want to use the new --image-extra-size flag[1] but my use-case
calls for megabyte granularity of this value. Rather than adding
60% to an 800MB image, maybe I only want to add 100 or 200MB, etc.
[1] https://review.opendev.org/#/c/655127/
Change-Id: I8fb9685d60ebb1260d5efcf03c5c23c561c24384
Use openSUSE 15.0 as default, which is the latest released stable
openSUSE release. Switch to https for accessing download.o.org
as encrypted transfers should be used by default.
Remove leftovers for definitely unmaintained openSUSE 13.x images
and split into old/new leap style versioning scheme for clarity.
Change-Id: Iab129eeee2b1a2563f0f0d2cb17bbad57c068e38
Harden sshd configuration by adding KexAlgorithms, Ciphers and MACs for sshd,
following good pratices on https://infosec.mozilla.org/guidelines/openssh
Change-Id: I3051320d867a5033e82deef10c5e723ca9829884
Co-Authored-By: Nicolas Hicher <nhicher@redhat.com>
Currently diskimage-builder supports two ways to specify the image
size. One is defining a fixed image size using DIB_IMAGE_SIZE, the
other one is auto-detection while adding a security margin of 60% as
free space. This means when building larger images (e.g. >100GB) with
unknown size upfront we end up with much wasted space, IO and network
traffic when uploading the images to several cloud providers. This can
be optimized by adding a third way by defining DIB_IMAGE_EXTRA_SIZE to
specify the free space in GB. This makes it possible to easily build
images of varying sizes while still minimizing the overhead by keeping
the free space constant to e.g. 1GB.
Change-Id: I114c739d11d0cfe3b8d8abc6df5ff989edfb67f2
In many cases, the statically sized 64MB journal is far below the
e2fstools default calculation[0] which calls for a 64MB journal only
on filesystems smaller than 16GB. On bare metal in particular, the
correct default journal size will often be in the 512MB-1GB range.
Since we cannot know what the target system is, this should be a
tunable parameter that the user can set depending on the intended
image usage.
Add a DIB_JOURNAL_SIZE envvar and --mkfs-journal-size parameter
to the image creation so users can override the default journal
size.
[0] https://git.kernel.org/pub/scm/fs/ext2/e2fsprogs.git/tree/lib/ext2fs/mkjournal.c#n333
Change-Id: I65fa13a088eecdfe61636678578577ea2cfb3c0c
This plumbs through an "--use-nm" flag to glean which instructs it to
setup interface bringup with NetworkManager rather than legacy network
enablement scripts.
In this case, install the NetworkManager package. In the non-nm case,
also install the network-scripts for Fedora 29 -- this has stopped
being installed by default (it's been deprecated since forever).
As noted in the docs, this is currently really only relevant on the
supported rpm distros which are using the ifcfg-rh NetworkManager
plugin to effectively re-use old config files. However,
NetworkManager has similar plugins for other platforms, so support can
be expanded if changes are proposed.
Depends-On: https://review.openstack.org/618964
Change-Id: I4d76e88ce25e5675fd5ef48924acd09915a62a4b
Provide a "when" option that provides for not installing packages
based on a = or != match on an environment variable.
Unit tests are added.
Change-Id: Ifa824dccaff69fd447f45d54cb4a3083bcabdd86
This allows nodes with remote devices configured via iBFT to be
correctly used during Ironic introspection and deployment,
at least for non-multipath configurations.
The new element is added as a dependency for ironic-agent.
Change-Id: If3dac6504d26535593f12e851092065b688ef696
install-packages is running before install.d phase, there is a chance
that installing a package like "container-selinux" will failed the
build, moving "selinux-permissive" to run at pre-install stage make
more sense.
Change-Id: I32f988be725d4b385c3765c47a00cd57c53d7d71
I'm not really sure why I originally had --logfile also log to stdout
in I202e1cb200bde17f6d7770cf1e2710bbf4cca64c, but it seem
counter-intuitive (indeed, I just tripped myself up thinking that in a
devstack job "--logfile" would put the logs into a separate file and
avoid the stdout logging, and I wrote it!).
Make it so specifying a --logfile puts dib into quiet mode for stdout.
Explicitly overriding DIB_QUIET will allow both if someone wants that.
Change-Id: I3279c9253eee1c9db69c958b87a0ce73efc0be9b
While trying to get docker image pre-caching to work we couldn't get a
docker daeomon to run within the chrooted environment. However we got
docker running with the help of bwrap outside of the chrooted
environment. The only option so far for this is the block-device.d
phase. But this has the problem that it runs after the image size has
been calculated. This leads to broken builds if the docker images
being pulled are big.
This can be solved by adding a post-root.d phase that runs outside the
chroot but before the image size calculation.
Change-Id: I36c2a81e2d9f5069f18ce5b0d52c5f1c7212c3ae
In exploring Gentoo caching, it was realised that we have no way to
bind mount the cache into the finalised image for the finalise.d
phases.
By adding a pre-finalise.d phase that runs outside the chroot, we can
mount outside things into the hierarchy at $TMP_BUILD_DIR/mnt which
are then seen by the in-chroot finalise.d phase.
This is similar to the pre-install phase
Change-Id: I9d782994843383ddf90f62c40498af9925fd9558
The grub.cfg has two variables [1]
GRUB_CMDLINE_LINUX : used on all boots
GRUB_CMDLINE_LINUX_DEFAULT : additionally used on all "normal" boots
The problem with I2298675dda1f699c572b3423e7274bc8bd7c1c9d is that it
appened the values in DIB_BOOTLOADER_DEFAULT_CMDLINE to both of these,
resulting in duplicated arguments. I don't think we considered that
GRUB_CMDLINE_LINUX_DEFAULT actually already appends to the
GRUB_CMDLINE_LINUX values.
Make DIB_BOOTLOADER_DEFAULT_CMDLINE only append itself to
GRUB_CMDLINE_LINUX_DEFAULT. That seems to line up sensibly with the
name of the variable.
Documentation is enhanced around this, and a releasenote added.
[1] https://help.ubuntu.com/community/Grub2/Setup
Change-Id: I76b5442a9090c19a6540ed2d4ab324546f241ebf
Closes: #1791736
This element will replace modprobe-blacklist element. It wil
still have the blacklist functionality, but it also adds
the feature of passing a complete file with settings to the
modprobe.d directory. Adding this functionality, that will
allow elements that depends on this module, to just copy the
specified files to the final directory.
Change-Id: I9a44f7d11520b8b1e604956d3c1db2fc7e2bf457
This adds support for a GPT label type to the partitioning code. This
is relatively straight-forward translation of the partition config
into a sgparted command-line and subsequent call.
A unit test is added based on a working GPT/EFI configuration and the
fedora-minimal functional test is updated to build a single-partition
GPT based using the new block-device-gpt override element. See notes
in the sample configuration files about partition requirements and
types.
Documentation has been updated.
Co-Authored-By: Marcin Juszkiewicz <marcin.juszkiewicz@linaro.org>
Change-Id: I6b819a8071389e7e4eb4874ff7750bd192695ff2
As described, we want to set the default label for XFS disks to the
shorter value.
For example, you hit this when setting the old FS_TYPE environment
variable to 'xfs' (which sets the "root-fs-type" parameter, which gets
passed through to 'type'; but does not set a default label).
Change-Id: I41dce6e25766562db4366021309b8c2b74a8ab80
Closes-Bug: 1742170
The installed pip can be an older version which does not support
the -c argument. Therefore, upgrade pip before using -c.
Change-Id: If18d8ea822a62c8551c9c4d47354d58b0299fed2
Closes-Bug: 1744403
This provides a basic LVM support to dib-block-device.
Co-Authored-By: Ian Wienand <iwienand@redhat.com>
Change-Id: Ibd624d9f95ee68b20a15891f639ddd5b3188cdf9
Deploying many nodes with the generated image shouldn't have the same
/etc/machine-id so clearing it and letting systemd generate a new
id upon first boot seems to be the best way to achieve this.
Change-Id: I73d0577d31464521b3989312fd9d982a1312a268
Closes-bug: 1707526
Closes-bug: 1672461
Fedora 26 is now the latest release:
https://fedoraproject.org/wiki/Releases/26/Schedule
We are building and using these in infra now
Change-Id: I012c2d28255be274e88abc2751d968bafaf76fbb
Depends-On: Ieba5f69020a13681074f72cfca2955071801b63a
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
We added the DIB_distro_DISTRIBUTION_MIRROR arguments with
I92964b17ec3e47cf97e3a3091f054b2a205ac768 as a way that we could
source a list of mirrors and then have the distro elements choose
which one applied to them.
However, this hasn't worked out to be so useful. The
openstack-ci-mirrors element is working as a mirror setup script -- it
translates the openstack CI mirror list variables into the generic
"DIB_DISTRIBUTION_MIRROR" as appropriate for each distro's build.
Also, it turns out there's other things that need to be done, such as
turning off gpg checking, which mean the idea of "just export
variables" hasn't turned out as valid ... you need actual code
involved to get it right.
AFAICT we never actually documented these, and they do not seem to be
in use. They have caused considerable confusion when dealing with new
platforms as we try to keep consistency. Remove them.
[1] http://codesearch.openstack.org/?q=DIB_.*_DISTRIBUTION_MIRROR&i=nope&files=&repos=
Change-Id: Ifc4ab700631ffdfbe790068558f670f9a11dde5e
Several people have popped up in IRC recently with failures in these
elements. Without Python 2.7 available in the image they are
unsupported (OpenStack hasn't supported it for a long time). Remove
these to avoid further confusion.
The centos/centos7 DISTRO split that has happened with centos-minimal
is unfortunate but I don't think it helps to rename centos7/rhel7 ATM.
To summarise; DISTRO=centos7 means image based build,
DISTRO=centos && DIB_RELEASE=7 means the minimal build.
In the future, I think it is important that the minimal builds and
image builds set the same DISTRO. This reflects that "upper" layers
shouldn't care about the exact building of the lower layers. I see
CentOS 8 going one of two ways
1) the changes are so significant, we start separate centos8 /
centos8-minimal elements. They both set DISTRO=centos8 (and
DIB_RELEASE to point-release maybe?). This means we have to update
all "if DISTRO == centos || DISTRO == centos7" branches to also check
for "centos8". Evenually (!) "centos" goes away for versioned DISTRO
only
2) we restore centos element with DISTRO=centos and DIB_RELEASE=8, and
centos-minimal remains the same. This means we have to audit all "if
DISTRO == centos" calls to make sure they're appropriate for version 8
(stick a "&& DIB_RELEASE=7" on them all basically).
I'm not sure we can fully decide until we start to see excatly how the
distro switching/matching bits look, but (2) is consistent with Ubuntu
and probably the preferred solution.
Some "rhel" parts have been cleaned up. More could be done in
rhel-common, but given our lack of coverage of that I'd prefer to
leave it for now.
Change-Id: I6ea784116ef59ca22878c8512c963f29c815a00a
This adds a devstack-inspired output filter to standardise
timestamping.
Currently, python tools timestamp always (timestamp setup in
logging_config.py) but all the surrounding bash does not.
We have extra timestamps added in run_functests.sh for our own
purposes to get the bash timestamps; but this ends up giving us
double-timestamps for the python bits. Additionally, callers such as
nodepool capture our output and put their own timestamps on it, and
again have the double-timestamps.
This uses a lightly modified outfilter.py from devstack to standardise
this.
All output is run through this filter, which will timestamp it. I
have removed the places where we double-timestamp -- logging_config.py
and the prefix in dib-run-parts.
An env option is added to turn timestamps off completely (does not
seem worth taking up a command-line option for). For callers like
nodepool, they can set this and will just have their own timestamps as
they collect the lines.
Since all logging is going through outfilter, it's easy to add a
--logfile option. I think this will be quite handy; personally I'm
always redirecting dib runs to files for debugging.
I've also added a "quiet" option. I think this could be useful in
run_tests.sh if we were to start logging the output of each test to
individual files. This would be much easier to deal with than the
very large log files we get (especially if we wanted to turn on
parallel running...)
Change-Id: I202e1cb200bde17f6d7770cf1e2710bbf4cca64c
This element has not been functioning correctly for some time due to
an incorrect path to select-boot-kernel-initrd (should be /usr/local/bin).
The dracut-regenerate element can be used to regenerate dracut ramdisks
and is more flexible than this element.
Change-Id: I33d555ffd4a92b2948b2ea4a66b151f0422ccb8c
Closes-Bug: #1688546
This patch finalizes the block device refactoring. It moves the three
remaining levels (filesystem creation, mount and fstab handling) into
the new python module.
Now it is possible to use any number of disk images, any number of
partitions and used them mounted to different directories.
Notes:
* unmount_dir : modified to only unmount the subdirs mounted by
mount_proc_sys_dev(). dib-block-device unmounts
$TMP_MOUNT_PATH/mnt (see I85e01f3898d3c043071de5fad82307cb091a64a9)
Change-Id: I592c0b1329409307197460cfa8fd69798013f1f8
Signed-off-by: Andreas Florath <andreas@florath.net>
Closes-Bug: #1664924
This change move "do_extra_package_install" from pre-install to install
phase.
Extra packages are added by user request using the flag "-p", This
package should not be something the elements depend on.
The reason behind this patch is to move the extra package install to
a proper phase, Also more reasonable if base element run package update
to be before we install extra packages.
Change-Id: I68cc773aba9aa01743f0dda9f4e635e4cac2a282
The dib-run-parts element was copying our internal version of
dib-run-parts into /usr/local/bin to be used running scripts inside
the target chroot. However, it never cleaned up after itself. This
means all images were left with an unmanaged local install of
dib-run-parts.
This copies dib-run-parts into the hooks directory of the chroot and
runs it from there. It is cleaned up automatically on the exit path.
The dib-run-parts element is no longer required and it has been
removed from all dependencies. It is left with a deprecation notice
in the README. For compatability we convert it to simply install
dib-utils.
Codesearch shows no users depending on this unintentional implicit
install. Note os-refresh-config depends on dib-utils and thus will
have an explicitly installed version.
Partial-Bug: #1673144
Change-Id: Ia2e96c00a4246c04beb96c17f83b8aefb69219ca
It was an oversight during v2 development for dib to start providing
dib-run-parts. The intention was for dib to use a vendored
dib-run-parts directly from $_LIB and have no dependencies on
dib-utils at all. By exporting dib-run-parts, we created an
unintentional conflict with the dib-utils package which provides the
same script.
Tools that depend on dib-utils are unaffected by this
(os-refresh-config).
The only tool that installs diskimage-builder and then assumes
dib-run-parts is available in the path is instack. I have proposed
Ibfe972208df40fa092b11b5419043524c903f1b4 to modify that to use our
internal version.
Change-Id: I149c345d38d761a49b3a6ccc4833482f09f1cd05
On Debian Jessie and Debian Stretch systemctl is in /bin.
If the package systemd-sysv is not installed the script
dib-init-system did not find the init system.
This patch fixes the problem: it also looks in /bin
for systemctl and if found decides for systemd.
Change-Id: I5a18052a070bad5e16b14672237a1e2b38513949
Signed-off-by: Andreas Florath <andreas@florath.net>
Currently there is no description of dependencies in the generated
documentation of the elements: therefore a user of an element does not
know which other elements are automatically included and e.g. which
configuration options are available. In addition there are some
copy&pastes of parts of the README.rst scattered thought different
Ubuntu and Debian specific elements.
This patch adds a semi-automatic generation of dependency information
of all elements. Nevertheless these are not automatically included.
The author of the element's README.rst can decide if and where the
dependency information should appear and can use the descriptor
.. element_deps::
for this.
This patch adds the dependency information for some Debian and
Ubuntu patches - and creates the base for later removing the
duplicated parts.
A call is added to element_dependencies._find_all_elements() to
populate reverse dependencies for Element objects.
(This is a reworking of I31d2b6050b6c46fefe37378698e9a330025db430 for
the feature/v2 branch)
Change-Id: Iebb83916fed71565071246baa550849eef40560b
During the creation of a disk image (e.g. for a VM), there is the need
to create, setup, configure and afterwards detach some kind of storage
where the newly installed OS can be copied to or directly installed
in.
This patch implements partitioning handling.
Change-Id: I0ca6a4ae3a2684d473b44e5f332ee4225ee30f8c
Signed-off-by: Andreas Florath <andreas@florath.net>
Cleaning logs was split, some was done in the
img-functions.finalise_base, some was done in the base element.
The version unifies tidy up logs in the lib/img-functions.
Especially when building docker container images the base element
cannot be used. This patch removes about some hundreds KB of
useless logs in cases when the base element is not used.
Change-Id: I165bafb73daf9144c2f3a83930e85e8d8cf5fae3
Signed-off-by: Andreas Florath <andreas@florath.net>
Every run we are doing a full tar.gz of the chroot environment that
never gets used.
It's not suitable for CI since we use fresh images each time there.
The cache in general isn't really isn't a very safe thing to have
around, because there's no invalidation procedure and no real way to
make one -- we've no guarantee that a new chroot build even moments
after a previous one wouldn't bring in or different packages, etc (of
course this is *unlikely*, but the longer you go between builds the
worse the problem becomes. Also, tons of packages get installed after
this not from any cache, so potential speed-up is rather marginal.
Debian turned this off with I58fc485aacacaa17243bf9ce760ed91256d1f182.
However, given the reasons above and it's complete lack of testing, I
don't see this as useful.
If we really want this type of thing, I think we should come up with a
way to use a persistent external yum/dnf cache that yum/dnf keeps in
sync with it's usual invalidation rules.
Change-Id: I66789c35db75c41bc45ea1ad2e26f87456de4e4d
