https://review.opendev.org/c/openstack/diskimage-builder/+/785138
adds the support for DIB_DNF_MODULE_STREAMS which is now available
for all Yum based distros.
This patch enhances the docs for using it for all Yum
based distributions.
Signed-off-by: Chandan Kumar (raukadah) <chkumar@redhat.com>
Change-Id: I29e726679c2b675b3c0cd95a3ff48fdad7cd5431
We've noticed that centos8 arm64 images have a root devices of
/dev/mapper/loop7p3 which make sense within a dib image build context
but not at boot time. Dib intends to use labels to set the root device
but when efi is used we end up running grub2-mkconfig against the efi
grub config path before we configure grub to use labels.
Fix this by running grub2-mkconfig after its configuration is set.
This should avoid confusion and complicated paths through the scripts
that configure this for us. We then copy the resulting config to the efi
specific grub.cfg location for platforms that have it.
There is also a small refactoring that is done to try and make the ~3
boot variants more clear:
1) Booting with legacy bios
2) Booting with uefi without a signed shim that directly calls grub
3) Booting with uefi and a signed shim that calls grub
Options 1 and 2 share the /boot/grub*/grub.cfg file. Option 3 needs its
grub.cfg to live alongside distro specific efi target.
Change-Id: Ie9790da9d1bbea58197b37b15a48e77f8a93c1ac
While building cloud images, it is common to set modules
for CentOS and RHEL images. Earlier it was part of rhel-common
which was specific to RHEL OS not for CentOS. Moving it
under yum element as module/stream can be enabled or disabled
via dnf itself.
Signed-off-by: Chandan Kumar (raukadah) <chkumar@redhat.com>
Change-Id: Idc0f277f97e92e4d003f059f01b59f1b5513da34
At this moment the IPA image building with OpenSuse is broken and here, it was failing during the release check for Opensue because etc/SuSE-release is not valid anymore and deprecated for openSuse. Its renamed to /etc/os-release for openSuse rlease 15. This PR will solve the issue to build IPA image with OpenSuse base image. There is another PR opened in ironic-python-agent-builder, which adds all the missing packages, setuptools upgrade and svc mapping to do the build successful.
https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/778726
Bug-Report: https://bugs.launchpad.net/diskimage-builder/+bug/1921510
Change-Id: Id2759be29bfcbf2ecf1ce67e171686924b506b1a
open-iscsi and open-isns need keywording to support gcc-10, move it out
of being keyworded only for musl profiles.
remove unneeded keywords for python-exec and python-exec-conf (marked
stable)
use the full package name for the dev-lang/python-exec-conf package
Change-Id: I44eaf8c2230e9e2089a72fce46954f4336626843
Signed-off-by: Matthew Thode <mthode@mthode.org>
As of grub2 >= 2.02-95 on redhat family distros, calling grub2-install
on an EFI partition will fail with: "this utility cannot be used for
EFI platforms because it does not support UEFI Secure Boot."
This version of grub is now in centos8-stream and non-eus repos of
RHEL-8. It is not currently possible to build whole-disk UEFI images
on these distros, and when this package is promoted this will also
affect centos8 and RHEL-8 eus. The grub maintainers made this change
because the grub2-install generated /boot/efi/EFI/BOOT/BOOTX64.EFI
will never be capable of booting with Secure Boot.
This change defines a $EFI_BOOT_DIR for every distro element. When
directory /boot/efi/$EFI_BOOT_DIR exists a grub.cfg file in will be
generated there. This change also installs the shim package on redhat
family distros, which installs a copy of the shim bootloader to
/boot/efi/EFI/BOOT/BOOTX64.EFI. Using centos as an example, this
allows UEFI to boot the shim /boot/efi/EFI/BOOT/BOOTX64.EFI which
then chains to /boot/efi/EFI/centos/grubx64.efi.
If /boot/efi/$EFI_BOOT_DIR doesn't exist (such as for Ubuntu,
/boot/efi/EFI/ubuntu) the current behaviour of running grub-install to
generate /boot/efi/EFI/BOOT/BOOTX64.EFI will continue. For distros
such as Ubutnu where packaging does not populate /boot/efi/EFI/ubuntu
with .efi files, secure boot can be added in the future by copying
.efi files to /boot/efi/EFI/ubuntu and copying the shim file to
/boot/efi/EFI/BOOT/BOOTX64.EFI.
Change-Id: I90925218ff2aa4c4daffcf86e686b6d98d6b0f21
The path for get-pip.py script in version 3.5 has been changed
with this commit [1].
[1] 2360f025eb
Change-Id: Ifde16e40b4e241c6c4c93df44330c403ee903e6f
As of grub2 >= 2.02-95 calling grub2-install on an EFI partition will
fail with: "this utility cannot be used for EFI platforms because it
does not support UEFI Secure Boot." In the case of ironic deployments,
ironic-python-agent will call efibootmgr to set the local boot for
subsequent boots.
This change adds efibootmgr to the image as well so any other UEFI
menu changes can be made manually.
Change-Id: I765ded15da07d6227d1e337960e54ad0e0d6ca39
The python3/python3-pyyaml packages both are never installed and dnf
itself never updated when $DIB_DISTRIBUTION_MIRROR set and used.
This change fix the order of the operations:
1. yum/dnf configure.
2. *.repo patching.
3. yum/dnf update/install execution.
Change-Id: Ifbbf1f0190fe8c8a77fb3be820e8056447e755f6
Signed-off-by: Maksim Malchuk <maksim.malchuk@gmail.com>
The trvial fix allow override to work in air-gapped envirments where
the command 'curl -s https://cloud.centos.org/...' would fail.
Change-Id: I84296d8816042e4cd4cb02f15746b86d600d13d6
Signed-off-by: Maksim Malchuk <maksim.malchuk@gmail.com>
Use the actual virtualenv to install proliant tools, and make sure the
dependencies needed for sum firmware upgrade are included.
Change-Id: Ie1dbb868450918567b2903cdeccda35af1904417
Closes-Bug: #1916346
This PR updates locations for files used by the bootloader depending on
the target operating system built. The current logic does not take into
account latest versions of operating systems and makes it impossible to
build ISOs against those.
With this change it is possible to correctly build CentOS 8, Ubuntu
18.04 and Ubuntu 20.04 images.
Closes-Bug: #1916913
Change-Id: I3ed0041640f539e82805d03ba26fe46217f3ac3c
The only difference between the rhel and redhat entries is rhel has
the extra grub-efi-x86_64 mapping. All redhat family releases would
benefit from having this too, so this change removes the whole rhel
entry and adds grub-efi-x86_64 to the redhat family.
The assumption is that anything which applies to rhel also applies to
centos-stream, and in this case doesn't harm centos or fedora either.
Change-Id: I0dc44c1f2b57516742f4c3e43cfc8874d6b90fa2
This package doesn't exist in the stream base repo, and neither does
centos-linux-repos.
These are presumably replaced by centos-stream-release and
centos-stream-repos. This change adds an else block to handle the
non-stream base packages.
Change-Id: I32249199c3dfa44fc24fba28d24f314112c2e200
This change will remove any existing interface configurations in the image. They are not necessary
and could interrupt with deployments. In any case they should not exist if we use
dhcp-all-interfaces element.
Change-Id: I35a4b5ea6e2315de3b0d9f8353ac2b6f4b995697
Set eus repositories if REG_RELEASE is set instead of the base repos
as the current behavior is to use the non-EUS repositories for RHEL
8.2 deployment which breaks image building for customers.
Change-Id: I8e687b27922c3f6fc3d69794866795ab89ecc346
The get-pip script does not work well with Python 2 and
it raise error during the installation [1].
[1] https://github.com/pypa/get-pip/issues/83
Change-Id: I3755c34da313ef647547c6ae18b59cc04c2cdd60
portage now generates /etc/python-exec/python-exec.conf based on the
order of PYTHON_TARGETS in /etc/portage/make.conf
fixes an issue where ARCH was being detected as amd64 not x86_64
fixes kernel installs (virtual/dist-kernel)
standardizes simple if statements (note, the 'shorthand' method will
pass the exit code back to shell but the 'longhand' does not).
Change-Id: I74041c232bc6ab4d6e67a4ecfaa759aa4a5feb6c
Signed-off-by: Matthew Thode <mthode@mthode.org>
* Add "DIB_UBUNTU_MIRROR_DISTS":
Default: ``updates,security,backports``
Notes: For some deployment, is may be required
to disable backport|update|etc packages
integration.
Change-Id: Ic7dcd29ea658a66763b4422915149e4d3fe663cc
Despite having several issues (like missing firmware), it is still
used by people. It seems that the only way to stop that is to remove it.
Change-Id: I4baed8e8ab663c624dcc8d06ff0293d57b082abb
This use the same workaround as element bootloader's[1] to fix CentOS
8.3 partition image building error with iscsi-boot:
...
2021-01-12 07:11:02.439 | + grub2-mkconfig -o /boot/grub2/grub.cfg
2021-01-12 07:11:02.654 | /usr/bin/grub2-editenv: error: cannot rename the file /boot/grub2/grubenv.new to /boot/grub2/grubenv: No such file or directory.
2021-01-12 07:11:02.665 | Generating grub configuration file ...
2021-01-12 07:11:03.112 | /usr/bin/grub2-editenv: error: cannot rename the file /boot/grub2/grubenv.new to /boot/grub2/grubenv: No such file or directory.
...
[1]: https://review.opendev.org/c/openstack/diskimage-builder/+/750279
Closes-Bug: 1911120
Change-Id: I2de5444f7e1a145df9abb03fa4c367e8bb914e03
This fixes below building error When build centos with element
dracut-regenerate on ubuntu by specifying the initrd and kernel version.
...
2021-01-12 03:44:15.758 | dracut: Cannot find module directory /lib/modules/5.4.0-58-generic/
2021-01-12 03:44:15.759 | dracut: and --no-kernel was not specified
2021-01-12 03:44:15.765 | Traceback (most recent call last):
2021-01-12 03:44:15.766 | File "/tmp/in_target.d/finalise.d/50-dracut-regenerate", line 102, in <module>
2021-01-12 03:44:15.766 | main()
2021-01-12 03:44:15.766 | File "/tmp/in_target.d/finalise.d/50-dracut-regenerate", line 82, in main
2021-01-12 03:44:15.766 | raise e
2021-01-12 03:44:15.766 | subprocess.CalledProcessError: Command '['dracut', '--list-modules']' returned non-zero exit status 1.
...
Change-Id: I87ae20b3fa2f291bb107e607137fcd7b1c0a4996
While looking at DIB logs, it is very hard to debug dracut
issues due to missing modules, listing dracut modules will
give an insight.
Related-Bug: #1907457
Signed-off-by: Chandan Kumar (raukadah) <chkumar@redhat.com>
Change-Id: I4d277bdfd648adba5a749d22d905c66b807e249a
With Centos 8.3, centos-repos package has been replaced by
other packages [1].
[1] https://lists.centos.org/pipermail/centos-devel/2020-September/056069.html
Also Increase flake8 and pyflakes version in lower-constraints.txt as
this was already broken.
Change-Id: Ife139fcaff0c2d944098ea353259971d2d3f18b8
The git package pulls in heavy dependencies for just a few rarely
used commands. git-core should be enough (and already uses for Suse).
Change-Id: I96b71072c22c26b3b651466053b9e9561527cbe5
Modern distros use more creative interface naming, e.g. CentOS 8
adds both eth0 and ens3 (!). Remove everything.
Change-Id: Ibdebdb09ea790787840cf9b817d4eb549ef18249
Grub2 messes up arguments with double quotes in them, changing from
key="value" to "key=value" [1]. Support this format as well.
[1] https://bugzilla.redhat.com/show_bug.cgi?id=950760
Change-Id: Icf4c9f86009d29a342d6e0e21644af680066b0b2
Autoremove can touch packages, that are assumed to be present
for post-install step.
A good example is dkms, which is dropped in
install.d, but dkms element try to use it in post-install step.
This results in the following log [1]
[1] http://paste.openstack.org/show/797809/
Change-Id: I635af230c6b250fee273039935cf19506e83b3d1
This is never called externally to dib, so doesn't need to be an
entrypoint. Call it from within dib using the running python
executable and from the lib/ directory; this means we do not need to
have the virtualenv activated to run disk-image-create.
Change-Id: Ie9b551824792864402b0c63ccc350dc5c92dcc3f
This is really an internal dib tool. Move it to the lib directory,
and call it with the python we are running under.
This is one less reason to require the virtualenv to be activated when
you run 'disk-image-create'.
Change-Id: Id689683a0b1fdcb446b04ba967284a216133d743
Provide ability to run dhcp client on VLAN interfaces created on
top of an Ethernet interface. See also
<https://storyboard.openstack.org/#!/story/2008298>`__ for further details.
Change-Id: Ic3ffd7b8e23b1e996cfe6c79ce0ff47e521f30be
centos-repos package is not available on centos-8-stream:
2020-11-02 15:35:20.962 | No Match for argument centos-repos
2020-11-02 15:35:20.992 | No Match for argument centos-release-stream
The build fails later on install_pkg_manager sed command:
2020-11-02 15:35:22.903 | sed: can't read
/var/cache/nodepool/dib_tmp/dib_build.iPSHptNW/mnt/etc/yum.repos.d/*repo: No
such file or directory
This change ensure centos-stream-repos package is installed to provide
/etc/yum.repos.d/*repo files
This change replace 'centos-release-stream' package with the new
'centos-stream-release' package [1]
[1] https://lists.centos.org/pipermail/centos-devel/2020-September/056069.html
Change-Id: I6c397bf7b5797a02e5f006c18ee63c9cdf66b38c
We are running yum-config-manager/dnf config-manager in the epel
element. Even though the yum-utils package is declared in the
package-installs.yaml file, the package-installs pre-install.d
script is executed after the one in the epel element, so image
build fails.
This commit ensures yum-utils or dnf-plugins-core are installed
before running the command.
Change-Id: Ib292b0b2b31bd966e0c5e8f2b2ce560bba89c45c
Initial patch for I78d7bcf214a45245e2073428120fcbdd968e1acd
works without the envvar set, however, 'set -eu' causes it to break
if unset.
This makes the module configuration variable not required to be
set, consistent with other DIB scripts.
Change-Id: I5ca80f518d0371a18c107c061dc923876463af57
Signed-off-by: Lon Hohberger <lhh@redhat.com>
This disables growpart module in cloud-init, not resizing / partition
to maximum disk free size by default.
Change-Id: I69984a9141fa8abb12dc5d51bd334f9280deca67
Originally it was added for missing python-cheetah dep
for openstack-nova. Nova has removed usage of it long
ago with [1]. rhel-7-server-rh-common-rpms should be
disabled once it's usage is over as it packages from
it can conflict with other openstack repos.
If some package which is needed by OpenStack Packages
is missing then instead of adding rhel-7-server-rh-common-rpms
repo consideration should be to add it in RDO.
[1] https://review.opendev.org/#/c/40205
This reverts commit c7219a5a60.
Change-Id: Iad3a1c353c10bb35f9c9ef4076b65f5c84b803b2
This file is present in both dpkg and debian-minimal element,
causing a failure to build anything with debian.
Change-Id: I8213d581a79bb432281f31955a44418e4047d9e1
Some OpenStack releases on RHEL require specific modules
in order to function correctly. This adds the ability
to set DIB_DNF_MODULE_STREAMS which then are selected
prior to package installation.
Change-Id: I78d7bcf214a45245e2073428120fcbdd968e1acd
Signed-off-by: Lon Hohberger <lhh@redhat.com>
e2fsprogs pulls in info which needs gzip. But that conflicts with the
preexisting busybox-gzip. Install normal gzip to ensure that things can
proceed without conflict.
Change-Id: I3bed5bc141eaa0f3a15bfbf9da3e2d6e7c964a76
busybox-grep is pulled in by the base image and it conflicts with rsync
package installs because rsync needs grep proper. Install grep proper
prior to installing the base distro to avoid this issue later when
trying to install rsync.
Change-Id: I2ec2bf39a2af214443a93776231657b25035e54f
The rhel8.2 .qcow2 images have moved from a single xfs partition into
an EFI style layout with separate /boot/efi partition.
This means the root partition has /boot/grub2/grubenv as a symlink to
../efi/EFI/redhat/grubenv, which is dangling when we are running the
bootloader element as /boot/efi is blank. grub-install tries to call
rename() in the process of re-writing this file, which fails and bails
out dib.
Remove this symlink if it exists
Change-Id: I5591144a3617dbae148b0c5d2a6a404942ffcd4e
Parital-Bug: #1893029
Redhat-Bug: https://bugzilla.redhat.com/show_bug.cgi?id=1871669
I088fc4284e889147ca9a375d4a159264cff53484 tried to slot the python3
install between the 00-dnf-update and before 01-package-installs;
however it also needs to run *after* the RHEL subscription
00-rhel-registration.
Thus a better place for it is 01-00-centos-python3, which will order
it after subscription and package updates, but before any use of
package-installs.
To avoid confusion over naming, move 00-0-dnf-update back to just
00-dnf-update.
Change-Id: Ib7c82895769e4889d47e10c4b37e06a42c053903
This avoids having to have gnupg2/apt-key dependencies in the base,
and is now well supported by modern Debuntu.
Signed-off-by: Matthew Thode <mthode@mthode.org>
Change-Id: I7065b2fab6125d9635ef99ff65d374b8b6b4c3a2
This patch makes iscsi-boot element support not only just
DISTRO_NAME centos7 but also centos and centos-minimal.
Change-Id: I8db8b01f35b2e572666badd8d2316d24a5e4287a
Update an rc-update call to only be made if running openrc instead of
all gentoo profiles (systemd does not have rc-update).
Add python3-pyyaml package mappings.
Update serial console to support multiple arches.
Update open-iscsi and open-isns keywords (looks like upstream merged
some musl fixes).
Update the kernel and initramfs file name globs for the
gentoo-kernel-bin usage.
Change-Id: I259bffed3a3e3f92be2210ead6bdfa383917d457
Signed-off-by: Matthew Thode <mthode@mthode.org>
This breaks having git as a package-install in other elements. And results in diskimage-builder always erasing git on the image.
This reverts commit 3c65025559.
Change-Id: Ifb18c4243aa08055d98602aa4fdb30b16dae061d
Adds:
1. grub-efi package mappings
2. efi-64 support
3. default (openrc) arm64 profile
4. systemd arm64 profile
Cleans up the keywords and use flags in 02-gentoo-02-flags. Most stuff
was stablized. Also cleaned up some formatting for the if statements.
Enables less trusted overlays (up to the end user to verify).
in 10-gentoo-image I cleaned up some bash lint things as well.
using && instead of -a and avoiding $?
Change-Id: I3dffe1aab4bbdc4946a9bf2269bf0cde49529a4e
For Bios and EFI compatibility, grub must be installed twice.
This patch adds the bios version when EFI is selected. The GPT EFI block partitioning
already adds the bios partition, but the bootloader only called grub once.
Change-Id: Iee6c8b3b97b3cfff4562bcb30a50800f5ade894a
Closes-Bug: #1889089
NetworkManager takes a distinctly differnet network management
approach and the bulk of the dhcp-all-interfaces code is largely
targetted at distribution specific configuration. Some which may
or may not override settings, or only partially assert desirable
settings.
As such, we need to set appropriate configuration, such as the
correct client to be used, and timeouts based upon user supplied
settings.
By default this change sets the client to be dhclient on redhat
styled machines, as the packaging default, while it works for
ramdisk usage, it does not reset the interface between retries,
which can be critical if the infrastucture operator is attempting
to configure LACP trunks to the end node.
Change-Id: I0e0cfbdbf7ef2b2861b934ccd7dab9d83a35c8f0
Story: 2008001
Task: 40648
Git is a build-time dependency that should not end up in the final
image should source-repositories be the only element requiring it at
image build time.
Change-Id: Id0798c0a753f893d02c2d13fddea0c04b28cb7ca
This patch adds support for CentOS 8 Stream [1] to the centos element
(cloud image). Users should set DIB_RELEASE=8-stream.
[1] https://www.centos.org/stream/
Change-Id: Ib8f542031c46326ffed812fa60cbc9e56db9d6fd
glance-registry service was deprecated in Queens release and has been
removed in Victoria [1].
[1] https://review.opendev.org/#/c/738671/
Change-Id: Id2dfc3455b377b885521556f37f4b0f2da197655
We are at the point that all distributions we are building have Python
3, so any tools running in the chroot can assume Python 3 exists.
This makes dib-python redundant; mark it as deprecated and start to
remove it from elements where it is no longer required.
Change-Id: I5d852843ec65d3b04444b77c54c5b82424455cd8
CentOS 7 is the only distro we support currently that doesn't have
Python 3 installed in some form in the base images. For centos 7 add
an early install of it in the yum element so we can have all the
in-chroot scripts assume Python 3. There is only one package that
causes issues; yaml which comes from EPEL. Everywhere else it is a
base package, but we don't have a way to say "enable epel to install
this". Just hack it in, we don't want to go reworking the world for
CentOS 7 at this point.
Also add python3 and it's yaml library to the centos 8 path. This
brings in the "user" python3 in /urs/bin/python3 (the "system" python3
is already installed). Again, this just lets us assume
/usr/bin/python3 in scripts for all platforms.
package-installs is one of these things running python in the chroot,
and unfortunately we have elements that use it at 01- level in
pre-installd. Thus to make sure python3 is there nice and early, run
it at 0 level, but make sure it comes after yum/dnf update.
Change-Id: I088fc4284e889147ca9a375d4a159264cff53484
Other architectures are stored under "altarch" for CentOS 7, update
the match.
Convert the delimiters to "," to avoid a subtle problem with "|" --
POSIX states
Within the BRE and the replacement, the BRE delimiter itself can be
used as a literal character if it is preceded by a backslash.
So "s|\(foo\|bar\)|moo|" doesn't do what you might think; the inner
pipe becomes a literal | and this will *not* match "foo" or "bar".
Change-Id: Ic1642325e3a59a10453c356d8d839ce649812af8
Simplify gpg checking by caching a keyring instead of keys to import.
Change-Id: I5ed74ec0e12732aec40ef31377e72d7ddc347f95
Signed-off-by: Matthew Thode <mthode@mthode.org>
Now that DIB is python3 only we can remove a hack that made sure
scripts outside the chroot ran with the correct version of python.
This is necessary as python3 does not resolve symbolic links to the
binary like python2.x did, which causes element scripts to fail finding
modules when DIB was run from inside a venv.
This patch does the following:
1. Reverts 9c7b8d1714 which was the
workaround for mixed python2/3 environments.
2. Updates the scripts to use "python3" instead of "python".
Change-Id: If2402bb02fc8a4778fa9434fa167ea1fafd87c28
With the removal of Python 2.x we can remove the unittest2 compat
wrappers and switch to assertCountEqual instead of assertItemsEqual
We have been able to use them since then, because
testtools required unittest2, which still included it. With testtools
removing Python 2.7 support [3][4], we will lose support for
assertItemsEqual, so we should switch to use assertCountEqual.
[1] - https://bugs.python.org/issue17866
[2] - https://hg.python.org/cpython/rev/d9921cb6e3cd
[3] - testing-cabal/testtools#286
[4] - testing-cabal/testtools#277
Change-Id: I870286a2557e41099597c22dc9747743e1077615
EPEL centos8 have different epel repos like
epel, epel-modular, epel-playground etc, so
we need to disable all not just epel.
Also ensure other epel repositories in CentOS7
like epel-testing are also disabled.
Related-Bug: #1885315
Change-Id: I02b3b83fa2047702d5f069d3ca1c9c0bcc1dab52
DIB was retrieving the oldest cloud image file which, presently, means
retrieving CentOS 8.1 instead of CentOS 8.2. Even though DIB runs a
system update and so catches up to latest, this takes bandwidth, time
and final image space (8.1 + system update = 765M qcow2, vs 8.2 + system
update = 518M qcow2).
This patch fixes that by taking the first image name in a descending
order list.
Change-Id: I648fe19f1f76c03c97492b6ac7be6381f6f9261b
This patch adds support for CentOS 8 Stream [1] to the centos-minimal
element. Users should set DIB_RELEASE=8-stream.
[1] https://www.centos.org/stream/
Change-Id: Id0825de735ab957c10daf35fb3c641f850cc6847
Since the original merge of this code
(04208e7c79) several things have
changed; particularly now we ship dib-run-parts as part of dib, not as
a separate package.
We setup $_LIB to point to the shipped library diretory via
pkg_resources lookups. We now call dib-run-parts (as mentioned,
shipped as a dib library now), source scripts, etc. via $_LIB and thus
do not rely on $PATH. Consequently we don't need this activation
part.
Which is helpful, because "venv" (as opposed to virtualenv) doesn't
have activate_this.py. So this fixes installation under that for
Python 3.
We update the functional tests to use the virtualenv_command exported
by the ensure-pip role, which will test the venv path. There is no
need for dib_python as we are Python 3 only now.
Change-Id: Iede929ea2d278008220aac8b1d678ba41eba0d8a
The standard Python venv module does not have this script, so currently
DIB unconditionally fails. While a real fix will be provided in
https://review.opendev.org/#/c/704478/ this change at least allows
users to try work around the problem.
Change-Id: I45b79d4d283f2b3ea909612e652672dcb6092488
This reverts commit 6ee2995214 and
e85c2a6f03.
I missed that if you pip install and then run dib-lint, it's not going
to pick up the .yamllint file shipped here. Thus it gives spurious
errors.
The reason for this was simply better duplicate key detection in yaml
files, which caused us problems with the kernel installs. However, at
this point it seems just the old "does it load" test from pyyaml will
be enough.
Change-Id: I87a9fc9bb119cfeffad48fc0fa0df31f0181825d
The main reason for using the stage4 is now gone (kernel compile).
Install and use the distro provided binary kernel package.
In addition to this, set the locale and timezone, beyond that very
little was done in the gentoo stage4.
Change-Id: I541b7d9b807e2357398ae1c249b1978958dd1137
Signed-off-by: Matthew Thode <mthode@mthode.org>
As of recently, opensuse-minimal images fail to build because of an
error installing the kernel-default package:
> Problem: kernel-default-5.6.12-1.3.x86_64 requires mkinitrd >= 2.7.1, but this requirement cannot be provided
> not installable providers: dracut-050+suse.61.g0fe0e854-1.1.i586[repo-oss]
> dracut-050+suse.61.g0fe0e854-1.1.x86_64[repo-oss]
The problem is there is a recently added package `busybox-links` which
provides a subpackage `busybox-xz` which provides the /usr/bin/xz
utility. Since this is available, the `aaa_base` package installs it
during the root.d base installation phase to fulfill it's dependency on
/usr/bin/xz. On the other hand, the dracut package explicitly requires
the `xz` package, and this is not co-installable with the `busybox-xz`
package, so the dracut package is not installable during the install.d
phase. This change explicitly adds the `xz` package to the initial
chroot provisioning phase so that the /usr/bin/xz requirement is already
fulfilled and `busybox-xz` does not get installed.
Change-Id: Iba8c301eb496657873963e1aa99736aacf87cb00
This reverts commit 6e549c33ac.
It uses the new multiple-parameter matching format from
Idff7b067ad4255e6fc4138f7eff313a81b75c8ba to actually do what it says.
Change-Id: I4656ff1a5c46bcfbd8587f2f541825f4ad08820f
The change Ia6f10741fa6be24b11d6991c8a6b6e07951ff68d introduced having
"when:" as a list of values. However, this was actually not
sufficient to express the logic required for arm64/x86_64/xenial
kernel matching we wanted.
Because the package name is a key, we can't have multiple entires in
the package-map YAML files. This means we can't do more advanced
matching and thus we need to be able to match through multiple
parameters. Similar to Ia6f10741fa6be24b11d6991c8a6b6e07951ff68d we
modify the matching rules to allow a list.
A an example of using this is provided in the README.rst, and this
same example worked through by the unit tests.
This also slightly updates the matching logic to be more sequential.
After each check we either continue on or log the failure and continue
to the next check (rather than set a list of flags then check that at
the end). This makes it much easier to understand what is being
matched in the logging output from the tool.
Change-Id: Idff7b067ad4255e6fc4138f7eff313a81b75c8ba
This gives us better linting of YAML files that just opening them.
This would have detected the duplicate keys in
I34e27d821fbefe274e7b007f37b0bd34db2e1d26.
The .yamllint is taken from zuul-jobs where it is also used as a
fairly sane set of default rules.
A few minor newline fixes are added.
Change-Id: I96d6644ae24f7deb84fa50fefbda0f0d33e0e009
This reverts commit 14ff8f942c.
This seems to not be installing the kernel at all, and needs further
investigation.
Change-Id: Ifd809d4b67aff5d80f979235db246a16af0375b3
Only install the HWE kernel by default for Xenial. This was actually
installing the 16.04 HWE kernel on Bionic by accident, since it seems
to have that package; however it was breaking Focal.
On the other distros, just install the default generic kernel. Let's
KISS for now if we can ...
Change-Id: I34e27d821fbefe274e7b007f37b0bd34db2e1d26
Allow the "when:" statements to be a list of values, which are
effectively anded together to filter the package install.
Change-Id: Ia6f10741fa6be24b11d6991c8a6b6e07951ff68d
Disable all repositories after attaching a pool as in some cases that
I still can't explain, the "-htb-" repositories are enabled by default
which causes problems later on when trying to install the packages.
Change-Id: I86b3baccd4b0932eb3686a17485f64ee09994dad
New versions of flake8 fail as "l" can be confused for "1" apparently
(E741) ... not sure I totally agree but since it's only one instance,
update it.
Change-Id: Ic5c47867facd56b53cc6534da4ae3a345c516202
This should be installing the python2 and python3 packages (that's
what pip-and-virtualenv is designed to do), but we dropped the +=
accidentally in ee9ad32b6f.
However, we've moved on anyway and after
I7a6a342461d6001c25e55638ba9b7438c28f2519 F31 doesn't support this
element. fedora-latest is already updated to f31 in the opendev gate.
Remove the testing as it is no longer relevant.
Change-Id: Id696a90baa1eb05cb4c08501f8dac3665d395682
This showed up with dnf in containers when TMPDIR was set; dnf started
trying to write to this directory while in the chroot.
We already do stripping like this in run_in_target -- but this is a
bit of a unique place because it's actually setting up the initial
chroot so the target doesn't actually exist yet; so we just hard-code
it in place here.
Change-Id: If7310cb820846da903bf60daa4486c8bf7cb0136
This is an alternative approach to commit
68bb43535e. I think this proposes a
better overall solution that the prior change which had the Python 3
packages being installed, but did not specify the _do_py3 flag to do
the installation steps that redirect the various tool installations.
Fedora 31+ doesn't have python2, and Tumbleweed does have some Python
2 support but there seems to be no reason to bother updating this
element for either with infra very close to removing this completely
[1]. Error out on these platforms, and add a release note.
The 15 path should include the python2 and python3 packages, along
with the flags to do the "cleanup"; i.e. forced removal of distutils
packages that pip 10+ won't touch. As mentioned in the original
change, the six package causes problems here, but we can clear that
too by explicitly listing it instead of letting it come in via
dependencies. Again, this element will be removed from the infra 15
builds ASAP; but we can release with this to provide a roll-back point
if we need to revert the removal to fix things temporarily.
Add it to the testing path as well.
[1] https://docs.opendev.org/opendev/infra-specs/latest/specs/cleanup-test-node-python.html
Change-Id: I7a6a342461d6001c25e55638ba9b7438c28f2519
When I tried to build CentOS8 image for AArch64 I got error saying that
MBR is not supported. So make sure that it will not be used by default.
Change-Id: Ib67ab7f808d727c3c61932c540d398dbe723972f
openSUSE Tumbleweed is dropping python2-* packages so we need
to stop intalling them. We can also stop installing those
for Leap 15. which avoids a pip uninstall issue (as python2-six
was still built with distribute).
Change-Id: Ie93c8addb26aab3d0154c4b5b52423799abede91
I don't see anywhere we bring this in, especially on a minimal build.
In 2020 it seems like a base dependency, put it alongside
software-properties-common that installs the other apt helper bits.
Change-Id: I5b079eac4912cb4a164e9aa6158ed106a28f576c
We're ending up with "centoscentos" in the mirror location and the
build fails; strip out the $contentdir from the original too.
Change-Id: If09dbbd8028ea510d2ab0d3d8afe484cea611df5
If the running kernel of the system building the image
matches the kernel that is to be removed dnf will fail.
Repalce use use of dnf with rpm -e.
Closes-Bug: #1623409
Change-Id: Ie2481ea8a02b7b0720e46fa179f24badf4aa25c5
This element is designed to install latest minor versions of different
python releases, like py27, py35, py36, py37, py38
into stow directory, and later easily enable them with stow.
Change-Id: Iab6d20e7643e549b53c629fb430e58b1c5e72991
Sometimes an element needs packages installed so that it can
perform tasks but those package are not appropriate for the
final image content. Add a "build-only" flag to package-install-squash
which will cause package to be installed at the beginning of the
phase and then uninstalled at the end of the phase.
Change-Id: Ie01b795991710c93f6b669c8f14b57eb4412c1d5
All the platforms we care about now have python3 with venv (even
centos7 now) packaged somehow. Add an ensure-venv element to make
sure that "python3 -m venv" works. Any other elements that wish to
install non-distribution-packaged Python utilities can use this to
keep them separate from the main system installs.
Port glean to use this, and drop its dependency on pip-and-virtualenv.
Change-Id: Ic16f134fe34293bb68e7c632dd320f523366320d
This causes problems for other projects incorporating dib; we don't
have a specific need for a cap.
Fix a few issues, mostly spacing or regex matches. No functional
changes.
W503 and W504 relate to leaving artithmetic operators at the start or
end of lines, and are mutually exclusive and, due to "ignore"
overriding the defaults both get enabled. It seems everyone gets this
wrong (https://gitlab.com/pycqa/flake8/issues/466). Don't take a
position on this and ignore both.
Use double # around comments including YAML snippets using "# type: "
which now gets detected as PEP484/mypy type hints.
Change-Id: I8b7ce6dee02dcce31c82427a2441c931d136ef57
* Add "centos" element, a CentOS version-independent element. This is in
line with the same work done for RHEL in Stein cycle.
* Deprecate the centos7 element. CentOS 7 support itself it not
deprecated though. The new "centos" element provides the same support
level as the "centos7" element.
* Add functional testing
The default CentOS version is 8. You can adjust it using the DIB_RELEASE
environment variable.
Change-Id: I373ba2296c4613765676e59aabd9c651345298d1
in CentOS build case building an image with "iscsi-boot" and "dracut-regenerate" will exit building because of statement "[ "$found" = 0 ]"
Change-Id: I1a6d60e9ec5f5cb508866c8376465c3e73551a30
On IPA we are using efivar and efibootmgr, we already added the
packages on ipa-builder.
Adding the pacakges on diskimage-builder, so that people who use
it to build the images won't get into trouble.
Change-Id: I9ab6588f20302b4808b09dc060aced5fd267a3d2
Debian default Python interpreter version is 2.7, but it's
possible to install a Python 3 interpreter from the base
repository.
With this change, if we set DIB_PYTHON_VERSION to 3, we install
the python3 package from base, with python3-libs, python3-pip and
python3-setuptools, and redefine python_path, effectively allowing
Python 3 interpreter to be used in Debian.
See a result of the job for building the ipa image here:
https://review.opendev.org/705773
Change-Id: Idabfa94c2bff6e0de6daa0866084d5db14d7dcb0
When there is a hashsum mismatch diskimage-builder forces downloads
with the -f switch of cache-url. This is currently broken because bash
escapes the quotes in curl_opts. This tricks curl trying to download
'no-cache' instead of the url. This can be fixed by using an array for
curl_opts which does the correct thing here.
Change-Id: Id9f1579dda9a3e0a2b08dd5faaeef0e2e580d419
Add a basic test to ensure that all elements have a README.rst file.
This way they will be exhaustively listed in the Sphinx documentation.
Add dummy README.rst for 'disable-selinux' and 'rpm-distro' elements.
Change-Id: Ia5252ddd89b5ae5c6e9a12a66ef10f912fd54da5
CentOS 8.1 split repositories and GPG keys out into subpackages. This
broke DIB support for CentOS 8.
7e41cef41826a0d73ced
Change-Id: If3de6efa6074e059dc9fdd47c7bdc19d26d4d7f2
The hook inside extra-data.d runs outside the chroot when
building the image which means that we need to prefix paths
inside the hook to avoid running things on the host.
We also run it with sudo because if we're running DIB not
as root, /etc is uid 0 and we'll get a permission denied.
Change-Id: I1838890fe124c84c879285a471bcc78fe47d6c23
Make sure rngd, a hardware RNG entropy gatherer daemon, is installed on
all DIB-built Red Hat family distro images. rngd comes installed by
default in a typical base installation as it's proven to help speed
things up.
Nova attaches the virtio-rng-pci device to VMs. virtio-rng-pci is a
device that provides feed random data. However, it is of little to no
use if the virtual machine is not configured to make use of given
device. That is where rngd can help by facilitating entropy to the pool
from virtio-rng-pci.
$ openstack image set --property hw_rng_model=virtio [...]
$ openstack flavor set --property hw_rng:allowed=True [...]
DIB-built minimal images do not come with rngd installed. This patch
makes sure the daemon is installed. Its systemd service comes already
enabled.
Change-Id: I34a989dbfc57d4c98113ac25c81dfb500945ff0a
Some phases of diskimage-builder run outside the chroot environment,
such as the extra-data.d scripts, and don't have access to dib-python.
This means these scripts may choose the wrong python version by using
"#!/usr/bin/env python" to execute. The svc-map element is an example.
This patch creates a temporary directory and symbolic link for the
correct version of python, then manipulates the environment PATH
to preference the symbolic link "python" command.
This will allow elements with these scripts to work correctly with
the version of python diskimage-builder is running under.
Change-Id: I289d621e1bfbba0eb174dff977d1a5c92c04e4fa
Co-Authored-By: Ian Wienand <iwienand@redhat.com>
As described inline, Bionic hosts will build invalid Trusty images.
Hack around this by disabling metadata_csum in the ext4 mkfs.
Change-Id: Ibd67d58ca830a9e60605d0700ee2b17906c804e6
The base URL of EPEL repository installed by the epel-release package in
CentOS 8 at least now defaults to https.
The error seen when building an CentOS 8 image was:
"Error: Cannot find a valid baseurl for repo: epel"
This patch fixes it so that it will always match regardless of being
http or https.
Change-Id: I9ec5536ee72047c929a1ef6950ff4e9092842a4c
The ndisc6 package is not yet available in EPEL 8.
See: https://bugzilla.redhat.com/show_bug.cgi?id=1779134
Until the package is available set the pkg-map to "" for
the ndisc6 package when distro is redhat and install the
package using || true in the element script instead so
that CentOS 8 build's do not fail because of the missing
package.
Once the package is in EPEL 8 this change can be reverted.
Related-Bug: #1754219
Change-Id: Icd4bad8852ce5ba40fb0e7b0d335191efbe88c67
After the introduction of 'Add output for mis-configured element
scripts' we started seeing CI failures in tripleo where
instack-undercloud is being used (rocky/queens):
/usr/lib/python2.7/site-packages/diskimage_builder/lib/dib-run-parts: line 108: DIB_DEBUG_TRACE: unbound variable
INFO: 2019-12-02 16:24:33,423 -- ############### End stdout/stderr logging ###############
ERROR: 2019-12-02 16:24:33,423 -- Hook FAILED.
Let's make sure that by default the env variable is set
to 0.
Change-Id: I38c76c0edee436f1e7dd0c9a868cea1e6ee3271d
Closes-Bug: #1854904
Without this change, operating system elements that use the
zypper-minimal element always must use download.opensuse.org as their
repository source. This change makes ZYPPER_REPOS overrideable, which
allows the user to create custom operating system elements that can use
private repositories as their source for base packages. For example,
with only this change, it is possible to create a sles-minimal element
that generates a SLE 15 SP1 image just by overriding DIB_ZYPPER_REPOS
and DIB_OPENSUSE_PATTERNS.
Change-Id: I46e40fbe4408d4204056a27b182b21213f1176ff
On openSUSE Tumbleweed, the login.defs config file was moved under
/usr[1]. This change allows the login.defs config change to work for
both old and new locations.
[1] https://build.opensuse.org/request/show/736424
Change-Id: Ia5eff5e7b0709836278361b1b8daa788619eff75
If rdisc6 is available, a node using this element will loops until
DIB_DHCP_TIMEOUT is reached because of a missing 'break' when rdisc6
return code is 0.
This will mark the dhcp-interface@.service unit as failed (because it
has the same timeout) and not bring any network interface online.
Change-Id: I034dcda94d765f236950ebcbee36789f5bdc515f
Closes-Bug: #1854717
Signed-off-by: Hervé Rousseau <hroussea@cern.ch>
Support for easy_install codepaths is increasingly broken, and now
putting allow-hosts in this file breaks most recent pip. Just stop
installing the file - people should be using pip anyway.
Change-Id: I0a6b2432f81d80fbcbb336403fe555003880fa9f
When running under nodepool in a foreground, non-daemonized situation
without a tty (i.e. within a container) we're seeing this "wait" hang
indefinitely.
It is probably related to "outfilter.py" and output file descriptors,
although TBH we haven't completely root-caused it. I won't claim this
is a great solution, but it should hopefully let the dib process
finish and just die, where outfilter will disappear.
Change-Id: If78da54df3d4c240fee16aee4413ec554b37c1d6
The current implementation evauates the dib-init-system
script too early. Also it looks that there is no simple
way of getting the info about the init system automatically:
another element can install (later on) a different
init system. Therefore the only reliable way of setting
this is manual.
Change-Id: I6e9ffa1bdb3154f488f4fd335b197699b86aacd4
Signed-off-by: Andreas Florath <andreas@florath.net>
I commonly get asked for help when people are attempting to create
local image elements and they cannot get them to work.
diskimage-builder silently ignores element scripts that it doesn't
find to it's liking, such as non-executable or files with extensions
(.sh is a common mistake).
This patch extends the '-x' tracing flag down to dib-run-parts and
will cause it to print out helpful messages when these files would
otherwise be silently ignored.
Examples:
Ignoring non-executable files: 10-do-not-run-me
Ignoring non-conforming filenames: 10-I-can-run.sh
I am not enabling these by default as they can create extra noise
and require additional filesystem IO to produce.
Change-Id: Ic804efca3015c199440b4b10da951d71a815c64f
When the rdisc6 utility is available probe for router
advertisement. configure eni and rhel-netscripts interfaces
to do IPv6 address configuration according to the flags
in the RA recived from the router.
The systemd service file timeout is DIB_DHCP_TIMEOUT * 2,
so that DHCPv4 can timout, and dhcpv6 run before the service
times out.
Retries are commented in dhclient.conf, without it we end up
trying DIB_DHCP_TIMEOUT * 60 before the client move on to
IPv6.
WHEN:
Stateful address conf. : No
Stateful other conf. : No
THEN:
Do not run dhclient at all, autoconfiguration via
SLAAC only.
WHEN:
Stateful address conf. : No
Stateful other conf. : Yes
THEN:
Run "dhclient -6 -S", The ``-S`` option makes the
dhcp client not request an address, only other
options such as DNS servers and NTP servers from
DHCPv6 server.
WHEN:
Stateful address conf. : Yes
Stateful other conf. : Yes
THEN:
The dhcp client should request an address _and_ other
options such as DNS servers and NTP servers from
DHCPv6 server.
NOTE: No IPv6 support added for suse-netscripts
Closes-Bug: 1754219
Change-Id: Icdc79875c33f894ab7eaec8afdfb33a731efff99
Currently DIB_ADD_APT_KEYS only supports GPG armor keys, while
default Debuntu apt gpg keys are in keyring format.
Change-Id: I361c375e25b03a08b19052b10c6733939c8df921
This reverts commit a3e9e7f89e.
We still have some issues with vhd creation on RAX
In short, it appears that images fail to resize unless they have a
specific "creator" field. Revert this while we consider the options.
[1] https://bugs.launchpad.net/nova/+bug/862653
Change-Id: I2b6a3bfbfe28432fbb6a2ce4a0211939d224b8d5
The "ironic-agent" is copied to ironic-python-agent-builder and
hence it is deprecated from DIB.
Remove from functional testing
Change-Id: Ibc4f75b9d7e2a31994fc86d05bd57975f00fb74f
Task: 36198
Story: 2005114
This package is not installed by default on Debuntu, but is on RH
platforms. This is causing a build breakage as DIB_PYTHON_VIRTUALENV
tries to use this (I3414fb9e503f94ff744b560eff9ec0f4afdbb50e).
Add the package.
Change-Id: I9a551c57dd128bbb4b095c847f634c777b2cb553
To ensure dracut does not load nouveau we need to explicitly disable it via
omit_drivers.
This change adds a method to drop in arbitary dracut conf files to an element
which are picked up by dracut-regenerate and included in the chroot where we
run dracut.
The disable-nouveau element just adds a conf file with
`omit_drivers += " nouveau"`
The default dracut conf files in /usr/lib include a similar file to omit the
nvidia kernel modules.
Change-Id: I6375e4843fd08d1410141fbbd8658042dcd5ad05
Closes-bug: 1842664
Seeing this at the end of the tripleo overcloud full build:
99-selinux-fixfiles-restore: line 69: [: too many arguments
Change-Id: I8fb10f3d3d38723b41190ae1898757e6df073945
The vhdutil utility is completely dead; the whole subsystem it relies
on was removed with [1] so it's not even vaguely possible to keep it
up-to-date.
I took the .raw images on a nb and used the qemu-img there (so Xenial)
and generated some VPC images; uploaded them to rackspace and the all
seemed to boot fine. If there was a problem, maybe it's been fixed on
either the qemu or RAX side in the previous few years.
Thus swith to qemu-img to generate the vhd images too.
[1] https://xenbits.xen.org/gitweb/?p=xen.git;a=commit;h=5c883cf036cf5ab8b1b79390549e2475f7a568dd
Change-Id: I3099d2ebb958370fcec623087a093b2c8dbdc6c4
Add option to set the suite subpath after the release name for the
security mirror URL independently in the debian-minimal element,
since this can differ between mirrors.
Change-Id: I4cc8f54fba012986423e30e19bff276208b8ad62
With the introduction of centos 8 we have constructs like
if [[ $DISTRO =~ (centos|fedora) && $DIB_RELEASE -ge 8 ]]
This is intended to match the "centos7" element (from the =~) but it
was missed that this is setting the DIB_RELEASE to "GenericCloud".
I think it makes more sense for this to be a numeric release, and
makes constructs like above work. There really isn't any other type
of image to choose here; thus we move it into a new, centos7
specific variable.
Note that when the centos 8 images are available, we want to move to a
generic "centos" element that will handle both 7 and 8 together (same
as rhel) based on DIB_RELEASE and deprecate centos7; this works with
that environment too.
Change-Id: I2e6b7848070d6452c0563e2a122447627c6e6bf7
It turns out that this breaks ipv6 config with NM. Instead what we want
is for glean to not up interfaces on boot (see the depends-on).
Change-Id: I6c5bc76c433e29f02d3266ab8f669015125ec954
Depends-On: https://review.opendev.org/#/c/688031
This adds CentOS 8 into functional and boot tests.
This completes centos-minimal support, documentation is updated and a
release note is added.
Change-Id: I435c2967b4f49faeb6d6edf189907b9f96e80357
As described inline, NetworkManager and dhcp-client make up the basic
networking for centos 8 installs; bring them into the base image.
Although in infra we then use simple-init, some other users find this
helpful.
Change-Id: Ib9f32e73bf9109cc1b659fe1deceb1a15301ffeb
By default network-scripts package isn't installed, so the directories
for these files don't exist either. Skip by default for Centos 8.
Change-Id: I194ec3735e17f27e586386541dc51f775b01e510
Use the wrapper calls from Ia267a60eecfa8f4071dd477d86daebe07e9a7e38
to install glean.
Using this wrapper means we cover all cases without more and more
branches; it should work for python2, python3 and also the special
case of RHEL/CentOS where dib-python points to the special
/usr/libexec/platform-python (which is python3.6 with inbuilt pip)
Change-Id: If624e8bb66ce0761fc0d5f34c2bed8b93a7daeee
