The latest Debian bullseye release doesn't provide yum any more, only
DNF. This breaks the minimal builds that are using on-host yum tools
to start the chroot. Probe for yumdownloader, and if it's not there,
use DNF.
Note this requires "dnf download" which may not be packaged. See
I21cfbd3935e48be4b92591ea36c7eed301230753 for a sample work-around
that installs this plugin in the nodepool-builder container.
Change-Id: Ia7f1e4d115cc67c378d865d91af94a07b8cdc6cc
Add openeuler-minimal element and add CI functional tests for both
x86_64 and arm64.
OpenEuler is an open source community driven YUM/DNF distro like
Fedora. It references Fedora and CentOS a lot for the rpm packages
building. So somewhat it can be treated as a redhat family distro
and reuse the YUM/DNF related elements to help build openEuler images.
For more info about openEuler, see: https://openeuler.org/en
Depends-On: https://review.opendev.org/c/zuul/zuul-jobs/+/803413
Change-Id: I3e06e49b524364c3a4edeba8bce7a8c06b9c7b76
This change permits the yum-minimal element to be used in downstream
custom distributions, which may have additional packages containing repo
config or GPG keys needed.
This could also be utilized at a later time to move the
distribution-specific logic in this method to each distribution element
separately.
Change-Id: Ic1434bb2fe7301086cf11ba6bd7f2ee187c5e6c8
As noted inline, this works around potential issues by being a strong
indication you are in a container (e.g. [1]). Since nothing should be
changing anything on the host/build system, this is a generically
safer way to operate.
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1975588
Change-Id: Ic6802c4ffc2e825f129af10717860a2d1770fe80
This package doesn't exist in the stream base repo, and neither does
centos-linux-repos.
These are presumably replaced by centos-stream-release and
centos-stream-repos. This change adds an else block to handle the
non-stream base packages.
Change-Id: I32249199c3dfa44fc24fba28d24f314112c2e200
With Centos 8.3, centos-repos package has been replaced by
other packages [1].
[1] https://lists.centos.org/pipermail/centos-devel/2020-September/056069.html
Also Increase flake8 and pyflakes version in lower-constraints.txt as
this was already broken.
Change-Id: Ife139fcaff0c2d944098ea353259971d2d3f18b8
centos-repos package is not available on centos-8-stream:
2020-11-02 15:35:20.962 | No Match for argument centos-repos
2020-11-02 15:35:20.992 | No Match for argument centos-release-stream
The build fails later on install_pkg_manager sed command:
2020-11-02 15:35:22.903 | sed: can't read
/var/cache/nodepool/dib_tmp/dib_build.iPSHptNW/mnt/etc/yum.repos.d/*repo: No
such file or directory
This change ensure centos-stream-repos package is installed to provide
/etc/yum.repos.d/*repo files
This change replace 'centos-release-stream' package with the new
'centos-stream-release' package [1]
[1] https://lists.centos.org/pipermail/centos-devel/2020-September/056069.html
Change-Id: I6c397bf7b5797a02e5f006c18ee63c9cdf66b38c
This patch adds support for CentOS 8 Stream [1] to the centos-minimal
element. Users should set DIB_RELEASE=8-stream.
[1] https://www.centos.org/stream/
Change-Id: Id0825de735ab957c10daf35fb3c641f850cc6847
This showed up with dnf in containers when TMPDIR was set; dnf started
trying to write to this directory while in the chroot.
We already do stripping like this in run_in_target -- but this is a
bit of a unique place because it's actually setting up the initial
chroot so the target doesn't actually exist yet; so we just hard-code
it in place here.
Change-Id: If7310cb820846da903bf60daa4486c8bf7cb0136
CentOS 8.1 split repositories and GPG keys out into subpackages. This
broke DIB support for CentOS 8.
7e41cef41826a0d73ced
Change-Id: If3de6efa6074e059dc9fdd47c7bdc19d26d4d7f2
Don't install the "yum" package, which is a backwards compat around
dnf. With 687003f we should not need the backwards compat links any
more.
Add libcurl to avoid conficts with in the curl "-minimal" packages
that happens on CentOS 8. But skip it on Fedora, because it seems to
create more problems there (not going to pretend it isn't all a
hack ... but it seems to work).
Change-Id: I1de2703eb5075a0a22837b6898bd8eb960d080dd
CentOS 8 has the "new" split-up locales packages. Fedora 24 is now
long gone, so take out the old branch and apply the lang package
install to Centos 8 as well.
The manual locale cleanup is not necessary on Centos 8; skip it.
Change-Id: Ib65fc15fe471348793fd6efb034517f11abd905e
The repo format has slightly changed for CentOS 8 (s/os/baseos/).
Make the chroot builder look for a more specific repos.d directory
first named for the distro variable, then fall back to to top-level
dir (this avoids having to constantly change fedora).
Update the gate mirror setup and roles for new Centos 8 paths too.
Change-Id: I5b7f0c3624cac1d7aa7ed8bf6286b85d808b9c9a
This patch adds option DIB_YUM_MINIMAL_EXTRA_REPOS to yum-minimal to
allow DIB users to include extra repositories to their final image.
Change-Id: I89549f4b0f4c9470143b5064817acab5043e31c5
Something (possibly [1], but that change is at best cryptic) has
changed such that we don't get correct /etc/os-release files
installed. This flows on to grub half-installing itself, enough to
not fail the build but not enough to make something bootable.
Installing the -cloud release package gets it back, and seems like a
sane choice for dib.
[1] 617b1bed34
Change-Id: Iff0413887fad798273b2bfcb140cc07f36d54a04
It looks like fedora-release on fedora 30+ has been split into sub
packages. Use fedora-release-common to avoid package conflicts.
Change-Id: I8f8711044fc4074b91939e0a6dfdac4d7a14a35b
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
In fedora-30 is when we migrate to dbus-broker, fedora-29 is still using
dbus-daemon.
Change-Id: I1e1d3a3826157b8b22386c211eaa58b6439b5f3c
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
As described in the comments, it seems the transition between
dbus-daemon -> dbus-broker in Fedora 29 has made it so the packages
can get into a state where neither service is enabled.
Explicitly install and enable dbus-broker for F29
Change-Id: I06753043a75be2f635653899c6c251b9fbdd7c67
As described, Fedora 27 has a curl-minimal package that comes in to
satisfy the rpm package dependency. It conflicts with the "real" curl
package -- which is so commonly installed (by infra elements, etc)
that this becomes an annoying problem. Just pre-install the full curl
package.
Fedora 24 is old enough to not worry about, so remove some old
workarounds to make the flow a little simpler.
Change-Id: I67baf96377109ac4521ba00243a0d91b35fafba0
The current implementation - as introduced in
Iee44703297a15b14c715f4bfb7bae67f613aceee - has some shortcomings / bugs,
like:
* the 'grep' check is too sloppy
* when /dev/pts is already mounted multiple times the current implementation
fails:
$ mount | grep devpts | sed 's/.*(\(.*\))/\1/'
rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000
rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000
rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000
* code duplication
* Using the undocumented and non-robust output
of 'mount'.
This patch fixed the above problems.
Change-Id: Ib0c7358772480c56d405659a6a32afd60c311686
Signed-off-by: Andreas Florath <andreas@florath.net>
This is a continuation for f2cc647dae ("diskimage_builder: lib:
common-functions: Fix options for devpts mount"). We also need to
respect the devpts mount options when the dib elements are mounting
this virtual filesystems themselves.
Change-Id: Iee44703297a15b14c715f4bfb7bae67f613aceee
In a couple of places we use flock for critical sections, but we leave
lockfiles around in various locations which can be confusing.
Introduce DIB_LOCKFILES global (under ~/.cache/dib/lockfiles) and
write lockfiles in there.
Fix up removal of the lockfile in the yum path; we just want to make
sure we cleanup the .rpmmacros file, but we don't need to remove the
lockfile as well.
Co-Authored-By: Andreas Florath <andreas@florath.net>
Change-Id: Ie810b2836be521325afe923708d046112e1e1e20
Currently a bind is used when mounting /dev/pts in chroot.
This leads to problems - especially when running DIB in parallel:
It was observed that the /dev/pts mount vanishes from the host
system.
This patch uses '-t devpts' - as it is done for /sys and /proc -
for handling /dev/pts.
Change-Id: Id7775ae6fca6502af800e7b73a00862ef320206b
Signed-off-by: Andreas Florath <andreas@florath.net>
As described in the referenced bug, the dependency solver in yum
doesn't handle weak dependencies well and in some cases, such as
Fedora 26, can end up choosing coreutils-single (the busybox-esque
single binary) instead of actual coreutils, which then causes problems
with conflicting packages later.
Change-Id: I2907bf3b74c146986b483d52cc6ac437036330b4
fedora/centos-minimal don't obey DIB_DISTRIBUTION_MIRROR currently. I
don't really want them too -- we want to be able to separate the
mirrors used during the build process from those embedded into the
final image. Add DIB_YUM_MINIMAL_BOOTSTRAP_REPOS which is a directory
with repo files to use during the install.
This introduces setup-gate-mirrors.sh which is intended to setup
repo/sources/whatever files in the openstack gate that point to the
local region mirror. It pulls the info from the mirror_info.sh script
on each CI node.
The openstack-ci-mirrors element is updated to export these variables.
elements are updated to depend on it. Tests are restored
Change-Id: I7604fc4d41cb1483be16b8d628a24e8fc764f515
Currently we have all our elements and library files in a top-level
directory and install them into
<root>/share/diskimage-builder/[elements|lib] (where root is either /
or the root of a virtualenv).
The problem with this is that editable/development installs (pip -e)
do *not* install data_files. Thus we have no canonical location to
look for elements -- leading to the various odd things we do such as a
whole bunch of guessing at the top of disk-image-create and having a
special test-loader in tests/test_elements.py so we can run python
unit tests on those elements that have it.
data_files is really the wrong thing to use for what are essentially
assets of the program. data_files install works well for things like
config-files, init.d files or dropping documentation files.
By moving the elements under the diskimage_builder package, we always
know where they are relative to where we import from. In fact,
pkg_resources has an api for this which we wrap in the new
diskimage_builder/paths.py helper [1].
We use this helper to find the correct path in the couple of places we
need to find the base-elements dir, and for the paths to import the
library shell functions.
Elements such as svc-map and pkg-map include python unit-tests, which
we do not need tests/test_elements.py to special-case load any more.
They just get found automatically by the normal subunit loader.
I have a follow-on change (I69ca3d26fede0506a6353c077c69f735c8d84d28)
to move disk-image-create to a regular python entry-point.
Unfortunately, this has to move to work with setuptools. You'd think
a symlink under diskimage_builder/[elements|lib] would work, but it
doesn't.
[1] this API handles stuff like getting files out of .zip archive
modules, which we don't do. Essentially for us it's returning
__file__.
Change-Id: I5e3e3c97f385b1a4ff2031a161a55b231895df5b
