Every run we are doing a full tar.gz of the chroot environment that
never gets used.
It's not suitable for CI since we use fresh images each time there.
The cache in general isn't really isn't a very safe thing to have
around, because there's no invalidation procedure and no real way to
make one -- we've no guarantee that a new chroot build even moments
after a previous one wouldn't bring in or different packages, etc (of
course this is *unlikely*, but the longer you go between builds the
worse the problem becomes. Also, tons of packages get installed after
this not from any cache, so potential speed-up is rather marginal.
Debian turned this off with I58fc485aacacaa17243bf9ce760ed91256d1f182.
However, given the reasons above and it's complete lack of testing, I
don't see this as useful.
If we really want this type of thing, I think we should come up with a
way to use a persistent external yum/dnf cache that yum/dnf keeps in
sync with it's usual invalidation rules.
Change-Id: I66789c35db75c41bc45ea1ad2e26f87456de4e4d
It seems in the grub cleanup in
Iafe3611f4eec3c6357587a6cae6a30a261686ead I managed to unintentionally
drop systemd from the yum-minimal builds. By not pre-installing grub
we dropped some dependencies; the path is tortured ... grub2 ->
os-prober -> udev -> systemd-udev -> systemd (we don't even want
os-prober! So this whole thing was working by accident).
This manifests in *very* confusing ways.
Currently centos-minimal builds are failing late in the build with
services unable to enabled. dib-init-system was actually trying to
tell us that it didn't know what init was installed (because systemd
wasn't actually installed), but unfortunately it was not really
failing. This meant the service files were not copied correctly from
other elements, and thus fail to be enabled. I have corrected this
with I076c08190d40c315ad6a6d96a3823e9fc52630be which would at least
alert us earlier.
For Fedora 24, due to a bug in dracut dependencies [1], missing the
systemd-udev package fails the build of the initrd during the kernel
install. This then results in an initrd-less, unbootable system (see
also Ibaaa81124098f3c6febe48e455d3e1cd0a5f1761).
Add these dependencies explicitly.
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1398505
Change-Id: I24ce648485c3d6f3c27ab8f87a638516b3727017
We have a pkg-map entry for lsb_release, but in package-installs.yaml
we refer to the actual package name instead. This will happen to
work on Red Hat platforms, but it's actually wrong.
Change-Id: Idb248f96e75fa1090422fa08e5fbb2385cc1f517
yumdownloader has to have all the repo XML files, etc, which adds up
to a not totally insignificant 150MiB or so. Currently we're leaking
this directory for every build, which adds up on regualar builders
like nodepool.
Isolate the call with a separate TMPDIR so we can clean it up after
the initial download.
Change-Id: Ic65e8ca837cc76b7a1bb9f83027b4a5bdd270f75
Because we are using the building platform's "yum" to do the initial
install into the chroot, it is affected by the base-system's
/etc/yum.conf.
pip-and-virtaulenv in I82acb865378a0fa5903a6267bfcee0e2962eced0 added
"exclude=python-pip..." in /etc/yum.conf to stop the package manager
overwriting the installed pip. Now our CI images have built with
this, we are now picking up this exclude on centos. Since on F24
dnf->python->python-pip we end up failing to build the the chroot
because python-pip can not be satisifed. In a general sense, however,
this could be caused by any configuration put into /etc/yum.conf that
is incompatible with installing into the chroot.
yum has the option to disable all excludes which is used here. This
seems to be the best way to isolate the chroot install from any
excludes that may have been done on the base system for various
reasons. I did consider using a completely separate yum.conf we ship
with dib ... but let's start simple.
This should fix the current gate failures on centos
Change-Id: I4e4cc8ed09a29c4057ade34ea93025139e191bf5
yum-minimal installs selinux but not libselinux-python, which makes
interacting with the node from ansible hard fail. Add it.
Change-Id: I403e7806ae10d5dd96d0727832f4da20e34b94c7
Kernels are built with auditing support, and without the audit deamon
logs bubble up to spam the console and /var/log/messages. This
package contains the audit daemon that catches these messages.
Change-Id: Ie3e216bab33b27f2d67a9379ddc3e89d66449251
We should be doing more to ensure initial configuration during
configuration. Taken from the steps done by [1], here we set
locale.conf and a general timezone.
The only reliable UTF8 locale is en_US.UTF-8; we don't want to use C
locale as it causes havoc with things like python3 and unicode. We
set locale.conf to this.
For Fedora 24 ensure we install the en_* locales too (this is really a
bug separate to this -- when you log in, by default ssh tries to copy
over your locale env variables, so logging into a F24 system would
result in using invalid locales for the most common en_* cases).
While we are here, setup a timezone link. It turns out infra puppet
overwrites this later, but at least we have a sane default.
[1] https://www.freedesktop.org/software/systemd/man/systemd-firstboot.html#
Change-Id: Ib8951a97f1772bc5228c682e88628ff53400a923
Fedora 24 has split locales into separate packages. Testing revealed
what is possibly a bug in the choosing of default packages, so add a
small work-around to ensure the minimal locale pack is installed.
This appears to be the only change required for fedora-minimal with
Fedora 24; at least to build with the project-config infra elements.
Change-Id: I64438c34c572ed96211384ae1bfb45b2949e4318
It turns out our manual locale cleanup is causing issues (see
I54490b17a7f8b2f977369044fcc6bb49cc13768e). Upon further
investigation, I think this is a better approach than manually
deleting repos.
glibc on Fedora obeys the %_install_langs macro for reducing the
installed locales (as mentioned in the comments, F24 has moved to
having different packages, but worry about that later).
So our existing clear-out is really only required for CentOS, whose
glibc does not have any way to indicate to build less locales.
However, %_install_langs is still correct there, as it restricts some
of the translation files and other things installed with the %lang
macro in spec files.
This is complicated by us having to set this at glibc-common install
time, which happens with the "yum" from outside the chroot (i.e. on
trusty). Since this is too old to have flags to pass this, we need to
fiddle with rpmmacros.
I've tested this with fedora-minimal builds and the locales file is
about 2MiB, which is what it was after the cleanups, and the listed
locales are only those we expect (i.e. it appears to be working).
Change-Id: I528a68beeb7b2ceec25ccbec1900670501608158
Export FS_TYPE from img-defaults and use it to remove hard-coded
defaults in the debootstrap mounting. Also, cleanup the suse element
as it should have access to the exported variable.
Change-Id: Ie9b671ca9336060a5ad294be48aa7eff442bf066
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
As motivation for this; we have had two breakouts of dib in recent
memory. One was a failure to unmount through symlinks in the core
code (I335316019ef948758392b03e91f9869102a472b9) and the other was
removing host keys on the build-system
(Ib01d71ff9415a0ae04d963f6e380aab9ac2260ce).
For the most part, dib runs unprivileged. Bits of the core code are
hopefully well tested (modulo bugs like the first one!). We give free
reign inside the chroot (although there is still some potential there
for adverse external affects via bind mounts). Where we could be a
bit safer (and could have prevented at least the second of these
breakouts) is with some better checking that the "sudo" calls
*outside* the chroot at least looked sane.
This adds a basic check that we're using chroot or image paths when
calling sudo in those parts of elements that run *outside* the chroot.
Various files are updated to accomodate this check; mostly by just
ignoring it for existing code (I have not audited these calls).
Nobody is pretending this type of checking makes dib magically safe,
or removes the issues with it needing to do things as root during the
build. But this can help find egregious errors like the key removal.
Change-Id: I161a5aea1d29dcdc7236f70d372c53246ec73749
Rather than removing all locale related stuff in cleanup, strip the
locale archive and rebuild it.
Building just en_US (along with POSIX/C) brings things inline with
debootstrap. As discussed in the bug referenced, this is about the
best we can do for Centos7.
Fedora 24 has split languages out into packages so we don't have to do
this, but I have not dealt with that yet. A guard is put in place so
we make sure we revisit this when we try to build F24.
Change-Id: I3f384d23e52effd6a09f47134746caa4a5c586be
If the initial yum install into the chroot fails, we can leave behind
a lockfile and an incorrectly modified rpmmacros.
Change this so we run the cleanup unconditionally.
Change-Id: Ia9f9c4c845e5f34d33ff9a4ab7226c9175283757
I have seem some occasional odd failures coming from the "dnf -y
update" done by elements/base/install.d/00-up-to-date.
dnf seems to sometimes think a package is not installed when it really
is. It then seems to try and re-install them, but notices they are
installed, and then bails with a failure exit [1]. The packages that
seem to cause this vary, but the common thread is that they seem to
have all been installed during the initial phase of installing the
package manager in the chroot.
I suspect that when we are building the chroot, we do our initial
install with the "external" yum & rpm. Then we start using the
dnf/yum in the chroot, but we're actually using meta-data created by
the *external* tools -- which could be vastly different versions or
who-knows-what. While I honestly I don't have an exact root cause,
empirically I've found rebuilding the rpm db always seems to fix
things up.
So this change takes care to rebuild the rpm db with the chroot
version of rpm, and clear out the package metadata for a refresh with
"update". This should hopefully put us in a consistent state.
[1] http://paste.openstack.org/show/487356/
Change-Id: I565df23897ae511356c4861fdbe63823fa6b6ff9
We were getting some subtle issues in fedora-minimal builds that
turned out to be because /var/run was not a symlink to /run.
Upon further investigation, it turns out that yum is creating a
/var/run directory for it's pid file when it starts working in the
empty chroot (which I verified by stracing it)
---
5905 stat("/home/ubuntu/tmp/dib-tmp/image.Ac4VZZsl/mnt/var/run", 0x7ffddffa0330) = -1 ENOENT (No such file or directory)
5905 mkdir("/home/ubuntu/tmp/dib-tmp/image.Ac4VZZsl/mnt/var/run", 0755) = 0
5905 open("/home/ubuntu/tmp/dib-tmp/image.Ac4VZZsl/mnt/var/run/yum.pid", O_WRONLY|O_CREAT|O_EXCL, 0644) = 6
---
Because this happens *before* we install "filesystem" (the package),
we mess up it's symlinking.
To work-around this, pre-install the trio of base packages (setup,
basesystem, filesystem) with rpm from outside the chroot.
Change-Id: I411b6ec9d91d95d3a0f98e76853086af3b70abe8
Install selinux policy packages as part of the base-installs. selinux
is part of the base-system and the kernel boots by default in selinux
mode.
Without both of these, we can get in a situation where later scripts
(particuarly, some of the infra scripts) might install systemd-policy
without a base policy (targeted), leading to a messed up situation
where systemd will halt during boot due to missing policy files.
Change-Id: I6bf156304d1134fb328fba9b12dc364701b13696
Add an environment variable to control the creation of eth0/1
interface enablement scripts.
With a tool such as glean, the presence of these scripts will indicate
the interface is configured and configuration-drive settings will not
be applied. This means in a non-dhcp situation like on Rackspace,
network is broken.
On Fedora, where later systemd provides "predictable network interface
names" [1] eth0 & eth1 ironically aren't predictable so this just
confuses things. You really need cloud-init or glean or something to
bring up your interfaces in a sane fashion.
This maintains the status-quo on centos-minimal, but disables creation
for fedora-minimal.
[1] http://www.freedesktop.org/wiki/Software/systemd/PredictableNetworkInterfaceNames/
Change-Id: I3f1ffeb6de3b1f952292a144efab9554f7f99a5f
As described in the comment, systemd will create a broken
/etc/resolv.conf link if there is no file in the base-image (as you
can read in the bug, it is debated if this is a bug or a feature).
The solution is to leave a dummy /etc/resolv.conf file in the image.
Whatever network manager you choose (NetworkManager, glean,
cloud-config, etc) will overwrite this anyway.
It's just that some tools, such as dhclient, get confused with the
broken symlink. This affects you if you're using glean to configure
the network in a DHCP situation, for example -- dhclient won't
configure nameservers and everything goes to heck.
Change-Id: I734834d03e7fdb13f9ab2e86f877b07bf4a84ff9
As described in the comments, CentOS overrides the "distroverpkg"
variable in yum.conf. This is the package that yum queries to
establish the value of the $releasever variable. On other platforms,
this defaults to "redhat-release" (which "fedora-release" provides) so
everything works. It is only when the base-system "distroverpkg"
refers to a package not in the chroot we hit the issue.
We can avoid this by setting the releasever variable via the
commandline.
Change-Id: I231c3277960992cd479b8aff7838f246397936f2
When the kernel gets installed on Fedora, the rpm post scripts call
"/bin/kernel-install" [1] to install it. This is a script provided by
systemd.
However, in [2], Fedora ships a patch to kernel-install that makes a
call-out to /sbin/new-kernel-pkg -- the install script provided by
grubby [3]
Without grubby installed, systemd's kernel-install script goes off and
runs dracut plugins directly [4], which eventually creates the initrd.
For reasons that are not clearly explained, the initrd will end up in
a a "machine-id" sub-directory of /boot (possibly, so you can symlink
it?). It is also called "initrd", even though it's an initramfs, for
historical reasons in dracut I think.
It is at this point that I think 99-ramdisk has been written to move
the generated initrd file back into /boot. Later on, when we build
the image, we run grub-install and it picks up the kernel and the
initrd and installs everything.
grubby's new-kernel-pkg [6] it's very similar -- it uses dracut to
make the initramfs ... but in this case it is put in /boot and is
actually called initramfs.
The subtle change that led me down this path is that dracut has been
modified to have a "Recommends" for grubby for >F22 [7]. After
discussing this change with the author, it turns out it was *always*
intended to use the grubby-based kernel install scripts for Fedora --
our builds have been incorrect in not including the package. The
author got sick of people removing the package and making unbootable
systems, hence the change.
Thus this removes the workarounds in 99-ramdisk and replace it with an
install of the grubby package. grubby's kernel install script will
put the kernel & generated initramfs in /boot, and it will be
installed correctly via the usual grub install later when we build the
disk image.
I have built F22 & F23 fedora-minimal images with this and they boot.
[1] http://pkgs.fedoraproject.org/cgit/kernel.git/tree/kernel.spec#n1832
[2] http://pkgs.fedoraproject.org/cgit/systemd.git/tree/kernel-install-grubby.patch
[3] http://linux.die.net/man/8/new-kernel-pkg
[4] https://github.com/haraldh/dracut/blob/master/50-dracut.install
[5] 81516adcb7
[6] https://github.com/rhinstaller/grubby/blob/master/new-kernel-pkg
[7] 47ff68e78b
Change-Id: I1a6e45d04755515286b3d49f8280c16b527e2f48
On Debian/Ubuntu installs of RPM, /usr/lib/rpm/macros sets
%_dbpath %(echo $HOME/.rpmdb)
which makes quite a bit of sense, because RPM is not the system
packager and thus RPM is setup to install things into a hierarchy in
the users homedir.
However, this messes things up when building a Fedora chroot on an
Ubuntu platform.
We use RPM & yum from the base-system to bootstrap the Fedora chroot.
While both obey --root flags, they still pick up the %_dbpath macro
and so end up creating the RPM database in <chroot>/home/user/.rpmdb
After we have bootstrapped yum/dnf, we execute further installation
commands from inside the chroot -- where we now have the Fedora
version of /usr/lib/rpm/macros and hence have _dbpath set to
/var/lib/rpm -- except there is no rpm database there.
Should anyone be finding this in the future, the actual issue that
appears is
$ sudo chroot /opt/dib_tmp/image.b6B5S3f6/mnt dnf makecache
Error: Failed to synchronize cache for repo 'fedora' from \
'https://mirrors.fedoraproject.org/metalink?repo=fedora-$releasever&arch=x86_64': \
Cannot prepare internal mirrorlist: file "repomd.xml" was not found in metalink
Note the issue there is that $releasever is not expanded, because the
rpmdb where this info is kept is not populated.
The trick is to make sure we override this value when using the host
rpm/yum to setup the chroot. The bare rpm calls, which we use to
install the repos, have a --dbpath argument where we can override
this. yum does not however, so we override this in the global
~/.rpmmacros while we are installing the packaging tools and
dependencies into the chroot.
Copious comments are included, because this is super-confusing.
Change-Id: I20801150ea02d1c64f118eb969fb2aec473476f7
fedora-minimal fails to build on Ubuntu Trusty due do being unable to
find the initrd (see Id4c04d7ae20068643df34d2fa31068e8a917a52d).
This is a rather obscure problem that comes from the intersection of
several things.
The first thing to note is that the post-install scripts of the
kernel-core package use kernel-install [1]. For whatever reason, this
installs the kernel to /boot/MACHINE-ID/KERNEL-VERSION
MACHINE-ID comes from /etc/machine-id; a UUID that should have been
created by the systemd post-inst scripts with systemd-machine-id-setup
[2].
The chroot environment provided for root.d elements has no kernel
file-systems like /proc or /dev mounted. This is where differences in
the base-system come into play -- on more recent systems that
implement getrandom() systemd does not need /dev/urandom to generate
the machine-id [3]; we get a value and /etc/machine-id is populated.
On older platforms (Trusty), systemd-machine-id-setup fails (unable to
access /dev/urandom) and we end up with a blank /etc/machine-id. This
ends up making kernel-install (the script) fail during yum's
installation of kernel-core, which means the initrd is not installed
correctly.
We end up bailing out in fedora-minimal/install.d/99-ramdisk, where we
try to put the installed ramdisk in /boot for the later grub install
scripts to find.
The solution here is to mount the standard kernel file-systems within
the chroot before we try installing.
[1] http://www.freedesktop.org/software/systemd/man/kernel-install.html
[2] http://www.freedesktop.org/software/systemd/man/systemd-machine-id-setup.html
[3] https://github.com/systemd/systemd/blob/master/src/basic/random-util.c
Change-Id: Ibcce35da928f64e6a719b070bcc833346ee7ee92
yum-minimal/root.d/08-yum-chroot runs before yum/root.d/50-yum-cache,
and thus if run on a completely fresh system will fail in
08-yum-chroot as the YUM_CACHE directory isn't made.
This is probably hidden by testing & nodepool builds, because it sets
DIB_IMAGE_CACHE. It was hidden from me because locally I have done
builds using the "yum" element previously, which had created the
cache.
Change-Id: I333f5f7e67d198f75a522cc296c118c2e94a5ecb
I'm not sure why we try to do an extra install of these, it is done
inside the chroot in _install_repos. Currently it just gets skipped
saying the packages are already installed.
Change-Id: Ic7aa8cbe13e4347b447e84bb9c12483a4e125228
Add basic F22/dnf support to yum-minimal path. We extract common
code, add some comments and reduce duplication.
Change-Id: If4bd5f88e26bd6f2168958f1ec1efff1072de7ba
Move yum-based install into a function, to make way for a second
related function where use dnf later
Change-Id: Iad09f3753ecdfa0c10cb8a0970a3c8e5a2dccab1
fedora-release >= 22 has acquired a dependency on /bin/sh. This comes
from a %posttrans section of the spec file, which is symlinking the
os-release file.
As discussed in [1], the links are setup correctly in the rpm, so the
post-install script isn't doing anything. Thus we can safely ignore
the dependency with --nodeps
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1265873
Change-Id: Icf17c84580a75d42d8e90d5d6e81ae7f5f576c32
The centos-minimal approach of using rinse does not, it turns out, work
on centos. That's a bummer. It's also rather heavyweight. Instead, with
minor machinations, we can just use yum itself pointed at a chroot.
Also adding fedora-minimal element which creates a fedora image using
the new yum-minimal approach.
Co-Authored-By: Gregory Haynes <greg@greghaynes.net>
Change-Id: I026fd9d323e786dae5bb67824c6501067e1ceaa3
