Commit Graph

37 Commits

Author SHA1 Message Date
Ian Wienand
ce410de834 yum-minimal: Disable excludes when installing pkg manager
Because we are using the building platform's "yum" to do the initial
install into the chroot, it is affected by the base-system's
/etc/yum.conf.

pip-and-virtaulenv in I82acb865378a0fa5903a6267bfcee0e2962eced0 added
"exclude=python-pip..." in /etc/yum.conf to stop the package manager
overwriting the installed pip.  Now our CI images have built with
this, we are now picking up this exclude on centos.  Since on F24
dnf->python->python-pip we end up failing to build the the chroot
because python-pip can not be satisifed.  In a general sense, however,
this could be caused by any configuration put into /etc/yum.conf that
is incompatible with installing into the chroot.

yum has the option to disable all excludes which is used here.  This
seems to be the best way to isolate the chroot install from any
excludes that may have been done on the base system for various
reasons.  I did consider using a completely separate yum.conf we ship
with dib ... but let's start simple.

This should fix the current gate failures on centos

Change-Id: I4e4cc8ed09a29c4057ade34ea93025139e191bf5
2016-09-20 09:31:44 +10:00
Ian Wienand
944b4fea0f Add "audit"package to yum-minimal
Kernels are built with auditing support, and without the audit deamon
logs bubble up to spam the console and /var/log/messages.  This
package contains the audit daemon that catches these messages.

Change-Id: Ie3e216bab33b27f2d67a9379ddc3e89d66449251
2016-08-08 17:54:20 +10:00
Ian Wienand
b745821353 yum-minimal: set locale.conf and tz in chroot
We should be doing more to ensure initial configuration during
configuration.  Taken from the steps done by [1], here we set
locale.conf and a general timezone.

The only reliable UTF8 locale is en_US.UTF-8; we don't want to use C
locale as it causes havoc with things like python3 and unicode.  We
set locale.conf to this.

For Fedora 24 ensure we install the en_* locales too (this is really a
bug separate to this -- when you log in, by default ssh tries to copy
over your locale env variables, so logging into a F24 system would
result in using invalid locales for the most common en_* cases).

While we are here, setup a timezone link.  It turns out infra puppet
overwrites this later, but at least we have a sane default.

[1] https://www.freedesktop.org/software/systemd/man/systemd-firstboot.html#

Change-Id: Ib8951a97f1772bc5228c682e88628ff53400a923
2016-07-15 13:35:10 +10:00
Ian Wienand
1b4e6eb5f7 Handle locales install on Fedora 24
Fedora 24 has split locales into separate packages.  Testing revealed
what is possibly a bug in the choosing of default packages, so add a
small work-around to ensure the minimal locale pack is installed.

This appears to be the only change required for fedora-minimal with
Fedora 24; at least to build with the project-config infra elements.

Change-Id: I64438c34c572ed96211384ae1bfb45b2949e4318
2016-06-24 10:28:06 +10:00
Jenkins
fb6fa48f85 Merge "Export FS_TYPE and remove hardcoded ext4 values" 2016-06-07 01:01:04 +00:00
Ian Wienand
f5dff9c52a Rework yum-minimal locale cleanup
It turns out our manual locale cleanup is causing issues (see
I54490b17a7f8b2f977369044fcc6bb49cc13768e).  Upon further
investigation, I think this is a better approach than manually
deleting repos.

glibc on Fedora obeys the %_install_langs macro for reducing the
installed locales (as mentioned in the comments, F24 has moved to
having different packages, but worry about that later).

So our existing clear-out is really only required for CentOS, whose
glibc does not have any way to indicate to build less locales.
However, %_install_langs is still correct there, as it restricts some
of the translation files and other things installed with the %lang
macro in spec files.

This is complicated by us having to set this at glibc-common install
time, which happens with the "yum" from outside the chroot (i.e. on
trusty).  Since this is too old to have flags to pass this, we need to
fiddle with rpmmacros.

I've tested this with fedora-minimal builds and the locales file is
about 2MiB, which is what it was after the cleanups, and the listed
locales are only those we expect (i.e. it appears to be working).

Change-Id: I528a68beeb7b2ceec25ccbec1900670501608158
2016-05-31 15:14:24 +10:00
Jenkins
5f4cac3303 Merge "Add a best-effort sudo safety check" 2016-05-26 17:31:00 +00:00
Paul Belanger
be521bdec6 Export FS_TYPE and remove hardcoded ext4 values
Export FS_TYPE from img-defaults and use it to remove hard-coded
defaults in the debootstrap mounting.  Also, cleanup the suse element
as it should have access to the exported variable.

Change-Id: Ie9b671ca9336060a5ad294be48aa7eff442bf066
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2016-05-19 17:33:36 +10:00
Paul Belanger
0478fb15db
Fix path issue for locale-archive.tmpl
Change-Id: Id589c16aab46d447b3c21f00f3acfd06890e43d2
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2016-05-16 09:23:58 -04:00
Jenkins
8cb8806227 Merge "yum-minimal : better cleanup of initial yum failure" 2016-05-13 01:04:33 +00:00
Ian Wienand
672705831f Add a best-effort sudo safety check
As motivation for this; we have had two breakouts of dib in recent
memory.  One was a failure to unmount through symlinks in the core
code (I335316019ef948758392b03e91f9869102a472b9) and the other was
removing host keys on the build-system
(Ib01d71ff9415a0ae04d963f6e380aab9ac2260ce).

For the most part, dib runs unprivileged.  Bits of the core code are
hopefully well tested (modulo bugs like the first one!).  We give free
reign inside the chroot (although there is still some potential there
for adverse external affects via bind mounts).  Where we could be a
bit safer (and could have prevented at least the second of these
breakouts) is with some better checking that the "sudo" calls
*outside* the chroot at least looked sane.

This adds a basic check that we're using chroot or image paths when
calling sudo in those parts of elements that run *outside* the chroot.
Various files are updated to accomodate this check; mostly by just
ignoring it for existing code (I have not audited these calls).

Nobody is pretending this type of checking makes dib magically safe,
or removes the issues with it needing to do things as root during the
build.  But this can help find egregious errors like the key removal.

Change-Id: I161a5aea1d29dcdc7236f70d372c53246ec73749
2016-05-09 15:41:38 +10:00
Ian Wienand
7aa9157c33 yum-minimal: strip locale archive
Rather than removing all locale related stuff in cleanup, strip the
locale archive and rebuild it.

Building just en_US (along with POSIX/C) brings things inline with
debootstrap.  As discussed in the bug referenced, this is about the
best we can do for Centos7.

Fedora 24 has split languages out into packages so we don't have to do
this, but I have not dealt with that yet.  A guard is put in place so
we make sure we revisit this when we try to build F24.

Change-Id: I3f384d23e52effd6a09f47134746caa4a5c586be
2016-04-21 15:00:13 +10:00
Ian Wienand
e2c0d16f84 yum-minimal : better cleanup of initial yum failure
If the initial yum install into the chroot fails, we can leave behind
a lockfile and an incorrectly modified rpmmacros.

Change this so we run the cleanup unconditionally.

Change-Id: Ia9f9c4c845e5f34d33ff9a4ab7226c9175283757
2016-04-20 09:42:42 +10:00
Ian Wienand
6f622fb7c9 yum-minimal: clear our rpm/dnf/yum data in chroot
I have seem some occasional odd failures coming from the "dnf -y
update" done by elements/base/install.d/00-up-to-date.

dnf seems to sometimes think a package is not installed when it really
is.  It then seems to try and re-install them, but notices they are
installed, and then bails with a failure exit [1].  The packages that
seem to cause this vary, but the common thread is that they seem to
have all been installed during the initial phase of installing the
package manager in the chroot.

I suspect that when we are building the chroot, we do our initial
install with the "external" yum & rpm.  Then we start using the
dnf/yum in the chroot, but we're actually using meta-data created by
the *external* tools -- which could be vastly different versions or
who-knows-what.  While I honestly I don't have an exact root cause,
empirically I've found rebuilding the rpm db always seems to fix
things up.

So this change takes care to rebuild the rpm db with the chroot
version of rpm, and clear out the package metadata for a refresh with
"update".  This should hopefully put us in a consistent state.

[1] http://paste.openstack.org/show/487356/

Change-Id: I565df23897ae511356c4861fdbe63823fa6b6ff9
2016-03-08 14:53:52 +11:00
Ian Wienand
349e8db2f7 yum-minimal: pre-install base packages
We were getting some subtle issues in fedora-minimal builds that
turned out to be because /var/run was not a symlink to /run.

Upon further investigation, it turns out that yum is creating a
/var/run directory for it's pid file when it starts working in the
empty chroot (which I verified by stracing it)

---
5905  stat("/home/ubuntu/tmp/dib-tmp/image.Ac4VZZsl/mnt/var/run", 0x7ffddffa0330) = -1 ENOENT (No such file or directory)
5905  mkdir("/home/ubuntu/tmp/dib-tmp/image.Ac4VZZsl/mnt/var/run", 0755) = 0
5905  open("/home/ubuntu/tmp/dib-tmp/image.Ac4VZZsl/mnt/var/run/yum.pid", O_WRONLY|O_CREAT|O_EXCL, 0644) = 6
---

Because this happens *before* we install "filesystem" (the package),
we mess up it's symlinking.

To work-around this, pre-install the trio of base packages (setup,
basesystem, filesystem) with rpm from outside the chroot.

Change-Id: I411b6ec9d91d95d3a0f98e76853086af3b70abe8
2016-02-11 15:42:10 +11:00
Jenkins
b0eff0df07 Merge "yum-minimal : install selinux policy packages" 2015-12-22 20:16:11 +00:00
Jenkins
d450751617 Merge "yum-minimal: leave behind dummy /etc/resolv.conf" 2015-12-22 20:16:05 +00:00
Ian Wienand
fd2f55ee41 yum-minimal : install selinux policy packages
Install selinux policy packages as part of the base-installs.  selinux
is part of the base-system and the kernel boots by default in selinux
mode.

Without both of these, we can get in a situation where later scripts
(particuarly, some of the infra scripts) might install systemd-policy
without a base policy (targeted), leading to a messed up situation
where systemd will halt during boot due to missing policy files.

Change-Id: I6bf156304d1134fb328fba9b12dc364701b13696
2015-12-22 08:45:20 +11:00
Ian Wienand
1f499360fc yum-minimal: do not configure eth0 & eth1 for DHCP automatically
Add an environment variable to control the creation of eth0/1
interface enablement scripts.

With a tool such as glean, the presence of these scripts will indicate
the interface is configured and configuration-drive settings will not
be applied.  This means in a non-dhcp situation like on Rackspace,
network is broken.

On Fedora, where later systemd provides "predictable network interface
names" [1] eth0 & eth1 ironically aren't predictable so this just
confuses things.  You really need cloud-init or glean or something to
bring up your interfaces in a sane fashion.

This maintains the status-quo on centos-minimal, but disables creation
for fedora-minimal.

[1] http://www.freedesktop.org/wiki/Software/systemd/PredictableNetworkInterfaceNames/

Change-Id: I3f1ffeb6de3b1f952292a144efab9554f7f99a5f
2015-12-22 08:39:04 +11:00
Ian Wienand
5f3855f6f5 yum-minimal: leave behind dummy /etc/resolv.conf
As described in the comment, systemd will create a broken
/etc/resolv.conf link if there is no file in the base-image (as you
can read in the bug, it is debated if this is a bug or a feature).

The solution is to leave a dummy /etc/resolv.conf file in the image.
Whatever network manager you choose (NetworkManager, glean,
cloud-config, etc) will overwrite this anyway.

It's just that some tools, such as dhclient, get confused with the
broken symlink.  This affects you if you're using glean to configure
the network in a DHCP situation, for example -- dhclient won't
configure nameservers and everything goes to heck.

Change-Id: I734834d03e7fdb13f9ab2e86f877b07bf4a84ff9
2015-12-21 15:28:52 +11:00
Ian Wienand
ce781fbbc4 Fix fedora-minimal on CentOS builds
As described in the comments, CentOS overrides the "distroverpkg"
variable in yum.conf.  This is the package that yum queries to
establish the value of the $releasever variable.  On other platforms,
this defaults to "redhat-release" (which "fedora-release" provides) so
everything works.  It is only when the base-system "distroverpkg"
refers to a package not in the chroot we hit the issue.

We can avoid this by setting the releasever variable via the
commandline.

Change-Id: I231c3277960992cd479b8aff7838f246397936f2
2015-12-02 12:16:43 +11:00
Ian Wienand
1d476dd994 Remove fedora-minimal/install.d/99-ramdisk
When the kernel gets installed on Fedora, the rpm post scripts call
"/bin/kernel-install" [1] to install it.  This is a script provided by
systemd.

However, in [2], Fedora ships a patch to kernel-install that makes a
call-out to /sbin/new-kernel-pkg -- the install script provided by
grubby [3]

Without grubby installed, systemd's kernel-install script goes off and
runs dracut plugins directly [4], which eventually creates the initrd.
For reasons that are not clearly explained, the initrd will end up in
a a "machine-id" sub-directory of /boot (possibly, so you can symlink
it?).  It is also called "initrd", even though it's an initramfs, for
historical reasons in dracut I think.

It is at this point that I think 99-ramdisk has been written to move
the generated initrd file back into /boot.  Later on, when we build
the image, we run grub-install and it picks up the kernel and the
initrd and installs everything.

grubby's new-kernel-pkg [6] it's very similar -- it uses dracut to
make the initramfs ... but in this case it is put in /boot and is
actually called initramfs.

The subtle change that led me down this path is that dracut has been
modified to have a "Recommends" for grubby for >F22 [7].  After
discussing this change with the author, it turns out it was *always*
intended to use the grubby-based kernel install scripts for Fedora --
our builds have been incorrect in not including the package.  The
author got sick of people removing the package and making unbootable
systems, hence the change.

Thus this removes the workarounds in 99-ramdisk and replace it with an
install of the grubby package.  grubby's kernel install script will
put the kernel & generated initramfs in /boot, and it will be
installed correctly via the usual grub install later when we build the
disk image.

I have built F22 & F23 fedora-minimal images with this and they boot.

[1] http://pkgs.fedoraproject.org/cgit/kernel.git/tree/kernel.spec#n1832
[2] http://pkgs.fedoraproject.org/cgit/systemd.git/tree/kernel-install-grubby.patch
[3] http://linux.die.net/man/8/new-kernel-pkg
[4] https://github.com/haraldh/dracut/blob/master/50-dracut.install
[5] 81516adcb7
[6] https://github.com/rhinstaller/grubby/blob/master/new-kernel-pkg
[7] 47ff68e78b

Change-Id: I1a6e45d04755515286b3d49f8280c16b527e2f48
2015-11-19 21:03:45 +11:00
Jenkins
5e571d9f44 Merge "Fixup RPM db path when building Fedora on Ubuntu" 2015-11-10 11:03:22 +00:00
Ian Wienand
3f3850aa0f Fixup RPM db path when building Fedora on Ubuntu
On Debian/Ubuntu installs of RPM, /usr/lib/rpm/macros sets

  %_dbpath  %(echo $HOME/.rpmdb)

which makes quite a bit of sense, because RPM is not the system
packager and thus RPM is setup to install things into a hierarchy in
the users homedir.

However, this messes things up when building a Fedora chroot on an
Ubuntu platform.

We use RPM & yum from the base-system to bootstrap the Fedora chroot.
While both obey --root flags, they still pick up the %_dbpath macro
and so end up creating the RPM database in <chroot>/home/user/.rpmdb

After we have bootstrapped yum/dnf, we execute further installation
commands from inside the chroot -- where we now have the Fedora
version of /usr/lib/rpm/macros and hence have _dbpath set to
/var/lib/rpm -- except there is no rpm database there.

Should anyone be finding this in the future, the actual issue that
appears is

  $ sudo chroot /opt/dib_tmp/image.b6B5S3f6/mnt dnf makecache
   Error: Failed to synchronize cache for repo 'fedora' from \
    'https://mirrors.fedoraproject.org/metalink?repo=fedora-$releasever&arch=x86_64': \
    Cannot prepare internal mirrorlist: file "repomd.xml" was not found in metalink

Note the issue there is that $releasever is not expanded, because the
rpmdb where this info is kept is not populated.

The trick is to make sure we override this value when using the host
rpm/yum to setup the chroot.  The bare rpm calls, which we use to
install the repos, have a --dbpath argument where we can override
this.  yum does not however, so we override this in the global
~/.rpmmacros while we are installing the packaging tools and
dependencies into the chroot.

Copious comments are included, because this is super-confusing.

Change-Id: I20801150ea02d1c64f118eb969fb2aec473476f7
2015-11-10 08:54:44 +00:00
Ian Wienand
8ee21cb9fd Remove unused RELEASE_RPMS variable
It was noticed during reviews of
Ic7aa8cbe13e4347b447e84bb9c12483a4e125228 these are unused

Change-Id: I9e0fa9d3e4864e54c6fe23f8b6e781e8d5d24bda
2015-11-10 07:17:52 +00:00
Jenkins
e90be5a595 Merge "Fix fedora-minimal kernel-install on older platforms" 2015-11-10 05:14:28 +00:00
Jenkins
654d80a40f Merge "Define a default for $YUM" 2015-11-03 20:33:38 +00:00
Ian Wienand
f307bb4d8b Fix fedora-minimal kernel-install on older platforms
fedora-minimal fails to build on Ubuntu Trusty due do being unable to
find the initrd (see Id4c04d7ae20068643df34d2fa31068e8a917a52d).

This is a rather obscure problem that comes from the intersection of
several things.

The first thing to note is that the post-install scripts of the
kernel-core package use kernel-install [1].  For whatever reason, this
installs the kernel to /boot/MACHINE-ID/KERNEL-VERSION

MACHINE-ID comes from /etc/machine-id; a UUID that should have been
created by the systemd post-inst scripts with systemd-machine-id-setup
[2].

The chroot environment provided for root.d elements has no kernel
file-systems like /proc or /dev mounted.  This is where differences in
the base-system come into play -- on more recent systems that
implement getrandom() systemd does not need /dev/urandom to generate
the machine-id [3]; we get a value and /etc/machine-id is populated.

On older platforms (Trusty), systemd-machine-id-setup fails (unable to
access /dev/urandom) and we end up with a blank /etc/machine-id.  This
ends up making kernel-install (the script) fail during yum's
installation of kernel-core, which means the initrd is not installed
correctly.

We end up bailing out in fedora-minimal/install.d/99-ramdisk, where we
try to put the installed ramdisk in /boot for the later grub install
scripts to find.

The solution here is to mount the standard kernel file-systems within
the chroot before we try installing.

[1] http://www.freedesktop.org/software/systemd/man/kernel-install.html
[2] http://www.freedesktop.org/software/systemd/man/systemd-machine-id-setup.html
[3] https://github.com/systemd/systemd/blob/master/src/basic/random-util.c

Change-Id: Ibcce35da928f64e6a719b070bcc833346ee7ee92
2015-11-04 06:23:17 +11:00
Jenkins
449b4e205e Merge "Preserve env when calling yum with sudo" 2015-11-03 10:36:33 +00:00
Ian Wienand
94a7a7cf67 Create YUM_CACHE_DIR in yum-minimal
yum-minimal/root.d/08-yum-chroot runs before yum/root.d/50-yum-cache,
and thus if run on a completely fresh system will fail in
08-yum-chroot as the YUM_CACHE directory isn't made.

This is probably hidden by testing & nodepool builds, because it sets
DIB_IMAGE_CACHE.  It was hidden from me because locally I have done
builds using the "yum" element previously, which had created the
cache.

Change-Id: I333f5f7e67d198f75a522cc296c118c2e94a5ecb
2015-10-23 15:17:18 +11:00
Derek Higgins
63641aa2ce Preserve env when calling yum with sudo
In particular we need [http|https]_proxy to be preserved.

Change-Id: I5bcd1b1deac917a1be8d8155f1283e330e3d5862
2015-10-16 17:36:09 +01:00
Derek Higgins
614661ec5b Define a default for $YUM
This is set by elements that require it not to be yum.

Change-Id: Ie01f357eef382bd1549fbe2b911129a3c48818e0
2015-10-16 09:55:00 +01:00
Ian Wienand
c40aa76d83 Remove extra install of release pkgs in fedora-minimal
I'm not sure why we try to do an extra install of these, it is done
inside the chroot in _install_repos.  Currently it just gets skipped
saying the packages are already installed.

Change-Id: Ic7aa8cbe13e4347b447e84bb9c12483a4e125228
2015-10-14 06:36:06 +11:00
Ian Wienand
ebdf48623e Add Fedora 22 support to yum-minimal
Add basic F22/dnf support to yum-minimal path.  We extract common
code, add some comments and reduce duplication.

Change-Id: If4bd5f88e26bd6f2168958f1ec1efff1072de7ba
2015-10-14 06:35:22 +11:00
Ian Wienand
829d626f0a Move yum-based install into function
Move yum-based install into a function, to make way for a second
related function where use dnf later

Change-Id: Iad09f3753ecdfa0c10cb8a0970a3c8e5a2dccab1
2015-10-09 12:03:19 +11:00
Ian Wienand
afaba5b42d Use --nodeps when installing fedora-release
fedora-release >= 22 has acquired a dependency on /bin/sh.  This comes
from a %posttrans section of the spec file, which is symlinking the
os-release file.

As discussed in [1], the links are setup correctly in the rpm, so the
post-install script isn't doing anything.  Thus we can safely ignore
the dependency with --nodeps

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1265873

Change-Id: Icf17c84580a75d42d8e90d5d6e81ae7f5f576c32
2015-09-24 21:03:36 +10:00
Monty Taylor
b5bcb3b60e Add a yum-minimal element that just uses yum
The centos-minimal approach of using rinse does not, it turns out, work
on centos. That's a bummer. It's also rather heavyweight. Instead, with
minor machinations, we can just use yum itself pointed at a chroot.

Also adding fedora-minimal element which creates a fedora image using
the new yum-minimal approach.

Co-Authored-By: Gregory Haynes <greg@greghaynes.net>

Change-Id: I026fd9d323e786dae5bb67824c6501067e1ceaa3
2015-04-14 13:39:18 -04:00