From 4585955a8b82889c61deb9ecb34b8713270406a7 Mon Sep 17 00:00:00 2001 From: Ian Wienand Date: Fri, 9 Dec 2016 14:41:24 +1100 Subject: [PATCH] Remove yum chroot caching Every run we are doing a full tar.gz of the chroot environment that never gets used. It's not suitable for CI since we use fresh images each time there. The cache in general isn't really isn't a very safe thing to have around, because there's no invalidation procedure and no real way to make one -- we've no guarantee that a new chroot build even moments after a previous one wouldn't bring in or different packages, etc (of course this is *unlikely*, but the longer you go between builds the worse the problem becomes. Also, tons of packages get installed after this not from any cache, so potential speed-up is rather marginal. Debian turned this off with I58fc485aacacaa17243bf9ce760ed91256d1f182. However, given the reasons above and it's complete lack of testing, I don't see this as useful. If we really want this type of thing, I think we should come up with a way to use a persistent external yum/dnf cache that yum/dnf keeps in sync with it's usual invalidation rules. Change-Id: I66789c35db75c41bc45ea1ad2e26f87456de4e4d --- elements/centos-minimal/README.rst | 12 +- elements/fedora-minimal/README.rst | 10 +- elements/yum-minimal/README.rst | 4 - elements/yum-minimal/root.d/08-yum-chroot | 192 ++++++++---------- .../yum-cache-removal-148c33012515e56e.yaml | 9 + 5 files changed, 106 insertions(+), 121 deletions(-) create mode 100644 releasenotes/notes/yum-cache-removal-148c33012515e56e.yaml diff --git a/elements/centos-minimal/README.rst b/elements/centos-minimal/README.rst index 3c83eef3..f43b9d91 100644 --- a/elements/centos-minimal/README.rst +++ b/elements/centos-minimal/README.rst @@ -6,13 +6,9 @@ Create a minimal image based on CentOS 7. Use of this element will require 'yum' and 'yum-utils' to be installed on Ubuntu and Debian. Nothing additional is needed on Fedora or CentOS. -The `DIB_OFFLINE` or more specific `DIB_YUMCHROOT_USE_CACHE` -variables can be set to prefer the use of a pre-cached root filesystem -tarball. - -By default, `DIB_YUM_MINIMAL_CREATE_INTERFACES` is set to enable the -creation of `/etc/sysconfig/network-scripts/ifcfg-eth[0|1]` scripts to -enable DHCP on the `eth0` & `eth1` interfaces. If you do not have +By default, ``DIB_YUM_MINIMAL_CREATE_INTERFACES`` is set to enable the +creation of ``/etc/sysconfig/network-scripts/ifcfg-eth[0|1]`` scripts to +enable DHCP on the ``eth0`` & ``eth1`` interfaces. If you do not have these interfaces, or if you are using something else to setup the network such as cloud-init, glean or network-manager, you would want -to set this to `0`. +to set this to ``0``. diff --git a/elements/fedora-minimal/README.rst b/elements/fedora-minimal/README.rst index 88c88ecd..f1c77370 100644 --- a/elements/fedora-minimal/README.rst +++ b/elements/fedora-minimal/README.rst @@ -11,10 +11,6 @@ Due to a bug in the released version of urlgrabber, on many systems an installation of urlgrabber from git is required. The git repository can be found here: http://yum.baseurl.org/gitweb?p=urlgrabber.git;a=summary -The `DIB_OFFLINE` or more specific `DIB_YUMCHROOT_USE_CACHE` -variables can be set to prefer the use of a pre-cached root filesystem -tarball. - -This element sets the `DIB_RELEASE` var to 'fedora'. The release of fedora -to be installed can be controlled through the `DIB_RELEASE` variable, which -defaults to '21'. +This element sets the ``DIB_RELEASE`` var to 'fedora'. The release of +fedora to be installed can be controlled through the ``DIB_RELEASE`` +variable, which defaults the latest supported release. diff --git a/elements/yum-minimal/README.rst b/elements/yum-minimal/README.rst index ebe6be3c..e68d704d 100644 --- a/elements/yum-minimal/README.rst +++ b/elements/yum-minimal/README.rst @@ -9,10 +9,6 @@ or fedora-minimal elements to get an actual base image. Use of this element will require 'yum' and 'yum-utils' to be installed on Ubuntu and Debian. Nothing additional is needed on Fedora or CentOS. -The `DIB_OFFLINE` or more specific `DIB_YUMCHROOT_USE_CACHE` -variables can be set to prefer the use of a pre-cached root filesystem -tarball. - If you wish to have DHCP networking setup for eth0 & eth1 via /etc/sysconfig/network-config scripts/ifcfg-eth[0|1], set the environment variable `DIB_YUM_MINIMAL_CREATE_INTERFACES` to `1`. diff --git a/elements/yum-minimal/root.d/08-yum-chroot b/elements/yum-minimal/root.d/08-yum-chroot index 7760897b..ee96e552 100755 --- a/elements/yum-minimal/root.d/08-yum-chroot +++ b/elements/yum-minimal/root.d/08-yum-chroot @@ -31,8 +31,6 @@ if [ $ARCH = amd64 ]; then ARCH=x86_64 fi # Calling elements will need to set DISTRO_NAME and DIB_RELEASE -DIB_YUMCHROOT_EXTRA_ARGS=${DIB_YUMCHROOT_EXTRA_ARGS:-} -YUMCHROOT_TARBALL=$DIB_IMAGE_CACHE/yumchroot-${DISTRO_NAME}-${DIB_RELEASE}-${ARCH}.tar.gz # TODO Maybe deal with DIB_DISTRIBUTION_MIRROR http_proxy=${http_proxy:-} YUM=${YUM:-yum} @@ -189,107 +187,97 @@ function _install_pkg_manager { $TARGET_ROOT/etc/yum.repos.d/*repo } -if [ -n "$DIB_OFFLINE" -o -n "${DIB_YUMCHROOT_USE_CACHE:-}" ] && [ -f $YUMCHROOT_TARBALL ] ; then - echo $YUMCHROOT_TARBALL found in cache. Using. - sudo tar -C $TARGET_ROOT --numeric-owner -xzf $YUMCHROOT_TARBALL +# Note this is not usually done for root.d elements (see +# lib/common-functions:mount_proc_dev_sys) but it's important that +# we have things like /dev/urandom around inside the chroot for +# the rpm [pre|post]inst scripts within the packages. +sudo mkdir -p $TARGET_ROOT/proc $TARGET_ROOT/dev $TARGET_ROOT/sys +sudo mount -t proc none $TARGET_ROOT/proc +sudo mount --bind /dev $TARGET_ROOT/dev +sudo mount --bind /dev/pts $TARGET_ROOT/dev/pts +sudo mount -t sysfs none $TARGET_ROOT/sys + +# initalize rpmdb +sudo mkdir -p $TARGET_ROOT/var/lib/rpm +sudo $_RPM --root $TARGET_ROOT --initdb + +# this makes sure that running yum/dnf in the chroot it can get +# out to download stuff +sudo mkdir $TARGET_ROOT/etc +sudo cp /etc/resolv.conf $TARGET_ROOT/etc/resolv.conf + +# Bind mount the external yum cache inside the chroot. Same logic +# as in the yum element to provide for yum caching copied here +# because the sequencing is wrong otherwise +sudo mkdir -p $TMP_MOUNT_PATH/tmp/yum +sudo mount --bind $YUM_CACHE $TMP_MOUNT_PATH/tmp/yum + +_install_repos + +# install dnf for >= f22 +if [ $DIB_RELEASE -ge 22 ]; then + _install_pkg_manager dnf dnf-plugins-core yum else - # Note this is not usually done for root.d elements (see - # lib/common-functions:mount_proc_dev_sys) but it's important that - # we have things like /dev/urandom around inside the chroot for - # the rpm [pre|post]inst scripts within the packages. - sudo mkdir -p $TARGET_ROOT/proc $TARGET_ROOT/dev $TARGET_ROOT/sys - sudo mount -t proc none $TARGET_ROOT/proc - sudo mount --bind /dev $TARGET_ROOT/dev - sudo mount --bind /dev/pts $TARGET_ROOT/dev/pts - sudo mount -t sysfs none $TARGET_ROOT/sys - - # initalize rpmdb - sudo mkdir -p $TARGET_ROOT/var/lib/rpm - sudo $_RPM --root $TARGET_ROOT --initdb - - # this makes sure that running yum/dnf in the chroot it can get - # out to download stuff - sudo mkdir $TARGET_ROOT/etc - sudo cp /etc/resolv.conf $TARGET_ROOT/etc/resolv.conf - - # Bind mount the external yum cache inside the chroot. Same logic - # as in the yum element to provide for yum caching copied here - # because the sequencing is wrong otherwise - sudo mkdir -p $TMP_MOUNT_PATH/tmp/yum - sudo mount --bind $YUM_CACHE $TMP_MOUNT_PATH/tmp/yum - - _install_repos - - # install dnf for >= f22 - if [ $DIB_RELEASE -ge 22 ]; then - _install_pkg_manager dnf dnf-plugins-core yum - else - _install_pkg_manager yum - fi - - # we just installed yum/dnf with "outside" tools (yum/rpm) which - # might have created /var/lib/[yum|rpm] (etc) that are slighlty - # incompatible. Refresh everything with the in-chroot tools - sudo -E chroot $TARGET_ROOT rpm --rebuilddb - sudo -E chroot $TARGET_ROOT ${YUM} clean all - - # populate the lang reduction macro in the chroot - echo "%_install_langs C:en_US:en_US.UTF-8" | \ - sudo tee -a $TARGET_ROOT/etc/rpm/macros.langs > /dev/null - - # bootstrap the environment within the chroot; bring in new - # metadata with an update and install some base packages we need. - sudo -E chroot $TARGET_ROOT ${YUM} -y update - sudo -E chroot $TARGET_ROOT ${YUM} -y \ - --setopt=cachedir=/tmp/yum/$ARCH/$DIB_RELEASE \ - install systemd passwd findutils sudo util-linux-ng - - # This package is split out from systemd on >F24, dracut is - # missing the dependency and will fail to make an initrd without - # it; see - # https://bugzilla.redhat.com/show_bug.cgi?id=1398505 - if [ $DISTRO_NAME = "fedora" -a $DIB_RELEASE -ge 24 ]; then - sudo -E chroot $TARGET_ROOT ${YUM} -y \ - --setopt=cachedir=/tmp/yum/$ARCH/$DIB_RELEASE \ - install systemd-udev - fi - - # Put in a dummy /etc/resolv.conf over the temporary one we used - # to bootstrap. systemd has a bug/feature [1] that it will assume - # you want systemd-networkd as the network manager and create a - # broken symlink to /run/... if the base image doesn't have one. - # This broken link confuses things like dhclient. - # [1] https://bugzilla.redhat.com/show_bug.cgi?id=1197204 - echo -e "# This file intentionally left blank\n" | \ - sudo tee $TARGET_ROOT/etc/resolv.conf - - # set the most reliable UTF-8 locale - echo -e 'LANG="en_US.UTF-8"' | \ - sudo tee $TARGET_ROOT/etc/locale.conf - # default to UTC - sudo -E chroot $TARGET_ROOT ln -sf /usr/share/zoneinfo/UTC \ - /etc/localtime - - # cleanup - # TODO : move this into a exit trap; and reconsider how - # this integrates with the global exit cleanup path. - sudo umount $TMP_MOUNT_PATH/tmp/yum - sudo umount $TARGET_ROOT/proc - sudo umount $TARGET_ROOT/dev/pts - sudo umount $TARGET_ROOT/dev - sudo umount $TARGET_ROOT/sys - - # RPM doesn't know whether files have been changed since install - # At this point though, we know for certain that we have changed no - # config files, so anything marked .rpmnew is just a bug. - for newfile in $(sudo find $TARGET_ROOT -type f -name '*rpmnew') ; do - sudo mv $newfile $(echo $newfile | sed 's/.rpmnew$//') - done - - echo Caching result in $YUMCHROOT_TARBALL - sudo tar --numeric-owner \ - -C $TARGET_ROOT \ - -zcf $YUMCHROOT_TARBALL --exclude='./tmp/*' . + _install_pkg_manager yum fi +# we just installed yum/dnf with "outside" tools (yum/rpm) which +# might have created /var/lib/[yum|rpm] (etc) that are slighlty +# incompatible. Refresh everything with the in-chroot tools +sudo -E chroot $TARGET_ROOT rpm --rebuilddb +sudo -E chroot $TARGET_ROOT ${YUM} clean all + +# populate the lang reduction macro in the chroot +echo "%_install_langs C:en_US:en_US.UTF-8" | \ + sudo tee -a $TARGET_ROOT/etc/rpm/macros.langs > /dev/null + +# bootstrap the environment within the chroot; bring in new +# metadata with an update and install some base packages we need. +sudo -E chroot $TARGET_ROOT ${YUM} -y update +sudo -E chroot $TARGET_ROOT ${YUM} -y \ + --setopt=cachedir=/tmp/yum/$ARCH/$DIB_RELEASE \ + install systemd passwd findutils sudo util-linux-ng + +# This package is split out from systemd on >F24, dracut is +# missing the dependency and will fail to make an initrd without +# it; see +# https://bugzilla.redhat.com/show_bug.cgi?id=1398505 +if [ $DISTRO_NAME = "fedora" -a $DIB_RELEASE -ge 24 ]; then + sudo -E chroot $TARGET_ROOT ${YUM} -y \ + --setopt=cachedir=/tmp/yum/$ARCH/$DIB_RELEASE \ + install systemd-udev +fi + +# Put in a dummy /etc/resolv.conf over the temporary one we used +# to bootstrap. systemd has a bug/feature [1] that it will assume +# you want systemd-networkd as the network manager and create a +# broken symlink to /run/... if the base image doesn't have one. +# This broken link confuses things like dhclient. +# [1] https://bugzilla.redhat.com/show_bug.cgi?id=1197204 +echo -e "# This file intentionally left blank\n" | \ + sudo tee $TARGET_ROOT/etc/resolv.conf + +# set the most reliable UTF-8 locale +echo -e 'LANG="en_US.UTF-8"' | \ + sudo tee $TARGET_ROOT/etc/locale.conf + # default to UTC +sudo -E chroot $TARGET_ROOT ln -sf /usr/share/zoneinfo/UTC \ + /etc/localtime + +# cleanup +# TODO : move this into a exit trap; and reconsider how +# this integrates with the global exit cleanup path. +sudo umount $TMP_MOUNT_PATH/tmp/yum +sudo umount $TARGET_ROOT/proc +sudo umount $TARGET_ROOT/dev/pts +sudo umount $TARGET_ROOT/dev +sudo umount $TARGET_ROOT/sys + +# RPM doesn't know whether files have been changed since install +# At this point though, we know for certain that we have changed no +# config files, so anything marked .rpmnew is just a bug. +for newfile in $(sudo find $TARGET_ROOT -type f -name '*rpmnew') ; do + sudo mv $newfile $(echo $newfile | sed 's/.rpmnew$//') +done + sudo rm -f ${TARGET_ROOT}/.extra_settings diff --git a/releasenotes/notes/yum-cache-removal-148c33012515e56e.yaml b/releasenotes/notes/yum-cache-removal-148c33012515e56e.yaml new file mode 100644 index 00000000..86727a08 --- /dev/null +++ b/releasenotes/notes/yum-cache-removal-148c33012515e56e.yaml @@ -0,0 +1,9 @@ +--- +deprecations: + - The ``DIB_YUMCHROOT_USE_CACHE`` variable has been removed and the + Fedora and CentOS ``-minimal`` initial chroot will always be + created by the package manager. The default creation of a chroot + tarball is stopped for these elements. This unused option was + unsafe; there is no guarantee that the base system will not change + even between runs. Getting the package manager to reuse the cache + for the initial chroot install is future work.