Remove yum chroot caching
Every run we are doing a full tar.gz of the chroot environment that never gets used. It's not suitable for CI since we use fresh images each time there. The cache in general isn't really isn't a very safe thing to have around, because there's no invalidation procedure and no real way to make one -- we've no guarantee that a new chroot build even moments after a previous one wouldn't bring in or different packages, etc (of course this is *unlikely*, but the longer you go between builds the worse the problem becomes. Also, tons of packages get installed after this not from any cache, so potential speed-up is rather marginal. Debian turned this off with I58fc485aacacaa17243bf9ce760ed91256d1f182. However, given the reasons above and it's complete lack of testing, I don't see this as useful. If we really want this type of thing, I think we should come up with a way to use a persistent external yum/dnf cache that yum/dnf keeps in sync with it's usual invalidation rules. Change-Id: I66789c35db75c41bc45ea1ad2e26f87456de4e4d
This commit is contained in:
Ian Wienand
parent
8c74c8e409
commit
4585955a8b
@ -6,13 +6,9 @@ Create a minimal image based on CentOS 7.
|
|||||||
Use of this element will require 'yum' and 'yum-utils' to be installed on
|
Use of this element will require 'yum' and 'yum-utils' to be installed on
|
||||||
Ubuntu and Debian. Nothing additional is needed on Fedora or CentOS.
|
Ubuntu and Debian. Nothing additional is needed on Fedora or CentOS.
|
||||||
|
|
||||||
The `DIB_OFFLINE` or more specific `DIB_YUMCHROOT_USE_CACHE`
|
By default, ``DIB_YUM_MINIMAL_CREATE_INTERFACES`` is set to enable the
|
||||||
variables can be set to prefer the use of a pre-cached root filesystem
|
creation of ``/etc/sysconfig/network-scripts/ifcfg-eth[0|1]`` scripts to
|
||||||
tarball.
|
enable DHCP on the ``eth0`` & ``eth1`` interfaces. If you do not have
|
||||||
|
|
||||||
By default, `DIB_YUM_MINIMAL_CREATE_INTERFACES` is set to enable the
|
|
||||||
creation of `/etc/sysconfig/network-scripts/ifcfg-eth[0|1]` scripts to
|
|
||||||
enable DHCP on the `eth0` & `eth1` interfaces. If you do not have
|
|
||||||
these interfaces, or if you are using something else to setup the
|
these interfaces, or if you are using something else to setup the
|
||||||
network such as cloud-init, glean or network-manager, you would want
|
network such as cloud-init, glean or network-manager, you would want
|
||||||
to set this to `0`.
|
to set this to ``0``.
|
||||||
|
|||||||
@ -11,10 +11,6 @@ Due to a bug in the released version of urlgrabber, on many systems an
|
|||||||
installation of urlgrabber from git is required. The git repository
|
installation of urlgrabber from git is required. The git repository
|
||||||
can be found here: http://yum.baseurl.org/gitweb?p=urlgrabber.git;a=summary
|
can be found here: http://yum.baseurl.org/gitweb?p=urlgrabber.git;a=summary
|
||||||
|
|
||||||
The `DIB_OFFLINE` or more specific `DIB_YUMCHROOT_USE_CACHE`
|
This element sets the ``DIB_RELEASE`` var to 'fedora'. The release of
|
||||||
variables can be set to prefer the use of a pre-cached root filesystem
|
fedora to be installed can be controlled through the ``DIB_RELEASE``
|
||||||
tarball.
|
variable, which defaults the latest supported release.
|
||||||
|
|
||||||
This element sets the `DIB_RELEASE` var to 'fedora'. The release of fedora
|
|
||||||
to be installed can be controlled through the `DIB_RELEASE` variable, which
|
|
||||||
defaults to '21'.
|
|
||||||
|
|||||||
@ -9,10 +9,6 @@ or fedora-minimal elements to get an actual base image.
|
|||||||
Use of this element will require 'yum' and 'yum-utils' to be installed on
|
Use of this element will require 'yum' and 'yum-utils' to be installed on
|
||||||
Ubuntu and Debian. Nothing additional is needed on Fedora or CentOS.
|
Ubuntu and Debian. Nothing additional is needed on Fedora or CentOS.
|
||||||
|
|
||||||
The `DIB_OFFLINE` or more specific `DIB_YUMCHROOT_USE_CACHE`
|
|
||||||
variables can be set to prefer the use of a pre-cached root filesystem
|
|
||||||
tarball.
|
|
||||||
|
|
||||||
If you wish to have DHCP networking setup for eth0 & eth1 via
|
If you wish to have DHCP networking setup for eth0 & eth1 via
|
||||||
/etc/sysconfig/network-config scripts/ifcfg-eth[0|1], set the
|
/etc/sysconfig/network-config scripts/ifcfg-eth[0|1], set the
|
||||||
environment variable `DIB_YUM_MINIMAL_CREATE_INTERFACES` to `1`.
|
environment variable `DIB_YUM_MINIMAL_CREATE_INTERFACES` to `1`.
|
||||||
|
|||||||
@ -31,8 +31,6 @@ if [ $ARCH = amd64 ]; then
|
|||||||
ARCH=x86_64
|
ARCH=x86_64
|
||||||
fi
|
fi
|
||||||
# Calling elements will need to set DISTRO_NAME and DIB_RELEASE
|
# Calling elements will need to set DISTRO_NAME and DIB_RELEASE
|
||||||
DIB_YUMCHROOT_EXTRA_ARGS=${DIB_YUMCHROOT_EXTRA_ARGS:-}
|
|
||||||
YUMCHROOT_TARBALL=$DIB_IMAGE_CACHE/yumchroot-${DISTRO_NAME}-${DIB_RELEASE}-${ARCH}.tar.gz
|
|
||||||
# TODO Maybe deal with DIB_DISTRIBUTION_MIRROR
|
# TODO Maybe deal with DIB_DISTRIBUTION_MIRROR
|
||||||
http_proxy=${http_proxy:-}
|
http_proxy=${http_proxy:-}
|
||||||
YUM=${YUM:-yum}
|
YUM=${YUM:-yum}
|
||||||
@ -189,107 +187,97 @@ function _install_pkg_manager {
|
|||||||
$TARGET_ROOT/etc/yum.repos.d/*repo
|
$TARGET_ROOT/etc/yum.repos.d/*repo
|
||||||
}
|
}
|
||||||
|
|
||||||
if [ -n "$DIB_OFFLINE" -o -n "${DIB_YUMCHROOT_USE_CACHE:-}" ] && [ -f $YUMCHROOT_TARBALL ] ; then
|
# Note this is not usually done for root.d elements (see
|
||||||
echo $YUMCHROOT_TARBALL found in cache. Using.
|
# lib/common-functions:mount_proc_dev_sys) but it's important that
|
||||||
sudo tar -C $TARGET_ROOT --numeric-owner -xzf $YUMCHROOT_TARBALL
|
# we have things like /dev/urandom around inside the chroot for
|
||||||
|
# the rpm [pre|post]inst scripts within the packages.
|
||||||
|
sudo mkdir -p $TARGET_ROOT/proc $TARGET_ROOT/dev $TARGET_ROOT/sys
|
||||||
|
sudo mount -t proc none $TARGET_ROOT/proc
|
||||||
|
sudo mount --bind /dev $TARGET_ROOT/dev
|
||||||
|
sudo mount --bind /dev/pts $TARGET_ROOT/dev/pts
|
||||||
|
sudo mount -t sysfs none $TARGET_ROOT/sys
|
||||||
|
|
||||||
|
# initalize rpmdb
|
||||||
|
sudo mkdir -p $TARGET_ROOT/var/lib/rpm
|
||||||
|
sudo $_RPM --root $TARGET_ROOT --initdb
|
||||||
|
|
||||||
|
# this makes sure that running yum/dnf in the chroot it can get
|
||||||
|
# out to download stuff
|
||||||
|
sudo mkdir $TARGET_ROOT/etc
|
||||||
|
sudo cp /etc/resolv.conf $TARGET_ROOT/etc/resolv.conf
|
||||||
|
|
||||||
|
# Bind mount the external yum cache inside the chroot. Same logic
|
||||||
|
# as in the yum element to provide for yum caching copied here
|
||||||
|
# because the sequencing is wrong otherwise
|
||||||
|
sudo mkdir -p $TMP_MOUNT_PATH/tmp/yum
|
||||||
|
sudo mount --bind $YUM_CACHE $TMP_MOUNT_PATH/tmp/yum
|
||||||
|
|
||||||
|
_install_repos
|
||||||
|
|
||||||
|
# install dnf for >= f22
|
||||||
|
if [ $DIB_RELEASE -ge 22 ]; then
|
||||||
|
_install_pkg_manager dnf dnf-plugins-core yum
|
||||||
else
|
else
|
||||||
# Note this is not usually done for root.d elements (see
|
_install_pkg_manager yum
|
||||||
# lib/common-functions:mount_proc_dev_sys) but it's important that
|
|
||||||
# we have things like /dev/urandom around inside the chroot for
|
|
||||||
# the rpm [pre|post]inst scripts within the packages.
|
|
||||||
sudo mkdir -p $TARGET_ROOT/proc $TARGET_ROOT/dev $TARGET_ROOT/sys
|
|
||||||
sudo mount -t proc none $TARGET_ROOT/proc
|
|
||||||
sudo mount --bind /dev $TARGET_ROOT/dev
|
|
||||||
sudo mount --bind /dev/pts $TARGET_ROOT/dev/pts
|
|
||||||
sudo mount -t sysfs none $TARGET_ROOT/sys
|
|
||||||
|
|
||||||
# initalize rpmdb
|
|
||||||
sudo mkdir -p $TARGET_ROOT/var/lib/rpm
|
|
||||||
sudo $_RPM --root $TARGET_ROOT --initdb
|
|
||||||
|
|
||||||
# this makes sure that running yum/dnf in the chroot it can get
|
|
||||||
# out to download stuff
|
|
||||||
sudo mkdir $TARGET_ROOT/etc
|
|
||||||
sudo cp /etc/resolv.conf $TARGET_ROOT/etc/resolv.conf
|
|
||||||
|
|
||||||
# Bind mount the external yum cache inside the chroot. Same logic
|
|
||||||
# as in the yum element to provide for yum caching copied here
|
|
||||||
# because the sequencing is wrong otherwise
|
|
||||||
sudo mkdir -p $TMP_MOUNT_PATH/tmp/yum
|
|
||||||
sudo mount --bind $YUM_CACHE $TMP_MOUNT_PATH/tmp/yum
|
|
||||||
|
|
||||||
_install_repos
|
|
||||||
|
|
||||||
# install dnf for >= f22
|
|
||||||
if [ $DIB_RELEASE -ge 22 ]; then
|
|
||||||
_install_pkg_manager dnf dnf-plugins-core yum
|
|
||||||
else
|
|
||||||
_install_pkg_manager yum
|
|
||||||
fi
|
|
||||||
|
|
||||||
# we just installed yum/dnf with "outside" tools (yum/rpm) which
|
|
||||||
# might have created /var/lib/[yum|rpm] (etc) that are slighlty
|
|
||||||
# incompatible. Refresh everything with the in-chroot tools
|
|
||||||
sudo -E chroot $TARGET_ROOT rpm --rebuilddb
|
|
||||||
sudo -E chroot $TARGET_ROOT ${YUM} clean all
|
|
||||||
|
|
||||||
# populate the lang reduction macro in the chroot
|
|
||||||
echo "%_install_langs C:en_US:en_US.UTF-8" | \
|
|
||||||
sudo tee -a $TARGET_ROOT/etc/rpm/macros.langs > /dev/null
|
|
||||||
|
|
||||||
# bootstrap the environment within the chroot; bring in new
|
|
||||||
# metadata with an update and install some base packages we need.
|
|
||||||
sudo -E chroot $TARGET_ROOT ${YUM} -y update
|
|
||||||
sudo -E chroot $TARGET_ROOT ${YUM} -y \
|
|
||||||
--setopt=cachedir=/tmp/yum/$ARCH/$DIB_RELEASE \
|
|
||||||
install systemd passwd findutils sudo util-linux-ng
|
|
||||||
|
|
||||||
# This package is split out from systemd on >F24, dracut is
|
|
||||||
# missing the dependency and will fail to make an initrd without
|
|
||||||
# it; see
|
|
||||||
# https://bugzilla.redhat.com/show_bug.cgi?id=1398505
|
|
||||||
if [ $DISTRO_NAME = "fedora" -a $DIB_RELEASE -ge 24 ]; then
|
|
||||||
sudo -E chroot $TARGET_ROOT ${YUM} -y \
|
|
||||||
--setopt=cachedir=/tmp/yum/$ARCH/$DIB_RELEASE \
|
|
||||||
install systemd-udev
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Put in a dummy /etc/resolv.conf over the temporary one we used
|
|
||||||
# to bootstrap. systemd has a bug/feature [1] that it will assume
|
|
||||||
# you want systemd-networkd as the network manager and create a
|
|
||||||
# broken symlink to /run/... if the base image doesn't have one.
|
|
||||||
# This broken link confuses things like dhclient.
|
|
||||||
# [1] https://bugzilla.redhat.com/show_bug.cgi?id=1197204
|
|
||||||
echo -e "# This file intentionally left blank\n" | \
|
|
||||||
sudo tee $TARGET_ROOT/etc/resolv.conf
|
|
||||||
|
|
||||||
# set the most reliable UTF-8 locale
|
|
||||||
echo -e 'LANG="en_US.UTF-8"' | \
|
|
||||||
sudo tee $TARGET_ROOT/etc/locale.conf
|
|
||||||
# default to UTC
|
|
||||||
sudo -E chroot $TARGET_ROOT ln -sf /usr/share/zoneinfo/UTC \
|
|
||||||
/etc/localtime
|
|
||||||
|
|
||||||
# cleanup
|
|
||||||
# TODO : move this into a exit trap; and reconsider how
|
|
||||||
# this integrates with the global exit cleanup path.
|
|
||||||
sudo umount $TMP_MOUNT_PATH/tmp/yum
|
|
||||||
sudo umount $TARGET_ROOT/proc
|
|
||||||
sudo umount $TARGET_ROOT/dev/pts
|
|
||||||
sudo umount $TARGET_ROOT/dev
|
|
||||||
sudo umount $TARGET_ROOT/sys
|
|
||||||
|
|
||||||
# RPM doesn't know whether files have been changed since install
|
|
||||||
# At this point though, we know for certain that we have changed no
|
|
||||||
# config files, so anything marked .rpmnew is just a bug.
|
|
||||||
for newfile in $(sudo find $TARGET_ROOT -type f -name '*rpmnew') ; do
|
|
||||||
sudo mv $newfile $(echo $newfile | sed 's/.rpmnew$//')
|
|
||||||
done
|
|
||||||
|
|
||||||
echo Caching result in $YUMCHROOT_TARBALL
|
|
||||||
sudo tar --numeric-owner \
|
|
||||||
-C $TARGET_ROOT \
|
|
||||||
-zcf $YUMCHROOT_TARBALL --exclude='./tmp/*' .
|
|
||||||
fi
|
fi
|
||||||
|
|
||||||
|
# we just installed yum/dnf with "outside" tools (yum/rpm) which
|
||||||
|
# might have created /var/lib/[yum|rpm] (etc) that are slighlty
|
||||||
|
# incompatible. Refresh everything with the in-chroot tools
|
||||||
|
sudo -E chroot $TARGET_ROOT rpm --rebuilddb
|
||||||
|
sudo -E chroot $TARGET_ROOT ${YUM} clean all
|
||||||
|
|
||||||
|
# populate the lang reduction macro in the chroot
|
||||||
|
echo "%_install_langs C:en_US:en_US.UTF-8" | \
|
||||||
|
sudo tee -a $TARGET_ROOT/etc/rpm/macros.langs > /dev/null
|
||||||
|
|
||||||
|
# bootstrap the environment within the chroot; bring in new
|
||||||
|
# metadata with an update and install some base packages we need.
|
||||||
|
sudo -E chroot $TARGET_ROOT ${YUM} -y update
|
||||||
|
sudo -E chroot $TARGET_ROOT ${YUM} -y \
|
||||||
|
--setopt=cachedir=/tmp/yum/$ARCH/$DIB_RELEASE \
|
||||||
|
install systemd passwd findutils sudo util-linux-ng
|
||||||
|
|
||||||
|
# This package is split out from systemd on >F24, dracut is
|
||||||
|
# missing the dependency and will fail to make an initrd without
|
||||||
|
# it; see
|
||||||
|
# https://bugzilla.redhat.com/show_bug.cgi?id=1398505
|
||||||
|
if [ $DISTRO_NAME = "fedora" -a $DIB_RELEASE -ge 24 ]; then
|
||||||
|
sudo -E chroot $TARGET_ROOT ${YUM} -y \
|
||||||
|
--setopt=cachedir=/tmp/yum/$ARCH/$DIB_RELEASE \
|
||||||
|
install systemd-udev
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Put in a dummy /etc/resolv.conf over the temporary one we used
|
||||||
|
# to bootstrap. systemd has a bug/feature [1] that it will assume
|
||||||
|
# you want systemd-networkd as the network manager and create a
|
||||||
|
# broken symlink to /run/... if the base image doesn't have one.
|
||||||
|
# This broken link confuses things like dhclient.
|
||||||
|
# [1] https://bugzilla.redhat.com/show_bug.cgi?id=1197204
|
||||||
|
echo -e "# This file intentionally left blank\n" | \
|
||||||
|
sudo tee $TARGET_ROOT/etc/resolv.conf
|
||||||
|
|
||||||
|
# set the most reliable UTF-8 locale
|
||||||
|
echo -e 'LANG="en_US.UTF-8"' | \
|
||||||
|
sudo tee $TARGET_ROOT/etc/locale.conf
|
||||||
|
# default to UTC
|
||||||
|
sudo -E chroot $TARGET_ROOT ln -sf /usr/share/zoneinfo/UTC \
|
||||||
|
/etc/localtime
|
||||||
|
|
||||||
|
# cleanup
|
||||||
|
# TODO : move this into a exit trap; and reconsider how
|
||||||
|
# this integrates with the global exit cleanup path.
|
||||||
|
sudo umount $TMP_MOUNT_PATH/tmp/yum
|
||||||
|
sudo umount $TARGET_ROOT/proc
|
||||||
|
sudo umount $TARGET_ROOT/dev/pts
|
||||||
|
sudo umount $TARGET_ROOT/dev
|
||||||
|
sudo umount $TARGET_ROOT/sys
|
||||||
|
|
||||||
|
# RPM doesn't know whether files have been changed since install
|
||||||
|
# At this point though, we know for certain that we have changed no
|
||||||
|
# config files, so anything marked .rpmnew is just a bug.
|
||||||
|
for newfile in $(sudo find $TARGET_ROOT -type f -name '*rpmnew') ; do
|
||||||
|
sudo mv $newfile $(echo $newfile | sed 's/.rpmnew$//')
|
||||||
|
done
|
||||||
|
|
||||||
sudo rm -f ${TARGET_ROOT}/.extra_settings
|
sudo rm -f ${TARGET_ROOT}/.extra_settings
|
||||||
|
|||||||
@ -0,0 +1,9 @@
|
|||||||
|
---
|
||||||
|
deprecations:
|
||||||
|
- The ``DIB_YUMCHROOT_USE_CACHE`` variable has been removed and the
|
||||||
|
Fedora and CentOS ``-minimal`` initial chroot will always be
|
||||||
|
created by the package manager. The default creation of a chroot
|
||||||
|
tarball is stopped for these elements. This unused option was
|
||||||
|
unsafe; there is no guarantee that the base system will not change
|
||||||
|
even between runs. Getting the package manager to reuse the cache
|
||||||
|
for the initial chroot install is future work.
|
||||||
Loading…
Reference in New Issue
Block a user