Remove yum chroot caching
Every run we are doing a full tar.gz of the chroot environment that never gets used. It's not suitable for CI since we use fresh images each time there. The cache in general isn't really isn't a very safe thing to have around, because there's no invalidation procedure and no real way to make one -- we've no guarantee that a new chroot build even moments after a previous one wouldn't bring in or different packages, etc (of course this is *unlikely*, but the longer you go between builds the worse the problem becomes. Also, tons of packages get installed after this not from any cache, so potential speed-up is rather marginal. Debian turned this off with I58fc485aacacaa17243bf9ce760ed91256d1f182. However, given the reasons above and it's complete lack of testing, I don't see this as useful. If we really want this type of thing, I think we should come up with a way to use a persistent external yum/dnf cache that yum/dnf keeps in sync with it's usual invalidation rules. Change-Id: I66789c35db75c41bc45ea1ad2e26f87456de4e4d
This commit is contained in:
Ian Wienand
parent
8c74c8e409
commit
4585955a8b
@ -6,13 +6,9 @@ Create a minimal image based on CentOS 7.
|
||||
Use of this element will require 'yum' and 'yum-utils' to be installed on
|
||||
Ubuntu and Debian. Nothing additional is needed on Fedora or CentOS.
|
||||
|
||||
The `DIB_OFFLINE` or more specific `DIB_YUMCHROOT_USE_CACHE`
|
||||
variables can be set to prefer the use of a pre-cached root filesystem
|
||||
tarball.
|
||||
|
||||
By default, `DIB_YUM_MINIMAL_CREATE_INTERFACES` is set to enable the
|
||||
creation of `/etc/sysconfig/network-scripts/ifcfg-eth[0|1]` scripts to
|
||||
enable DHCP on the `eth0` & `eth1` interfaces. If you do not have
|
||||
By default, ``DIB_YUM_MINIMAL_CREATE_INTERFACES`` is set to enable the
|
||||
creation of ``/etc/sysconfig/network-scripts/ifcfg-eth[0|1]`` scripts to
|
||||
enable DHCP on the ``eth0`` & ``eth1`` interfaces. If you do not have
|
||||
these interfaces, or if you are using something else to setup the
|
||||
network such as cloud-init, glean or network-manager, you would want
|
||||
to set this to `0`.
|
||||
to set this to ``0``.
|
||||
|
||||
@ -11,10 +11,6 @@ Due to a bug in the released version of urlgrabber, on many systems an
|
||||
installation of urlgrabber from git is required. The git repository
|
||||
can be found here: http://yum.baseurl.org/gitweb?p=urlgrabber.git;a=summary
|
||||
|
||||
The `DIB_OFFLINE` or more specific `DIB_YUMCHROOT_USE_CACHE`
|
||||
variables can be set to prefer the use of a pre-cached root filesystem
|
||||
tarball.
|
||||
|
||||
This element sets the `DIB_RELEASE` var to 'fedora'. The release of fedora
|
||||
to be installed can be controlled through the `DIB_RELEASE` variable, which
|
||||
defaults to '21'.
|
||||
This element sets the ``DIB_RELEASE`` var to 'fedora'. The release of
|
||||
fedora to be installed can be controlled through the ``DIB_RELEASE``
|
||||
variable, which defaults the latest supported release.
|
||||
|
||||
@ -9,10 +9,6 @@ or fedora-minimal elements to get an actual base image.
|
||||
Use of this element will require 'yum' and 'yum-utils' to be installed on
|
||||
Ubuntu and Debian. Nothing additional is needed on Fedora or CentOS.
|
||||
|
||||
The `DIB_OFFLINE` or more specific `DIB_YUMCHROOT_USE_CACHE`
|
||||
variables can be set to prefer the use of a pre-cached root filesystem
|
||||
tarball.
|
||||
|
||||
If you wish to have DHCP networking setup for eth0 & eth1 via
|
||||
/etc/sysconfig/network-config scripts/ifcfg-eth[0|1], set the
|
||||
environment variable `DIB_YUM_MINIMAL_CREATE_INTERFACES` to `1`.
|
||||
|
||||
@ -31,8 +31,6 @@ if [ $ARCH = amd64 ]; then
|
||||
ARCH=x86_64
|
||||
fi
|
||||
# Calling elements will need to set DISTRO_NAME and DIB_RELEASE
|
||||
DIB_YUMCHROOT_EXTRA_ARGS=${DIB_YUMCHROOT_EXTRA_ARGS:-}
|
||||
YUMCHROOT_TARBALL=$DIB_IMAGE_CACHE/yumchroot-${DISTRO_NAME}-${DIB_RELEASE}-${ARCH}.tar.gz
|
||||
# TODO Maybe deal with DIB_DISTRIBUTION_MIRROR
|
||||
http_proxy=${http_proxy:-}
|
||||
YUM=${YUM:-yum}
|
||||
@ -189,107 +187,97 @@ function _install_pkg_manager {
|
||||
$TARGET_ROOT/etc/yum.repos.d/*repo
|
||||
}
|
||||
|
||||
if [ -n "$DIB_OFFLINE" -o -n "${DIB_YUMCHROOT_USE_CACHE:-}" ] && [ -f $YUMCHROOT_TARBALL ] ; then
|
||||
echo $YUMCHROOT_TARBALL found in cache. Using.
|
||||
sudo tar -C $TARGET_ROOT --numeric-owner -xzf $YUMCHROOT_TARBALL
|
||||
# Note this is not usually done for root.d elements (see
|
||||
# lib/common-functions:mount_proc_dev_sys) but it's important that
|
||||
# we have things like /dev/urandom around inside the chroot for
|
||||
# the rpm [pre|post]inst scripts within the packages.
|
||||
sudo mkdir -p $TARGET_ROOT/proc $TARGET_ROOT/dev $TARGET_ROOT/sys
|
||||
sudo mount -t proc none $TARGET_ROOT/proc
|
||||
sudo mount --bind /dev $TARGET_ROOT/dev
|
||||
sudo mount --bind /dev/pts $TARGET_ROOT/dev/pts
|
||||
sudo mount -t sysfs none $TARGET_ROOT/sys
|
||||
|
||||
# initalize rpmdb
|
||||
sudo mkdir -p $TARGET_ROOT/var/lib/rpm
|
||||
sudo $_RPM --root $TARGET_ROOT --initdb
|
||||
|
||||
# this makes sure that running yum/dnf in the chroot it can get
|
||||
# out to download stuff
|
||||
sudo mkdir $TARGET_ROOT/etc
|
||||
sudo cp /etc/resolv.conf $TARGET_ROOT/etc/resolv.conf
|
||||
|
||||
# Bind mount the external yum cache inside the chroot. Same logic
|
||||
# as in the yum element to provide for yum caching copied here
|
||||
# because the sequencing is wrong otherwise
|
||||
sudo mkdir -p $TMP_MOUNT_PATH/tmp/yum
|
||||
sudo mount --bind $YUM_CACHE $TMP_MOUNT_PATH/tmp/yum
|
||||
|
||||
_install_repos
|
||||
|
||||
# install dnf for >= f22
|
||||
if [ $DIB_RELEASE -ge 22 ]; then
|
||||
_install_pkg_manager dnf dnf-plugins-core yum
|
||||
else
|
||||
# Note this is not usually done for root.d elements (see
|
||||
# lib/common-functions:mount_proc_dev_sys) but it's important that
|
||||
# we have things like /dev/urandom around inside the chroot for
|
||||
# the rpm [pre|post]inst scripts within the packages.
|
||||
sudo mkdir -p $TARGET_ROOT/proc $TARGET_ROOT/dev $TARGET_ROOT/sys
|
||||
sudo mount -t proc none $TARGET_ROOT/proc
|
||||
sudo mount --bind /dev $TARGET_ROOT/dev
|
||||
sudo mount --bind /dev/pts $TARGET_ROOT/dev/pts
|
||||
sudo mount -t sysfs none $TARGET_ROOT/sys
|
||||
|
||||
# initalize rpmdb
|
||||
sudo mkdir -p $TARGET_ROOT/var/lib/rpm
|
||||
sudo $_RPM --root $TARGET_ROOT --initdb
|
||||
|
||||
# this makes sure that running yum/dnf in the chroot it can get
|
||||
# out to download stuff
|
||||
sudo mkdir $TARGET_ROOT/etc
|
||||
sudo cp /etc/resolv.conf $TARGET_ROOT/etc/resolv.conf
|
||||
|
||||
# Bind mount the external yum cache inside the chroot. Same logic
|
||||
# as in the yum element to provide for yum caching copied here
|
||||
# because the sequencing is wrong otherwise
|
||||
sudo mkdir -p $TMP_MOUNT_PATH/tmp/yum
|
||||
sudo mount --bind $YUM_CACHE $TMP_MOUNT_PATH/tmp/yum
|
||||
|
||||
_install_repos
|
||||
|
||||
# install dnf for >= f22
|
||||
if [ $DIB_RELEASE -ge 22 ]; then
|
||||
_install_pkg_manager dnf dnf-plugins-core yum
|
||||
else
|
||||
_install_pkg_manager yum
|
||||
fi
|
||||
|
||||
# we just installed yum/dnf with "outside" tools (yum/rpm) which
|
||||
# might have created /var/lib/[yum|rpm] (etc) that are slighlty
|
||||
# incompatible. Refresh everything with the in-chroot tools
|
||||
sudo -E chroot $TARGET_ROOT rpm --rebuilddb
|
||||
sudo -E chroot $TARGET_ROOT ${YUM} clean all
|
||||
|
||||
# populate the lang reduction macro in the chroot
|
||||
echo "%_install_langs C:en_US:en_US.UTF-8" | \
|
||||
sudo tee -a $TARGET_ROOT/etc/rpm/macros.langs > /dev/null
|
||||
|
||||
# bootstrap the environment within the chroot; bring in new
|
||||
# metadata with an update and install some base packages we need.
|
||||
sudo -E chroot $TARGET_ROOT ${YUM} -y update
|
||||
sudo -E chroot $TARGET_ROOT ${YUM} -y \
|
||||
--setopt=cachedir=/tmp/yum/$ARCH/$DIB_RELEASE \
|
||||
install systemd passwd findutils sudo util-linux-ng
|
||||
|
||||
# This package is split out from systemd on >F24, dracut is
|
||||
# missing the dependency and will fail to make an initrd without
|
||||
# it; see
|
||||
# https://bugzilla.redhat.com/show_bug.cgi?id=1398505
|
||||
if [ $DISTRO_NAME = "fedora" -a $DIB_RELEASE -ge 24 ]; then
|
||||
sudo -E chroot $TARGET_ROOT ${YUM} -y \
|
||||
--setopt=cachedir=/tmp/yum/$ARCH/$DIB_RELEASE \
|
||||
install systemd-udev
|
||||
fi
|
||||
|
||||
# Put in a dummy /etc/resolv.conf over the temporary one we used
|
||||
# to bootstrap. systemd has a bug/feature [1] that it will assume
|
||||
# you want systemd-networkd as the network manager and create a
|
||||
# broken symlink to /run/... if the base image doesn't have one.
|
||||
# This broken link confuses things like dhclient.
|
||||
# [1] https://bugzilla.redhat.com/show_bug.cgi?id=1197204
|
||||
echo -e "# This file intentionally left blank\n" | \
|
||||
sudo tee $TARGET_ROOT/etc/resolv.conf
|
||||
|
||||
# set the most reliable UTF-8 locale
|
||||
echo -e 'LANG="en_US.UTF-8"' | \
|
||||
sudo tee $TARGET_ROOT/etc/locale.conf
|
||||
# default to UTC
|
||||
sudo -E chroot $TARGET_ROOT ln -sf /usr/share/zoneinfo/UTC \
|
||||
/etc/localtime
|
||||
|
||||
# cleanup
|
||||
# TODO : move this into a exit trap; and reconsider how
|
||||
# this integrates with the global exit cleanup path.
|
||||
sudo umount $TMP_MOUNT_PATH/tmp/yum
|
||||
sudo umount $TARGET_ROOT/proc
|
||||
sudo umount $TARGET_ROOT/dev/pts
|
||||
sudo umount $TARGET_ROOT/dev
|
||||
sudo umount $TARGET_ROOT/sys
|
||||
|
||||
# RPM doesn't know whether files have been changed since install
|
||||
# At this point though, we know for certain that we have changed no
|
||||
# config files, so anything marked .rpmnew is just a bug.
|
||||
for newfile in $(sudo find $TARGET_ROOT -type f -name '*rpmnew') ; do
|
||||
sudo mv $newfile $(echo $newfile | sed 's/.rpmnew$//')
|
||||
done
|
||||
|
||||
echo Caching result in $YUMCHROOT_TARBALL
|
||||
sudo tar --numeric-owner \
|
||||
-C $TARGET_ROOT \
|
||||
-zcf $YUMCHROOT_TARBALL --exclude='./tmp/*' .
|
||||
_install_pkg_manager yum
|
||||
fi
|
||||
|
||||
# we just installed yum/dnf with "outside" tools (yum/rpm) which
|
||||
# might have created /var/lib/[yum|rpm] (etc) that are slighlty
|
||||
# incompatible. Refresh everything with the in-chroot tools
|
||||
sudo -E chroot $TARGET_ROOT rpm --rebuilddb
|
||||
sudo -E chroot $TARGET_ROOT ${YUM} clean all
|
||||
|
||||
# populate the lang reduction macro in the chroot
|
||||
echo "%_install_langs C:en_US:en_US.UTF-8" | \
|
||||
sudo tee -a $TARGET_ROOT/etc/rpm/macros.langs > /dev/null
|
||||
|
||||
# bootstrap the environment within the chroot; bring in new
|
||||
# metadata with an update and install some base packages we need.
|
||||
sudo -E chroot $TARGET_ROOT ${YUM} -y update
|
||||
sudo -E chroot $TARGET_ROOT ${YUM} -y \
|
||||
--setopt=cachedir=/tmp/yum/$ARCH/$DIB_RELEASE \
|
||||
install systemd passwd findutils sudo util-linux-ng
|
||||
|
||||
# This package is split out from systemd on >F24, dracut is
|
||||
# missing the dependency and will fail to make an initrd without
|
||||
# it; see
|
||||
# https://bugzilla.redhat.com/show_bug.cgi?id=1398505
|
||||
if [ $DISTRO_NAME = "fedora" -a $DIB_RELEASE -ge 24 ]; then
|
||||
sudo -E chroot $TARGET_ROOT ${YUM} -y \
|
||||
--setopt=cachedir=/tmp/yum/$ARCH/$DIB_RELEASE \
|
||||
install systemd-udev
|
||||
fi
|
||||
|
||||
# Put in a dummy /etc/resolv.conf over the temporary one we used
|
||||
# to bootstrap. systemd has a bug/feature [1] that it will assume
|
||||
# you want systemd-networkd as the network manager and create a
|
||||
# broken symlink to /run/... if the base image doesn't have one.
|
||||
# This broken link confuses things like dhclient.
|
||||
# [1] https://bugzilla.redhat.com/show_bug.cgi?id=1197204
|
||||
echo -e "# This file intentionally left blank\n" | \
|
||||
sudo tee $TARGET_ROOT/etc/resolv.conf
|
||||
|
||||
# set the most reliable UTF-8 locale
|
||||
echo -e 'LANG="en_US.UTF-8"' | \
|
||||
sudo tee $TARGET_ROOT/etc/locale.conf
|
||||
# default to UTC
|
||||
sudo -E chroot $TARGET_ROOT ln -sf /usr/share/zoneinfo/UTC \
|
||||
/etc/localtime
|
||||
|
||||
# cleanup
|
||||
# TODO : move this into a exit trap; and reconsider how
|
||||
# this integrates with the global exit cleanup path.
|
||||
sudo umount $TMP_MOUNT_PATH/tmp/yum
|
||||
sudo umount $TARGET_ROOT/proc
|
||||
sudo umount $TARGET_ROOT/dev/pts
|
||||
sudo umount $TARGET_ROOT/dev
|
||||
sudo umount $TARGET_ROOT/sys
|
||||
|
||||
# RPM doesn't know whether files have been changed since install
|
||||
# At this point though, we know for certain that we have changed no
|
||||
# config files, so anything marked .rpmnew is just a bug.
|
||||
for newfile in $(sudo find $TARGET_ROOT -type f -name '*rpmnew') ; do
|
||||
sudo mv $newfile $(echo $newfile | sed 's/.rpmnew$//')
|
||||
done
|
||||
|
||||
sudo rm -f ${TARGET_ROOT}/.extra_settings
|
||||
|
||||
@ -0,0 +1,9 @@
|
||||
---
|
||||
deprecations:
|
||||
- The ``DIB_YUMCHROOT_USE_CACHE`` variable has been removed and the
|
||||
Fedora and CentOS ``-minimal`` initial chroot will always be
|
||||
created by the package manager. The default creation of a chroot
|
||||
tarball is stopped for these elements. This unused option was
|
||||
unsafe; there is no guarantee that the base system will not change
|
||||
even between runs. Getting the package manager to reuse the cache
|
||||
for the initial chroot install is future work.
|
||||
Loading…
Reference in New Issue
Block a user