testing/os-autoinst-distri-rocky-migrated
10
0
Fork 0
Code Issues 12 Pull Requests 1 Packages Projects Releases Wiki Activity

Use mirrorlist instead of baseurl for updates tests

Browse Source 
The reason we have all this horrible code to use the commented-
out baseurl lines in the repo files instead of the metalinks
that are usually used is a timing issue with the metalink
system. As a protection against stale mirrors, the metalink
system sends the package manager a list of mirrors *and a list
of recent checksums for the repo metadata*. The package manager
goes out and gets the metadata from the first mirror on the
list, then checksums it; if the checksum isn't on the list of
checksums it got from mirrormanager, it assumes that means the
mirror is stale, and tries the next on the list instead.

The problem is that MM's list of checksums is currently only
updated once an hour (by a cron job). So we kept running into
a problem where, when a test ran just after one of the repos
had been regenerated, the infra mirror it's supposed to use
would be rejected because the checksum wasn't on the list - but
not because the mirror was stale, but because it was too fresh,
it had got the new packages and metadata but mirrormanager's
list of checksums hadn't been updated to include the checksum
for the latest metadata.

All this baseurl munging code was getting ridiculous, though,
what with the tests getting more complicated and errors showing
up in the actual repo files and stuff. It occurred to me that
instead of using the baseurl we can just use the 'mirrorlist'
system instead of 'metalink'. mirrorlist is the dumber, older
system which just provides the package manager a list of mirrors
and nothing else - the whole stale-mirror-detection-checksum
thing does not happen with mirrorlists, the package manager just
tries all the mirrors in order and uses the first that works.
And happily, it's very easy to convert the metalink URLs into
mirrorlist URLs, and it saves all that faffing around trying to
fix up baseurls.

Also, adjust upgrade_boot to do the s/metalink/mirrorlist/
substitution, so upgrade tests don't run into the timing issue
in the steps before the main repo_setup run is done by
upgrade_run, and adjust repo_setup_compose to sub this line out
later.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
This commit is contained in:
Adam Williamson 2018-05-09 12:35:59 -07:00
parent d661eaaad4
commit 33ac181955
3 changed files with 19 additions and 54 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

68
lib/utils.pm
View File

@ -339,8 +339,8 @@ sub _repo_setup_compose {
# won't always exist and we don't want to bother testing or # won't always exist and we don't want to bother testing or
# predicting their existence; assert_script_run doesn't buy you # predicting their existence; assert_script_run doesn't buy you
# much with sed as it'll return 0 even if it replaced nothing # much with sed as it'll return 0 even if it replaced nothing
script_run "sed -i -e 's,^metalink,#metalink,g' -e 's,^#baseurl.*basearch,baseurl=${location}/Everything/\$basearch,g' -e 's,^#baseurl.*source,baseurl=${location}/Everything/source,g' /etc/yum.repos.d/{fedora,fedora-rawhide}.repo", 0; script_run "sed -i -e 's,^metalink,#metalink,g' -e 's,^mirrorlist,#mirrorlist,g' -e 's,^#baseurl.*basearch,baseurl=${location}/Everything/\$basearch,g' -e 's,^#baseurl.*source,baseurl=${location}/Everything/source,g' /etc/yum.repos.d/{fedora,fedora-rawhide}.repo", 0;
script_run "sed -i -e 's,^metalink,#metalink,g' -e 's,^#baseurl.*basearch,baseurl=${location}/Modular/\$basearch,g' -e 's,^#baseurl.*source,baseurl=${location}/Modular/source,g' /etc/yum.repos.d/{fedora-modular,fedora-rawhide-modular}.repo", 0; script_run "sed -i -e 's,^metalink,#metalink,g' -e 's,^mirrorlist,#mirrorlist,g' -e 's,^#baseurl.*basearch,baseurl=${location}/Modular/\$basearch,g' -e 's,^#baseurl.*source,baseurl=${location}/Modular/source,g' /etc/yum.repos.d/{fedora-modular,fedora-rawhide-modular}.repo", 0;
# this can be used for debugging if something is going wrong # this can be used for debugging if something is going wrong
# unless (script_run 'pushd /etc/yum.repos.d && tar czvf yumreposd.tar.gz * && popd') { # unless (script_run 'pushd /etc/yum.repos.d && tar czvf yumreposd.tar.gz * && popd') {
@ -348,56 +348,27 @@ sub _repo_setup_compose {
# } # }
} }
sub _repo_setup_updates_development {
# Fix URL for fedora.repo if this is a development release
# This is rather icky, but I can't think of any better way
# The problem is that the 'baseurl' line in fedora.repo is
# always left as the correct URL for a *stable* release, we
# don't change it to the URL for a Branched release while the
# release is Branched, as it's too much annoying package work
assert_script_run "sed -i -e 's,/releases/,/development/,g' /etc/yum.repos.d/fedora.repo";
# Disable updates-testing so other bad updates don't break us
assert_script_run "dnf config-manager --set-disabled updates-testing";
# https://pagure.io/fedora-repos/issue/70
# this is the easiest workaround, it's not wrong as the repo
# is empty for branched anyway
assert_script_run "dnf config-manager --set-disabled updates";
# same for Modular, if appropriate
unless (script_run 'test -f /etc/yum.repos.d/fedora-updates-modular.repo') {
assert_script_run "sed -i -e 's,/releases/,/development/,g' /etc/yum.repos.d/fedora-modular.repo";
assert_script_run "dnf config-manager --set-disabled updates-testing-modular";
assert_script_run "dnf config-manager --set-disabled updates-modular";
}
}
sub _repo_setup_updates_28_errors {
# fix up some errors in fedora-repos baseurls in F28:
# https://pagure.io/fedora-repos/issue/70
assert_script_run 'sed -i -e "s,^\(baseurl.*/\)os/$,\1,g" /etc/yum.repos.d/fedora*updates*.repo';
assert_script_run 'sed -i -e "s,/testing-modular/,/testing/,g" /etc/yum.repos.d/fedora*updates*.repo';
}
sub _repo_setup_updates { sub _repo_setup_updates {
# Appropriate repo setup steps for testing a Bodhi update # Appropriate repo setup steps for testing a Bodhi update
# Check if we already ran, bail if so # Check if we already ran, bail if so
return unless script_run "test -f /etc/yum.repos.d/advisory.repo"; return unless script_run "test -f /etc/yum.repos.d/advisory.repo";
# Use baseurl not metalink so we don't hit the timing issue where # Use mirrorlist not metalink so we don't hit the timing issue where
# the infra repo is updated but mirrormanager metadata checksums # the infra repo is updated but mirrormanager metadata checksums
# have not been updated, and the infra repo is rejected as its # have not been updated, and the infra repo is rejected as its
# metadata checksum isn't known to MM # metadata checksum isn't known to MM
assert_script_run "sed -i -e 's,^metalink,#metalink,g' -e 's,^#baseurl,baseurl,g' /etc/yum.repos.d/fedora*.repo"; assert_script_run "sed -i -e 's,metalink,mirrorlist,g' /etc/yum.repos.d/fedora*.repo";
if (get_var("OFW")) { if (get_var("DEVELOPMENT")) {
# the uncommented baseurl line must be changed for PowerPC # Disable updates-testing so other bad updates don't break us
# from download.fedoraproject.org/pub/fedora/linux/... # this will do nothing on upgrade tests as we're on a stable
# to download.fedoraproject.org/pub/fedora-secondary/... # release at this point, but it won't *hurt* anything, so no
script_run "sed -i -e 's,/pub/fedora/linux/,/pub/fedora-secondary/,g' /etc/yum.repos.d/fedora*.repo", 0; # need to except that case really
} assert_script_run "dnf config-manager --set-disabled updates-testing";
# for non-upgrade tests, we want to do the 'development' changes # same for Modular, if appropriate
# and f28 fixups *before* we set up the update repo... unless (script_run 'test -f /etc/yum.repos.d/fedora-updates-modular.repo') {
unless (get_var("UPGRADE")) { assert_script_run "dnf config-manager --set-disabled updates-testing-modular";
_repo_setup_updates_development if (get_var("DEVELOPMENT")); }
_repo_setup_updates_28_errors if (get_var("VERSION") == 28);
} }
# Set up an additional repo containing the update packages. We do # Set up an additional repo containing the update packages. We do
# this rather than simply running a one-time update because it may # this rather than simply running a one-time update because it may
# be the case that a package from the update isn't installed *now* # be the case that a package from the update isn't installed *now*
@ -412,15 +383,6 @@ sub _repo_setup_updates {
# for upgrade tests, we want to do the 'development' changes *after* we # for upgrade tests, we want to do the 'development' changes *after* we
# set up the update repo. We don't do the f28 fixups as we don't have # set up the update repo. We don't do the f28 fixups as we don't have
# f28 fedora-repos. # f28 fedora-repos.
if (get_var("UPGRADE")) {
_repo_setup_updates_development if (get_var("DEVELOPMENT"));
if ( get_var("VERSION") > 27 && get_var("CURRREL") < 28) {
# this gets really ugly, but the repo URLs changed between 27 and
# 28, so because we're using baseurl not metalink, we need to
# fix the locations up when doing upgrade tests to 28...
assert_script_run 'sed -i -e "s,/\$releasever/\$basearch/,/\$releasever/Everything/\$basearch/,g" /etc/yum.repos.d/fedora*updates*.repo';
}
}
# this can be used for debugging if something is going wrong # this can be used for debugging if something is going wrong
# unless (script_run 'pushd /etc/yum.repos.d && tar czvf yumreposd.tar.gz * && popd') { # unless (script_run 'pushd /etc/yum.repos.d && tar czvf yumreposd.tar.gz * && popd') {

3
tests/upgrade_boot.pm
View File

@ -20,6 +20,9 @@ sub run {
$self->root_console(tty=>3); $self->root_console(tty=>3);
# disable screen blanking (update can take a long time) # disable screen blanking (update can take a long time)
script_run "setterm -blank 0"; script_run "setterm -blank 0";
# adjust repo files to use mirrorlist not metalink (to avoid
# metalink checksum timing issue causing infra repo to be rejected)
assert_script_run "sed -i -e 's,metalink,mirrorlist,g' /etc/yum.repos.d/fedora*.repo";
} }

2
tests/upgrade_run.pm
View File

@ -13,7 +13,7 @@ sub run {
# disable screen blanking (download can take a long time) # disable screen blanking (download can take a long time)
script_run "setterm -blank 0"; script_run "setterm -blank 0";
# use compose repo # use compose repo (compose tests) or set up update repo (update tests)
repo_setup(); repo_setup();
my $params = "-y --releasever=${relnum}"; my $params = "-y --releasever=${relnum}";
if ($release eq "rawhide") { if ($release eq "rawhide") {