Commit Graph

160 Commits

Author SHA1 Message Date
Adam Williamson
764c6dbd95 Notice when update package should have been installed but wasn't
We hit an interesting case in update testing recently:

https://bodhi.fedoraproject.org/updates/FEDORA-2018-115068f60e

An earlier version of that update failed testing. When we dug
into it a bit, we found that the test was failing because an
earlier version of the `pki-server` package was installed than
the version that was in the update; when asked (as part of
FreeIPA deployment) to install it, dnf had noticed that there
were dependency issues with the version of the package from the
update, but it happened to be able to install the version from
the frozen 'stable' repo...so it just went ahead and did that.

In this case, the 'missed' package resulted in a test failure,
but it'd actually be possible for this to happen and the test
to complete; we really ought to notice when this happens, and
treat it as a test failure.

So what this attempts to do is: at the end of all update tests,
check for all installed packages with the same name as a package
from the update, and compare their full NEVR to the one of the
package from the update. If a package with the same name as one
of the update packages is installed, but does not appear to be
the *same NEVR*, we fail, and upload the lists of packages for
manual investigation as to what the heck's going on.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-12-12 22:17:29 -08:00
Adam Williamson
65b4755358 Simplify anaconda error screen handling, add new needle
There's really no point having separate error and error_report
needles. Just match on error_report as well as clicking on it.
Also add a new error_report needle for latest Rawhide fonts.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-12-12 22:06:20 -08:00
Adam Williamson
0639468de6 Use -l for systemctl status (to avoid ellipsization)
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-12-12 12:36:21 -08:00
Adam Williamson
cb035c7737 Still fixing up this serial logging stuff
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-12-12 12:28:23 -08:00
Adam Williamson
95b227b97a Sigh, fix a syntax error in previous commit
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-12-12 11:57:53 -08:00
Adam Williamson
0f5281f389 In post_fail_hooks, try using serial line if no network
Sometimes we get a test failing because the SUT isn't connecting
to the network for some reason. In this case we never get any
logs, because `upload_logs` relies on being able to reacht at
least the worker host system via the network.

This attempts to detect when we can't ping the worker host, and
in that case, send some info out over the serial line instead.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-12-12 11:40:58 -08:00
Adam Williamson
25d8b56fe9 Drop 'get_milestone' from utils (it's unimplemented and unused)
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-11-30 14:22:08 -08:00
Adam Williamson
56060ff8bd Simplify and improve how we get to 'linux' line in grub
That whole creaky edifice of conditionals that figured out how
many times to press 'down' was a mess I always hated, and I just
found out that the fix for BLS wasn't complete - I'd assumed in
writing it that systems weren't being migrated to BLS on upgrade
to F30, but actually they are. This makes that design very hard
as we'd have had to find a way to change the number of 'down'
presses part-way through update tests, and all the ways I can
think of to do that would've made this even sillier.

Happily I managed to come up with what looks like a much simpler
approach: just go from the bottom. It seems that in every setup
I can think of to check - all three arches, BLS, no BLS, pre-
install, post-install - the linux line is two lines up from the
bottom of the config stanza (the last line is blank, and the
last line but one is the initramfs line). So we can just press
down 50 times (to make damn sure we're at the bottom) then press
up twice and we should be in the right place, no matter the arch,
the release, or if BLS is in use or not. Whew.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-11-30 14:17:01 -08:00
Adam Williamson
0d54e222f7 Pull in the update that fixes the FreeIPA / authselect bug
This bug is breaking all update FreeIPA tests; until the updates
go stable, let's pull them in to update tests so the results
are useful.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-11-28 17:42:58 -08:00
Adam Williamson
7dc50d3285 Drop no-longer-needed F29 update test hotfix
The update went stable months ago.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-11-27 17:30:25 -08:00
Adam Williamson
d6f26403a6 Use different file for checking post-upgrade release version
The one we were using before doesn't seem to exist any more in
Rawhide. /etc/os-release should be fine.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-11-15 12:46:24 -08:00
Adam Williamson
9d50f4f4a8 Tweak bootloader handling for BLS
Now the BLS stuff is enabled in Rawhide, we need to press 'down'
a different number of times to reach the 'linux' line when
editing the boot params (I really, really wish there was a
better way to do this :<). It gets tricky as there are all sorts
of cases here (support_server tests use a CURRREL disk image,
and then there's upgrade tests)...I think this covers things for
now.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-11-15 12:06:56 -08:00
Adam Williamson
6ddb475928 RHBZ#1644919: allow longer for console login, with a soft fail
Since a recent sssd update, console login during FreeIPA tests
is taking unusually long. We don't want this to fail all the
tests, so let's extend the timeout, but with a soft fail.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-10-31 16:33:32 -07:00
Adam Williamson
e6c8c5f0ff Work around Firefox 'close multiple tabs' warning
Somehow, recently, FreeIPA tests are running into Firefox not
quitting because it's showing a warning about closing multiple
tabs. (I think we didn't *get* multiple tabs before but now we
do, for some reason). So let's work around this by clicking
"Close tabs" if the warning appears.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-10-30 18:34:37 -07:00
Adam Williamson
6f1d9565d7 Add a temp hotfix for #1636633 for F29 update tests
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-10-09 16:40:31 -07:00
Adam Williamson
52b0fbaa32 D'oh, fix syntax error in previous commit
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-10-06 08:52:08 -07:00
Adam Williamson
9869920f5b Use longer timeout for root console switch after liveinst
For some reason, in recent tests, switching to a console after
live install completes is taking a long time, and tests are
failing because we 'only' allow 10 seconds for the login prompt
to appear. This seems to indicate some kind of performance bug,
but we don't really want all liveinst tests to fail on in, this
is not primarily a performance testing framework. So let's
tweak the root_console / console_login bits a bit to allow a
configurable timeout for the login prompt to appear, and use
that to wait 30 secs instead of 10 in this case.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-10-06 08:44:34 -07:00
Adam Williamson
1d6491d143 Workaround Workstation live on tty2 not tty1 (RHBZ #1635033)
In recent Rawhide, it seems the Workstation live session runs on
tty2 not tty1 for some reason. This throws off anacondatest
root_console, which assumes there'll be a vt on tty2. Handle it
by using tty3 instead if we're in a GNOME live environment.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-10-01 14:24:40 -07:00
Adam Williamson
e9e44a749e Add download_modularity_tests to exports from utils
This is probably my fault from when I was fixing up the PR.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-09-30 09:33:22 -07:00
Adam Williamson
a49f328dc6 Tweak how update-upgrade tests are handled a bit
Looking at this, it's a bit weird: the updated packages are
actually included in the upgrade process, but we still run
_advisory_update, which does basically nothing...then reboots.
That's kinda silly and makes the tests a bit flaky, let's fix
it. I don't think there's actually any problem with doing the
upload of updatepkgs.txt in _repo_setup_updates, becase that
already guards against being run more than once, it just bails
very early if it's already been run.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-09-28 17:23:51 -07:00
Lukas Ruzicka
24e68aa8a2 Create openqa tests to test modularity. 2018-09-26 23:09:36 -07:00
Adam Williamson
ad99c3fc23 Handle Xorg starting unusually slowly, but soft fail (#1622254)
There seems to be a bug in Rawhide lately where, when our tests
want to install a bare X and run Firefox on it, this takes an
unusually long time to start up, with SELinux in enforcing mode.
With SELinux in permissive mode it starts as fast as usual. This
isn't a hard failure and we don't want it to block all later
tests, so let's handle it and treat it as a soft fail.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-08-24 15:39:16 -07:00
Adam Williamson
5d99d56f8d Factor out the grub error loop, apply it to decrypt-on-boot
OK, we now need to work around this goddamn grub bug in *three*
places, so let's stop copying the loop around and factor it out
instead. The third place is encrypted installs, as they wait
for the decryption prompt on boot.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-08-19 10:29:23 -04:00
Adam Williamson
7df2bfb4f3 Workaround RHBZ#1618928 (hit space till errors go away)
Per Neal Gompa boot will proceed if we just page through the
error(?) messages displayed when #1618928 happens, so let's do
that to let the tests get further and see what else is broken.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-08-18 17:37:42 -04:00
Adam Williamson
8273d584e6 yay more fking magic sleeps
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-06-25 10:31:53 -07:00
Adam Williamson
861ad5d4aa Load us layout before doing post-install aarch64 cmdline hack
It seems that for some reason the localized layout gets loaded
on the installer VTs by this point in time, so we need to load
'us' again for this complex command to work.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-06-25 09:35:56 -07:00
Adam Williamson
b1140dafde Same workaround as last commit, for device type
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-06-15 14:34:56 -07:00
Adam Williamson
9b7a9c9cd8 Work around a partition scheme selection bug on aarch64
Sometimes on aarch64 clicking the partition scheme drop-down
just doesn't seem to make the menu appear, instead the button
goes active but that's all. It's very unlikely we'll be able
to track down why as this doesn't happen in manual testing on
aarch64 (according to @pwhalen), so instead let's just work
around it.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-06-15 13:40:13 -07:00
Adam Williamson
05c9f4fbcd Make sure all check_screen calls have explicit timeout
Upstream is gonna change the default from 30 to 0, it seems:
https://github.com/os-autoinst/os-autoinst/pull/965
so let's go ahead and change these two cases where we have no
explicit timeout to have one.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-05-24 14:17:24 -07:00
Adam Williamson
33ac181955 Use mirrorlist instead of baseurl for updates tests
The reason we have all this horrible code to use the commented-
out baseurl lines in the repo files instead of the metalinks
that are usually used is a timing issue with the metalink
system. As a protection against stale mirrors, the metalink
system sends the package manager a list of mirrors *and a list
of recent checksums for the repo metadata*. The package manager
goes out and gets the metadata from the first mirror on the
list, then checksums it; if the checksum isn't on the list of
checksums it got from mirrormanager, it assumes that means the
mirror is stale, and tries the next on the list instead.

The problem is that MM's list of checksums is currently only
updated once an hour (by a cron job). So we kept running into
a problem where, when a test ran just after one of the repos
had been regenerated, the infra mirror it's supposed to use
would be rejected because the checksum wasn't on the list - but
not because the mirror was stale, but because it was too fresh,
it had got the new packages and metadata but mirrormanager's
list of checksums hadn't been updated to include the checksum
for the latest metadata.

All this baseurl munging code was getting ridiculous, though,
what with the tests getting more complicated and errors showing
up in the actual repo files and stuff. It occurred to me that
instead of using the baseurl we can just use the 'mirrorlist'
system instead of 'metalink'. mirrorlist is the dumber, older
system which just provides the package manager a list of mirrors
and nothing else - the whole stale-mirror-detection-checksum
thing does not happen with mirrorlists, the package manager just
tries all the mirrors in order and uses the first that works.
And happily, it's very easy to convert the metalink URLs into
mirrorlist URLs, and it saves all that faffing around trying to
fix up baseurls.

Also, adjust upgrade_boot to do the s/metalink/mirrorlist/
substitution, so upgrade tests don't run into the timing issue
in the steps before the main repo_setup run is done by
upgrade_run, and adjust repo_setup_compose to sub this line out
later.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-05-18 16:41:13 -07:00
Adam Williamson
452140fdbb Re-arrange the repo fixups to be a bit clearer and more correct
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-05-03 11:51:50 -07:00
Adam Williamson
7912648e5b Escape the $ in previous commit, tweak repo debugging bits
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-05-03 11:28:30 -07:00
Adam Williamson
6089ec6c55 More repo baseurl munging, for update upgrade tests to F28 now
Now F28 went stable, we're not disabling updates on upgrade any
more, and this bug got exposed: the location of the updates and
updates-testing repos actually changed between F27 and F28, so
the `baseurl` line from fedora-repos in F27 isn't correct for
F28. When doing an upgrade from < 28 to > 27, we need to correct
the URL when we're done installing stuff from the old release
repos but before we start trying to pull stuff from the new
release repos.

This repo munging crap is really getting fragile, it'd be great
if we could get that metadata timing issue resolved so we could
reliably use mirrormanager...

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-05-03 10:06:58 -07:00
Adam Williamson
6b62cb87ed Another fix for the repo fix...
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-05-02 15:55:59 -07:00
Adam Williamson
c236fbcd21 Drop a stray # in previous commit, which made it not work
This is from my local test, sigh.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-05-02 15:48:24 -07:00
Adam Williamson
cef45346b6 Try to fix up some errors in fedora-repos baseurls in F28
This is breaking all f28 update tests, sigh.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-05-02 15:35:50 -07:00
Adam Williamson
cec1da9372 Remove bodhi-client 0.9 support in repo_setup_updates
...don't need it any more, F25 went EOL ages ago.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-05-02 15:08:34 -07:00
Adam Williamson
e931cfa0a5 Test FreeIPA upgrade on updates
This adds the FreeIPA server and client upgrade tests to a new
updates-server-upgrade flavor which fedora_openqa will schedule
for updates. This way, we can test whether updates break
FreeIPA upgrades, which is a request the FreeIPA team made to
me. This has been deployed on staging for the last week or so
and appears to work fine.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-04-26 11:35:18 -07:00
Adam Williamson
7e8a3010bb Fix bracket matching in previous commit
Syntax checks? In a text editor? THAT'S CRAZY TALK

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-04-09 17:19:03 -07:00
Adam Williamson
51b46bab7b g-i-s suppression change now applies to FAW 28+
Since gnome-initial-setup-3.28.0-5.fc28 , the g-i-s screens
that are supposed to be suppressed as part of
https://fedoraproject.org/wiki/Changes/ReduceInitialSetupRedundancy
are now suppressed on FAW installs as well as traditional ones.
So adjust the logic accordingly.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-04-09 16:50:16 -07:00
Adam Williamson
23192baa5d Use better pattern for checking if there are coredumps
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-04-04 14:13:39 -07:00
Adam Williamson
cdf33cc2ae Catch and upload coredumps in post-fail hook
We were doing this in a post-install test, but not on failures.
We need it to figure out why Firefox is crashing on aarch64...

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-04-04 12:10:33 -07:00
Adam Williamson
89bdea410f Rejig how we count the g-i-s screens we'll encounter
Trying to keep track of what these magic numbers mean is really
getting messy, so let's do it a bit more explicitly, using the
page names g-i-s uses internally, and lots of comments. This
should make it clearer and more maintainable when stuff changes.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-03-29 19:06:51 -07:00
Adam Williamson
baa7ac4e39 Handle KDE update test when KDE has aleady found updates
We do the 'desktop update' test for KDE via the notification
icon thingy, and it behaves differently depending on whether it
has already detected there are updates or not. The test only
works at present in the case where it *hasn't* - it expects the
notification icon to be in the extended panel and it expects to
see a 'refresh' button, neither of which is the case if it's
already noticed there are updates to install.

We should also force PackageKit to update its list of available
updates after we set up our 'special' update, otherwise on this
path KDE will only install the updates it found *before* we did
our stuff, and the test will fail as our special update won't be
there.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-03-28 19:53:19 -07:00
Adam Williamson
75d1b80db7 Bump a magic timeout a bit for aarch64
aarch64 managed to hit the problem this 'magic timeout' tries
to avoid, so let's extend it :(

e.g. https://openqa.stg.fedoraproject.org/tests/267174

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-03-27 11:45:22 -07:00
Adam Williamson
81540165e8 Adjust repo_setup for add-on Modularity
I believe this should do all the right repo modifications for
add-on Modularity (i.e. F28+ Server installs, for now).

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-03-26 13:27:31 -07:00
Adam Williamson
2c26ce1ede Simply repo_setup by removing MODULAR bit
We don't need all the MODULAR stuff any more as we no longer do
oldModularity composes like that.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-03-26 12:56:10 -07:00
Adam Williamson
669df14566 FreeIPA: handle unexpected auto-login to webUI
There are cases where we get logged back into the FreeIPA web UI
automatically by a stale kerberos ticket or something. If we're
logged in as the *right* user, let's just treat this as a soft
failure and continue with the test.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-03-21 15:43:40 -07:00
Adam Williamson
e685f00bd0 More tweaking for UEFI boot params on aarch64
Seems we're dealing with a bug, here.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-03-20 15:07:02 -07:00
Adam Williamson
20c361999b Fix post-install bootloader args on aarch64
Seems aarch64 needs 12 'down' key presses like ppc64, not 13
like x86_64. Tweak how this is done a bit; the ternary wasn't
elegant any more with the aarch64 change, so just get rid of it.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-03-20 14:05:00 -07:00