Commit Graph

90 Commits

Author SHA1 Message Date
Adam Williamson
e87348abca Be more strict in console_login
Because we use check_screen not assert_screen here, the match
can actually fail, but the match_has_tag conditional can pass
on the *previous* match, if that happened to also be matching
on a console tag. We don't want that. Let's just make these
into assert_screens to avoid it; I don't think there's any path
where we're actually expecting this to work if those
check_screens didn't match.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-04-10 10:44:15 -07:00
Lukas Ruzicka
a412f37a50 Use keyboard to navigate in menu. 2019-03-19 12:15:03 -07:00
Lukas Ruzicka
c9a25cb2a6 Create an application start-stop test for KDE.
Add png files as a background for the needles.

Rewrite the test handling methods to make them ready for KDE

Add the ABRT startstop test.

Make changes and corrections to the abrt test.

Add test for Firewall.

Add test for DNF dragora.

Add wait time for dnfdragora test.

Correct syntax.

Add Language test.

Make some changes to the DNF dragora test.

Add Users test case.

Add needles for DBUS viewer test.

Add Dbusviewer test.

Add Mahjong test and needles for games.

Add Minesweeper tests.

Add Patience test.

Add test for Document Viewer.

add test for gwenview

Add test for koulourpaint.

Add test for Kruler

Add test for Kcolorchooser

Reneedle failing needle.

Add ktorrent tests.

Add tests for CPT editors.

Add test for Krfb

Fix names for those files.

Add test for Kget

Add Akregator test.

Add test for Konversation.

Make Konversation end really.

Add tests for Kmail

Add test for PIM exporter.

Add test for KTnef and Krdc.

Fix problems after test runs.

Make more tries.

Fix needle to be found better.

Fix more errors.

Add test for Falkon.

Add tests for browsers.

Add support for closing tabs into Firefox test.

Add tests for K3B

move needles to correct directory

Add Kaddressbook test.

Add Kontakt text.

Add test for korganizer.

Add menu office needle and correct konqueror needle.

Add test for calligra stage.

Add test for Calligra.

Add test for network connections.

Modify needle for kaddressbook to prevent failing.

Add test for system settings and fix others.

Add test for FMW.

Add test for Dolphin

Add test for Infocenter.

Add test for kparted.

Fix a wrong needle.

Test relnotes.

Fix some errors in tests.

Add test for Discover.

Add test for Ksysguard.

Add tests for Konsole.

Add tests for KDE wallet.

Add tests for several utilities.

Add Krusader test.

Finish utility tests.

Fix some errors.

Fix needle for spectacle.

Add wait time to let Dragora wait for network.
2019-03-19 12:14:47 -07:00
Adam Williamson
067a728363 Give 'dnf -y update' a bit longer in update tests
For big updates, 10 minutes seems not to always be enough, viz
https://openqa.stg.fedoraproject.org/tests/494515

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-03-14 08:38:54 -07:00
Adam Williamson
375575cd07 Use a second disk for the update repo in image build tests
This tweaks the update live and installer image build tests to
store the update repo on a separate disk. This should hopefully
avoid the tests failing due to insufficient disk space when the
update is huge (e.g. KDE or GNOME megaupdates).

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-03-13 12:57:57 -07:00
Adam Williamson
6dde0ea0c9 Fix #1674045 workaround for bootloader case
The initial implementation here has a problem if we spot the
'successful' screen briefly, then the system reboots normally,
reaches the bootloader and proceeds past it all within 10
seconds; in this case we'll never actually spot the bootloader
and do our stuff. This tweak should continue through the code
block immediately if the bootloader shows up during the ten
seconds, otherwise check again for the 'successful' screen and
reboot.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-03-11 18:34:13 -07:00
Adam Williamson
bb8d9d26d7 Workaround f30 compose/buildroot mismatch
The f30 tag is currently a lot fresher than what's in the repos,
because we haven't had a compose for five days. This causes
problems in some F30 update tests because the update works fine
with what's in the `f30` tag but not what's on the mirrors. To
deal with this until we get a successful compose, let's just
enable the f30-build repo in the update test process...

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-03-06 10:41:16 -08:00
Adam Williamson
0c63065ebd Add the #1674045 workaround to one more place
Sigh, so much copypasta makes me sad, should've factored it out.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-03-05 07:35:11 -08:00
Adam Williamson
86fe913b84 Soft fail workaround the 'upgrade hangs at end' bug (#1674045)
We want to get accurate results from the rest of the upgrade
test and we know about this bug, so let's make it a soft fail
so we can see how the test is functioning otherwise.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-03-04 18:07:04 -08:00
Adam Williamson
b23dd7e14c Use correct name for file to disable Firefox shield studies
Dunno where I got distribution.ini from...

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-02-26 10:57:26 -08:00
Adam Williamson
8f0108176b D'oh, move disable_firefox_studies up in utils
...so things that appear earlier in the file can use it.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-02-21 10:35:51 -08:00
Adam Williamson
3b236c7a39 Try and disable Firefox 'shield studies'
I've seen a few tests fail lately because Firefox suddenly
opened a tab with some "Privacy Study" in it and switched to it.
Per https://bugzilla.mozilla.org/show_bug.cgi?id=1529626 , this
should be the way to disable these...

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-02-21 09:51:07 -08:00
Adam Williamson
f50f6d66b1 Fix up the 'skip update on support server' stuff (a20ea59)
The previous attempt at this (in a20ea59) was wrong, because it
meant we didn't create the update repo at all, and we do need
the repo to exist in the support_server test every time (as the
point of the support server is to serve it out via NFS). We just
don't want to *use it on the support_server system itself* if
the releases don't match. So rejig things to do that properly.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-02-08 08:20:24 -08:00
Adam Williamson
8d1d150798 Handle running update tests on Koji tasks
We quite often want to run the update tests on a Koji task (not
a Bodhi update) for some reason - usually to test a potential
fix for an issue, or at a maintainer's request to test a change
before it is merged upstream and officially sent out as an
update. Up till now I've always hacked up utils.pm on the
staging server by hand to do this, which is horrible. Together
with a commit to fedora_openqa, this should allow us to do it in
a nice, sane way via the CLI. It's mostly just tweaking the
"updates" repo setup in utils.pm as you'd expect, but there's a
bit of subtlety to it because of the installer tests that use
%ADVISORY% as a variable substitution in the disk image name;
you can't do something like `%ADVISORY or KOJITASK%`, sadly, so
I had to have almost-redundant variables ADVISORY, KOJITASK and
ADVISORY_OR_TASK (we could kinda just live with ADVISORY_OR_TASK
except I didn't want to drop ADVISORY as it's an unnecessary
change from previous behavior).

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-01-29 11:54:15 +01:00
Adam Williamson
a1bb5bd1f7 Drop an update test hotfix (no longer needed, breaks F28)
The packages from this update seem to be breaking F28 update
tests for some reason; a later update has gone stable anyway, so
this is no longer needed.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-01-07 18:13:21 -08:00
Lukas Ruzicka
39d3427471 Create a test suite for application start stop testing.
Merges #86
Fixes #85
2019-01-04 15:23:03 -08:00
Adam Williamson
ca6e1105c1 Work around RHBZ #1663050 (tty quitting on exit from startx)
A bug showed up in Rawhide where, when you run startx in a tty,
when you exit that X session, the tty quits and returns to the
login prompt. This is a slightly sloppy workaround for that
problem.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-01-02 13:53:31 -08:00
Adam Williamson
e9642eace5 Give console login even longer on aarch64 for #1644919
Even an extra 30 secs doesn't seem to be long enough for the
aarch64 tests.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-12-18 11:01:35 -08:00
Adam Williamson
d6de57c6de Drop RHBZ#1618928 workaround
Bug was fixed back in August.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-12-17 14:10:58 -08:00
Adam Williamson
451f35feec Remove another case of RHBZ#1622254 workaround
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-12-17 14:04:53 -08:00
Adam Williamson
b171c0a1c4 Try to fix the 'matching update package' check for kernel
That clever-clever 'check the packages from the update were
installed' thing from yesterday breaks on kernel updates, as
they're installonly; after the update, the new version of the
package is installed, but the *old* version is too, and the way
I implemented the check, it treats that as a failure. Let's try
and handle this a somewhat-clever way (if this fails, I'm just
going to grep out lines with 'kernel' in them, as a *dumb* way).

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-12-13 18:52:51 -08:00
Adam Williamson
12e103e3da Factor meat out of advisory_post and do it in postfail too
If an update test fails before reaching advisory_post, we don't
generate the 'what update packages were installed' and 'were
any update packages *not* installed when they should have been'
logs, but these may well be useful for diagnosing the failure -
so let's also do the same stuff there. Only let's not do it all
twice.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-12-12 22:17:29 -08:00
Adam Williamson
764c6dbd95 Notice when update package should have been installed but wasn't
We hit an interesting case in update testing recently:

https://bodhi.fedoraproject.org/updates/FEDORA-2018-115068f60e

An earlier version of that update failed testing. When we dug
into it a bit, we found that the test was failing because an
earlier version of the `pki-server` package was installed than
the version that was in the update; when asked (as part of
FreeIPA deployment) to install it, dnf had noticed that there
were dependency issues with the version of the package from the
update, but it happened to be able to install the version from
the frozen 'stable' repo...so it just went ahead and did that.

In this case, the 'missed' package resulted in a test failure,
but it'd actually be possible for this to happen and the test
to complete; we really ought to notice when this happens, and
treat it as a test failure.

So what this attempts to do is: at the end of all update tests,
check for all installed packages with the same name as a package
from the update, and compare their full NEVR to the one of the
package from the update. If a package with the same name as one
of the update packages is installed, but does not appear to be
the *same NEVR*, we fail, and upload the lists of packages for
manual investigation as to what the heck's going on.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-12-12 22:17:29 -08:00
Adam Williamson
25d8b56fe9 Drop 'get_milestone' from utils (it's unimplemented and unused)
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-11-30 14:22:08 -08:00
Adam Williamson
56060ff8bd Simplify and improve how we get to 'linux' line in grub
That whole creaky edifice of conditionals that figured out how
many times to press 'down' was a mess I always hated, and I just
found out that the fix for BLS wasn't complete - I'd assumed in
writing it that systems weren't being migrated to BLS on upgrade
to F30, but actually they are. This makes that design very hard
as we'd have had to find a way to change the number of 'down'
presses part-way through update tests, and all the ways I can
think of to do that would've made this even sillier.

Happily I managed to come up with what looks like a much simpler
approach: just go from the bottom. It seems that in every setup
I can think of to check - all three arches, BLS, no BLS, pre-
install, post-install - the linux line is two lines up from the
bottom of the config stanza (the last line is blank, and the
last line but one is the initramfs line). So we can just press
down 50 times (to make damn sure we're at the bottom) then press
up twice and we should be in the right place, no matter the arch,
the release, or if BLS is in use or not. Whew.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-11-30 14:17:01 -08:00
Adam Williamson
0d54e222f7 Pull in the update that fixes the FreeIPA / authselect bug
This bug is breaking all update FreeIPA tests; until the updates
go stable, let's pull them in to update tests so the results
are useful.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-11-28 17:42:58 -08:00
Adam Williamson
7dc50d3285 Drop no-longer-needed F29 update test hotfix
The update went stable months ago.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-11-27 17:30:25 -08:00
Adam Williamson
d6f26403a6 Use different file for checking post-upgrade release version
The one we were using before doesn't seem to exist any more in
Rawhide. /etc/os-release should be fine.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-11-15 12:46:24 -08:00
Adam Williamson
9d50f4f4a8 Tweak bootloader handling for BLS
Now the BLS stuff is enabled in Rawhide, we need to press 'down'
a different number of times to reach the 'linux' line when
editing the boot params (I really, really wish there was a
better way to do this :<). It gets tricky as there are all sorts
of cases here (support_server tests use a CURRREL disk image,
and then there's upgrade tests)...I think this covers things for
now.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-11-15 12:06:56 -08:00
Adam Williamson
6ddb475928 RHBZ#1644919: allow longer for console login, with a soft fail
Since a recent sssd update, console login during FreeIPA tests
is taking unusually long. We don't want this to fail all the
tests, so let's extend the timeout, but with a soft fail.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-10-31 16:33:32 -07:00
Adam Williamson
e6c8c5f0ff Work around Firefox 'close multiple tabs' warning
Somehow, recently, FreeIPA tests are running into Firefox not
quitting because it's showing a warning about closing multiple
tabs. (I think we didn't *get* multiple tabs before but now we
do, for some reason). So let's work around this by clicking
"Close tabs" if the warning appears.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-10-30 18:34:37 -07:00
Adam Williamson
6f1d9565d7 Add a temp hotfix for #1636633 for F29 update tests
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-10-09 16:40:31 -07:00
Adam Williamson
52b0fbaa32 D'oh, fix syntax error in previous commit
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-10-06 08:52:08 -07:00
Adam Williamson
9869920f5b Use longer timeout for root console switch after liveinst
For some reason, in recent tests, switching to a console after
live install completes is taking a long time, and tests are
failing because we 'only' allow 10 seconds for the login prompt
to appear. This seems to indicate some kind of performance bug,
but we don't really want all liveinst tests to fail on in, this
is not primarily a performance testing framework. So let's
tweak the root_console / console_login bits a bit to allow a
configurable timeout for the login prompt to appear, and use
that to wait 30 secs instead of 10 in this case.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-10-06 08:44:34 -07:00
Adam Williamson
e9e44a749e Add download_modularity_tests to exports from utils
This is probably my fault from when I was fixing up the PR.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-09-30 09:33:22 -07:00
Adam Williamson
a49f328dc6 Tweak how update-upgrade tests are handled a bit
Looking at this, it's a bit weird: the updated packages are
actually included in the upgrade process, but we still run
_advisory_update, which does basically nothing...then reboots.
That's kinda silly and makes the tests a bit flaky, let's fix
it. I don't think there's actually any problem with doing the
upload of updatepkgs.txt in _repo_setup_updates, becase that
already guards against being run more than once, it just bails
very early if it's already been run.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-09-28 17:23:51 -07:00
Lukas Ruzicka
24e68aa8a2 Create openqa tests to test modularity. 2018-09-26 23:09:36 -07:00
Adam Williamson
ad99c3fc23 Handle Xorg starting unusually slowly, but soft fail (#1622254)
There seems to be a bug in Rawhide lately where, when our tests
want to install a bare X and run Firefox on it, this takes an
unusually long time to start up, with SELinux in enforcing mode.
With SELinux in permissive mode it starts as fast as usual. This
isn't a hard failure and we don't want it to block all later
tests, so let's handle it and treat it as a soft fail.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-08-24 15:39:16 -07:00
Adam Williamson
5d99d56f8d Factor out the grub error loop, apply it to decrypt-on-boot
OK, we now need to work around this goddamn grub bug in *three*
places, so let's stop copying the loop around and factor it out
instead. The third place is encrypted installs, as they wait
for the decryption prompt on boot.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-08-19 10:29:23 -04:00
Adam Williamson
7df2bfb4f3 Workaround RHBZ#1618928 (hit space till errors go away)
Per Neal Gompa boot will proceed if we just page through the
error(?) messages displayed when #1618928 happens, so let's do
that to let the tests get further and see what else is broken.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-08-18 17:37:42 -04:00
Adam Williamson
8273d584e6 yay more fking magic sleeps
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-06-25 10:31:53 -07:00
Adam Williamson
861ad5d4aa Load us layout before doing post-install aarch64 cmdline hack
It seems that for some reason the localized layout gets loaded
on the installer VTs by this point in time, so we need to load
'us' again for this complex command to work.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-06-25 09:35:56 -07:00
Adam Williamson
05c9f4fbcd Make sure all check_screen calls have explicit timeout
Upstream is gonna change the default from 30 to 0, it seems:
https://github.com/os-autoinst/os-autoinst/pull/965
so let's go ahead and change these two cases where we have no
explicit timeout to have one.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-05-24 14:17:24 -07:00
Adam Williamson
33ac181955 Use mirrorlist instead of baseurl for updates tests
The reason we have all this horrible code to use the commented-
out baseurl lines in the repo files instead of the metalinks
that are usually used is a timing issue with the metalink
system. As a protection against stale mirrors, the metalink
system sends the package manager a list of mirrors *and a list
of recent checksums for the repo metadata*. The package manager
goes out and gets the metadata from the first mirror on the
list, then checksums it; if the checksum isn't on the list of
checksums it got from mirrormanager, it assumes that means the
mirror is stale, and tries the next on the list instead.

The problem is that MM's list of checksums is currently only
updated once an hour (by a cron job). So we kept running into
a problem where, when a test ran just after one of the repos
had been regenerated, the infra mirror it's supposed to use
would be rejected because the checksum wasn't on the list - but
not because the mirror was stale, but because it was too fresh,
it had got the new packages and metadata but mirrormanager's
list of checksums hadn't been updated to include the checksum
for the latest metadata.

All this baseurl munging code was getting ridiculous, though,
what with the tests getting more complicated and errors showing
up in the actual repo files and stuff. It occurred to me that
instead of using the baseurl we can just use the 'mirrorlist'
system instead of 'metalink'. mirrorlist is the dumber, older
system which just provides the package manager a list of mirrors
and nothing else - the whole stale-mirror-detection-checksum
thing does not happen with mirrorlists, the package manager just
tries all the mirrors in order and uses the first that works.
And happily, it's very easy to convert the metalink URLs into
mirrorlist URLs, and it saves all that faffing around trying to
fix up baseurls.

Also, adjust upgrade_boot to do the s/metalink/mirrorlist/
substitution, so upgrade tests don't run into the timing issue
in the steps before the main repo_setup run is done by
upgrade_run, and adjust repo_setup_compose to sub this line out
later.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-05-18 16:41:13 -07:00
Adam Williamson
452140fdbb Re-arrange the repo fixups to be a bit clearer and more correct
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-05-03 11:51:50 -07:00
Adam Williamson
7912648e5b Escape the $ in previous commit, tweak repo debugging bits
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-05-03 11:28:30 -07:00
Adam Williamson
6089ec6c55 More repo baseurl munging, for update upgrade tests to F28 now
Now F28 went stable, we're not disabling updates on upgrade any
more, and this bug got exposed: the location of the updates and
updates-testing repos actually changed between F27 and F28, so
the `baseurl` line from fedora-repos in F27 isn't correct for
F28. When doing an upgrade from < 28 to > 27, we need to correct
the URL when we're done installing stuff from the old release
repos but before we start trying to pull stuff from the new
release repos.

This repo munging crap is really getting fragile, it'd be great
if we could get that metadata timing issue resolved so we could
reliably use mirrormanager...

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-05-03 10:06:58 -07:00
Adam Williamson
6b62cb87ed Another fix for the repo fix...
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-05-02 15:55:59 -07:00
Adam Williamson
c236fbcd21 Drop a stray # in previous commit, which made it not work
This is from my local test, sigh.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-05-02 15:48:24 -07:00
Adam Williamson
cef45346b6 Try to fix up some errors in fedora-repos baseurls in F28
This is breaking all f28 update tests, sigh.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-05-02 15:35:50 -07:00