Commit Graph

192 Commits

Author SHA1 Message Date
Lukas Ruzicka
a412f37a50 Use keyboard to navigate in menu. 2019-03-19 12:15:03 -07:00
Lukas Ruzicka
c9a25cb2a6 Create an application start-stop test for KDE.
Add png files as a background for the needles.

Rewrite the test handling methods to make them ready for KDE

Add the ABRT startstop test.

Make changes and corrections to the abrt test.

Add test for Firewall.

Add test for DNF dragora.

Add wait time for dnfdragora test.

Correct syntax.

Add Language test.

Make some changes to the DNF dragora test.

Add Users test case.

Add needles for DBUS viewer test.

Add Dbusviewer test.

Add Mahjong test and needles for games.

Add Minesweeper tests.

Add Patience test.

Add test for Document Viewer.

add test for gwenview

Add test for koulourpaint.

Add test for Kruler

Add test for Kcolorchooser

Reneedle failing needle.

Add ktorrent tests.

Add tests for CPT editors.

Add test for Krfb

Fix names for those files.

Add test for Kget

Add Akregator test.

Add test for Konversation.

Make Konversation end really.

Add tests for Kmail

Add test for PIM exporter.

Add test for KTnef and Krdc.

Fix problems after test runs.

Make more tries.

Fix needle to be found better.

Fix more errors.

Add test for Falkon.

Add tests for browsers.

Add support for closing tabs into Firefox test.

Add tests for K3B

move needles to correct directory

Add Kaddressbook test.

Add Kontakt text.

Add test for korganizer.

Add menu office needle and correct konqueror needle.

Add test for calligra stage.

Add test for Calligra.

Add test for network connections.

Modify needle for kaddressbook to prevent failing.

Add test for system settings and fix others.

Add test for FMW.

Add test for Dolphin

Add test for Infocenter.

Add test for kparted.

Fix a wrong needle.

Test relnotes.

Fix some errors in tests.

Add test for Discover.

Add test for Ksysguard.

Add tests for Konsole.

Add tests for KDE wallet.

Add tests for several utilities.

Add Krusader test.

Finish utility tests.

Fix some errors.

Fix needle for spectacle.

Add wait time to let Dragora wait for network.
2019-03-19 12:14:47 -07:00
Adam Williamson
067a728363 Give 'dnf -y update' a bit longer in update tests
For big updates, 10 minutes seems not to always be enough, viz
https://openqa.stg.fedoraproject.org/tests/494515

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-03-14 08:38:54 -07:00
Adam Williamson
375575cd07 Use a second disk for the update repo in image build tests
This tweaks the update live and installer image build tests to
store the update repo on a separate disk. This should hopefully
avoid the tests failing due to insufficient disk space when the
update is huge (e.g. KDE or GNOME megaupdates).

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-03-13 12:57:57 -07:00
Adam Williamson
6dde0ea0c9 Fix #1674045 workaround for bootloader case
The initial implementation here has a problem if we spot the
'successful' screen briefly, then the system reboots normally,
reaches the bootloader and proceeds past it all within 10
seconds; in this case we'll never actually spot the bootloader
and do our stuff. This tweak should continue through the code
block immediately if the bootloader shows up during the ten
seconds, otherwise check again for the 'successful' screen and
reboot.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-03-11 18:34:13 -07:00
Adam Williamson
bb8d9d26d7 Workaround f30 compose/buildroot mismatch
The f30 tag is currently a lot fresher than what's in the repos,
because we haven't had a compose for five days. This causes
problems in some F30 update tests because the update works fine
with what's in the `f30` tag but not what's on the mirrors. To
deal with this until we get a successful compose, let's just
enable the f30-build repo in the update test process...

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-03-06 10:41:16 -08:00
Adam Williamson
0c63065ebd Add the #1674045 workaround to one more place
Sigh, so much copypasta makes me sad, should've factored it out.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-03-05 07:35:11 -08:00
Adam Williamson
86fe913b84 Soft fail workaround the 'upgrade hangs at end' bug (#1674045)
We want to get accurate results from the rest of the upgrade
test and we know about this bug, so let's make it a soft fail
so we can see how the test is functioning otherwise.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-03-04 18:07:04 -08:00
Adam Williamson
6f24fd452c Seems we need a 15 sec sleep for the Firefox cert workaround
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-02-26 12:05:55 -08:00
Adam Williamson
b23dd7e14c Use correct name for file to disable Firefox shield studies
Dunno where I got distribution.ini from...

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-02-26 10:57:26 -08:00
Adam Williamson
855e54d8df Refresh workaround for occasional FreeIPA web UI cert issue
Investigation suggests that the Firefox certificate validation
failure we sometimes get when loading the FreeIPA web UI in the
freeipa_webui test module is a race issue in Firefox itself:

https://bugzilla.mozilla.org/show_bug.cgi?id=1530429

refreshing the page seems to work around the problem, so let's
do that.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-02-25 10:13:33 -08:00
Adam Williamson
945a93aa05 Get more info on failures caused by Firefox certificate issues
We're getting an intermittent case where FreeIPA tests fail
because of a web server certificate issue. Click 'Advanced' in
Firefox when this happens so we can get a bit more info on the
problem.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-02-21 12:16:42 -08:00
Adam Williamson
8f0108176b D'oh, move disable_firefox_studies up in utils
...so things that appear earlier in the file can use it.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-02-21 10:35:51 -08:00
Adam Williamson
3b236c7a39 Try and disable Firefox 'shield studies'
I've seen a few tests fail lately because Firefox suddenly
opened a tab with some "Privacy Study" in it and switched to it.
Per https://bugzilla.mozilla.org/show_bug.cgi?id=1529626 , this
should be the way to disable these...

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-02-21 09:51:07 -08:00
Adam Williamson
f50f6d66b1 Fix up the 'skip update on support server' stuff (a20ea59)
The previous attempt at this (in a20ea59) was wrong, because it
meant we didn't create the update repo at all, and we do need
the repo to exist in the support_server test every time (as the
point of the support server is to serve it out via NFS). We just
don't want to *use it on the support_server system itself* if
the releases don't match. So rejig things to do that properly.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-02-08 08:20:24 -08:00
Adam Williamson
99302c6fd4 Add a live image build test for updates
Just like the installer image build test, only...it builds a live
image. This involves reimplementing quite a chunk of the Koji
livemedia task. Ah, well. Also involves rethinking the flavor
names a bit here, these seem...better.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-02-07 18:28:24 -08:00
Adam Williamson
8d1d150798 Handle running update tests on Koji tasks
We quite often want to run the update tests on a Koji task (not
a Bodhi update) for some reason - usually to test a potential
fix for an issue, or at a maintainer's request to test a change
before it is merged upstream and officially sent out as an
update. Up till now I've always hacked up utils.pm on the
staging server by hand to do this, which is horrible. Together
with a commit to fedora_openqa, this should allow us to do it in
a nice, sane way via the CLI. It's mostly just tweaking the
"updates" repo setup in utils.pm as you'd expect, but there's a
bit of subtlety to it because of the installer tests that use
%ADVISORY% as a variable substitution in the disk image name;
you can't do something like `%ADVISORY or KOJITASK%`, sadly, so
I had to have almost-redundant variables ADVISORY, KOJITASK and
ADVISORY_OR_TASK (we could kinda just live with ADVISORY_OR_TASK
except I didn't want to drop ADVISORY as it's an unnecessary
change from previous behavior).

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-01-29 11:54:15 +01:00
Adam Williamson
12affb145f Add update tests to build and test a netinst image
This adds a test which builds a netinst image potentially with
the package(s) from the update, and uploads that image. It also
adds a test which runs a default install using that image. This
is intended to check whether the update breaks the creation or
use of install images; particularly this will let us test
anaconda etc. updates. We also update the minimal disk image
name, as we have to make it bigger to accommodate this test,
and making it bigger changes its name - the actual change to
the disk image itself is in createhdds. We also have to redo a
bunch of installer needles for F28 fonts, after I removed them
a month or so back...

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-01-18 08:24:44 -08:00
Adam Williamson
2c7562eea4 Log more network status stuff for anaconda failures without net
This was in installedtest.pm but not anacondatest.pm, may as well
have it in both.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-01-16 11:25:03 -08:00
Adam Williamson
536f699013 Post fail: handle landing in dracut shell, upload rdsosreport
If a test fails to the dracut shell, we currently don't do
anything useful. This should recognize when that happens, and
upload rdsosreport.txt.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-01-16 11:24:06 -08:00
Adam Williamson
9df8be32cb Upload dbus.log on anaconda failure
This log file was quite recently added to anaconda, let's upload
it with all the others.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-01-14 12:57:38 -08:00
Adam Williamson
a1bb5bd1f7 Drop an update test hotfix (no longer needed, breaks F28)
The packages from this update seem to be breaking F28 update
tests for some reason; a later update has gone stable anyway, so
this is no longer needed.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-01-07 18:13:21 -08:00
Lukas Ruzicka
39d3427471 Create a test suite for application start stop testing.
Merges #86
Fixes #85
2019-01-04 15:23:03 -08:00
Adam Williamson
141f29c7cc fix syntax of advisory_check_nonmatching_packages in post-fail
You...can't do this like that.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-01-03 11:35:40 -08:00
Adam Williamson
ca6e1105c1 Work around RHBZ #1663050 (tty quitting on exit from startx)
A bug showed up in Rawhide where, when you run startx in a tty,
when you exit that X session, the tty quits and returns to the
login prompt. This is a slightly sloppy workaround for that
problem.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-01-02 13:53:31 -08:00
Adam Williamson
e9642eace5 Give console login even longer on aarch64 for #1644919
Even an extra 30 secs doesn't seem to be long enough for the
aarch64 tests.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-12-18 11:01:35 -08:00
Adam Williamson
d6de57c6de Drop RHBZ#1618928 workaround
Bug was fixed back in August.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-12-17 14:10:58 -08:00
Adam Williamson
451f35feec Remove another case of RHBZ#1622254 workaround
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-12-17 14:04:53 -08:00
Adam Williamson
777de30179 Remove RHBZ#1622254 workaround
Fixed since 2018-09-12.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-12-17 13:47:46 -08:00
Adam Williamson
b171c0a1c4 Try to fix the 'matching update package' check for kernel
That clever-clever 'check the packages from the update were
installed' thing from yesterday breaks on kernel updates, as
they're installonly; after the update, the new version of the
package is installed, but the *old* version is too, and the way
I implemented the check, it treats that as a failure. Let's try
and handle this a somewhat-clever way (if this fails, I'm just
going to grep out lines with 'kernel' in them, as a *dumb* way).

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-12-13 18:52:51 -08:00
Adam Williamson
4629e5b740 Fix console keyboard layout in installedtest post_fail_hook
This should fix log collection when a French or Japanese test
fails before the test itself would have done this.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-12-13 16:29:53 -08:00
Adam Williamson
12e103e3da Factor meat out of advisory_post and do it in postfail too
If an update test fails before reaching advisory_post, we don't
generate the 'what update packages were installed' and 'were
any update packages *not* installed when they should have been'
logs, but these may well be useful for diagnosing the failure -
so let's also do the same stuff there. Only let's not do it all
twice.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-12-12 22:17:29 -08:00
Adam Williamson
764c6dbd95 Notice when update package should have been installed but wasn't
We hit an interesting case in update testing recently:

https://bodhi.fedoraproject.org/updates/FEDORA-2018-115068f60e

An earlier version of that update failed testing. When we dug
into it a bit, we found that the test was failing because an
earlier version of the `pki-server` package was installed than
the version that was in the update; when asked (as part of
FreeIPA deployment) to install it, dnf had noticed that there
were dependency issues with the version of the package from the
update, but it happened to be able to install the version from
the frozen 'stable' repo...so it just went ahead and did that.

In this case, the 'missed' package resulted in a test failure,
but it'd actually be possible for this to happen and the test
to complete; we really ought to notice when this happens, and
treat it as a test failure.

So what this attempts to do is: at the end of all update tests,
check for all installed packages with the same name as a package
from the update, and compare their full NEVR to the one of the
package from the update. If a package with the same name as one
of the update packages is installed, but does not appear to be
the *same NEVR*, we fail, and upload the lists of packages for
manual investigation as to what the heck's going on.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-12-12 22:17:29 -08:00
Adam Williamson
65b4755358 Simplify anaconda error screen handling, add new needle
There's really no point having separate error and error_report
needles. Just match on error_report as well as clicking on it.
Also add a new error_report needle for latest Rawhide fonts.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-12-12 22:06:20 -08:00
Adam Williamson
0639468de6 Use -l for systemctl status (to avoid ellipsization)
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-12-12 12:36:21 -08:00
Adam Williamson
cb035c7737 Still fixing up this serial logging stuff
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-12-12 12:28:23 -08:00
Adam Williamson
95b227b97a Sigh, fix a syntax error in previous commit
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-12-12 11:57:53 -08:00
Adam Williamson
0f5281f389 In post_fail_hooks, try using serial line if no network
Sometimes we get a test failing because the SUT isn't connecting
to the network for some reason. In this case we never get any
logs, because `upload_logs` relies on being able to reacht at
least the worker host system via the network.

This attempts to detect when we can't ping the worker host, and
in that case, send some info out over the serial line instead.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-12-12 11:40:58 -08:00
Adam Williamson
25d8b56fe9 Drop 'get_milestone' from utils (it's unimplemented and unused)
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-11-30 14:22:08 -08:00
Adam Williamson
56060ff8bd Simplify and improve how we get to 'linux' line in grub
That whole creaky edifice of conditionals that figured out how
many times to press 'down' was a mess I always hated, and I just
found out that the fix for BLS wasn't complete - I'd assumed in
writing it that systems weren't being migrated to BLS on upgrade
to F30, but actually they are. This makes that design very hard
as we'd have had to find a way to change the number of 'down'
presses part-way through update tests, and all the ways I can
think of to do that would've made this even sillier.

Happily I managed to come up with what looks like a much simpler
approach: just go from the bottom. It seems that in every setup
I can think of to check - all three arches, BLS, no BLS, pre-
install, post-install - the linux line is two lines up from the
bottom of the config stanza (the last line is blank, and the
last line but one is the initramfs line). So we can just press
down 50 times (to make damn sure we're at the bottom) then press
up twice and we should be in the right place, no matter the arch,
the release, or if BLS is in use or not. Whew.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-11-30 14:17:01 -08:00
Adam Williamson
0d54e222f7 Pull in the update that fixes the FreeIPA / authselect bug
This bug is breaking all update FreeIPA tests; until the updates
go stable, let's pull them in to update tests so the results
are useful.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-11-28 17:42:58 -08:00
Adam Williamson
7dc50d3285 Drop no-longer-needed F29 update test hotfix
The update went stable months ago.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-11-27 17:30:25 -08:00
Adam Williamson
d6f26403a6 Use different file for checking post-upgrade release version
The one we were using before doesn't seem to exist any more in
Rawhide. /etc/os-release should be fine.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-11-15 12:46:24 -08:00
Adam Williamson
9d50f4f4a8 Tweak bootloader handling for BLS
Now the BLS stuff is enabled in Rawhide, we need to press 'down'
a different number of times to reach the 'linux' line when
editing the boot params (I really, really wish there was a
better way to do this :<). It gets tricky as there are all sorts
of cases here (support_server tests use a CURRREL disk image,
and then there's upgrade tests)...I think this covers things for
now.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-11-15 12:06:56 -08:00
Adam Williamson
6ddb475928 RHBZ#1644919: allow longer for console login, with a soft fail
Since a recent sssd update, console login during FreeIPA tests
is taking unusually long. We don't want this to fail all the
tests, so let's extend the timeout, but with a soft fail.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-10-31 16:33:32 -07:00
Adam Williamson
e6c8c5f0ff Work around Firefox 'close multiple tabs' warning
Somehow, recently, FreeIPA tests are running into Firefox not
quitting because it's showing a warning about closing multiple
tabs. (I think we didn't *get* multiple tabs before but now we
do, for some reason). So let's work around this by clicking
"Close tabs" if the warning appears.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-10-30 18:34:37 -07:00
Adam Williamson
6f1d9565d7 Add a temp hotfix for #1636633 for F29 update tests
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-10-09 16:40:31 -07:00
Adam Williamson
52b0fbaa32 D'oh, fix syntax error in previous commit
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-10-06 08:52:08 -07:00
Adam Williamson
9869920f5b Use longer timeout for root console switch after liveinst
For some reason, in recent tests, switching to a console after
live install completes is taking a long time, and tests are
failing because we 'only' allow 10 seconds for the login prompt
to appear. This seems to indicate some kind of performance bug,
but we don't really want all liveinst tests to fail on in, this
is not primarily a performance testing framework. So let's
tweak the root_console / console_login bits a bit to allow a
configurable timeout for the login prompt to appear, and use
that to wait 30 secs instead of 10 in this case.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-10-06 08:44:34 -07:00
Adam Williamson
1d6491d143 Workaround Workstation live on tty2 not tty1 (RHBZ #1635033)
In recent Rawhide, it seems the Workstation live session runs on
tty2 not tty1 for some reason. This throws off anacondatest
root_console, which assumes there'll be a vt on tty2. Handle it
by using tty3 instead if we're in a GNOME live environment.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-10-01 14:24:40 -07:00