Commit Graph

383 Commits

Author SHA1 Message Date
Adam Williamson
fb439767f5 Fix ostree root
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-08-16 11:00:15 -07:00
Adam Williamson
092d7dd9c3 installedtest: try and fix log uploads for ostree installs
The log files are all under the ostree deploy root, the 'real'
system root has nothing useful. Try and find the deploy root
and prepend it to all the relevant commands if we're a CANNED
install.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-08-16 09:05:26 -07:00
Adam Williamson
150812c15c short-circuit copy_devcdrom_as_isofile if no ISO_URL
There are some situations in which ISO_URL is not set (e.g. a
manual POST to re-run tests after download has been done). If
there is no ISO_URL just skip the checksum check here.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-08-15 19:05:59 -07:00
Adam Williamson
04b716329a sigh, can't use get_release_number before defining it...
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-08-15 12:00:26 -07:00
Adam Williamson
d7d3d40d83 Work around lorax breakage in F29
A broken lorax got into F29 stable and is causing live image
builds to fail in all F29 update tests. Work around that by
downloading a newer fixed package in _repo_setup_updates.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-08-15 11:52:12 -07:00
Adam Williamson
9dd5465f83 Go back to git master for modularity test script
As my fixes got merged. Yay.
This reverts commit 3d3bc1e2d2.
2019-08-12 18:18:52 -07:00
Adam Williamson
4dae3d2168 Use some nmcli con down/ups to workaround RHBZ #1739148
Restarting NetworkManager.service doesn't seem to apply our
newly-created static config in Rawhide any more (again!), since
NetworkManager 1.20.0-0.5.fc31 landed. This seems to work around
the bug.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-08-08 08:54:23 -07:00
Adam Williamson
3d3bc1e2d2 Use my fork of modularity test script for now
As upstream master doesn't run.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-08-07 17:50:09 -07:00
Adam Williamson
d9816b0ddb Current modularity test script requires python3-yaml
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-08-07 17:49:12 -07:00
Lukas Ruzicka
d06b30e8dc Add new testcases and methods for release identification. 2019-07-30 15:45:02 -07:00
Adam Williamson
85c74f214e Use lower similarity level for type_safely wait_still_screen
Just like all the others, to handle flashing cursors.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-07-25 14:57:00 -07:00
Adam Williamson
0206324115 start_cockpit: wait a bit longer for login screen
Current tests sometimes seem to fail because X takes quite a
while to come up for some reason. I can't figure out why, and
this isn't properly a failure of the test (starting the X server
is an implementation detail), so let's just wait a bit longer
for now.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-07-25 14:55:07 -07:00
Adam Williamson
2a8acb6acb start_cockpit: lower similarity level for wait_still_screen
Flashing cursor, again.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-07-25 14:28:17 -07:00
Adam Williamson
be12f07900 GNOME apps: dismiss new LibreOffice 'tip of the day' if it shows
Also add a new needle for Impress startup.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-07-23 20:15:24 -07:00
Adam Williamson
30102df930 Fix download_modularity_tests arg evaluation
Evaluating an array in scalar context gives the length of the
array, not its first value...

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-07-17 10:40:03 -07:00
Adam Williamson
25f408ea7e Factor out clicking of update and akonadi notifications in KDE
There are three places where we basically want to click away
pop-up update notifications and the buggy akonadi_migration_agent
notification if it's there, in KDE tests. Let's share this code
between them, and also let's record soft failures for the buggy
cases in the desktop_notifications test.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-07-17 09:02:24 -07:00
Lukas Ruzicka
1475813b75 Update the modularity tests.
* Add test to check module defaults.
* Add whitelist download.
* Fix install test to include selected profile to be on the safe side.
* Add test into templates.
2019-07-15 13:19:06 +02:00
Adam Williamson
d34b5a3567 Revert "Use some nmcli con down/ups to workaround RHBZ #1727411"
This reverts commit 7ec143382f.
The bug it works around was fixed, so we shouldn't need the work
around any more.
2019-07-09 17:40:01 -07:00
Adam Williamson
7ec143382f Use some nmcli con down/ups to workaround RHBZ #1727411
Restarting NetworkManager.service doesn't seem to apply our
newly-created static config in Rawhide any more, since
NetworkManager 1.20.0-0.3.fc31 landed. This seems to work around
the bug.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-07-05 16:40:43 -07:00
Adam Williamson
670c2d4c9d type_very_safely: use looser still screen similarity
type_very_safely commonly gets stuck waiting on a still screen
at the end because when you're typing, there's often a flashing
cursor, and the default similarity for wait_still_screen is 47
which is tight enough that a flashing cursor usually fails it.
So back it off to 45.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-07-05 15:58:22 -07:00
Adam Williamson
e1cad438ce Remove the Bodhi workaround again now the update is stable
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-06-27 08:39:26 -07:00
Adam Williamson
6656f1659d apps_activities is a dupe with overview_app_grid, kill it
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-06-25 14:01:39 -07:00
Adam Williamson
b2848c5040 Fix fucking missing fucking -y fuck it I just want fucking dinner
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-06-17 19:29:41 -07:00
Adam Williamson
57938e1663 Add a workaround for a Bodhi bug that's causing problems
We're not downloading all the packages from updates that contain
more than one builds ATM, which makes the test invalid and has
caused some false fails (and may even have caused false passes,
though I can't tell yet). Install a fixed Bodhi as a workaround
for this.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-06-17 19:12:01 -07:00
Adam Williamson
3c8316241e Out, damned tab, out!
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-05-28 16:45:41 -07:00
Adam Williamson
43b990195c Add a wrapper around assert_and_click to handle signature change
So, turns out new os-autoinst does *not* still accept the old
argument style for assert_and_click...and old os-autoinst
doesn't accept the new one. This adds a wrapper that handles
both, so our tests can work with old and new os-autoinst. We can
drop this once both deployments are on newer os-autoinst.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-05-28 09:41:33 -07:00
Adam Williamson
87456f6b1d Use new arg style for assert_and_click
In the new os-autoinst I just sent to staging, the old style
doesn't work any more, breaks all tests. This style should also
work with the older os-autoinst on stable.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-05-27 14:28:17 -07:00
Adam Williamson
a8bcc191bb Drop FreeIPA upgrade workaround as the update is stable now
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-05-24 16:43:13 -07:00
Adam Williamson
e1ffd8aabd Fix text install test for 'use password' now being default
'use password' now seems to be default when creating a user in
text mode, so we need to handle that.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-05-22 15:18:08 -07:00
Adam Williamson
333757eadc Pull in fix for BIND bug in F30 upgrade tests
A BIND update broke a test, then went stable, so it's failing on
all updates right now. This subsequent update fixes the bug, so
let's pull it in here until it's pushed stable.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-05-09 09:31:47 +07:00
Adam Williamson
0d38d3f292 FreeIPA: get verbose logs from BIND
This was necessary for debugging the FreeIPA 4.8 pre-release
update bug, so let's have it for all runs, just in case.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-05-01 18:59:52 -07:00
Adam Williamson
68cb3eb464 Remove branched buildroot repo workaround
For now, anyway...thinking of bringing this back in a more
generic form later.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-05-01 08:33:02 -07:00
Adam Williamson
7d4db97aa3 remove #1698200 workaround (no longer needed)
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-05-01 08:29:57 -07:00
Adam Williamson
6c974ce2ac Remove workarounds for #1674045 (upgrade hang at end)
It's fixed now.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-04-29 09:49:03 -07:00
Michel Normand
37e2f134f1 Bypass dup chars after snapshot brc#1691487
eg failing jobs:
https://openqa.stg.fedoraproject.org/tests/499908#step/_boot_to_anaconda/7
https://openqa.stg.fedoraproject.org/tests/519326#step/_post_network_static/4

Signed-off-by: Michel Normand <normand@linux.vnet.ibm.com>
2019-04-15 16:20:13 +02:00
Adam Williamson
d33aa591a6 Update the workaround for #1698200
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-04-12 19:01:09 -07:00
Adam Williamson
2754deb28a Don't do live build postinstall steps on the install tests
It's not going to work there. Use the test name not the flavor
name to match on.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-04-12 18:54:36 -07:00
Guy Menanteau
29070a54f1 New install_repository_hd_variation
use a new PREINSTALL to execute a rescue script before install.

This is covering the  Hard_drive_variation test as per:
https://fedoraproject.org/wiki/QA:Testcase_install_repository_Hard_drive_variation

Signed-off-by: Guy Menanteau <menantea@linux.vnet.ibm.com>
2019-04-12 15:49:54 -07:00
Michel Normand
4a00301725 new select_rescue_mode in lib/utils from rescue_mode_encrypted.pm
placed in lib/utils.pm to be used by another commit.

Signed-off-by: Michel Normand <normand@linux.vnet.ibm.com>
2019-04-12 15:49:54 -07:00
Adam Williamson
8688b91fb2 F30 updates tests: pull in fix for #1698200
Otherwise base_services_start test always fails.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-04-11 09:52:53 -07:00
Adam Williamson
e87348abca Be more strict in console_login
Because we use check_screen not assert_screen here, the match
can actually fail, but the match_has_tag conditional can pass
on the *previous* match, if that happened to also be matching
on a console tag. We don't want that. Let's just make these
into assert_screens to avoid it; I don't think there's any path
where we're actually expecting this to work if those
check_screens didn't match.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-04-10 10:44:15 -07:00
Lukas Ruzicka
a412f37a50 Use keyboard to navigate in menu. 2019-03-19 12:15:03 -07:00
Lukas Ruzicka
c9a25cb2a6 Create an application start-stop test for KDE.
Add png files as a background for the needles.

Rewrite the test handling methods to make them ready for KDE

Add the ABRT startstop test.

Make changes and corrections to the abrt test.

Add test for Firewall.

Add test for DNF dragora.

Add wait time for dnfdragora test.

Correct syntax.

Add Language test.

Make some changes to the DNF dragora test.

Add Users test case.

Add needles for DBUS viewer test.

Add Dbusviewer test.

Add Mahjong test and needles for games.

Add Minesweeper tests.

Add Patience test.

Add test for Document Viewer.

add test for gwenview

Add test for koulourpaint.

Add test for Kruler

Add test for Kcolorchooser

Reneedle failing needle.

Add ktorrent tests.

Add tests for CPT editors.

Add test for Krfb

Fix names for those files.

Add test for Kget

Add Akregator test.

Add test for Konversation.

Make Konversation end really.

Add tests for Kmail

Add test for PIM exporter.

Add test for KTnef and Krdc.

Fix problems after test runs.

Make more tries.

Fix needle to be found better.

Fix more errors.

Add test for Falkon.

Add tests for browsers.

Add support for closing tabs into Firefox test.

Add tests for K3B

move needles to correct directory

Add Kaddressbook test.

Add Kontakt text.

Add test for korganizer.

Add menu office needle and correct konqueror needle.

Add test for calligra stage.

Add test for Calligra.

Add test for network connections.

Modify needle for kaddressbook to prevent failing.

Add test for system settings and fix others.

Add test for FMW.

Add test for Dolphin

Add test for Infocenter.

Add test for kparted.

Fix a wrong needle.

Test relnotes.

Fix some errors in tests.

Add test for Discover.

Add test for Ksysguard.

Add tests for Konsole.

Add tests for KDE wallet.

Add tests for several utilities.

Add Krusader test.

Finish utility tests.

Fix some errors.

Fix needle for spectacle.

Add wait time to let Dragora wait for network.
2019-03-19 12:14:47 -07:00
Adam Williamson
067a728363 Give 'dnf -y update' a bit longer in update tests
For big updates, 10 minutes seems not to always be enough, viz
https://openqa.stg.fedoraproject.org/tests/494515

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-03-14 08:38:54 -07:00
Adam Williamson
375575cd07 Use a second disk for the update repo in image build tests
This tweaks the update live and installer image build tests to
store the update repo on a separate disk. This should hopefully
avoid the tests failing due to insufficient disk space when the
update is huge (e.g. KDE or GNOME megaupdates).

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-03-13 12:57:57 -07:00
Adam Williamson
6dde0ea0c9 Fix #1674045 workaround for bootloader case
The initial implementation here has a problem if we spot the
'successful' screen briefly, then the system reboots normally,
reaches the bootloader and proceeds past it all within 10
seconds; in this case we'll never actually spot the bootloader
and do our stuff. This tweak should continue through the code
block immediately if the bootloader shows up during the ten
seconds, otherwise check again for the 'successful' screen and
reboot.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-03-11 18:34:13 -07:00
Adam Williamson
bb8d9d26d7 Workaround f30 compose/buildroot mismatch
The f30 tag is currently a lot fresher than what's in the repos,
because we haven't had a compose for five days. This causes
problems in some F30 update tests because the update works fine
with what's in the `f30` tag but not what's on the mirrors. To
deal with this until we get a successful compose, let's just
enable the f30-build repo in the update test process...

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-03-06 10:41:16 -08:00
Adam Williamson
0c63065ebd Add the #1674045 workaround to one more place
Sigh, so much copypasta makes me sad, should've factored it out.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-03-05 07:35:11 -08:00
Adam Williamson
86fe913b84 Soft fail workaround the 'upgrade hangs at end' bug (#1674045)
We want to get accurate results from the rest of the upgrade
test and we know about this bug, so let's make it a soft fail
so we can see how the test is functioning otherwise.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-03-04 18:07:04 -08:00
Adam Williamson
6f24fd452c Seems we need a 15 sec sleep for the Firefox cert workaround
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-02-26 12:05:55 -08:00
Adam Williamson
b23dd7e14c Use correct name for file to disable Firefox shield studies
Dunno where I got distribution.ini from...

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-02-26 10:57:26 -08:00
Adam Williamson
855e54d8df Refresh workaround for occasional FreeIPA web UI cert issue
Investigation suggests that the Firefox certificate validation
failure we sometimes get when loading the FreeIPA web UI in the
freeipa_webui test module is a race issue in Firefox itself:

https://bugzilla.mozilla.org/show_bug.cgi?id=1530429

refreshing the page seems to work around the problem, so let's
do that.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-02-25 10:13:33 -08:00
Adam Williamson
945a93aa05 Get more info on failures caused by Firefox certificate issues
We're getting an intermittent case where FreeIPA tests fail
because of a web server certificate issue. Click 'Advanced' in
Firefox when this happens so we can get a bit more info on the
problem.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-02-21 12:16:42 -08:00
Adam Williamson
8f0108176b D'oh, move disable_firefox_studies up in utils
...so things that appear earlier in the file can use it.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-02-21 10:35:51 -08:00
Adam Williamson
3b236c7a39 Try and disable Firefox 'shield studies'
I've seen a few tests fail lately because Firefox suddenly
opened a tab with some "Privacy Study" in it and switched to it.
Per https://bugzilla.mozilla.org/show_bug.cgi?id=1529626 , this
should be the way to disable these...

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-02-21 09:51:07 -08:00
Adam Williamson
f50f6d66b1 Fix up the 'skip update on support server' stuff (a20ea59)
The previous attempt at this (in a20ea59) was wrong, because it
meant we didn't create the update repo at all, and we do need
the repo to exist in the support_server test every time (as the
point of the support server is to serve it out via NFS). We just
don't want to *use it on the support_server system itself* if
the releases don't match. So rejig things to do that properly.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-02-08 08:20:24 -08:00
Adam Williamson
99302c6fd4 Add a live image build test for updates
Just like the installer image build test, only...it builds a live
image. This involves reimplementing quite a chunk of the Koji
livemedia task. Ah, well. Also involves rethinking the flavor
names a bit here, these seem...better.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-02-07 18:28:24 -08:00
Adam Williamson
8d1d150798 Handle running update tests on Koji tasks
We quite often want to run the update tests on a Koji task (not
a Bodhi update) for some reason - usually to test a potential
fix for an issue, or at a maintainer's request to test a change
before it is merged upstream and officially sent out as an
update. Up till now I've always hacked up utils.pm on the
staging server by hand to do this, which is horrible. Together
with a commit to fedora_openqa, this should allow us to do it in
a nice, sane way via the CLI. It's mostly just tweaking the
"updates" repo setup in utils.pm as you'd expect, but there's a
bit of subtlety to it because of the installer tests that use
%ADVISORY% as a variable substitution in the disk image name;
you can't do something like `%ADVISORY or KOJITASK%`, sadly, so
I had to have almost-redundant variables ADVISORY, KOJITASK and
ADVISORY_OR_TASK (we could kinda just live with ADVISORY_OR_TASK
except I didn't want to drop ADVISORY as it's an unnecessary
change from previous behavior).

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-01-29 11:54:15 +01:00
Adam Williamson
12affb145f Add update tests to build and test a netinst image
This adds a test which builds a netinst image potentially with
the package(s) from the update, and uploads that image. It also
adds a test which runs a default install using that image. This
is intended to check whether the update breaks the creation or
use of install images; particularly this will let us test
anaconda etc. updates. We also update the minimal disk image
name, as we have to make it bigger to accommodate this test,
and making it bigger changes its name - the actual change to
the disk image itself is in createhdds. We also have to redo a
bunch of installer needles for F28 fonts, after I removed them
a month or so back...

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-01-18 08:24:44 -08:00
Adam Williamson
2c7562eea4 Log more network status stuff for anaconda failures without net
This was in installedtest.pm but not anacondatest.pm, may as well
have it in both.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-01-16 11:25:03 -08:00
Adam Williamson
536f699013 Post fail: handle landing in dracut shell, upload rdsosreport
If a test fails to the dracut shell, we currently don't do
anything useful. This should recognize when that happens, and
upload rdsosreport.txt.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-01-16 11:24:06 -08:00
Adam Williamson
9df8be32cb Upload dbus.log on anaconda failure
This log file was quite recently added to anaconda, let's upload
it with all the others.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-01-14 12:57:38 -08:00
Adam Williamson
a1bb5bd1f7 Drop an update test hotfix (no longer needed, breaks F28)
The packages from this update seem to be breaking F28 update
tests for some reason; a later update has gone stable anyway, so
this is no longer needed.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-01-07 18:13:21 -08:00
Lukas Ruzicka
39d3427471 Create a test suite for application start stop testing.
Merges #86
Fixes #85
2019-01-04 15:23:03 -08:00
Adam Williamson
141f29c7cc fix syntax of advisory_check_nonmatching_packages in post-fail
You...can't do this like that.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-01-03 11:35:40 -08:00
Adam Williamson
ca6e1105c1 Work around RHBZ #1663050 (tty quitting on exit from startx)
A bug showed up in Rawhide where, when you run startx in a tty,
when you exit that X session, the tty quits and returns to the
login prompt. This is a slightly sloppy workaround for that
problem.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-01-02 13:53:31 -08:00
Adam Williamson
e9642eace5 Give console login even longer on aarch64 for #1644919
Even an extra 30 secs doesn't seem to be long enough for the
aarch64 tests.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-12-18 11:01:35 -08:00
Adam Williamson
d6de57c6de Drop RHBZ#1618928 workaround
Bug was fixed back in August.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-12-17 14:10:58 -08:00
Adam Williamson
451f35feec Remove another case of RHBZ#1622254 workaround
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-12-17 14:04:53 -08:00
Adam Williamson
777de30179 Remove RHBZ#1622254 workaround
Fixed since 2018-09-12.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-12-17 13:47:46 -08:00
Adam Williamson
b171c0a1c4 Try to fix the 'matching update package' check for kernel
That clever-clever 'check the packages from the update were
installed' thing from yesterday breaks on kernel updates, as
they're installonly; after the update, the new version of the
package is installed, but the *old* version is too, and the way
I implemented the check, it treats that as a failure. Let's try
and handle this a somewhat-clever way (if this fails, I'm just
going to grep out lines with 'kernel' in them, as a *dumb* way).

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-12-13 18:52:51 -08:00
Adam Williamson
4629e5b740 Fix console keyboard layout in installedtest post_fail_hook
This should fix log collection when a French or Japanese test
fails before the test itself would have done this.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-12-13 16:29:53 -08:00
Adam Williamson
12e103e3da Factor meat out of advisory_post and do it in postfail too
If an update test fails before reaching advisory_post, we don't
generate the 'what update packages were installed' and 'were
any update packages *not* installed when they should have been'
logs, but these may well be useful for diagnosing the failure -
so let's also do the same stuff there. Only let's not do it all
twice.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-12-12 22:17:29 -08:00
Adam Williamson
764c6dbd95 Notice when update package should have been installed but wasn't
We hit an interesting case in update testing recently:

https://bodhi.fedoraproject.org/updates/FEDORA-2018-115068f60e

An earlier version of that update failed testing. When we dug
into it a bit, we found that the test was failing because an
earlier version of the `pki-server` package was installed than
the version that was in the update; when asked (as part of
FreeIPA deployment) to install it, dnf had noticed that there
were dependency issues with the version of the package from the
update, but it happened to be able to install the version from
the frozen 'stable' repo...so it just went ahead and did that.

In this case, the 'missed' package resulted in a test failure,
but it'd actually be possible for this to happen and the test
to complete; we really ought to notice when this happens, and
treat it as a test failure.

So what this attempts to do is: at the end of all update tests,
check for all installed packages with the same name as a package
from the update, and compare their full NEVR to the one of the
package from the update. If a package with the same name as one
of the update packages is installed, but does not appear to be
the *same NEVR*, we fail, and upload the lists of packages for
manual investigation as to what the heck's going on.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-12-12 22:17:29 -08:00
Adam Williamson
65b4755358 Simplify anaconda error screen handling, add new needle
There's really no point having separate error and error_report
needles. Just match on error_report as well as clicking on it.
Also add a new error_report needle for latest Rawhide fonts.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-12-12 22:06:20 -08:00
Adam Williamson
0639468de6 Use -l for systemctl status (to avoid ellipsization)
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-12-12 12:36:21 -08:00
Adam Williamson
cb035c7737 Still fixing up this serial logging stuff
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-12-12 12:28:23 -08:00
Adam Williamson
95b227b97a Sigh, fix a syntax error in previous commit
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-12-12 11:57:53 -08:00
Adam Williamson
0f5281f389 In post_fail_hooks, try using serial line if no network
Sometimes we get a test failing because the SUT isn't connecting
to the network for some reason. In this case we never get any
logs, because `upload_logs` relies on being able to reacht at
least the worker host system via the network.

This attempts to detect when we can't ping the worker host, and
in that case, send some info out over the serial line instead.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-12-12 11:40:58 -08:00
Adam Williamson
25d8b56fe9 Drop 'get_milestone' from utils (it's unimplemented and unused)
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-11-30 14:22:08 -08:00
Adam Williamson
56060ff8bd Simplify and improve how we get to 'linux' line in grub
That whole creaky edifice of conditionals that figured out how
many times to press 'down' was a mess I always hated, and I just
found out that the fix for BLS wasn't complete - I'd assumed in
writing it that systems weren't being migrated to BLS on upgrade
to F30, but actually they are. This makes that design very hard
as we'd have had to find a way to change the number of 'down'
presses part-way through update tests, and all the ways I can
think of to do that would've made this even sillier.

Happily I managed to come up with what looks like a much simpler
approach: just go from the bottom. It seems that in every setup
I can think of to check - all three arches, BLS, no BLS, pre-
install, post-install - the linux line is two lines up from the
bottom of the config stanza (the last line is blank, and the
last line but one is the initramfs line). So we can just press
down 50 times (to make damn sure we're at the bottom) then press
up twice and we should be in the right place, no matter the arch,
the release, or if BLS is in use or not. Whew.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-11-30 14:17:01 -08:00
Adam Williamson
0d54e222f7 Pull in the update that fixes the FreeIPA / authselect bug
This bug is breaking all update FreeIPA tests; until the updates
go stable, let's pull them in to update tests so the results
are useful.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-11-28 17:42:58 -08:00
Adam Williamson
7dc50d3285 Drop no-longer-needed F29 update test hotfix
The update went stable months ago.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-11-27 17:30:25 -08:00
Adam Williamson
d6f26403a6 Use different file for checking post-upgrade release version
The one we were using before doesn't seem to exist any more in
Rawhide. /etc/os-release should be fine.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-11-15 12:46:24 -08:00
Adam Williamson
9d50f4f4a8 Tweak bootloader handling for BLS
Now the BLS stuff is enabled in Rawhide, we need to press 'down'
a different number of times to reach the 'linux' line when
editing the boot params (I really, really wish there was a
better way to do this :<). It gets tricky as there are all sorts
of cases here (support_server tests use a CURRREL disk image,
and then there's upgrade tests)...I think this covers things for
now.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-11-15 12:06:56 -08:00
Adam Williamson
6ddb475928 RHBZ#1644919: allow longer for console login, with a soft fail
Since a recent sssd update, console login during FreeIPA tests
is taking unusually long. We don't want this to fail all the
tests, so let's extend the timeout, but with a soft fail.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-10-31 16:33:32 -07:00
Adam Williamson
e6c8c5f0ff Work around Firefox 'close multiple tabs' warning
Somehow, recently, FreeIPA tests are running into Firefox not
quitting because it's showing a warning about closing multiple
tabs. (I think we didn't *get* multiple tabs before but now we
do, for some reason). So let's work around this by clicking
"Close tabs" if the warning appears.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-10-30 18:34:37 -07:00
Adam Williamson
6f1d9565d7 Add a temp hotfix for #1636633 for F29 update tests
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-10-09 16:40:31 -07:00
Adam Williamson
52b0fbaa32 D'oh, fix syntax error in previous commit
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-10-06 08:52:08 -07:00
Adam Williamson
9869920f5b Use longer timeout for root console switch after liveinst
For some reason, in recent tests, switching to a console after
live install completes is taking a long time, and tests are
failing because we 'only' allow 10 seconds for the login prompt
to appear. This seems to indicate some kind of performance bug,
but we don't really want all liveinst tests to fail on in, this
is not primarily a performance testing framework. So let's
tweak the root_console / console_login bits a bit to allow a
configurable timeout for the login prompt to appear, and use
that to wait 30 secs instead of 10 in this case.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-10-06 08:44:34 -07:00
Adam Williamson
1d6491d143 Workaround Workstation live on tty2 not tty1 (RHBZ #1635033)
In recent Rawhide, it seems the Workstation live session runs on
tty2 not tty1 for some reason. This throws off anacondatest
root_console, which assumes there'll be a vt on tty2. Handle it
by using tty3 instead if we're in a GNOME live environment.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-10-01 14:24:40 -07:00
Adam Williamson
e9e44a749e Add download_modularity_tests to exports from utils
This is probably my fault from when I was fixing up the PR.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-09-30 09:33:22 -07:00
Adam Williamson
a49f328dc6 Tweak how update-upgrade tests are handled a bit
Looking at this, it's a bit weird: the updated packages are
actually included in the upgrade process, but we still run
_advisory_update, which does basically nothing...then reboots.
That's kinda silly and makes the tests a bit flaky, let's fix
it. I don't think there's actually any problem with doing the
upload of updatepkgs.txt in _repo_setup_updates, becase that
already guards against being run more than once, it just bails
very early if it's already been run.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-09-28 17:23:51 -07:00
Lukas Ruzicka
24e68aa8a2 Create openqa tests to test modularity. 2018-09-26 23:09:36 -07:00
Adam Williamson
ad99c3fc23 Handle Xorg starting unusually slowly, but soft fail (#1622254)
There seems to be a bug in Rawhide lately where, when our tests
want to install a bare X and run Firefox on it, this takes an
unusually long time to start up, with SELinux in enforcing mode.
With SELinux in permissive mode it starts as fast as usual. This
isn't a hard failure and we don't want it to block all later
tests, so let's handle it and treat it as a soft fail.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-08-24 15:39:16 -07:00
Adam Williamson
5d99d56f8d Factor out the grub error loop, apply it to decrypt-on-boot
OK, we now need to work around this goddamn grub bug in *three*
places, so let's stop copying the loop around and factor it out
instead. The third place is encrypted installs, as they wait
for the decryption prompt on boot.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-08-19 10:29:23 -04:00
Adam Williamson
7df2bfb4f3 Workaround RHBZ#1618928 (hit space till errors go away)
Per Neal Gompa boot will proceed if we just page through the
error(?) messages displayed when #1618928 happens, so let's do
that to let the tests get further and see what else is broken.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-08-18 17:37:42 -04:00
Adam Williamson
8273d584e6 yay more fking magic sleeps
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-06-25 10:31:53 -07:00
Adam Williamson
861ad5d4aa Load us layout before doing post-install aarch64 cmdline hack
It seems that for some reason the localized layout gets loaded
on the installer VTs by this point in time, so we need to load
'us' again for this complex command to work.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-06-25 09:35:56 -07:00
Adam Williamson
b1140dafde Same workaround as last commit, for device type
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-06-15 14:34:56 -07:00
Adam Williamson
9b7a9c9cd8 Work around a partition scheme selection bug on aarch64
Sometimes on aarch64 clicking the partition scheme drop-down
just doesn't seem to make the menu appear, instead the button
goes active but that's all. It's very unlikely we'll be able
to track down why as this doesn't happen in manual testing on
aarch64 (according to @pwhalen), so instead let's just work
around it.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-06-15 13:40:13 -07:00
Adam Williamson
05c9f4fbcd Make sure all check_screen calls have explicit timeout
Upstream is gonna change the default from 30 to 0, it seems:
https://github.com/os-autoinst/os-autoinst/pull/965
so let's go ahead and change these two cases where we have no
explicit timeout to have one.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-05-24 14:17:24 -07:00
Adam Williamson
33ac181955 Use mirrorlist instead of baseurl for updates tests
The reason we have all this horrible code to use the commented-
out baseurl lines in the repo files instead of the metalinks
that are usually used is a timing issue with the metalink
system. As a protection against stale mirrors, the metalink
system sends the package manager a list of mirrors *and a list
of recent checksums for the repo metadata*. The package manager
goes out and gets the metadata from the first mirror on the
list, then checksums it; if the checksum isn't on the list of
checksums it got from mirrormanager, it assumes that means the
mirror is stale, and tries the next on the list instead.

The problem is that MM's list of checksums is currently only
updated once an hour (by a cron job). So we kept running into
a problem where, when a test ran just after one of the repos
had been regenerated, the infra mirror it's supposed to use
would be rejected because the checksum wasn't on the list - but
not because the mirror was stale, but because it was too fresh,
it had got the new packages and metadata but mirrormanager's
list of checksums hadn't been updated to include the checksum
for the latest metadata.

All this baseurl munging code was getting ridiculous, though,
what with the tests getting more complicated and errors showing
up in the actual repo files and stuff. It occurred to me that
instead of using the baseurl we can just use the 'mirrorlist'
system instead of 'metalink'. mirrorlist is the dumber, older
system which just provides the package manager a list of mirrors
and nothing else - the whole stale-mirror-detection-checksum
thing does not happen with mirrorlists, the package manager just
tries all the mirrors in order and uses the first that works.
And happily, it's very easy to convert the metalink URLs into
mirrorlist URLs, and it saves all that faffing around trying to
fix up baseurls.

Also, adjust upgrade_boot to do the s/metalink/mirrorlist/
substitution, so upgrade tests don't run into the timing issue
in the steps before the main repo_setup run is done by
upgrade_run, and adjust repo_setup_compose to sub this line out
later.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-05-18 16:41:13 -07:00
Adam Williamson
452140fdbb Re-arrange the repo fixups to be a bit clearer and more correct
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-05-03 11:51:50 -07:00
Adam Williamson
7912648e5b Escape the $ in previous commit, tweak repo debugging bits
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-05-03 11:28:30 -07:00
Adam Williamson
6089ec6c55 More repo baseurl munging, for update upgrade tests to F28 now
Now F28 went stable, we're not disabling updates on upgrade any
more, and this bug got exposed: the location of the updates and
updates-testing repos actually changed between F27 and F28, so
the `baseurl` line from fedora-repos in F27 isn't correct for
F28. When doing an upgrade from < 28 to > 27, we need to correct
the URL when we're done installing stuff from the old release
repos but before we start trying to pull stuff from the new
release repos.

This repo munging crap is really getting fragile, it'd be great
if we could get that metadata timing issue resolved so we could
reliably use mirrormanager...

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-05-03 10:06:58 -07:00
Adam Williamson
6b62cb87ed Another fix for the repo fix...
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-05-02 15:55:59 -07:00
Adam Williamson
c236fbcd21 Drop a stray # in previous commit, which made it not work
This is from my local test, sigh.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-05-02 15:48:24 -07:00
Adam Williamson
cef45346b6 Try to fix up some errors in fedora-repos baseurls in F28
This is breaking all f28 update tests, sigh.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-05-02 15:35:50 -07:00
Adam Williamson
cec1da9372 Remove bodhi-client 0.9 support in repo_setup_updates
...don't need it any more, F25 went EOL ages ago.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-05-02 15:08:34 -07:00
Adam Williamson
e931cfa0a5 Test FreeIPA upgrade on updates
This adds the FreeIPA server and client upgrade tests to a new
updates-server-upgrade flavor which fedora_openqa will schedule
for updates. This way, we can test whether updates break
FreeIPA upgrades, which is a request the FreeIPA team made to
me. This has been deployed on staging for the last week or so
and appears to work fine.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-04-26 11:35:18 -07:00
Adam Williamson
7e8a3010bb Fix bracket matching in previous commit
Syntax checks? In a text editor? THAT'S CRAZY TALK

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-04-09 17:19:03 -07:00
Adam Williamson
51b46bab7b g-i-s suppression change now applies to FAW 28+
Since gnome-initial-setup-3.28.0-5.fc28 , the g-i-s screens
that are supposed to be suppressed as part of
https://fedoraproject.org/wiki/Changes/ReduceInitialSetupRedundancy
are now suppressed on FAW installs as well as traditional ones.
So adjust the logic accordingly.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-04-09 16:50:16 -07:00
Adam Williamson
23192baa5d Use better pattern for checking if there are coredumps
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-04-04 14:13:39 -07:00
Adam Williamson
cdf33cc2ae Catch and upload coredumps in post-fail hook
We were doing this in a post-install test, but not on failures.
We need it to figure out why Firefox is crashing on aarch64...

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-04-04 12:10:33 -07:00
Adam Williamson
89bdea410f Rejig how we count the g-i-s screens we'll encounter
Trying to keep track of what these magic numbers mean is really
getting messy, so let's do it a bit more explicitly, using the
page names g-i-s uses internally, and lots of comments. This
should make it clearer and more maintainable when stuff changes.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-03-29 19:06:51 -07:00
Adam Williamson
baa7ac4e39 Handle KDE update test when KDE has aleady found updates
We do the 'desktop update' test for KDE via the notification
icon thingy, and it behaves differently depending on whether it
has already detected there are updates or not. The test only
works at present in the case where it *hasn't* - it expects the
notification icon to be in the extended panel and it expects to
see a 'refresh' button, neither of which is the case if it's
already noticed there are updates to install.

We should also force PackageKit to update its list of available
updates after we set up our 'special' update, otherwise on this
path KDE will only install the updates it found *before* we did
our stuff, and the test will fail as our special update won't be
there.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-03-28 19:53:19 -07:00
Adam Williamson
75d1b80db7 Bump a magic timeout a bit for aarch64
aarch64 managed to hit the problem this 'magic timeout' tries
to avoid, so let's extend it :(

e.g. https://openqa.stg.fedoraproject.org/tests/267174

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-03-27 11:45:22 -07:00
Adam Williamson
81540165e8 Adjust repo_setup for add-on Modularity
I believe this should do all the right repo modifications for
add-on Modularity (i.e. F28+ Server installs, for now).

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-03-26 13:27:31 -07:00
Adam Williamson
2c26ce1ede Simply repo_setup by removing MODULAR bit
We don't need all the MODULAR stuff any more as we no longer do
oldModularity composes like that.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-03-26 12:56:10 -07:00
Adam Williamson
669df14566 FreeIPA: handle unexpected auto-login to webUI
There are cases where we get logged back into the FreeIPA web UI
automatically by a stale kerberos ticket or something. If we're
logged in as the *right* user, let's just treat this as a soft
failure and continue with the test.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-03-21 15:43:40 -07:00
Adam Williamson
e685f00bd0 More tweaking for UEFI boot params on aarch64
Seems we're dealing with a bug, here.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-03-20 15:07:02 -07:00
Adam Williamson
20c361999b Fix post-install bootloader args on aarch64
Seems aarch64 needs 12 'down' key presses like ppc64, not 13
like x86_64. Tweak how this is done a bit; the ternary wasn't
elegant any more with the aarch64 change, so just get rid of it.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-03-20 14:05:00 -07:00
Adam Williamson
d162e4df16 Disable 'updates' repo for Branched update tests
This works around RHBZ #1552814, and it's not incorrect really
because the repo is always empty for Branched. I didn't do it
before because we might theoretically start using the repo for
Branched at some point in the future, and if we did that we'd
probably want it enabled for this test. But to get F28 update
tests working, let's just turn it off for now.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-03-10 11:01:35 -08:00
Adam Williamson
f1913e8305 Change g-i-s 'nexts' count for F28+
Several screens are now skipped in g-i-s pre-login mode, we
have to adjust the number we expect.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-03-09 18:37:01 -08:00
Adam Williamson
0915b857f9 More tweaking for this damn no root password spoke situation
Previous approach wouldn't work for tests that run after the
install test...let's just set a password from a chroot after
install completes. Don't really like this as it changes the
'real' install process a bit, but it's the least invasive short
term fix at least. We can maybe do something more sudo-y later
with a bit more thought.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-03-08 20:31:14 -08:00
Adam Williamson
83c32fe04e More tweaking for Workstation live scenario
It's really INSTALL_NO_USER, not USER_LOGIN='false'. Also, we
need to make root_console work with no root password, sigh.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-03-08 19:22:21 -08:00
Michel Normand
18602d72f3 Change baseurl for PowerPC repo as "fedora-secondary"
in _repo_setup_updates function

Signed-off-by: Michel Normand <normand@linux.vnet.ibm.com>
2017-11-28 08:18:16 +01:00
Adam Williamson
2a7b2923f7 Handle BGO#790811 (GNOME 'getting started' crash)
There's a bug causing the 'getting started' screen to crash.
This doesn't really make the system unusable, so treating it
as a soft failure seems appropriate, especially as this will
unblock all the post-install tests on Workstation.
2017-11-24 16:36:31 -08:00
Adam Williamson
da7eed3be6 Handle unbranded Cockpit login screen, record soft failure
There's a bug in current Modular Server which results in Cockpit
having an unbranded login screen, let's handle this but record
it as a soft failure.
2017-11-07 16:01:07 -08:00
Adam Williamson
442c143144 Try to fix quoting in printf 2017-11-07 15:00:08 -08:00
Adam Williamson
ce07c418d0 Fix (hackily) install of X, Firefox and haveged on Modular
Modular composes don't include these packages, but we need them
to run the web UI tests for FreeIPA and Cockpit. This is the
most reasonable hack I can come up with for now: just use a
non-modular fedora repo to source these packages when doing
Modular compose testing.

If we ever reach an all-Modular future, these packages should
be available in Modular composes I guess, but for now they are
not.
2017-11-07 14:20:36 -08:00
Adam Williamson
72395f1f67 Remove stray apostrophe in _repo_setup_compose 2017-11-06 17:42:30 -08:00
Adam Williamson
5e628551d8 Use Server not Everything variant for http repos on modular
Modular Server composes don't contain an Everything variant, so
we can't use it for the installation repo.
2017-11-06 16:25:40 -08:00
Adam Williamson
833502c880 Don't comment out metalink lines in repo files for modular
We can't easily do the 'point to the compose' fixup.
2017-11-06 16:20:40 -08:00
Adam Williamson
aaba51768e Don't use dnf config-manager on modular images
It's not there and can't be installed, at least at present.
2017-11-06 16:12:55 -08:00
Michel Normand
4610408963 Do not try to login if already done
the comment in console_login was already valid
but there was a missing return after test.

exemple of incorrect output is:
https://openqa.stg.fedoraproject.org/tests/182970#step/_collect_data/4

Signed-off-by: Michel Normand <normand@linux.vnet.ibm.com>
2017-10-18 18:53:11 +02:00
Michel Normand
e6fe996613 Revert "Modify "install_multi" test for PowerPC"
This reverts commit 8b2977f1d618316ded61420df4fc7d2afd07cbf4.

The initial commit was required for PowerPC
until qemu 2.7.1-6 (in f25) not required anymore
since qemu 2.9.0-5 (in f26)
2017-09-06 08:43:04 +02:00
Michel Normand
b1662c5f9d Add a 1s delay between each down key in do_bootloader
and call save_screenshot to visually check
for debug purpose only
Also change for PowerPC the number of down key to 12
(rather than 12)
Seems to be mandatory since 20170327.

Signed-off-by: Michel Normand <normand@linux.vnet.ibm.com>
2017-09-06 08:43:04 +02:00
Guy Menanteau
9c8a4949fa Modify "install_multi" test for PowerPC
PowerPC arches have the empty disk automatically
mounted on the second position in anaconda (vdb).
Thus, trig installation on second disk.
Change disk checking to point on correct disk.

Warning: this is a workaround specific correction
addressing a specific case.
This will have to be improved/changed with a more
generic code as suggested by Adam Williamson in
https://pagure.io/fedora-qa/os-autoinst-distri-fedora/pull-request/1#comment-31858
proposal for a next commit :)

Signed-off-by: Guy Menanteau <menantea@linux.vnet.ibm.com>
Signed-off-by: Michel Normand <normand@linux.vnet.ibm.com>
2017-09-06 08:43:04 +02:00
Michel Normand
6659e264b7 update repo_setup for PowerPC f25
to avoid upgrade_server test to fail with:
"Repository fedora-source has no mirror or baseurl set."

Signed-off-by: Michel Normand <normand@linux.vnet.ibm.com>
2017-09-06 08:43:04 +02:00
Guy Menanteau
06839422e5 First PowerPC specific changes (not templates)
* New OFW variable to identify Open Firmware (used by PowerPC)
* Few needles changes for PowerPC support
* as requested do not change the timers value below for PowerPC
  tests/install_source_graphical.pm (300 to 600)
  tests/_boot_to_anaconda.pm (300 to 1200)
  This will be handled by TIMEOUT_SCALE in templates

Signed-off-by: Guy Menanteau <menantea@linux.vnet.ibm.com>
2017-09-06 08:43:04 +02:00
Adam Williamson
c2bb74c64d Explicitly install 'koji' package in update repo setup
bodhi-client doesn't depend on the 'koji' package but does need
it to do 'bodhi updates download', which we want to do. So we
must explicitly install it here.
2017-09-04 11:05:00 -07:00
Adam Williamson
b6c23ff4cf Hide cursor at start of prelogin gnome-initial-setup
For the same reason we do it at the start of GDM (recent mutter
dumps a mouse cursor in the middle of the screen on startup).
2017-09-02 16:16:54 -07:00
Adam Williamson
9b91c839a7 Treat failure to source bash profile as a soft fail
There's a bug in current Rawhide causing sourcing of /etc/bashrc
to fail when logging in as a regular user. This results in the
bash prompt looking different, which is currently a hard fail,
and causes most tests to die. It's better to treat this as a
soft fail so the rest of the test can run. So add a needle to
spot this case, and a little finish function the console login
function calls whenever it's successfully logged in, to check
whether it got the no-profile prompt and register a soft fail.
2017-09-02 12:49:56 -07:00
Adam Williamson
ad6186daf1 Use super not alt-f1 for the 'clean desktop' workaround
Using alt-f1 might cause problems if we do it while we're at a
TTY (it'll switch to TTY1, which may not be what we want).
2017-07-10 15:05:36 -07:00
Adam Williamson
aca7de2861 Change up 'clean desktop' check again (use a util function)
Well, that OCR needle isn't working out so great, as it seems
to match when it shouldn't:

https://openqa.fedoraproject.org/tests/119217#step/_graphical_wait_login/5

So let's try another approach. Ditch the OCR needle and have a
function for checking we're at a clean desktop. It does the
normal needle match, but if we're on GNOME, it also tries
hitting alt+f1 and seeing if we're at the overview; if so, it
hits alt+f1 again (to go back to the desktop) and returns.
2017-07-10 11:47:07 -07:00
Adam Williamson
c3fd611f2d Finally revert anaconda root_console workaround
RHBZ #1222413 was fixed long ago. This workaround is, I think,
the cause of openQA failures to run commands properly with an
extraneous '2' at the start of the command (e.g. 116864).
2017-07-04 09:48:40 -07:00
Jan Sedlák
5e2a76efde fix blivet so that correct unit for size is selected
Differential Revision: https://phab.qa.fedoraproject.org/D1193
2017-05-05 09:29:55 +02:00
Jan Sedlák
140c5f0a42 Add custom partitioning tests for blivet
Differential Revision: https://phab.qa.fedoraproject.org/D1188
2017-04-24 14:23:35 +02:00
Adam Williamson
654534e694 Work around RHBZ #1439429 when running Firefox on X 2017-04-05 18:22:09 -07:00
Adam Williamson
1444c5030b Add tests with no user created during install
Summary:
This adds a new test suite, run for Workstation and KDE live
images, which does not create a user during install. It then
expects initial-setup (KDE) or gnome-initial-setup (Workstation)
to appear after install, creates a user, and proceeds with
normal boot.

Note the ARM image test already covers the initial-setup text
mode, and the ARM minimal image is the only case where that
actually matters (it's not included in Server).

Test Plan:
Run the new tests, check they work. Run all old
tests, check the changes didn't break them.

Reviewers: jsedlak, jskladan

Reviewed By: jsedlak

Subscribers: tflink

Differential Revision: https://phab.qa.fedoraproject.org/D1185
2017-04-05 09:43:26 -07:00
Adam Williamson
7977b41804 Give update download yet more time (we have some big updates) 2017-03-24 14:10:29 -07:00
Adam Williamson
ba3a5152c1 Improve FreeIPA debug logging a bit
Committing without review as this is pretty trivial and I've
had it on staging for the last few days without issue. Just gets
us somewhat better info for debugging FreeIPA issues.
2017-03-16 12:36:33 -07:00
Adam Williamson
e4a8929465 Delete openh264 repo definition during repo_setup
This repo is causing problems for Branched update tests. The
repo is not available for 26 at all yet. This shouldn't be a
problem as the repo is disabled by default, but it seems that
some things - at least realmd, as used in the FreeIPA enrolment
tests - still try to update the repo's metadata when installing
packages, and fail because it 404s.

Since none of our tests actually needs this repo AFAIK, let's
just delete it in repo_setup.
2017-03-15 10:02:25 -07:00
Adam Williamson
bb1be2f0b1 Disable updates-testing for updates tests on Branched 2017-03-13 14:49:27 -07:00
Adam Williamson
da01d15406 Fix repo setup for Branched update tests (I hope)
Branched update tests are all failing because the baseurl in
fedora.repo is incorrect for Branched. This is a rather hacky
fix for this problem. It relies on the scheduler setting the
DEVELOPMENT variable when the update is for Branched (I named
the variable DEVELOPMENT rather than BRANCHED to be more
future-proof).

Alternative options I rejected were:

i) stick with MM links
ii) do something 'clever' to retrieve the URLs from MM

Rejected i) because the timing problem where the infra repo gets
updated before MM has the updated repodata checksums is just too
much of a problem; whenever that happens, dnf will refuse to use
the metadata from the infra repo and go pull it from an external
mirror, which can wind up timing out.

Rejected ii) because it seemed too fancy and not really any more
robust than just doing this and adapting it if Things Change In
Future (TM).
2017-03-13 12:43:54 -07:00
Adam Williamson
8eea2a5d1f Bump the timeout on the initial 'dnf -y update' for update tests 2017-03-03 15:55:31 -08:00
Adam Williamson
461f3a6132 Update testing: log packages in update and installed packages
Summary:
This adds some logging related to the update testing workflow,
so we have some idea what we actually tested. We log precisely
which packages were actually downloaded from the update - this
is important as updates can be edited and when examining results
we'll want to know which packages actually got used. We also
add a new module which runs at the end of postinstall and tries
to figure out which packages from the update were installed in
the course of the test. This still isn't a guarantee the test
actually *tested them* in any way, but it at least means they
got installed successfully and didn't interfere with the test.

Test Plan:
Run the update test workflow, check the logs get
uploaded and seem accurate (sometimes some RPM garbage messages
wind up in the package log, I'm not too worried about that at
present). Run the compose test workflow and check it didn't
break.

Reviewers: jsedlak

Reviewed By: jsedlak

Subscribers: tflink

Differential Revision: https://phab.qa.fedoraproject.org/D1149
2017-02-23 14:51:19 -08:00
Adam Williamson
92d588f245 Add support for testing updates
Summary:
This adds an entirely new workflow for testing distribution
updates. The `ADVISORY` variable is introduced: when set,
`main.pm` will load an early post-install test that sets up
a repository containing the packages from the specified update,
runs `dnf -y update`, and reboots. A new templates file is
added, `templates-updates`, which adds two new flavors called
`updates-server` and `updates-workstation`, each containing
job templates for appropriate post-install tests. Scheduler is
expected to post `ADVISORY=(update ID) HDD_1=(base image)
FLAVOR=updates-(server|workstation)`, where (base image) is one
of the stable release base disk images produced by `createhdds`
and usually used for upgrade testing. This will result in the
appropriate job templates being loaded.

We rejig postinstall test loading and static network config a
bit so that this works for both the 'compose' and 'updates' test
flows: we have to ensure we bring up networking for the tap
tests before we try and install the updates, but still allow
later adjustment of the configuration. We take advantage of the
openQA feature that was added a few months back to run the same
module multiple times, so the `_advisory_update` module can
reboot after installing the updates and the modules that take
care of bootloader, encryption and login get run again. This
looks slightly wacky in the web UI, though - it doesn't show the
later runs of each module.

We also use the recently added feature to specify `+HDD_1` in
the test suites which use a disk image uploaded by an earlier
post-install test, so the test suite value will take priority
over the value POSTed by the scheduler for those tests, and we
will use the uploaded disk image (and not the clean base image
POSTed by the scheduler) for those tests.

My intent here is to enhance the scheduler, adding a consumer
which listens out for critpath updates, and runs this test flow
for each one, then reports the results to ResultsDB where Bodhi
could query and display them. We could also add a list of other
packages to have one or both sets of update tests run on it, I
guess.

Test Plan:
Try a post something like:
HDD_1=disk_f25_server_3_x86_64.img DISTRI=fedora VERSION=25
FLAVOR=updates-server ARCH=x86_64 BUILD=FEDORA-2017-376ae2b92c
ADVISORY=FEDORA-2017-376ae2b92c CURRREL=25 PREVREL=24

Pick an appropriate `ADVISORY` (ideally, one containing some
packages which might actually be involved in the tests), and
matching `FLAVOR` and `HDD_1`. The appropriate tests should run,
a repo with the update packages should be created and enabled
(and dnf update run), and the tests should work properly. Also
test a regular compose run to make sure I didn't break anything.

Reviewers: jskladan, jsedlak

Reviewed By: jsedlak

Subscribers: tflink

Differential Revision: https://phab.qa.fedoraproject.org/D1143
2017-02-22 11:33:32 -08:00
Adam Williamson
186678e98b Make log upload work when installed system hits emergency mode
Summary:
This is to handle cases like #1414904 , where the system boots
to emergency mode. We really need logs to try and debug this.

Test Plan:
Force a test to hit emergency mode somehow (right now
you can just run base_services_start on Rawhide over and over
until you hit #1414904, but there's probably an easier way to
do it, I think there's a systemd boot arg to tell it which target
to boot for e.g.) and check logs get uploaded. Also check this
doesn't break log upload for a 'normal' failure.

Reviewers: garretraziel_but_actually_jsedlak_who_uses_stupid_nicknames

Reviewed By: garretraziel_but_actually_jsedlak_who_uses_stupid_nicknames

Subscribers: tflink

Differential Revision: https://phab.qa.fedoraproject.org/D1103
2017-02-01 12:30:21 +01:00
Adam Williamson
e5dc67126d Fix check_release, menu_launch_type, start_cockpit
I accidentally left the `my $self = shift` lines in these when
changing them from methods into functions, so they don't work
right at all. Whoops. Sorry.
2017-01-26 14:32:42 +01:00
Adam Williamson
b67f604894 Move all remaining utility functions into exporter modules
Summary:
This adds a couple of new exporter modules, renames main_common
to utils (this is a better name: openSUSE's main_common is
functions used in main.pm, utils is what they call their module
full of miscellaneous commonly-used functions), and moves a
bunch of utility functions that were previously needlessly
implemented as instance methods in base classes into the
exporter modules. That means we can get rid of all the annoying
$self-> syntax for calling them.

We get rid of `fedorabase` entirely, as it's no longer useful
for anything. Other base classes keep the 'standard' methods
(like `post_fail_hook`) and methods which actually need to be
methods (like `root_console`, whose behaviour is different in
anacondatest and installedtest).

Test Plan:
Do a full test suite run and check everything lines
up. There should be no functional differences from before at all,
this is just a re-org.

Reviewers: jskladan, garretraziel_but_actually_jsedlak_who_uses_stupid_nicknames

Reviewed By: garretraziel_but_actually_jsedlak_who_uses_stupid_nicknames

Subscribers: tflink

Differential Revision: https://phab.qa.fedoraproject.org/D1080
2017-01-17 23:15:44 -08:00
Adam Williamson
e848ce768e Fix copy_host_file for files containing quotes (I hope)
`man printf` says \" is treated as a quote, but not \'. So
let's have the command use double quotes to wrap the format,
and escape any double quotes in the text. Hope this works.
2017-01-15 10:52:14 -08:00
Adam Williamson
e0418b3328 Update and unwrap README, move function docs in-line
The README looks pretty ugly on Pagure. So let's unwrap it.
Let's also move the function docs into the source files. We're
much more likely to keep them up to date that way, I think. We
should probably change over to proper perl POD documentation at
some point, but comments in-line are OK for now I think.
2017-01-12 14:27:42 -08:00
Adam Williamson
c4f32ab5ad add an Asian (Japanese) language install test
Summary:
Include some basic testing of Japanese input, and split the
input testing (including Russian) into a separate module, since
it's not really part of 'login' testing.

Test Plan:
Run the test, and the Russian and French tests too to
make sure they didn't break. Tested on staging. Note the Japanese
test soft fails, intentionally, at present, as I discovered a bug
while working on it:
https://bugzilla.gnome.org/show_bug.cgi?id=776189

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D1072
2016-12-21 08:41:00 -08:00
Adam Williamson
7135a4e05d iSCSI: test user authentication
Summary:
This isn't in the criteria, but it's commonly used, so we ought
to test this way. Require authentication for the iSCSI target
and have the test provide the appropriate auth info.

Test Plan:
Run the iscsi test and check it works (you need the
recent fixes for support_server to make *that* work). Nothing
else should be affected.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D1070
2016-12-20 08:34:30 -08:00
Adam Williamson
0fa6138448 Have non-English tests do graphical install and login
Summary:
The non-English tests so far did not test that graphical login
worked as expected, which is a fairly large hole. With this
change, they should do a Workstation install and test login to
both GNOME and the console works as expected. KDE is not yet
tested.

As part of this we tweak the implementation of keyboard layout
switching in graphical environments to use a generic function
in main_common which can handle both anaconda and desktops
(just GNOME at present, but should extend easily to any desktop
with a known switcher key and a visible layout indicator),
replacing the anacondatest class method. I kinda don't like that
the test has to specifically tell the function when it's in
anaconda, but I don't think I want to start experimenting with
a global 'test phase' openQA variable or anything like that at
present.

Fixes T842.

Test Plan:
Run the French and Russian install tests and check
they work as expected. Also run an English Workstation install
if you like, and make sure that didn't break. This change is
live on staging ATM, seems to work fine.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Maniphest Tasks: T842

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D1071
2016-12-16 09:40:29 -08:00
Adam Williamson
332a955814 disable updates as well as updates-testing in repo_setup
This should solve all those annoying "Failed to synchronize
cache for repo 'updates'" failures we've had: there's no need
for the 'updates' repository to be enabled when we've decided
we want the `repo_setup` changes to be made, and having it
enabled causes problems when we run right after the Rawhide
compose completes. We hit the awkward period where the rawhide
repo has been synced but mirrormanager has not been updated
with the new metadata checksums, so mirrormanager rejects the
metadata from dl.fp.o and DNF has to go out and hit other
mirrors until it finds one which didn't sync yet. Since the
point of `repo_setup` is specifically to hack up the config so
we only use packages from the compose *anyway*, there's no
reason at all to worry about leaving 'updates' enabled and
nerfing it like we do 'fedora' and 'rawhide', we can just turn
it off.
2016-12-15 16:11:37 -08:00
Adam Williamson
cc7fe4b971 Tweaks to AVC test loading / run
Committing without review as this causes failures...try to make
sure we only run the AVC test when it makes sense, and fix
running it on the French install test.
2016-12-08 12:03:26 -08:00
Adam Williamson
c4edf8009e Improve and simplify post_fail_hook existence checks
Summary:
The current installedtest post_fail_hook assumes /var/tmp/abrt
exists at all, and dies if it doesn't, leading to no /var/log
upload. We can also avoid using openQA `script_output` - which
is annoyingly indirect and slow - by using this neat `test -n`
trick I found on SO. Let's also use it in the anacondatest
post_fail_hook to avoid uploading /var/tmp when it's empty
(which we currently do). This also drops the 0 arg from a few
more script_run calls, because it's safe to wait for the run
to complete and we should probably do so to avoid later typing
errors if the commands are slow.

Test Plan:
Cause both anaconda and installed tests to fail and
check the hooks work as intended. Maybe twiddle the failures to
ensure directories do and don't exist and/or have contents and
make sure things work OK. I've tested this to some degree and
I'm pretty sure it works right.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D1041
2016-10-31 11:41:10 -07:00
Adam Williamson
600dd39a64 install tar in installedtest post_fail_hook
It's not always in minimal installs. This is a simple change
and needed to make the post-fail hook work for minimal installs,
so pushing without review.
2016-10-27 17:08:27 -07:00
Adam Williamson
dcb68d93c8 drop our implementation of script_run in favour of os-autoinst
Summary:
os-autoinst implements `script_run` itself now, we aren't
required to implement it ourselves any more. os-autoinst's
implementation is better than ours, as it allows for verifying
the script actually ran (via the redirect-output-to-serial-
console trick).

So this drops our implementation so we'll just use the upstream
one. Where I judged we don't want to bother with the 'check
the command actually ran' feature I've adjusted our direct
`script_run` calls to pass a wait time of 0, which skips the
'wait for command to run' stuff entirely and just does a simple
'type the string and hit enter'.

Because of how the inheritance works, our `assert_script_run`
calls already used the os-autoinst `script_run`, rather than
the one from our distribution.

This should prevent `prepare_test_packages` sometimes going
wrong right after removing the python3-kickstart package, as
we'll properly wait for that removal to complete now (before
we weren't, we'd just start typing the next command while it
was still running, which could result in lost keypresses).

Test Plan:
Check all tests still run OK (I've tried this on
staging and it seems fine).

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D1034
2016-10-20 09:24:48 -07:00
Adam Williamson
7869c3d07a drop check_type_string, use upstream merged code instead
Summary:
The code from `check_type_string` was effectively merged into
os-autoinst's `testapi::type_string` as an optional argument,
so let's drop this downstream version and just have the 'safe
typing' functions use `type_string`.

Test Plan:
Run tests, check they pass and work the same (i.e.
make sure they're actually checking for screen change when
typing).

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D1033
2016-10-20 09:12:55 -07:00
Adam Williamson
cf2ce903c5 drop cockpit workaround entirely
now 119 hit F25, this is no longer needed (and breaks the test
in fact).
2016-10-17 16:52:17 -07:00
Adam Williamson
660dde164f disable cockpit login workaround for Rawhide
Rawhide now has cockpit 119, where this is fixed.
2016-10-08 10:59:57 -07:00
Adam Williamson
bacb6f1f7b redo console_login with multiple matches, move to main_common
Summary:
Since we can match on multiple needles, we can drop the loop
from console_login and instead do it this way, which is simpler
and should work better on ARM (the timeouts will scale and
allow ARM to be slow here). Also move it to main_common as
there's no logical reason for it to be a class method.

Also remove the `check` arg. `check` was only set to 0 by two
tests, _console_shutdown and anacondatest's _post_fail_hook.

For _console_shutdown, I think I just wanted to give it the
best possible chance of succeeding. But we're really not going
to lose anything significant by checking, the only case where
check=>0 would've helped is if the 'good' needle had stopped
matching, and all sorts of other tests will fail in that case.

anacondatest was only using it to save a screenshot of whatever
was on the tty if it didn't reach a root console, which doesn't
seem that useful, and we'll get screenshots from check_screen
and assert_screen anyway.

Test Plan:
Run all tests, check they behave as expected and
none inappropriately fails on console login.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D1016
2016-09-30 08:42:45 -07:00
Adam Williamson
acb20f1a70 lengthen the wait_idle timeout in prepare_test_packages
we're still having trouble, the idle wait is timing out...
2016-09-28 18:26:29 -07:00
Adam Williamson
e9ce14a891 consolidate login waits, use postinstall not entrypoint for base
Summary:
I started out wanting to fix an issue I noticed today where
graphical upgrade tests were failing because they didn't wait
for the graphical login screen properly; the test was sitting
at the 'full Fedora logo' state of plymouth for a long time,
so the current boot_to_login_screen's wait_still_screen was
triggered by it and the function wound up failing on the
assert_screen, because it was still some time before the real
login screen appeared.

So I tweaked the boot_to_login_screen implementation to work
slightly differently (look for a login screen match, *then* -
if we're dealing with a graphical login - wait_still_screen
to defeat the 'old GPU buffer showing login screen' problem
and assert the login screen again). But while working on it,
I figured we really should consolidate all the various places
that handle the bootloader -> login, we were doing it quite
differently in all sorts of different places. And as part of
that, I converted the base tests to use POSTINSTALL (and thus
go through the shared _wait_login tests) instead of handling
boot themselves. As part of *that*, I tweaked main.pm to not
require all POSTINSTALL tests have the _postinstall suffix on
their names, as it really doesn't make sense, and renamed the
tests.

Test Plan: Run all tests, see if they work.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D1015
2016-09-27 11:48:15 -07:00
Adam Williamson
b4dc92ce67 wait_idle in prepare_test_packages, hope this fixes typing
See https://openqa.stg.fedoraproject.org/tests/46189 etc. - we
often seem to lose keystrokes in the command right after
removing python3-kickstart, let's hope a wait_idle helps.
2016-09-26 11:35:07 -07:00
Adam Williamson
ddc91efeff detect (rather than guessing) desktop vt
use 'ps' output for Xorg and Xwayland. We'd need some new
openQA var to get this right by 'guessing', as it's vt1 for
Workstation when running live - so long as autologin worked -
but vt2 after install. We'd need a var or some other thing to
detect which case we're running in. LIVE doesn't do it, it's
set even when running a post-install test from a live image.

So instead let's just do it a bit more cleverly. This also
gives us a bit of insurance against changes in GDM, SDDM etc.
behaviour, so long as Xwayland or Xorg is running (and we can
add additional processes to the list, like gnome-shell, if
needed/appropriate). We assume the *final* listed process -
i.e. the most recently-started one - will be the desktop;
this covers gdm's behaviour of starting up on vt1 then running
the user session on vt2. We can make this function more complex
and add args if we ever get to the point where we have multi-
user tests running or anything (e.g. allow to pass a username
and only look for that user's processes).

Landing without review as this broke the live variant of the
test on Workstation in production (kinda not sure why it worked
in testing, or I didn't notice that it failed, but never mind).
I've tested it on staging.
2016-09-24 13:04:04 -07:00
Adam Williamson
e8b20ec73f add a desktop_update_graphical test
Summary:
Very similar to the CLI update test, but using the desktops'
update applications. This is based off the CLI update test
branch as it uses the shared functions that branch introduced.
We do not use the fake update packages, as they don't really
do anything useful for these tests; for dnf they can help us
distinguish between issues with the dnf mechanism and issues
with the repos, but we can't really tell that in the graphical
case. So we only use the python3-kickstart package here.

Test Plan:
Run the test on both KDE and GNOME and ensure it
performs as intended. I've been testing it on staging, so you
can see it there.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D1010
2016-09-22 11:38:51 -07:00
Adam Williamson
44ec3d84c3 add a base_update_cli test
Summary:
this uses a couple of test repos with fake packages to test the
basic dnf mechanisms are working, then messes around with the
python3-kickstart package a bit to try and test the default repo
configuration is working, keys are in place and so on. We use
python3-kickstart because we should be able to rely on the copy
of that package in the 'stable' repo being installable (or else
the compose would have failed), but it shouldn't be vital to
the operation of the system.

Test Plan: Run the test, see if it works.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D1006
2016-09-22 10:57:12 -07:00
Adam Williamson
1cdd8e18b7 make sure we get logs from failed kickstart installs
Summary:
by waiting for the bootloader in _boot_to_anaconda rather than
_console_wait_login, we can ensure that we use the anaconda
post-fail hook and thus get logs uploaded when a kickstart
install fails.

Test Plan:
Run a kickstart install test that fails and check
anaconda logs get uploaded. Then run one that works and make
sure it...still works.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D1005
2016-09-20 10:51:51 -07:00
Adam Williamson
af00b71790 use tty6 for installedtest post_fail_hook
Summary:
tty2 is where wayland desktop sessions run. I think it makes
sense to use a high tty for the post_fail_hook, so we know the
lower ones can be used by the tests...

Test Plan:
Run a Workstation post-install test that fails, check
the hook works.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D1003
2016-09-19 16:17:39 -07:00
Adam Williamson
016074d247 make cockpit login screen workaround unconditional for now
cockpit 118 hit F25 stable, so we need this on both branches
now...next up, 119 will hit Rawhide, and we'll need to make
it F25-only...
2016-09-12 17:47:21 -07:00
Adam Williamson
e1ec1997af try to be safer when typing in X: slower, more checks
Summary:
the main thing this does is try and type slower in X - this
should cover nearly everywhere we type anything in X, and make
it type slower. We also add a bit more safety checking to some
old tests which didn't have it (mainly _do_install_and_reboot)
- wait_still_screen after typing to make sure all the keypresses
were registered before continuing.

This is an attempt to mitigate the problems we've seen where
the wrong text gets typed into the wrong places and the tests
break.

This branch is live on staging atm. It still has *some* issues,
but I do think it's an improvement.

Test Plan:
run the tests (probably several times), compare to
runs without the change, see if it's better or worse...

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D993
2016-09-12 10:24:30 -07:00
Adam Williamson
3672973b9d handle cockpit 'user name' field not pre-selected on Rawhide
cockpit 118 just landed in Rawhide, and it seems the username
field on the login screen is no longer selected by default,
you have to hit tab to navigate to it. We could get smart and
store the cockpit version in a variable or something, but it
doesn't seem worth it for now, let's just use a simple 'if
rawhide' conditional which can be adjusted as necessary as
things change.

https://github.com/cockpit-project/cockpit/issues/5000
2016-09-08 21:42:11 -07:00
Adam Williamson
03af57e0da start firefox at 1024x768 resolution
when we run firefox in a bare X session, by default we get an
800x600 firefox in a 1024x768 X server with some dead black
space to the right and bottom of the screen. Now it turns out
that if the mouse is in the dead space, Firefox will not get
any keystrokes we send.

This didn't used to be a problem, but I made it into one with
this os-autoinst change:

https://github.com/os-autoinst/os-autoinst/pull/559

that makes os-autoinst move the cursor to 1023,767 after each
`assert_and_click`, instead of 0x0 as it did before, unless the
cursor has previously been explicitly place somewhere. So in
this case it gets moved to the dead space, and Firefox stops
responding to keypresses after the first `assert_and_click`.

We could equally well fix this by setting the cursor to 0,0
after running Firefox, but I like this more as it makes sure
we won't run into the same problem some other way, and makes
the videos and screenshots look nicer.

This fixes the realmd_join_cockpit test that's been failing
ever since I installed an os-autoinst with that fix. Committing
without review as it's a straightforward fix and I want the
test working again...
2016-09-08 09:15:43 -07:00
Jan Sedlák
db95bccd52 add anaconda text UI test
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D980
2016-09-07 10:34:54 +02:00
Adam Williamson
ef689e75a9 use compose repository (not master repo) for most tests
Summary:
we have a long-standing problem with all the tests that hit
the repositories. The tests are triggered as soon as a compose
completes. At this point in time, the compose is not synced to
the mirrors, where the default 'fedora' repo definition looks;
the sync happens after the compose completes, and there is also
a metadata sync step that must happen after *that* before any
operation that uses the 'fedora' repository definition will
actually use the packages from the new compose. Thus all net
install tests and tests that installed packages have been
effectively testing the previous compose, not the current one.

We have some thoughts about how to fix this 'properly' (such
that the openQA tests wouldn't have to do anything special,
but their 'fedora' repository would somehow reflect the compose
under test), but none of them is in place right now or likely
to happen in the short term, so in the mean time this should
deal with most of the issues. With this change, everything but
the default_install tests for the netinst images should use
the compose-under-test's Everything tree instead of the 'fedora'
repository, and thus should install and test the correct
packages.

This relies on a corresponding change to openqa_fedora_tools
to set the LOCATION openQA setting (which is simply the base
location of the compose under test).

Test Plan:
Do a full test run, check (as far as you can) tests run sensibly
and use appropriate repositories.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D989
2016-09-01 08:22:59 -07:00
Adam Williamson
a901fce4ab add QA:Testcase_FreeIPA_password_change test
Summary:
again, added as a non-fatal module for realmd_join_cockpit as
it's convenient to do it here. Also abstract a couple of ipa
bits into a new exporter package in the style of SUSE's
mm_network, rather than using ill-fitting class inheritance as
we have before - we should probably convert our existing class
based stuff to work this way.

Also a few minor tweaks and clean-ups of the other tests:

The path in console_login() where we detect login of a regular
user when we want root or vice versa and log out was actually
broken because it would 'wait' for the result of the 'exit'
command, which obviously doesn't work (as it relies on running
another command afterwards, and we're no longer at a shell).
This commit no longer actually uses that path, but I spotted
the bug with an earlier version of this which did, and we may
as well keep the fix.

/var/log/lastlog is an apparently-extremely-large sparse file.
A couple of times it seemed to cause tar to run very slowly
while creating the /var/log archive for upload on failure. It's
no use for diagnosing bugs, so we may as well exclude it from
the archive.

I caught cockpit webUI login failing one time when testing the
test, so threw in a wait_still_screen before starting to type
the URL, as we have for the FreeIPA webUI.

I also caught a timing issue with the openQA webUI policy add
step; the test flips from the Users screen to the HBAC screen
then clicks the 'add' button, but there's actually an identical
'add' button on *both* screens, so it could wind up trying to
click the one on the Users screen instead, if the web UI took
a few milliseconds to switch. So we throw in a needle match to
make sure we're actually on the HBAC screen before clicking the
button.

We make the freeipa_webui test a 'milestone' so that if the
new test fails, restoring to the last-known-good milestone
doesn't take so long; it actually seems like openQA can get
confused and try to cancel the test if restoring the milestone
takes a *really* long time, and wind up with a zombie qemu
process, which isn't good. This seems to avoid that happening.

Test Plan:
In the simple case, just run all the FreeIPA-related
tests on Fedora 24 (as Rawhide is broken) and make sure they all
work properly. To get a bit more advanced you can throw in an
`assert_script_run 'false'` in either of the non-fatal tests to
break it and make sure things go properly when that happens (the
last milestone should be restored - which should be right after
freeipa_webui, sitting at tty1 - and run properly; things are
set up so each test starts with root logged in on tty1).

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D935
2016-08-03 13:21:12 -07:00
Adam Williamson
aacd01ea8b add encrypted workstation upgrade tests (current and previous)
Summary:
This requires us to handle decryption each time we reboot in
the upgrade process, so factor that little block out into the
base class so we don't have to keep pasting it. It's also a
bit tricky to integrate into the 'catch a boot loop' code we
have to deal with #1349721, but I think this should work. There
is a matching openqa_fedora_tools diff to generate the disk
image.

Test Plan:
Run the tests, check that they work, run the other
upgrade and encrypted install tests and check they still work
properly too.

Reviewers: garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D922
2016-07-08 08:56:57 -07:00
Adam Williamson
0da6652287 add NFS tests (and DHCP/DNS in the support server)
Summary:
Set up the support server to provide DHCP/DNS functionality and
an NFS server, providing a kickstart. Add a kickstart test just
like the other root-user-crypted-net kickstart tests except it
gets the kickstart from the support server via NFS. Also add NFS
repository tests and a second support server for Server-dvd-iso
flavor: this test must run on that flavor to ensure that packages
are actually available. The support server just mounts the
attached 'DVD' and exports it via NFS.

Note we don't need to do anything clever to avoid IP conflicts
between the two support servers, because os-autoinst-openvswitch
ensures each worker group is on its own VLAN.

As part of adding the NFS repo tests, I did a bit of cleanup,
moving little things we were repeating a lot into anacondatest,
and sharing the 'check if the repo was used' logic between all
the tests (by making it into a test step that's loaded for all
of them). I also simplified the 'was repo used' checks a bit,
it seems silly to run a 'grep' command inside the VM then have
os-autoinst do a grep on the output (which is effectively what
we were doing before), instead we'll just use a single grep
within the VM, and clean up the messy quoting/escaping a bit.

Test Plan:
Run all tests - at least all repository tests - and
check they work (make sure the tests are actually still sane,
not just that they pass). I've done runs of all the repo tests
and they look good to me, but please double-check. I'm currently
re-running the whole 24-20160609.n.0 test on staging with these
changes.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D888
2016-06-13 08:42:30 -07:00
Adam Williamson
7a8ae3a357 add an iscsi test, and a support_server test to support it
Summary:
this is following a SUSE model for tests where we need a server
end but don't want setting up the server to constitute a real
test in itself, we want it to be stable. The 'support_server'
test just boots a pre-built (by createhdds) disk image, sets up
networking, and runs the iSCSI server.

To run the iSCSI test we need to handle networking config in
anaconda (or we would need to set the support server up as a
DHCP server, which may be worth considering), so this adds that.
We also need to be able to specify the target device for a
volume in custom partitioning, so this adds that too.

Test Plan:
Build the necessary support server disk image (use
D883), then run the test and make sure it works. Also make sure
all other tests continue to work.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D884
2016-06-09 08:43:46 -07:00
Adam Williamson
66fc3cc7d4 add a cockpit realmd FreeIPA join test
Summary:
This requires a few other changes:

* turn clone_host_resolv into clone_host_file, letting you clone
  any given host file (cloning /etc/hosts seems to make both
  server deployment and client enrolment faster/more reliable)
* allow loading of multiple POSTINSTALL tests (so we can share
  the freeipa_client_postinstall test). Note this is compatible,
  existing uses will work fine
* move initial password change for the IPA test users into the
  server deployment test (so the client tests don't conflict over
  doing that)
* add GRUB_POSTINSTALL, for specifying boot parameters for boot of
  the installed system, and make it work by tweaking _console_wait
  _login (doesn't work for _graphical_wait_login yet, as I didn't
  need that)
* make the static networking config for tap tests into a library
  function so the tests can share it
* handle ABRT problem dirs showing up in /var/spool/abrt as well
  as /var/tmp/abrt (because the enrol attempt hits #1330766 and
  the crash report shows up in /var/spool/abrt, don't ask me why
  the difference, I just work here)
* specify the DNS servers from the worker host's resolv.conf as
  the forwarders for the FreeIPA server when deploying it; if we
  don't do this, rolekit defaults to using the root servers as
  forwarders(!) and thus we get the public, not phx2-appropriate,
  results for e.g. mirrors.fedoraproject.org, some of which the
  workers can't reach, so PackageKit package install always fails
  (boy, was it fun figuring THAT mess out)

Even after all that, the test still doesn't actually pass, but
I'm reasonably confident this is because it's hitting actual bugs,
not because it's broken. It runs into #1330766 nearly every time
(I think I saw *one* time the enrolment actually succeeded), and
seems to run into a subsequent bug I hadn't seen before when
trying to work around that by trying the join again (see
https://bugzilla.redhat.com/show_bug.cgi?id=1330766#c37 ).

Test Plan:
Run the test, see what happens. If you're really lucky,
it'll actually pass. But you'll probably run into #1330766#c37,
I'm mostly posting for comment. You'll need a tap-capable openQA
instance to test this.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D880
2016-06-07 13:00:39 -07:00
Adam Williamson
87da472f03 fix root console login with switched keyboard layout
we only 'double' the password for the user account, not for root
this is a trivial fix so just pushing it out to get cyrillic
test working at last (hopefully)
2016-06-06 12:00:47 -07:00
Adam Williamson
aba2814611 add cockpit_default and cockpit_basic tests
Summary:
This adds tests for the Server_cockpit_default and cockpit_basic
test cases. Some notes: I was initially thinking of combining
these into a single test with multiple test modules and coming
up with a system for doing wiki reporting based on individual
test module status, but because we'll also want to do a cockpit
FreeIPA enrol test, I decided against it. We don't really want
to combine all three because then we would skip the cockpit
tests whenever FreeIPA server deployment failed, which isn't
ideal. So since we'll need a separate FreeIPA enrolment test
anyway it doesn't really make sense to go to the trouble of
designing a system for loading multiple postinstall tests (though
I have an idea for that!) and a per-module wiki reporting system.

This was the most minimal and hopefully reliable method for
running Cockpit from a stock Server install that I could think
of. An alternative approach would be to have, say, the most
recent stable Workstation live as a 'stock' asset and have two
tests, one which runs a stock Server install and just waits and
another which boots the live image and accesses the cockpit
running on the other box, but that seems a bit over-complex. It
is not possible to have dependencies between tests for different
ISOs, in case you were wondering about having a Workstation live
test which runs parallel with a Server DVD test, we can't do
that. One funny thing is the font that winds up getting used for
the desktop, but I don't *think* that should be a problem.

Picking needles was a bit tricky; any improvement suggestions
are welcome. I'm hoping it turns out to be safe to rely on some
dbus log messages being present; I think logging into Cockpit
triggers activation of the realmd dbus interface, so there
*should* always be some messages related to that. An alternative
would just be to match on a sliver of the dark grey table header
and the light grey row beneath it and assume that'll always be
the first message (whatever the message is), but then we have to
find some area of the message details screen which is always
present for any message, and it just seems a tad more likely to
result in false passes. Similary I'm making an assumption that
auditd is always going to show up on the first page of the
Services screen and the details screen will always show that
'loaded...enabled' text.

Test Plan:
Run the tests and see if they work! See
https://openqa.stg.fedoraproject.org/tests/21373 and
https://openqa.stg.fedoraproject.org/tests/21371 for my tests.

Reviewers: garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D874
2016-06-01 09:05:33 -07:00
Adam Williamson
a1d240b135 add needles to treat 'Custom Operating System' as minimal
Summary:
the Server DVD now just has 'Fedora Server' and 'Custom
Operating System' environments. Custom is basically minimal.
So we can use the DVD for 'universal' testing again, these
needles match the anaconda_minimal tags.

Test Plan:
Run the 'universal' tests on a DVD ISO with these
needles added, test that they work OK and use the 'Custom' env.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D844
2016-05-20 12:56:44 -07:00
Adam Williamson
a4f3267534 add Russian install test
Summary:
Requires new needles and test suite and job template, plus a
few tweaks to handle 'switched' keyboard layouts (so we use the
switched layout in the username and password).

Test Plan:
Run the test and see that it...fails. But that's OK!
It's a genuine bug: RHBZ #1333998 . At least make sure it gets
to that point and no other tests have broken and all the needles
look sane.

Reviewers: garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D846
2016-05-20 07:52:55 -07:00