Commit Graph

137 Commits

Author SHA1 Message Date
Lukas Ruzicka
d82f4ecd7f Develop a new test for system updates via Cockpit.
* server-cockpit-updates tests that Cockpit can be used to update the system.
* server-cockpit-autoupdate tests that users can use dnf-automatic for system
updates.
* cockpit functions were removed from utils.pm and put into an extra library
for cockpit - cockpit.pm which all cockpit tests are now using.

Review cockpit.pm

Review autoupdate test.

Review the update test.

Fix typo in cockpit.pm

Add sleep.

Add missing command.

Delete an unused needle.
2019-10-01 16:04:01 +02:00
Adam Williamson
b32449f72a Add missing bracket
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-09-20 11:04:31 -07:00
Adam Williamson
6538a82096 Download by ADVISORY_NVRS with koji for updates, not ADVISORY
This goes with a corresponding fedora_openqa commit that passes
ADVISORY_NVRS when scheduling update jobs. This is to address
https://pagure.io/fedora-qa/fedora_openqa/issue/78 so we can
publish spec-compliant and correct CI Messages: we will decide
what NVRs are in the update at scheduling time and always get
and test those NVRs, rather than the test downloading whatever
is in the update when it is run.

Some consequences: manual restarting or cloning of an update
test scheduled before this change lands will result in failure;
to do this you'll have to add the ADVISORY_NVRS value manually
with clone_job.pl, or simply reschedule the update with
fedora-openqa.

Manual restarting or cloning of an update test scheduled with
this change will always test the same NVRs. If the update has
changed and you want to re-test with the new packages in the
update, you must change ADVISORY_NVRS manually with clone_job,
or reschedule with fedora-openqa.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-09-20 10:58:27 -07:00
Adam Williamson
7f96c3aa49 Remove a now-unneeded workaround
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-09-20 10:51:16 -07:00
Michel Normand
6455319ef5 typo in lib/utils.pm
reported in autoinst-log.txt
===
"my" variable $files masks earlier declaration in same scope at /var/lib/openqa/share/tests/fedora/lib/utils.pm line 352.
"my" variable $beta masks earlier declaration in same scope at /var/lib/openqa/share/tests/fedora/lib/utils.pm line 995.
===

Signed-off-by: Michel Normand <normand@linux.vnet.ibm.com>
2019-09-20 07:15:21 +02:00
Adam Williamson
dc8ed1c9ba Yet more pre-release check fixes
We need another needle for French installs, and we need to use
ISO_URL for checking the ISO file name if ISO isn't set.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-09-11 14:31:54 -07:00
Adam Williamson
243dfbdacd desktop_vt: hackaround 'tty?' problem on F31
On F31 update tests, desktop_vt is broken because the 'ps -C'
command is showing 'tty?' as the tty on which Xwayland/Xorg is
running. Let's try using loginctl as a workaround for this.
This ordering of commands should ensure the ps -C output takes
precedence when it's correct.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-09-03 16:19:08 -07:00
Adam Williamson
567b2f1a3f Wait longer for console to appear after quitting Firefox
A couple of tests failed in F31/Rawhide lately because quitting
Firefox (running on a bare X server) took more than 10 seconds
to get back to a console...not sure why, but it's not something
we ought to waste a lot of time on, let's just bump the timeout.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-08-28 11:07:13 -07:00
Adam Williamson
b9934c9816 Fix prerelease check for Branched update tests
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-08-27 08:36:40 -07:00
Adam Williamson
bf5691e69d Fix and simplify fedora_release test
Also simplify os_release a bit thanks to the improved spell_
version_number.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-08-21 13:09:40 -07:00
Adam Williamson
efbfe95213 Fix wrong repos_mirrorlist call which broke all update tests
Ugh. Sorry. Teach me to fire off changes without testing them.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-08-21 11:29:30 -07:00
Adam Williamson
86488d2611 Expect Workstation branding on Silverblue installer
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-08-21 00:01:47 -07:00
Adam Williamson
397dc9a9cf Don't redefine $prerelease
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-08-20 23:42:01 -07:00
Adam Williamson
a177a63ec1 Factor out s/metalink/mirrorlist/ into a util function
We do this in quite a few places and need to do it in another,
so let's just have a function for it. It takes a file glob
so we can have it run on a different one for _live_build.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-08-20 14:16:23 -07:00
Adam Williamson
150812c15c short-circuit copy_devcdrom_as_isofile if no ISO_URL
There are some situations in which ISO_URL is not set (e.g. a
manual POST to re-run tests after download has been done). If
there is no ISO_URL just skip the checksum check here.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-08-15 19:05:59 -07:00
Adam Williamson
04b716329a sigh, can't use get_release_number before defining it...
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-08-15 12:00:26 -07:00
Adam Williamson
d7d3d40d83 Work around lorax breakage in F29
A broken lorax got into F29 stable and is causing live image
builds to fail in all F29 update tests. Work around that by
downloading a newer fixed package in _repo_setup_updates.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-08-15 11:52:12 -07:00
Adam Williamson
9dd5465f83 Go back to git master for modularity test script
As my fixes got merged. Yay.
This reverts commit 3d3bc1e2d2.
2019-08-12 18:18:52 -07:00
Adam Williamson
3d3bc1e2d2 Use my fork of modularity test script for now
As upstream master doesn't run.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-08-07 17:50:09 -07:00
Adam Williamson
d9816b0ddb Current modularity test script requires python3-yaml
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-08-07 17:49:12 -07:00
Lukas Ruzicka
d06b30e8dc Add new testcases and methods for release identification. 2019-07-30 15:45:02 -07:00
Adam Williamson
85c74f214e Use lower similarity level for type_safely wait_still_screen
Just like all the others, to handle flashing cursors.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-07-25 14:57:00 -07:00
Adam Williamson
0206324115 start_cockpit: wait a bit longer for login screen
Current tests sometimes seem to fail because X takes quite a
while to come up for some reason. I can't figure out why, and
this isn't properly a failure of the test (starting the X server
is an implementation detail), so let's just wait a bit longer
for now.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-07-25 14:55:07 -07:00
Adam Williamson
2a8acb6acb start_cockpit: lower similarity level for wait_still_screen
Flashing cursor, again.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-07-25 14:28:17 -07:00
Adam Williamson
be12f07900 GNOME apps: dismiss new LibreOffice 'tip of the day' if it shows
Also add a new needle for Impress startup.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-07-23 20:15:24 -07:00
Adam Williamson
30102df930 Fix download_modularity_tests arg evaluation
Evaluating an array in scalar context gives the length of the
array, not its first value...

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-07-17 10:40:03 -07:00
Adam Williamson
25f408ea7e Factor out clicking of update and akonadi notifications in KDE
There are three places where we basically want to click away
pop-up update notifications and the buggy akonadi_migration_agent
notification if it's there, in KDE tests. Let's share this code
between them, and also let's record soft failures for the buggy
cases in the desktop_notifications test.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-07-17 09:02:24 -07:00
Lukas Ruzicka
1475813b75 Update the modularity tests.
* Add test to check module defaults.
* Add whitelist download.
* Fix install test to include selected profile to be on the safe side.
* Add test into templates.
2019-07-15 13:19:06 +02:00
Adam Williamson
670c2d4c9d type_very_safely: use looser still screen similarity
type_very_safely commonly gets stuck waiting on a still screen
at the end because when you're typing, there's often a flashing
cursor, and the default similarity for wait_still_screen is 47
which is tight enough that a flashing cursor usually fails it.
So back it off to 45.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-07-05 15:58:22 -07:00
Adam Williamson
e1cad438ce Remove the Bodhi workaround again now the update is stable
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-06-27 08:39:26 -07:00
Adam Williamson
6656f1659d apps_activities is a dupe with overview_app_grid, kill it
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-06-25 14:01:39 -07:00
Adam Williamson
b2848c5040 Fix fucking missing fucking -y fuck it I just want fucking dinner
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-06-17 19:29:41 -07:00
Adam Williamson
57938e1663 Add a workaround for a Bodhi bug that's causing problems
We're not downloading all the packages from updates that contain
more than one builds ATM, which makes the test invalid and has
caused some false fails (and may even have caused false passes,
though I can't tell yet). Install a fixed Bodhi as a workaround
for this.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-06-17 19:12:01 -07:00
Adam Williamson
3c8316241e Out, damned tab, out!
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-05-28 16:45:41 -07:00
Adam Williamson
43b990195c Add a wrapper around assert_and_click to handle signature change
So, turns out new os-autoinst does *not* still accept the old
argument style for assert_and_click...and old os-autoinst
doesn't accept the new one. This adds a wrapper that handles
both, so our tests can work with old and new os-autoinst. We can
drop this once both deployments are on newer os-autoinst.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-05-28 09:41:33 -07:00
Adam Williamson
87456f6b1d Use new arg style for assert_and_click
In the new os-autoinst I just sent to staging, the old style
doesn't work any more, breaks all tests. This style should also
work with the older os-autoinst on stable.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-05-27 14:28:17 -07:00
Adam Williamson
a8bcc191bb Drop FreeIPA upgrade workaround as the update is stable now
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-05-24 16:43:13 -07:00
Adam Williamson
e1ffd8aabd Fix text install test for 'use password' now being default
'use password' now seems to be default when creating a user in
text mode, so we need to handle that.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-05-22 15:18:08 -07:00
Adam Williamson
333757eadc Pull in fix for BIND bug in F30 upgrade tests
A BIND update broke a test, then went stable, so it's failing on
all updates right now. This subsequent update fixes the bug, so
let's pull it in here until it's pushed stable.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-05-09 09:31:47 +07:00
Adam Williamson
68cb3eb464 Remove branched buildroot repo workaround
For now, anyway...thinking of bringing this back in a more
generic form later.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-05-01 08:33:02 -07:00
Adam Williamson
7d4db97aa3 remove #1698200 workaround (no longer needed)
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-05-01 08:29:57 -07:00
Adam Williamson
6c974ce2ac Remove workarounds for #1674045 (upgrade hang at end)
It's fixed now.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-04-29 09:49:03 -07:00
Michel Normand
37e2f134f1 Bypass dup chars after snapshot brc#1691487
eg failing jobs:
https://openqa.stg.fedoraproject.org/tests/499908#step/_boot_to_anaconda/7
https://openqa.stg.fedoraproject.org/tests/519326#step/_post_network_static/4

Signed-off-by: Michel Normand <normand@linux.vnet.ibm.com>
2019-04-15 16:20:13 +02:00
Adam Williamson
d33aa591a6 Update the workaround for #1698200
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-04-12 19:01:09 -07:00
Guy Menanteau
29070a54f1 New install_repository_hd_variation
use a new PREINSTALL to execute a rescue script before install.

This is covering the  Hard_drive_variation test as per:
https://fedoraproject.org/wiki/QA:Testcase_install_repository_Hard_drive_variation

Signed-off-by: Guy Menanteau <menantea@linux.vnet.ibm.com>
2019-04-12 15:49:54 -07:00
Michel Normand
4a00301725 new select_rescue_mode in lib/utils from rescue_mode_encrypted.pm
placed in lib/utils.pm to be used by another commit.

Signed-off-by: Michel Normand <normand@linux.vnet.ibm.com>
2019-04-12 15:49:54 -07:00
Adam Williamson
8688b91fb2 F30 updates tests: pull in fix for #1698200
Otherwise base_services_start test always fails.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-04-11 09:52:53 -07:00
Adam Williamson
e87348abca Be more strict in console_login
Because we use check_screen not assert_screen here, the match
can actually fail, but the match_has_tag conditional can pass
on the *previous* match, if that happened to also be matching
on a console tag. We don't want that. Let's just make these
into assert_screens to avoid it; I don't think there's any path
where we're actually expecting this to work if those
check_screens didn't match.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-04-10 10:44:15 -07:00
Lukas Ruzicka
a412f37a50 Use keyboard to navigate in menu. 2019-03-19 12:15:03 -07:00
Lukas Ruzicka
c9a25cb2a6 Create an application start-stop test for KDE.
Add png files as a background for the needles.

Rewrite the test handling methods to make them ready for KDE

Add the ABRT startstop test.

Make changes and corrections to the abrt test.

Add test for Firewall.

Add test for DNF dragora.

Add wait time for dnfdragora test.

Correct syntax.

Add Language test.

Make some changes to the DNF dragora test.

Add Users test case.

Add needles for DBUS viewer test.

Add Dbusviewer test.

Add Mahjong test and needles for games.

Add Minesweeper tests.

Add Patience test.

Add test for Document Viewer.

add test for gwenview

Add test for koulourpaint.

Add test for Kruler

Add test for Kcolorchooser

Reneedle failing needle.

Add ktorrent tests.

Add tests for CPT editors.

Add test for Krfb

Fix names for those files.

Add test for Kget

Add Akregator test.

Add test for Konversation.

Make Konversation end really.

Add tests for Kmail

Add test for PIM exporter.

Add test for KTnef and Krdc.

Fix problems after test runs.

Make more tries.

Fix needle to be found better.

Fix more errors.

Add test for Falkon.

Add tests for browsers.

Add support for closing tabs into Firefox test.

Add tests for K3B

move needles to correct directory

Add Kaddressbook test.

Add Kontakt text.

Add test for korganizer.

Add menu office needle and correct konqueror needle.

Add test for calligra stage.

Add test for Calligra.

Add test for network connections.

Modify needle for kaddressbook to prevent failing.

Add test for system settings and fix others.

Add test for FMW.

Add test for Dolphin

Add test for Infocenter.

Add test for kparted.

Fix a wrong needle.

Test relnotes.

Fix some errors in tests.

Add test for Discover.

Add test for Ksysguard.

Add tests for Konsole.

Add tests for KDE wallet.

Add tests for several utilities.

Add Krusader test.

Finish utility tests.

Fix some errors.

Fix needle for spectacle.

Add wait time to let Dragora wait for network.
2019-03-19 12:14:47 -07:00