Commit Graph

407 Commits

Author SHA1 Message Date
Adam Williamson fd26d347fe Workaround unexpected authentication required dialog in g-i-s
https://gitlab.gnome.org/GNOME/gnome-initial-setup/-/issues/106
This started happening after the data center move. Really not
sure why.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-06-12 12:28:18 -07:00
Adam Williamson 1b37200446 Work around GNOME Shell app submenu bug
See https://gitlab.gnome.org/GNOME/gnome-shell/-/issues/2878 .
GNOME 3.37.2 seems to have a bug with submenus in the app menu;
the first time you open one you can't scroll through it using
the keyboard. On every open after the first it works fine. This
is a quick and dirty workaround - when we're dealing with a
submenu, open it then close it then open it again.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-06-11 16:00:56 -07:00
Adam Williamson 2e21cfac15 Update cockpit needles and select_cockpit_update for 220
In Cockpit 220, the Updates entry is off the bottom of the screen
so we need to scroll the left bar down before we can click it.
Also update some other needles.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-05-30 10:27:20 -07:00
Lukáš Růžička 4d4b2d7739 Finish the post review changes. 2020-05-28 13:32:42 -07:00
Lukáš Růžička b5169e602b Use existing needles for fs selection. 2020-05-27 15:54:10 -07:00
Lukáš Růžička 9efb0e10e6 Create a test to resize LVM partitions using Blivet. 2020-05-27 15:54:09 -07:00
Adam Williamson b59f875af1 Drop F32 workaround updates (both now stable)
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-05-27 13:15:37 -07:00
Adam Williamson acc46464f8 Revise release variable handling, prerelease checks, os-release
I started out trying to fix os-release for the recent change to
add "Prerelease" tags to the VERSION and PRETTY_NAME fields, then
things spiralled. It got me thinking about the awkward DEVELOPMENT
variable we use, so I decided to get rid of it and refactor the
few things that use it. I refactored the anaconda prerelease tag
check, and wrote a new giant comment that gives details about
exactly how anaconda decides whether to show those tags, to give
context to our choices about when to expect them. This check now
uses a new LABEL variable the scheduler now sets. I also wound up
creating new UP1REL and UP2REL vars to define the 'source' release
for upgrade tests, separate from CURRREL and PREVREL, which are
now never lies - they really are the current stable and previous
stable release, even for update upgrade tests.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-05-07 15:42:08 -07:00
Adam Williamson 8eaddd8c3d Set OfferToSaveLogins policy to false in Firefox config
This seems to work around the Firefox 76 bug with password entry
sometimes breaking:
https://bugzilla.mozilla.org/show_bug.cgi?id=1635833

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-05-06 13:25:46 -07:00
Adam Williamson e67d3e7c01 Drop similarity level on Cockpit login screen wait_still_screen
We have a flashing cursor, folks.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-05-06 12:58:59 -07:00
Adam Williamson d66b185471 solidify_wallpaper: restore a wait_still_screen in GNOME path
The old code waited after launching the terminal, the new code
doesn't, which led to a 'g' being swallowed in the first command
in https://openqa.fedoraproject.org/tests/592759 .

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-05-05 10:19:31 -07:00
Adam Williamson 8866b5289e Fix solidify_wallpaper for KDE
The desktop check used the wrong case.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-05-05 10:08:47 -07:00
Lukáš Růžička 61f845dbbf Make solidify_wallpaper more universal to handle different desktops. 2020-05-05 00:14:35 +00:00
Adam Williamson ea9ac508ac Fix check_desktop variable timeouts
I forgot that tries was configurable. Sigh. Convert it to a
timeout argument.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-04-18 14:54:48 -07:00
Lukáš Růžička f3d6a9574c Add desktop login test, revise and rename check_desktop
This adds a new test that implementsQA:Testcase_desktop_login
on both GNOME and KDE.

While working on this, we realized that the "desktop_clean"
needles were really "app menu" needles, and for KDE, this was
a duplication with the new "system menu" needles, because on KDE
the app menu and the system menu are the same. So I (Adam)
started to de-duplicate that, but also realized that "app menu
button" is a much more accurate name for these needles, so I was
renaming the old desktop_clean needles to app_menu_button. That
led me to the realization that "check_desktop_clean" is itself a
dumb name, because we don't (at least, any more, way back in the
mists of time we may have done) do anything to check that the
desktop is "clean" - we're really just asserting that we're at a
desktop *at all*. While thinking *that* through, I *also* realized
that the whole "open the overview and look for the app grid icon"
workaround it did is no longer necessary, because GNOME doesn't
use a translucent top bar any more. That went away in GNOME 3.32,
which is in Fedora 30, our oldest supported release.

So I threw that away, renamed the function "check_desktop",
cleaned up all the needle naming and tagging, and also added an
app menu needle for GNOME in Japanese because we were missing
one (the Japanese tests have been using the "app grid icon"
workaround the whole time).
2020-04-17 17:27:04 -07:00
Michel Normand cc5c85a3e4 Increase timeout for getting_started ppc64le f32
Signed-off-by: Michel Normand <normand@linux.vnet.ibm.com>
2020-04-03 19:38:04 +02:00
Adam Williamson dc7b7a7241 Great Needle Cleanup 2020
Remove a bunch of needles that have not been used for some time,
plus a few workarounds that are similarly stale.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-03-20 14:02:10 -07:00
Lukáš Růžička d19cf9553e Reneedle the Gnome Application StartStop tests. 2020-03-18 18:06:34 +01:00
Adam Williamson e8ea8a7fe5 OK adam remember how we make an empty array? you got this buddy
I believe in you! We ALL believe in you!

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-03-17 08:33:05 -07:00
Adam Williamson 8832c5b81e Add FEDORA-2020-5f05c3ec46 as a workaround to fix FreeIPA upgrade
FreeIPA F31 -> F32 upgrade test is currently failing because
a new pki-core hit F31 stable but not F32 stable yet. It can't go
backwards on upgrade, that breaks stuff. The F32 update has been
pushed stable but just hasn't made mirrors yet as the last F32
nightly compose failed, so let's add it to the workarounds for
now.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-03-17 08:18:11 -07:00
Adam Williamson d9438e8042 Drop workaround updates that have been pushed stable
Except one that's pushed stable but hasn't made repos yet (as the
last F32 nightly compose failed).

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-03-17 08:17:00 -07:00
Adam Williamson 5fdf0041cf Bump another blivet timeout
Seems 5 seconds isn't long enough to wait here on aarch64, the
previous dialog hasn't always cleared by then. See e.g.
https://openqa.stg.fedoraproject.org/tests/753802

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-03-11 17:43:23 -07:00
Adam Williamson 3444d2f959 Try and fix blivet test fails caused by too-early OK click
We seem to quite often get a failure in the blivet_lvmthin test
here which seems to be caused by trying to click 'OK' while the
'Device type' menu is still changing state or something. Let's
throw in a little delay.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-03-11 17:22:26 -07:00
Adam Williamson 5c6ee14ed3 Add F32 backgrounds update as an update test workaround
So at least GNOME background test passes.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-03-06 21:51:32 -08:00
Adam Williamson e7c6501737 Extend update test workaround mechanism to upgrade tests
To cover cases like #1767351 where we need to apply a workaround
to the pre-upgrade environment.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-03-06 19:21:44 -08:00
Adam Williamson 9df8fdb00e Drop workaround package that's gone stable
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-03-06 12:54:16 -08:00
Adam Williamson 884f71bb81 Revert "Workaround #1805553 (single disk not selected as install target)"
This reverts commit adf3f91818.
The bug has been fixed in anaconda, so we can drop this, which
is good as it has timing issues producing false positives on
Rawhide...
2020-02-26 14:30:08 -08:00
Adam Williamson 3d29d7b746 Add efivar update for F32 as workaround to fix UEFI update tests
This is pending stable, but looks like the update push won't
happen for a few hours, so I'm adding it as a workaround so we
can re-run the tests and get them to pass.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-02-26 12:48:17 -08:00
Adam Williamson 8e43dfa304 Drop F31 workaround that's gone stable, add F32 lmod workaround
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-02-25 17:40:22 -08:00
Adam Williamson adf3f91818 Workaround #1805553 (single disk not selected as install target)
Rawhide seems to have developed a bug where a single disk is no
longer automatically selected as the install target when you
enter the INSTALLATION DESTINATION spoke. We need to work around
this (but register it as a soft failure).

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-02-20 17:41:34 -08:00
Adam Williamson 1eb4e3dca5 Add perl syntax check test, add it to CI
Inspired by openQA's 01-compile-check-all.t, this adds a perl
test which checks the syntax of main.pm and all lib and test
files, and hooks it up to CI. Requires os-autoinst and
perl-Test-Strict.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-02-13 15:28:09 -08:00
Adam Williamson 66e53feb4d Update workarounds (drop update now stable, add F31 libdnf fix)
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-02-05 02:51:37 -08:00
Adam Williamson 0a4a787c03 Give Firefox longer to start
It seems to take a long time sometimes for some reason. Can't
pin it down but it's causing test flakes, so let's just let it
be. It *may* happen when chrony adjusts the system clock just as
Firefox is starting, for some reason.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-01-22 10:40:46 +01:00
Adam Williamson d236c48152 Doh, fix stray quote mark from last commit
Where did I leave my brown paper bag...

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-01-16 18:57:17 +00:00
Adam Williamson 1098450fd6 Pull in F30 Cockpit 210 update as a workaround
Otherwise the new cockpit main screen needle match fails, because
'machine' is not on the top part of the screen.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-01-16 18:51:40 +00:00
Adam Williamson ba40ba3e4a Drop #1663050 workaround (fixed a while ago)
This block is kinda weird, but I don't want to fiddle with it
any more right now. I don't know why we do this check_screen
exactly like this.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-01-10 14:41:41 -08:00
Adam Williamson 0714143673 Drop a stray old comment and move a debug block up
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-01-10 14:34:58 -08:00
Adam Williamson fc93309b41 Drop _assert_and_click shim
We're on newer code on both deployments now, no longer needed.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-01-10 14:22:20 -08:00
Adam Williamson dba4e8c57d Drop old version conditionals
We're well past 27 now...

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-01-10 14:20:44 -08:00
Adam Williamson fcb06a874c Drop no-longer-needed update workarounds
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-01-10 14:18:54 -08:00
Adam Williamson 0926be0520 Drop similarity level on a flashing cursor wait_still_screen
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-01-08 15:31:32 -08:00
Adam Williamson 478a55f057 Drop one more mention of wait_idle
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-01-03 14:20:18 -08:00
Adam Williamson 34dc2751a5 Replace remaining wait_idles with sleeps
wait_idle is finally removed upstream in recent git os-autoinst.
This replaces all remaining wait_idles with sleeps, except for
one which is removed (I'm hoping improvements to typing in the
last few years should mean it isn't necessary any more, if it
turns out to be, I'll put it back as a sleep).

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-01-02 17:44:45 -08:00
Adam Williamson 501b4bccc8 Fix 'clear' after login in French
...because it comes out as 'cleqr'. Note, this may be fragile if
we start doing more stuff post-install, but for now I think it's
safe, I don't *think* we should ever hit this after running
`loadkeys us`.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-12-14 09:17:34 -08:00
Adam Williamson de2c44e7ae Work around Lmod bug in KDE update packages, with new mechanism
It seems like the situation where we need to pull an update from
updates-testing into all update tests to work around some known
issue is going to keep happening. So instead of constantly
adding and then entirely removing bespoke lines for each specific
workaround, let's have a permanent mechanism for doing this: a
hash with release numbers as keys, and arrayrefs of update IDs
as values, and a block to call `bodhi updates download` on the
appropriate array for the release under test. This way, to add
or remove a workaround you just update the hash. If we're at a
point where *no* workarounds are needed the %workarounds hash
can be made entirely empty (it must exist, though) and the code
will be a clean no-op.

The actual workaround here pulls in Lmod updates I just sent out
to work around this issue in one of the KDE update tests:

https://openqa.fedoraproject.org/tests/497160#step/base_update_cli/11

there's some code in Lmod that gets sourced in bash profiles
which breaks openQA's `validate_script_output` by blurping two
lines of informational output into the output of the script.
The update backports a change from upstream Lmod master that
sends that informational output to stderr instead of stdout.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-12-11 12:02:21 -08:00
Adam Williamson c9ae9c4d67 Drop SELinux and NSS update workarounds (went stable)
All these updates have gone stable now, so we no longer need
these workarounds.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-12-10 18:12:13 -08:00
Adam Williamson 3511a5b57b Replace jss workaround with selinux and nss workarounds
The jss updates all went stable already. Now we have a problem
with SELinux, upower and container-selinux (we need a newer
selinux-policy to avoid upower failures in the services_start
test, but the first attempt to fix it caused the desktop_updates
test to start failing because container-selinux needed adapting
to changes in selinux-policy...let's just pull in the updates
with the latest versions of both to be safe), and one with NSS
that causes Firefox to give false certificate errors sometimes
(this is particularly affecting the FreeIPA browser test). As
usual these should be dropped once the updates go stable.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-12-06 11:51:07 -08:00
Adam Williamson e8802935f9 Tweak text_console_login match area
On Rawhide Cloud_Base boots, there's some SSH key and network
information printed above the 'login:' prompt, so we can't
expect empty space there. Also tweak console_login() to clear
the screen after logging in, so the login prompt is cleared and
doesn't confuse things on subsequent runs (like it did first
time we tried this). And add a new user logged in needle, as it
seems after we clear the screen the tilde appears in a slightly
different position and the existing needle doesn't match.

https://openqa.fedoraproject.org/tests/489003#step/_console_wait_login/7

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-11-27 10:39:29 -08:00
Adam Williamson defab3bfcb Correct renaming of the virtio serial consoles
That second string didn't work how I thought it did.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-11-20 09:15:58 -08:00
Adam Williamson 48b6c9d3e9 Change name we use for virtio serial consoles
There is nothing inherently 'root'-y about these so it makes no
sense to prefix their names with 'root-'. And why change from
'console' to 'terminal' compared to the naming used in the
actual qemu command and the log files? It's just confusing.
Let's be consistent (except for using - instead of _ here...
but - is easier to type!)

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-11-20 08:39:28 -08:00
Lukáš Růžička b93a197c22 Enable Anaconda Text install via serial console.
This adds the Anaconda text installation test over
serial console and FIXES #115.
2019-11-19 22:54:55 -08:00
Adam Williamson 9bec7ee797 Bump timeout on exiting Firefox again...
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-11-07 07:35:32 -08:00
Adam Williamson b82a1dec6f Add test for QA:Testcase_Anaconda_User_Interface_VNC
This adds a test for QA:Testcase_Anaconda_User_Interface_VNC -
the VNC install test case. It's implemented as a server/client
pair, with the server booting from the Server DVD image with
`inst.vnc` and the client booting from the desktop base disk
image, setting up networking, then running Boxes to connect to
the server and run the install.

There are various little tweaks to test loading and logic to
handle this, mostly pretty clear. We also move the workaround
for 'spurious auth prompt appears on desktop after you switch
away to a VT and back' out of the desktop update test and into
the `desktop_vt` helper function, since now this test can hit
it as well. We enhance _graphical_wait_login to handle the boot
loader if needed (it has never needed to until now).

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-11-05 14:46:10 -08:00
Adam Williamson efec7010cb Handle an 'akonadi did something' notification in KDE, etc.
This handles a case where KDE shows a notification saying
'PIM Maintenance (Finished)', like this:

https://openqa.fedoraproject.org/tests/477345#step/desktop_notifications/34

we need to click it away for the desktop_notification test to
pass. It also clarifies the difference between this notification
and the eternal 'akonadi_migration_agent is doing something'
popup in the needle names and comments. It also replaces the
'check_screen then assert_and_click if found' pattern in several
notifications-related places with the better 'check_screen then
click_lastmatch if found' pattern now available upstream.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-10-30 09:05:06 -07:00
Adam Williamson 8c07149ec7 Workaround #1766451 in update tests
Download the fixed jss to the advisory repo.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-10-29 18:03:59 -07:00
Adam Williamson 764a2bdc33 Drop #1757948 workaround, shouldn't be needed any more
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-10-29 17:59:33 -07:00
Adam Williamson 33e3224411 Drop stray `start_cockpit` export from utils.pm
It was moved to cockpit.pm, but this wasn't removed.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-10-29 15:12:04 -07:00
Lukas Ruzicka 22b185abf6 Enable tracking of executed applications in Application Start/Stop tests (FIXES #116)
This PR adds the following:
* moves out the presetting procedures, so that two long terminal tests do not have
to run twice
* add methods for application to register when successfully started
* adds a test that checks if all required applications have registered
2019-10-16 17:36:37 -07:00
Adam Williamson 079b969fab Workaround RHBZ#1757948 in F30 update tests
We need to pull in the fixed fwupd or else the desktop_update
test always fails.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-10-15 19:23:55 -07:00
Lukas Ruzicka d82f4ecd7f Develop a new test for system updates via Cockpit.
* server-cockpit-updates tests that Cockpit can be used to update the system.
* server-cockpit-autoupdate tests that users can use dnf-automatic for system
updates.
* cockpit functions were removed from utils.pm and put into an extra library
for cockpit - cockpit.pm which all cockpit tests are now using.

Review cockpit.pm

Review autoupdate test.

Review the update test.

Fix typo in cockpit.pm

Add sleep.

Add missing command.

Delete an unused needle.
2019-10-01 16:04:01 +02:00
Adam Williamson b32449f72a Add missing bracket
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-09-20 11:04:31 -07:00
Adam Williamson 6538a82096 Download by ADVISORY_NVRS with koji for updates, not ADVISORY
This goes with a corresponding fedora_openqa commit that passes
ADVISORY_NVRS when scheduling update jobs. This is to address
https://pagure.io/fedora-qa/fedora_openqa/issue/78 so we can
publish spec-compliant and correct CI Messages: we will decide
what NVRs are in the update at scheduling time and always get
and test those NVRs, rather than the test downloading whatever
is in the update when it is run.

Some consequences: manual restarting or cloning of an update
test scheduled before this change lands will result in failure;
to do this you'll have to add the ADVISORY_NVRS value manually
with clone_job.pl, or simply reschedule the update with
fedora-openqa.

Manual restarting or cloning of an update test scheduled with
this change will always test the same NVRs. If the update has
changed and you want to re-test with the new packages in the
update, you must change ADVISORY_NVRS manually with clone_job,
or reschedule with fedora-openqa.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-09-20 10:58:27 -07:00
Adam Williamson 7f96c3aa49 Remove a now-unneeded workaround
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-09-20 10:51:16 -07:00
Michel Normand 6455319ef5 typo in lib/utils.pm
reported in autoinst-log.txt
===
"my" variable $files masks earlier declaration in same scope at /var/lib/openqa/share/tests/fedora/lib/utils.pm line 352.
"my" variable $beta masks earlier declaration in same scope at /var/lib/openqa/share/tests/fedora/lib/utils.pm line 995.
===

Signed-off-by: Michel Normand <normand@linux.vnet.ibm.com>
2019-09-20 07:15:21 +02:00
Adam Williamson dc8ed1c9ba Yet more pre-release check fixes
We need another needle for French installs, and we need to use
ISO_URL for checking the ISO file name if ISO isn't set.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-09-11 14:31:54 -07:00
Adam Williamson 243dfbdacd desktop_vt: hackaround 'tty?' problem on F31
On F31 update tests, desktop_vt is broken because the 'ps -C'
command is showing 'tty?' as the tty on which Xwayland/Xorg is
running. Let's try using loginctl as a workaround for this.
This ordering of commands should ensure the ps -C output takes
precedence when it's correct.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-09-03 16:19:08 -07:00
Adam Williamson 567b2f1a3f Wait longer for console to appear after quitting Firefox
A couple of tests failed in F31/Rawhide lately because quitting
Firefox (running on a bare X server) took more than 10 seconds
to get back to a console...not sure why, but it's not something
we ought to waste a lot of time on, let's just bump the timeout.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-08-28 11:07:13 -07:00
Adam Williamson b9934c9816 Fix prerelease check for Branched update tests
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-08-27 08:36:40 -07:00
Adam Williamson bf5691e69d Fix and simplify fedora_release test
Also simplify os_release a bit thanks to the improved spell_
version_number.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-08-21 13:09:40 -07:00
Adam Williamson efbfe95213 Fix wrong repos_mirrorlist call which broke all update tests
Ugh. Sorry. Teach me to fire off changes without testing them.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-08-21 11:29:30 -07:00
Adam Williamson 86488d2611 Expect Workstation branding on Silverblue installer
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-08-21 00:01:47 -07:00
Adam Williamson 397dc9a9cf Don't redefine $prerelease
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-08-20 23:42:01 -07:00
Adam Williamson a177a63ec1 Factor out s/metalink/mirrorlist/ into a util function
We do this in quite a few places and need to do it in another,
so let's just have a function for it. It takes a file glob
so we can have it run on a different one for _live_build.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-08-20 14:16:23 -07:00
Adam Williamson 81387cc840 Revert "installedtest: try and fix log uploads for ostree installs"
This reverts commit 092d7dd9c3 and
the follow-up, they turn out not to be necessary or useful...
2019-08-16 11:29:53 -07:00
Adam Williamson fb439767f5 Fix ostree root
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-08-16 11:00:15 -07:00
Adam Williamson 092d7dd9c3 installedtest: try and fix log uploads for ostree installs
The log files are all under the ostree deploy root, the 'real'
system root has nothing useful. Try and find the deploy root
and prepend it to all the relevant commands if we're a CANNED
install.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-08-16 09:05:26 -07:00
Adam Williamson 150812c15c short-circuit copy_devcdrom_as_isofile if no ISO_URL
There are some situations in which ISO_URL is not set (e.g. a
manual POST to re-run tests after download has been done). If
there is no ISO_URL just skip the checksum check here.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-08-15 19:05:59 -07:00
Adam Williamson 04b716329a sigh, can't use get_release_number before defining it...
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-08-15 12:00:26 -07:00
Adam Williamson d7d3d40d83 Work around lorax breakage in F29
A broken lorax got into F29 stable and is causing live image
builds to fail in all F29 update tests. Work around that by
downloading a newer fixed package in _repo_setup_updates.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-08-15 11:52:12 -07:00
Adam Williamson 9dd5465f83 Go back to git master for modularity test script
As my fixes got merged. Yay.
This reverts commit 3d3bc1e2d2.
2019-08-12 18:18:52 -07:00
Adam Williamson 4dae3d2168 Use some nmcli con down/ups to workaround RHBZ #1739148
Restarting NetworkManager.service doesn't seem to apply our
newly-created static config in Rawhide any more (again!), since
NetworkManager 1.20.0-0.5.fc31 landed. This seems to work around
the bug.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-08-08 08:54:23 -07:00
Adam Williamson 3d3bc1e2d2 Use my fork of modularity test script for now
As upstream master doesn't run.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-08-07 17:50:09 -07:00
Adam Williamson d9816b0ddb Current modularity test script requires python3-yaml
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-08-07 17:49:12 -07:00
Lukas Ruzicka d06b30e8dc Add new testcases and methods for release identification. 2019-07-30 15:45:02 -07:00
Adam Williamson 85c74f214e Use lower similarity level for type_safely wait_still_screen
Just like all the others, to handle flashing cursors.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-07-25 14:57:00 -07:00
Adam Williamson 0206324115 start_cockpit: wait a bit longer for login screen
Current tests sometimes seem to fail because X takes quite a
while to come up for some reason. I can't figure out why, and
this isn't properly a failure of the test (starting the X server
is an implementation detail), so let's just wait a bit longer
for now.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-07-25 14:55:07 -07:00
Adam Williamson 2a8acb6acb start_cockpit: lower similarity level for wait_still_screen
Flashing cursor, again.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-07-25 14:28:17 -07:00
Adam Williamson be12f07900 GNOME apps: dismiss new LibreOffice 'tip of the day' if it shows
Also add a new needle for Impress startup.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-07-23 20:15:24 -07:00
Adam Williamson 30102df930 Fix download_modularity_tests arg evaluation
Evaluating an array in scalar context gives the length of the
array, not its first value...

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-07-17 10:40:03 -07:00
Adam Williamson 25f408ea7e Factor out clicking of update and akonadi notifications in KDE
There are three places where we basically want to click away
pop-up update notifications and the buggy akonadi_migration_agent
notification if it's there, in KDE tests. Let's share this code
between them, and also let's record soft failures for the buggy
cases in the desktop_notifications test.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-07-17 09:02:24 -07:00
Lukas Ruzicka 1475813b75 Update the modularity tests.
* Add test to check module defaults.
* Add whitelist download.
* Fix install test to include selected profile to be on the safe side.
* Add test into templates.
2019-07-15 13:19:06 +02:00
Adam Williamson d34b5a3567 Revert "Use some nmcli con down/ups to workaround RHBZ #1727411"
This reverts commit 7ec143382f.
The bug it works around was fixed, so we shouldn't need the work
around any more.
2019-07-09 17:40:01 -07:00
Adam Williamson 7ec143382f Use some nmcli con down/ups to workaround RHBZ #1727411
Restarting NetworkManager.service doesn't seem to apply our
newly-created static config in Rawhide any more, since
NetworkManager 1.20.0-0.3.fc31 landed. This seems to work around
the bug.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-07-05 16:40:43 -07:00
Adam Williamson 670c2d4c9d type_very_safely: use looser still screen similarity
type_very_safely commonly gets stuck waiting on a still screen
at the end because when you're typing, there's often a flashing
cursor, and the default similarity for wait_still_screen is 47
which is tight enough that a flashing cursor usually fails it.
So back it off to 45.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-07-05 15:58:22 -07:00
Adam Williamson e1cad438ce Remove the Bodhi workaround again now the update is stable
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-06-27 08:39:26 -07:00
Adam Williamson 6656f1659d apps_activities is a dupe with overview_app_grid, kill it
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-06-25 14:01:39 -07:00
Adam Williamson b2848c5040 Fix fucking missing fucking -y fuck it I just want fucking dinner
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-06-17 19:29:41 -07:00
Adam Williamson 57938e1663 Add a workaround for a Bodhi bug that's causing problems
We're not downloading all the packages from updates that contain
more than one builds ATM, which makes the test invalid and has
caused some false fails (and may even have caused false passes,
though I can't tell yet). Install a fixed Bodhi as a workaround
for this.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-06-17 19:12:01 -07:00
Adam Williamson 3c8316241e Out, damned tab, out!
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-05-28 16:45:41 -07:00
Adam Williamson 43b990195c Add a wrapper around assert_and_click to handle signature change
So, turns out new os-autoinst does *not* still accept the old
argument style for assert_and_click...and old os-autoinst
doesn't accept the new one. This adds a wrapper that handles
both, so our tests can work with old and new os-autoinst. We can
drop this once both deployments are on newer os-autoinst.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-05-28 09:41:33 -07:00
Adam Williamson 87456f6b1d Use new arg style for assert_and_click
In the new os-autoinst I just sent to staging, the old style
doesn't work any more, breaks all tests. This style should also
work with the older os-autoinst on stable.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-05-27 14:28:17 -07:00
Adam Williamson a8bcc191bb Drop FreeIPA upgrade workaround as the update is stable now
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-05-24 16:43:13 -07:00
Adam Williamson e1ffd8aabd Fix text install test for 'use password' now being default
'use password' now seems to be default when creating a user in
text mode, so we need to handle that.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-05-22 15:18:08 -07:00
Adam Williamson 333757eadc Pull in fix for BIND bug in F30 upgrade tests
A BIND update broke a test, then went stable, so it's failing on
all updates right now. This subsequent update fixes the bug, so
let's pull it in here until it's pushed stable.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-05-09 09:31:47 +07:00
Adam Williamson 0d38d3f292 FreeIPA: get verbose logs from BIND
This was necessary for debugging the FreeIPA 4.8 pre-release
update bug, so let's have it for all runs, just in case.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-05-01 18:59:52 -07:00
Adam Williamson 68cb3eb464 Remove branched buildroot repo workaround
For now, anyway...thinking of bringing this back in a more
generic form later.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-05-01 08:33:02 -07:00
Adam Williamson 7d4db97aa3 remove #1698200 workaround (no longer needed)
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-05-01 08:29:57 -07:00
Adam Williamson 6c974ce2ac Remove workarounds for #1674045 (upgrade hang at end)
It's fixed now.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-04-29 09:49:03 -07:00
Michel Normand 37e2f134f1 Bypass dup chars after snapshot brc#1691487
eg failing jobs:
https://openqa.stg.fedoraproject.org/tests/499908#step/_boot_to_anaconda/7
https://openqa.stg.fedoraproject.org/tests/519326#step/_post_network_static/4

Signed-off-by: Michel Normand <normand@linux.vnet.ibm.com>
2019-04-15 16:20:13 +02:00
Adam Williamson d33aa591a6 Update the workaround for #1698200
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-04-12 19:01:09 -07:00
Adam Williamson 2754deb28a Don't do live build postinstall steps on the install tests
It's not going to work there. Use the test name not the flavor
name to match on.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-04-12 18:54:36 -07:00
Guy Menanteau 29070a54f1 New install_repository_hd_variation
use a new PREINSTALL to execute a rescue script before install.

This is covering the  Hard_drive_variation test as per:
https://fedoraproject.org/wiki/QA:Testcase_install_repository_Hard_drive_variation

Signed-off-by: Guy Menanteau <menantea@linux.vnet.ibm.com>
2019-04-12 15:49:54 -07:00
Michel Normand 4a00301725 new select_rescue_mode in lib/utils from rescue_mode_encrypted.pm
placed in lib/utils.pm to be used by another commit.

Signed-off-by: Michel Normand <normand@linux.vnet.ibm.com>
2019-04-12 15:49:54 -07:00
Adam Williamson 8688b91fb2 F30 updates tests: pull in fix for #1698200
Otherwise base_services_start test always fails.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-04-11 09:52:53 -07:00
Adam Williamson e87348abca Be more strict in console_login
Because we use check_screen not assert_screen here, the match
can actually fail, but the match_has_tag conditional can pass
on the *previous* match, if that happened to also be matching
on a console tag. We don't want that. Let's just make these
into assert_screens to avoid it; I don't think there's any path
where we're actually expecting this to work if those
check_screens didn't match.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-04-10 10:44:15 -07:00
Lukas Ruzicka a412f37a50 Use keyboard to navigate in menu. 2019-03-19 12:15:03 -07:00
Lukas Ruzicka c9a25cb2a6 Create an application start-stop test for KDE.
Add png files as a background for the needles.

Rewrite the test handling methods to make them ready for KDE

Add the ABRT startstop test.

Make changes and corrections to the abrt test.

Add test for Firewall.

Add test for DNF dragora.

Add wait time for dnfdragora test.

Correct syntax.

Add Language test.

Make some changes to the DNF dragora test.

Add Users test case.

Add needles for DBUS viewer test.

Add Dbusviewer test.

Add Mahjong test and needles for games.

Add Minesweeper tests.

Add Patience test.

Add test for Document Viewer.

add test for gwenview

Add test for koulourpaint.

Add test for Kruler

Add test for Kcolorchooser

Reneedle failing needle.

Add ktorrent tests.

Add tests for CPT editors.

Add test for Krfb

Fix names for those files.

Add test for Kget

Add Akregator test.

Add test for Konversation.

Make Konversation end really.

Add tests for Kmail

Add test for PIM exporter.

Add test for KTnef and Krdc.

Fix problems after test runs.

Make more tries.

Fix needle to be found better.

Fix more errors.

Add test for Falkon.

Add tests for browsers.

Add support for closing tabs into Firefox test.

Add tests for K3B

move needles to correct directory

Add Kaddressbook test.

Add Kontakt text.

Add test for korganizer.

Add menu office needle and correct konqueror needle.

Add test for calligra stage.

Add test for Calligra.

Add test for network connections.

Modify needle for kaddressbook to prevent failing.

Add test for system settings and fix others.

Add test for FMW.

Add test for Dolphin

Add test for Infocenter.

Add test for kparted.

Fix a wrong needle.

Test relnotes.

Fix some errors in tests.

Add test for Discover.

Add test for Ksysguard.

Add tests for Konsole.

Add tests for KDE wallet.

Add tests for several utilities.

Add Krusader test.

Finish utility tests.

Fix some errors.

Fix needle for spectacle.

Add wait time to let Dragora wait for network.
2019-03-19 12:14:47 -07:00
Adam Williamson 067a728363 Give 'dnf -y update' a bit longer in update tests
For big updates, 10 minutes seems not to always be enough, viz
https://openqa.stg.fedoraproject.org/tests/494515

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-03-14 08:38:54 -07:00
Adam Williamson 375575cd07 Use a second disk for the update repo in image build tests
This tweaks the update live and installer image build tests to
store the update repo on a separate disk. This should hopefully
avoid the tests failing due to insufficient disk space when the
update is huge (e.g. KDE or GNOME megaupdates).

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-03-13 12:57:57 -07:00
Adam Williamson 6dde0ea0c9 Fix #1674045 workaround for bootloader case
The initial implementation here has a problem if we spot the
'successful' screen briefly, then the system reboots normally,
reaches the bootloader and proceeds past it all within 10
seconds; in this case we'll never actually spot the bootloader
and do our stuff. This tweak should continue through the code
block immediately if the bootloader shows up during the ten
seconds, otherwise check again for the 'successful' screen and
reboot.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-03-11 18:34:13 -07:00
Adam Williamson bb8d9d26d7 Workaround f30 compose/buildroot mismatch
The f30 tag is currently a lot fresher than what's in the repos,
because we haven't had a compose for five days. This causes
problems in some F30 update tests because the update works fine
with what's in the `f30` tag but not what's on the mirrors. To
deal with this until we get a successful compose, let's just
enable the f30-build repo in the update test process...

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-03-06 10:41:16 -08:00
Adam Williamson 0c63065ebd Add the #1674045 workaround to one more place
Sigh, so much copypasta makes me sad, should've factored it out.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-03-05 07:35:11 -08:00
Adam Williamson 86fe913b84 Soft fail workaround the 'upgrade hangs at end' bug (#1674045)
We want to get accurate results from the rest of the upgrade
test and we know about this bug, so let's make it a soft fail
so we can see how the test is functioning otherwise.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-03-04 18:07:04 -08:00
Adam Williamson 6f24fd452c Seems we need a 15 sec sleep for the Firefox cert workaround
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-02-26 12:05:55 -08:00
Adam Williamson b23dd7e14c Use correct name for file to disable Firefox shield studies
Dunno where I got distribution.ini from...

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-02-26 10:57:26 -08:00
Adam Williamson 855e54d8df Refresh workaround for occasional FreeIPA web UI cert issue
Investigation suggests that the Firefox certificate validation
failure we sometimes get when loading the FreeIPA web UI in the
freeipa_webui test module is a race issue in Firefox itself:

https://bugzilla.mozilla.org/show_bug.cgi?id=1530429

refreshing the page seems to work around the problem, so let's
do that.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-02-25 10:13:33 -08:00
Adam Williamson 945a93aa05 Get more info on failures caused by Firefox certificate issues
We're getting an intermittent case where FreeIPA tests fail
because of a web server certificate issue. Click 'Advanced' in
Firefox when this happens so we can get a bit more info on the
problem.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-02-21 12:16:42 -08:00
Adam Williamson 8f0108176b D'oh, move disable_firefox_studies up in utils
...so things that appear earlier in the file can use it.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-02-21 10:35:51 -08:00
Adam Williamson 3b236c7a39 Try and disable Firefox 'shield studies'
I've seen a few tests fail lately because Firefox suddenly
opened a tab with some "Privacy Study" in it and switched to it.
Per https://bugzilla.mozilla.org/show_bug.cgi?id=1529626 , this
should be the way to disable these...

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-02-21 09:51:07 -08:00
Adam Williamson f50f6d66b1 Fix up the 'skip update on support server' stuff (a20ea59)
The previous attempt at this (in a20ea59) was wrong, because it
meant we didn't create the update repo at all, and we do need
the repo to exist in the support_server test every time (as the
point of the support server is to serve it out via NFS). We just
don't want to *use it on the support_server system itself* if
the releases don't match. So rejig things to do that properly.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-02-08 08:20:24 -08:00
Adam Williamson 99302c6fd4 Add a live image build test for updates
Just like the installer image build test, only...it builds a live
image. This involves reimplementing quite a chunk of the Koji
livemedia task. Ah, well. Also involves rethinking the flavor
names a bit here, these seem...better.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-02-07 18:28:24 -08:00
Adam Williamson 8d1d150798 Handle running update tests on Koji tasks
We quite often want to run the update tests on a Koji task (not
a Bodhi update) for some reason - usually to test a potential
fix for an issue, or at a maintainer's request to test a change
before it is merged upstream and officially sent out as an
update. Up till now I've always hacked up utils.pm on the
staging server by hand to do this, which is horrible. Together
with a commit to fedora_openqa, this should allow us to do it in
a nice, sane way via the CLI. It's mostly just tweaking the
"updates" repo setup in utils.pm as you'd expect, but there's a
bit of subtlety to it because of the installer tests that use
%ADVISORY% as a variable substitution in the disk image name;
you can't do something like `%ADVISORY or KOJITASK%`, sadly, so
I had to have almost-redundant variables ADVISORY, KOJITASK and
ADVISORY_OR_TASK (we could kinda just live with ADVISORY_OR_TASK
except I didn't want to drop ADVISORY as it's an unnecessary
change from previous behavior).

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-01-29 11:54:15 +01:00
Adam Williamson 12affb145f Add update tests to build and test a netinst image
This adds a test which builds a netinst image potentially with
the package(s) from the update, and uploads that image. It also
adds a test which runs a default install using that image. This
is intended to check whether the update breaks the creation or
use of install images; particularly this will let us test
anaconda etc. updates. We also update the minimal disk image
name, as we have to make it bigger to accommodate this test,
and making it bigger changes its name - the actual change to
the disk image itself is in createhdds. We also have to redo a
bunch of installer needles for F28 fonts, after I removed them
a month or so back...

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-01-18 08:24:44 -08:00
Adam Williamson 2c7562eea4 Log more network status stuff for anaconda failures without net
This was in installedtest.pm but not anacondatest.pm, may as well
have it in both.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-01-16 11:25:03 -08:00
Adam Williamson 536f699013 Post fail: handle landing in dracut shell, upload rdsosreport
If a test fails to the dracut shell, we currently don't do
anything useful. This should recognize when that happens, and
upload rdsosreport.txt.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-01-16 11:24:06 -08:00
Adam Williamson 9df8be32cb Upload dbus.log on anaconda failure
This log file was quite recently added to anaconda, let's upload
it with all the others.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-01-14 12:57:38 -08:00
Adam Williamson a1bb5bd1f7 Drop an update test hotfix (no longer needed, breaks F28)
The packages from this update seem to be breaking F28 update
tests for some reason; a later update has gone stable anyway, so
this is no longer needed.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-01-07 18:13:21 -08:00
Lukas Ruzicka 39d3427471 Create a test suite for application start stop testing.
Merges #86
Fixes #85
2019-01-04 15:23:03 -08:00
Adam Williamson 141f29c7cc fix syntax of advisory_check_nonmatching_packages in post-fail
You...can't do this like that.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-01-03 11:35:40 -08:00
Adam Williamson ca6e1105c1 Work around RHBZ #1663050 (tty quitting on exit from startx)
A bug showed up in Rawhide where, when you run startx in a tty,
when you exit that X session, the tty quits and returns to the
login prompt. This is a slightly sloppy workaround for that
problem.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-01-02 13:53:31 -08:00
Adam Williamson e9642eace5 Give console login even longer on aarch64 for #1644919
Even an extra 30 secs doesn't seem to be long enough for the
aarch64 tests.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-12-18 11:01:35 -08:00
Adam Williamson d6de57c6de Drop RHBZ#1618928 workaround
Bug was fixed back in August.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-12-17 14:10:58 -08:00
Adam Williamson 451f35feec Remove another case of RHBZ#1622254 workaround
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-12-17 14:04:53 -08:00
Adam Williamson 777de30179 Remove RHBZ#1622254 workaround
Fixed since 2018-09-12.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-12-17 13:47:46 -08:00
Adam Williamson b171c0a1c4 Try to fix the 'matching update package' check for kernel
That clever-clever 'check the packages from the update were
installed' thing from yesterday breaks on kernel updates, as
they're installonly; after the update, the new version of the
package is installed, but the *old* version is too, and the way
I implemented the check, it treats that as a failure. Let's try
and handle this a somewhat-clever way (if this fails, I'm just
going to grep out lines with 'kernel' in them, as a *dumb* way).

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-12-13 18:52:51 -08:00
Adam Williamson 4629e5b740 Fix console keyboard layout in installedtest post_fail_hook
This should fix log collection when a French or Japanese test
fails before the test itself would have done this.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-12-13 16:29:53 -08:00
Adam Williamson 12e103e3da Factor meat out of advisory_post and do it in postfail too
If an update test fails before reaching advisory_post, we don't
generate the 'what update packages were installed' and 'were
any update packages *not* installed when they should have been'
logs, but these may well be useful for diagnosing the failure -
so let's also do the same stuff there. Only let's not do it all
twice.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-12-12 22:17:29 -08:00
Adam Williamson 764c6dbd95 Notice when update package should have been installed but wasn't
We hit an interesting case in update testing recently:

https://bodhi.fedoraproject.org/updates/FEDORA-2018-115068f60e

An earlier version of that update failed testing. When we dug
into it a bit, we found that the test was failing because an
earlier version of the `pki-server` package was installed than
the version that was in the update; when asked (as part of
FreeIPA deployment) to install it, dnf had noticed that there
were dependency issues with the version of the package from the
update, but it happened to be able to install the version from
the frozen 'stable' repo...so it just went ahead and did that.

In this case, the 'missed' package resulted in a test failure,
but it'd actually be possible for this to happen and the test
to complete; we really ought to notice when this happens, and
treat it as a test failure.

So what this attempts to do is: at the end of all update tests,
check for all installed packages with the same name as a package
from the update, and compare their full NEVR to the one of the
package from the update. If a package with the same name as one
of the update packages is installed, but does not appear to be
the *same NEVR*, we fail, and upload the lists of packages for
manual investigation as to what the heck's going on.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-12-12 22:17:29 -08:00
Adam Williamson 65b4755358 Simplify anaconda error screen handling, add new needle
There's really no point having separate error and error_report
needles. Just match on error_report as well as clicking on it.
Also add a new error_report needle for latest Rawhide fonts.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-12-12 22:06:20 -08:00
Adam Williamson 0639468de6 Use -l for systemctl status (to avoid ellipsization)
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-12-12 12:36:21 -08:00