Commit Graph

344 Commits

Author SHA1 Message Date
Adam Williamson
12affb145f Add update tests to build and test a netinst image
This adds a test which builds a netinst image potentially with
the package(s) from the update, and uploads that image. It also
adds a test which runs a default install using that image. This
is intended to check whether the update breaks the creation or
use of install images; particularly this will let us test
anaconda etc. updates. We also update the minimal disk image
name, as we have to make it bigger to accommodate this test,
and making it bigger changes its name - the actual change to
the disk image itself is in createhdds. We also have to redo a
bunch of installer needles for F28 fonts, after I removed them
a month or so back...

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-01-18 08:24:44 -08:00
Adam Williamson
510662c1ec Drop all RHBZ #1663040 workarounds, should be fixed now
In Fedora-Rawhide-20190117.n.0 #1663040 should be fixed, so drop
all workarounds for it.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-01-17 08:35:30 -08:00
Adam Williamson
ec762117d0 Wait an extra 2 minutes for the installer 'beta nag' screen
There seems to be an issue in Rawhide ATM which can cause the
'beta nag' screen to take a very long time to appear. Bump the
timeout to avoid tests failing on this.

https://bugzilla.redhat.com/show_bug.cgi?id=1666112

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-01-14 15:22:01 -08:00
Lukas Ruzicka
39d3427471 Create a test suite for application start stop testing.
Merges #86
Fixes #85
2019-01-04 15:23:03 -08:00
Adam Williamson
c6fac00698 More fixing of the workaround...
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-01-04 13:55:31 -08:00
Adam Williamson
42a38009eb Fix workaround (have to wait for desktop to appear)
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-01-04 13:27:01 -08:00
Adam Williamson
4e537684b1 Work around #1663040 in Workstation live installs
We don't want the tests to fail on this now we know what the
bug is, really - we want to find if there are any subsequent
fails, and allow the post-install tests to run also. So, let's
make it a soft failure.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-01-04 13:01:36 -08:00
Adam Williamson
2b29ecbc18 FreeIPA client: set longer timeout for login(1) (#1661273)
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-12-20 09:02:36 -08:00
Adam Williamson
3d35ec9670 Update a comment to not mention a removed workaround
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-12-17 14:36:10 -08:00
Adam Williamson
00d1fdbd2c Drop #1600823 workaround
Bug was fixed months ago.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-12-17 14:30:09 -08:00
Adam Williamson
f40599ee15 Drop all #1594402 workarounds
I'm pretty sure we got all the bugs this was working around
fixed. Again, if not, we can put this back!

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-12-17 14:27:18 -08:00
Adam Williamson
01fd92cb34 Drop #1444225 workaround
The bug never got explicitly addressed, but it's very old and
anaconda has been substantially changed since. The workaround
sometimes triggers erroneously now (because the icon sometimes
goes black while the spoke is still in 'Checking storage
configuration...' state), which is awkward. I can't be 100% sure
the bug doesn't sometimes still happen, but if it does, we'll
notice fairly soon, and we can tweak this and put it back.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-12-17 14:18:05 -08:00
Adam Williamson
d6de57c6de Drop RHBZ#1618928 workaround
Bug was fixed back in August.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-12-17 14:10:58 -08:00
Adam Williamson
7cbd859d81 Remove another old workaround (for #1606541)
This was fixed long ago as well...

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-12-17 13:36:13 -08:00
Adam Williamson
e04ee00e19 Drop various old workarounds from _support_server
I tested a run with these commented out, it worked fine. Seems
the bugs are all fixed now.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-12-17 13:06:19 -08:00
Adam Williamson
ef3555aaa1 Drop another old workaround (KDE package signature check)
It seems Rawhide auto-signing is working fine now: openQA claims
this needle has 'never' been matched (which really means 'not
for a long time'), and I can't find any test fail in the last
year which looks like it landed on this 'authenticate' screen
but the needle failed to match, or anything like that.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-12-17 12:24:09 -08:00
Adam Williamson
1fd0097d1d Simplify an 'if >F26' thing
We don't care about anything older than 27 any more.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-12-17 12:19:44 -08:00
Adam Williamson
d1e7b89efd Fix a potential race in desktop update test
https://openqa.stg.fedoraproject.org/tests/424393 is a failure
where the 'Download' [updates] button was already visible when
we went to the tab. We already checked whether an 'apply' button
is visible and skipped the 'refresh' click if so, but because
the 'download' button is a new thing, we weren't skipping the
'refresh' click if 'download' was already visible.

So in this case, even though we could already see 'download', we
went ahead and clicked 'refresh'...then *immediately* started
looking for 'download'. It seems that Software did not refresh
and remove the 'Download' button *immediately* when we pressed
'refresh' - it left the 'Download' button visible briefly, and
*in this brief window*, we clicked it. *Then* Software kinda
'noticed' we'd clicked 'Update', and it seems it just sort of
throws away our click on 'Download' at that point and does the
refresh.

So at that point, the test thinks it's clicked 'Download' and
expects to see 'Apply', but actually the 'Download' click got
more or less thrown away, so the test fails, sitting at the
'Download' button.

To solve this, let's just extend the existing check to skip the
'refresh' click if 'download' *or* 'apply' are already visible.

There is a sort of possibility here that we could wind up
downloading and installing some updates that existed and were
noticed *before* we did our python3-kickstart trick, but not
install the python3-kickstart update, and cause the test to fail
because of that, but that doesn't seem to have happened before
when we were seeing the 'update' button, so I think I'm not
going to borrow trouble. If it happens, we'll deal with it I
guess.

The comment talks only about KDE, but clearly it can be the case
that an automatic check makes the button visible on GNOME too,
so let's rewrite the comment too.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-12-17 12:10:06 -08:00
Adam Williamson
517750443e Remove RHBZ #1638563 workaround
The fix for this bug was sent to all releases now, so we should
not need the workaround any more. Let's kill it.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-12-17 12:02:57 -08:00
Adam Williamson
0498f4db94 Use 30s timeout on the check_screen in the workaround
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-12-13 15:29:35 -08:00
Adam Williamson
b3f51e6108 Gah come back missing bracket
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-12-13 15:23:50 -08:00
Adam Williamson
1f7e5ebe20 Attempt a workaround for RHBZ#1659266
This is breaking the memory_check tests. I just reproduced it
manually and the UI *does* come back to life if you wait some
time; let's see if we can work around the bug this way.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-12-13 15:15:49 -08:00
Adam Williamson
12e103e3da Factor meat out of advisory_post and do it in postfail too
If an update test fails before reaching advisory_post, we don't
generate the 'what update packages were installed' and 'were
any update packages *not* installed when they should have been'
logs, but these may well be useful for diagnosing the failure -
so let's also do the same stuff there. Only let's not do it all
twice.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-12-12 22:17:29 -08:00
Adam Williamson
764c6dbd95 Notice when update package should have been installed but wasn't
We hit an interesting case in update testing recently:

https://bodhi.fedoraproject.org/updates/FEDORA-2018-115068f60e

An earlier version of that update failed testing. When we dug
into it a bit, we found that the test was failing because an
earlier version of the `pki-server` package was installed than
the version that was in the update; when asked (as part of
FreeIPA deployment) to install it, dnf had noticed that there
were dependency issues with the version of the package from the
update, but it happened to be able to install the version from
the frozen 'stable' repo...so it just went ahead and did that.

In this case, the 'missed' package resulted in a test failure,
but it'd actually be possible for this to happen and the test
to complete; we really ought to notice when this happens, and
treat it as a test failure.

So what this attempts to do is: at the end of all update tests,
check for all installed packages with the same name as a package
from the update, and compare their full NEVR to the one of the
package from the update. If a package with the same name as one
of the update packages is installed, but does not appear to be
the *same NEVR*, we fail, and upload the lists of packages for
manual investigation as to what the heck's going on.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-12-12 22:17:29 -08:00
Adam Williamson
e76ad4bbc1 Crank sss debuglevel up to 9, and also do it for cockpit test
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-11-30 14:58:53 -08:00
Adam Williamson
14ad5b97f1 Still try and upload testedpkgs even if comm 'fails'
From local experimentation, it still actually produces the
output, even though it prints the message about the order being
wrong and exits 1.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-11-08 17:46:45 -08:00
Adam Williamson
ddf6ba5a6b update tests: don't fail if comm is unhappy about the alphabet
Weirdly, occasionally some update tests seem to fail because
the 'comm' util we use to produce the list of packages from the
update that were actually tested during the job doesn't think
one of the input files is in alphabetical order, even though we
sort them both when they're produced. I don't know if this is
possibly due to the definition of 'alphabetical order' changing
as part of the update, or what. But we really shouldn't *fail*
the test when this happens, as it's not part of the functional
test, we're just producing convenience data. So, let's handle
the command failing, and if it happens, upload the input files
so we can maybe figure out why it's unhappy...

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-11-08 16:59:06 -08:00
Adam Williamson
70ef3404f0 Tweak previous commit to avoid some bugs
The previous commit would lead to the 'workaround' getting hit
incorrectly, and might have had some other issues...tweak it a
bit.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-10-31 12:45:11 -07:00
Adam Williamson
fd753b2e3a Handle split of 'download' and 'apply' phases in gnome-software
GNOME Software 3.30.5 split the offline update process into two
separate 'download' and 'apply' phases. So we need to handle
clicking 'download' before 'apply', if that happens.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-10-31 11:50:14 -07:00
Adam Williamson
e6c8c5f0ff Work around Firefox 'close multiple tabs' warning
Somehow, recently, FreeIPA tests are running into Firefox not
quitting because it's showing a warning about closing multiple
tabs. (I think we didn't *get* multiple tabs before but now we
do, for some reason). So let's work around this by clicking
"Close tabs" if the warning appears.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-10-30 18:34:37 -07:00
Adam Williamson
129d1316a6 Fix tty switching for desktop_notifications
Lately, we can't be sure the desktop will be on tty1 after we
do 'systemctl isolate graphical.target'. For recent Workstation
lives it actually shows up on tty2.

We could be 'clever' and switch to tty2 on F29+ Workstation
lives...but actually it seems like if we just don't do anything,
systemd switches us to the correct tty. So let's rely on that,
at least as long as it's working.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-10-12 21:16:33 -07:00
Adam Williamson
22c0b5bc04 Disable hidden grub menu for uploaded base installs
At least one test (desktop_notifications_postinstall) boots from
the disk image uploaded by install_default_upload, and needs to
access the grub menu. On F29+ Workstation this is failing,
because the grub menu is now hidden by default, so when the test
boots, it never sees the bootloader screen, and fails.

I considered trying to teach it to hold down shift or hit f8 or
esc at the right time, but that seems like it might be hard. So
instead let's just try to disable the hidden menu when we're
about to upload the installed system image. This is kinda going
against the 'preserve natural system behaviour' principle we try
to use for openQA, but I think it's OK as we do have other tests
that will exercise the 'hidden boot menu' stuff to some extent.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-10-12 15:36:55 -07:00
Adam Williamson
17b6d9f708 Tweak the workaround to work for F27 too
On F27 we don't get a 'Software is up to date' screen because
there's an upgrade available. Let's work with the refresh button
instead.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-10-11 22:23:53 -07:00
Adam Williamson
63d8f34a0e Tweak the workaround loop a bit, refresh the comments
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-10-11 16:18:04 -07:00
Adam Williamson
db4ab638da Restore modified version of the #1314991 workaround for #1638563
We're not seeing *exactly* #1314991 any more, but we're seeing
something that looks quite similar: the first attempt to find
updates just doesn't find any. No error message, no updates. I
have reported a bug for this and am investigating it, in the
meantime, let's restore the workaround, elaborated a bit, and
looking for the 'Software is up to date' screen instead of the
error message.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-10-11 16:07:58 -07:00
Adam Williamson
25ad8a6aeb Drop workaround for #1314991, it doesn't work any more
I rather suspect the *bug* is still basically present and it's
why this test often fails, but we no longer seem to see the
*error message* which lets us detect the bug happening. This
needle has not been hit by any test for six months. So let's
remove the workaround as it adds complexity.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-10-10 13:45:51 -07:00
Adam Williamson
9869920f5b Use longer timeout for root console switch after liveinst
For some reason, in recent tests, switching to a console after
live install completes is taking a long time, and tests are
failing because we 'only' allow 10 seconds for the login prompt
to appear. This seems to indicate some kind of performance bug,
but we don't really want all liveinst tests to fail on in, this
is not primarily a performance testing framework. So let's
tweak the root_console / console_login bits a bit to allow a
configurable timeout for the login prompt to appear, and use
that to wait 30 secs instead of 10 in this case.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-10-06 08:44:34 -07:00
Adam Williamson
6e262be28b Add a check that FreeIPA is actually up after upgrade
The FreeIPA upgrade test didn't actually check that FreeIPA is
actually running after the upgrade and reboot, it just kinda
assumed it is. Let's add a check to the start of the 'check'
test module that makes sure ipa.service actually comes up to
'active' state. This'll make it clearer when tests are failing
because FreeIPA didn't come up right after the upgrade. The
check will run on non-upgrade tests too, but that's fine.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-10-05 23:53:17 -07:00
Adam Williamson
0bf76db7d5 Add a test of bootchain stuff for updates
This adds a new test intended to just check boot chain things
for updates. It doesn't run any test modules besides the stock
update ones, but sets a variable, ADVISORY_BOOT_TEST, which
causes _advisory_update to do some additional stuff after
installing the updates but before rebooting: it forces regen
of the initramfs and bootloader config, and reinstalls the
bootloader on BIOS (not UEFI as it's not relevant). If the
following boot fails, we probably have a bug somewhere.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-10-04 16:43:15 -07:00
Adam Williamson
d2d6bfa695 Try something different for screen corruption after IPA uninstall
There's this annoying problem where the screen sometimes goes
messed up after ipa-server-uninstall. 'clear' doesn't seem to
really work to fix it up either. Let's try flipping between
ttys. I don't like this much as it's already a pain trying to
work out / remember what tty we might possibly be on at any
given time, but I think we're always on either 1 or 3 here, so
let's do ctrl-alt-f1 ctrl-alt-f3 to ensure at least one change
and wind up on tty3...

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-09-28 18:09:58 -07:00
Adam Williamson
c2fb886d6f Add another wait to avoid a transition animation in anaconda
I swear, these transitions drive me nuts.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-09-28 17:46:36 -07:00
Adam Williamson
a49f328dc6 Tweak how update-upgrade tests are handled a bit
Looking at this, it's a bit weird: the updated packages are
actually included in the upgrade process, but we still run
_advisory_update, which does basically nothing...then reboots.
That's kinda silly and makes the tests a bit flaky, let's fix
it. I don't think there's actually any problem with doing the
upload of updatepkgs.txt in _repo_setup_updates, becase that
already guards against being run more than once, it just bails
very early if it's already been run.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-09-28 17:23:51 -07:00
Lukas Ruzicka
24e68aa8a2 Create openqa tests to test modularity. 2018-09-26 23:09:36 -07:00
Adam Williamson
923267574d Drop the workaround for #1625572
It's fixed everywhere now, and the workaround can misfire if
the first g-i-s is slow quitting (see
https://openqa.fedoraproject.org/tests/280482)
2018-09-16 08:39:30 -07:00
Adam Williamson
f334df1337 Bump ipa-replica-install timeout a bit (it takes a long time)
I'm going to figure out if it's a bug that it takes so long, but
for now let's just bump the timeout.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-09-07 15:56:11 -07:00
Adam Williamson
e2eb794a87 Tweak the workaround yet again (with soft fail this time)
Sigh, sorry, just perfecting. This way it won't fail when the
bug is fixed (hopefully).

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-09-06 20:36:29 -07:00
Adam Williamson
7dff1843db Tweak that workaround again (forgot the version check)
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-09-06 19:53:07 -07:00
Adam Williamson
211cc221b3 Rejig the 1625572 workaround to just use an existing conditional
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-09-06 19:00:49 -07:00
Adam Williamson
af6b9b15aa Gah, fix version testing in previous commit
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-09-06 18:23:13 -07:00
Adam Williamson
5a48086e61 Don't expect GDM when doing GNOME no-user install on F28+
GNOME now transitions straight from the g-i-s 'user creation'
mode to a logged-in desktop.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-09-06 18:21:12 -07:00