Commit Graph

409 Commits

Author SHA1 Message Date
Adam Williamson
25ad8a6aeb Drop workaround for #1314991, it doesn't work any more
I rather suspect the *bug* is still basically present and it's
why this test often fails, but we no longer seem to see the
*error message* which lets us detect the bug happening. This
needle has not been hit by any test for six months. So let's
remove the workaround as it adds complexity.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-10-10 13:45:51 -07:00
Adam Williamson
9869920f5b Use longer timeout for root console switch after liveinst
For some reason, in recent tests, switching to a console after
live install completes is taking a long time, and tests are
failing because we 'only' allow 10 seconds for the login prompt
to appear. This seems to indicate some kind of performance bug,
but we don't really want all liveinst tests to fail on in, this
is not primarily a performance testing framework. So let's
tweak the root_console / console_login bits a bit to allow a
configurable timeout for the login prompt to appear, and use
that to wait 30 secs instead of 10 in this case.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-10-06 08:44:34 -07:00
Adam Williamson
6e262be28b Add a check that FreeIPA is actually up after upgrade
The FreeIPA upgrade test didn't actually check that FreeIPA is
actually running after the upgrade and reboot, it just kinda
assumed it is. Let's add a check to the start of the 'check'
test module that makes sure ipa.service actually comes up to
'active' state. This'll make it clearer when tests are failing
because FreeIPA didn't come up right after the upgrade. The
check will run on non-upgrade tests too, but that's fine.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-10-05 23:53:17 -07:00
Adam Williamson
0bf76db7d5 Add a test of bootchain stuff for updates
This adds a new test intended to just check boot chain things
for updates. It doesn't run any test modules besides the stock
update ones, but sets a variable, ADVISORY_BOOT_TEST, which
causes _advisory_update to do some additional stuff after
installing the updates but before rebooting: it forces regen
of the initramfs and bootloader config, and reinstalls the
bootloader on BIOS (not UEFI as it's not relevant). If the
following boot fails, we probably have a bug somewhere.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-10-04 16:43:15 -07:00
Adam Williamson
d2d6bfa695 Try something different for screen corruption after IPA uninstall
There's this annoying problem where the screen sometimes goes
messed up after ipa-server-uninstall. 'clear' doesn't seem to
really work to fix it up either. Let's try flipping between
ttys. I don't like this much as it's already a pain trying to
work out / remember what tty we might possibly be on at any
given time, but I think we're always on either 1 or 3 here, so
let's do ctrl-alt-f1 ctrl-alt-f3 to ensure at least one change
and wind up on tty3...

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-09-28 18:09:58 -07:00
Adam Williamson
c2fb886d6f Add another wait to avoid a transition animation in anaconda
I swear, these transitions drive me nuts.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-09-28 17:46:36 -07:00
Adam Williamson
a49f328dc6 Tweak how update-upgrade tests are handled a bit
Looking at this, it's a bit weird: the updated packages are
actually included in the upgrade process, but we still run
_advisory_update, which does basically nothing...then reboots.
That's kinda silly and makes the tests a bit flaky, let's fix
it. I don't think there's actually any problem with doing the
upload of updatepkgs.txt in _repo_setup_updates, becase that
already guards against being run more than once, it just bails
very early if it's already been run.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-09-28 17:23:51 -07:00
Lukas Ruzicka
24e68aa8a2 Create openqa tests to test modularity. 2018-09-26 23:09:36 -07:00
Adam Williamson
923267574d Drop the workaround for #1625572
It's fixed everywhere now, and the workaround can misfire if
the first g-i-s is slow quitting (see
https://openqa.fedoraproject.org/tests/280482)
2018-09-16 08:39:30 -07:00
Adam Williamson
f334df1337 Bump ipa-replica-install timeout a bit (it takes a long time)
I'm going to figure out if it's a bug that it takes so long, but
for now let's just bump the timeout.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-09-07 15:56:11 -07:00
Adam Williamson
e2eb794a87 Tweak the workaround yet again (with soft fail this time)
Sigh, sorry, just perfecting. This way it won't fail when the
bug is fixed (hopefully).

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-09-06 20:36:29 -07:00
Adam Williamson
7dff1843db Tweak that workaround again (forgot the version check)
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-09-06 19:53:07 -07:00
Adam Williamson
211cc221b3 Rejig the 1625572 workaround to just use an existing conditional
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-09-06 19:00:49 -07:00
Adam Williamson
af6b9b15aa Gah, fix version testing in previous commit
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-09-06 18:23:13 -07:00
Adam Williamson
5a48086e61 Don't expect GDM when doing GNOME no-user install on F28+
GNOME now transitions straight from the g-i-s 'user creation'
mode to a logged-in desktop.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-09-06 18:21:12 -07:00
Adam Williamson
e3887c5a83 Work around RHBZ#1625572 (both g-i-s modes running)
RHBZ#1625572 is for gnome-initial-setup running in 'first login'
mode after it's already run in 'user creation' mode (which isn't
meant to happen). This works around that so the subsequent tests
can run. We don't soft-fail because meh.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-09-06 17:59:40 -07:00
Adam Williamson
c62a04bac8 Remove old workaround for g-i-s failing to run on F26
This hasn't been a problem for ages.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-09-06 17:49:03 -07:00
Adam Williamson
cc2f1a3cec _graphical_wait_login: drop an old FAW workaround
This bug was fixed long ago, we no longer need the workaround.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-09-06 15:02:20 -07:00
Adam Williamson
64c5070b06 Simplify _graphical_wait_login by dropping a huge conditional
If USER_LOGIN is false we can just return; when we reach the
login screen. We don't need a huge conditional when we don't do
anything *after* it, in the false case.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-09-06 15:01:08 -07:00
Adam Williamson
cfe2a33038 Have domain controller upload logs *before* decommissioning
It transpires that decommissioning wipes some stuff, like the
dirsrv logs. Obviously we want these included in the logs we
upload for reference purposes, so let's upload earlier.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-08-24 14:42:52 -07:00
Adam Williamson
bdd26a09ee Have kickstart tests handle RHBZ#1618928 too
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-08-19 10:03:27 -04:00
Adam Williamson
9df64398ee Add a FreeIPA replication job set
This adds a set of jobs to test FreeIPA replication. We deploy
a server, deploy a replica of that server, then enrol a client
against the replica and run the client tests.

At first I was planning to add the replica testing into the
main set of FreeIPA tests, but the test ordering/blocking (via
mutexes and barriers and what-have-you) just turns into a big
nightmare that way. This way seems rather simpler to deal with.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-08-02 09:59:40 -07:00
Adam Williamson
35caea44bc Revert "Try downgrading softhsm as a workaround for #1607635"
This reverts commit 0289716d70.
Turns out the downgrade doesn't avoid the crash. :(
2018-07-30 11:58:12 -07:00
Adam Williamson
0289716d70 Try downgrading softhsm as a workaround for #1607635
We'd really like to know if FreeIPA is working aside from this
crasher bug, so let's workaround it.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-07-30 11:33:02 -07:00
Adam Williamson
acc4ccd7cc Add a sleep to desktop_update_graphical
Try and avoid failure to launch alt-f1 dialog...

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-07-30 11:23:59 -07:00
Adam Williamson
a479774026 Correct previous commit
Should be if, not unless...damn exit codes.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-07-23 14:08:37 -07:00
Adam Williamson
3817e7128d Work around RHBZ #1606541 (on Rawhide, not updates)
We kind of want to know if FreeIPA is working aside from this
known bug, so let's treat it as a soft failure and work around
it. But only for Rawhide, not for F27/F28 updates tests.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-07-23 11:12:57 -07:00
Adam Williamson
7621cd7e57 Move the new base_services_start check to the start
...otherwise we actually return before it, if no services fail.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-07-13 14:02:31 -07:00
Adam Williamson
b4cd1b4a9e Check for services deleted to break loops in base_services_start
https://bugzilla.redhat.com/show_bug.cgi?id=1600823 shows a
case where systemd throws a service that would usually have been
started out of the boot process *entirely* in order to resolve a
dependency loop. This means the service won't show up as failed,
it will just be inactive when it should be active. This still
should constitute a failure of this test, so let's add a check
for the log message that indicates this situation.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-07-13 13:37:04 -07:00
Adam Williamson
a362ecb2a0 Add a softfail workaround for #1600823
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-07-13 00:39:53 -07:00
Adam Williamson
9ae2c249f2 Stop using rolekit for database server role on F29+
As for the domain controller role, stop using rolekit on F29+,
as it's going away. Continue using it on <F29.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-07-10 17:54:30 -07:00
Adam Williamson
5999058d07 We should *not* check CURRREL for rolekit in _check
...because by this point in the upgrade test, the system is
upgraded, and rolekit won't be there on F29+.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-07-10 16:21:54 -07:00
Adam Williamson
7e7016ea14 Convert domain controller test not to use rolekit
Rolekit is going away. At least for the F29 cycle, though, we
still want to test basically the same functionality. This ports
the 'domain controller role' test to use ipa-server-install
directly rather than rolectl.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-07-09 19:54:49 -07:00
Adam Williamson
1c1b33840f Work around an anaconda logging bug that showed up today
https://github.com/rhinstaller/anaconda/pull/1519 should fix it
on the anaconda side, till that's merged, we need this.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-06-28 15:13:06 -07:00
Adam Williamson
b81f681f02 Load us console layout in uefi_postinstall too
...since it runs during non-English tests on aarch64.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-06-25 11:28:12 -07:00
Adam Williamson
861ad5d4aa Load us layout before doing post-install aarch64 cmdline hack
It seems that for some reason the localized layout gets loaded
on the installer VTs by this point in time, so we need to load
'us' again for this complex command to work.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-06-25 09:35:56 -07:00
Adam Williamson
8e0764bc80 Brown paper bag fix: need use utils for type_safely
...sigh.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-06-24 17:21:08 -07:00
Adam Williamson
d8f5f56fff Also do 'console=tty0' workaround for rescue boot on aarch64
...sigh, another place this is needed, and it's a bit ugly
here. Ah, well.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-06-24 17:17:10 -07:00
Adam Williamson
e200e29fff Add another #1594402 comment
Just commenting why we do this again.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-06-24 17:16:53 -07:00
Adam Williamson
f30b7517ce Correct the text install workaround for aarch64
I sorta screwed up the brackets there a bit.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-06-24 17:09:19 -07:00
Adam Williamson
ab32b75aba Note bug related to 'console=tty0 quiet' workaround
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-06-22 13:51:03 -07:00
Adam Williamson
1b51987478 Add 'console=tty0' for anaconda text install on aarch64
We need this at least till #1594402 is fixed.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-06-22 13:48:36 -07:00
Adam Williamson
4d803eda52 Add 'console=tty0 quiet' to cmdline for aarch64 installs
We need this as part of the fix for #1593028, at least until
the kernel package is changed to no longer have
CONFIG_CMDLINE="console=ttyAMA0" in the config for aarch64
builds. Fully fixing the bug also requires some change to the
kernel or dracut or something.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-06-22 11:56:21 -07:00
Adam Williamson
faef957bac Revert workaround for RHBZ#1553935 now it's fixed
We don't need this any more, so let's remove the complication.
2018-06-18 11:02:08 -07:00
Adam Williamson
6324db0b87 Fix some minor syntax errors in previous commit
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-06-13 10:11:53 -07:00
Lukas Ruzicka
8897fe45a6 Add test for Arabic installation (revisited). 2018-06-13 11:56:26 +02:00
Adam Williamson
bd1b951f71 Bump a magic sleep a bit
Seems we need to wait a bit longer for this stupid transition
to happen on aarch64...

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-06-11 15:59:44 -07:00
Adam Williamson
06fbeca281 Avoid unconditional wait for Timbuktu screen in _boot_to_anaconda
The way this currently works, the test unconditionally waits 60
seconds for the "Timbuktu screen" (the warning dialog shown on
pre-release images) to appear when anaconda is starting up, even
if it's testing an image where it doesn't show up. Now we test
Atomic nightlies and live respins and stuff this happens quite a
lot, so let's avoid it. This way if the hub appears during those
60 seconds we'll spot it right away and continue, otherwise we
behave the same as before.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-05-24 14:18:21 -07:00
Adam Williamson
05c9f4fbcd Make sure all check_screen calls have explicit timeout
Upstream is gonna change the default from 30 to 0, it seems:
https://github.com/os-autoinst/os-autoinst/pull/965
so let's go ahead and change these two cases where we have no
explicit timeout to have one.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-05-24 14:17:24 -07:00
Adam Williamson
33ac181955 Use mirrorlist instead of baseurl for updates tests
The reason we have all this horrible code to use the commented-
out baseurl lines in the repo files instead of the metalinks
that are usually used is a timing issue with the metalink
system. As a protection against stale mirrors, the metalink
system sends the package manager a list of mirrors *and a list
of recent checksums for the repo metadata*. The package manager
goes out and gets the metadata from the first mirror on the
list, then checksums it; if the checksum isn't on the list of
checksums it got from mirrormanager, it assumes that means the
mirror is stale, and tries the next on the list instead.

The problem is that MM's list of checksums is currently only
updated once an hour (by a cron job). So we kept running into
a problem where, when a test ran just after one of the repos
had been regenerated, the infra mirror it's supposed to use
would be rejected because the checksum wasn't on the list - but
not because the mirror was stale, but because it was too fresh,
it had got the new packages and metadata but mirrormanager's
list of checksums hadn't been updated to include the checksum
for the latest metadata.

All this baseurl munging code was getting ridiculous, though,
what with the tests getting more complicated and errors showing
up in the actual repo files and stuff. It occurred to me that
instead of using the baseurl we can just use the 'mirrorlist'
system instead of 'metalink'. mirrorlist is the dumber, older
system which just provides the package manager a list of mirrors
and nothing else - the whole stale-mirror-detection-checksum
thing does not happen with mirrorlists, the package manager just
tries all the mirrors in order and uses the first that works.
And happily, it's very easy to convert the metalink URLs into
mirrorlist URLs, and it saves all that faffing around trying to
fix up baseurls.

Also, adjust upgrade_boot to do the s/metalink/mirrorlist/
substitution, so upgrade tests don't run into the timing issue
in the steps before the main repo_setup run is done by
upgrade_run, and adjust repo_setup_compose to sub this line out
later.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-05-18 16:41:13 -07:00
Adam Williamson
8754611eef Extend some boot timeouts in upgrade tests
Sometimes rebooting during upgrade tests seems to take longer
than these timeouts allow, so let's bump them a bit.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-05-11 09:02:21 -07:00
Adam Williamson
fdaa4783e7 _check_install_source: handle 'added repo' and 'enabled repo'
The text changed from 'added repo' to 'enabled repo' in Rawhide
after F28, so let's handle both at least till F28 is EOL.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-05-07 15:50:47 -07:00
Adam Williamson
c3a511e052 support_server: workaround RHBZ#1554390 (breaks dnsmasq on F28)
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-05-03 14:00:41 -07:00
Adam Williamson
0ab8c6bd3f Click on language select screen (RHBZ #1566066)
We've been seeing an odd case lately where the language select
screen is not foregrounded when it appears (so all text is
grey). It happens very occasionally on x86_64, but a lot on
ppc64. To work around this, let's add a needle that matches the
inactive screen, and click on the screen when it appears just
to make sure it's active.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-04-25 15:04:52 -07:00
Adam Williamson
70d3f0269c Of course, we have to actually auth *properly*...
...just clicking the button ain't gonna work. D'oh.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-03-28 22:17:33 -07:00
Adam Williamson
1f2e333043 Handle an auth dialog appearing after login on FAW
This will happen until a fedora-release PR is merged and sent
out for F28.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-03-28 21:24:31 -07:00
Adam Williamson
baa7ac4e39 Handle KDE update test when KDE has aleady found updates
We do the 'desktop update' test for KDE via the notification
icon thingy, and it behaves differently depending on whether it
has already detected there are updates or not. The test only
works at present in the case where it *hasn't* - it expects the
notification icon to be in the extended panel and it expects to
see a 'refresh' button, neither of which is the case if it's
already noticed there are updates to install.

We should also force PackageKit to update its list of available
updates after we set up our 'special' update, otherwise on this
path KDE will only install the updates it found *before* we did
our stuff, and the test will fail as our special update won't be
there.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-03-28 19:53:19 -07:00
Adam Williamson
de3f4a873c Sleep a bit less in notification tests (to fix KDE postinst)
In F28 tests, the notification 'counter' thing that we rely on
to check there's only *one* notification seems to suddenly
disappear...right around 10 minutes after the desktop starts up,
which is just how long our test idles for to catch crashes that
happen a little after boot. That causes test fails. Let's try
just cutting the wait down to 8 minutes to see if that helps.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-03-28 19:20:29 -07:00
Adam Williamson
c2a5846064 Fix KDE live notification test (dismiss network notification)
KDE in F28+ seems to show a network connection notification on
live boot, for some reason. Just dismiss it to help the test
pass.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-03-28 19:01:46 -07:00
Adam Williamson
24f2eb39d9 Check for test3's existence at start of freeipa_password_change
This test expects to pick up from freeipa_webui, but that test
is not fatal (i.e. it can fail and we still carry on to this
one). We should probably make them independent, but for now,
just check if 'test3' (one of the users freeipa_webui creates
and that this test requires) actually exists, at the start. If
not, we can just die right away.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-03-23 09:18:39 -07:00
Adam Williamson
4d997d7323 Move install of Firefox into its own milestone module
This reduces duplication, but it also means that if the FreeIPA
web UI module fails, the password change module will pick up
from a point where Firefox is set up and won't fail in a bogus
way because it isn't.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-03-23 09:10:51 -07:00
Adam Williamson
46a41d8d67 More debug logging for FreeIPA server (from ab)
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-03-21 11:34:45 -07:00
Adam Williamson
e2a8c34e86 Tweak mouse placement in _graphical_wait_login for KDE browser
This mouse placement is in the middle of where the 'install
addon' popover appears in Firefox, and that seems like it
sometimes causes the popover to immediately disappear in KDE.
This is pretty corner-case-y so I don't wanna report it as a
bug, let's just tweak the cursor hiding location and see if it
solves the problem.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-03-16 15:10:34 -07:00
Adam Williamson
7c1f59cfe8 Move UEFI postinstall to tty4 too
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-03-10 08:38:20 -08:00
Adam Williamson
a20270e430 Move collect_data and console_shutdown to tty4...
...as somehow a Workstation live install currently has the
desktop on tty3, I have no idea why (g-i-s not quitting right?)

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-03-09 22:06:18 -08:00
Adam Williamson
ed9945de71 chpasswd -R is blocked by SELinux, sigh...
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-03-09 17:06:01 -08:00
Adam Williamson
eedcec1c02 Try and fix root password setting approach
assert_script_run and chroot don't mix...let's try chpasswd.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-03-09 16:25:58 -08:00
Adam Williamson
ed62801202 Fix the conditionals for the Workstation live check
Sigh, perl. Sigh, me.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-03-09 15:47:39 -08:00
Adam Williamson
f09330e897 Work around RHBZ #1553807 by checking if anaconda still running
This is the best workaround I can think of for RHBZ #1553807 -
just check (in the 60 second 'move the mouse' loop) if anaconda
is still running, based on whether its icon is in the top bar
(on Workstation live installs only, obviously).

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-03-09 15:22:29 -08:00
Adam Williamson
71bfb896a4 Wiggle the mouse during install
Seems with the long period of not doing anything and possibly
with very aggressive timeouts in Fedora 28, Workstation live
wants to blank the screen while we're installing. Stop it.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-03-08 21:21:23 -08:00
Adam Williamson
0915b857f9 More tweaking for this damn no root password spoke situation
Previous approach wouldn't work for tests that run after the
install test...let's just set a password from a chroot after
install completes. Don't really like this as it changes the
'real' install process a bit, but it's the least invasive short
term fix at least. We can maybe do something more sudo-y later
with a bit more thought.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-03-08 20:31:14 -08:00
Adam Williamson
7c28cfc909 Adapt for no user/root panes on Workstation live install
Workstation live installs for F28+ drop the user creation and
root password panes from anaconda, so we need to not try and
use them any more. But we still want the old behaviour for F27.
I'm hoping this approach will work, if not, we'll find out soon
enough. This removes the install_no_user test for F28+ as it
will no longer differ from the install_default test.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-03-08 18:32:16 -08:00
Adam Williamson
d070ad44a5 Check default package selection is correct
This adds a check that the default package set selection is
actually correct, where possible and appropriate, as part of
the `_software_selection` test. We do this by examining the
`packaging.log` log file and checking which environment group
was selected.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-01-05 15:18:09 -08:00
Adam Williamson
ffa7ca2447 Add a check that correct filesystem was used on Server installs
This is currently broken, but openQA doesn't notice; we really
should. We could also check the default in other cases, but I
think that's less clear-cut, as it's kind of an anaconda design
choice, it's not mandated in Fedora requirements anywhere.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-01-05 15:18:05 -08:00
Adam Williamson
f70416c6a1 Use release number not "rawhide" for Rawhide upgrades (#1531356)
There's a problem with using `--releasever=rawhide` for upgrade
tests ATM - see #1531356 . To avoid this, we'll try using the
real Rawhide release number (which I'm adapting the scheduler
code to discover and pass in as `RAWREL`).

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-01-04 18:34:20 -08:00
Adam Williamson
2a7b2923f7 Handle BGO#790811 (GNOME 'getting started' crash)
There's a bug causing the 'getting started' screen to crash.
This doesn't really make the system unusable, so treating it
as a soft failure seems appropriate, especially as this will
unblock all the post-install tests on Workstation.
2017-11-24 16:36:31 -08:00
Adam Williamson
e2beb4f046 Bump up some more DNF timeouts due to #1516990
All of these seem vulnerable.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2017-11-23 16:01:58 -08:00
Adam Williamson
a0a63256d0 Whoops, do that last thing properly
We need the 60 second timeout here, not the other place.
2017-11-23 13:42:21 -08:00
Adam Williamson
f20086f59c Wait 60 seconds for 'timbuktu screen' (#1516990)
It takes an unusally long time for Modular images to get from
language selection to the 'timbuktu screen', so give 'em a bit
more time. See bug report for more info.
2017-11-23 11:44:57 -08:00
Adam Williamson
ce07c418d0 Fix (hackily) install of X, Firefox and haveged on Modular
Modular composes don't include these packages, but we need them
to run the web UI tests for FreeIPA and Cockpit. This is the
most reasonable hack I can come up with for now: just use a
non-modular fedora repo to source these packages when doing
Modular compose testing.

If we ever reach an all-Modular future, these packages should
be available in Modular composes I guess, but for now they are
not.
2017-11-07 14:20:36 -08:00
Adam Williamson
14bbfaaa3c Skip software selection on Modular until it's any use
With current Modular composes it does nothing.
2017-11-07 12:44:52 -08:00
Adam Williamson
bd27cc69d8 Don't install haveged for modular tests (yet)
It's not there, I don't think.
2017-11-06 18:01:23 -08:00
Adam Williamson
7b7b63abc1 Workaround F26 base image g-i-s issue for upgrade tests too
Since April there's been some kind of issue in the F26 base
image which means gnome-initial-setup doesn't run on the first
user login (as it should). The F25 base image is fine. I've
not yet had the time to look into this.

I put a workaround in place to prevent this problem causing
false fails of update tests that boot from the F26 base image,
but didn't apply the same workaround to upgrade tests, which
is why upgrade tests from F26 Workstation always fail - they
expect g-i-s to run on first login (which happens after the
upgrade, in upgrade tests) and it doesn't. So let's just extend
the workaround to apply to upgrade tests too, for now, until we
can figure out why this happens.
2017-10-20 12:21:31 -07:00
Adam Williamson
4323d8f533 Fix desktop_update_graphical for recent F27+ GNOME change
The default action on the reboot confirmation dialog changed
from Reboot to Cancel, so when we hit enter, we just cancel the
reboot. Tweak this to hit tab on F27+ (but not <F26, so update
tests continue to work too).
2017-10-19 12:46:40 -07:00
Adam Williamson
859f4acf4c Fix install_mirrorlist_graphical test
anaconda's interface changed, so we need to adapt to that.
2017-10-16 14:23:56 -07:00
Adam Williamson
97d4c8b85c Install dejavu fonts for Cockpit tests
The font Firefox uses when we don't ensure dejavu is installed
seems to bounce around a bit, so let's ensure the dejavu fonts
are there before we start Firefox. Also update a needle for
this.
2017-10-10 16:52:02 -07:00
Adam Williamson
14c1bae747 Disable SELinux in _support_server
tgtd is failing to start due to an SELinux denial; I'm filing a
bug now.
2017-09-16 17:04:32 -07:00
Adam Williamson
99a203b8e0 Catch RHBZ #1314991 and handle it, with a soft failure
We can deal with this annoying bug by looking out for the error
we see when it happens, hitting the 'refresh' button again, and
resetting the loop counter to 1 (requires changing the loop to
a C-style loop).
2017-09-06 14:19:37 -07:00
Guy Menanteau
5fcfb353b3 Revert "iscsi backing store not same disk for PowerPC"
previously required on f25 host with qemu 2.7.1-6
it is not needed anymore on f26 with qemu 2.9.0-5

This reverts commit 0eb15266117aae47f663297f5f332d480d8549b9.
2017-09-06 08:43:04 +02:00
Michel Normand
e6fe996613 Revert "Modify "install_multi" test for PowerPC"
This reverts commit 8b2977f1d618316ded61420df4fc7d2afd07cbf4.

The initial commit was required for PowerPC
until qemu 2.7.1-6 (in f25) not required anymore
since qemu 2.9.0-5 (in f26)
2017-09-06 08:43:04 +02:00
Michel Normand
f3878c4380 Change checking nfs mount for PowerPC
by direct grep of mount command
because nfs mounting not traced in ananconda or packaging log.

Signed-off-by: Michel Normand <normand@linux.vnet.ibm.com>
2017-09-06 08:43:04 +02:00
Michel Normand
2fceef0eb3 Add "PPC PReP Boot" filesystem for PowerPC blivet tests
This is required because anaconda is still checking for it
even if not mandatory. Already tracked by bug
https://bugzilla.redhat.com/show_bug.cgi?id=1172791

Signed-off-by: Michel Normand <normand@linux.vnet.ibm.com>
2017-09-06 08:43:04 +02:00
Guy Menanteau
4b77b71771 iscsi backing store not same disk for PowerPC
Signed-off-by: Guy Menanteau <menantea@linux.vnet.ibm.com>
2017-09-06 08:43:04 +02:00
Michel Normand
47b2855d55 Add --refresh to force dnf synchronisation
because started qcow2 may be obsolete for update repo.

Note: despite deprecated "update" alias,
continue to use it rather than "upgrade" command.

Signed-off-by: Michel Normand <normand@linux.vnet.ibm.com>
2017-09-06 08:43:04 +02:00
Michel Normand
9aa83cffd1 support nfsvers=x parameter as nfs option
Required to validate bypass for pending bug
https://bugzilla.redhat.com/show_bug.cgi?id=1386059

Signed-off-by: Michel Normand <normand@linux.vnet.ibm.com>
2017-09-06 08:43:04 +02:00
Michel Normand
c086b3c0c1 rpcbind not same path for PowerPC
A temporary patch on top of workaround RHBZ #1402427
(I do not have access to it)

Signed-off-by: Michel Normand <normand@linux.vnet.ibm.com>
2017-09-06 08:43:04 +02:00
Guy Menanteau
9c8a4949fa Modify "install_multi" test for PowerPC
PowerPC arches have the empty disk automatically
mounted on the second position in anaconda (vdb).
Thus, trig installation on second disk.
Change disk checking to point on correct disk.

Warning: this is a workaround specific correction
addressing a specific case.
This will have to be improved/changed with a more
generic code as suggested by Adam Williamson in
https://pagure.io/fedora-qa/os-autoinst-distri-fedora/pull-request/1#comment-31858
proposal for a next commit :)

Signed-off-by: Guy Menanteau <menantea@linux.vnet.ibm.com>
Signed-off-by: Michel Normand <normand@linux.vnet.ibm.com>
2017-09-06 08:43:04 +02:00
Guy Menanteau
06839422e5 First PowerPC specific changes (not templates)
* New OFW variable to identify Open Firmware (used by PowerPC)
* Few needles changes for PowerPC support
* as requested do not change the timers value below for PowerPC
  tests/install_source_graphical.pm (300 to 600)
  tests/_boot_to_anaconda.pm (300 to 1200)
  This will be handled by TIMEOUT_SCALE in templates

Signed-off-by: Guy Menanteau <menantea@linux.vnet.ibm.com>
2017-09-06 08:43:04 +02:00
Adam Williamson
004121a38d Upload all logs when AVCs and/or crashes detected
Previously we just got a screenshot of some AVCs or coredumps,
which told us something was wrong but didn't really help debug
it. So, let's upload the output of the commands and then also
use the post-fail hook to upload the system logs, which should
give us much more info to work with.
2017-09-05 09:35:59 -07:00
Adam Williamson
60913a20e0 Always upload logs at end of domain_controller test
We often want to see the logs from the FreeIPA deployment test
even if that test passes - to look for some detail that doesn't
cause a test to fail, for instance, or if one of the *client*
tests failed for a reason that involves the server. So, let's
do that.
2017-09-05 09:32:33 -07:00
Michel Normand
53f0acea50 Add allowerasing for upgrade if failed without it
Note:
The dnf.log and dnf.rpm.log are already uploaded
to help for investigation.

Signed-off-by: Michel Normand <normand@linux.vnet.ibm.com>
2017-09-05 13:38:14 +02:00
Adam Williamson
a7a6219276 Re-confirm passphrase when working around #1444225
When we hit #1444225, as well as hitting Done a second time, we
have to confirm the passphrase again to work around it.
2017-07-26 15:05:55 -07:00
Adam Williamson
81ebcd882c Install libglvnd-egl when installing base-x, X seems to need it
On Rawhide, X fails to start unless this is installed. Should
probably be in the base-x group, I'll file a bug or add it.
2017-07-24 18:58:31 -07:00
Adam Williamson
6443d364b9 Fix a typo in the RHBZ#1444225 workaround 2017-07-10 15:15:41 -07:00
Adam Williamson
9a7d51f841 Tweak _graphical_input to work with transparent top bar
At least, I hope this will work.
2017-07-10 14:41:57 -07:00
Adam Williamson
e61cfa699c Try a different workaround for RHBZ #1444225
That other one didn't help, so let's try this - try and spot if
the spoke is in the unexpected state (the needle should only
match if the spoke is done processing and still in warning
state, it shouldn't match while the needle is still thinking)
and click through it again if so.
2017-07-10 12:24:41 -07:00
Adam Williamson
aca7de2861 Change up 'clean desktop' check again (use a util function)
Well, that OCR needle isn't working out so great, as it seems
to match when it shouldn't:

https://openqa.fedoraproject.org/tests/119217#step/_graphical_wait_login/5

So let's try another approach. Ditch the OCR needle and have a
function for checking we're at a clean desktop. It does the
normal needle match, but if we're on GNOME, it also tries
hitting alt+f1 and seeing if we're at the overview; if so, it
hits alt+f1 again (to go back to the desktop) and returns.
2017-07-10 11:47:07 -07:00
Adam Williamson
ddb6f8e416 Adjust cockpit log needles for Rawhide change
The Logs screen changed a bit. Also, it seems wrong to have
'notices' in all these needle names, so let's fix that.
2017-07-10 10:35:27 -07:00
Adam Williamson
d119e1e53e Check for NFS repo message in anaconda.log too
It seems to have moved there in F27.
2017-07-10 10:21:57 -07:00
Adam Williamson
ee2a3daefd Try to work around #1444225 with a sleep in disk_guided_empty
IIRC disk_guided_empty is the only storage test that clicks
through INSTALLATION DESTINATION *really* fast, so let's try
adding a 2-second sleep to it to see if it works around the
'sometimes spoke shows as incomplete' bug that cropped up in
F26 and hasn't been fixed yet and tends to cause failures.
2017-07-06 13:15:20 -07:00
Adam Williamson
09c264fe16 Add Workstation dvd-ostree flavor and tests
Summary:
As we're getting the Workstation dvd-ostree (OStree installer
image) built for Rawhide now, let's try testing it.

Test Plan:
Run the tests on a Rawhide compose that works and
has the image (e.g. 20170615.n.0). Check that new tests work
as expected and old tests are not adversely affected. A
corresponding diff for fedora_openqa will be coming to take
care of scheduling. Note that the tests will often soft fail
for now; this is intentional due to RHBZ#1193590, the bash
prompt for root is incorrect on ostree installs, so I have
added a needle that matches the incorrect prompt but which is
flagged as a workaround needle (so causing the test result to
be a soft fail).

Reviewers: jsedlak, jskladan

Reviewed By: jsedlak

Subscribers: tflink

Differential Revision: https://phab.qa.fedoraproject.org/D1211
2017-06-26 18:48:27 -07:00
Adam Williamson
a29875a2ff Upgrade tests: Run FreeIPA webUI and password change modules
Summary:
This just adds the FreeIPA web UI and password change
test modules to the FreeIPA upgrade test (client end). It's
useful to check out these features too. We don't need to
separate these into separate jobs, as we're not trying to
fill out different matrix checkboxes here, we just want to
know whether everything works.

Test Plan:
Run the test, see that the modules work properly.
I was actually expecting this to fail given the issues with
the upgrade on the server end, but it seems to pass.

Reviewers: jsedlak, jskladan

Reviewed By: jsedlak

Subscribers: tflink

Differential Revision: https://phab.qa.fedoraproject.org/D1207
2017-06-13 10:06:15 -07:00
Adam Williamson
25b910135b Workaround cursor showing up on GDM in desktop_notifications 2017-06-06 18:16:52 -07:00
Adam Williamson
df2c3cd906 Test upgrade of FreeIPA server and client deployment
Summary:
This adds an upgrade variant of the FreeIPA tests, with only
the simplest client enrolment (sssd) for now. The server test
starts from the N-1 release and deploys the domain controller
role. The client test similarly starts from the N-1 release
and, when the server is deployed, enrols as a domain client.
Then the server upgrades itself, while the client waits (as the
server is its name server). Then the client upgrades itself,
while the server does some self-checks. The server then waits
for the client to do its checks before decommissioning itself,
as usual. So, summary: *deployment* of both server and client
occurs on N-1, then both are upgraded, then the actual *checks*
occur on N.

In my testing, this all more or less works, except the role
decommission step fails. This failure seems to be a genuine one
so far as I can tell; I intend to file a bug for it soon.

Test Plan:
Run the new tests, check they work. Run the existing
FreeIPA tests (both the compose and the update variants), check
they both behave the same.

Reviewers: jsedlak, jskladan

Reviewed By: jsedlak

Subscribers: tflink

Differential Revision: https://phab.qa.fedoraproject.org/D1204
2017-06-02 12:17:07 -07:00
Jan Sedlák
48b99a2291 add base system logging test
Differential Revision: https://phab.qa.fedoraproject.org/D1202
2017-06-01 11:06:04 +02:00
Jan Sedlák
4114406668 add UEFI for blivet tests
Differential Revision: https://phab.qa.fedoraproject.org/D1201
2017-05-22 09:26:58 +02:00
Jan Sedlák
0b5f865c8f add custom btrfs partitioning test for blivet-gui
Differential Revision: https://phab.qa.fedoraproject.org/D1194
2017-05-19 13:58:16 +02:00
Jan Sedlák
ea5296b306 add postinstalls to custom partitioning tests
Differential Revision: https://phab.qa.fedoraproject.org/D1192
2017-05-09 09:11:34 +02:00
Adam Williamson
c2b1ea4e8c Use test3 not test1 for FreeIPA password change test module
Previously this module changed test1's password, so that it
would still be able to work even if the webui test module
failed (so test3/test4 didn't get created). But this means
that, for about 30 seconds, test1's password is 'loremipsum'
not 'batterystaple', and if one of the *other* client test
jobs happens to hit a point where it has to auth as test1
during the 30 seconds test1's password is different, it will
fail. This looks to be what happened to the join_kickstart
test the last few days - it failed because it tried to login
as test1 during the password change window.

By using the test3 user instead (which is only used by the
join_cockpit test, currently) we avoid this problem, at the
cost that the password_change module will always fail if the
webui module fails.
2017-05-04 17:29:35 -07:00
Adam Williamson
7d07147b7e Work around F26 base image bug: g-i-s failing to run
With the latest F26 base images, it seems like g-i-s fails to
run at first login. This is clearly some kind of bug somewhere
and I'll investigate it, but it shouldn't be causing the update
tests to fail - we can still validly run the tests with g-i-s
not running. So for now, adjust the _graphical_wait_login test
to tolerate this behaviour when running update tests.
2017-04-24 18:49:25 -07:00
Jan Sedlák
140c5f0a42 Add custom partitioning tests for blivet
Differential Revision: https://phab.qa.fedoraproject.org/D1188
2017-04-24 14:23:35 +02:00
Adam Williamson
e6076e14b8 Workaround issues with loading same module more than once
Summary:
Loading the same module more than once *kinda* works, but it
shows up all kinds of funky in the openQA web interface. There's
a POO for this:
https://progress.opensuse.org/issues/10514
But it doesn't seem like it's going to be resolved immediately,
so in the mean time maybe we should avoid doing it so we don't
have to deal with the weirdness it produces in the web UI. So
here's a kinda icky hack that uses symlinks and stuff to load
multiple instances of 'the same' test module.

Test Plan:
Run an update test, look at how it looks in the web
UI and confirm it's a lot clearer and less buggy. Check there
aren't any bugs in the loading approach. This is deployed on stg
so you can look at it there.

Reviewers: jsedlak, jskladan

Reviewed By: jsedlak

Subscribers: tflink

Differential Revision: https://phab.qa.fedoraproject.org/D1186
2017-04-18 10:38:16 -07:00
Adam Williamson
9da72de4ed Hide mouse on login screen (BGO #781294)
GDM 3.24.1 dumps a cursor in the middle of the login screen even
if you didn't move the mouse yet, so let's workaround that.
2017-04-13 15:28:37 -07:00
Adam Williamson
ee283b1411 Tweak _console_avc_crash so we get soft failures
We do want soft failures (but not hard) when _console_avc_crash
fails. So let's tweak it to ensure that's what happens.
2017-04-10 15:44:13 -07:00
Adam Williamson
e68e113f76 Remove test_flags comments, add ignore_failure flag
It's not really a good idea to have the comments that explain
the test_flags in *every* test, because they can go stale and
then we either have to live with them being old or update them
all. Like, now. So let's just take 'em all out. There's always
a reference in the openQA and os-autoinst docs, and those get
updated faster.

More importantly, add the new `ignore_failure` flag to relevant
tests - all the tests that don't have the 'important' or
'fatal' flag at present. Upstream killed the 'important' flag
(making all tests 'important' by default), I got it replaced
with the 'ignore_failure' flag, we now need to explicitly mark
all modules we want the 'ignore_failure' behaviour for.
2017-04-10 15:00:10 -07:00
Adam Williamson
5b4ce80487 Tweak needle matches at start of _do_install_and_reboot
The way this was set up before, if `anaconda_main_hub` matched
immediately but some spoke was still in a 'processing' state,
it only had 30 seconds (default `assert_and_click` timeout) to
complete and allow the 'Begin Installation' button to appear.
It seems unnecessary to match on *both* needles, really, so
let's just give 300 seconds for the `begin_installation` needle
to appear. It's not going to appear on any other screen.

This problem caused a couple of spurious failures today -
https://openqa.fedoraproject.org/tests/77839 and
https://openqa.fedoraproject.org/tests/77858 - because they
took a bit too long for the INSTALLATION DESTINATION spoke to
clear.
2017-04-07 11:31:34 -07:00
Adam Williamson
1444c5030b Add tests with no user created during install
Summary:
This adds a new test suite, run for Workstation and KDE live
images, which does not create a user during install. It then
expects initial-setup (KDE) or gnome-initial-setup (Workstation)
to appear after install, creates a user, and proceeds with
normal boot.

Note the ARM image test already covers the initial-setup text
mode, and the ARM minimal image is the only case where that
actually matters (it's not included in Server).

Test Plan:
Run the new tests, check they work. Run all old
tests, check the changes didn't break them.

Reviewers: jsedlak, jskladan

Reviewed By: jsedlak

Subscribers: tflink

Differential Revision: https://phab.qa.fedoraproject.org/D1185
2017-04-05 09:43:26 -07:00
Adam Williamson
4ea5129312 Goddamnit, perl, and your goddamn brackets 2017-03-29 15:09:25 -07:00
Adam Williamson
4f014d4f11 Use tty2 not tty3 for freeipa_client login tests
There's an issue where the follow-on _advisory_post test tries
to log in before the 'login failed' error has cleared. We can
easily avoid this by using tty2 for the login tests, then
_advisory_post will switch to tty3 for its stuff.
2017-03-29 14:33:25 -07:00
Adam Williamson
b6d4fd7d4c Don't create user when USER_LOGIN is false, but for KDE install
Summary:
For some reason, we have `USER_LOGIN` set to 'false' for the KDE
package set install test. I really don't know / remember why
that would be; I'd think we should create a user and log in as
that user to make sure it works properly when installing KDE
from the traditional installer. It's not strictly part of the
package set test, true, but still, seems worth doing.

Also, when `USER_LOGIN` is set to 'false' and the installer runs,
we create a user called 'false'. This doesn't seem like what we
wanted, so let's not do that. I dunno if there are any other
cases besides the KDE one that this commit changes, but still.

Test Plan:
Run the full test suite and look for weirdness, check
KDE package set test works as intended (now creates a user called
'test' and logs in as that user).

Reviewers: jsedlak, jskladan

Reviewed By: jsedlak

Subscribers: tflink

Differential Revision: https://phab.qa.fedoraproject.org/D1182
2017-03-29 09:30:16 -07:00
Adam Williamson
cc2d53b4ed Wait a bit after input switch before doing Japanese typing 2017-03-20 11:32:50 -07:00
Adam Williamson
ba3a5152c1 Improve FreeIPA debug logging a bit
Committing without review as this is pretty trivial and I've
had it on staging for the last few days without issue. Just gets
us somewhat better info for debugging FreeIPA issues.
2017-03-16 12:36:33 -07:00
Adam Williamson
1b26f7c04c Add missing 'use utils' to disk_guided_encrypted_postinstall 2017-03-09 13:57:06 -08:00
Adam Williamson
a581d20f8e Desktop update: Wait before clicking update apply button
We used to do this only for KDE, but I've seen the new update
tests sometimes fail at this point for no apparent reason, and
I'm thinking a wait may help (in case they're clicking the
button before it's really 'ready').
2017-03-02 15:33:23 -08:00
Adam Williamson
8d55935b41 Bump up the ARM boot timeout
Let's see if it can get a bit further, at least...
2017-03-02 15:16:02 -08:00
Adam Williamson
e9fda18016 Explicit timeout for the 'top' in _collect_data
This keeps failing because the default `assert_script_run`
timeout changed from 90 to 30 in the last os-autoinst update
(an unintended consequence of a change I made). This has been
fixed upstream, but in the mean time, let's just set an
explicit timeout on the call.
2017-03-02 14:47:44 -08:00
Adam Williamson
602a1ee101 Call boot_decrypt properly (function not method)
Noticed in e.g. https://openqa.fedoraproject.org/tests/58798
we're doing this wrong, `boot_decrypt` was moved into utils as
a function, but we were still calling it as a method...
2017-03-02 14:32:17 -08:00
Adam Williamson
9aed11a77b Work around RHBZ #1400293 (breaks IPA tests on Fedora 24) 2017-02-28 11:46:21 -08:00
Adam Williamson
461f3a6132 Update testing: log packages in update and installed packages
Summary:
This adds some logging related to the update testing workflow,
so we have some idea what we actually tested. We log precisely
which packages were actually downloaded from the update - this
is important as updates can be edited and when examining results
we'll want to know which packages actually got used. We also
add a new module which runs at the end of postinstall and tries
to figure out which packages from the update were installed in
the course of the test. This still isn't a guarantee the test
actually *tested them* in any way, but it at least means they
got installed successfully and didn't interfere with the test.

Test Plan:
Run the update test workflow, check the logs get
uploaded and seem accurate (sometimes some RPM garbage messages
wind up in the package log, I'm not too worried about that at
present). Run the compose test workflow and check it didn't
break.

Reviewers: jsedlak

Reviewed By: jsedlak

Subscribers: tflink

Differential Revision: https://phab.qa.fedoraproject.org/D1149
2017-02-23 14:51:19 -08:00
Adam Williamson
92d588f245 Add support for testing updates
Summary:
This adds an entirely new workflow for testing distribution
updates. The `ADVISORY` variable is introduced: when set,
`main.pm` will load an early post-install test that sets up
a repository containing the packages from the specified update,
runs `dnf -y update`, and reboots. A new templates file is
added, `templates-updates`, which adds two new flavors called
`updates-server` and `updates-workstation`, each containing
job templates for appropriate post-install tests. Scheduler is
expected to post `ADVISORY=(update ID) HDD_1=(base image)
FLAVOR=updates-(server|workstation)`, where (base image) is one
of the stable release base disk images produced by `createhdds`
and usually used for upgrade testing. This will result in the
appropriate job templates being loaded.

We rejig postinstall test loading and static network config a
bit so that this works for both the 'compose' and 'updates' test
flows: we have to ensure we bring up networking for the tap
tests before we try and install the updates, but still allow
later adjustment of the configuration. We take advantage of the
openQA feature that was added a few months back to run the same
module multiple times, so the `_advisory_update` module can
reboot after installing the updates and the modules that take
care of bootloader, encryption and login get run again. This
looks slightly wacky in the web UI, though - it doesn't show the
later runs of each module.

We also use the recently added feature to specify `+HDD_1` in
the test suites which use a disk image uploaded by an earlier
post-install test, so the test suite value will take priority
over the value POSTed by the scheduler for those tests, and we
will use the uploaded disk image (and not the clean base image
POSTed by the scheduler) for those tests.

My intent here is to enhance the scheduler, adding a consumer
which listens out for critpath updates, and runs this test flow
for each one, then reports the results to ResultsDB where Bodhi
could query and display them. We could also add a list of other
packages to have one or both sets of update tests run on it, I
guess.

Test Plan:
Try a post something like:
HDD_1=disk_f25_server_3_x86_64.img DISTRI=fedora VERSION=25
FLAVOR=updates-server ARCH=x86_64 BUILD=FEDORA-2017-376ae2b92c
ADVISORY=FEDORA-2017-376ae2b92c CURRREL=25 PREVREL=24

Pick an appropriate `ADVISORY` (ideally, one containing some
packages which might actually be involved in the tests), and
matching `FLAVOR` and `HDD_1`. The appropriate tests should run,
a repo with the update packages should be created and enabled
(and dnf update run), and the tests should work properly. Also
test a regular compose run to make sure I didn't break anything.

Reviewers: jskladan, jsedlak

Reviewed By: jsedlak

Subscribers: tflink

Differential Revision: https://phab.qa.fedoraproject.org/D1143
2017-02-22 11:33:32 -08:00
Adam Williamson
186678e98b Make log upload work when installed system hits emergency mode
Summary:
This is to handle cases like #1414904 , where the system boots
to emergency mode. We really need logs to try and debug this.

Test Plan:
Force a test to hit emergency mode somehow (right now
you can just run base_services_start on Rawhide over and over
until you hit #1414904, but there's probably an easier way to
do it, I think there's a systemd boot arg to tell it which target
to boot for e.g.) and check logs get uploaded. Also check this
doesn't break log upload for a 'normal' failure.

Reviewers: garretraziel_but_actually_jsedlak_who_uses_stupid_nicknames

Reviewed By: garretraziel_but_actually_jsedlak_who_uses_stupid_nicknames

Subscribers: tflink

Differential Revision: https://phab.qa.fedoraproject.org/D1103
2017-02-01 12:30:21 +01:00
Adam Williamson
b67f604894 Move all remaining utility functions into exporter modules
Summary:
This adds a couple of new exporter modules, renames main_common
to utils (this is a better name: openSUSE's main_common is
functions used in main.pm, utils is what they call their module
full of miscellaneous commonly-used functions), and moves a
bunch of utility functions that were previously needlessly
implemented as instance methods in base classes into the
exporter modules. That means we can get rid of all the annoying
$self-> syntax for calling them.

We get rid of `fedorabase` entirely, as it's no longer useful
for anything. Other base classes keep the 'standard' methods
(like `post_fail_hook`) and methods which actually need to be
methods (like `root_console`, whose behaviour is different in
anacondatest and installedtest).

Test Plan:
Do a full test suite run and check everything lines
up. There should be no functional differences from before at all,
this is just a re-org.

Reviewers: jskladan, garretraziel_but_actually_jsedlak_who_uses_stupid_nicknames

Reviewed By: garretraziel_but_actually_jsedlak_who_uses_stupid_nicknames

Subscribers: tflink

Differential Revision: https://phab.qa.fedoraproject.org/D1080
2017-01-17 23:15:44 -08:00
Adam Williamson
a091185af1 Handle 'non-ASCII in password' warning for Russian installs
Summary:
Since 26.17, anaconda shows a warning when the user password
contains non-ASCII characters, and requires a second Done click
to confirm. This change should handle that.

On the 'catch cases where password typing went wrong and re-try'
bit: to keep that, but not re-type the password *every single
time* on the Russian install test, we'd have to make the needle
match the text of the warning. This is problematic because then
that needle will be able to break without us easily noticing;
that's why I wanted to keep the 'warning bar' needle text-free.
Unfortunately, that means we have to skip the protection for
switched-layout installs.

Note the protection was actually not working for any non-English
install anyhow, because the needle had `LANGUAGE-english` as a
tag. We never noticed that. Failed password typing is pretty
rare now, so we can live without the protection - it's just nice
to have it for the English install tests because there's so many
of them.

Test Plan:
Run the Russian install with a recent Rawhide image,
check it clicks 'Done' twice. Note, it will still fail, because
of RHBZ #1413813.

Reviewers: jskladan, garretraziel_but_actually_jsedlak_who_uses_stupid_nicknames

Reviewed By: garretraziel_but_actually_jsedlak_who_uses_stupid_nicknames

Subscribers: tflink

Differential Revision: https://phab.qa.fedoraproject.org/D1084
2017-01-17 23:13:47 -08:00
Adam Williamson
062d9f8f5e Add jobs to gather memory usage data
Summary:
This adds a new test, memory_check, which just does a default
package set install with `inst.debug` parameter then uploads
the memory usage file (`/tmp/memory.dat`) at the end. We can
have check-compose use the data to analyze changes in memory
usage over time.

Test Plan:
Fire off the Workstation network install image tests
and make sure the memory usage test runs and works on all three
machines. This is live on staging already.

Reviewers: jskladan, garretraziel_but_actually_jsedlak_who_uses_stupid_nicknames

Reviewed By: garretraziel_but_actually_jsedlak_who_uses_stupid_nicknames

Subscribers: tflink

Differential Revision: https://phab.qa.fedoraproject.org/D1082
2017-01-16 09:30:14 -08:00
Adam Williamson
460a6e5439 Simplify the services test fix
I forgot `script_run` actually returns the exit code. That makes
it easier.
2017-01-11 16:03:09 -08:00
Adam Williamson
c9ede993e5 Fix base_services_start
The old version wasn't working - it was passing even though two
services fail to start in Workstation currently. I'm really not
sure why the old approach wasn't working, but it wasn't, and I
rather hate `script_output` anyway, so here's a different way
of doing it which relies on `eval`ing `assert_script_output`
instead. (I really should send a PR for a non-fatal version of
assert_script_output...)
2017-01-11 13:49:17 -08:00
Adam Williamson
dac3c2dd38 Use systemctl --no-legend when collecting service info
Without this, when there are failed services, we get an extra
column to the left of the service names with a unicode dot for
each failed service, which is awkward and screws up the parsing.
2017-01-11 11:58:37 -08:00
Adam Williamson
c4f32ab5ad add an Asian (Japanese) language install test
Summary:
Include some basic testing of Japanese input, and split the
input testing (including Russian) into a separate module, since
it's not really part of 'login' testing.

Test Plan:
Run the test, and the Russian and French tests too to
make sure they didn't break. Tested on staging. Note the Japanese
test soft fails, intentionally, at present, as I discovered a bug
while working on it:
https://bugzilla.gnome.org/show_bug.cgi?id=776189

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D1072
2016-12-21 08:41:00 -08:00
Adam Williamson
7135a4e05d iSCSI: test user authentication
Summary:
This isn't in the criteria, but it's commonly used, so we ought
to test this way. Require authentication for the iSCSI target
and have the test provide the appropriate auth info.

Test Plan:
Run the iscsi test and check it works (you need the
recent fixes for support_server to make *that* work). Nothing
else should be affected.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D1070
2016-12-20 08:34:30 -08:00
Adam Williamson
0fa6138448 Have non-English tests do graphical install and login
Summary:
The non-English tests so far did not test that graphical login
worked as expected, which is a fairly large hole. With this
change, they should do a Workstation install and test login to
both GNOME and the console works as expected. KDE is not yet
tested.

As part of this we tweak the implementation of keyboard layout
switching in graphical environments to use a generic function
in main_common which can handle both anaconda and desktops
(just GNOME at present, but should extend easily to any desktop
with a known switcher key and a visible layout indicator),
replacing the anacondatest class method. I kinda don't like that
the test has to specifically tell the function when it's in
anaconda, but I don't think I want to start experimenting with
a global 'test phase' openQA variable or anything like that at
present.

Fixes T842.

Test Plan:
Run the French and Russian install tests and check
they work as expected. Also run an English Workstation install
if you like, and make sure that didn't break. This change is
live on staging ATM, seems to work fine.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Maniphest Tasks: T842

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D1071
2016-12-16 09:40:29 -08:00