Commit Graph

737 Commits

Author SHA1 Message Date
Lukas Ruzicka
d82f4ecd7f Develop a new test for system updates via Cockpit.
* server-cockpit-updates tests that Cockpit can be used to update the system.
* server-cockpit-autoupdate tests that users can use dnf-automatic for system
updates.
* cockpit functions were removed from utils.pm and put into an extra library
for cockpit - cockpit.pm which all cockpit tests are now using.

Review cockpit.pm

Review autoupdate test.

Review the update test.

Fix typo in cockpit.pm

Add sleep.

Add missing command.

Delete an unused needle.
2019-10-01 16:04:01 +02:00
Adam Williamson
8be0541ce7 Tweak live_build mock handling to work with templates
mock is changing the config files to be based on templates,
which breaks how we generate the live build mock config. This
should work with both the old and new config files.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-09-23 16:00:08 -07:00
Adam Williamson
c0f911067f Disable prerelease text check in disk_guided_empty
This is causing all kinds of trouble, because when the test is
run on the Server DVD - with the 'orange to blue' gradient - the
prerelease note is dark red text on a dark blue background.
os-autoinst actually reduces the color depth of images/needles
and greyscales them before performing the match...but for this
dark red text on dark blue background, the result seems to be
that the text and background come out *the same grey*, so *any*
text will match the needle (even if it's completely different
text), as will *no text at all*. I've tried finessing around
this a few times but it just keeps happening, so for now I'm
just disabling the pre-release text check at this point. We still
have the check during _boot_to_anaconda, when the text appears
on a *grey* background and so isn't a problem. I'm not removing
the needles yet, until we hear back from upstream:

https://progress.opensuse.org/issues/56822

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-09-11 15:01:54 -07:00
Lukas Ruzicka
46bc71afd6 Delete the KDE test for Release Notes due to retirement. 2019-09-05 11:41:18 +02:00
Adam Williamson
8b236592d9 Disable SELinux for live image build test
Building images with SELinux in permissive mode worked fine in
F29 and F30, but it seems to be broken in F31. Releng seems to
have it set up so the build environment has SELinux disabled,
so let's do the same. Will file a bug for this, as bcl says it
should be considered a bug.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-08-29 17:49:27 -07:00
Lukas Ruzicka
022865ab9f KDE app start/stop: launch apps with commands not menus
This is doing slightly less to exercise the launcher menus and see
whether icons appear or disappear, but it's much faster and more
reliable. We do still use menu launch for one app, just to check
the mechanism works in general.
2019-08-23 12:41:49 -07:00
Adam Williamson
bf5691e69d Fix and simplify fedora_release test
Also simplify os_release a bit thanks to the improved spell_
version_number.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-08-21 13:09:40 -07:00
Adam Williamson
2f51f3ef20 Add PXE boot test
This adds a whole wodge of stuff to support_server to make it
act as a PXE server, then adds a new test which boots from PXE
and so should hit the PXE server. We use the NFS install repo as
that can be relied on to work for a support_server install.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-08-20 18:17:57 -07:00
Adam Williamson
a177a63ec1 Factor out s/metalink/mirrorlist/ into a util function
We do this in quite a few places and need to do it in another,
so let's just have a function for it. It takes a file glob
so we can have it run on a different one for _live_build.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-08-20 14:16:23 -07:00
Adam Williamson
d4206ee732 Drop one old regex bit from install source check
Not needed now F28 is EOL.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-08-20 12:06:33 -07:00
Adam Williamson
d71daa1fe4 Do the metalink->mirrorlist dodge in live_build mock config
This avoids hitting slow metadata downloads in mock env.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-08-15 13:36:25 -07:00
Adam Williamson
cac08c170f Remove RHBZ #1704488 workaround (was fixed a while back)
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-08-15 11:45:16 -07:00
Adam Williamson
cf5704eb3d Goddamn boilerplate...
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-08-14 14:43:23 -07:00
Adam Williamson
7739bdbbd8 D'oh - check_desktop_clean is in utils...
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-08-14 14:42:18 -07:00
Adam Williamson
469a1395d0 Revise desktop background checks
Split this out of install_default, because it really is not a
part of that test and we do not want that test to fail because
the desktop background is wrong. Make it its own test module
and test suite instead. Don't do it on Rawhide, because we
really can't assert anything worthwhile about Rawhide at the
moment at least (this means the test runs but is a no-op and
will always pass on Rawhide, unfortunately). Move the needles
to a more appropriate location (this has nothing to do with
anaconda) and use 'background' not 'wallpaper' naming (that's
the name we use elsewhere in the project, e.g. package names).
Also, run the test on updates, and add an F29 needle for this
purpose.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-08-14 14:30:25 -07:00
Lukas Ruzicka
d06b30e8dc Add new testcases and methods for release identification. 2019-07-30 15:45:02 -07:00
Adam Williamson
c4c998c5a8 Actually set longer timeout before services entry click
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-07-25 14:25:11 -07:00
Adam Williamson
cf709af13c cockpit: retry the 'services entry' click if it fails
cockpit test often seems to fail because the click on a services
entry to open the services detail screen just gets lost. Let's
wait longer before trying it, and retry once if it fails.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-07-25 14:17:33 -07:00
Adam Williamson
be12f07900 GNOME apps: dismiss new LibreOffice 'tip of the day' if it shows
Also add a new needle for Impress startup.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-07-23 20:15:24 -07:00
Adam Williamson
19268619fd Adapt _do_install_and_reboot for Rawhide user/root spoke change
The user and root spokes were moved from during-install to pre-
install hub in Rawhide. This should cope with that, while still
working for older images.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-07-23 12:14:03 -07:00
Adam Williamson
317aadeb2b potential_conflict: still need to call the interpreter
script isn't executable as downloaded, this is easier than chmod.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-07-22 15:00:25 -07:00
Adam Williamson
5a10c537f7 Update potential_conflict invocation for Python 3
The script got ported. Yay.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-07-22 14:54:36 -07:00
Lukáš Růžička
2db77367f9 Add new action to the install test. 2019-07-19 14:22:49 +02:00
Adam Williamson
25f408ea7e Factor out clicking of update and akonadi notifications in KDE
There are three places where we basically want to click away
pop-up update notifications and the buggy akonadi_migration_agent
notification if it's there, in KDE tests. Let's share this code
between them, and also let's record soft failures for the buggy
cases in the desktop_notifications test.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-07-17 09:02:24 -07:00
Adam Williamson
eba6818f5b KDE app start/stop: also get rid of akonadi notification
...because it screws with the test. See
https://bugzilla.redhat.com/show_bug.cgi?id=1716005

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-07-16 15:53:02 -07:00
Adam Williamson
4cc51a51a0 Dismiss multiple update notifications in app start/stop test too
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-07-16 14:40:10 -07:00
Adam Williamson
43417f925c Whoops...actually decrease the counter
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-07-16 14:05:00 -07:00
Adam Williamson
34321dae8f desktop_notifications: handle multiple KDE update notifications
Current KDE seems to like showing us multiple update available
notifications. So the test must dismiss all of them. See:
https://bugzilla.redhat.com/show_bug.cgi?id=1730482

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-07-16 13:58:15 -07:00
Lukas Ruzicka
1475813b75 Update the modularity tests.
* Add test to check module defaults.
* Add whitelist download.
* Fix install test to include selected profile to be on the safe side.
* Add test into templates.
2019-07-15 13:19:06 +02:00
Adam Williamson
dfce3e522d desktop_update: handle multiple KDE update available notifications
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-07-10 12:37:26 -07:00
Adam Williamson
9078bca871 cockpit: update needles for 198, rejig Services page handling
Cockpit 198 comes with a UI overhaul, so almost all needles
need an update.

The 'auditd' service is no longer on the first page. To make
this less fragile (at the cost of not testing that clicking on
a service actually opens the detail page *for that service*,
tweak the needles to just look for *any* running service, click
on it, and check we got to a 'details' page. We also redo the
existing needles for this design.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-07-10 11:43:03 -07:00
Adam Williamson
2ecf97e7a7 Revert "desktop_browser: remove the new tab workaround"
This reverts commit a0c6d1c88e.
Sadly we're still getting problems opening new tabs, even though
I thought the os-autoinst change should solve it.
2019-07-08 10:25:46 -07:00
Adam Williamson
a0c6d1c88e desktop_browser: remove the new tab workaround
I'm pretty sure I worked around the modifier bug in os-autoinst,
so this shouldn't be necessary any more. See:
https://bugzilla.redhat.com/show_bug.cgi?id=1727388

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-07-05 16:42:58 -07:00
Adam Williamson
78dd42b876 Automate two 'mediakit' tests (repoclosure and file conflicts)
We can run these tests much like support_server - have a test
which boots from the support_server disk image but with the ISO
to be tested attached, then mounts it and runs the scripts.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-07-04 13:44:20 -07:00
Adam Williamson
de4a58438b Use existing needle for dismissing KDE update notification
@lruzicka added this to the KDE app tests in 71c4e273, but there
was no need for a new needle as I'd already done the same thing
in the desktop updates tests; let's just use the same needle.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-07-03 08:39:01 -07:00
Lukas Ruzicka
71c4e273ef Click off updates notification if it appears. 2019-07-03 10:52:47 +02:00
Lukas Ruzicka
d538959011 Update modules to test. 2019-07-03 10:42:33 +02:00
Adam Williamson
5bb9f26b5f Try and fix root password setting for ostree installs
This is kind of a pain. I hope this works...

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-06-28 14:09:55 -07:00
Adam Williamson
604a476295 Don't setenforce before chpasswd for root
This shouldn't be needed any more (the referenced bug was fixed
in F28) and doesn't work in non-live installer.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-06-28 14:09:55 -07:00
Adam Williamson
c0bad87836 Revamp post-install Doin' Stuff
This whole block where we do various things at the console after
install completes was becoming a real mess. I had secret hopes
of killing it entirely at some point, but...that doesn't look
like it's gonna happen this century. So let's make it better
instead. The conditionals were getting very nested and icky and
it was hard to see what was actually going on. This rationalizes
things so first we figure out all the things we might want to do
at a console, then if we don't have anything to do at a console
we go ahead and hit the reboot button; otherwise we go to the
console and do all the things we need to do, including rebooting
unless this is the memory check test.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-06-28 14:09:55 -07:00
Lukas Ruzicka
9e3b16bfcf Change Settings test due to change functionality. 2019-06-21 14:54:18 +02:00
Adam Williamson
7c78f9a710 Don't try bypass_1691487 for graphical base_selinux runs
This just doesn't work when running the test on a graphical
install because we're at a desktop when we try and do it, not
a text console. So let's only try to do it if there is no
DESKTOP. See e.g.:

https://openqa.fedoraproject.org/tests/408484#step/base_selinux/2

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-06-03 15:32:25 -07:00
Adam Williamson
7953fb91d1 Try to handle spurious 'auth required' on KDE update test too
Sigh, now *this* is showing up in KDE too. Fun.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-05-30 17:54:27 -07:00
Adam Williamson
2e56facb68 Try to handle changes to KDE update notifications in Rawhide
The way KDE does update notifications has changed - it's now a
permanent pop-up notification. This is a bit awkward for our
logic; it's hard to define a needle that proves this pop-up is
the only notification. Instead, let's dismiss it, then open the
notification tray and assert that there aren't any others. But
we also retain the old behaviour (more or less) for testing old
releases.

The popup notification also blocks the 'refresh' needle in the
systray and so breaks the desktop update test, so we deal with
that too.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-05-30 17:34:09 -07:00
Adam Williamson
43b990195c Add a wrapper around assert_and_click to handle signature change
So, turns out new os-autoinst does *not* still accept the old
argument style for assert_and_click...and old os-autoinst
doesn't accept the new one. This adds a wrapper that handles
both, so our tests can work with old and new os-autoinst. We can
drop this once both deployments are on newer os-autoinst.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-05-28 09:41:33 -07:00
Adam Williamson
87456f6b1d Use new arg style for assert_and_click
In the new os-autoinst I just sent to staging, the old style
doesn't work any more, breaks all tests. This style should also
work with the older os-autoinst on stable.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-05-27 14:28:17 -07:00
Adam Williamson
66917e791d Fix the check from the last commit
D'oh.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-05-22 15:27:59 -07:00
Adam Williamson
e1ffd8aabd Fix text install test for 'use password' now being default
'use password' now seems to be default when creating a user in
text mode, so we need to handle that.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-05-22 15:18:08 -07:00
Adam Williamson
0d38d3f292 FreeIPA: get verbose logs from BIND
This was necessary for debugging the FreeIPA 4.8 pre-release
update bug, so let's have it for all runs, just in case.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-05-01 18:59:52 -07:00
Adam Williamson
6c22b55be5 Drop another part of branched buildroot repo workaround
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-05-01 13:49:18 -07:00
Adam Williamson
f4c3c4c2cd Workaround #1704488 in the live image build test
Odd bug which breaks mock until you reboot or restart dbus-broker.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-04-29 16:24:19 -07:00
Adam Williamson
2d709a99ad Remove #1699099 workaround
We got a compose with the fix included, so this should no longer
be needed.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-04-18 07:59:29 -07:00
Adam Williamson
ccb67bd5a7 Extend gnome-software/issues/582 workaround to 2 auth requests
It seems that when this problem happens now we get *two* auth
requests, so we need to handle that.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-04-17 10:32:04 -07:00
Adam Williamson
d0d99946ce Exclude fix for #1699099 from the workaround...
...so we can check it works.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-04-16 22:49:49 -07:00
Michel Normand
37e2f134f1 Bypass dup chars after snapshot brc#1691487
eg failing jobs:
https://openqa.stg.fedoraproject.org/tests/499908#step/_boot_to_anaconda/7
https://openqa.stg.fedoraproject.org/tests/519326#step/_post_network_static/4

Signed-off-by: Michel Normand <normand@linux.vnet.ibm.com>
2019-04-15 16:20:13 +02:00
Adam Williamson
a9c6e69689 Make the 1699099 workaround not assert_script_run
File doesn't exist for ostree installs. No point failing if this
fails, we may as well just continue and see what happens.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-04-13 08:40:57 -07:00
Adam Williamson
33dd7ded6b Sigh fix a syntax error in a previous fix I am bad at this
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-04-13 08:02:52 -07:00
Adam Williamson
d1cb937ea3 Load correct keyboard layout before doing 1699099 workaround
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-04-13 08:00:23 -07:00
Adam Williamson
2e68250bb1 Fix a logic bug in the 1699099 workaround which broke lots
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-04-13 07:56:57 -07:00
Adam Williamson
785eaffb3a Add a workaround for #1699099 so update tests don't fail on it
No purpose is being served by all update tests failing on this
bug, so let's try and work around it.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-04-12 18:55:17 -07:00
Adam Williamson
33f63c3d95 rescue_shell_prompt is really just a root console needle...
...so make it look like all the other ones we have already.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-04-12 15:54:26 -07:00
Guy Menanteau
33fd05f3d0 New install_repository_nfsiso_variation
This is covering the  NFSISO_variation test as per:
https://fedoraproject.org/wiki/QA:Testcase_install_repository_NFSISO_variation

Change support_server to use support_5 disk image in templates
supposed to be previously created by updated createhdds tool
required for new test.

Signed-off-by: Guy Menanteau <menantea@linux.vnet.ibm.com>
2019-04-12 15:49:54 -07:00
Guy Menanteau
29070a54f1 New install_repository_hd_variation
use a new PREINSTALL to execute a rescue script before install.

This is covering the  Hard_drive_variation test as per:
https://fedoraproject.org/wiki/QA:Testcase_install_repository_Hard_drive_variation

Signed-off-by: Guy Menanteau <menantea@linux.vnet.ibm.com>
2019-04-12 15:49:54 -07:00
Michel Normand
4a00301725 new select_rescue_mode in lib/utils from rescue_mode_encrypted.pm
placed in lib/utils.pm to be used by another commit.

Signed-off-by: Michel Normand <normand@linux.vnet.ibm.com>
2019-04-12 15:49:54 -07:00
Adam Williamson
6bd24c3caa Set up FreeIPA reverse DNS on F28, where we still use rolekit
Otherwise the replica tests don't work.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-04-10 13:16:56 -07:00
Adam Williamson
dc81c64f4f Use type_very_safely for the desktop auth workaround
type_safely just ain't safe enough, it seems.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-04-10 12:46:45 -07:00
Lukas Ruzicka
99f93945b3 Change KDE desktop background at start of KDE app test
Since the KDE menus have transparency set, any time the wallpaper
changes the menus will look different, and the app tests may
fail. This sets the desktop wallpaper to black at the start of
the test suite to avoid this problem.
2019-04-09 08:57:43 -07:00
Adam Williamson
794c245786 Restore #1594402 / #1661288 workarounds (aarch64 console)
This reverts commit f40599ee15,
with a few tweaks. It turns out we still need these workarounds
- see https://bugzilla.redhat.com/show_bug.cgi?id=1661288 .
2019-03-29 15:22:47 -07:00
Adam Williamson
b1a0fd7dc3 Add tests for QA:Testcase Remote Logging test case
This commit adds server and client tests to cover the remote
logging Server test case.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-03-28 12:52:54 -07:00
Lukas Ruzicka
662f89f9d4 Fix Firefox failure. 2019-03-28 11:50:41 +01:00
Lukas Ruzicka
f9201fb7d8 Fix kmail and kontakt tests
Fix to pass kontakt tests properly.
2019-03-27 18:00:05 +01:00
Lukas Ruzicka
7dd43daab8 Fix Krusader
It seems that there was a lag in button drawing, which prevented
OpenQA from fully recognizing the krusader_welcome needle because
it was evaluated too soon.
I added "wait_still_screen 3" to make sure, there is time for a
button to be drawn correctly before assertion.
2019-03-27 11:30:01 +01:00
Lukas Ruzicka
ff4ceb55ff Enable a switch for systemwide ABRT during installation.
Change method from assert_script_run to script_run

Change to reboot from console.

Move code in submethod, use ifs to make sure everything runs.

Fix chroot rebooting problem.
2019-03-26 20:10:00 +00:00
Adam Williamson
e396341f31 Handle 'authentication required' appearing on Boxes startup
In the last few weeks Boxes is showing an auth dialog on start.
I've filed a bug on this; let's have the test handle it as a
soft failure, since this isn't a fatal problem. Do this by
making an existing needle for this dialog a bit more generic
in name and using it.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-03-26 12:30:47 -07:00
Adam Williamson
5a1a71c67f Use the existing 'oh hai this is firefox' needle for app tests
The KDE and GNOME 'apps' tests for Firefox both invented their
own 'oh look Firefox is running' needle, even though we already
had one. The GNOME one was broken by the removal of the app
title bar in Firefox 66.

Instead of having three needles for the same thing, let's just
throw the 'apps' ones out and use the pre-existing one from
needles/firefox for all cases.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-03-26 12:23:45 -07:00
Lukas Ruzicka
9ae60de422 Reneedle menus and fix some issues in tests.
This update fixes the strange behaviour in KDE which causes wrong clicking.
With new logic, this is not an issue anymore.
2019-03-20 10:26:33 +01:00
Lukas Ruzicka
426b3447da Delete tests for non-existing menu items. 2019-03-19 12:14:52 -07:00
Lukas Ruzicka
c9a25cb2a6 Create an application start-stop test for KDE.
Add png files as a background for the needles.

Rewrite the test handling methods to make them ready for KDE

Add the ABRT startstop test.

Make changes and corrections to the abrt test.

Add test for Firewall.

Add test for DNF dragora.

Add wait time for dnfdragora test.

Correct syntax.

Add Language test.

Make some changes to the DNF dragora test.

Add Users test case.

Add needles for DBUS viewer test.

Add Dbusviewer test.

Add Mahjong test and needles for games.

Add Minesweeper tests.

Add Patience test.

Add test for Document Viewer.

add test for gwenview

Add test for koulourpaint.

Add test for Kruler

Add test for Kcolorchooser

Reneedle failing needle.

Add ktorrent tests.

Add tests for CPT editors.

Add test for Krfb

Fix names for those files.

Add test for Kget

Add Akregator test.

Add test for Konversation.

Make Konversation end really.

Add tests for Kmail

Add test for PIM exporter.

Add test for KTnef and Krdc.

Fix problems after test runs.

Make more tries.

Fix needle to be found better.

Fix more errors.

Add test for Falkon.

Add tests for browsers.

Add support for closing tabs into Firefox test.

Add tests for K3B

move needles to correct directory

Add Kaddressbook test.

Add Kontakt text.

Add test for korganizer.

Add menu office needle and correct konqueror needle.

Add test for calligra stage.

Add test for Calligra.

Add test for network connections.

Modify needle for kaddressbook to prevent failing.

Add test for system settings and fix others.

Add test for FMW.

Add test for Dolphin

Add test for Infocenter.

Add test for kparted.

Fix a wrong needle.

Test relnotes.

Fix some errors in tests.

Add test for Discover.

Add test for Ksysguard.

Add tests for Konsole.

Add tests for KDE wallet.

Add tests for several utilities.

Add Krusader test.

Finish utility tests.

Fix some errors.

Fix needle for spectacle.

Add wait time to let Dragora wait for network.
2019-03-19 12:14:47 -07:00
Adam Williamson
1ed87382a9 abrt back under utilities, drop sundry
As of Fedora-30-20190316.n.1, gnome-abrt is back under the
Utilities menu and the 'Sundry' menu seems to have disappeared.
Change the test back to using utilities and drop the sundry
needle.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-03-19 10:11:32 -07:00
Michel Normand
ff3768b1d0 grub2-install parm set to /dev/vda1 for PowerPC
Signed-off-by: Michel Normand <normand@linux.vnet.ibm.com>
2019-03-12 19:41:20 +00:00
Adam Williamson
b777094b7b Give live image build command another 10 minutes
40 seems not long enough, it's timing out while the build is
still running sometimes.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-03-06 19:53:31 -08:00
Adam Williamson
d6c339e9ff Disable the 'f30-build' repo for the desktop update test
...GNOME Software doesn't like it.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-03-06 19:51:09 -08:00
Adam Williamson
f3caa1fc53 Bump timeout for install_lang_selected to appear a bit
It seems 3 secs was a bit tight for recent Branched and Rawhide,
test was failing when the screen was just a bit slow to update
for some reason.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-03-04 17:32:25 -08:00
Adam Williamson
450f843389 desktop_browser: try both button and ctrl-t for opening new tab
Sigh. It seems sometimes the button isn't visible because GNOME
pops up a fricking notification that blocks it.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-03-01 09:13:54 -08:00
Adam Williamson
03a56eb0da Avoid spurious repo check error for universal tests on netinst
As there is *always* a base repo error when booting a netinst
image, this check was giving us spurious failures when running
universal tests on a netinst (which we are currently doing on
ppc64 it seems). Tweak it a bit to hopefully avoid this.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-03-01 08:49:08 -08:00
Adam Williamson
6d2031dbfd desktop_browser: fix wait_still_screen, drop wait_idle
wait_idle was deprecated and effectively it just sleeps, so
replace these wait_idles with explicit sleeps - I can't think
of a better choice here as we just need to wait for Firefox to
calm down after various operations.

The wait_still_screens were always hitting the 30 second time
out because of the flashing cursor; setting the similarity level
required to lower than the default 47 should fix this.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-02-13 09:51:34 -08:00
Adam Williamson
965a1a6546 Click instead of using ctrl-t for new browser tab
ctrl-t seems to be failing often in Firefox tests, and I can't
figure out why. Let's try clicking the 'new tab' button instead
as a workaround.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-02-13 09:35:32 -08:00
Adam Williamson
99302c6fd4 Add a live image build test for updates
Just like the installer image build test, only...it builds a live
image. This involves reimplementing quite a chunk of the Koji
livemedia task. Ah, well. Also involves rethinking the flavor
names a bit here, these seem...better.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-02-07 18:28:24 -08:00
Adam Williamson
5cbc7df810 Work around GGO gnome-software/582 till it's fixed
We're getting a spurious "Authentication required" for a
background refresh of the system repos sometimes, depending on
whether it happens while we're at a tty. This is now accepted
by upstream, so let's handle it and make it a soft failure for
now.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-02-07 09:58:10 -08:00
Adam Williamson
8d1d150798 Handle running update tests on Koji tasks
We quite often want to run the update tests on a Koji task (not
a Bodhi update) for some reason - usually to test a potential
fix for an issue, or at a maintainer's request to test a change
before it is merged upstream and officially sent out as an
update. Up till now I've always hacked up utils.pm on the
staging server by hand to do this, which is horrible. Together
with a commit to fedora_openqa, this should allow us to do it in
a nice, sane way via the CLI. It's mostly just tweaking the
"updates" repo setup in utils.pm as you'd expect, but there's a
bit of subtlety to it because of the installer tests that use
%ADVISORY% as a variable substitution in the disk image name;
you can't do something like `%ADVISORY or KOJITASK%`, sadly, so
I had to have almost-redundant variables ADVISORY, KOJITASK and
ADVISORY_OR_TASK (we could kinda just live with ADVISORY_OR_TASK
except I didn't want to drop ADVISORY as it's an unnecessary
change from previous behavior).

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-01-29 11:54:15 +01:00
Lukas Ruzicka
c97c23d485 Recreate needles to make them check different area.
This commit adjusts the area that needles look for to avoid checking
based on a menu item that has disappeared from the system.
2019-01-28 11:11:36 +01:00
Adam Williamson
7aed615a8d Only do the addrepourl check if we're on F28+
inst.addrepo was added in F29. In F28 it should just sorta not
do anything at all. It's not worth doing a big workaround to
make F28 graphically add the repo in this case; let's just allow
F28 to not actually add the repo. It's fine! This is fine.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-01-26 20:49:59 +01:00
Adam Williamson
d1006a38e5 Have update installer test install the update packages (#89)
In https://bugzilla.redhat.com/show_bug.cgi?id=1669256 it became
obvious that there's a missing feature in the new installer test
for updates: the update is both used in the image build process
and built into the installer environment itself, but it is not
actually included in the installed package set. This can be a
problem if the update has a bug that manifests *only* at install
time if it is in the install transaction (which is exactly the
case there), because the test will not catch this, and nor will
any other test.

So this commit makes `support_server` set up the update repo and
serve it out via NFS when it's run in an update context, and
makes the actual update install test run parallel with it and
use that repository. This way the install should include the
package(s) from the update. (It also of course means the test
fails if an update breaks NFS or something like that, but hey,
we want to know that!)

A parallel commit for fedora_openqa is necessary to add the
required CURRREL setting for the updates-installer flavor.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-01-26 19:14:04 +01:00
Adam Williamson
12affb145f Add update tests to build and test a netinst image
This adds a test which builds a netinst image potentially with
the package(s) from the update, and uploads that image. It also
adds a test which runs a default install using that image. This
is intended to check whether the update breaks the creation or
use of install images; particularly this will let us test
anaconda etc. updates. We also update the minimal disk image
name, as we have to make it bigger to accommodate this test,
and making it bigger changes its name - the actual change to
the disk image itself is in createhdds. We also have to redo a
bunch of installer needles for F28 fonts, after I removed them
a month or so back...

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-01-18 08:24:44 -08:00
Adam Williamson
510662c1ec Drop all RHBZ #1663040 workarounds, should be fixed now
In Fedora-Rawhide-20190117.n.0 #1663040 should be fixed, so drop
all workarounds for it.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-01-17 08:35:30 -08:00
Adam Williamson
ec762117d0 Wait an extra 2 minutes for the installer 'beta nag' screen
There seems to be an issue in Rawhide ATM which can cause the
'beta nag' screen to take a very long time to appear. Bump the
timeout to avoid tests failing on this.

https://bugzilla.redhat.com/show_bug.cgi?id=1666112

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-01-14 15:22:01 -08:00
Lukas Ruzicka
39d3427471 Create a test suite for application start stop testing.
Merges #86
Fixes #85
2019-01-04 15:23:03 -08:00
Adam Williamson
c6fac00698 More fixing of the workaround...
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-01-04 13:55:31 -08:00
Adam Williamson
42a38009eb Fix workaround (have to wait for desktop to appear)
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-01-04 13:27:01 -08:00
Adam Williamson
4e537684b1 Work around #1663040 in Workstation live installs
We don't want the tests to fail on this now we know what the
bug is, really - we want to find if there are any subsequent
fails, and allow the post-install tests to run also. So, let's
make it a soft failure.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-01-04 13:01:36 -08:00
Adam Williamson
2b29ecbc18 FreeIPA client: set longer timeout for login(1) (#1661273)
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-12-20 09:02:36 -08:00
Adam Williamson
3d35ec9670 Update a comment to not mention a removed workaround
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-12-17 14:36:10 -08:00
Adam Williamson
00d1fdbd2c Drop #1600823 workaround
Bug was fixed months ago.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-12-17 14:30:09 -08:00
Adam Williamson
f40599ee15 Drop all #1594402 workarounds
I'm pretty sure we got all the bugs this was working around
fixed. Again, if not, we can put this back!

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-12-17 14:27:18 -08:00
Adam Williamson
01fd92cb34 Drop #1444225 workaround
The bug never got explicitly addressed, but it's very old and
anaconda has been substantially changed since. The workaround
sometimes triggers erroneously now (because the icon sometimes
goes black while the spoke is still in 'Checking storage
configuration...' state), which is awkward. I can't be 100% sure
the bug doesn't sometimes still happen, but if it does, we'll
notice fairly soon, and we can tweak this and put it back.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-12-17 14:18:05 -08:00
Adam Williamson
d6de57c6de Drop RHBZ#1618928 workaround
Bug was fixed back in August.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-12-17 14:10:58 -08:00
Adam Williamson
7cbd859d81 Remove another old workaround (for #1606541)
This was fixed long ago as well...

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-12-17 13:36:13 -08:00
Adam Williamson
e04ee00e19 Drop various old workarounds from _support_server
I tested a run with these commented out, it worked fine. Seems
the bugs are all fixed now.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-12-17 13:06:19 -08:00
Adam Williamson
ef3555aaa1 Drop another old workaround (KDE package signature check)
It seems Rawhide auto-signing is working fine now: openQA claims
this needle has 'never' been matched (which really means 'not
for a long time'), and I can't find any test fail in the last
year which looks like it landed on this 'authenticate' screen
but the needle failed to match, or anything like that.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-12-17 12:24:09 -08:00
Adam Williamson
1fd0097d1d Simplify an 'if >F26' thing
We don't care about anything older than 27 any more.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-12-17 12:19:44 -08:00
Adam Williamson
d1e7b89efd Fix a potential race in desktop update test
https://openqa.stg.fedoraproject.org/tests/424393 is a failure
where the 'Download' [updates] button was already visible when
we went to the tab. We already checked whether an 'apply' button
is visible and skipped the 'refresh' click if so, but because
the 'download' button is a new thing, we weren't skipping the
'refresh' click if 'download' was already visible.

So in this case, even though we could already see 'download', we
went ahead and clicked 'refresh'...then *immediately* started
looking for 'download'. It seems that Software did not refresh
and remove the 'Download' button *immediately* when we pressed
'refresh' - it left the 'Download' button visible briefly, and
*in this brief window*, we clicked it. *Then* Software kinda
'noticed' we'd clicked 'Update', and it seems it just sort of
throws away our click on 'Download' at that point and does the
refresh.

So at that point, the test thinks it's clicked 'Download' and
expects to see 'Apply', but actually the 'Download' click got
more or less thrown away, so the test fails, sitting at the
'Download' button.

To solve this, let's just extend the existing check to skip the
'refresh' click if 'download' *or* 'apply' are already visible.

There is a sort of possibility here that we could wind up
downloading and installing some updates that existed and were
noticed *before* we did our python3-kickstart trick, but not
install the python3-kickstart update, and cause the test to fail
because of that, but that doesn't seem to have happened before
when we were seeing the 'update' button, so I think I'm not
going to borrow trouble. If it happens, we'll deal with it I
guess.

The comment talks only about KDE, but clearly it can be the case
that an automatic check makes the button visible on GNOME too,
so let's rewrite the comment too.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-12-17 12:10:06 -08:00
Adam Williamson
517750443e Remove RHBZ #1638563 workaround
The fix for this bug was sent to all releases now, so we should
not need the workaround any more. Let's kill it.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-12-17 12:02:57 -08:00
Adam Williamson
0498f4db94 Use 30s timeout on the check_screen in the workaround
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-12-13 15:29:35 -08:00
Adam Williamson
b3f51e6108 Gah come back missing bracket
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-12-13 15:23:50 -08:00
Adam Williamson
1f7e5ebe20 Attempt a workaround for RHBZ#1659266
This is breaking the memory_check tests. I just reproduced it
manually and the UI *does* come back to life if you wait some
time; let's see if we can work around the bug this way.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-12-13 15:15:49 -08:00
Adam Williamson
12e103e3da Factor meat out of advisory_post and do it in postfail too
If an update test fails before reaching advisory_post, we don't
generate the 'what update packages were installed' and 'were
any update packages *not* installed when they should have been'
logs, but these may well be useful for diagnosing the failure -
so let's also do the same stuff there. Only let's not do it all
twice.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-12-12 22:17:29 -08:00
Adam Williamson
764c6dbd95 Notice when update package should have been installed but wasn't
We hit an interesting case in update testing recently:

https://bodhi.fedoraproject.org/updates/FEDORA-2018-115068f60e

An earlier version of that update failed testing. When we dug
into it a bit, we found that the test was failing because an
earlier version of the `pki-server` package was installed than
the version that was in the update; when asked (as part of
FreeIPA deployment) to install it, dnf had noticed that there
were dependency issues with the version of the package from the
update, but it happened to be able to install the version from
the frozen 'stable' repo...so it just went ahead and did that.

In this case, the 'missed' package resulted in a test failure,
but it'd actually be possible for this to happen and the test
to complete; we really ought to notice when this happens, and
treat it as a test failure.

So what this attempts to do is: at the end of all update tests,
check for all installed packages with the same name as a package
from the update, and compare their full NEVR to the one of the
package from the update. If a package with the same name as one
of the update packages is installed, but does not appear to be
the *same NEVR*, we fail, and upload the lists of packages for
manual investigation as to what the heck's going on.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-12-12 22:17:29 -08:00
Adam Williamson
e76ad4bbc1 Crank sss debuglevel up to 9, and also do it for cockpit test
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-11-30 14:58:53 -08:00
Adam Williamson
14ad5b97f1 Still try and upload testedpkgs even if comm 'fails'
From local experimentation, it still actually produces the
output, even though it prints the message about the order being
wrong and exits 1.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-11-08 17:46:45 -08:00
Adam Williamson
ddf6ba5a6b update tests: don't fail if comm is unhappy about the alphabet
Weirdly, occasionally some update tests seem to fail because
the 'comm' util we use to produce the list of packages from the
update that were actually tested during the job doesn't think
one of the input files is in alphabetical order, even though we
sort them both when they're produced. I don't know if this is
possibly due to the definition of 'alphabetical order' changing
as part of the update, or what. But we really shouldn't *fail*
the test when this happens, as it's not part of the functional
test, we're just producing convenience data. So, let's handle
the command failing, and if it happens, upload the input files
so we can maybe figure out why it's unhappy...

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-11-08 16:59:06 -08:00
Adam Williamson
70ef3404f0 Tweak previous commit to avoid some bugs
The previous commit would lead to the 'workaround' getting hit
incorrectly, and might have had some other issues...tweak it a
bit.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-10-31 12:45:11 -07:00
Adam Williamson
fd753b2e3a Handle split of 'download' and 'apply' phases in gnome-software
GNOME Software 3.30.5 split the offline update process into two
separate 'download' and 'apply' phases. So we need to handle
clicking 'download' before 'apply', if that happens.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-10-31 11:50:14 -07:00
Adam Williamson
e6c8c5f0ff Work around Firefox 'close multiple tabs' warning
Somehow, recently, FreeIPA tests are running into Firefox not
quitting because it's showing a warning about closing multiple
tabs. (I think we didn't *get* multiple tabs before but now we
do, for some reason). So let's work around this by clicking
"Close tabs" if the warning appears.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-10-30 18:34:37 -07:00
Adam Williamson
129d1316a6 Fix tty switching for desktop_notifications
Lately, we can't be sure the desktop will be on tty1 after we
do 'systemctl isolate graphical.target'. For recent Workstation
lives it actually shows up on tty2.

We could be 'clever' and switch to tty2 on F29+ Workstation
lives...but actually it seems like if we just don't do anything,
systemd switches us to the correct tty. So let's rely on that,
at least as long as it's working.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-10-12 21:16:33 -07:00
Adam Williamson
22c0b5bc04 Disable hidden grub menu for uploaded base installs
At least one test (desktop_notifications_postinstall) boots from
the disk image uploaded by install_default_upload, and needs to
access the grub menu. On F29+ Workstation this is failing,
because the grub menu is now hidden by default, so when the test
boots, it never sees the bootloader screen, and fails.

I considered trying to teach it to hold down shift or hit f8 or
esc at the right time, but that seems like it might be hard. So
instead let's just try to disable the hidden menu when we're
about to upload the installed system image. This is kinda going
against the 'preserve natural system behaviour' principle we try
to use for openQA, but I think it's OK as we do have other tests
that will exercise the 'hidden boot menu' stuff to some extent.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-10-12 15:36:55 -07:00
Adam Williamson
17b6d9f708 Tweak the workaround to work for F27 too
On F27 we don't get a 'Software is up to date' screen because
there's an upgrade available. Let's work with the refresh button
instead.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-10-11 22:23:53 -07:00
Adam Williamson
63d8f34a0e Tweak the workaround loop a bit, refresh the comments
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-10-11 16:18:04 -07:00
Adam Williamson
db4ab638da Restore modified version of the #1314991 workaround for #1638563
We're not seeing *exactly* #1314991 any more, but we're seeing
something that looks quite similar: the first attempt to find
updates just doesn't find any. No error message, no updates. I
have reported a bug for this and am investigating it, in the
meantime, let's restore the workaround, elaborated a bit, and
looking for the 'Software is up to date' screen instead of the
error message.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-10-11 16:07:58 -07:00
Adam Williamson
25ad8a6aeb Drop workaround for #1314991, it doesn't work any more
I rather suspect the *bug* is still basically present and it's
why this test often fails, but we no longer seem to see the
*error message* which lets us detect the bug happening. This
needle has not been hit by any test for six months. So let's
remove the workaround as it adds complexity.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-10-10 13:45:51 -07:00
Adam Williamson
9869920f5b Use longer timeout for root console switch after liveinst
For some reason, in recent tests, switching to a console after
live install completes is taking a long time, and tests are
failing because we 'only' allow 10 seconds for the login prompt
to appear. This seems to indicate some kind of performance bug,
but we don't really want all liveinst tests to fail on in, this
is not primarily a performance testing framework. So let's
tweak the root_console / console_login bits a bit to allow a
configurable timeout for the login prompt to appear, and use
that to wait 30 secs instead of 10 in this case.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-10-06 08:44:34 -07:00
Adam Williamson
6e262be28b Add a check that FreeIPA is actually up after upgrade
The FreeIPA upgrade test didn't actually check that FreeIPA is
actually running after the upgrade and reboot, it just kinda
assumed it is. Let's add a check to the start of the 'check'
test module that makes sure ipa.service actually comes up to
'active' state. This'll make it clearer when tests are failing
because FreeIPA didn't come up right after the upgrade. The
check will run on non-upgrade tests too, but that's fine.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-10-05 23:53:17 -07:00
Adam Williamson
0bf76db7d5 Add a test of bootchain stuff for updates
This adds a new test intended to just check boot chain things
for updates. It doesn't run any test modules besides the stock
update ones, but sets a variable, ADVISORY_BOOT_TEST, which
causes _advisory_update to do some additional stuff after
installing the updates but before rebooting: it forces regen
of the initramfs and bootloader config, and reinstalls the
bootloader on BIOS (not UEFI as it's not relevant). If the
following boot fails, we probably have a bug somewhere.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-10-04 16:43:15 -07:00
Adam Williamson
d2d6bfa695 Try something different for screen corruption after IPA uninstall
There's this annoying problem where the screen sometimes goes
messed up after ipa-server-uninstall. 'clear' doesn't seem to
really work to fix it up either. Let's try flipping between
ttys. I don't like this much as it's already a pain trying to
work out / remember what tty we might possibly be on at any
given time, but I think we're always on either 1 or 3 here, so
let's do ctrl-alt-f1 ctrl-alt-f3 to ensure at least one change
and wind up on tty3...

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-09-28 18:09:58 -07:00
Adam Williamson
c2fb886d6f Add another wait to avoid a transition animation in anaconda
I swear, these transitions drive me nuts.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-09-28 17:46:36 -07:00
Adam Williamson
a49f328dc6 Tweak how update-upgrade tests are handled a bit
Looking at this, it's a bit weird: the updated packages are
actually included in the upgrade process, but we still run
_advisory_update, which does basically nothing...then reboots.
That's kinda silly and makes the tests a bit flaky, let's fix
it. I don't think there's actually any problem with doing the
upload of updatepkgs.txt in _repo_setup_updates, becase that
already guards against being run more than once, it just bails
very early if it's already been run.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-09-28 17:23:51 -07:00
Lukas Ruzicka
24e68aa8a2 Create openqa tests to test modularity. 2018-09-26 23:09:36 -07:00
Adam Williamson
923267574d Drop the workaround for #1625572
It's fixed everywhere now, and the workaround can misfire if
the first g-i-s is slow quitting (see
https://openqa.fedoraproject.org/tests/280482)
2018-09-16 08:39:30 -07:00
Adam Williamson
f334df1337 Bump ipa-replica-install timeout a bit (it takes a long time)
I'm going to figure out if it's a bug that it takes so long, but
for now let's just bump the timeout.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-09-07 15:56:11 -07:00
Adam Williamson
e2eb794a87 Tweak the workaround yet again (with soft fail this time)
Sigh, sorry, just perfecting. This way it won't fail when the
bug is fixed (hopefully).

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-09-06 20:36:29 -07:00
Adam Williamson
7dff1843db Tweak that workaround again (forgot the version check)
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-09-06 19:53:07 -07:00
Adam Williamson
211cc221b3 Rejig the 1625572 workaround to just use an existing conditional
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-09-06 19:00:49 -07:00
Adam Williamson
af6b9b15aa Gah, fix version testing in previous commit
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-09-06 18:23:13 -07:00
Adam Williamson
5a48086e61 Don't expect GDM when doing GNOME no-user install on F28+
GNOME now transitions straight from the g-i-s 'user creation'
mode to a logged-in desktop.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-09-06 18:21:12 -07:00
Adam Williamson
e3887c5a83 Work around RHBZ#1625572 (both g-i-s modes running)
RHBZ#1625572 is for gnome-initial-setup running in 'first login'
mode after it's already run in 'user creation' mode (which isn't
meant to happen). This works around that so the subsequent tests
can run. We don't soft-fail because meh.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-09-06 17:59:40 -07:00
Adam Williamson
c62a04bac8 Remove old workaround for g-i-s failing to run on F26
This hasn't been a problem for ages.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-09-06 17:49:03 -07:00
Adam Williamson
cc2f1a3cec _graphical_wait_login: drop an old FAW workaround
This bug was fixed long ago, we no longer need the workaround.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-09-06 15:02:20 -07:00
Adam Williamson
64c5070b06 Simplify _graphical_wait_login by dropping a huge conditional
If USER_LOGIN is false we can just return; when we reach the
login screen. We don't need a huge conditional when we don't do
anything *after* it, in the false case.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-09-06 15:01:08 -07:00
Adam Williamson
cfe2a33038 Have domain controller upload logs *before* decommissioning
It transpires that decommissioning wipes some stuff, like the
dirsrv logs. Obviously we want these included in the logs we
upload for reference purposes, so let's upload earlier.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-08-24 14:42:52 -07:00
Adam Williamson
bdd26a09ee Have kickstart tests handle RHBZ#1618928 too
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-08-19 10:03:27 -04:00
Adam Williamson
9df64398ee Add a FreeIPA replication job set
This adds a set of jobs to test FreeIPA replication. We deploy
a server, deploy a replica of that server, then enrol a client
against the replica and run the client tests.

At first I was planning to add the replica testing into the
main set of FreeIPA tests, but the test ordering/blocking (via
mutexes and barriers and what-have-you) just turns into a big
nightmare that way. This way seems rather simpler to deal with.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-08-02 09:59:40 -07:00
Adam Williamson
35caea44bc Revert "Try downgrading softhsm as a workaround for #1607635"
This reverts commit 0289716d70.
Turns out the downgrade doesn't avoid the crash. :(
2018-07-30 11:58:12 -07:00
Adam Williamson
0289716d70 Try downgrading softhsm as a workaround for #1607635
We'd really like to know if FreeIPA is working aside from this
crasher bug, so let's workaround it.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-07-30 11:33:02 -07:00
Adam Williamson
acc4ccd7cc Add a sleep to desktop_update_graphical
Try and avoid failure to launch alt-f1 dialog...

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-07-30 11:23:59 -07:00
Adam Williamson
a479774026 Correct previous commit
Should be if, not unless...damn exit codes.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-07-23 14:08:37 -07:00
Adam Williamson
3817e7128d Work around RHBZ #1606541 (on Rawhide, not updates)
We kind of want to know if FreeIPA is working aside from this
known bug, so let's treat it as a soft failure and work around
it. But only for Rawhide, not for F27/F28 updates tests.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-07-23 11:12:57 -07:00
Adam Williamson
7621cd7e57 Move the new base_services_start check to the start
...otherwise we actually return before it, if no services fail.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-07-13 14:02:31 -07:00
Adam Williamson
b4cd1b4a9e Check for services deleted to break loops in base_services_start
https://bugzilla.redhat.com/show_bug.cgi?id=1600823 shows a
case where systemd throws a service that would usually have been
started out of the boot process *entirely* in order to resolve a
dependency loop. This means the service won't show up as failed,
it will just be inactive when it should be active. This still
should constitute a failure of this test, so let's add a check
for the log message that indicates this situation.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-07-13 13:37:04 -07:00
Adam Williamson
a362ecb2a0 Add a softfail workaround for #1600823
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-07-13 00:39:53 -07:00
Adam Williamson
9ae2c249f2 Stop using rolekit for database server role on F29+
As for the domain controller role, stop using rolekit on F29+,
as it's going away. Continue using it on <F29.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-07-10 17:54:30 -07:00
Adam Williamson
5999058d07 We should *not* check CURRREL for rolekit in _check
...because by this point in the upgrade test, the system is
upgraded, and rolekit won't be there on F29+.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-07-10 16:21:54 -07:00
Adam Williamson
7e7016ea14 Convert domain controller test not to use rolekit
Rolekit is going away. At least for the F29 cycle, though, we
still want to test basically the same functionality. This ports
the 'domain controller role' test to use ipa-server-install
directly rather than rolectl.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-07-09 19:54:49 -07:00
Adam Williamson
1c1b33840f Work around an anaconda logging bug that showed up today
https://github.com/rhinstaller/anaconda/pull/1519 should fix it
on the anaconda side, till that's merged, we need this.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-06-28 15:13:06 -07:00
Adam Williamson
b81f681f02 Load us console layout in uefi_postinstall too
...since it runs during non-English tests on aarch64.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-06-25 11:28:12 -07:00
Adam Williamson
861ad5d4aa Load us layout before doing post-install aarch64 cmdline hack
It seems that for some reason the localized layout gets loaded
on the installer VTs by this point in time, so we need to load
'us' again for this complex command to work.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-06-25 09:35:56 -07:00
Adam Williamson
8e0764bc80 Brown paper bag fix: need use utils for type_safely
...sigh.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-06-24 17:21:08 -07:00
Adam Williamson
d8f5f56fff Also do 'console=tty0' workaround for rescue boot on aarch64
...sigh, another place this is needed, and it's a bit ugly
here. Ah, well.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-06-24 17:17:10 -07:00
Adam Williamson
e200e29fff Add another #1594402 comment
Just commenting why we do this again.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-06-24 17:16:53 -07:00
Adam Williamson
f30b7517ce Correct the text install workaround for aarch64
I sorta screwed up the brackets there a bit.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-06-24 17:09:19 -07:00
Adam Williamson
ab32b75aba Note bug related to 'console=tty0 quiet' workaround
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-06-22 13:51:03 -07:00
Adam Williamson
1b51987478 Add 'console=tty0' for anaconda text install on aarch64
We need this at least till #1594402 is fixed.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-06-22 13:48:36 -07:00
Adam Williamson
4d803eda52 Add 'console=tty0 quiet' to cmdline for aarch64 installs
We need this as part of the fix for #1593028, at least until
the kernel package is changed to no longer have
CONFIG_CMDLINE="console=ttyAMA0" in the config for aarch64
builds. Fully fixing the bug also requires some change to the
kernel or dracut or something.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-06-22 11:56:21 -07:00
Adam Williamson
faef957bac Revert workaround for RHBZ#1553935 now it's fixed
We don't need this any more, so let's remove the complication.
2018-06-18 11:02:08 -07:00
Adam Williamson
6324db0b87 Fix some minor syntax errors in previous commit
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-06-13 10:11:53 -07:00
Lukas Ruzicka
8897fe45a6 Add test for Arabic installation (revisited). 2018-06-13 11:56:26 +02:00
Adam Williamson
bd1b951f71 Bump a magic sleep a bit
Seems we need to wait a bit longer for this stupid transition
to happen on aarch64...

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-06-11 15:59:44 -07:00
Adam Williamson
06fbeca281 Avoid unconditional wait for Timbuktu screen in _boot_to_anaconda
The way this currently works, the test unconditionally waits 60
seconds for the "Timbuktu screen" (the warning dialog shown on
pre-release images) to appear when anaconda is starting up, even
if it's testing an image where it doesn't show up. Now we test
Atomic nightlies and live respins and stuff this happens quite a
lot, so let's avoid it. This way if the hub appears during those
60 seconds we'll spot it right away and continue, otherwise we
behave the same as before.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-05-24 14:18:21 -07:00
Adam Williamson
05c9f4fbcd Make sure all check_screen calls have explicit timeout
Upstream is gonna change the default from 30 to 0, it seems:
https://github.com/os-autoinst/os-autoinst/pull/965
so let's go ahead and change these two cases where we have no
explicit timeout to have one.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-05-24 14:17:24 -07:00
Adam Williamson
33ac181955 Use mirrorlist instead of baseurl for updates tests
The reason we have all this horrible code to use the commented-
out baseurl lines in the repo files instead of the metalinks
that are usually used is a timing issue with the metalink
system. As a protection against stale mirrors, the metalink
system sends the package manager a list of mirrors *and a list
of recent checksums for the repo metadata*. The package manager
goes out and gets the metadata from the first mirror on the
list, then checksums it; if the checksum isn't on the list of
checksums it got from mirrormanager, it assumes that means the
mirror is stale, and tries the next on the list instead.

The problem is that MM's list of checksums is currently only
updated once an hour (by a cron job). So we kept running into
a problem where, when a test ran just after one of the repos
had been regenerated, the infra mirror it's supposed to use
would be rejected because the checksum wasn't on the list - but
not because the mirror was stale, but because it was too fresh,
it had got the new packages and metadata but mirrormanager's
list of checksums hadn't been updated to include the checksum
for the latest metadata.

All this baseurl munging code was getting ridiculous, though,
what with the tests getting more complicated and errors showing
up in the actual repo files and stuff. It occurred to me that
instead of using the baseurl we can just use the 'mirrorlist'
system instead of 'metalink'. mirrorlist is the dumber, older
system which just provides the package manager a list of mirrors
and nothing else - the whole stale-mirror-detection-checksum
thing does not happen with mirrorlists, the package manager just
tries all the mirrors in order and uses the first that works.
And happily, it's very easy to convert the metalink URLs into
mirrorlist URLs, and it saves all that faffing around trying to
fix up baseurls.

Also, adjust upgrade_boot to do the s/metalink/mirrorlist/
substitution, so upgrade tests don't run into the timing issue
in the steps before the main repo_setup run is done by
upgrade_run, and adjust repo_setup_compose to sub this line out
later.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-05-18 16:41:13 -07:00
Adam Williamson
8754611eef Extend some boot timeouts in upgrade tests
Sometimes rebooting during upgrade tests seems to take longer
than these timeouts allow, so let's bump them a bit.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-05-11 09:02:21 -07:00
Adam Williamson
fdaa4783e7 _check_install_source: handle 'added repo' and 'enabled repo'
The text changed from 'added repo' to 'enabled repo' in Rawhide
after F28, so let's handle both at least till F28 is EOL.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-05-07 15:50:47 -07:00
Adam Williamson
c3a511e052 support_server: workaround RHBZ#1554390 (breaks dnsmasq on F28)
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-05-03 14:00:41 -07:00
Adam Williamson
0ab8c6bd3f Click on language select screen (RHBZ #1566066)
We've been seeing an odd case lately where the language select
screen is not foregrounded when it appears (so all text is
grey). It happens very occasionally on x86_64, but a lot on
ppc64. To work around this, let's add a needle that matches the
inactive screen, and click on the screen when it appears just
to make sure it's active.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-04-25 15:04:52 -07:00
Adam Williamson
70d3f0269c Of course, we have to actually auth *properly*...
...just clicking the button ain't gonna work. D'oh.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-03-28 22:17:33 -07:00
Adam Williamson
1f2e333043 Handle an auth dialog appearing after login on FAW
This will happen until a fedora-release PR is merged and sent
out for F28.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-03-28 21:24:31 -07:00
Adam Williamson
baa7ac4e39 Handle KDE update test when KDE has aleady found updates
We do the 'desktop update' test for KDE via the notification
icon thingy, and it behaves differently depending on whether it
has already detected there are updates or not. The test only
works at present in the case where it *hasn't* - it expects the
notification icon to be in the extended panel and it expects to
see a 'refresh' button, neither of which is the case if it's
already noticed there are updates to install.

We should also force PackageKit to update its list of available
updates after we set up our 'special' update, otherwise on this
path KDE will only install the updates it found *before* we did
our stuff, and the test will fail as our special update won't be
there.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-03-28 19:53:19 -07:00
Adam Williamson
de3f4a873c Sleep a bit less in notification tests (to fix KDE postinst)
In F28 tests, the notification 'counter' thing that we rely on
to check there's only *one* notification seems to suddenly
disappear...right around 10 minutes after the desktop starts up,
which is just how long our test idles for to catch crashes that
happen a little after boot. That causes test fails. Let's try
just cutting the wait down to 8 minutes to see if that helps.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-03-28 19:20:29 -07:00
Adam Williamson
c2a5846064 Fix KDE live notification test (dismiss network notification)
KDE in F28+ seems to show a network connection notification on
live boot, for some reason. Just dismiss it to help the test
pass.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-03-28 19:01:46 -07:00
Adam Williamson
24f2eb39d9 Check for test3's existence at start of freeipa_password_change
This test expects to pick up from freeipa_webui, but that test
is not fatal (i.e. it can fail and we still carry on to this
one). We should probably make them independent, but for now,
just check if 'test3' (one of the users freeipa_webui creates
and that this test requires) actually exists, at the start. If
not, we can just die right away.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-03-23 09:18:39 -07:00
Adam Williamson
4d997d7323 Move install of Firefox into its own milestone module
This reduces duplication, but it also means that if the FreeIPA
web UI module fails, the password change module will pick up
from a point where Firefox is set up and won't fail in a bogus
way because it isn't.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-03-23 09:10:51 -07:00
Adam Williamson
46a41d8d67 More debug logging for FreeIPA server (from ab)
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-03-21 11:34:45 -07:00
Adam Williamson
e2a8c34e86 Tweak mouse placement in _graphical_wait_login for KDE browser
This mouse placement is in the middle of where the 'install
addon' popover appears in Firefox, and that seems like it
sometimes causes the popover to immediately disappear in KDE.
This is pretty corner-case-y so I don't wanna report it as a
bug, let's just tweak the cursor hiding location and see if it
solves the problem.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-03-16 15:10:34 -07:00
Adam Williamson
7c1f59cfe8 Move UEFI postinstall to tty4 too
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-03-10 08:38:20 -08:00
Adam Williamson
a20270e430 Move collect_data and console_shutdown to tty4...
...as somehow a Workstation live install currently has the
desktop on tty3, I have no idea why (g-i-s not quitting right?)

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-03-09 22:06:18 -08:00
Adam Williamson
ed9945de71 chpasswd -R is blocked by SELinux, sigh...
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-03-09 17:06:01 -08:00
Adam Williamson
eedcec1c02 Try and fix root password setting approach
assert_script_run and chroot don't mix...let's try chpasswd.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-03-09 16:25:58 -08:00
Adam Williamson
ed62801202 Fix the conditionals for the Workstation live check
Sigh, perl. Sigh, me.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-03-09 15:47:39 -08:00
Adam Williamson
f09330e897 Work around RHBZ #1553807 by checking if anaconda still running
This is the best workaround I can think of for RHBZ #1553807 -
just check (in the 60 second 'move the mouse' loop) if anaconda
is still running, based on whether its icon is in the top bar
(on Workstation live installs only, obviously).

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-03-09 15:22:29 -08:00
Adam Williamson
71bfb896a4 Wiggle the mouse during install
Seems with the long period of not doing anything and possibly
with very aggressive timeouts in Fedora 28, Workstation live
wants to blank the screen while we're installing. Stop it.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-03-08 21:21:23 -08:00
Adam Williamson
0915b857f9 More tweaking for this damn no root password spoke situation
Previous approach wouldn't work for tests that run after the
install test...let's just set a password from a chroot after
install completes. Don't really like this as it changes the
'real' install process a bit, but it's the least invasive short
term fix at least. We can maybe do something more sudo-y later
with a bit more thought.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-03-08 20:31:14 -08:00
Adam Williamson
7c28cfc909 Adapt for no user/root panes on Workstation live install
Workstation live installs for F28+ drop the user creation and
root password panes from anaconda, so we need to not try and
use them any more. But we still want the old behaviour for F27.
I'm hoping this approach will work, if not, we'll find out soon
enough. This removes the install_no_user test for F28+ as it
will no longer differ from the install_default test.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-03-08 18:32:16 -08:00
Adam Williamson
d070ad44a5 Check default package selection is correct
This adds a check that the default package set selection is
actually correct, where possible and appropriate, as part of
the `_software_selection` test. We do this by examining the
`packaging.log` log file and checking which environment group
was selected.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-01-05 15:18:09 -08:00
Adam Williamson
ffa7ca2447 Add a check that correct filesystem was used on Server installs
This is currently broken, but openQA doesn't notice; we really
should. We could also check the default in other cases, but I
think that's less clear-cut, as it's kind of an anaconda design
choice, it's not mandated in Fedora requirements anywhere.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-01-05 15:18:05 -08:00
Adam Williamson
f70416c6a1 Use release number not "rawhide" for Rawhide upgrades (#1531356)
There's a problem with using `--releasever=rawhide` for upgrade
tests ATM - see #1531356 . To avoid this, we'll try using the
real Rawhide release number (which I'm adapting the scheduler
code to discover and pass in as `RAWREL`).

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-01-04 18:34:20 -08:00
Adam Williamson
2a7b2923f7 Handle BGO#790811 (GNOME 'getting started' crash)
There's a bug causing the 'getting started' screen to crash.
This doesn't really make the system unusable, so treating it
as a soft failure seems appropriate, especially as this will
unblock all the post-install tests on Workstation.
2017-11-24 16:36:31 -08:00
Adam Williamson
e2beb4f046 Bump up some more DNF timeouts due to #1516990
All of these seem vulnerable.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2017-11-23 16:01:58 -08:00
Adam Williamson
a0a63256d0 Whoops, do that last thing properly
We need the 60 second timeout here, not the other place.
2017-11-23 13:42:21 -08:00
Adam Williamson
f20086f59c Wait 60 seconds for 'timbuktu screen' (#1516990)
It takes an unusally long time for Modular images to get from
language selection to the 'timbuktu screen', so give 'em a bit
more time. See bug report for more info.
2017-11-23 11:44:57 -08:00
Adam Williamson
ce07c418d0 Fix (hackily) install of X, Firefox and haveged on Modular
Modular composes don't include these packages, but we need them
to run the web UI tests for FreeIPA and Cockpit. This is the
most reasonable hack I can come up with for now: just use a
non-modular fedora repo to source these packages when doing
Modular compose testing.

If we ever reach an all-Modular future, these packages should
be available in Modular composes I guess, but for now they are
not.
2017-11-07 14:20:36 -08:00
Adam Williamson
14bbfaaa3c Skip software selection on Modular until it's any use
With current Modular composes it does nothing.
2017-11-07 12:44:52 -08:00
Adam Williamson
bd27cc69d8 Don't install haveged for modular tests (yet)
It's not there, I don't think.
2017-11-06 18:01:23 -08:00
Adam Williamson
7b7b63abc1 Workaround F26 base image g-i-s issue for upgrade tests too
Since April there's been some kind of issue in the F26 base
image which means gnome-initial-setup doesn't run on the first
user login (as it should). The F25 base image is fine. I've
not yet had the time to look into this.

I put a workaround in place to prevent this problem causing
false fails of update tests that boot from the F26 base image,
but didn't apply the same workaround to upgrade tests, which
is why upgrade tests from F26 Workstation always fail - they
expect g-i-s to run on first login (which happens after the
upgrade, in upgrade tests) and it doesn't. So let's just extend
the workaround to apply to upgrade tests too, for now, until we
can figure out why this happens.
2017-10-20 12:21:31 -07:00
Adam Williamson
4323d8f533 Fix desktop_update_graphical for recent F27+ GNOME change
The default action on the reboot confirmation dialog changed
from Reboot to Cancel, so when we hit enter, we just cancel the
reboot. Tweak this to hit tab on F27+ (but not <F26, so update
tests continue to work too).
2017-10-19 12:46:40 -07:00
Adam Williamson
859f4acf4c Fix install_mirrorlist_graphical test
anaconda's interface changed, so we need to adapt to that.
2017-10-16 14:23:56 -07:00
Adam Williamson
97d4c8b85c Install dejavu fonts for Cockpit tests
The font Firefox uses when we don't ensure dejavu is installed
seems to bounce around a bit, so let's ensure the dejavu fonts
are there before we start Firefox. Also update a needle for
this.
2017-10-10 16:52:02 -07:00
Adam Williamson
14c1bae747 Disable SELinux in _support_server
tgtd is failing to start due to an SELinux denial; I'm filing a
bug now.
2017-09-16 17:04:32 -07:00
Adam Williamson
99a203b8e0 Catch RHBZ #1314991 and handle it, with a soft failure
We can deal with this annoying bug by looking out for the error
we see when it happens, hitting the 'refresh' button again, and
resetting the loop counter to 1 (requires changing the loop to
a C-style loop).
2017-09-06 14:19:37 -07:00
Guy Menanteau
5fcfb353b3 Revert "iscsi backing store not same disk for PowerPC"
previously required on f25 host with qemu 2.7.1-6
it is not needed anymore on f26 with qemu 2.9.0-5

This reverts commit 0eb15266117aae47f663297f5f332d480d8549b9.
2017-09-06 08:43:04 +02:00
Michel Normand
e6fe996613 Revert "Modify "install_multi" test for PowerPC"
This reverts commit 8b2977f1d618316ded61420df4fc7d2afd07cbf4.

The initial commit was required for PowerPC
until qemu 2.7.1-6 (in f25) not required anymore
since qemu 2.9.0-5 (in f26)
2017-09-06 08:43:04 +02:00
Michel Normand
f3878c4380 Change checking nfs mount for PowerPC
by direct grep of mount command
because nfs mounting not traced in ananconda or packaging log.

Signed-off-by: Michel Normand <normand@linux.vnet.ibm.com>
2017-09-06 08:43:04 +02:00
Michel Normand
2fceef0eb3 Add "PPC PReP Boot" filesystem for PowerPC blivet tests
This is required because anaconda is still checking for it
even if not mandatory. Already tracked by bug
https://bugzilla.redhat.com/show_bug.cgi?id=1172791

Signed-off-by: Michel Normand <normand@linux.vnet.ibm.com>
2017-09-06 08:43:04 +02:00
Guy Menanteau
4b77b71771 iscsi backing store not same disk for PowerPC
Signed-off-by: Guy Menanteau <menantea@linux.vnet.ibm.com>
2017-09-06 08:43:04 +02:00
Michel Normand
47b2855d55 Add --refresh to force dnf synchronisation
because started qcow2 may be obsolete for update repo.

Note: despite deprecated "update" alias,
continue to use it rather than "upgrade" command.

Signed-off-by: Michel Normand <normand@linux.vnet.ibm.com>
2017-09-06 08:43:04 +02:00
Michel Normand
9aa83cffd1 support nfsvers=x parameter as nfs option
Required to validate bypass for pending bug
https://bugzilla.redhat.com/show_bug.cgi?id=1386059

Signed-off-by: Michel Normand <normand@linux.vnet.ibm.com>
2017-09-06 08:43:04 +02:00
Michel Normand
c086b3c0c1 rpcbind not same path for PowerPC
A temporary patch on top of workaround RHBZ #1402427
(I do not have access to it)

Signed-off-by: Michel Normand <normand@linux.vnet.ibm.com>
2017-09-06 08:43:04 +02:00
Guy Menanteau
9c8a4949fa Modify "install_multi" test for PowerPC
PowerPC arches have the empty disk automatically
mounted on the second position in anaconda (vdb).
Thus, trig installation on second disk.
Change disk checking to point on correct disk.

Warning: this is a workaround specific correction
addressing a specific case.
This will have to be improved/changed with a more
generic code as suggested by Adam Williamson in
https://pagure.io/fedora-qa/os-autoinst-distri-fedora/pull-request/1#comment-31858
proposal for a next commit :)

Signed-off-by: Guy Menanteau <menantea@linux.vnet.ibm.com>
Signed-off-by: Michel Normand <normand@linux.vnet.ibm.com>
2017-09-06 08:43:04 +02:00
Guy Menanteau
06839422e5 First PowerPC specific changes (not templates)
* New OFW variable to identify Open Firmware (used by PowerPC)
* Few needles changes for PowerPC support
* as requested do not change the timers value below for PowerPC
  tests/install_source_graphical.pm (300 to 600)
  tests/_boot_to_anaconda.pm (300 to 1200)
  This will be handled by TIMEOUT_SCALE in templates

Signed-off-by: Guy Menanteau <menantea@linux.vnet.ibm.com>
2017-09-06 08:43:04 +02:00
Adam Williamson
004121a38d Upload all logs when AVCs and/or crashes detected
Previously we just got a screenshot of some AVCs or coredumps,
which told us something was wrong but didn't really help debug
it. So, let's upload the output of the commands and then also
use the post-fail hook to upload the system logs, which should
give us much more info to work with.
2017-09-05 09:35:59 -07:00
Adam Williamson
60913a20e0 Always upload logs at end of domain_controller test
We often want to see the logs from the FreeIPA deployment test
even if that test passes - to look for some detail that doesn't
cause a test to fail, for instance, or if one of the *client*
tests failed for a reason that involves the server. So, let's
do that.
2017-09-05 09:32:33 -07:00
Michel Normand
53f0acea50 Add allowerasing for upgrade if failed without it
Note:
The dnf.log and dnf.rpm.log are already uploaded
to help for investigation.

Signed-off-by: Michel Normand <normand@linux.vnet.ibm.com>
2017-09-05 13:38:14 +02:00
Adam Williamson
a7a6219276 Re-confirm passphrase when working around #1444225
When we hit #1444225, as well as hitting Done a second time, we
have to confirm the passphrase again to work around it.
2017-07-26 15:05:55 -07:00
Adam Williamson
81ebcd882c Install libglvnd-egl when installing base-x, X seems to need it
On Rawhide, X fails to start unless this is installed. Should
probably be in the base-x group, I'll file a bug or add it.
2017-07-24 18:58:31 -07:00
Adam Williamson
6443d364b9 Fix a typo in the RHBZ#1444225 workaround 2017-07-10 15:15:41 -07:00
Adam Williamson
9a7d51f841 Tweak _graphical_input to work with transparent top bar
At least, I hope this will work.
2017-07-10 14:41:57 -07:00
Adam Williamson
e61cfa699c Try a different workaround for RHBZ #1444225
That other one didn't help, so let's try this - try and spot if
the spoke is in the unexpected state (the needle should only
match if the spoke is done processing and still in warning
state, it shouldn't match while the needle is still thinking)
and click through it again if so.
2017-07-10 12:24:41 -07:00
Adam Williamson
aca7de2861 Change up 'clean desktop' check again (use a util function)
Well, that OCR needle isn't working out so great, as it seems
to match when it shouldn't:

https://openqa.fedoraproject.org/tests/119217#step/_graphical_wait_login/5

So let's try another approach. Ditch the OCR needle and have a
function for checking we're at a clean desktop. It does the
normal needle match, but if we're on GNOME, it also tries
hitting alt+f1 and seeing if we're at the overview; if so, it
hits alt+f1 again (to go back to the desktop) and returns.
2017-07-10 11:47:07 -07:00
Adam Williamson
ddb6f8e416 Adjust cockpit log needles for Rawhide change
The Logs screen changed a bit. Also, it seems wrong to have
'notices' in all these needle names, so let's fix that.
2017-07-10 10:35:27 -07:00
Adam Williamson
d119e1e53e Check for NFS repo message in anaconda.log too
It seems to have moved there in F27.
2017-07-10 10:21:57 -07:00
Adam Williamson
ee2a3daefd Try to work around #1444225 with a sleep in disk_guided_empty
IIRC disk_guided_empty is the only storage test that clicks
through INSTALLATION DESTINATION *really* fast, so let's try
adding a 2-second sleep to it to see if it works around the
'sometimes spoke shows as incomplete' bug that cropped up in
F26 and hasn't been fixed yet and tends to cause failures.
2017-07-06 13:15:20 -07:00
Adam Williamson
09c264fe16 Add Workstation dvd-ostree flavor and tests
Summary:
As we're getting the Workstation dvd-ostree (OStree installer
image) built for Rawhide now, let's try testing it.

Test Plan:
Run the tests on a Rawhide compose that works and
has the image (e.g. 20170615.n.0). Check that new tests work
as expected and old tests are not adversely affected. A
corresponding diff for fedora_openqa will be coming to take
care of scheduling. Note that the tests will often soft fail
for now; this is intentional due to RHBZ#1193590, the bash
prompt for root is incorrect on ostree installs, so I have
added a needle that matches the incorrect prompt but which is
flagged as a workaround needle (so causing the test result to
be a soft fail).

Reviewers: jsedlak, jskladan

Reviewed By: jsedlak

Subscribers: tflink

Differential Revision: https://phab.qa.fedoraproject.org/D1211
2017-06-26 18:48:27 -07:00
Adam Williamson
a29875a2ff Upgrade tests: Run FreeIPA webUI and password change modules
Summary:
This just adds the FreeIPA web UI and password change
test modules to the FreeIPA upgrade test (client end). It's
useful to check out these features too. We don't need to
separate these into separate jobs, as we're not trying to
fill out different matrix checkboxes here, we just want to
know whether everything works.

Test Plan:
Run the test, see that the modules work properly.
I was actually expecting this to fail given the issues with
the upgrade on the server end, but it seems to pass.

Reviewers: jsedlak, jskladan

Reviewed By: jsedlak

Subscribers: tflink

Differential Revision: https://phab.qa.fedoraproject.org/D1207
2017-06-13 10:06:15 -07:00
Adam Williamson
25b910135b Workaround cursor showing up on GDM in desktop_notifications 2017-06-06 18:16:52 -07:00
Adam Williamson
df2c3cd906 Test upgrade of FreeIPA server and client deployment
Summary:
This adds an upgrade variant of the FreeIPA tests, with only
the simplest client enrolment (sssd) for now. The server test
starts from the N-1 release and deploys the domain controller
role. The client test similarly starts from the N-1 release
and, when the server is deployed, enrols as a domain client.
Then the server upgrades itself, while the client waits (as the
server is its name server). Then the client upgrades itself,
while the server does some self-checks. The server then waits
for the client to do its checks before decommissioning itself,
as usual. So, summary: *deployment* of both server and client
occurs on N-1, then both are upgraded, then the actual *checks*
occur on N.

In my testing, this all more or less works, except the role
decommission step fails. This failure seems to be a genuine one
so far as I can tell; I intend to file a bug for it soon.

Test Plan:
Run the new tests, check they work. Run the existing
FreeIPA tests (both the compose and the update variants), check
they both behave the same.

Reviewers: jsedlak, jskladan

Reviewed By: jsedlak

Subscribers: tflink

Differential Revision: https://phab.qa.fedoraproject.org/D1204
2017-06-02 12:17:07 -07:00
Jan Sedlák
48b99a2291 add base system logging test
Differential Revision: https://phab.qa.fedoraproject.org/D1202
2017-06-01 11:06:04 +02:00
Jan Sedlák
4114406668 add UEFI for blivet tests
Differential Revision: https://phab.qa.fedoraproject.org/D1201
2017-05-22 09:26:58 +02:00
Jan Sedlák
0b5f865c8f add custom btrfs partitioning test for blivet-gui
Differential Revision: https://phab.qa.fedoraproject.org/D1194
2017-05-19 13:58:16 +02:00
Jan Sedlák
ea5296b306 add postinstalls to custom partitioning tests
Differential Revision: https://phab.qa.fedoraproject.org/D1192
2017-05-09 09:11:34 +02:00
Adam Williamson
c2b1ea4e8c Use test3 not test1 for FreeIPA password change test module
Previously this module changed test1's password, so that it
would still be able to work even if the webui test module
failed (so test3/test4 didn't get created). But this means
that, for about 30 seconds, test1's password is 'loremipsum'
not 'batterystaple', and if one of the *other* client test
jobs happens to hit a point where it has to auth as test1
during the 30 seconds test1's password is different, it will
fail. This looks to be what happened to the join_kickstart
test the last few days - it failed because it tried to login
as test1 during the password change window.

By using the test3 user instead (which is only used by the
join_cockpit test, currently) we avoid this problem, at the
cost that the password_change module will always fail if the
webui module fails.
2017-05-04 17:29:35 -07:00
Adam Williamson
7d07147b7e Work around F26 base image bug: g-i-s failing to run
With the latest F26 base images, it seems like g-i-s fails to
run at first login. This is clearly some kind of bug somewhere
and I'll investigate it, but it shouldn't be causing the update
tests to fail - we can still validly run the tests with g-i-s
not running. So for now, adjust the _graphical_wait_login test
to tolerate this behaviour when running update tests.
2017-04-24 18:49:25 -07:00
Jan Sedlák
140c5f0a42 Add custom partitioning tests for blivet
Differential Revision: https://phab.qa.fedoraproject.org/D1188
2017-04-24 14:23:35 +02:00
Adam Williamson
e6076e14b8 Workaround issues with loading same module more than once
Summary:
Loading the same module more than once *kinda* works, but it
shows up all kinds of funky in the openQA web interface. There's
a POO for this:
https://progress.opensuse.org/issues/10514
But it doesn't seem like it's going to be resolved immediately,
so in the mean time maybe we should avoid doing it so we don't
have to deal with the weirdness it produces in the web UI. So
here's a kinda icky hack that uses symlinks and stuff to load
multiple instances of 'the same' test module.

Test Plan:
Run an update test, look at how it looks in the web
UI and confirm it's a lot clearer and less buggy. Check there
aren't any bugs in the loading approach. This is deployed on stg
so you can look at it there.

Reviewers: jsedlak, jskladan

Reviewed By: jsedlak

Subscribers: tflink

Differential Revision: https://phab.qa.fedoraproject.org/D1186
2017-04-18 10:38:16 -07:00
Adam Williamson
9da72de4ed Hide mouse on login screen (BGO #781294)
GDM 3.24.1 dumps a cursor in the middle of the login screen even
if you didn't move the mouse yet, so let's workaround that.
2017-04-13 15:28:37 -07:00
Adam Williamson
ee283b1411 Tweak _console_avc_crash so we get soft failures
We do want soft failures (but not hard) when _console_avc_crash
fails. So let's tweak it to ensure that's what happens.
2017-04-10 15:44:13 -07:00
Adam Williamson
e68e113f76 Remove test_flags comments, add ignore_failure flag
It's not really a good idea to have the comments that explain
the test_flags in *every* test, because they can go stale and
then we either have to live with them being old or update them
all. Like, now. So let's just take 'em all out. There's always
a reference in the openQA and os-autoinst docs, and those get
updated faster.

More importantly, add the new `ignore_failure` flag to relevant
tests - all the tests that don't have the 'important' or
'fatal' flag at present. Upstream killed the 'important' flag
(making all tests 'important' by default), I got it replaced
with the 'ignore_failure' flag, we now need to explicitly mark
all modules we want the 'ignore_failure' behaviour for.
2017-04-10 15:00:10 -07:00
Adam Williamson
5b4ce80487 Tweak needle matches at start of _do_install_and_reboot
The way this was set up before, if `anaconda_main_hub` matched
immediately but some spoke was still in a 'processing' state,
it only had 30 seconds (default `assert_and_click` timeout) to
complete and allow the 'Begin Installation' button to appear.
It seems unnecessary to match on *both* needles, really, so
let's just give 300 seconds for the `begin_installation` needle
to appear. It's not going to appear on any other screen.

This problem caused a couple of spurious failures today -
https://openqa.fedoraproject.org/tests/77839 and
https://openqa.fedoraproject.org/tests/77858 - because they
took a bit too long for the INSTALLATION DESTINATION spoke to
clear.
2017-04-07 11:31:34 -07:00
Adam Williamson
1444c5030b Add tests with no user created during install
Summary:
This adds a new test suite, run for Workstation and KDE live
images, which does not create a user during install. It then
expects initial-setup (KDE) or gnome-initial-setup (Workstation)
to appear after install, creates a user, and proceeds with
normal boot.

Note the ARM image test already covers the initial-setup text
mode, and the ARM minimal image is the only case where that
actually matters (it's not included in Server).

Test Plan:
Run the new tests, check they work. Run all old
tests, check the changes didn't break them.

Reviewers: jsedlak, jskladan

Reviewed By: jsedlak

Subscribers: tflink

Differential Revision: https://phab.qa.fedoraproject.org/D1185
2017-04-05 09:43:26 -07:00
Adam Williamson
4ea5129312 Goddamnit, perl, and your goddamn brackets 2017-03-29 15:09:25 -07:00
Adam Williamson
4f014d4f11 Use tty2 not tty3 for freeipa_client login tests
There's an issue where the follow-on _advisory_post test tries
to log in before the 'login failed' error has cleared. We can
easily avoid this by using tty2 for the login tests, then
_advisory_post will switch to tty3 for its stuff.
2017-03-29 14:33:25 -07:00
Adam Williamson
b6d4fd7d4c Don't create user when USER_LOGIN is false, but for KDE install
Summary:
For some reason, we have `USER_LOGIN` set to 'false' for the KDE
package set install test. I really don't know / remember why
that would be; I'd think we should create a user and log in as
that user to make sure it works properly when installing KDE
from the traditional installer. It's not strictly part of the
package set test, true, but still, seems worth doing.

Also, when `USER_LOGIN` is set to 'false' and the installer runs,
we create a user called 'false'. This doesn't seem like what we
wanted, so let's not do that. I dunno if there are any other
cases besides the KDE one that this commit changes, but still.

Test Plan:
Run the full test suite and look for weirdness, check
KDE package set test works as intended (now creates a user called
'test' and logs in as that user).

Reviewers: jsedlak, jskladan

Reviewed By: jsedlak

Subscribers: tflink

Differential Revision: https://phab.qa.fedoraproject.org/D1182
2017-03-29 09:30:16 -07:00
Adam Williamson
cc2d53b4ed Wait a bit after input switch before doing Japanese typing 2017-03-20 11:32:50 -07:00
Adam Williamson
ba3a5152c1 Improve FreeIPA debug logging a bit
Committing without review as this is pretty trivial and I've
had it on staging for the last few days without issue. Just gets
us somewhat better info for debugging FreeIPA issues.
2017-03-16 12:36:33 -07:00
Adam Williamson
1b26f7c04c Add missing 'use utils' to disk_guided_encrypted_postinstall 2017-03-09 13:57:06 -08:00
Adam Williamson
a581d20f8e Desktop update: Wait before clicking update apply button
We used to do this only for KDE, but I've seen the new update
tests sometimes fail at this point for no apparent reason, and
I'm thinking a wait may help (in case they're clicking the
button before it's really 'ready').
2017-03-02 15:33:23 -08:00
Adam Williamson
8d55935b41 Bump up the ARM boot timeout
Let's see if it can get a bit further, at least...
2017-03-02 15:16:02 -08:00
Adam Williamson
e9fda18016 Explicit timeout for the 'top' in _collect_data
This keeps failing because the default `assert_script_run`
timeout changed from 90 to 30 in the last os-autoinst update
(an unintended consequence of a change I made). This has been
fixed upstream, but in the mean time, let's just set an
explicit timeout on the call.
2017-03-02 14:47:44 -08:00
Adam Williamson
602a1ee101 Call boot_decrypt properly (function not method)
Noticed in e.g. https://openqa.fedoraproject.org/tests/58798
we're doing this wrong, `boot_decrypt` was moved into utils as
a function, but we were still calling it as a method...
2017-03-02 14:32:17 -08:00
Adam Williamson
9aed11a77b Work around RHBZ #1400293 (breaks IPA tests on Fedora 24) 2017-02-28 11:46:21 -08:00
Adam Williamson
461f3a6132 Update testing: log packages in update and installed packages
Summary:
This adds some logging related to the update testing workflow,
so we have some idea what we actually tested. We log precisely
which packages were actually downloaded from the update - this
is important as updates can be edited and when examining results
we'll want to know which packages actually got used. We also
add a new module which runs at the end of postinstall and tries
to figure out which packages from the update were installed in
the course of the test. This still isn't a guarantee the test
actually *tested them* in any way, but it at least means they
got installed successfully and didn't interfere with the test.

Test Plan:
Run the update test workflow, check the logs get
uploaded and seem accurate (sometimes some RPM garbage messages
wind up in the package log, I'm not too worried about that at
present). Run the compose test workflow and check it didn't
break.

Reviewers: jsedlak

Reviewed By: jsedlak

Subscribers: tflink

Differential Revision: https://phab.qa.fedoraproject.org/D1149
2017-02-23 14:51:19 -08:00
Adam Williamson
92d588f245 Add support for testing updates
Summary:
This adds an entirely new workflow for testing distribution
updates. The `ADVISORY` variable is introduced: when set,
`main.pm` will load an early post-install test that sets up
a repository containing the packages from the specified update,
runs `dnf -y update`, and reboots. A new templates file is
added, `templates-updates`, which adds two new flavors called
`updates-server` and `updates-workstation`, each containing
job templates for appropriate post-install tests. Scheduler is
expected to post `ADVISORY=(update ID) HDD_1=(base image)
FLAVOR=updates-(server|workstation)`, where (base image) is one
of the stable release base disk images produced by `createhdds`
and usually used for upgrade testing. This will result in the
appropriate job templates being loaded.

We rejig postinstall test loading and static network config a
bit so that this works for both the 'compose' and 'updates' test
flows: we have to ensure we bring up networking for the tap
tests before we try and install the updates, but still allow
later adjustment of the configuration. We take advantage of the
openQA feature that was added a few months back to run the same
module multiple times, so the `_advisory_update` module can
reboot after installing the updates and the modules that take
care of bootloader, encryption and login get run again. This
looks slightly wacky in the web UI, though - it doesn't show the
later runs of each module.

We also use the recently added feature to specify `+HDD_1` in
the test suites which use a disk image uploaded by an earlier
post-install test, so the test suite value will take priority
over the value POSTed by the scheduler for those tests, and we
will use the uploaded disk image (and not the clean base image
POSTed by the scheduler) for those tests.

My intent here is to enhance the scheduler, adding a consumer
which listens out for critpath updates, and runs this test flow
for each one, then reports the results to ResultsDB where Bodhi
could query and display them. We could also add a list of other
packages to have one or both sets of update tests run on it, I
guess.

Test Plan:
Try a post something like:
HDD_1=disk_f25_server_3_x86_64.img DISTRI=fedora VERSION=25
FLAVOR=updates-server ARCH=x86_64 BUILD=FEDORA-2017-376ae2b92c
ADVISORY=FEDORA-2017-376ae2b92c CURRREL=25 PREVREL=24

Pick an appropriate `ADVISORY` (ideally, one containing some
packages which might actually be involved in the tests), and
matching `FLAVOR` and `HDD_1`. The appropriate tests should run,
a repo with the update packages should be created and enabled
(and dnf update run), and the tests should work properly. Also
test a regular compose run to make sure I didn't break anything.

Reviewers: jskladan, jsedlak

Reviewed By: jsedlak

Subscribers: tflink

Differential Revision: https://phab.qa.fedoraproject.org/D1143
2017-02-22 11:33:32 -08:00
Adam Williamson
186678e98b Make log upload work when installed system hits emergency mode
Summary:
This is to handle cases like #1414904 , where the system boots
to emergency mode. We really need logs to try and debug this.

Test Plan:
Force a test to hit emergency mode somehow (right now
you can just run base_services_start on Rawhide over and over
until you hit #1414904, but there's probably an easier way to
do it, I think there's a systemd boot arg to tell it which target
to boot for e.g.) and check logs get uploaded. Also check this
doesn't break log upload for a 'normal' failure.

Reviewers: garretraziel_but_actually_jsedlak_who_uses_stupid_nicknames

Reviewed By: garretraziel_but_actually_jsedlak_who_uses_stupid_nicknames

Subscribers: tflink

Differential Revision: https://phab.qa.fedoraproject.org/D1103
2017-02-01 12:30:21 +01:00
Adam Williamson
b67f604894 Move all remaining utility functions into exporter modules
Summary:
This adds a couple of new exporter modules, renames main_common
to utils (this is a better name: openSUSE's main_common is
functions used in main.pm, utils is what they call their module
full of miscellaneous commonly-used functions), and moves a
bunch of utility functions that were previously needlessly
implemented as instance methods in base classes into the
exporter modules. That means we can get rid of all the annoying
$self-> syntax for calling them.

We get rid of `fedorabase` entirely, as it's no longer useful
for anything. Other base classes keep the 'standard' methods
(like `post_fail_hook`) and methods which actually need to be
methods (like `root_console`, whose behaviour is different in
anacondatest and installedtest).

Test Plan:
Do a full test suite run and check everything lines
up. There should be no functional differences from before at all,
this is just a re-org.

Reviewers: jskladan, garretraziel_but_actually_jsedlak_who_uses_stupid_nicknames

Reviewed By: garretraziel_but_actually_jsedlak_who_uses_stupid_nicknames

Subscribers: tflink

Differential Revision: https://phab.qa.fedoraproject.org/D1080
2017-01-17 23:15:44 -08:00
Adam Williamson
a091185af1 Handle 'non-ASCII in password' warning for Russian installs
Summary:
Since 26.17, anaconda shows a warning when the user password
contains non-ASCII characters, and requires a second Done click
to confirm. This change should handle that.

On the 'catch cases where password typing went wrong and re-try'
bit: to keep that, but not re-type the password *every single
time* on the Russian install test, we'd have to make the needle
match the text of the warning. This is problematic because then
that needle will be able to break without us easily noticing;
that's why I wanted to keep the 'warning bar' needle text-free.
Unfortunately, that means we have to skip the protection for
switched-layout installs.

Note the protection was actually not working for any non-English
install anyhow, because the needle had `LANGUAGE-english` as a
tag. We never noticed that. Failed password typing is pretty
rare now, so we can live without the protection - it's just nice
to have it for the English install tests because there's so many
of them.

Test Plan:
Run the Russian install with a recent Rawhide image,
check it clicks 'Done' twice. Note, it will still fail, because
of RHBZ #1413813.

Reviewers: jskladan, garretraziel_but_actually_jsedlak_who_uses_stupid_nicknames

Reviewed By: garretraziel_but_actually_jsedlak_who_uses_stupid_nicknames

Subscribers: tflink

Differential Revision: https://phab.qa.fedoraproject.org/D1084
2017-01-17 23:13:47 -08:00
Adam Williamson
062d9f8f5e Add jobs to gather memory usage data
Summary:
This adds a new test, memory_check, which just does a default
package set install with `inst.debug` parameter then uploads
the memory usage file (`/tmp/memory.dat`) at the end. We can
have check-compose use the data to analyze changes in memory
usage over time.

Test Plan:
Fire off the Workstation network install image tests
and make sure the memory usage test runs and works on all three
machines. This is live on staging already.

Reviewers: jskladan, garretraziel_but_actually_jsedlak_who_uses_stupid_nicknames

Reviewed By: garretraziel_but_actually_jsedlak_who_uses_stupid_nicknames

Subscribers: tflink

Differential Revision: https://phab.qa.fedoraproject.org/D1082
2017-01-16 09:30:14 -08:00
Adam Williamson
460a6e5439 Simplify the services test fix
I forgot `script_run` actually returns the exit code. That makes
it easier.
2017-01-11 16:03:09 -08:00
Adam Williamson
c9ede993e5 Fix base_services_start
The old version wasn't working - it was passing even though two
services fail to start in Workstation currently. I'm really not
sure why the old approach wasn't working, but it wasn't, and I
rather hate `script_output` anyway, so here's a different way
of doing it which relies on `eval`ing `assert_script_output`
instead. (I really should send a PR for a non-fatal version of
assert_script_output...)
2017-01-11 13:49:17 -08:00
Adam Williamson
dac3c2dd38 Use systemctl --no-legend when collecting service info
Without this, when there are failed services, we get an extra
column to the left of the service names with a unicode dot for
each failed service, which is awkward and screws up the parsing.
2017-01-11 11:58:37 -08:00
Adam Williamson
c4f32ab5ad add an Asian (Japanese) language install test
Summary:
Include some basic testing of Japanese input, and split the
input testing (including Russian) into a separate module, since
it's not really part of 'login' testing.

Test Plan:
Run the test, and the Russian and French tests too to
make sure they didn't break. Tested on staging. Note the Japanese
test soft fails, intentionally, at present, as I discovered a bug
while working on it:
https://bugzilla.gnome.org/show_bug.cgi?id=776189

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D1072
2016-12-21 08:41:00 -08:00
Adam Williamson
7135a4e05d iSCSI: test user authentication
Summary:
This isn't in the criteria, but it's commonly used, so we ought
to test this way. Require authentication for the iSCSI target
and have the test provide the appropriate auth info.

Test Plan:
Run the iscsi test and check it works (you need the
recent fixes for support_server to make *that* work). Nothing
else should be affected.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D1070
2016-12-20 08:34:30 -08:00
Adam Williamson
0fa6138448 Have non-English tests do graphical install and login
Summary:
The non-English tests so far did not test that graphical login
worked as expected, which is a fairly large hole. With this
change, they should do a Workstation install and test login to
both GNOME and the console works as expected. KDE is not yet
tested.

As part of this we tweak the implementation of keyboard layout
switching in graphical environments to use a generic function
in main_common which can handle both anaconda and desktops
(just GNOME at present, but should extend easily to any desktop
with a known switcher key and a visible layout indicator),
replacing the anacondatest class method. I kinda don't like that
the test has to specifically tell the function when it's in
anaconda, but I don't think I want to start experimenting with
a global 'test phase' openQA variable or anything like that at
present.

Fixes T842.

Test Plan:
Run the French and Russian install tests and check
they work as expected. Also run an English Workstation install
if you like, and make sure that didn't break. This change is
live on staging ATM, seems to work fine.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Maniphest Tasks: T842

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D1071
2016-12-16 09:40:29 -08:00
Adam Williamson
d8ba53aefc workaround #1402427 in support_server
support_server was failing because of #1402427. I re-generated
the disk image with latest F25 so the fixed selinux-policy is
used, but even then, it seems we have to run 'restorecon' on
rpcbind manually before starting nfs.
2016-12-13 16:38:22 -08:00
Jan Sedlák
44cf1cd89c reintroduce rescue on UEFI
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D1060
2016-12-13 09:18:34 +01:00
Adam Williamson
5c94924c59 guh, main_common 2016-12-08 12:20:09 -08:00
Adam Williamson
6dc20de0b0 move 'loadkeys us' call into avc_crash test for now
Ugh, there's no good way to do this ATM. If this doesn't work
I might just skip the test for this case.
2016-12-08 12:18:45 -08:00
Adam Williamson
cc7fe4b971 Tweaks to AVC test loading / run
Committing without review as this causes failures...try to make
sure we only run the AVC test when it makes sense, and fix
running it on the French install test.
2016-12-08 12:03:26 -08:00
Adam Williamson
7687a3aa34 Add a non-fatal console AVC / crash post-install check
Summary:
This has all console tests check for AVCs (with ausearch) and
crashes (with coredumpctl) at post-install stage. It's non-
fatal as this doesn't really mean the test failed, but we want
to spot when there are unexpected AVCs or crashes.

Test Plan:
Run some console tests, check it works right. I only
tested with one test, since so many are broken on Rawhide ATM

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D1066
2016-12-08 08:58:29 -08:00
Adam Williamson
62ef578758 Extend a timeout in the graphical update test
The 'refresh' button takes a bit longer to appear on F24 respin
tests.
2016-12-07 17:27:35 -08:00
Adam Williamson
6f90ff0143 collect_data: run top *before* everything else
Committing without review as this is pretty trivial. Running
top *after* all the other collect_data tasks leads to some
meaningless fluctuations in the output; I intended to look for
activity caused by stuff running 'as usual', not activity from
the other collect_data tasks.
2016-12-02 15:01:40 -08:00
Adam Williamson
571cf3c035 bump the initial ARM boot timeout a bit 2016-11-09 07:21:03 -08:00
Adam Williamson
65cadb11df Collect some data on freshly-installed systems after some tests
Summary:
I've been wanting to do this for a while; I think it'll let us
check for some significant changes between composes. This should
cause runs of a few test cases to collect and upload info on:

* installed packages
* free memory
* disk space
* active services
* 1 minute of CPU usage info (via top)

immediately after install and initial login. In some cases this
will be useful / interesting simply to look at directly, but
we can also have check-compose analyze the data and include
significant changes in its reports.

Test Plan:
Run affected tests, make sure the data collection
works.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D1046
2016-11-09 07:20:17 -08:00
Adam Williamson
7b31b8263e Force GNOME to notify updates, re-enable test on Workstation
Summary:
GNOME's update notification criteria are pretty braindead: it
fires the update check timer once on login then once every hour
thereafter, but only actually checks for and notifies of updates
once a day if it's after 6am(?!?!?!). So we have to do a bunch
of fiddling around to ensure we reliably get a notification.
Move the clock to 6am if it's earlier than that, and reset the
'last update check' timer to 48 hours ago, then log in to GNOME
after that.

Note: I thought this still wasn't fully reliable, but I've looked
into all the recent failures of either test on staging and
there's only one which was really 'no update notification came
up', and the logs clearly indicate PK did run an update check,
so I don't think that was a test bug (I think something went
wrong with the update check). The other failures are all 'GNOME
did something wacky', plus one case where the needle didn't quite
match because I think the match area is slightly too tall; I'll
fix that in a second.

Test Plan:
Run the tests on both KDE and GNOME and check they
work properly now (assuming nothing unrelated breaks, like KDE
crashing...)

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D1039
2016-10-27 16:23:59 -07:00
Adam Williamson
1ae1b6e2cb wait longer for system to shutdown in _console_shutdown
We're not really *testing* shutdown here, we're just shutting
down to make sure the uploaded disk image is clean. So we don't
really mind if shutdown takes a while. It often seems to take
longer than 1 minute on KDE installs and cause a soft fail, so
let's bump the timeout to 3 minutes.
2016-10-26 14:03:15 -07:00
Adam Williamson
b129bf5487 don't wait for dnf system-upgrade reboot command to return
because it won't.
2016-10-21 17:43:04 -07:00
Adam Williamson
f95e1d55a8 give IPA server decommission longer to complete (RHBZ #1387425)
It's suddenly taking longer in Rawhide - reported to RHBZ - so
let it take longer.
2016-10-20 13:44:08 -07:00
Adam Williamson
dcb68d93c8 drop our implementation of script_run in favour of os-autoinst
Summary:
os-autoinst implements `script_run` itself now, we aren't
required to implement it ourselves any more. os-autoinst's
implementation is better than ours, as it allows for verifying
the script actually ran (via the redirect-output-to-serial-
console trick).

So this drops our implementation so we'll just use the upstream
one. Where I judged we don't want to bother with the 'check
the command actually ran' feature I've adjusted our direct
`script_run` calls to pass a wait time of 0, which skips the
'wait for command to run' stuff entirely and just does a simple
'type the string and hit enter'.

Because of how the inheritance works, our `assert_script_run`
calls already used the os-autoinst `script_run`, rather than
the one from our distribution.

This should prevent `prepare_test_packages` sometimes going
wrong right after removing the python3-kickstart package, as
we'll properly wait for that removal to complete now (before
we weren't, we'd just start typing the next command while it
was still running, which could result in lost keypresses).

Test Plan:
Check all tests still run OK (I've tried this on
staging and it seems fine).

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D1034
2016-10-20 09:24:48 -07:00
Adam Williamson
3d84e2a8c2 type root password faster on ostree installs
Summary:
Since we started using `type_very_safely` for typing the root
password, we starting hitting a race issue. If we complete the
root password spoke so slowly that the software deployment
process completes in the meantime, anaconda will wait until we
complete the spoke then immediately flip to the 'post-install
configuration' step, at which point access to the USER CREATION
spoke is blocked.

We don't hit this case on regular RPM installs or live installs
as the deployment phase still takes a while for both of those,
but we are sometimes hitting it for the Atomic ostree install
image, as the software deployment phase is pretty fast there.
We *could* just not bother creating a user and testing we can
log in as a user for that test, but I don't like that approach,
we *should* be testing that user creation and login works OK
for ostree installs. So instead, let's just type the root
password a bit less safely for ostree installs; this will be
more vulnerable to typing errors but hopefully will avoid the
race problem.

Test Plan:
Run a few Atomic installs, see if they hit the race.
Might need to run other tests at the same time, and you may not
be able to hit it, this is obviously dependent on the I/O of
the worker host...

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D1022
2016-10-13 18:39:33 -07:00
Adam Williamson
fc1dc167f9 support_server: give the DVD copy a bit longer to complete
seems like it took more than 2 minutes in F25 testing today.
2016-10-11 18:11:34 -07:00
Adam Williamson
fdeb333ff3 fix desktop_notifications to use assert_screen not check
whoops...this test should have been failing...
2016-09-30 09:21:01 -07:00
Adam Williamson
bacb6f1f7b redo console_login with multiple matches, move to main_common
Summary:
Since we can match on multiple needles, we can drop the loop
from console_login and instead do it this way, which is simpler
and should work better on ARM (the timeouts will scale and
allow ARM to be slow here). Also move it to main_common as
there's no logical reason for it to be a class method.

Also remove the `check` arg. `check` was only set to 0 by two
tests, _console_shutdown and anacondatest's _post_fail_hook.

For _console_shutdown, I think I just wanted to give it the
best possible chance of succeeding. But we're really not going
to lose anything significant by checking, the only case where
check=>0 would've helped is if the 'good' needle had stopped
matching, and all sorts of other tests will fail in that case.

anacondatest was only using it to save a screenshot of whatever
was on the tty if it didn't reach a root console, which doesn't
seem that useful, and we'll get screenshots from check_screen
and assert_screen anyway.

Test Plan:
Run all tests, check they behave as expected and
none inappropriately fails on console login.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D1016
2016-09-30 08:42:45 -07:00
Adam Williamson
8a6529526f fix ARM initial-setup handling for extra choice
As with the text install test, there's an additional choice on
the 'Time settings' path compared to whatever image garret
developed the test on; right after 'Time settings' you have to
pick 'Set timezone' or 'Configure NTP servers'. So adjust the
test to handle this, just like we did there.
2016-09-28 13:15:56 -07:00
Adam Williamson
e9ce14a891 consolidate login waits, use postinstall not entrypoint for base
Summary:
I started out wanting to fix an issue I noticed today where
graphical upgrade tests were failing because they didn't wait
for the graphical login screen properly; the test was sitting
at the 'full Fedora logo' state of plymouth for a long time,
so the current boot_to_login_screen's wait_still_screen was
triggered by it and the function wound up failing on the
assert_screen, because it was still some time before the real
login screen appeared.

So I tweaked the boot_to_login_screen implementation to work
slightly differently (look for a login screen match, *then* -
if we're dealing with a graphical login - wait_still_screen
to defeat the 'old GPU buffer showing login screen' problem
and assert the login screen again). But while working on it,
I figured we really should consolidate all the various places
that handle the bootloader -> login, we were doing it quite
differently in all sorts of different places. And as part of
that, I converted the base tests to use POSTINSTALL (and thus
go through the shared _wait_login tests) instead of handling
boot themselves. As part of *that*, I tweaked main.pm to not
require all POSTINSTALL tests have the _postinstall suffix on
their names, as it really doesn't make sense, and renamed the
tests.

Test Plan: Run all tests, see if they work.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D1015
2016-09-27 11:48:15 -07:00
Adam Williamson
683b819bf9 extend the wait_still_screen time in upgrade_pre too
I have a better fix for this coming, but it's a big change that
requires review, so for now, do this. The tests are actually
failing because the 30 second wait_still_screen is too short(!)
- it's triggering while the test is sitting at the 'full Fedora
logo' bootsplash screen then doing a 30 second assert_screen
graphical_login, which is failing because it actually takes
more than 60 seconds to get from the 'full F' screen to the
login screen. So let's just make the wait_still_screen 45
seconds for now.
2016-09-24 16:39:52 -07:00
Adam Williamson
032d8f76a3 give upgrade_preinstall a bit longer for desktop startup
90 seems too tight. See:
https://openqa.fedoraproject.org/tests/36079
2016-09-24 14:28:20 -07:00
Adam Williamson
ddc91efeff detect (rather than guessing) desktop vt
use 'ps' output for Xorg and Xwayland. We'd need some new
openQA var to get this right by 'guessing', as it's vt1 for
Workstation when running live - so long as autologin worked -
but vt2 after install. We'd need a var or some other thing to
detect which case we're running in. LIVE doesn't do it, it's
set even when running a post-install test from a live image.

So instead let's just do it a bit more cleverly. This also
gives us a bit of insurance against changes in GDM, SDDM etc.
behaviour, so long as Xwayland or Xorg is running (and we can
add additional processes to the list, like gnome-shell, if
needed/appropriate). We assume the *final* listed process -
i.e. the most recently-started one - will be the desktop;
this covers gdm's behaviour of starting up on vt1 then running
the user session on vt2. We can make this function more complex
and add args if we ever get to the point where we have multi-
user tests running or anything (e.g. allow to pass a username
and only look for that user's processes).

Landing without review as this broke the live variant of the
test on Workstation in production (kinda not sure why it worked
in testing, or I didn't notice that it failed, but never mind).
I've tested it on staging.
2016-09-24 13:04:04 -07:00
Adam Williamson
e33b635f41 handle auth request for unsigned updates on KDE
When a package is unsigned, KDE will prompt for authentication.
Let's handle this. But count it as a soft fail, because
puiterwijk claims that Rawhide packages will be autosigned
from next week, so this *should* not happen and would indicate
an unsigned package in the repos. We make the KDE 'update
complete' needle area smaller because the wider area includes
some transparency and so will only match if the update applet
is open; this area will match whether it's open (no auth case)
or not open (auth case - the applet seems to disappear after
you provide the password in the auth prompt).

Pushing without review as the test is in production so I want
to make sure it works correctly.

(Also, hey, check out that array match for assert_screen and
that match_has_tag! This is gonna make some things so much
easier...thanks upstream)
2016-09-23 18:20:28 -07:00
Adam Williamson
a067d0655f add a desktop notifications test
Summary:
this more or less covers desktop_error_checks and desktop_
update_notification, though it can't really distinguish
between them easily. All we know is that if both the live and
postinstall versions of this test pass, both of those tests
pass. Any fails will have to be investigated manually.

Test Plan:
Run the tests for both KDE and Workstation, see
what happens. Workstation will fail for F25 and Rawhide at
present, due to SELinux/abrt notifications.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D1004
2016-09-23 16:03:13 -07:00
Adam Williamson
e8b20ec73f add a desktop_update_graphical test
Summary:
Very similar to the CLI update test, but using the desktops'
update applications. This is based off the CLI update test
branch as it uses the shared functions that branch introduced.
We do not use the fake update packages, as they don't really
do anything useful for these tests; for dnf they can help us
distinguish between issues with the dnf mechanism and issues
with the repos, but we can't really tell that in the graphical
case. So we only use the python3-kickstart package here.

Test Plan:
Run the test on both KDE and GNOME and ensure it
performs as intended. I've been testing it on staging, so you
can see it there.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D1010
2016-09-22 11:38:51 -07:00
Adam Williamson
44ec3d84c3 add a base_update_cli test
Summary:
this uses a couple of test repos with fake packages to test the
basic dnf mechanisms are working, then messes around with the
python3-kickstart package a bit to try and test the default repo
configuration is working, keys are in place and so on. We use
python3-kickstart because we should be able to rely on the copy
of that package in the 'stable' repo being installable (or else
the compose would have failed), but it shouldn't be vital to
the operation of the system.

Test Plan: Run the test, see if it works.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D1006
2016-09-22 10:57:12 -07:00
Adam Williamson
1cdd8e18b7 make sure we get logs from failed kickstart installs
Summary:
by waiting for the bootloader in _boot_to_anaconda rather than
_console_wait_login, we can ensure that we use the anaconda
post-fail hook and thus get logs uploaded when a kickstart
install fails.

Test Plan:
Run a kickstart install test that fails and check
anaconda logs get uploaded. Then run one that works and make
sure it...still works.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D1005
2016-09-20 10:51:51 -07:00
Jan Sedlák
c0b9bdb543 add anaconda rescue test on encrypted disk
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D995
2016-09-16 14:44:03 +02:00
Adam Williamson
ec00f3c540 fix install_text test's timezone spoke handling
I guess this was developed on F24, or an older F25? There is
an additional choice showing up in production: before the tz
choice, it asks if you want to set the tz or configure NTP. So
we have to pick tz.
2016-09-13 12:58:22 -07:00
Adam Williamson
e1ec1997af try to be safer when typing in X: slower, more checks
Summary:
the main thing this does is try and type slower in X - this
should cover nearly everywhere we type anything in X, and make
it type slower. We also add a bit more safety checking to some
old tests which didn't have it (mainly _do_install_and_reboot)
- wait_still_screen after typing to make sure all the keypresses
were registered before continuing.

This is an attempt to mitigate the problems we've seen where
the wrong text gets typed into the wrong places and the tests
break.

This branch is live on staging atm. It still has *some* issues,
but I do think it's an improvement.

Test Plan:
run the tests (probably several times), compare to
runs without the change, see if it's better or worse...

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D993
2016-09-12 10:24:30 -07:00
Adam Williamson
1c5fffa859 add a database server role test (and client test)
Summary:
Pretty straightforward tests which deploy the database
server role and exercise it a bit.

Test Plan: Run the tests, check they work properly.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D991
2016-09-08 14:19:09 -07:00
Adam Williamson
9af88e3f23 destroy kerberos cache before firefox in freeipa password change
In F25+, Firefox seems to do kerberos auth automatically, so if
we go to the FreeIPA admin URL while kerberos-authed as test4,
we are logged in right away as test4 - neat! But not what we
wanted. So let's kdestroy.
2016-09-08 10:28:31 -07:00
Adam Williamson
03af57e0da start firefox at 1024x768 resolution
when we run firefox in a bare X session, by default we get an
800x600 firefox in a 1024x768 X server with some dead black
space to the right and bottom of the screen. Now it turns out
that if the mouse is in the dead space, Firefox will not get
any keystrokes we send.

This didn't used to be a problem, but I made it into one with
this os-autoinst change:

https://github.com/os-autoinst/os-autoinst/pull/559

that makes os-autoinst move the cursor to 1023,767 after each
`assert_and_click`, instead of 0x0 as it did before, unless the
cursor has previously been explicitly place somewhere. So in
this case it gets moved to the dead space, and Firefox stops
responding to keypresses after the first `assert_and_click`.

We could equally well fix this by setting the cursor to 0,0
after running Firefox, but I like this more as it makes sure
we won't run into the same problem some other way, and makes
the videos and screenshots look nicer.

This fixes the realmd_join_cockpit test that's been failing
ever since I installed an os-autoinst with that fix. Committing
without review as it's a straightforward fix and I want the
test working again...
2016-09-08 09:15:43 -07:00
Jan Sedlák
db95bccd52 add anaconda text UI test
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D980
2016-09-07 10:34:54 +02:00
Adam Williamson
1c882f612e lengthen that last wait_still_screen a bit...
...per https://openqa.stg.fedoraproject.org/tests/38809 , one
second isn't long enough.
2016-09-05 17:50:06 -07:00
Adam Williamson
21a5f23206 add a short wait_still_screen to _graphical_wait_login
looks like we had a good ol' 'sliding-in-from-the-side' fail
in https://openqa.fedoraproject.org/tests/32163 .
2016-09-04 09:46:58 -07:00
Adam Williamson
82b8b0a053 handle REPOSITORY_GRAPHICAL being http as well as https
I assumed the 'compose location' sent by fedmsg was https, but
in fact it's http (you get redirected to https when you access
it). Could just change the default back to 4, but why not make
it properly robust. Sending without review so this doesn't go
wrong all weekend.
2016-09-02 08:51:36 -07:00
Adam Williamson
ef689e75a9 use compose repository (not master repo) for most tests
Summary:
we have a long-standing problem with all the tests that hit
the repositories. The tests are triggered as soon as a compose
completes. At this point in time, the compose is not synced to
the mirrors, where the default 'fedora' repo definition looks;
the sync happens after the compose completes, and there is also
a metadata sync step that must happen after *that* before any
operation that uses the 'fedora' repository definition will
actually use the packages from the new compose. Thus all net
install tests and tests that installed packages have been
effectively testing the previous compose, not the current one.

We have some thoughts about how to fix this 'properly' (such
that the openQA tests wouldn't have to do anything special,
but their 'fedora' repository would somehow reflect the compose
under test), but none of them is in place right now or likely
to happen in the short term, so in the mean time this should
deal with most of the issues. With this change, everything but
the default_install tests for the netinst images should use
the compose-under-test's Everything tree instead of the 'fedora'
repository, and thus should install and test the correct
packages.

This relies on a corresponding change to openqa_fedora_tools
to set the LOCATION openQA setting (which is simply the base
location of the compose under test).

Test Plan:
Do a full test run, check (as far as you can) tests run sensibly
and use appropriate repositories.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D989
2016-09-01 08:22:59 -07:00
Adam Williamson
9311146f95 drop dictionary error workaround
Summary:
the dictionary error bug was fixed some time back, so drop this
workaround for it.

Test Plan:
Run all tests for F25 and Rawhide and verify they don't need
this workaround any longer.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D988
2016-08-30 11:18:46 -07:00
Adam Williamson
67cb387e9a drop RHBZ #1349721 workaround, as the bug is fixed
Summary: Pretty simple!

Test Plan:
Check the upgrade tests work the same as before the
change.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D979
2016-08-29 11:45:34 -07:00
Adam Williamson
88e63f1593 fix upgrade test (broken by recent nogpgcheck commit)
in the recent commit to always use nogpgcheck I inadvertently
broke the upgrade tests, by dropping the `--releasever` from
the `dnf system-upgrade download` command. So fix that.
2016-08-12 10:08:46 -07:00
Adam Williamson
b7abdf81a9 always use --nogpgcheck when installing packages
Summary:
Except when running on the pre-upgrade release in the upgrade
tests (where GPG check should always be OK).

Currently we always need to use --nogpgcheck on Rawhide, and we
must also use it on Branched prior to the Bodhi activation
point. At present we don't really have any simple way to know
when the Bodhi activation point has kicked in. We could assume
that it's safe to do GPG checking for 'candidate' (not nightly)
composes, but even that isn't 100% safe and isn't really the
*right* thing to do. So I think for now it's best to just always
use --nogpgcheck , until we come up with a decent way to check
for Bodhi enablement, or releng figures things out so we can
rely on packages being signed in Rawhide and in Branched before
Bodhi enablement.

Test Plan:
Check the tests all still run, make sure I didn't
miss any dnf calls.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D964
2016-08-10 07:56:13 -07:00
Adam Williamson
a625eac820 support_server: give the ISO copy a bit longer to complete
this is a big copy, could take longer than the default.
2016-08-09 17:11:53 -07:00
Adam Williamson
a8ddc002f8 add QA:Testcase_desktop_browser test
Summary:
pretty simple stuff here. The distinction between 'firefox' and
'browser' is that the 'browser' needles I expect would also be
correct for other default browsers, while the 'firefox' needles
are specific to Firefox. We need '-kde' variants of some Firefox
needles where interface text is included, because the font is
Cantarell in GNOME but whatever the default 'sans' font is in
KDE - I suppose we should really use -thatfontsname rather than
-kde, but I can't think what it's called...

I couldn't do the 'log in to FAS' bit of the test since we don't
really have a sane way to provide a password while not exposing
it to the public.

Test Plan:
Run the test, check it works - for both KDE and
Workstation.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D938
2016-08-03 13:22:29 -07:00
Adam Williamson
a901fce4ab add QA:Testcase_FreeIPA_password_change test
Summary:
again, added as a non-fatal module for realmd_join_cockpit as
it's convenient to do it here. Also abstract a couple of ipa
bits into a new exporter package in the style of SUSE's
mm_network, rather than using ill-fitting class inheritance as
we have before - we should probably convert our existing class
based stuff to work this way.

Also a few minor tweaks and clean-ups of the other tests:

The path in console_login() where we detect login of a regular
user when we want root or vice versa and log out was actually
broken because it would 'wait' for the result of the 'exit'
command, which obviously doesn't work (as it relies on running
another command afterwards, and we're no longer at a shell).
This commit no longer actually uses that path, but I spotted
the bug with an earlier version of this which did, and we may
as well keep the fix.

/var/log/lastlog is an apparently-extremely-large sparse file.
A couple of times it seemed to cause tar to run very slowly
while creating the /var/log archive for upload on failure. It's
no use for diagnosing bugs, so we may as well exclude it from
the archive.

I caught cockpit webUI login failing one time when testing the
test, so threw in a wait_still_screen before starting to type
the URL, as we have for the FreeIPA webUI.

I also caught a timing issue with the openQA webUI policy add
step; the test flips from the Users screen to the HBAC screen
then clicks the 'add' button, but there's actually an identical
'add' button on *both* screens, so it could wind up trying to
click the one on the Users screen instead, if the web UI took
a few milliseconds to switch. So we throw in a needle match to
make sure we're actually on the HBAC screen before clicking the
button.

We make the freeipa_webui test a 'milestone' so that if the
new test fails, restoring to the last-known-good milestone
doesn't take so long; it actually seems like openQA can get
confused and try to cancel the test if restoring the milestone
takes a *really* long time, and wind up with a zombie qemu
process, which isn't good. This seems to avoid that happening.

Test Plan:
In the simple case, just run all the FreeIPA-related
tests on Fedora 24 (as Rawhide is broken) and make sure they all
work properly. To get a bit more advanced you can throw in an
`assert_script_run 'false'` in either of the non-fatal tests to
break it and make sure things go properly when that happens (the
last milestone should be restored - which should be right after
freeipa_webui, sitting at tty1 - and run properly; things are
set up so each test starts with root logged in on tty1).

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D935
2016-08-03 13:21:12 -07:00
Adam Williamson
c6d8b54d58 add QA:Testcase_Anaconda_updates.img_via_installation_source
Summary:
we can test this quite easily by placing the standard openQA
updates image in the NFS repo used for the NFS repo install
tests. We just have to copy the contents of the ISO (instead of
directly exporting the ISO loop mount as an NFS share) so we
can add this extra file.

At first I planned to combine this with the NFS repo variation
test, but when you use a remote stage2 like this it changes repo
setup such that the packaging.log line we look for to verify
the remote repo was used does not show up, and there's enough
fuzziness in how anaconda-dracut fudges inst.repo and
inst.stage2 that it's probably a good idea to test them
separately anyhow.

Test Plan:
Run the new test and the other NFS tests, make sure
this one works and the others don't break.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D929
2016-08-03 13:19:18 -07:00
Adam Williamson
aacd01ea8b add encrypted workstation upgrade tests (current and previous)
Summary:
This requires us to handle decryption each time we reboot in
the upgrade process, so factor that little block out into the
base class so we don't have to keep pasting it. It's also a
bit tricky to integrate into the 'catch a boot loop' code we
have to deal with #1349721, but I think this should work. There
is a matching openqa_fedora_tools diff to generate the disk
image.

Test Plan:
Run the tests, check that they work, run the other
upgrade and encrypted install tests and check they still work
properly too.

Reviewers: garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D922
2016-07-08 08:56:57 -07:00
Adam Williamson
29c3fab71e add a softfail workaround for RHBZ #1349721
Summary:
try to catch a boot loop after `dnf system-upgrade reboot`, if
we do, set the test to soft_fail and pass enforcing=0 to work
around it.

Test Plan:
Run the upgrade_foo tests and see that they now soft
fail instead of hard failing (unless there are any other issues).
Run the upgrade_2_foo tests and make sure they still pass (i.e.
we don't erroneously soft fail them).

Reviewers: garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D912
2016-07-01 08:16:47 -07:00
Adam Williamson
e24c377b01 add FreeIPA web UI testing
Summary:
as a new, non-fatal test step in the cockpit enrolment test,
because it kinda fits in there; we have an enrolled system with
a web browser *right there*. This will require making the wiki
reporting stuff slightly cleverer so we can say 'report a pass
for this wiki test instance if this test step passed', but that
should be possible. Making this non-fatal means the rest of the
cockpit enrolment test will go ahead even if the freeipa web UI
fails.

The 'check if we can log in' stuff is identical to freeipa_
client_postinstall except with different user names, so we could
potentially factor that out somehow, but I couldn't think of a
super clean way to do it so for now it's just copied.

Note this diff is on top of the freeipa-realmd branch which
is for D894, it's not on top of develop.

Test Plan:
Run the modified test and see if it works. No other
tests are modified, so they should be OK.

Reviewers: garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D895
2016-06-28 12:01:31 -07:00
Adam Williamson
5f5021530d add a realmd_join_sssd test, and use DHCP for it and cockpit
Summary:
This is a pretty straightforward IPA joining test. Since I
figured out how to set up a DHCP server for support_server,
let's do the same for the domain controller so we can simplify
these enrolment tests a bit.

We also extend the timeout on installing haveged on the server
a bit (2 minutes is a bit low when it hits a slow metadata
download), and drop an unnecessary clone_host_file from the
cockpit join test (it was only there for testing and the next
operation immediately overwrites it).

Test Plan:
Do a full server DVD test run, check the new test
works and none of the others broke.

Reviewers: garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D894
2016-06-28 12:00:13 -07:00
Adam Williamson
fe507b9d46 bump haveged install timeout to 5 mins
2 mins is a bit short sometimes, it seems...Rawhide keeps failing
on this.
2016-06-21 21:22:50 -07:00
Adam Williamson
0da6652287 add NFS tests (and DHCP/DNS in the support server)
Summary:
Set up the support server to provide DHCP/DNS functionality and
an NFS server, providing a kickstart. Add a kickstart test just
like the other root-user-crypted-net kickstart tests except it
gets the kickstart from the support server via NFS. Also add NFS
repository tests and a second support server for Server-dvd-iso
flavor: this test must run on that flavor to ensure that packages
are actually available. The support server just mounts the
attached 'DVD' and exports it via NFS.

Note we don't need to do anything clever to avoid IP conflicts
between the two support servers, because os-autoinst-openvswitch
ensures each worker group is on its own VLAN.

As part of adding the NFS repo tests, I did a bit of cleanup,
moving little things we were repeating a lot into anacondatest,
and sharing the 'check if the repo was used' logic between all
the tests (by making it into a test step that's loaded for all
of them). I also simplified the 'was repo used' checks a bit,
it seems silly to run a 'grep' command inside the VM then have
os-autoinst do a grep on the output (which is effectively what
we were doing before), instead we'll just use a single grep
within the VM, and clean up the messy quoting/escaping a bit.

Test Plan:
Run all tests - at least all repository tests - and
check they work (make sure the tests are actually still sane,
not just that they pass). I've done runs of all the repo tests
and they look good to me, but please double-check. I'm currently
re-running the whole 24-20160609.n.0 test on staging with these
changes.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D888
2016-06-13 08:42:30 -07:00
Adam Williamson
7a8ae3a357 add an iscsi test, and a support_server test to support it
Summary:
this is following a SUSE model for tests where we need a server
end but don't want setting up the server to constitute a real
test in itself, we want it to be stable. The 'support_server'
test just boots a pre-built (by createhdds) disk image, sets up
networking, and runs the iSCSI server.

To run the iSCSI test we need to handle networking config in
anaconda (or we would need to set the support server up as a
DHCP server, which may be worth considering), so this adds that.
We also need to be able to specify the target device for a
volume in custom partitioning, so this adds that too.

Test Plan:
Build the necessary support server disk image (use
D883), then run the test and make sure it works. Also make sure
all other tests continue to work.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D884
2016-06-09 08:43:46 -07:00
Adam Williamson
3a31f9639d use sudo when rebooting after live install
on Rawhide it seems user cannot run 'reboot' any more. 'sudo
reboot' seems to do the trick (tested KDE and GNOME).
2016-06-08 12:20:43 -07:00
Adam Williamson
66fc3cc7d4 add a cockpit realmd FreeIPA join test
Summary:
This requires a few other changes:

* turn clone_host_resolv into clone_host_file, letting you clone
  any given host file (cloning /etc/hosts seems to make both
  server deployment and client enrolment faster/more reliable)
* allow loading of multiple POSTINSTALL tests (so we can share
  the freeipa_client_postinstall test). Note this is compatible,
  existing uses will work fine
* move initial password change for the IPA test users into the
  server deployment test (so the client tests don't conflict over
  doing that)
* add GRUB_POSTINSTALL, for specifying boot parameters for boot of
  the installed system, and make it work by tweaking _console_wait
  _login (doesn't work for _graphical_wait_login yet, as I didn't
  need that)
* make the static networking config for tap tests into a library
  function so the tests can share it
* handle ABRT problem dirs showing up in /var/spool/abrt as well
  as /var/tmp/abrt (because the enrol attempt hits #1330766 and
  the crash report shows up in /var/spool/abrt, don't ask me why
  the difference, I just work here)
* specify the DNS servers from the worker host's resolv.conf as
  the forwarders for the FreeIPA server when deploying it; if we
  don't do this, rolekit defaults to using the root servers as
  forwarders(!) and thus we get the public, not phx2-appropriate,
  results for e.g. mirrors.fedoraproject.org, some of which the
  workers can't reach, so PackageKit package install always fails
  (boy, was it fun figuring THAT mess out)

Even after all that, the test still doesn't actually pass, but
I'm reasonably confident this is because it's hitting actual bugs,
not because it's broken. It runs into #1330766 nearly every time
(I think I saw *one* time the enrolment actually succeeded), and
seems to run into a subsequent bug I hadn't seen before when
trying to work around that by trying the join again (see
https://bugzilla.redhat.com/show_bug.cgi?id=1330766#c37 ).

Test Plan:
Run the test, see what happens. If you're really lucky,
it'll actually pass. But you'll probably run into #1330766#c37,
I'm mostly posting for comment. You'll need a tap-capable openQA
instance to test this.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D880
2016-06-07 13:00:39 -07:00
Adam Williamson
aba2814611 add cockpit_default and cockpit_basic tests
Summary:
This adds tests for the Server_cockpit_default and cockpit_basic
test cases. Some notes: I was initially thinking of combining
these into a single test with multiple test modules and coming
up with a system for doing wiki reporting based on individual
test module status, but because we'll also want to do a cockpit
FreeIPA enrol test, I decided against it. We don't really want
to combine all three because then we would skip the cockpit
tests whenever FreeIPA server deployment failed, which isn't
ideal. So since we'll need a separate FreeIPA enrolment test
anyway it doesn't really make sense to go to the trouble of
designing a system for loading multiple postinstall tests (though
I have an idea for that!) and a per-module wiki reporting system.

This was the most minimal and hopefully reliable method for
running Cockpit from a stock Server install that I could think
of. An alternative approach would be to have, say, the most
recent stable Workstation live as a 'stock' asset and have two
tests, one which runs a stock Server install and just waits and
another which boots the live image and accesses the cockpit
running on the other box, but that seems a bit over-complex. It
is not possible to have dependencies between tests for different
ISOs, in case you were wondering about having a Workstation live
test which runs parallel with a Server DVD test, we can't do
that. One funny thing is the font that winds up getting used for
the desktop, but I don't *think* that should be a problem.

Picking needles was a bit tricky; any improvement suggestions
are welcome. I'm hoping it turns out to be safe to rely on some
dbus log messages being present; I think logging into Cockpit
triggers activation of the realmd dbus interface, so there
*should* always be some messages related to that. An alternative
would just be to match on a sliver of the dark grey table header
and the light grey row beneath it and assume that'll always be
the first message (whatever the message is), but then we have to
find some area of the message details screen which is always
present for any message, and it just seems a tad more likely to
result in false passes. Similary I'm making an assumption that
auditd is always going to show up on the first page of the
Services screen and the details screen will always show that
'loaded...enabled' text.

Test Plan:
Run the tests and see if they work! See
https://openqa.stg.fedoraproject.org/tests/21373 and
https://openqa.stg.fedoraproject.org/tests/21371 for my tests.

Reviewers: garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D874
2016-06-01 09:05:33 -07:00
Jan Sedlák
05f33bbd39 workaround mcelog failing
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D843
2016-06-01 08:12:16 +02:00
Adam Williamson
cf543adf72 add a server default firewall test
Summary:
The test here is a bit ugly, but it should work. Better ideas
welcome =)

Test Plan:
Run the test, check it works (and maybe hack it up
a bit and check it fails properly too, it worked first time for
me which is always suspicious)

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D870
2016-05-31 14:24:34 -07:00
Jan Sedlák
802c9298fd add base_services_start test for ARM
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D861
2016-05-27 12:36:15 +02:00
Adam Williamson
ede7713a6c disk_guided_encrypted: only switch layout when appropriate
this stuff is only appropriate when a switched keyboard layout
is in play. Left unconditional like this, it breaks the normal
encrypted install test case.
2016-05-20 11:56:48 -07:00
Adam Williamson
efb0a916f6 _graphical_wait_login: inherit 'installedtest' not 'basetest'
Summary:
We'll often be able to reach a root console and upload logs if
this test fails, and we want to do that so we can figure out
what's going on.

Test Plan:
Have the test fail and check that it manages to upload
logs.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D856
2016-05-20 07:53:45 -07:00
Adam Williamson
a4f3267534 add Russian install test
Summary:
Requires new needles and test suite and job template, plus a
few tweaks to handle 'switched' keyboard layouts (so we use the
switched layout in the username and password).

Test Plan:
Run the test and see that it...fails. But that's OK!
It's a genuine bug: RHBZ #1333998 . At least make sure it gets
to that point and no other tests have broken and all the needles
look sane.

Reviewers: garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D846
2016-05-20 07:52:55 -07:00
Jan Sedlák
dfc58f1b73 add ARM initial-setup test
ARM actually doesn't have "install" test, but in install matrix,
there is test whether ARM disk boots into initial_setup. HDD is saved
after this test for Base tests.

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D853
2016-05-18 14:04:45 +02:00
Adam Williamson
60ceeb469e whoops, fix install_done
I broke everything in an unreviewed commit! Go me.
2016-05-06 16:16:28 -07:00
Adam Williamson
92ba718de3 dnf-plugin-system-upgrade: don't use updates-testing, 10 mins
the updates-testing here was never meant to be permanent, it
was only added when the plugin was very new and we knew the
version in stable was busted. The test cases do not say to use
u-t, so let's not.

We've seen this test fail several times recently because of very
slow metadata downloads at this point, so let's give it longer
to run.
2016-05-06 13:37:19 -07:00
Adam Williamson
dc1f3dcdf3 wait_still_screen after install completes
This should be a better fix for the problem of the install_done
needle matching in the sidebar gradient while the button is
transitioning. We just assert_screen first, then once we hit it,
we wait_still_screen for 3 seconds, then assert_and_click the
button.
2016-05-06 13:15:13 -07:00
Adam Williamson
db34513524 ok, still fixing the g-i-s loop...but I know what was wrong now
the first param to assert_and_click is *the button number*, not
the wait time. So as soon as I set that for anything it was not
gonna work. So going back to the original approach but fixing
that, hope this works.
2016-05-05 23:20:02 -07:00
Adam Williamson
7c6ad4f29f damnit, i swear to god, this time 2016-05-05 19:07:40 -07:00
Adam Williamson
bdc69381ba one more fix to g-i-s loop, nearly there... 2016-05-05 18:09:39 -07:00
Adam Williamson
9297282b9e tweak mouse_set 2016-05-05 17:46:39 -07:00