Commit Graph

209 Commits

Author SHA1 Message Date
Adam Williamson
a067d0655f add a desktop notifications test
Summary:
this more or less covers desktop_error_checks and desktop_
update_notification, though it can't really distinguish
between them easily. All we know is that if both the live and
postinstall versions of this test pass, both of those tests
pass. Any fails will have to be investigated manually.

Test Plan:
Run the tests for both KDE and Workstation, see
what happens. Workstation will fail for F25 and Rawhide at
present, due to SELinux/abrt notifications.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D1004
2016-09-23 16:03:13 -07:00
Adam Williamson
e8b20ec73f add a desktop_update_graphical test
Summary:
Very similar to the CLI update test, but using the desktops'
update applications. This is based off the CLI update test
branch as it uses the shared functions that branch introduced.
We do not use the fake update packages, as they don't really
do anything useful for these tests; for dnf they can help us
distinguish between issues with the dnf mechanism and issues
with the repos, but we can't really tell that in the graphical
case. So we only use the python3-kickstart package here.

Test Plan:
Run the test on both KDE and GNOME and ensure it
performs as intended. I've been testing it on staging, so you
can see it there.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D1010
2016-09-22 11:38:51 -07:00
Adam Williamson
1cdd8e18b7 make sure we get logs from failed kickstart installs
Summary:
by waiting for the bootloader in _boot_to_anaconda rather than
_console_wait_login, we can ensure that we use the anaconda
post-fail hook and thus get logs uploaded when a kickstart
install fails.

Test Plan:
Run a kickstart install test that fails and check
anaconda logs get uploaded. Then run one that works and make
sure it...still works.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D1005
2016-09-20 10:51:51 -07:00
Jan Sedlák
c0b9bdb543 add anaconda rescue test on encrypted disk
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D995
2016-09-16 14:44:03 +02:00
Adam Williamson
4d3ed25d79 add missing screenshots for new font needles (whoops) 2016-09-13 21:08:29 -07:00
Adam Williamson
f26d70bc58 add some variant needles for new freetype in Rawhide
seems to affect the network spoke a lot but not anywhere else.
Odd. Anyhow, here we go. Keeping the old needles for now as
F25 still has the old rendering.
2016-09-13 13:02:05 -07:00
Adam Williamson
72074868a8 clean up lots of no-longer-used needles
Summary:
I've been wanting to do this for a while, but
https://github.com/os-autoinst/openQA/issues/786 is making it
difficult. Still, I think the quantity of needles is slowing
down openQA, so I'm doing it the old-fashioned way - looking
through test results and seeing what needles are actually used
now.

Test Plan:
Run full test suite for 24 Atomic, 25 and Rawhide and
make sure all tests still work. This is currently deployed on
staging (along with key-fixes) and I'm testing it there.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D994
2016-09-12 10:25:24 -07:00
Adam Williamson
e1ec1997af try to be safer when typing in X: slower, more checks
Summary:
the main thing this does is try and type slower in X - this
should cover nearly everywhere we type anything in X, and make
it type slower. We also add a bit more safety checking to some
old tests which didn't have it (mainly _do_install_and_reboot)
- wait_still_screen after typing to make sure all the keypresses
were registered before continuing.

This is an attempt to mitigate the problems we've seen where
the wrong text gets typed into the wrong places and the tests
break.

This branch is live on staging atm. It still has *some* issues,
but I do think it's an improvement.

Test Plan:
run the tests (probably several times), compare to
runs without the change, see if it's better or worse...

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D993
2016-09-12 10:24:30 -07:00
Adam Williamson
276e38b561 more variant software selection needles for BGO #771127
so what seems to be going on on the software selection screen
is some kind of GTK+ bug:

https://bugzilla.gnome.org/show_bug.cgi?id=771127

the radio button bullets don't always render correctly - I think
they're not always completing a transition they go through on
selection. I think they can get stuck in any state between 'grey
and small' and 'black and big', but for now these are the needles
I've managed to create from failures; we're missing a 'greyer
KDE' variant, but if that happens, we can add it. If the bug
gets fixed we should be able to drop all these.
2016-09-09 10:41:54 -07:00
Adam Williamson
2f5cc741f9 update fs_xfs and fs_ext4_preselected needles
these rarely get used any more. they're only used when we're
running the filesystem tests on a non-Server image, where the
default fs is ext4 not xfs. With the old nightly process, where
we only got a generic boot.iso, we used them all the time, but
now we get product-ized nightlies, we rarely do. But I did hit
them today running the universal tests with a generic boot.iso
I hand-built to test an anaconda update, so in case we need to
do that kind of thing again in future, we may as well update
the needles.
2016-09-08 13:46:26 -07:00
Adam Williamson
c92e94b116 some more variant needles for freeipa webui 2016-09-08 10:57:04 -07:00
Adam Williamson
2c644aa204 add a bunch of variant needles for cockpit and freeipa webuis 2016-09-08 09:48:23 -07:00
Jan Sedlák
db95bccd52 add anaconda text UI test
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D980
2016-09-07 10:34:54 +02:00
Adam Williamson
e56ab793de add another kde_selected needle variant
seems like the size of the radio bullet is very slightly
different? whatevers.
2016-09-03 17:42:36 -07:00
Adam Williamson
f99411e0cc add new font rendering needles for GNOME login 2016-09-01 10:41:22 -07:00
Adam Williamson
ef689e75a9 use compose repository (not master repo) for most tests
Summary:
we have a long-standing problem with all the tests that hit
the repositories. The tests are triggered as soon as a compose
completes. At this point in time, the compose is not synced to
the mirrors, where the default 'fedora' repo definition looks;
the sync happens after the compose completes, and there is also
a metadata sync step that must happen after *that* before any
operation that uses the 'fedora' repository definition will
actually use the packages from the new compose. Thus all net
install tests and tests that installed packages have been
effectively testing the previous compose, not the current one.

We have some thoughts about how to fix this 'properly' (such
that the openQA tests wouldn't have to do anything special,
but their 'fedora' repository would somehow reflect the compose
under test), but none of them is in place right now or likely
to happen in the short term, so in the mean time this should
deal with most of the issues. With this change, everything but
the default_install tests for the netinst images should use
the compose-under-test's Everything tree instead of the 'fedora'
repository, and thus should install and test the correct
packages.

This relies on a corresponding change to openqa_fedora_tools
to set the LOCATION openQA setting (which is simply the base
location of the compose under test).

Test Plan:
Do a full test run, check (as far as you can) tests run sensibly
and use appropriate repositories.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D989
2016-09-01 08:22:59 -07:00
Adam Williamson
ea2f71b2cb SDDM appearance changed again in Rawhide, add needles 2016-08-30 12:55:57 -07:00
Adam Williamson
9311146f95 drop dictionary error workaround
Summary:
the dictionary error bug was fixed some time back, so drop this
workaround for it.

Test Plan:
Run all tests for F25 and Rawhide and verify they don't need
this workaround any longer.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D988
2016-08-30 11:18:46 -07:00
Jan Sedlák
518f996b10 add changed english filtered needle 2016-08-30 15:57:52 +02:00
Adam Williamson
19fccde4f0 add a few more needle updates for font rendering changes 2016-08-29 15:23:24 -07:00
Adam Williamson
5d7b46cb9d widen reclaim_space_delete_btn area to make it unambiguous
this needle can actually also match the 'Delete all' button,
resulting in the test not doing what it should. openQA should
prefer the match closer to the area's location in the needle,
but https://openqa.fedoraproject.org/tests/31136 seems to show
this not happening, so let's make the match area wider in all
versions of the needle so it should no longer match the Delete
all button.
2016-08-29 11:00:53 -07:00
Adam Williamson
232dad604c add a bunch of needles for changed font rendering
font rendering has changed somehow in both Rawhide and F25
updates-testing, retake a bunch of needles for this. Some more
may be needed later in tests that are currently failed or
skipped for other reasons.
2016-08-25 16:01:26 -07:00
Adam Williamson
ecd7ce6b3f add changed font rendering next_button variant 2016-08-23 22:14:47 -07:00
Adam Williamson
1286eb79d7 tweak name of that last needle
this isn't actually f25 specific, I don't think...the tests
that failed are the Workstation network install tests, which
use updates-testing, so I think they're getting the same font
rendering change as just showed up in Rawhide. So these should
actually be correct for both branches, let's just leave the
name as the date.
2016-08-23 13:27:52 -07:00
Adam Williamson
038b2ceb25 add another gdm_input variant for current f25
something changed here again...
2016-08-23 12:22:02 -07:00
Adam Williamson
bc3402397d add a current rawhide join_domain needle
looks like some font change occurred at some point, this test
hasn't reached this point for a long time so I can't really pin
down what changed, so just name it for the date.
2016-08-11 18:25:46 -07:00
Adam Williamson
f62c68147b add new cockpit join link needle
recent cockpit changed the Join Domain from a button to a link,
add a needle.
2016-08-10 16:17:33 -07:00
Adam Williamson
e57c7c9805 update firefox download_complete needle screenshot
seems like the overlay of the little 'speech bubble' triangle
changed a bit, with a recent GTK+ or something I guess. Since
this test is quite new we shouldn't need to keep the old
version, so just replace the screenshot.
2016-08-09 10:48:44 -07:00
Jan Sedlák
6fdc9934d4 add new graphical_desktop_clean needle
See https://openqa.fedoraproject.org/tests/27661#step/_graphical_wait_login/11

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D962
2016-08-09 12:39:10 +02:00
Adam Williamson
a8ddc002f8 add QA:Testcase_desktop_browser test
Summary:
pretty simple stuff here. The distinction between 'firefox' and
'browser' is that the 'browser' needles I expect would also be
correct for other default browsers, while the 'firefox' needles
are specific to Firefox. We need '-kde' variants of some Firefox
needles where interface text is included, because the font is
Cantarell in GNOME but whatever the default 'sans' font is in
KDE - I suppose we should really use -thatfontsname rather than
-kde, but I can't think what it's called...

I couldn't do the 'log in to FAS' bit of the test since we don't
really have a sane way to provide a password while not exposing
it to the public.

Test Plan:
Run the test, check it works - for both KDE and
Workstation.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D938
2016-08-03 13:22:29 -07:00
Adam Williamson
a901fce4ab add QA:Testcase_FreeIPA_password_change test
Summary:
again, added as a non-fatal module for realmd_join_cockpit as
it's convenient to do it here. Also abstract a couple of ipa
bits into a new exporter package in the style of SUSE's
mm_network, rather than using ill-fitting class inheritance as
we have before - we should probably convert our existing class
based stuff to work this way.

Also a few minor tweaks and clean-ups of the other tests:

The path in console_login() where we detect login of a regular
user when we want root or vice versa and log out was actually
broken because it would 'wait' for the result of the 'exit'
command, which obviously doesn't work (as it relies on running
another command afterwards, and we're no longer at a shell).
This commit no longer actually uses that path, but I spotted
the bug with an earlier version of this which did, and we may
as well keep the fix.

/var/log/lastlog is an apparently-extremely-large sparse file.
A couple of times it seemed to cause tar to run very slowly
while creating the /var/log archive for upload on failure. It's
no use for diagnosing bugs, so we may as well exclude it from
the archive.

I caught cockpit webUI login failing one time when testing the
test, so threw in a wait_still_screen before starting to type
the URL, as we have for the FreeIPA webUI.

I also caught a timing issue with the openQA webUI policy add
step; the test flips from the Users screen to the HBAC screen
then clicks the 'add' button, but there's actually an identical
'add' button on *both* screens, so it could wind up trying to
click the one on the Users screen instead, if the web UI took
a few milliseconds to switch. So we throw in a needle match to
make sure we're actually on the HBAC screen before clicking the
button.

We make the freeipa_webui test a 'milestone' so that if the
new test fails, restoring to the last-known-good milestone
doesn't take so long; it actually seems like openQA can get
confused and try to cancel the test if restoring the milestone
takes a *really* long time, and wind up with a zombie qemu
process, which isn't good. This seems to avoid that happening.

Test Plan:
In the simple case, just run all the FreeIPA-related
tests on Fedora 24 (as Rawhide is broken) and make sure they all
work properly. To get a bit more advanced you can throw in an
`assert_script_run 'false'` in either of the non-fatal tests to
break it and make sure things go properly when that happens (the
last milestone should be restored - which should be right after
freeipa_webui, sitting at tty1 - and run properly; things are
set up so each test starts with root logged in on tty1).

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D935
2016-08-03 13:21:12 -07:00
Adam Williamson
aacd01ea8b add encrypted workstation upgrade tests (current and previous)
Summary:
This requires us to handle decryption each time we reboot in
the upgrade process, so factor that little block out into the
base class so we don't have to keep pasting it. It's also a
bit tricky to integrate into the 'catch a boot loop' code we
have to deal with #1349721, but I think this should work. There
is a matching openqa_fedora_tools diff to generate the disk
image.

Test Plan:
Run the tests, check that they work, run the other
upgrade and encrypted install tests and check they still work
properly too.

Reviewers: garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D922
2016-07-08 08:56:57 -07:00
Adam Williamson
31328c17a8 drop RHBZ #1346390 workaround needle
The bug seems fixed and the needle is causing false softfails
2016-07-05 16:46:21 -07:00
Adam Williamson
8e94dd3f16 add a login_sddm for updated F24 KDE
seems the text has moved about a bit and it's now matching the
workaround needle only, which we don't want.
2016-07-01 08:17:51 -07:00
Adam Williamson
b475c15c76 update some KDE needles for Rawhide
SDDM got a makeover, apparently, and the background color of
the panel has changed a bit.
2016-06-28 14:45:24 -07:00
Adam Williamson
e24c377b01 add FreeIPA web UI testing
Summary:
as a new, non-fatal test step in the cockpit enrolment test,
because it kinda fits in there; we have an enrolled system with
a web browser *right there*. This will require making the wiki
reporting stuff slightly cleverer so we can say 'report a pass
for this wiki test instance if this test step passed', but that
should be possible. Making this non-fatal means the rest of the
cockpit enrolment test will go ahead even if the freeipa web UI
fails.

The 'check if we can log in' stuff is identical to freeipa_
client_postinstall except with different user names, so we could
potentially factor that out somehow, but I couldn't think of a
super clean way to do it so for now it's just copied.

Note this diff is on top of the freeipa-realmd branch which
is for D894, it's not on top of develop.

Test Plan:
Run the modified test and see if it works. No other
tests are modified, so they should be OK.

Reviewers: garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D895
2016-06-28 12:01:31 -07:00
Jan Sedlák
8066beba91 add new GNOME login input
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D878
2016-06-27 12:17:45 +02:00
Adam Williamson
83828425e1 tweak install_lang needle to match Fedora-Atomic
now we're getting composes with the product name 'Fedora-Atomic'
this needle doesn't match them. Tweak it so it should match both
'Fedora' and 'Fedora-Atomic'.
2016-06-22 20:54:54 -07:00
Adam Williamson
9531beff2a new cockpit logs_notices_entry needle for Rawhide font change
something changed in Rawhide's font for this needle, so we need
a variant
2016-06-16 11:39:52 -07:00
Adam Williamson
b101042dae add a workaround needle for RHBZ #1346390
in current Rawhide, the desktop selector dropdown in SDDM is
blank; add a workaround needle to handle that.
2016-06-14 09:57:08 -07:00
Adam Williamson
3a76630a09 add similar variant for Russian layout needle 2016-06-13 15:36:59 -07:00
Adam Williamson
b5c3cd9a9c add variant us layout indicator needle
Somehow it seems this can look a bit different sometimes - see
https://openqa.fedoraproject.org/tests/22000 . The text seems
to be in a slightly different vertical position.
2016-06-13 15:23:25 -07:00
Adam Williamson
0da6652287 add NFS tests (and DHCP/DNS in the support server)
Summary:
Set up the support server to provide DHCP/DNS functionality and
an NFS server, providing a kickstart. Add a kickstart test just
like the other root-user-crypted-net kickstart tests except it
gets the kickstart from the support server via NFS. Also add NFS
repository tests and a second support server for Server-dvd-iso
flavor: this test must run on that flavor to ensure that packages
are actually available. The support server just mounts the
attached 'DVD' and exports it via NFS.

Note we don't need to do anything clever to avoid IP conflicts
between the two support servers, because os-autoinst-openvswitch
ensures each worker group is on its own VLAN.

As part of adding the NFS repo tests, I did a bit of cleanup,
moving little things we were repeating a lot into anacondatest,
and sharing the 'check if the repo was used' logic between all
the tests (by making it into a test step that's loaded for all
of them). I also simplified the 'was repo used' checks a bit,
it seems silly to run a 'grep' command inside the VM then have
os-autoinst do a grep on the output (which is effectively what
we were doing before), instead we'll just use a single grep
within the VM, and clean up the messy quoting/escaping a bit.

Test Plan:
Run all tests - at least all repository tests - and
check they work (make sure the tests are actually still sane,
not just that they pass). I've done runs of all the repo tests
and they look good to me, but please double-check. I'm currently
re-running the whole 24-20160609.n.0 test on staging with these
changes.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D888
2016-06-13 08:42:30 -07:00
Adam Williamson
7a8ae3a357 add an iscsi test, and a support_server test to support it
Summary:
this is following a SUSE model for tests where we need a server
end but don't want setting up the server to constitute a real
test in itself, we want it to be stable. The 'support_server'
test just boots a pre-built (by createhdds) disk image, sets up
networking, and runs the iSCSI server.

To run the iSCSI test we need to handle networking config in
anaconda (or we would need to set the support server up as a
DHCP server, which may be worth considering), so this adds that.
We also need to be able to specify the target device for a
volume in custom partitioning, so this adds that too.

Test Plan:
Build the necessary support server disk image (use
D883), then run the test and make sure it works. Also make sure
all other tests continue to work.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D884
2016-06-09 08:43:46 -07:00
Adam Williamson
046d7e128e add a workaround needle for KDE with upstream kicker (1344118)
Rawhide KDE has an upstream icon for the kicker; softfail on
this so other tests can run.
2016-06-08 12:53:16 -07:00
Adam Williamson
7606131d09 add updated kde anaconda installer icon for current Rawhide
the icon changed
2016-06-08 11:39:24 -07:00
Adam Williamson
66fc3cc7d4 add a cockpit realmd FreeIPA join test
Summary:
This requires a few other changes:

* turn clone_host_resolv into clone_host_file, letting you clone
  any given host file (cloning /etc/hosts seems to make both
  server deployment and client enrolment faster/more reliable)
* allow loading of multiple POSTINSTALL tests (so we can share
  the freeipa_client_postinstall test). Note this is compatible,
  existing uses will work fine
* move initial password change for the IPA test users into the
  server deployment test (so the client tests don't conflict over
  doing that)
* add GRUB_POSTINSTALL, for specifying boot parameters for boot of
  the installed system, and make it work by tweaking _console_wait
  _login (doesn't work for _graphical_wait_login yet, as I didn't
  need that)
* make the static networking config for tap tests into a library
  function so the tests can share it
* handle ABRT problem dirs showing up in /var/spool/abrt as well
  as /var/tmp/abrt (because the enrol attempt hits #1330766 and
  the crash report shows up in /var/spool/abrt, don't ask me why
  the difference, I just work here)
* specify the DNS servers from the worker host's resolv.conf as
  the forwarders for the FreeIPA server when deploying it; if we
  don't do this, rolekit defaults to using the root servers as
  forwarders(!) and thus we get the public, not phx2-appropriate,
  results for e.g. mirrors.fedoraproject.org, some of which the
  workers can't reach, so PackageKit package install always fails
  (boy, was it fun figuring THAT mess out)

Even after all that, the test still doesn't actually pass, but
I'm reasonably confident this is because it's hitting actual bugs,
not because it's broken. It runs into #1330766 nearly every time
(I think I saw *one* time the enrolment actually succeeded), and
seems to run into a subsequent bug I hadn't seen before when
trying to work around that by trying the join again (see
https://bugzilla.redhat.com/show_bug.cgi?id=1330766#c37 ).

Test Plan:
Run the test, see what happens. If you're really lucky,
it'll actually pass. But you'll probably run into #1330766#c37,
I'm mostly posting for comment. You'll need a tap-capable openQA
instance to test this.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D880
2016-06-07 13:00:39 -07:00
Adam Williamson
46c6ed3cdb add needle for qwerty user logged into console
we may use this for multiple languages in future though it's
for Russian only ATM, so placing in the generic console/ dir.
2016-06-02 08:23:02 -07:00
Adam Williamson
aba2814611 add cockpit_default and cockpit_basic tests
Summary:
This adds tests for the Server_cockpit_default and cockpit_basic
test cases. Some notes: I was initially thinking of combining
these into a single test with multiple test modules and coming
up with a system for doing wiki reporting based on individual
test module status, but because we'll also want to do a cockpit
FreeIPA enrol test, I decided against it. We don't really want
to combine all three because then we would skip the cockpit
tests whenever FreeIPA server deployment failed, which isn't
ideal. So since we'll need a separate FreeIPA enrolment test
anyway it doesn't really make sense to go to the trouble of
designing a system for loading multiple postinstall tests (though
I have an idea for that!) and a per-module wiki reporting system.

This was the most minimal and hopefully reliable method for
running Cockpit from a stock Server install that I could think
of. An alternative approach would be to have, say, the most
recent stable Workstation live as a 'stock' asset and have two
tests, one which runs a stock Server install and just waits and
another which boots the live image and accesses the cockpit
running on the other box, but that seems a bit over-complex. It
is not possible to have dependencies between tests for different
ISOs, in case you were wondering about having a Workstation live
test which runs parallel with a Server DVD test, we can't do
that. One funny thing is the font that winds up getting used for
the desktop, but I don't *think* that should be a problem.

Picking needles was a bit tricky; any improvement suggestions
are welcome. I'm hoping it turns out to be safe to rely on some
dbus log messages being present; I think logging into Cockpit
triggers activation of the realmd dbus interface, so there
*should* always be some messages related to that. An alternative
would just be to match on a sliver of the dark grey table header
and the light grey row beneath it and assume that'll always be
the first message (whatever the message is), but then we have to
find some area of the message details screen which is always
present for any message, and it just seems a tad more likely to
result in false passes. Similary I'm making an assumption that
auditd is always going to show up on the first page of the
Services screen and the details screen will always show that
'loaded...enabled' text.

Test Plan:
Run the tests and see if they work! See
https://openqa.stg.fedoraproject.org/tests/21373 and
https://openqa.stg.fedoraproject.org/tests/21371 for my tests.

Reviewers: garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D874
2016-06-01 09:05:33 -07:00
Adam Williamson
a1d240b135 add needles to treat 'Custom Operating System' as minimal
Summary:
the Server DVD now just has 'Fedora Server' and 'Custom
Operating System' environments. Custom is basically minimal.
So we can use the DVD for 'universal' testing again, these
needles match the anaconda_minimal tags.

Test Plan:
Run the 'universal' tests on a DVD ISO with these
needles added, test that they work OK and use the 'Custom' env.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D844
2016-05-20 12:56:44 -07:00
Adam Williamson
a4f3267534 add Russian install test
Summary:
Requires new needles and test suite and job template, plus a
few tweaks to handle 'switched' keyboard layouts (so we use the
switched layout in the username and password).

Test Plan:
Run the test and see that it...fails. But that's OK!
It's a genuine bug: RHBZ #1333998 . At least make sure it gets
to that point and no other tests have broken and all the needles
look sane.

Reviewers: garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D846
2016-05-20 07:52:55 -07:00
Jan Sedlák
6dcb20e022 add new french anaconda needles
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D855
2016-05-20 10:29:40 +02:00
Adam Williamson
8cb5a494ca move new sddm needles into kde directory 2016-05-19 06:33:39 -07:00
Adam Williamson
a71f821983 add updated sddm login needle
Font rendering has changed since this was taken. We still need
the old one for pre-upgrade cases.
2016-05-18 08:27:16 -07:00
Jan Sedlák
dfc58f1b73 add ARM initial-setup test
ARM actually doesn't have "install" test, but in install matrix,
there is test whether ARM disk boots into initial_setup. HDD is saved
after this test for Base tests.

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D853
2016-05-18 14:04:45 +02:00
Josef Skladanka
0d488890ba Organize needles to directories (step 2/2)
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D744
2016-05-13 13:43:31 +02:00
Josef Skladanka
ad044c0b3f Organize needles to directories (step 1/2)
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D744
2016-05-13 13:43:25 +02:00
Adam Williamson
2df55efb49 add desktop_terminal test, refactor test loading a bit
Summary:
I really just want to add the desktop_terminal test, but I think
this refactor is in order now. It splits up loading of the
various test phases (much as SUSE do it) and allows us to run
the post-install tests without the install tests, for e.g. I
tweaked things to allow the upgrade tests to use the existing
_wait_login tests for final login and combine the two upgrade
postinstall tests into one simple one.

This comes with a bit of a behaviour change to make graphical
wait login behave the same as console wait login: it will log
in unless USER_LOGIN is set to 'false'. Previously it only
logged in if both USER_LOGIN and USER_PASSWORD were set, which
I don't think ever happened in a graphical test, so we never
actually did a graphical login. The intent here is we should do
a login on the default_install tests. That's going a bit beyond
the test case, but it seems like a reasonable thing to test. We
can set USER_LOGIN to false if we don't want to do it.

Test Plan:
Do a full test run, make sure the new tests work and
no old tests break.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D839
2016-05-05 16:39:47 -07:00
Adam Williamson
559b5f07b9 update resize slider needle for GTK+ 3.21.1 2016-05-04 17:36:31 -07:00
Adam Williamson
f59343403a add FreeIPA server role deploy and kickstart enrolment tests
Summary:
These require openQA tap networking to allow the server and
client boxes to communicate, and require masquerading (NAT) so
the server at least can reach a repository (dnf/rolekit really,
really do not want to work without a repo connection).

They use the 'parallel' test support to have the server deploy
run first while the client enrol test waits at the grub menu
until the server is done before it goes ahead.

This is all deployed and working on stg. The really tricky bit
was getting all the openvswitch and firewall config right in
ansible.

We *could* do the server deploy test as a follow-on from the
default install test to save the install, but then we'd have to
teach it to change the hostname and set up static networking
post-install. I'm not sure if it's worth doing that.

This requires the corresponding openqa_fedora_tools commit that
adds the hard disks (containing the kickstarts - it's possible
to get them from remote during install, but we have to set up
name resolution or hard code the IP of the server).

Test Plan:
Deploy this and the openqa_fedora_tools commit,
generate the disks, configure the networking (good luck! See
the docs in openqa_fedora_tools) and see if you can run the
tests. If you're using Docker, uh...sorry. You somehow need to
set things up so the workers can use tap interfaces that can
talk to each other and are NATed to the outside world. Have fun.
I can talk you through it on IRC...

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D831
2016-05-04 11:53:11 -07:00
Jan Sedlák
d4466100d4 add necessary needles and fix code for KDE upgrade tests
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D837
2016-05-04 09:58:35 +02:00
Adam Williamson
100c133dda include some grey in the reboot button match
Summary:
There's a weird problem that we hit occasionally only on the
Server image, see e.g.:
https://openqa.fedoraproject.org/tests/14300
I *think* what's happening (thanks hergertme from #gtk+ IRC for
help with this) is that when the 'install done!' text and the
reboot button appears, there's a short transition for the button
so it doesn't look *exactly* like the screenshot right away. If
openQA's needle match loop happens to kick in and check the
match right after the text and button appear, while the
transition is happening, it decides that some random area of the
gradient down the left hand side of the screen matches *better*
than the button, so it 'clicks' that, and we don't get a reboot.
If the openQA needle match loop happens to kick in a millisecond
or whatever later, after the button is done transitioning, the
button is now a 100% match and the bug doesn't happen. This only
happens on Server because the left-hand side gradient differs in
each flavor, and only the Server one just happens to have a spot
which is nearly identical to the needle area.

So to avoid that, let's include a bit of the grey space next to
the button. This is a bit more fragile against GTK+ changes to
button design or padding or whatever, but it will at least avoid
the match issue.

The other option would be to try and hit the button with tab and
enter or something, I guess.

Test Plan:
bit hard since this is an intermittent bug, I guess
run install_default on a Server image a bunch of times and see if
it always works.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D824
2016-04-26 19:08:51 -07:00
Adam Williamson
da58dc694b add white variant of kde desktop runner, remove old one
according to rdieter the runner dialog could be white or grey
background, we should match on either. One is probably a bug
but it's best to just work with it. Removing the February
version without the 'search' text.
2016-04-25 12:21:52 -07:00
Adam Williamson
31c17235f5 KDE desktop runner changed *again*
clean up the old needles while we're at it
2016-04-19 08:15:07 -07:00
Adam Williamson
90b5acf72a handle 'weak password' due to dictionary load error
Summary:
Rawhide currently seems to have a bug in spell check dictionary
load, which causes the test to fail as it requires another Done
click. So add a workaround needle that handles this case.

Test Plan:
Apply the patch, run some tests, see if they work. I
did a test run on staging:
https://openqa.stg.fedoraproject.org/tests/13331

Reviewers: garretraziel, jskladan

Reviewed By: jskladan

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D815
2016-04-14 23:37:22 -07:00
Adam Williamson
3dca7cbf55 add new Workstation welcome app needles
the colors in this seem to change slightly a lot. I think
GNOME does something clever based on the background image or
something? Anyway. New needles.
2016-04-09 07:36:55 -07:00
Adam Williamson
77c07ae803 remove gtk+ 3.20 bug workaround needles
this seems to be resolved in both F24 and Rawhide now.
2016-04-09 07:19:15 -07:00
Adam Williamson
591b153238 revise updates.img test to work with new anaconda (T759)
Summary:
per details in T759, the 'unipony' updates image we use to test
the updates image features doesn't work with latest anaconda (f24
and Rawhide). I've built a new updates image which uses a neat
anaconda feature that allows you to override CSS with a file in
a special location; it sets the background for disk capacity
texts on the INSTALLATION DESTINATION spoke to be pink. This
lets us use a simple needle that just looks for a pink blob on
that spoke, on the basis that it's unlikely there'll ever be a
pink blob there for any other reason, so if there is one, the
updates image worked. There will be an accompanying tools diff
to change the updates disk image to use the new updates image.

Test Plan:
Do a test run and check the updates image tests pass
and no other tests are broken. You'll need to pull in the tools
diff and re-generate the updates disk image to check that test,
the scsi_updates_img test should work with just this diff.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D799
2016-04-01 08:00:47 -07:00
Adam Williamson
9992375d0a needles: disk selection GTK+ 3.20 workaround, part select update
Some needle changes to account for latest Rawhide. There seems
to be a bug between anaconda and GTK+ 3.20 causing text in the
disk selection widget not to be styled correctly: this commit
adds needles for that which are tagged as 'workaround', meaning
openQA will complete the tests but mark them as requiring a
workaround to work ('soft fail'). The bug is #1322036. There is
also an intentional change to the volume list in custom part;
the mount point names are now in dark grey rather than black (in
fact this has always been intended, but it's been broken for a
long time, according to davidshea). I cleaned up the part select
needles at the same time; the new ones have the non-variant
names, as davidshea says the same change is coming to F24 and
the original needles haven't matched for months. The F23 Atomic
install test doesn't hit any of these needles, so we don't need
to keep the original needles around for that. The most recent
variants are kept, as they'll be needed for F24 tests until the
new anaconda build goes stable for F24. The 'freetype262'
variant of part_select_swap hasn't matched for two months and is
dropped.
2016-03-29 11:37:55 -07:00
Jan Sedlák
49411aeb97 update "KDE selected" needle in software selection
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D781
2016-03-21 15:00:16 +01:00
Jan Sedlák
d4a7124ff6 add english filtered cantarell22 needle 2016-03-15 10:36:38 +01:00
Adam Williamson
0fdd706734 needles: the resize slider changed yet again (GTK+ 3.19.11) 2016-03-09 17:57:15 -08:00
Adam Williamson
0305c5fcd0 anaconda crash reporting: update needles, click Report again
Summary:
First off, this revises the anaconda crash handling needles a
bit. We ditch gtk3195 and update anaconda_error to reflect
current F24/Rawhide. We keep the old anaconda_error around for
now as anaconda_error-23, to handle crashes in the F23 two-week
Atomic nightlies. We also add an 'early' variant, which is for
when (I think) the installer crashes very early, before it's
loaded in GTK+ settings; when that happens, the dialog uses a
different font. The screenshot comes from a recent Rawhide test
that crashed.

We also restore the anaconda `post_fail_hook` code to click
the Report button when a crash happens. This was erroneously
removed in D637. Before the Report button is clicked, the
`anaconda-tb` file exists but the libreport stuff in `/var/tmp`
does not. By removing this, we lost the libreport bits from
the uploaded files, which makes it harder to report crashes. So
let's add it back.

Finally we fix the actual tarring and uploading of `/var/tmp`;
also in D637 this got broken because it was being tarred up in
whatever directory the commands happened to be running in, but
we were still trying to upload it from `/var/tmp`.

https://openqa.stg.fedoraproject.org/tests/8444 was run with
these changes, and has `/var/tmp` correctly uploaded.

Test Plan:
Run some test that crashes, make sure the crash
handling all works correctly.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D768
2016-03-08 08:10:51 -08:00
Adam Williamson
2c1890c52e revise anaconda_install_done needles
GTK+ 3.19.10(?) has changed things in current F24/Rawhide;
weirdly, done-freetype262 seems to match sometimes but not
always. Add a new needle from a current screenshot but keep
freetype262 around for now just in case. Drop some variants
that should never be needed any more, rename the original
needle to -23 to mark that we're only keeping it around for
the F23 two-week Atomic tests. For now keep french-cantarell20
as the 'official' French needle, we'll see if it hits any
failures when the tests are working a bit better.
2016-03-07 16:57:20 -08:00
Adam Williamson
abbba24ca4 add french user created needle for cantarell22 2016-02-18 13:33:23 -08:00
Adam Williamson
684f7b4e5e add some GTK+ 3.19.9 / Cantarell 0.0.22 needles 2016-02-18 12:49:11 -08:00
Adam Williamson
741883f00b add needle for newest KDE desktop runner 2016-02-18 12:45:33 -08:00
Jan Sedlák
596772f163 fix Cantarell 0.0.21 also for english filtered needles
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D739
2016-02-09 10:21:01 +01:00
Adam Williamson
41adcd87d4 add needle variants for GTK+ 3.19.8
it changes radio buttons and checkboxes a bit.
2016-02-04 02:16:03 +01:00
Adam Williamson
a0f75b4227 needle update for Cantarell 0.0.21 and GTK+ 3.19.7
mostly cantarell changes. GTK+ changes are bugs, so put 'bug'
in the name to remind us of this. There's one non-typical
change, to main_hub_installation_source; I removed the match
on the text label of the spoke entirely instead of updating it,
in line with other similar needles, it's unnecessary and only
serves to make the needle more fragile.
2016-01-21 17:58:01 -08:00
Adam Williamson
f855516818 new needles for cantarell 0.0.20 / GTK+ 3.19.6
both of these landed in Rawhide overnight and busted stuff, so
new needles!
2016-01-12 17:07:09 -08:00
Jan Sedlák
7b3bb5c252 add freetype262 needle for english_filtered tag
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D706
2015-12-23 08:45:54 +01:00
Adam Williamson
13f58b0426 add GTK+ 3.19.5 variant of GNOME live initial screen needle 2015-12-18 07:58:51 -08:00
Adam Williamson
d06f775d5c add a bunch of needle variants for GTK+ 3.19.5
3.19.5 seems to change the colors of buttons slightly and also
the rendering of checkboxes/radio buttons, they're now placed
somewhat differently relative to their labels.
2015-12-17 11:59:11 -08:00
Adam Williamson
43e655eae4 add freetype 2.6.2 variants of KDE package set needles
garretraziel took these with F23, I think. Theoretically we
could just have these ones and ditch his, as we're not running
KDE install tests on F23, but we might want to discuss that kind
of question more generally, so for now let's just add the
variant.

We may need more variants for the post-install matches, but the
test doesn't actually get that far ATM because of dependency
issues in the KDE package set, so I'll deal with that once we
can actually do a successful KDE install.
2015-12-08 12:03:53 -08:00
Adam Williamson
f34d01ee01 add BIOS font variant for 'user logged in' needle
Summary:
As I figured out a while back, the seabios firmware font is
nearly the same as our console font (eurlatgr) but not quite,
significantly its @ character is different (shorter). We
already have a root_logged_in variant for the firmware font,
but we now need one for user_logged_in as well. The Atomic
installer image has had the Linux console fonts stripped from
its payload, so it's now using the firmware font, and user_
logged_in isn't matching - see
https://openqa.fedoraproject.org/tests/461 .

Test Plan:
Run the cloud_atomic_canned test (on an F23
post-release nightly, as it's not appearing for Rawhide
nightlies ATM) and confirm it now passes.

Reviewers: garretraziel, jskladan

Reviewed By: jskladan

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D684
2015-12-08 08:05:46 -08:00
Jan Sedlák
fd2ed44162 add KDE package set testcase
add KDE package set testcase - this only adds needles and
new testcase configuration, because with needles cleanup from D670,
it should work without change in code.

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D673
2015-12-08 11:49:38 +01:00
Jan Sedlák
6e801bcd52 remove ENV-DESKTOP- tag, set DESKTOP- tag instead where necessary
Let's use the same principle with DESKTOP tag as we use with
LANGUAGE tag. Where there are any GNOME-only (or KDE-only) needles, tag
them with `DESKTOP-gnome` and delete then when different `DESKTOP` value
is specified. This will help with `QA:Testcase_Package_Sets_KDE_Package_Install`
testcase - we will be able to use almost the same code and check that KDE
got installed properly.

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D670
2015-12-08 11:32:48 +01:00
Adam Williamson
163a8fd94a make 'first partition' needles more reliable / universal
Summary:
The 'reclaim_space_first_partition' needles aren't very good.
Using the area with the filesystem means we need to duplicate
the needle for NTFS, and the needles in fact also match the
*second* partition in most cases, sometimes causing the tests
to delete the second partition instead of the first and fail.
This design should work better: a single needle should match
all filesystems, and by including some of the highlighted top
row in the match, we should never match on the second row.
Including 'da' in the needle makes this really 'first disk,
first partition', but that's all the cases we currently use
it for. We don't really need this needle to work hard to make
sure we're on the right screen, as the subsequent needles
(which match buttons that only exist on this screen) do that
job fine.

We drop some needles that should no longer be necessary: F23
should now be using the cantarell17 version of the needle,
and F24 the freetype262 needle. We don't run installer tests
for any other release.

We could tweak second_partition needle to match, but nothing
is actually using it anyway.

Test Plan:
Do a full test and run and make sure the tests
which use this needle work, and always select the correct
partition.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D681
2015-12-07 09:41:49 -08:00
Adam Williamson
2e42e6e478 one more freetype 2.6.2 needle (desktop_clean) 2015-12-05 14:09:52 -08:00
Adam Williamson
301b1e3177 one more freetype 2.6.2 needle - missed this one 2015-12-04 14:53:48 -08:00
Adam Williamson
31513e4e89 update needles for freetype 2.6.2
freetype 2.6.2 changes font rendering quite significantly.
Most text needles needed redoing. Committing without review
so maybe we finally get some damn useful test results...
2015-12-04 13:43:40 -08:00
Adam Williamson
a9c45df445 add needle for lighter-coloured resize slider
Summary: seems like this got lighter for F24 too.

Test Plan:
Run the shrink tests and see if they get past this
screen.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D666
2015-11-27 08:29:58 -08:00
Jan Sedlák
7240ce774f add custom partitioning xfs tests
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D663
2015-11-26 13:50:45 +01:00
Adam Williamson
d3b4e30595 don't match on left border of language list (french filtered)
Summary:
this needle seems to be fragile not because of cantarell in
fact, but because the difference between the the left edge of
the language box and the 'F' is not consistent. So, let's just
not include it in the match.

Test Plan:
Check if French install test works (see test #8847
failure in BOS).

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D661
2015-11-24 08:35:19 -08:00
Jan Sedlák
cf6fa5d957 another needles fixing
This adds another Cantarrell 1.8 missing needle as well as
lighter variant of Done button (it's perhaps caused by displayed
warning about missing swap partition on no_swap test?). With this
in place, all tests on Rawhide passes except of KDE (but it seems
that KDE Live images are borked) and desktop upgrade tests.

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D651
2015-11-19 10:11:07 +01:00
Adam Williamson
3ecef54b51 explicitly check for DNF "system is not ready for upgrade"
Summary:
Instead of sitting there waiting 6000 seconds twice, when DNF
explicitly tells us it failed, just die.

This is why we haven't been getting proper compose check reports
lately; the upgrade tests are failing, waiting 6000 seconds to
time out, then being cloned and tried again, waiting another 6000
seconds. This is just barely going beyond check-compose's 8 hour
wait limit, as it's some time before the upgrade tests even get
started (they're low in the priority list). We're still going to
have that problem if the tests fail any other way, but this at
least catches that case.

Test Plan:
Run the upgrade tests and see that they fail quicker
(assuming the dependency problems they're dying on aren't fixed).
Maybe also do a 22-23 upgrade test and check it still succeeds
properly.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D650
2015-11-18 11:17:49 -08:00
Jan Sedlák
df7a4582d0 add another cantarell18 needle
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D646
2015-11-13 10:05:02 +01:00
Adam Williamson
29b9ffc405 add Cantarell 0.0.18 variant needles
Cantarell 0.0.18 showed up in Rawhide recently and changes
rendering again, so we need some more variant needles. Once
Cantarell change quiets down a bit we'll have to clean these
all up.
2015-11-10 12:35:30 -08:00
Jan Sedlák
da7bdda462 add azerty user logged in needle
Since we are now logging to user account every time,
I've encountered new bug - we are missing needle for azerty
user logged in. This DR adds it.

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D640
2015-11-09 09:49:21 +01:00