Commit Graph

468 Commits

Author SHA1 Message Date
Adam Williamson
a4f3267534 add Russian install test
Summary:
Requires new needles and test suite and job template, plus a
few tweaks to handle 'switched' keyboard layouts (so we use the
switched layout in the username and password).

Test Plan:
Run the test and see that it...fails. But that's OK!
It's a genuine bug: RHBZ #1333998 . At least make sure it gets
to that point and no other tests have broken and all the needles
look sane.

Reviewers: garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D846
2016-05-20 07:52:55 -07:00
Jan Sedlák
6dcb20e022 add new french anaconda needles
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D855
2016-05-20 10:29:40 +02:00
Adam Williamson
8cb5a494ca move new sddm needles into kde directory 2016-05-19 06:33:39 -07:00
Adam Williamson
a71f821983 add updated sddm login needle
Font rendering has changed since this was taken. We still need
the old one for pre-upgrade cases.
2016-05-18 08:27:16 -07:00
Jan Sedlák
dfc58f1b73 add ARM initial-setup test
ARM actually doesn't have "install" test, but in install matrix,
there is test whether ARM disk boots into initial_setup. HDD is saved
after this test for Base tests.

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D853
2016-05-18 14:04:45 +02:00
Josef Skladanka
0d488890ba Organize needles to directories (step 2/2)
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D744
2016-05-13 13:43:31 +02:00
Josef Skladanka
ad044c0b3f Organize needles to directories (step 1/2)
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D744
2016-05-13 13:43:25 +02:00
Adam Williamson
60ceeb469e whoops, fix install_done
I broke everything in an unreviewed commit! Go me.
2016-05-06 16:16:28 -07:00
Adam Williamson
9871c48433 allow failure when uploading anaconda logs
Summary:
This requires a PR I sent upstream:
https://github.com/os-autoinst/os-autoinst/pull/490
This change is in os-autoinst -10. However, with older packages
it won't crash or anything, it'll just behave as before.

With the change, this allows log upload to fail, so if one of
the logs is missing, the hook doesn't immediately die and fail
to upload the rest of the logs. Various anaconda logs are not
always present: the DNF logs are not present for Atomic or live
installs, and the X.log and syslog are not present for live
installs. Adding a fail-tolerant mode to upstream upload_logs
seemed a better option than testing for the existence of each
log file prior to uploading it, or adding a bunch of GET_VAR
calls to try and figure out which log files 'should' exist.

Test Plan:
Run an Atomic or live install test that fails, and
check what logs get uploaded. You can just test a current
Rawhide Atomic installer ISO, as they're crashing right now.
Without this patch (and the os-autoinst update) the hook dies
when it tries to upload dnf.log, so the traceback and /var/tmp
archive don't get uploaded; with this patch all the present
logs should get uploaded. Compare:
https://openqa.fedoraproject.org/tests/14834
https://openqa.stg.fedoraproject.org/tests/15371
(I tested this out on staging).

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D834
2016-05-06 13:44:06 -07:00
Adam Williamson
92ba718de3 dnf-plugin-system-upgrade: don't use updates-testing, 10 mins
the updates-testing here was never meant to be permanent, it
was only added when the plugin was very new and we knew the
version in stable was busted. The test cases do not say to use
u-t, so let's not.

We've seen this test fail several times recently because of very
slow metadata downloads at this point, so let's give it longer
to run.
2016-05-06 13:37:19 -07:00
Adam Williamson
dc1f3dcdf3 wait_still_screen after install completes
This should be a better fix for the problem of the install_done
needle matching in the sidebar gradient while the button is
transitioning. We just assert_screen first, then once we hit it,
we wait_still_screen for 3 seconds, then assert_and_click the
button.
2016-05-06 13:15:13 -07:00
Adam Williamson
db34513524 ok, still fixing the g-i-s loop...but I know what was wrong now
the first param to assert_and_click is *the button number*, not
the wait time. So as soon as I set that for anything it was not
gonna work. So going back to the original approach but fixing
that, hope this works.
2016-05-05 23:20:02 -07:00
Adam Williamson
7c6ad4f29f damnit, i swear to god, this time 2016-05-05 19:07:40 -07:00
Adam Williamson
bdc69381ba one more fix to g-i-s loop, nearly there... 2016-05-05 18:09:39 -07:00
Adam Williamson
9297282b9e tweak mouse_set 2016-05-05 17:46:39 -07:00
Adam Williamson
db78559a91 another attempt to fix the damn g-i-s handler 2016-05-05 17:44:49 -07:00
Adam Williamson
79bc1e515d tweak the g-i-s handling a bit
grr, this is annoying...it fails in upgrade tests, let's try to
be more forgiving
2016-05-05 17:13:21 -07:00
Adam Williamson
880bc477a0 fix mistake: graphical_login still time should stay 30 secs
I didn't mean to drop it to 15, only meant to extend the timeout
to 90.
2016-05-05 16:41:08 -07:00
Adam Williamson
2df55efb49 add desktop_terminal test, refactor test loading a bit
Summary:
I really just want to add the desktop_terminal test, but I think
this refactor is in order now. It splits up loading of the
various test phases (much as SUSE do it) and allows us to run
the post-install tests without the install tests, for e.g. I
tweaked things to allow the upgrade tests to use the existing
_wait_login tests for final login and combine the two upgrade
postinstall tests into one simple one.

This comes with a bit of a behaviour change to make graphical
wait login behave the same as console wait login: it will log
in unless USER_LOGIN is set to 'false'. Previously it only
logged in if both USER_LOGIN and USER_PASSWORD were set, which
I don't think ever happened in a graphical test, so we never
actually did a graphical login. The intent here is we should do
a login on the default_install tests. That's going a bit beyond
the test case, but it seems like a reasonable thing to test. We
can set USER_LOGIN to false if we don't want to do it.

Test Plan:
Do a full test run, make sure the new tests work and
no old tests break.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D839
2016-05-05 16:39:47 -07:00
Adam Williamson
e5d169b67c set worker_class 'tap' for IPA tests
this will make them only-run on the tap-capable hosts in infra.
2016-05-05 14:42:16 -07:00
Adam Williamson
559b5f07b9 update resize slider needle for GTK+ 3.21.1 2016-05-04 17:36:31 -07:00
Adam Williamson
f59343403a add FreeIPA server role deploy and kickstart enrolment tests
Summary:
These require openQA tap networking to allow the server and
client boxes to communicate, and require masquerading (NAT) so
the server at least can reach a repository (dnf/rolekit really,
really do not want to work without a repo connection).

They use the 'parallel' test support to have the server deploy
run first while the client enrol test waits at the grub menu
until the server is done before it goes ahead.

This is all deployed and working on stg. The really tricky bit
was getting all the openvswitch and firewall config right in
ansible.

We *could* do the server deploy test as a follow-on from the
default install test to save the install, but then we'd have to
teach it to change the hostname and set up static networking
post-install. I'm not sure if it's worth doing that.

This requires the corresponding openqa_fedora_tools commit that
adds the hard disks (containing the kickstarts - it's possible
to get them from remote during install, but we have to set up
name resolution or hard code the IP of the server).

Test Plan:
Deploy this and the openqa_fedora_tools commit,
generate the disks, configure the networking (good luck! See
the docs in openqa_fedora_tools) and see if you can run the
tests. If you're using Docker, uh...sorry. You somehow need to
set things up so the workers can use tap interfaces that can
talk to each other and are NATed to the outside world. Have fun.
I can talk you through it on IRC...

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D831
2016-05-04 11:53:11 -07:00
Jan Sedlák
d4466100d4 add necessary needles and fix code for KDE upgrade tests
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D837
2016-05-04 09:58:35 +02:00
Adam Williamson
ba90ef631a fix installedtest post-fail hook when no /var/tmp/abrt
Summary:
you can't use validate_script_output like this, it doesn't just
return a bool, it actually kills the test if the output doesn't
validate. So any time we didn't have anything in /var/tmp/abrt,
the post_fail_hook would die at that point and not upload the
contents of /var/log as we wanted it to.

Note that even script_output will kill the test if the return
code is >0. That's OK in this case, but always be careful with
these subroutines.

Test Plan:
Run a test that fails in post-install but doesn't
produce anything in /var/tmp/abrt (you may have to artificially
break something). Check that it now uploads /var/log. I noticed
this while working on FreeIPA tests...

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D830
2016-04-27 09:36:28 -07:00
Adam Williamson
100c133dda include some grey in the reboot button match
Summary:
There's a weird problem that we hit occasionally only on the
Server image, see e.g.:
https://openqa.fedoraproject.org/tests/14300
I *think* what's happening (thanks hergertme from #gtk+ IRC for
help with this) is that when the 'install done!' text and the
reboot button appears, there's a short transition for the button
so it doesn't look *exactly* like the screenshot right away. If
openQA's needle match loop happens to kick in and check the
match right after the text and button appear, while the
transition is happening, it decides that some random area of the
gradient down the left hand side of the screen matches *better*
than the button, so it 'clicks' that, and we don't get a reboot.
If the openQA needle match loop happens to kick in a millisecond
or whatever later, after the button is done transitioning, the
button is now a 100% match and the bug doesn't happen. This only
happens on Server because the left-hand side gradient differs in
each flavor, and only the Server one just happens to have a spot
which is nearly identical to the needle area.

So to avoid that, let's include a bit of the grey space next to
the button. This is a bit more fragile against GTK+ changes to
button design or padding or whatever, but it will at least avoid
the match issue.

The other option would be to try and hit the button with tab and
enter or something, I guess.

Test Plan:
bit hard since this is an intermittent bug, I guess
run install_default on a Server image a bunch of times and see if
it always works.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D824
2016-04-26 19:08:51 -07:00
Adam Williamson
da58dc694b add white variant of kde desktop runner, remove old one
according to rdieter the runner dialog could be white or grey
background, we should match on either. One is probably a bug
but it's best to just work with it. Removing the February
version without the 'search' text.
2016-04-25 12:21:52 -07:00
Jan Sedlák
5afc97d0bf add kde and server upgrade tests to templates
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D822
2016-04-25 10:00:34 +02:00
Adam Williamson
31c17235f5 KDE desktop runner changed *again*
clean up the old needles while we're at it
2016-04-19 08:15:07 -07:00
Adam Williamson
90b5acf72a handle 'weak password' due to dictionary load error
Summary:
Rawhide currently seems to have a bug in spell check dictionary
load, which causes the test to fail as it requires another Done
click. So add a workaround needle that handles this case.

Test Plan:
Apply the patch, run some tests, see if they work. I
did a test run on staging:
https://openqa.stg.fedoraproject.org/tests/13331

Reviewers: garretraziel, jskladan

Reviewed By: jskladan

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D815
2016-04-14 23:37:22 -07:00
Adam Williamson
3dca7cbf55 add new Workstation welcome app needles
the colors in this seem to change slightly a lot. I think
GNOME does something clever based on the background image or
something? Anyway. New needles.
2016-04-09 07:36:55 -07:00
Adam Williamson
77c07ae803 remove gtk+ 3.20 bug workaround needles
this seems to be resolved in both F24 and Rawhide now.
2016-04-09 07:19:15 -07:00
Adam Williamson
df195a7853 rename BOOT_UPDATES_IMG_URL to TEST_UPDATES, add GRUBADD
Summary:
BOOT_UPDATES_IMG_URL is a pretty misleading name - it used to
be the actual URL, but now it's simply a boolean that decides
whether we look for the effect of the openQA updates image or
not. TEST_UPDATES seems clearer.

GRUBADD does the same thing as GRUB, on top of it. The point of
this is so we can add an option to the scheduler CLI that lets
you say 'run the normal tests, but with this updates image' -
so we can easily (albeit manually triggered) check the impact
of some anaconda change that needs testing. It should never be
set in the templates or the tests, it's there strictly for the
scheduler (whether that's fedora_openqa_schedule or literally a
person calling `client isos post`) to use as a kind of override.
The tests that test updates image loading will probably fail
when doing this, but all other tests should work as intended,
including ones that specify GRUB, becase the extra params will
just get added on top. That's why I invented a new var instead
of just letting the scheduler override GRUB's value when POST
ing.

Test Plan:
Check the rename didn't break anything (updates tests
still work). Run tests with GRUBADD param, make sure value is
correctly appended to cmdline both when GRUB is also specified
and when it is not.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D801
2016-04-08 13:21:29 -07:00
Adam Williamson
591b153238 revise updates.img test to work with new anaconda (T759)
Summary:
per details in T759, the 'unipony' updates image we use to test
the updates image features doesn't work with latest anaconda (f24
and Rawhide). I've built a new updates image which uses a neat
anaconda feature that allows you to override CSS with a file in
a special location; it sets the background for disk capacity
texts on the INSTALLATION DESTINATION spoke to be pink. This
lets us use a simple needle that just looks for a pink blob on
that spoke, on the basis that it's unlikely there'll ever be a
pink blob there for any other reason, so if there is one, the
updates image worked. There will be an accompanying tools diff
to change the updates disk image to use the new updates image.

Test Plan:
Do a test run and check the updates image tests pass
and no other tests are broken. You'll need to pull in the tools
diff and re-generate the updates disk image to check that test,
the scsi_updates_img test should work with just this diff.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D799
2016-04-01 08:00:47 -07:00
Adam Williamson
9992375d0a needles: disk selection GTK+ 3.20 workaround, part select update
Some needle changes to account for latest Rawhide. There seems
to be a bug between anaconda and GTK+ 3.20 causing text in the
disk selection widget not to be styled correctly: this commit
adds needles for that which are tagged as 'workaround', meaning
openQA will complete the tests but mark them as requiring a
workaround to work ('soft fail'). The bug is #1322036. There is
also an intentional change to the volume list in custom part;
the mount point names are now in dark grey rather than black (in
fact this has always been intended, but it's been broken for a
long time, according to davidshea). I cleaned up the part select
needles at the same time; the new ones have the non-variant
names, as davidshea says the same change is coming to F24 and
the original needles haven't matched for months. The F23 Atomic
install test doesn't hit any of these needles, so we don't need
to keep the original needles around for that. The most recent
variants are kept, as they'll be needed for F24 tests until the
new anaconda build goes stable for F24. The 'freetype262'
variant of part_select_swap hasn't matched for two months and is
dropped.
2016-03-29 11:37:55 -07:00
Adam Williamson
4cae3e56a9 document POSTINSTALL variable (recently added)
meant to document this on commit, forgot - it's just a simple
doc explanation, so committing without review.
2016-03-24 14:07:23 -07:00
Adam Williamson
10783f7280 split install_default into upload/no-upload variants
Summary:
this should avoid unnecessary disk uploads and hopefully help
further reduce the incidence of weird failures in the chained
tests.

With this change we should only upload disk images for the cases
where we're actually going to run the chained tests: we won't
upload disk images for default_install runs on images we don't
run the chained tests for, or for the UEFI job for images we
*do* run the chained tests for.

We only actually need to run the current chained tests
for Server DVD, Workstation live and KDE live x86_64; there's
no need to run them for Everything boot, so we drop that.

Test Plan:
Do a full test run and make sure all tests run
properly and we now only upload disk images where we really need
to.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D794
2016-03-24 13:38:46 -07:00
Adam Williamson
3e435182dd add firewall kickstart tests (disabled and configured)
Summary:
these together test QA:Testcase_kickstart_firewall from the
Server matrix. I'll have to come up with some kinda way to
handle reporting that, might be tricky.

Couple of tweaks to overall test flow: tests can now specify
a POSTINSTALL variable which will load a post-install test
following a naming convention, and tests can specify USER_LOGIN
as 'false' to disable the 'log in as a user' step entirely. We
could easily adjust the kickstarts to create a user so the test
could log in as one, but it seems like an unnecessary step and
I liked the idea of allowing the user login to be skipped.

Test Plan:
Schedule 'universal' tests, check the new tests run
and pass or fail as they should, check no other test is broken
by the logic flow changes.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D792
2016-03-23 13:52:00 -07:00
Adam Williamson
ca60b9abb8 adjust openQA test suite names (install_foo not server_foo)
Summary:
We named a bunch of the	tests 'server_foo' back	when we	were
starting out and didn't	really know what we were doing.	They're
all really installation	tests, not 'server' tests. I actually
want to	start adding some Server tests now, so it seems	like a
good time to fix that mess. This standardizes on 'install_' as
a prefix for installation tests, converting all	the 'server_'
tests and fixing up a couple of	odd cases to use 'install_'.

Test Plan:
Apply along with the matching commit for tools, do
a full test run and report test, and make sure all tests run
with the new names and correct ResTups are generated for wiki
submission.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D790
2016-03-23 13:46:48 -07:00
Adam Williamson
bf8c827107 shutdown before uploading disk images
Summary:
I believe the failures in the Server DVD chained Base tests are
happening because the VM is not cleanly shut down before the disk
image is uploaded. This adds a shutdown step to all tests that
upload a disk image (so, for now, just default_install). To keep
things simple it just runs 'shutdown' from a root console, rather
than using graphical desktop shutdown methods, as the aim is only
to make the disk state clean, not to test shutdown exactly.

I've tested this on staging; a Server DVD test run with this
change produced a full set of passed tests, as opposed to all
the Base tests failing because the system didn't boot properly.
Workstation and KDE tests seem to work fine also.

For the record, SUSE does much the same thing as this commit.

Test Plan:
Do a full test run and make sure everything that worked
before still does. Check that all default_install tests have a
_console_shutdown step added, and it works, and all chained tests
work (or fail for some unrelated reason, but make sure this
doesn't break them).

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D787
2016-03-22 07:19:47 -07:00
Jan Sedlák
49411aeb97 update "KDE selected" needle in software selection
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D781
2016-03-21 15:00:16 +01:00
Jan Sedlák
d4a7124ff6 add english filtered cantarell22 needle 2016-03-15 10:36:38 +01:00
Adam Williamson
0fdd706734 needles: the resize slider changed yet again (GTK+ 3.19.11) 2016-03-09 17:57:15 -08:00
Adam Williamson
0305c5fcd0 anaconda crash reporting: update needles, click Report again
Summary:
First off, this revises the anaconda crash handling needles a
bit. We ditch gtk3195 and update anaconda_error to reflect
current F24/Rawhide. We keep the old anaconda_error around for
now as anaconda_error-23, to handle crashes in the F23 two-week
Atomic nightlies. We also add an 'early' variant, which is for
when (I think) the installer crashes very early, before it's
loaded in GTK+ settings; when that happens, the dialog uses a
different font. The screenshot comes from a recent Rawhide test
that crashed.

We also restore the anaconda `post_fail_hook` code to click
the Report button when a crash happens. This was erroneously
removed in D637. Before the Report button is clicked, the
`anaconda-tb` file exists but the libreport stuff in `/var/tmp`
does not. By removing this, we lost the libreport bits from
the uploaded files, which makes it harder to report crashes. So
let's add it back.

Finally we fix the actual tarring and uploading of `/var/tmp`;
also in D637 this got broken because it was being tarred up in
whatever directory the commands happened to be running in, but
we were still trying to upload it from `/var/tmp`.

https://openqa.stg.fedoraproject.org/tests/8444 was run with
these changes, and has `/var/tmp` correctly uploaded.

Test Plan:
Run some test that crashes, make sure the crash
handling all works correctly.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D768
2016-03-08 08:10:51 -08:00
Adam Williamson
2c1890c52e revise anaconda_install_done needles
GTK+ 3.19.10(?) has changed things in current F24/Rawhide;
weirdly, done-freetype262 seems to match sometimes but not
always. Add a new needle from a current screenshot but keep
freetype262 around for now just in case. Drop some variants
that should never be needed any more, rename the original
needle to -23 to mark that we're only keeping it around for
the F23 two-week Atomic tests. For now keep french-cantarell20
as the 'official' French needle, we'll see if it hits any
failures when the tests are working a bit better.
2016-03-07 16:57:20 -08:00
Adam Williamson
d7a8b5d112 do the /Everything/ fix for REPOSITORY_GRAPHICAL too 2016-03-02 10:25:51 -08:00
Adam Williamson
4cb0a99ec3 add /Everything/ to REPOSITORY_VARIATION
with Pungi 4, the public repos are product-y, we need to add
/Everything/ to the path between the release and the arch.
Again pushing without review to get the tests working.
2016-03-02 09:12:41 -08:00
Adam Williamson
81f2463234 drop stray use of non-existent get_release()
committing without review as the fix is trivial and it breaks
a couple of tests.
2016-03-02 09:08:35 -08:00
Adam Williamson
d6470af4cb ISO_URL not ISOURL, now... 2016-02-23 18:04:54 -08:00
Adam Williamson
0d2f0cb58d we don't need $self here 2016-02-23 18:01:16 -08:00
Adam Williamson
c92ea1b9bd install_source_graphical wasn't properly switched to VERSION 2016-02-23 17:59:44 -08:00