Commit Graph

554 Commits

Author SHA1 Message Date
Josef Skladanka
0d488890ba Organize needles to directories (step 2/2)
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D744
2016-05-13 13:43:31 +02:00
Josef Skladanka
ad044c0b3f Organize needles to directories (step 1/2)
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D744
2016-05-13 13:43:25 +02:00
Adam Williamson
2df55efb49 add desktop_terminal test, refactor test loading a bit
Summary:
I really just want to add the desktop_terminal test, but I think
this refactor is in order now. It splits up loading of the
various test phases (much as SUSE do it) and allows us to run
the post-install tests without the install tests, for e.g. I
tweaked things to allow the upgrade tests to use the existing
_wait_login tests for final login and combine the two upgrade
postinstall tests into one simple one.

This comes with a bit of a behaviour change to make graphical
wait login behave the same as console wait login: it will log
in unless USER_LOGIN is set to 'false'. Previously it only
logged in if both USER_LOGIN and USER_PASSWORD were set, which
I don't think ever happened in a graphical test, so we never
actually did a graphical login. The intent here is we should do
a login on the default_install tests. That's going a bit beyond
the test case, but it seems like a reasonable thing to test. We
can set USER_LOGIN to false if we don't want to do it.

Test Plan:
Do a full test run, make sure the new tests work and
no old tests break.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D839
2016-05-05 16:39:47 -07:00
Adam Williamson
559b5f07b9 update resize slider needle for GTK+ 3.21.1 2016-05-04 17:36:31 -07:00
Adam Williamson
f59343403a add FreeIPA server role deploy and kickstart enrolment tests
Summary:
These require openQA tap networking to allow the server and
client boxes to communicate, and require masquerading (NAT) so
the server at least can reach a repository (dnf/rolekit really,
really do not want to work without a repo connection).

They use the 'parallel' test support to have the server deploy
run first while the client enrol test waits at the grub menu
until the server is done before it goes ahead.

This is all deployed and working on stg. The really tricky bit
was getting all the openvswitch and firewall config right in
ansible.

We *could* do the server deploy test as a follow-on from the
default install test to save the install, but then we'd have to
teach it to change the hostname and set up static networking
post-install. I'm not sure if it's worth doing that.

This requires the corresponding openqa_fedora_tools commit that
adds the hard disks (containing the kickstarts - it's possible
to get them from remote during install, but we have to set up
name resolution or hard code the IP of the server).

Test Plan:
Deploy this and the openqa_fedora_tools commit,
generate the disks, configure the networking (good luck! See
the docs in openqa_fedora_tools) and see if you can run the
tests. If you're using Docker, uh...sorry. You somehow need to
set things up so the workers can use tap interfaces that can
talk to each other and are NATed to the outside world. Have fun.
I can talk you through it on IRC...

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D831
2016-05-04 11:53:11 -07:00
Jan Sedlák
d4466100d4 add necessary needles and fix code for KDE upgrade tests
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D837
2016-05-04 09:58:35 +02:00
Adam Williamson
100c133dda include some grey in the reboot button match
Summary:
There's a weird problem that we hit occasionally only on the
Server image, see e.g.:
https://openqa.fedoraproject.org/tests/14300
I *think* what's happening (thanks hergertme from #gtk+ IRC for
help with this) is that when the 'install done!' text and the
reboot button appears, there's a short transition for the button
so it doesn't look *exactly* like the screenshot right away. If
openQA's needle match loop happens to kick in and check the
match right after the text and button appear, while the
transition is happening, it decides that some random area of the
gradient down the left hand side of the screen matches *better*
than the button, so it 'clicks' that, and we don't get a reboot.
If the openQA needle match loop happens to kick in a millisecond
or whatever later, after the button is done transitioning, the
button is now a 100% match and the bug doesn't happen. This only
happens on Server because the left-hand side gradient differs in
each flavor, and only the Server one just happens to have a spot
which is nearly identical to the needle area.

So to avoid that, let's include a bit of the grey space next to
the button. This is a bit more fragile against GTK+ changes to
button design or padding or whatever, but it will at least avoid
the match issue.

The other option would be to try and hit the button with tab and
enter or something, I guess.

Test Plan:
bit hard since this is an intermittent bug, I guess
run install_default on a Server image a bunch of times and see if
it always works.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D824
2016-04-26 19:08:51 -07:00
Adam Williamson
da58dc694b add white variant of kde desktop runner, remove old one
according to rdieter the runner dialog could be white or grey
background, we should match on either. One is probably a bug
but it's best to just work with it. Removing the February
version without the 'search' text.
2016-04-25 12:21:52 -07:00
Adam Williamson
31c17235f5 KDE desktop runner changed *again*
clean up the old needles while we're at it
2016-04-19 08:15:07 -07:00
Adam Williamson
90b5acf72a handle 'weak password' due to dictionary load error
Summary:
Rawhide currently seems to have a bug in spell check dictionary
load, which causes the test to fail as it requires another Done
click. So add a workaround needle that handles this case.

Test Plan:
Apply the patch, run some tests, see if they work. I
did a test run on staging:
https://openqa.stg.fedoraproject.org/tests/13331

Reviewers: garretraziel, jskladan

Reviewed By: jskladan

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D815
2016-04-14 23:37:22 -07:00
Adam Williamson
3dca7cbf55 add new Workstation welcome app needles
the colors in this seem to change slightly a lot. I think
GNOME does something clever based on the background image or
something? Anyway. New needles.
2016-04-09 07:36:55 -07:00
Adam Williamson
77c07ae803 remove gtk+ 3.20 bug workaround needles
this seems to be resolved in both F24 and Rawhide now.
2016-04-09 07:19:15 -07:00
Adam Williamson
591b153238 revise updates.img test to work with new anaconda (T759)
Summary:
per details in T759, the 'unipony' updates image we use to test
the updates image features doesn't work with latest anaconda (f24
and Rawhide). I've built a new updates image which uses a neat
anaconda feature that allows you to override CSS with a file in
a special location; it sets the background for disk capacity
texts on the INSTALLATION DESTINATION spoke to be pink. This
lets us use a simple needle that just looks for a pink blob on
that spoke, on the basis that it's unlikely there'll ever be a
pink blob there for any other reason, so if there is one, the
updates image worked. There will be an accompanying tools diff
to change the updates disk image to use the new updates image.

Test Plan:
Do a test run and check the updates image tests pass
and no other tests are broken. You'll need to pull in the tools
diff and re-generate the updates disk image to check that test,
the scsi_updates_img test should work with just this diff.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D799
2016-04-01 08:00:47 -07:00
Adam Williamson
9992375d0a needles: disk selection GTK+ 3.20 workaround, part select update
Some needle changes to account for latest Rawhide. There seems
to be a bug between anaconda and GTK+ 3.20 causing text in the
disk selection widget not to be styled correctly: this commit
adds needles for that which are tagged as 'workaround', meaning
openQA will complete the tests but mark them as requiring a
workaround to work ('soft fail'). The bug is #1322036. There is
also an intentional change to the volume list in custom part;
the mount point names are now in dark grey rather than black (in
fact this has always been intended, but it's been broken for a
long time, according to davidshea). I cleaned up the part select
needles at the same time; the new ones have the non-variant
names, as davidshea says the same change is coming to F24 and
the original needles haven't matched for months. The F23 Atomic
install test doesn't hit any of these needles, so we don't need
to keep the original needles around for that. The most recent
variants are kept, as they'll be needed for F24 tests until the
new anaconda build goes stable for F24. The 'freetype262'
variant of part_select_swap hasn't matched for two months and is
dropped.
2016-03-29 11:37:55 -07:00
Jan Sedlák
49411aeb97 update "KDE selected" needle in software selection
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D781
2016-03-21 15:00:16 +01:00
Jan Sedlák
d4a7124ff6 add english filtered cantarell22 needle 2016-03-15 10:36:38 +01:00
Adam Williamson
0fdd706734 needles: the resize slider changed yet again (GTK+ 3.19.11) 2016-03-09 17:57:15 -08:00
Adam Williamson
0305c5fcd0 anaconda crash reporting: update needles, click Report again
Summary:
First off, this revises the anaconda crash handling needles a
bit. We ditch gtk3195 and update anaconda_error to reflect
current F24/Rawhide. We keep the old anaconda_error around for
now as anaconda_error-23, to handle crashes in the F23 two-week
Atomic nightlies. We also add an 'early' variant, which is for
when (I think) the installer crashes very early, before it's
loaded in GTK+ settings; when that happens, the dialog uses a
different font. The screenshot comes from a recent Rawhide test
that crashed.

We also restore the anaconda `post_fail_hook` code to click
the Report button when a crash happens. This was erroneously
removed in D637. Before the Report button is clicked, the
`anaconda-tb` file exists but the libreport stuff in `/var/tmp`
does not. By removing this, we lost the libreport bits from
the uploaded files, which makes it harder to report crashes. So
let's add it back.

Finally we fix the actual tarring and uploading of `/var/tmp`;
also in D637 this got broken because it was being tarred up in
whatever directory the commands happened to be running in, but
we were still trying to upload it from `/var/tmp`.

https://openqa.stg.fedoraproject.org/tests/8444 was run with
these changes, and has `/var/tmp` correctly uploaded.

Test Plan:
Run some test that crashes, make sure the crash
handling all works correctly.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D768
2016-03-08 08:10:51 -08:00
Adam Williamson
2c1890c52e revise anaconda_install_done needles
GTK+ 3.19.10(?) has changed things in current F24/Rawhide;
weirdly, done-freetype262 seems to match sometimes but not
always. Add a new needle from a current screenshot but keep
freetype262 around for now just in case. Drop some variants
that should never be needed any more, rename the original
needle to -23 to mark that we're only keeping it around for
the F23 two-week Atomic tests. For now keep french-cantarell20
as the 'official' French needle, we'll see if it hits any
failures when the tests are working a bit better.
2016-03-07 16:57:20 -08:00
Adam Williamson
abbba24ca4 add french user created needle for cantarell22 2016-02-18 13:33:23 -08:00
Adam Williamson
684f7b4e5e add some GTK+ 3.19.9 / Cantarell 0.0.22 needles 2016-02-18 12:49:11 -08:00
Adam Williamson
741883f00b add needle for newest KDE desktop runner 2016-02-18 12:45:33 -08:00
Jan Sedlák
596772f163 fix Cantarell 0.0.21 also for english filtered needles
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D739
2016-02-09 10:21:01 +01:00
Adam Williamson
41adcd87d4 add needle variants for GTK+ 3.19.8
it changes radio buttons and checkboxes a bit.
2016-02-04 02:16:03 +01:00
Adam Williamson
a0f75b4227 needle update for Cantarell 0.0.21 and GTK+ 3.19.7
mostly cantarell changes. GTK+ changes are bugs, so put 'bug'
in the name to remind us of this. There's one non-typical
change, to main_hub_installation_source; I removed the match
on the text label of the spoke entirely instead of updating it,
in line with other similar needles, it's unnecessary and only
serves to make the needle more fragile.
2016-01-21 17:58:01 -08:00
Adam Williamson
f855516818 new needles for cantarell 0.0.20 / GTK+ 3.19.6
both of these landed in Rawhide overnight and busted stuff, so
new needles!
2016-01-12 17:07:09 -08:00
Jan Sedlák
7b3bb5c252 add freetype262 needle for english_filtered tag
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D706
2015-12-23 08:45:54 +01:00
Adam Williamson
13f58b0426 add GTK+ 3.19.5 variant of GNOME live initial screen needle 2015-12-18 07:58:51 -08:00
Adam Williamson
d06f775d5c add a bunch of needle variants for GTK+ 3.19.5
3.19.5 seems to change the colors of buttons slightly and also
the rendering of checkboxes/radio buttons, they're now placed
somewhat differently relative to their labels.
2015-12-17 11:59:11 -08:00
Adam Williamson
43e655eae4 add freetype 2.6.2 variants of KDE package set needles
garretraziel took these with F23, I think. Theoretically we
could just have these ones and ditch his, as we're not running
KDE install tests on F23, but we might want to discuss that kind
of question more generally, so for now let's just add the
variant.

We may need more variants for the post-install matches, but the
test doesn't actually get that far ATM because of dependency
issues in the KDE package set, so I'll deal with that once we
can actually do a successful KDE install.
2015-12-08 12:03:53 -08:00
Adam Williamson
f34d01ee01 add BIOS font variant for 'user logged in' needle
Summary:
As I figured out a while back, the seabios firmware font is
nearly the same as our console font (eurlatgr) but not quite,
significantly its @ character is different (shorter). We
already have a root_logged_in variant for the firmware font,
but we now need one for user_logged_in as well. The Atomic
installer image has had the Linux console fonts stripped from
its payload, so it's now using the firmware font, and user_
logged_in isn't matching - see
https://openqa.fedoraproject.org/tests/461 .

Test Plan:
Run the cloud_atomic_canned test (on an F23
post-release nightly, as it's not appearing for Rawhide
nightlies ATM) and confirm it now passes.

Reviewers: garretraziel, jskladan

Reviewed By: jskladan

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D684
2015-12-08 08:05:46 -08:00
Jan Sedlák
fd2ed44162 add KDE package set testcase
add KDE package set testcase - this only adds needles and
new testcase configuration, because with needles cleanup from D670,
it should work without change in code.

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D673
2015-12-08 11:49:38 +01:00
Jan Sedlák
6e801bcd52 remove ENV-DESKTOP- tag, set DESKTOP- tag instead where necessary
Let's use the same principle with DESKTOP tag as we use with
LANGUAGE tag. Where there are any GNOME-only (or KDE-only) needles, tag
them with `DESKTOP-gnome` and delete then when different `DESKTOP` value
is specified. This will help with `QA:Testcase_Package_Sets_KDE_Package_Install`
testcase - we will be able to use almost the same code and check that KDE
got installed properly.

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D670
2015-12-08 11:32:48 +01:00
Adam Williamson
163a8fd94a make 'first partition' needles more reliable / universal
Summary:
The 'reclaim_space_first_partition' needles aren't very good.
Using the area with the filesystem means we need to duplicate
the needle for NTFS, and the needles in fact also match the
*second* partition in most cases, sometimes causing the tests
to delete the second partition instead of the first and fail.
This design should work better: a single needle should match
all filesystems, and by including some of the highlighted top
row in the match, we should never match on the second row.
Including 'da' in the needle makes this really 'first disk,
first partition', but that's all the cases we currently use
it for. We don't really need this needle to work hard to make
sure we're on the right screen, as the subsequent needles
(which match buttons that only exist on this screen) do that
job fine.

We drop some needles that should no longer be necessary: F23
should now be using the cantarell17 version of the needle,
and F24 the freetype262 needle. We don't run installer tests
for any other release.

We could tweak second_partition needle to match, but nothing
is actually using it anyway.

Test Plan:
Do a full test and run and make sure the tests
which use this needle work, and always select the correct
partition.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D681
2015-12-07 09:41:49 -08:00
Adam Williamson
2e42e6e478 one more freetype 2.6.2 needle (desktop_clean) 2015-12-05 14:09:52 -08:00
Adam Williamson
301b1e3177 one more freetype 2.6.2 needle - missed this one 2015-12-04 14:53:48 -08:00
Adam Williamson
31513e4e89 update needles for freetype 2.6.2
freetype 2.6.2 changes font rendering quite significantly.
Most text needles needed redoing. Committing without review
so maybe we finally get some damn useful test results...
2015-12-04 13:43:40 -08:00
Adam Williamson
a9c45df445 add needle for lighter-coloured resize slider
Summary: seems like this got lighter for F24 too.

Test Plan:
Run the shrink tests and see if they get past this
screen.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D666
2015-11-27 08:29:58 -08:00
Jan Sedlák
7240ce774f add custom partitioning xfs tests
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D663
2015-11-26 13:50:45 +01:00
Adam Williamson
d3b4e30595 don't match on left border of language list (french filtered)
Summary:
this needle seems to be fragile not because of cantarell in
fact, but because the difference between the the left edge of
the language box and the 'F' is not consistent. So, let's just
not include it in the match.

Test Plan:
Check if French install test works (see test #8847
failure in BOS).

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D661
2015-11-24 08:35:19 -08:00
Jan Sedlák
cf6fa5d957 another needles fixing
This adds another Cantarrell 1.8 missing needle as well as
lighter variant of Done button (it's perhaps caused by displayed
warning about missing swap partition on no_swap test?). With this
in place, all tests on Rawhide passes except of KDE (but it seems
that KDE Live images are borked) and desktop upgrade tests.

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D651
2015-11-19 10:11:07 +01:00
Adam Williamson
3ecef54b51 explicitly check for DNF "system is not ready for upgrade"
Summary:
Instead of sitting there waiting 6000 seconds twice, when DNF
explicitly tells us it failed, just die.

This is why we haven't been getting proper compose check reports
lately; the upgrade tests are failing, waiting 6000 seconds to
time out, then being cloned and tried again, waiting another 6000
seconds. This is just barely going beyond check-compose's 8 hour
wait limit, as it's some time before the upgrade tests even get
started (they're low in the priority list). We're still going to
have that problem if the tests fail any other way, but this at
least catches that case.

Test Plan:
Run the upgrade tests and see that they fail quicker
(assuming the dependency problems they're dying on aren't fixed).
Maybe also do a 22-23 upgrade test and check it still succeeds
properly.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D650
2015-11-18 11:17:49 -08:00
Jan Sedlák
df7a4582d0 add another cantarell18 needle
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D646
2015-11-13 10:05:02 +01:00
Adam Williamson
29b9ffc405 add Cantarell 0.0.18 variant needles
Cantarell 0.0.18 showed up in Rawhide recently and changes
rendering again, so we need some more variant needles. Once
Cantarell change quiets down a bit we'll have to clean these
all up.
2015-11-10 12:35:30 -08:00
Jan Sedlák
da7bdda462 add azerty user logged in needle
Since we are now logging to user account every time,
I've encountered new bug - we are missing needle for azerty
user logged in. This DR adds it.

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D640
2015-11-09 09:49:21 +01:00
Jan Sedlák
ec6b3ff4a3 use validate_script_output instead of typing and needles matching
Use validate_script_output and regex matching instead
of type_string and needles.

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D637
2015-11-04 14:38:36 +01:00
Jan Sedlák
7a4eff8acb add new krunner needles
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D626
2015-10-27 13:52:00 +01:00
Adam Williamson
1f9685d058 another Cantarell 0.0.17.2 variant
Summary: didn't catch this before (as it depends on the image tested)

Test Plan: Test a Server ISO, the ext3 test fails without this

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D622
2015-10-23 11:27:49 -07:00
Adam Williamson
676175023d add another cantarell variant needle
Summary:
So another couple of Cantarell updates hit Rawhide and changed
rendering a bit more, leading to this needle breaking. This is
for Cantarell 0.0.17.2. No other needles seem to have problems.

Test Plan:
Already in place on BOS - note the 2015-10-15 Rawhide
tests that initially failed but now pass.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D621
2015-10-16 23:37:48 -07:00
Josef Skladanka
75b50ae17f Moved needles to the right dir 2015-10-15 09:58:04 +02:00
Jan Sedlák
26fd74490c yet another KDE fixes - login manager changed
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D598
2015-09-30 13:31:03 +02:00
Adam Williamson
b97c019ae9 revise language tag handling to be easier to use (T617)
Summary:
T617 makes some good points about the language tags; this is my
suggestion for an improvement. It requires a bit of cleverness
in unregister_prefix_tags(), but the upshot is that you don't
need to know to set any special tags when creating needles, a
needle with no language-related tags will be considered as valid
for all languages. You have to explicitly add LANGUAGE- tag(s)
to a needle for the language filtering to 'kick in' in any way.
If a needle has at least one LANGUAGE- tag, it will be filtered
unless it has the appropriate tag for the job's specified
language (default is still 'english').

With this approach, only needles which we specifically want to
*only* match their tagged language(s) need the tags, so we can
drop all those -ALL tags.

We're using LANGUAGE- instead of ENV-LANGUAGE- now because the
ENV- tag names denote tags that are treated slightly specially
by openQA, and this is not one. We cannot cleanly use
ENV-INSTLANG because openQA has a hardwired default of 'en_US'
for that.

Test Plan:
Check both English and French tests still work as
intended.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D589
2015-09-29 15:52:50 -07:00
Adam Williamson
d4ddfe167a add a default_install test for cloud atomic installer image
Summary:
We have these 'atomic installer' images (so far just Cloud),
and maxamillion wanted to get them tested. Turns out it's
pretty trivial - they look much like other installs. Only
little wrinkle is they have a reduced hub (no repository
needles) like live images, but are not like live images in
any other way, so I rejigged the 'small hub needle filtering'
handling a bit.

There will be an accompanying diff for tools, and also some
changes in fedfind (these images are getting built nightly
for *current stable*, and it'd be good to test those).

Because we'd like to test the 22 nightlies, I had to add some
needles for 'olddpi' versions of a few screens. See 2e4c1c2 -
the 22 Atomic installer images still have the old GTK+ code
meaning they run at 96.09dpi. I only retook the necessary
needles for the default-install test, if we add any others we
made need to retake a few more needles.

Test Plan:
Schedule jobs for a compose with the atomic installer
image. You will need the matching openqa_fedora_tools diff and
the very latest git fedfind. Check the test for that image runs,
all other tests run as usual, excessive images are not
downloaded, and the atomic installer is not used for running
universal tests.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D595
2015-09-29 11:36:11 -07:00
Jan Sedlák
533dd806e2 fix KDE x11_start_program needle, add Gnome needle
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D592
2015-09-17 09:49:55 +02:00
Adam Williamson
f0f4482976 needle clean up: no ENV-OFW, remove unused needles
Summary:
This removes a couple of unused needles. password_required
needles were dupes of console_password_required, and no tests
actually use password_required. tmp_failed_proc just isn't
used by anything at all (I think it was at some point used in
fedup tests).

This also drops the ENV-OFW-1 tag from all needles, which I
think got in because someone based a needle on a SUSE needle
which has that tag, then it spread. It's of no use or meaning
in Fedora.

Test Plan: Check all tests still run as normal.

Reviewers: garretraziel, jskladan

Reviewed By: jskladan

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D579
2015-09-15 16:05:00 -07:00
Jan Sedlák
7a608985dc fix KDE needle language tag
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D582
2015-09-15 13:51:47 +02:00
Petr Schindler
338b4bf513 Adds uefi support to tests where it makes sense
What changed:
* There is a new needle for uefi bootloader.
* UEFI postinstallation phase
* UEFI tests and machine added to template
2015-09-15 11:04:01 +02:00
Adam Williamson
b3aa968575 add a french (encrypted) test
Summary:
this handles Non-English European Language Install. Basically
it's a bunch of new screenshots for existing tag names, plus
a bit of configurability in _boot_to_anaconda and tweaking some
existing needles to do non-text matches. The weird 'half-the-
icon' needles are for cases where there may or may not be a
warning triangle but we want to click it either way (saves
duplicating the needle).

This also sets up a convention for tagging what languages a
needle is appropriate for. If it's specifically appropriate for
one or more languages, a tag ENV-LANGUAGE-(LANGUAGE) should be
applied for each language, where (LANGUAGE) is the install
language in upper-case ('LANGUAGE' variable, which should also
be the string that will be typed into the language selection
screen). If the needle ought to be used for *all* languages -
i.e. it's not a text match, or any text in the match is known
not to be translated - the tag ENV-INSTLANG-ALL should be
applied.

To back this, main.pm now unregisters all needles that are not
tagged with either ENV-LANGUAGE-ALL or the tag for the language
actually being used (if the LANGUAGE var is not set, we assume
english). The point of this is to check the install is actually
translated; if we allow all needles to match, the test would
pass even if no translations appeared at all.

Test Plan:
Run all tests and make sure you get the expected
results. You can schedule a run against 23 Beta TC1 to see the
French test fails 'correctly' when translations are missing.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D577
2015-09-14 18:08:58 -07:00
Jan Sedlák
7017486d43 add KDE live default install test
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D576
2015-09-14 08:52:37 +02:00
Adam Williamson
68acecb6d4 convert upgrade tests to dnf-plugin-system-upgrade
Summary:
This is a first cut which more or less works for now. Issues:

1) We're not really testing the BUILD, here. All the test does
is try and upgrade to the specified VERSION - so it'll be using
the latest 'stable' for the given VERSION at the time the test
runs. This isn't really that terrible, but especially for TC/RC
validation, we might want to make things a bit more elaborate
and set up the repo for the actual BUILD (and disable the main
repos).

2) We'd actually need --nogpgcheck for non-Rawhide, at one
specific point in the release cycle - after Branching but
before Bodhi activation (which is when we can be sure all
packages are signed). This won't matter until 24 branches, and
maybe releng will have it fixed by then...if not, I'll tweak
it.

3) We don't really test that the upgrade actually *happened*
for desktop, at the moment - the only thing in the old test
that really checked that was where we checked for the fedup
boot menu entry, but that has no analog in dnf. What we should
probably do is check that GUI login works, then switch to a
console and check /etc/fedora-release just as the minimal test
does.

Test Plan:
Run the tests. Note that creating the desktop disk
image doesn't work ATM, so I can't verify the desktop test
works, but the minimal one seems to (with D565). There'll be
a matching diff for openqa_fedora_tools to update the test
case names there.

Reviewers: jskladan, garretraziel

Reviewed By: jskladan, garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D567
2015-09-10 14:49:13 -07:00
Adam Williamson
c9ec17b2b7 make Done button match area smaller to handle dotted line
Summary:
To understand this change see test 1092 on BOS. When you click
Done once and then get the 'click Done again to proceed' warning,
the Done button seems to grows a little dotted line (I think
this indicates it's selected), which causes the match to drop to
94% and fail. To counter that, shrink the match area to the bit
of the button inside the dotted line, which will be the same
whether it's there or not.

Test Plan:
See if no_swap stops failing sometimes now, maybe,
at last? Yeesh. Also see if all other tests still run OK.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D549
2015-09-08 16:31:43 -07:00
Jan Sedlák
f8f242b7e0 add guided shrink test
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D561
2015-09-08 15:54:22 +02:00
Adam Williamson
95bc2657fb add a 'no swap partition' test
Summary:
since we did this live at Flock today, I figured I'd tidy it
up and submit it. This is an 'optional' test, but some people
do run this way so it'd be nice to have it. This adds another
little helper method in anacondatest.pm, for deleting partitions,
which works much like the others added in previous commits.

Test Plan: Schedule a test run, see if the test runs and works.

Reviewers: jskladan, garretraziel

Reviewed By: jskladan, garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D503
2015-08-19 17:41:41 -04:00
Jan Sedlák
f71faaa43d add updates.img via local media test
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D496
2015-08-17 17:36:40 +02:00
Adam Williamson
8fde224dc5 add ext3, btrfs and LVM thin tests, tweaks to custom methods
Summary:
This adds three new custom storage tests and some needles to
support them, and tweaks the custom storage methods a bit to
address some things that cropped up in writing the tests. A
new method is added for changing the filesystem, as that's
a distinct operation from changing the device type.

This also restores the previous behaviour of select_disks()
where it handled selecting custom partitioning when needed.
Turns out it's pretty common to use regex'es in perl! Who'd'a
thought.

A corresponding commit to add the tests to openqa_fedora_tools
is coming.

There's no post-install step for the tests yet; I'll try and
write those up and add them soon.

Test Plan:
Do a full run, including the new tests, on Alpha RC2 and check
all are scheduled correctly and run correctly. The LVM thinp
test is expected to fail as it catches a genuine bug.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D490
2015-08-10 11:01:12 -07:00
Adam Williamson
3df993404c needle tweaks for eurlatgr font in anaconda
Summary:
I discovered another fun font issue today. Current anaconda
images don't use the intended 'default' console font, eurlatgr.
Neither do live images, but installed systems *do*.

The font they use is the system BIOS font, which in openQA
cases means the qemu firmware font. The easiest way to spot the
difference is the @ character; the shorter version is from the
system BIOS, the slightly taller one is what it looks like in
eurlatgr and latarcyrheb-sun16 (the old default).

In a test image I built, for some reason, I *did* get eurlatgr
in the tmux console, and that broke some needle matches. After
figuring all this out, bcl has sent a lorax patch to use
eurlatgr in the installer, so it makes sense to add these fixes
to the repo for when that kicks in.

We shrink the match on root_logged_in.json by one line. This
screenshot is taken from a post-install case where the prompt
appears in the middle of the screen, and has three black rows
above the prompt; in anaconda, when the prompt appears right at
the top of the screen, there's only *two* rows of black above
it, so the match fails. This fixes that. It's been working so
far because installs have been matching root_logged_in_
rawhide20150311, which is taken with the firmware font, but
once the installer starts using eurlatgr, that won't match any
more.

We also add a new needle for the anaconda_install_source_check
_repo_added tag, taken with eurlatgr. The existing screenshot
was taken either with the firmware font or with latarcyrheb.
They both use a curly glyph for a single quote ('), while
eurlatgr uses a straight line.

This also renames the root_logged_in variant needle to be
clearer about why it's there. We'll probably need variants of
some needles until we're sure lives, anaconda env, and installed
systems are all using eurlatgr. RHBZ #1250262 is a bug I filed
for the live images not using eurlatgr.

Test Plan:
Run the tests with both BIOS font and eurlatgr as
the anaconda font and make sure they all work. The latter
might be a bit tricky till the change lands upstream, I've no
idea how it worked out that way in my test boot.iso.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D483
2015-08-05 09:15:41 -07:00
Adam Williamson
ce4cf5d022 drop the user_logged_in_20150721f23 variant needle
Summary:
I actually have no idea why I created this, now. Examining it
closely there is no difference in the matched area between it
and user_logged_in. Both are evidently using the eurlatgr font
(tall @ sign). I tested that the kickstart_user_creation test
still works on F23 Alpha TC2 with this needle gone, and the
kickstart tests are the only ones that use it.

Test Plan: Run the kickstart tests and verify they still work.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D484
2015-08-05 03:25:55 -07:00
Adam Williamson
248b7a9536 revise storage: better test loading, shared disk selection
Summary:
This contains several tweaks to storage handling. It adds a
method for disk selection which all the storage tests can
share. It sets up a more extensible approach for main.pm to
run the storage tests, instead of an ever-growing forest of
'else' clauses. Finally it sets up a couple of methods for
changing partitioning schemes on the custom part screen and
uses one of them in the software RAID test; the other will
be used for other custom storage tests.

This kills the two_disks needle. I could keep it and work
it into select_disks, but it doesn't fit naturally and I
really just don't see the point of the needle. The only thing
we lose is we don't check that anaconda actually sees two
disks in the 'attach two disks, only install to one' test
(that's server_sata_multi), but the other multi-disk tests
will serve to catch that case failing for some reason.

What I actually intended to do was add some more tests for
different custom part storage types, but it seemed a good
idea to do some of this cleanup so that can be implemented
efficiently. I'll have followups for that.

Test Plan:
Run all tests and ensure they work exactly as
before (not just that they still pass, but that the correct
test steps are actually scheduled in each case.)

Reviewers: garretraziel, jskladan

Reviewed By: garretraziel, jskladan

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D475
2015-07-31 01:31:27 -07:00
Adam Williamson
c11e7bb99b retake another screenshot for 96dpi
Summary:
Missed this one in D456. It causes the software RAID test to
fail.

Test Plan: Run the software RAID test on Alpha TC2.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D476
2015-07-26 23:47:39 -07:00
Adam Williamson
25b9271e37 tweak two disk matching to be more robust
Summary:
Depending on the disks attached to the system and some other
idiosyncracries (like what the disk 'description' currently
happens to be), the distance between the two disk icons can
vary, so a needle that matches on the two disk icons is fragile
(whenever the distance differs, the match fails). So instead of
that, let's have a needle that matches on a single icon with
a lot of grey space to its left, plus the two little matches
we later use in the select_disk(1/2) needles. This should be
robust enough and doesn't need us to keep duplicating it when
the distance between the icons changes. With this tweak, the
'2' variant is no longer needed.

Test Plan:
Run the various tests that use two disks and check this
needle always matches. You'll also want to have D456 applied.

Reviewers: garretraziel, jskladan

Reviewed By: garretraziel, jskladan

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D463
2015-07-22 11:25:48 -07:00
Adam Williamson
4b8e411479 create fedora base class, factor out console login
Summary:
Root console in anaconda got broken by RHBZ #1222413 - no
shell on tty2. Decided to clean up console use in general as
part of fixing it.

This creates a class 'fedorabase' and has 'anacondalog' and
'fedoralog' both inherit from it. boot_to_login_screen is
moved there (as it seems appropriate) and it has a new
method, console_login, which basically handles 'get me a
shell on a console': if we're already at one it returns,
if not it'll type the user name and the password *if
necessary* (sometimes it's not) and return once it sees a
prompt. It takes a hash of named parameters for user,
password and 'check', which is whether it should die if it
fails to reach a console or not (some users don't want it
to).

anacondalog and fedoralog both get 'root_console' methods
which do something appropriate and then call
console_login; both have a hash of named parameters,
anacondalog's version only bothers with 'check', while
fedoralog's also accepts 'tty' to pick the tty to use.

This also adjusts all things which try to get to a console
prompt to use either root_console or console_login as
appropriate.

It also tweaks the needle tags a bit, drops some unneeded
needles, and adds a new 'user console prompt' needle; we
really just need two versions of the root prompt needle
and two of the user prompt needle (one for <F23, one for
F23+ - the console font changed in F23, and the @ character
at least doesn't match between the two). I think we still
need the <F23 case for upgrade tests, for now.

Test Plan:
Do a full test run and see that more tests
succeed. I've done a run on happyassassin with a hack to
workaround the SELinux issue for interactive installs,
and the results look good. I also fiddled about a bit to
test some different cases, like forcing a failure in a
live test to test post_fail_hook (and hence root_console)
in that scenario, and forcing failures after some console
commands had been run to check that it DTRT when we've
already reached a console, etc.

Reviewers: jskladan, garretraziel

Reviewed By: jskladan, garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D462
2015-07-22 11:24:40 -07:00
Adam Williamson
2e4c1c2325 re-do most anaconda screenshots for 96dpi
Summary:
Most of the anaconda screenshots were taken with older netinst
images, at 96.09dpi. Workstation lives use 96dpi exactly. This
led to some needles failing on the Workstation live as fonts
render slightly differently at 96 and 96.09dpi. LONG STORY
SHORT, we got a change in GTK+ which means traditional install
images now use 96dpi exactly too; so we should be consistent
between lives and trad images now, but we need to retake all
the screenshots taken at 96.09dpi. That's what this is.

I also cleaned up a couple of other things. The anaconda_error
needle was effectively a duplicate of anaconda_report_btn.
Neither is actually used atm, but I kept report_btn in case we
use it in future.

anaconda_main_hub_installdest_selected isn't needed any more,
since my tweaks to the matching in anaconda_main_hub_nonlive;
it will now match the case this needle was created for initially.

anaconda_software_select_box is no longer used by any test. The
software selection tests were changed to use key sequences and
dropped their use of this needle, but the needle wasn't removed.

anaconda_select_install_lang_english_filtered_english_selected_
as_default was never really necessary, the situation it's meant
for can be handled by simply adding the anaconda_select_install_
lang_english_filtered tag to the anaconda_select_install_lang_
english_selected needle (as this commit does). That achieves
the needed effect and doesn't break the case where the UI is in
Czech.

Most of the match area changes are just because the layout
changed a bit since the screenshot was taken. I did reduce some
over-precision in some needles, notably the multi-disk needles
- there is no reason we need to match on the disk descriptions
and sizes when we're just selecting the appropriate disks. We
aren't testing anaconda's description of the disks (which may
change unexpectedly), only that it *works* with the disk
types used in the test.

After this commit a little under half of the tests pass with
the 23 Branched 20150717 nightly. Most of the ones that fail
fail because of the anaconda tty2 bug (there's no console on
tty2) or the selinux bug (#1243168) that prevents console login
post-install with SELinux enabled. There are a few other fails
too, but none of them seems to be caused by a needle mismatch,
so I'll investigate and fix them separately.

Test Plan:
Run the full test suite on a current nightly, see that some
tests actually pass now!

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D456
2015-07-20 09:10:37 -07:00
Garret Raziel
c8f02969f2 add fedup desktop test
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D451
2015-07-17 12:52:56 +02:00
Adam Williamson
2f20e8e103 don't include radio button state in anaconda_minimal_highlighted
Summary:
There's no need to include the radio button state here, which
causes the match to fail if Minimal is highlighted and already
selected. Hitting Space in this case won't de-select it (it's
a radio button, not a checkbox), and will give the desired
result. So we should accept it.

Test Plan:
Run tests against Rawhide (and maybe F22 as a regression check)
and ensure the package set selection step works, in tests where
it's included.

Reviewers: garretraziel, jskladan

Reviewed By: jskladan

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D385
2015-06-11 09:59:51 -07:00
Adam Williamson
53fcc86ae3 avoid branded areas, tighten 'Done' button matches
Summary:
Two needles (at least, there may be more) fail to match properly
with Rawhide (F23) because the top part of some screens now has
branding, while before it was plain grey. On root_password_screen
we can just throw that match away, we don't really need it, the
text 'Root Password:' seems specific enough to be reliable. On
user_creation_screen let's replace the branded area match with a
couple of the text box labels.

This also tightens the 'Done' button match area on both spokes;
the original area is too loose and covers some of the now-branded
area outside the button, so the match fails.

This fixes the default_install test for 2015-06-07 nightly.

Test Plan:
Try a test run against the 2015-06-07 nightly and see if the
default_install test passes. Maybe also check that it still
passes for an F22 build.

Reviewers: garretraziel, jskladan

Reviewed By: jskladan

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D386
2015-06-11 09:57:40 -07:00
Garret Raziel
1422d2c0e2 Add fedup_minimal test
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D358
2015-05-13 13:03:23 +02:00
Garret Raziel
f7d4bc69ce changes to solve 173 and 167
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D335
2015-04-10 10:38:28 +02:00
Garret Raziel
469d8d2e92 upload Anaconda ABRT directory
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D310
2015-04-03 13:42:59 +02:00
Garret Raziel
3146a72d99 try to fix anaconda user creation
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D326
2015-04-03 10:27:31 +02:00
Garret Raziel
c866851e4b select environment programmatically
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D312
2015-03-25 11:56:48 +01:00
Adam Williamson
84a14cdb67 Support for live image testing
This requires adding products, flavors and needles and test
cases, and tweaking some existing ones to handle the
slightly different behaviour of live images in shared tests.

To handle the different main hub screens in live and non-live,
a less stringent needle is added which is unregistered for
non-live tests, so they don't match on it before they've
finished updating repository metadata.

There are a few small bugfix tweaks in this too, like some
delays in user creation to try and avoid intermittent failures
there.

A new root_logged_in needle is also included, to handle a new
console font in Rawhide - that has nothing strictly to do with
live testing, it just happened to come up while working on
this.
2015-03-18 14:28:03 -07:00
Garret Raziel
f8f6d2f8bc fixing some issues - doublequotes and type_string 2015-03-13 12:58:03 +01:00
Garret Raziel
b8e011e2ff wait for Password: prompt 2015-03-11 16:35:54 +01:00
Garret Raziel
3863d840b0 add software RAID test suite 2015-03-06 10:36:25 +01:00
Garret Raziel
2038451b96 add multiple disks test 2015-03-05 13:57:47 +01:00
Garret Raziel
f51b1c916e click on input box in select lang 2015-02-20 10:30:23 +01:00
Garret Raziel
7eb9eaa090 changes in user creation, http mirrors 2015-02-19 14:15:29 +01:00
Garret Raziel
235377708f beginning of test logging 2015-02-13 15:08:29 +01:00
Garret Raziel
faf54a2055 completed delete partial testsuite 2015-02-13 12:11:47 +01:00
Garret Raziel
a00cf9e530 New tests 2015-02-13 09:51:24 +01:00
Josef Skladanka
98462fc270 Minor fix to the check_repo_added needle 2015-02-11 12:05:25 +01:00
Josef Skladanka
fe36bf1c83 Disk guided encrypted 2015-02-04 17:16:21 +01:00
Josef Skladanka
376ec0d81b Inst-repo HTTP variation 2015-02-04 14:45:37 +01:00
Josef Skladanka
23aee07c68 Install source graphical mirrorlist + repo 2015-02-04 13:02:54 +01:00
Josef Skladanka
a4aec6b123 Fixed select-lang when geolocation pre-selects english 2015-02-03 18:11:29 +01:00
Garret Raziel
45ec446ba5 add package select minimal 2015-02-03 15:02:52 +01:00
Josef Skladanka
dfd6123859 update needles 2015-02-03 13:59:40 +01:00
Josef Skladanka
f9e75e315c another bunch of tests 2015-01-30 10:35:13 +01:00
Josef Skladanka
8e6c972893 fixed disk guided multiselect 2015-01-27 16:24:12 +01:00
Garret Raziel
0c18688663 Added 'server_delete_pata' testsuite 2015-01-27 14:22:35 +01:00
Josef Skladanka
47e8c38dca Added few more tests 2015-01-27 13:35:27 +01:00
Josef Skladanka
780a9afbba Added 'server_sata_multi' testsuite
The testsuite covers guided installation on multiple devices.

TODO: change from VirtIO to SATA driver
2015-01-26 17:21:32 +01:00
Josef Skladanka
5264388dc5 Added 'server_simple' testsuite 2015-01-26 15:58:07 +01:00
Josef Skladanka
f17b37c292 Decoupled tools from tests 2015-01-26 14:43:01 +01:00