Commit Graph

29 Commits

Author SHA1 Message Date
Adam Williamson
c4f32ab5ad add an Asian (Japanese) language install test
Summary:
Include some basic testing of Japanese input, and split the
input testing (including Russian) into a separate module, since
it's not really part of 'login' testing.

Test Plan:
Run the test, and the Russian and French tests too to
make sure they didn't break. Tested on staging. Note the Japanese
test soft fails, intentionally, at present, as I discovered a bug
while working on it:
https://bugzilla.gnome.org/show_bug.cgi?id=776189

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D1072
2016-12-21 08:41:00 -08:00
Adam Williamson
7135a4e05d iSCSI: test user authentication
Summary:
This isn't in the criteria, but it's commonly used, so we ought
to test this way. Require authentication for the iSCSI target
and have the test provide the appropriate auth info.

Test Plan:
Run the iscsi test and check it works (you need the
recent fixes for support_server to make *that* work). Nothing
else should be affected.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D1070
2016-12-20 08:34:30 -08:00
Adam Williamson
0fa6138448 Have non-English tests do graphical install and login
Summary:
The non-English tests so far did not test that graphical login
worked as expected, which is a fairly large hole. With this
change, they should do a Workstation install and test login to
both GNOME and the console works as expected. KDE is not yet
tested.

As part of this we tweak the implementation of keyboard layout
switching in graphical environments to use a generic function
in main_common which can handle both anaconda and desktops
(just GNOME at present, but should extend easily to any desktop
with a known switcher key and a visible layout indicator),
replacing the anacondatest class method. I kinda don't like that
the test has to specifically tell the function when it's in
anaconda, but I don't think I want to start experimenting with
a global 'test phase' openQA variable or anything like that at
present.

Fixes T842.

Test Plan:
Run the French and Russian install tests and check
they work as expected. Also run an English Workstation install
if you like, and make sure that didn't break. This change is
live on staging ATM, seems to work fine.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Maniphest Tasks: T842

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D1071
2016-12-16 09:40:29 -08:00
Adam Williamson
f11d5265b7 add F24 variants of some needles
To fix tests of the live-respins (which are still F24 ATM).
2016-12-07 16:01:22 -08:00
Adam Williamson
638aa01de2 update some iscsi needles for current f25 (and rawhide) 2016-10-27 00:35:42 -07:00
Adam Williamson
e317244d1e revise install_done needles to include button text
OK, we once again hit an annoying case where the button match
that doesn't include text got matched in the sidebar:

https://openqa.fedoraproject.org/tests/39576#step/_do_install_and_reboot/13

so I think it's time to give up on that approach and just go
ahead and deal with the button text varying between live and
traditional installs. So let's change the existing needles to
include the button text and add one new needle to cover the
English live F25/F26 case. I don't think we need a 'quit'
needle for F24 (as the only thing we test for F24 is the Atomic
installer image, which says 'Reboot') or for French or Russian
(as we only test a traditional installer image for those, not
a live image).
2016-10-07 10:10:10 -07:00
Adam Williamson
ab5c97907e add 'finish configuration' needle variants for f24
the Atomic installer test is hitting 'finish configuration' now
(thanks, slow typing!) so we need an F24 needle variant (font
rendering changed).
2016-09-29 15:23:40 -07:00
Adam Williamson
4d3ed25d79 add missing screenshots for new font needles (whoops) 2016-09-13 21:08:29 -07:00
Adam Williamson
f26d70bc58 add some variant needles for new freetype in Rawhide
seems to affect the network spoke a lot but not anywhere else.
Odd. Anyhow, here we go. Keeping the old needles for now as
F25 still has the old rendering.
2016-09-13 13:02:05 -07:00
Adam Williamson
72074868a8 clean up lots of no-longer-used needles
Summary:
I've been wanting to do this for a while, but
https://github.com/os-autoinst/openQA/issues/786 is making it
difficult. Still, I think the quantity of needles is slowing
down openQA, so I'm doing it the old-fashioned way - looking
through test results and seeing what needles are actually used
now.

Test Plan:
Run full test suite for 24 Atomic, 25 and Rawhide and
make sure all tests still work. This is currently deployed on
staging (along with key-fixes) and I'm testing it there.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D994
2016-09-12 10:25:24 -07:00
Adam Williamson
e1ec1997af try to be safer when typing in X: slower, more checks
Summary:
the main thing this does is try and type slower in X - this
should cover nearly everywhere we type anything in X, and make
it type slower. We also add a bit more safety checking to some
old tests which didn't have it (mainly _do_install_and_reboot)
- wait_still_screen after typing to make sure all the keypresses
were registered before continuing.

This is an attempt to mitigate the problems we've seen where
the wrong text gets typed into the wrong places and the tests
break.

This branch is live on staging atm. It still has *some* issues,
but I do think it's an improvement.

Test Plan:
run the tests (probably several times), compare to
runs without the change, see if it's better or worse...

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D993
2016-09-12 10:24:30 -07:00
Adam Williamson
276e38b561 more variant software selection needles for BGO #771127
so what seems to be going on on the software selection screen
is some kind of GTK+ bug:

https://bugzilla.gnome.org/show_bug.cgi?id=771127

the radio button bullets don't always render correctly - I think
they're not always completing a transition they go through on
selection. I think they can get stuck in any state between 'grey
and small' and 'black and big', but for now these are the needles
I've managed to create from failures; we're missing a 'greyer
KDE' variant, but if that happens, we can add it. If the bug
gets fixed we should be able to drop all these.
2016-09-09 10:41:54 -07:00
Adam Williamson
2f5cc741f9 update fs_xfs and fs_ext4_preselected needles
these rarely get used any more. they're only used when we're
running the filesystem tests on a non-Server image, where the
default fs is ext4 not xfs. With the old nightly process, where
we only got a generic boot.iso, we used them all the time, but
now we get product-ized nightlies, we rarely do. But I did hit
them today running the universal tests with a generic boot.iso
I hand-built to test an anaconda update, so in case we need to
do that kind of thing again in future, we may as well update
the needles.
2016-09-08 13:46:26 -07:00
Adam Williamson
e56ab793de add another kde_selected needle variant
seems like the size of the radio bullet is very slightly
different? whatevers.
2016-09-03 17:42:36 -07:00
Adam Williamson
ef689e75a9 use compose repository (not master repo) for most tests
Summary:
we have a long-standing problem with all the tests that hit
the repositories. The tests are triggered as soon as a compose
completes. At this point in time, the compose is not synced to
the mirrors, where the default 'fedora' repo definition looks;
the sync happens after the compose completes, and there is also
a metadata sync step that must happen after *that* before any
operation that uses the 'fedora' repository definition will
actually use the packages from the new compose. Thus all net
install tests and tests that installed packages have been
effectively testing the previous compose, not the current one.

We have some thoughts about how to fix this 'properly' (such
that the openQA tests wouldn't have to do anything special,
but their 'fedora' repository would somehow reflect the compose
under test), but none of them is in place right now or likely
to happen in the short term, so in the mean time this should
deal with most of the issues. With this change, everything but
the default_install tests for the netinst images should use
the compose-under-test's Everything tree instead of the 'fedora'
repository, and thus should install and test the correct
packages.

This relies on a corresponding change to openqa_fedora_tools
to set the LOCATION openQA setting (which is simply the base
location of the compose under test).

Test Plan:
Do a full test run, check (as far as you can) tests run sensibly
and use appropriate repositories.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D989
2016-09-01 08:22:59 -07:00
Adam Williamson
9311146f95 drop dictionary error workaround
Summary:
the dictionary error bug was fixed some time back, so drop this
workaround for it.

Test Plan:
Run all tests for F25 and Rawhide and verify they don't need
this workaround any longer.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D988
2016-08-30 11:18:46 -07:00
Jan Sedlák
518f996b10 add changed english filtered needle 2016-08-30 15:57:52 +02:00
Adam Williamson
19fccde4f0 add a few more needle updates for font rendering changes 2016-08-29 15:23:24 -07:00
Adam Williamson
5d7b46cb9d widen reclaim_space_delete_btn area to make it unambiguous
this needle can actually also match the 'Delete all' button,
resulting in the test not doing what it should. openQA should
prefer the match closer to the area's location in the needle,
but https://openqa.fedoraproject.org/tests/31136 seems to show
this not happening, so let's make the match area wider in all
versions of the needle so it should no longer match the Delete
all button.
2016-08-29 11:00:53 -07:00
Adam Williamson
232dad604c add a bunch of needles for changed font rendering
font rendering has changed somehow in both Rawhide and F25
updates-testing, retake a bunch of needles for this. Some more
may be needed later in tests that are currently failed or
skipped for other reasons.
2016-08-25 16:01:26 -07:00
Adam Williamson
83828425e1 tweak install_lang needle to match Fedora-Atomic
now we're getting composes with the product name 'Fedora-Atomic'
this needle doesn't match them. Tweak it so it should match both
'Fedora' and 'Fedora-Atomic'.
2016-06-22 20:54:54 -07:00
Adam Williamson
3a76630a09 add similar variant for Russian layout needle 2016-06-13 15:36:59 -07:00
Adam Williamson
b5c3cd9a9c add variant us layout indicator needle
Somehow it seems this can look a bit different sometimes - see
https://openqa.fedoraproject.org/tests/22000 . The text seems
to be in a slightly different vertical position.
2016-06-13 15:23:25 -07:00
Adam Williamson
0da6652287 add NFS tests (and DHCP/DNS in the support server)
Summary:
Set up the support server to provide DHCP/DNS functionality and
an NFS server, providing a kickstart. Add a kickstart test just
like the other root-user-crypted-net kickstart tests except it
gets the kickstart from the support server via NFS. Also add NFS
repository tests and a second support server for Server-dvd-iso
flavor: this test must run on that flavor to ensure that packages
are actually available. The support server just mounts the
attached 'DVD' and exports it via NFS.

Note we don't need to do anything clever to avoid IP conflicts
between the two support servers, because os-autoinst-openvswitch
ensures each worker group is on its own VLAN.

As part of adding the NFS repo tests, I did a bit of cleanup,
moving little things we were repeating a lot into anacondatest,
and sharing the 'check if the repo was used' logic between all
the tests (by making it into a test step that's loaded for all
of them). I also simplified the 'was repo used' checks a bit,
it seems silly to run a 'grep' command inside the VM then have
os-autoinst do a grep on the output (which is effectively what
we were doing before), instead we'll just use a single grep
within the VM, and clean up the messy quoting/escaping a bit.

Test Plan:
Run all tests - at least all repository tests - and
check they work (make sure the tests are actually still sane,
not just that they pass). I've done runs of all the repo tests
and they look good to me, but please double-check. I'm currently
re-running the whole 24-20160609.n.0 test on staging with these
changes.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D888
2016-06-13 08:42:30 -07:00
Adam Williamson
7a8ae3a357 add an iscsi test, and a support_server test to support it
Summary:
this is following a SUSE model for tests where we need a server
end but don't want setting up the server to constitute a real
test in itself, we want it to be stable. The 'support_server'
test just boots a pre-built (by createhdds) disk image, sets up
networking, and runs the iSCSI server.

To run the iSCSI test we need to handle networking config in
anaconda (or we would need to set the support server up as a
DHCP server, which may be worth considering), so this adds that.
We also need to be able to specify the target device for a
volume in custom partitioning, so this adds that too.

Test Plan:
Build the necessary support server disk image (use
D883), then run the test and make sure it works. Also make sure
all other tests continue to work.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D884
2016-06-09 08:43:46 -07:00
Adam Williamson
a1d240b135 add needles to treat 'Custom Operating System' as minimal
Summary:
the Server DVD now just has 'Fedora Server' and 'Custom
Operating System' environments. Custom is basically minimal.
So we can use the DVD for 'universal' testing again, these
needles match the anaconda_minimal tags.

Test Plan:
Run the 'universal' tests on a DVD ISO with these
needles added, test that they work OK and use the 'Custom' env.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D844
2016-05-20 12:56:44 -07:00
Adam Williamson
a4f3267534 add Russian install test
Summary:
Requires new needles and test suite and job template, plus a
few tweaks to handle 'switched' keyboard layouts (so we use the
switched layout in the username and password).

Test Plan:
Run the test and see that it...fails. But that's OK!
It's a genuine bug: RHBZ #1333998 . At least make sure it gets
to that point and no other tests have broken and all the needles
look sane.

Reviewers: garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D846
2016-05-20 07:52:55 -07:00
Jan Sedlák
6dcb20e022 add new french anaconda needles
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D855
2016-05-20 10:29:40 +02:00
Josef Skladanka
0d488890ba Organize needles to directories (step 2/2)
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D744
2016-05-13 13:43:31 +02:00