These are no use :/ The Rawhide default background is a 'changes
with the time of day' one, and with the transparent top bar,
that makes a proper needle match impossible to reliably achieve,
as there is not a single pixel of the 'clean desktop' which
will always be the same any more. We'll have to come up with
something else. I'm trying out the use of an OCR match now.
The GNOME top bar is now transparent in Rawhide, which broke the
'clean desktop' needle. It also means it'll break every time
the background changes, from now on :( Update the needle.
Seems like the top '[patch]' link on kernel.org may now be a
raw patch file which the browser just displays, not a compressed
patch it offers to download. So tweak the needle so we should
click on the *bottom* link instead. openQA looks for matches
as close as possible to the location in the needle.
When running universal tests with a non-Server image, which is
not common but *does* happen (e.g. manually built test image),
the pre-selected filesystem in blivet-gui will be ext4 not xfs
(just as with custom partitioning). So we need xfs and ext4
variants of this needle. Renamed the XFS one for consistency.
Summary:
This adds a new test suite, run for Workstation and KDE live
images, which does not create a user during install. It then
expects initial-setup (KDE) or gnome-initial-setup (Workstation)
to appear after install, creates a user, and proceeds with
normal boot.
Note the ARM image test already covers the initial-setup text
mode, and the ARM minimal image is the only case where that
actually matters (it's not included in Server).
Test Plan:
Run the new tests, check they work. Run all old
tests, check the changes didn't break them.
Reviewers: jsedlak, jskladan
Reviewed By: jsedlak
Subscribers: tflink
Differential Revision: https://phab.qa.fedoraproject.org/D1185
They got rid of the 'Dashboard' text we were matching on, so
let's change this needle. This 'Hardware' text should show up
in all cockpit versions, I think.
Summary:
This is to handle cases like #1414904 , where the system boots
to emergency mode. We really need logs to try and debug this.
Test Plan:
Force a test to hit emergency mode somehow (right now
you can just run base_services_start on Rawhide over and over
until you hit #1414904, but there's probably an easier way to
do it, I think there's a systemd boot arg to tell it which target
to boot for e.g.) and check logs get uploaded. Also check this
doesn't break log upload for a 'normal' failure.
Reviewers: garretraziel_but_actually_jsedlak_who_uses_stupid_nicknames
Reviewed By: garretraziel_but_actually_jsedlak_who_uses_stupid_nicknames
Subscribers: tflink
Differential Revision: https://phab.qa.fedoraproject.org/D1103
openQA has some problems handling needles with identical names
in different subdirectories. We haven't had the cycles to send
fixes for this yet, so for now, let's just rename all such
needles we have.
Not sure if the color change will occur in Fedora 25 nightlies
or not, so not sure if we can remove the old needles. But we
*can* remove the F24 needles now I think.
Summary:
Since 26.17, anaconda shows a warning when the user password
contains non-ASCII characters, and requires a second Done click
to confirm. This change should handle that.
On the 'catch cases where password typing went wrong and re-try'
bit: to keep that, but not re-type the password *every single
time* on the Russian install test, we'd have to make the needle
match the text of the warning. This is problematic because then
that needle will be able to break without us easily noticing;
that's why I wanted to keep the 'warning bar' needle text-free.
Unfortunately, that means we have to skip the protection for
switched-layout installs.
Note the protection was actually not working for any non-English
install anyhow, because the needle had `LANGUAGE-english` as a
tag. We never noticed that. Failed password typing is pretty
rare now, so we can live without the protection - it's just nice
to have it for the English install tests because there's so many
of them.
Test Plan:
Run the Russian install with a recent Rawhide image,
check it clicks 'Done' twice. Note, it will still fail, because
of RHBZ #1413813.
Reviewers: jskladan, garretraziel_but_actually_jsedlak_who_uses_stupid_nicknames
Reviewed By: garretraziel_but_actually_jsedlak_who_uses_stupid_nicknames
Subscribers: tflink
Differential Revision: https://phab.qa.fedoraproject.org/D1084
The F25 'desktopencrypt' images built by createhdds boot to a
console passphrase entry screen, not graphical. F24 images
built the same way boot to a graphical passphrase entry screen.
I'm not sure why, but it's not really worth spending a lot of
energy on, I don't think - let's just add a needle to cover
this case.
Summary:
Include some basic testing of Japanese input, and split the
input testing (including Russian) into a separate module, since
it's not really part of 'login' testing.
Test Plan:
Run the test, and the Russian and French tests too to
make sure they didn't break. Tested on staging. Note the Japanese
test soft fails, intentionally, at present, as I discovered a bug
while working on it:
https://bugzilla.gnome.org/show_bug.cgi?id=776189
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D1072
Summary:
This isn't in the criteria, but it's commonly used, so we ought
to test this way. Require authentication for the iSCSI target
and have the test provide the appropriate auth info.
Test Plan:
Run the iscsi test and check it works (you need the
recent fixes for support_server to make *that* work). Nothing
else should be affected.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D1070
Summary:
The non-English tests so far did not test that graphical login
worked as expected, which is a fairly large hole. With this
change, they should do a Workstation install and test login to
both GNOME and the console works as expected. KDE is not yet
tested.
As part of this we tweak the implementation of keyboard layout
switching in graphical environments to use a generic function
in main_common which can handle both anaconda and desktops
(just GNOME at present, but should extend easily to any desktop
with a known switcher key and a visible layout indicator),
replacing the anacondatest class method. I kinda don't like that
the test has to specifically tell the function when it's in
anaconda, but I don't think I want to start experimenting with
a global 'test phase' openQA variable or anything like that at
present.
Fixes T842.
Test Plan:
Run the French and Russian install tests and check
they work as expected. Also run an English Workstation install
if you like, and make sure that didn't break. This change is
live on staging ATM, seems to work fine.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Maniphest Tasks: T842
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D1071
Summary:
GNOME's update notification criteria are pretty braindead: it
fires the update check timer once on login then once every hour
thereafter, but only actually checks for and notifies of updates
once a day if it's after 6am(?!?!?!). So we have to do a bunch
of fiddling around to ensure we reliably get a notification.
Move the clock to 6am if it's earlier than that, and reset the
'last update check' timer to 48 hours ago, then log in to GNOME
after that.
Note: I thought this still wasn't fully reliable, but I've looked
into all the recent failures of either test on staging and
there's only one which was really 'no update notification came
up', and the logs clearly indicate PK did run an update check,
so I don't think that was a test bug (I think something went
wrong with the update check). The other failures are all 'GNOME
did something wacky', plus one case where the needle didn't quite
match because I think the match area is slightly too tall; I'll
fix that in a second.
Test Plan:
Run the tests on both KDE and GNOME and check they
work properly now (assuming nothing unrelated breaks, like KDE
crashing...)
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D1039
OK, we once again hit an annoying case where the button match
that doesn't include text got matched in the sidebar:
https://openqa.fedoraproject.org/tests/39576#step/_do_install_and_reboot/13
so I think it's time to give up on that approach and just go
ahead and deal with the button text varying between live and
traditional installs. So let's change the existing needles to
include the button text and add one new needle to cover the
English live F25/F26 case. I don't think we need a 'quit'
needle for F24 (as the only thing we test for F24 is the Atomic
installer image, which says 'Reboot') or for French or Russian
(as we only test a traditional installer image for those, not
a live image).
Summary:
I started out wanting to fix an issue I noticed today where
graphical upgrade tests were failing because they didn't wait
for the graphical login screen properly; the test was sitting
at the 'full Fedora logo' state of plymouth for a long time,
so the current boot_to_login_screen's wait_still_screen was
triggered by it and the function wound up failing on the
assert_screen, because it was still some time before the real
login screen appeared.
So I tweaked the boot_to_login_screen implementation to work
slightly differently (look for a login screen match, *then* -
if we're dealing with a graphical login - wait_still_screen
to defeat the 'old GPU buffer showing login screen' problem
and assert the login screen again). But while working on it,
I figured we really should consolidate all the various places
that handle the bootloader -> login, we were doing it quite
differently in all sorts of different places. And as part of
that, I converted the base tests to use POSTINSTALL (and thus
go through the shared _wait_login tests) instead of handling
boot themselves. As part of *that*, I tweaked main.pm to not
require all POSTINSTALL tests have the _postinstall suffix on
their names, as it really doesn't make sense, and renamed the
tests.
Test Plan: Run all tests, see if they work.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D1015
When a package is unsigned, KDE will prompt for authentication.
Let's handle this. But count it as a soft fail, because
puiterwijk claims that Rawhide packages will be autosigned
from next week, so this *should* not happen and would indicate
an unsigned package in the repos. We make the KDE 'update
complete' needle area smaller because the wider area includes
some transparency and so will only match if the update applet
is open; this area will match whether it's open (no auth case)
or not open (auth case - the applet seems to disappear after
you provide the password in the auth prompt).
Pushing without review as the test is in production so I want
to make sure it works correctly.
(Also, hey, check out that array match for assert_screen and
that match_has_tag! This is gonna make some things so much
easier...thanks upstream)
Summary:
this more or less covers desktop_error_checks and desktop_
update_notification, though it can't really distinguish
between them easily. All we know is that if both the live and
postinstall versions of this test pass, both of those tests
pass. Any fails will have to be investigated manually.
Test Plan:
Run the tests for both KDE and Workstation, see
what happens. Workstation will fail for F25 and Rawhide at
present, due to SELinux/abrt notifications.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D1004
Summary:
Very similar to the CLI update test, but using the desktops'
update applications. This is based off the CLI update test
branch as it uses the shared functions that branch introduced.
We do not use the fake update packages, as they don't really
do anything useful for these tests; for dnf they can help us
distinguish between issues with the dnf mechanism and issues
with the repos, but we can't really tell that in the graphical
case. So we only use the python3-kickstart package here.
Test Plan:
Run the test on both KDE and GNOME and ensure it
performs as intended. I've been testing it on staging, so you
can see it there.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D1010
Summary:
by waiting for the bootloader in _boot_to_anaconda rather than
_console_wait_login, we can ensure that we use the anaconda
post-fail hook and thus get logs uploaded when a kickstart
install fails.
Test Plan:
Run a kickstart install test that fails and check
anaconda logs get uploaded. Then run one that works and make
sure it...still works.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D1005
seems to affect the network spoke a lot but not anywhere else.
Odd. Anyhow, here we go. Keeping the old needles for now as
F25 still has the old rendering.
Summary:
I've been wanting to do this for a while, but
https://github.com/os-autoinst/openQA/issues/786 is making it
difficult. Still, I think the quantity of needles is slowing
down openQA, so I'm doing it the old-fashioned way - looking
through test results and seeing what needles are actually used
now.
Test Plan:
Run full test suite for 24 Atomic, 25 and Rawhide and
make sure all tests still work. This is currently deployed on
staging (along with key-fixes) and I'm testing it there.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D994
Summary:
the main thing this does is try and type slower in X - this
should cover nearly everywhere we type anything in X, and make
it type slower. We also add a bit more safety checking to some
old tests which didn't have it (mainly _do_install_and_reboot)
- wait_still_screen after typing to make sure all the keypresses
were registered before continuing.
This is an attempt to mitigate the problems we've seen where
the wrong text gets typed into the wrong places and the tests
break.
This branch is live on staging atm. It still has *some* issues,
but I do think it's an improvement.
Test Plan:
run the tests (probably several times), compare to
runs without the change, see if it's better or worse...
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D993
so what seems to be going on on the software selection screen
is some kind of GTK+ bug:
https://bugzilla.gnome.org/show_bug.cgi?id=771127
the radio button bullets don't always render correctly - I think
they're not always completing a transition they go through on
selection. I think they can get stuck in any state between 'grey
and small' and 'black and big', but for now these are the needles
I've managed to create from failures; we're missing a 'greyer
KDE' variant, but if that happens, we can add it. If the bug
gets fixed we should be able to drop all these.
these rarely get used any more. they're only used when we're
running the filesystem tests on a non-Server image, where the
default fs is ext4 not xfs. With the old nightly process, where
we only got a generic boot.iso, we used them all the time, but
now we get product-ized nightlies, we rarely do. But I did hit
them today running the universal tests with a generic boot.iso
I hand-built to test an anaconda update, so in case we need to
do that kind of thing again in future, we may as well update
the needles.
Summary:
we have a long-standing problem with all the tests that hit
the repositories. The tests are triggered as soon as a compose
completes. At this point in time, the compose is not synced to
the mirrors, where the default 'fedora' repo definition looks;
the sync happens after the compose completes, and there is also
a metadata sync step that must happen after *that* before any
operation that uses the 'fedora' repository definition will
actually use the packages from the new compose. Thus all net
install tests and tests that installed packages have been
effectively testing the previous compose, not the current one.
We have some thoughts about how to fix this 'properly' (such
that the openQA tests wouldn't have to do anything special,
but their 'fedora' repository would somehow reflect the compose
under test), but none of them is in place right now or likely
to happen in the short term, so in the mean time this should
deal with most of the issues. With this change, everything but
the default_install tests for the netinst images should use
the compose-under-test's Everything tree instead of the 'fedora'
repository, and thus should install and test the correct
packages.
This relies on a corresponding change to openqa_fedora_tools
to set the LOCATION openQA setting (which is simply the base
location of the compose under test).
Test Plan:
Do a full test run, check (as far as you can) tests run sensibly
and use appropriate repositories.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D989
Summary:
the dictionary error bug was fixed some time back, so drop this
workaround for it.
Test Plan:
Run all tests for F25 and Rawhide and verify they don't need
this workaround any longer.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D988
this needle can actually also match the 'Delete all' button,
resulting in the test not doing what it should. openQA should
prefer the match closer to the area's location in the needle,
but https://openqa.fedoraproject.org/tests/31136 seems to show
this not happening, so let's make the match area wider in all
versions of the needle so it should no longer match the Delete
all button.
font rendering has changed somehow in both Rawhide and F25
updates-testing, retake a bunch of needles for this. Some more
may be needed later in tests that are currently failed or
skipped for other reasons.
this isn't actually f25 specific, I don't think...the tests
that failed are the Workstation network install tests, which
use updates-testing, so I think they're getting the same font
rendering change as just showed up in Rawhide. So these should
actually be correct for both branches, let's just leave the
name as the date.
looks like some font change occurred at some point, this test
hasn't reached this point for a long time so I can't really pin
down what changed, so just name it for the date.
seems like the overlay of the little 'speech bubble' triangle
changed a bit, with a recent GTK+ or something I guess. Since
this test is quite new we shouldn't need to keep the old
version, so just replace the screenshot.
Summary:
pretty simple stuff here. The distinction between 'firefox' and
'browser' is that the 'browser' needles I expect would also be
correct for other default browsers, while the 'firefox' needles
are specific to Firefox. We need '-kde' variants of some Firefox
needles where interface text is included, because the font is
Cantarell in GNOME but whatever the default 'sans' font is in
KDE - I suppose we should really use -thatfontsname rather than
-kde, but I can't think what it's called...
I couldn't do the 'log in to FAS' bit of the test since we don't
really have a sane way to provide a password while not exposing
it to the public.
Test Plan:
Run the test, check it works - for both KDE and
Workstation.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D938
Summary:
again, added as a non-fatal module for realmd_join_cockpit as
it's convenient to do it here. Also abstract a couple of ipa
bits into a new exporter package in the style of SUSE's
mm_network, rather than using ill-fitting class inheritance as
we have before - we should probably convert our existing class
based stuff to work this way.
Also a few minor tweaks and clean-ups of the other tests:
The path in console_login() where we detect login of a regular
user when we want root or vice versa and log out was actually
broken because it would 'wait' for the result of the 'exit'
command, which obviously doesn't work (as it relies on running
another command afterwards, and we're no longer at a shell).
This commit no longer actually uses that path, but I spotted
the bug with an earlier version of this which did, and we may
as well keep the fix.
/var/log/lastlog is an apparently-extremely-large sparse file.
A couple of times it seemed to cause tar to run very slowly
while creating the /var/log archive for upload on failure. It's
no use for diagnosing bugs, so we may as well exclude it from
the archive.
I caught cockpit webUI login failing one time when testing the
test, so threw in a wait_still_screen before starting to type
the URL, as we have for the FreeIPA webUI.
I also caught a timing issue with the openQA webUI policy add
step; the test flips from the Users screen to the HBAC screen
then clicks the 'add' button, but there's actually an identical
'add' button on *both* screens, so it could wind up trying to
click the one on the Users screen instead, if the web UI took
a few milliseconds to switch. So we throw in a needle match to
make sure we're actually on the HBAC screen before clicking the
button.
We make the freeipa_webui test a 'milestone' so that if the
new test fails, restoring to the last-known-good milestone
doesn't take so long; it actually seems like openQA can get
confused and try to cancel the test if restoring the milestone
takes a *really* long time, and wind up with a zombie qemu
process, which isn't good. This seems to avoid that happening.
Test Plan:
In the simple case, just run all the FreeIPA-related
tests on Fedora 24 (as Rawhide is broken) and make sure they all
work properly. To get a bit more advanced you can throw in an
`assert_script_run 'false'` in either of the non-fatal tests to
break it and make sure things go properly when that happens (the
last milestone should be restored - which should be right after
freeipa_webui, sitting at tty1 - and run properly; things are
set up so each test starts with root logged in on tty1).
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D935
Summary:
This requires us to handle decryption each time we reboot in
the upgrade process, so factor that little block out into the
base class so we don't have to keep pasting it. It's also a
bit tricky to integrate into the 'catch a boot loop' code we
have to deal with #1349721, but I think this should work. There
is a matching openqa_fedora_tools diff to generate the disk
image.
Test Plan:
Run the tests, check that they work, run the other
upgrade and encrypted install tests and check they still work
properly too.
Reviewers: garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D922
Summary:
as a new, non-fatal test step in the cockpit enrolment test,
because it kinda fits in there; we have an enrolled system with
a web browser *right there*. This will require making the wiki
reporting stuff slightly cleverer so we can say 'report a pass
for this wiki test instance if this test step passed', but that
should be possible. Making this non-fatal means the rest of the
cockpit enrolment test will go ahead even if the freeipa web UI
fails.
The 'check if we can log in' stuff is identical to freeipa_
client_postinstall except with different user names, so we could
potentially factor that out somehow, but I couldn't think of a
super clean way to do it so for now it's just copied.
Note this diff is on top of the freeipa-realmd branch which
is for D894, it's not on top of develop.
Test Plan:
Run the modified test and see if it works. No other
tests are modified, so they should be OK.
Reviewers: garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D895
now we're getting composes with the product name 'Fedora-Atomic'
this needle doesn't match them. Tweak it so it should match both
'Fedora' and 'Fedora-Atomic'.
