This is a bit complex to automate, because we cannot really use
the production Zezere server (provision.fedoraproject.org) as
the test case shows, as we'd have to solve authentication and
we also don't really want to constantly keep registering new
hosts to it that are going to disappear and never be seen again.
So, instead we'll do it by setting up our *own* Zezere, and
provisioning our IoT system in that. We run two tests. The
'ignition' test is the actual IoT 'device'; all it really does
is boot up, sit around, and wait to be provisioned. The 'server'
test first sets up a Zezere server, then logs into it, adds an
ssh key, claims the IoT device, provisions it, and connects to
it to create a special file which tells the 'ignition' test
everything worked and it can close out.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This is to make the infra folks happy, apparently using 10.0.x.x
and 10.1.x.x is causing conflicts since our actual infra network
uses those ranges too.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
It actually is supposed to be installed by default, so if it's
missing that's a bug. It's been added to comps now so it should
be there from now on.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
It got split out and is not installed by default in 33-0.8. This
is intentional, not a bug, see https://pagure.io/fesco/issue/2114
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This adds a test that automates
https://fedoraproject.org/wiki/QA:Testcase_Clevis. It requires
os-autoinst-4.6-18.20200623git5038d8c or newer, and a worker
host in the 'tpm' class which is set up to have an instance of
swtpm running at /tmp/mytpmX , where X is the worker instance
number, for each worker. The Fedora infrastructure ansible
plays have been updated to handle this via an instantiated
systemd service, which other instances can also adopt.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
`cp -R foo/*` doesn't get all files in `foo/`, it misses hidden
files. This turns out to be a problem with recent anaconda, as
it expects to find a .treeinfo file here. So, let's use rsync.
We could probably do this with cp too but I can't think of the
right arguments right now...
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This adds a pair of tests, one which does almost all the work
from the test case, the other just a client test to check that
we can connect to an HTTP server running in a container on the
host. We also have to bump the _console_wait_login timeout on
this path a bit as we're booting a disk image that was installed
with DHCP working, but we change the network setup so DHCP does
not work any more, and the system spends quite some time trying
to bring the network up on boot before eventually giving up and
proceeding.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Here we are creating ~/.config for a newly-created user with root
ownership. We can't leave it that way, as commands run as the
user account won't be able to change it, as they should be able
to. So we need to change the ownership (and, just in case, fix
SELinux contexts) afterwards.
This was the real source of the problem we were seeing (the test
failing early due to the gsettings command which should turn the
screen background black failing).
Signed-off-by: Adam Williamson <awilliam@redhat.com>
We're trying to launch stuff the instant we see a desktop, and
it seems to be failing quite often in GNOME. Let's give it a few
seconds.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
IoT does nightly Branched and Rawhide composes that are built
as RC candidates, for some reason. So let's except it from this
check.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
I started out trying to fix os-release for the recent change to
add "Prerelease" tags to the VERSION and PRETTY_NAME fields, then
things spiralled. It got me thinking about the awkward DEVELOPMENT
variable we use, so I decided to get rid of it and refactor the
few things that use it. I refactored the anaconda prerelease tag
check, and wrote a new giant comment that gives details about
exactly how anaconda decides whether to show those tags, to give
context to our choices about when to expect them. This check now
uses a new LABEL variable the scheduler now sets. I also wound up
creating new UP1REL and UP2REL vars to define the 'source' release
for upgrade tests, separate from CURRREL and PREVREL, which are
now never lies - they really are the current stable and previous
stable release, even for update upgrade tests.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The update live image build test keeps running out of disk space.
We've bumped the minimal disk image from 12GB all the way up to
20GB so far but it keeps happening. So let's try a different
strategy: use a scratch disk to mount /var/lib/mock. That's where
all the space gets used. This should allow us to reduce the size
of the minimal disk image again, and giving it 25GB of empty disk
should avoid it running out of space again for a while.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This is failing on every update and that's not telling us anything
useful - we already know about the bug - so let's work around it.
Not adding a softfail as it's a bit more awkward to do that.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
We can get tripped up by the tutorial screen when launching
Boxes; borrow some code from the app start/stop test to check for
and handle it.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The default timeout for check_screen is 0, so we were only giving
the enter key press a fraction of a second to take effect before
expecting to see locked_screen_switch_user. This is too tight,
see https://openqa.fedoraproject.org/tests/586257 . Let's give it
five seconds before we give up.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Since GDM shows the "system-menu-button", it could not correctly
switch users on a locked screen. I added a check to see
if we are on a locked screen and behave accordingly.
This adds a new test that implementsQA:Testcase_desktop_login
on both GNOME and KDE.
While working on this, we realized that the "desktop_clean"
needles were really "app menu" needles, and for KDE, this was
a duplication with the new "system menu" needles, because on KDE
the app menu and the system menu are the same. So I (Adam)
started to de-duplicate that, but also realized that "app menu
button" is a much more accurate name for these needles, so I was
renaming the old desktop_clean needles to app_menu_button. That
led me to the realization that "check_desktop_clean" is itself a
dumb name, because we don't (at least, any more, way back in the
mists of time we may have done) do anything to check that the
desktop is "clean" - we're really just asserting that we're at a
desktop *at all*. While thinking *that* through, I *also* realized
that the whole "open the overview and look for the app grid icon"
workaround it did is no longer necessary, because GNOME doesn't
use a translucent top bar any more. That went away in GNOME 3.32,
which is in Fedora 30, our oldest supported release.
So I threw that away, renamed the function "check_desktop",
cleaned up all the needle naming and tagging, and also added an
app menu needle for GNOME in Japanese because we were missing
one (the Japanese tests have been using the "app grid icon"
workaround the whole time).
This stuff is kinda broken in various ways and halfline thinks
he can fix the underlying bug anyway. So let's go back to just
the GNOME live test being broken for now.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
I tested this workaround on staging before pushing it to git and
it worked, but then when I pushed it to prod it didn't work. On
stg I also had this to set GDM to debugging mode, so maybe this
is also needed for the workaround to work for some reason?
Signed-off-by: Adam Williamson <awilliam@redhat.com>
A GNOME bug seems to result in us getting to GDM, not a liveuser
desktop, after running 'systemctl isolate graphical.target' from
a live boot to runlevel 3 since the end of March. This works
around that to let the test run, as it's not really a failure of
the test per se.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Previous commit same summary had some side effect
solved by this new one.
And avoid a warning in autoinst-log when ABRT var not defined.
Signed-off-by: Michel Normand <normand@linux.vnet.ibm.com>
I merged the previous commit before realizing the ordering was
wrong. All other 'actions' lines have to come *before* the one
that adds 'reboot', because one of the conditions for that is
whether @actions is populated - basically, if we're taking any
actions, we also have to reboot afterwards. If we add an action
*after* that line (but no actions were added before that line),
we'll do it but then not reboot and the test will break.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This reverts commit 00b756f0e2.
Unfortunately, I made a typo in the script and the fix did not
work. I do not want to rebase the master (in order not to break
things for everyone) so I am reverting again.
Sorry.
This reverts commit d784bf54ca.
It turned out that Locations are not connected to Konqueror
at all. The reason why the test is failing is that the
application has been removed to limit the number of
web browsers.
We seem to be seeing the bug this works around:
https://bugzilla.redhat.com/show_bug.cgi?id=1765685
in F30 and F31 update tests even with this wait. At least, it
looks that way. Trying this to see if a longer wait helps.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
RHBZ #1692972 was fixed long ago, so we don't need to worry
about that any more. But this test failed on the recent F31 live
respin compose because it was changed to assume the tutorial
would appear on startup, which only happens on F32+. To make the
test work on F31 respins, let's handle both paths. Once F32 is
stable we can drop this as we won't run the test on F31 any more
after that.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
IoT created a branch that's basically Rawhide but is versioned
33. This causes the release_identification tests to fail. I don't
think they'll change this on their end, so let's just have the
test cope with it and expect branches versioned as the Rawhide
release number to behave as Rawhide does here.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
It seems to be actually installing fedora-release-silverblue now
so we get correct identification here. Update the tests.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
https://bodhi.fedoraproject.org/updates/FEDORA-2020-1070052d10#comment-1284223
It seems the rootfs on the Fedora 30 installer images we build at
present has gotten very big, so big that an update which contains
some very slightly larger firmware packages causes the rootfs to
be completely full (though lorax doesn't fail) and the image
doesn't boot.
I don't yet know when or why the rootfs got that big, but it's
not really a bug in this update, so for now let's just tell
lorax to use a bigger rootfs so the tests pass for this and any
similar future updates, until I can maybe find time to pinpoint
the culprit more precisely.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Initial implementation wasn't correct, I forgot CURRREL is not
'the pre-upgrade release version' but just 'the current stable
release'. This is a dumb way to figure out the correct release
number for this context but off-hand I can't think of a better
one.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
When there's a failed service we get a stupid bullet-point char
at the start of its line, and all the other lines are space-
padded to match indentation. Which is annoying! This (I hope)
ditches that crap without losing anything of value.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Also comment this better. We need to index from the end of the
string here, not the start, because going from the start breaks
when the compose shortname has a dash in it.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Per https://github.com/weldr/lorax/pull/881 it wasn't doing
anything anyway, and using it causes the command to fail in F32
and later.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
https://github.com/rpm-software-management/mock/commit/cb25d3ba
changes existing mock configs for stable releases to have a
`dnf.conf` section instead of a `yum.conf` section, and this
change got pushed out to F30 and F31, which breaks us :( Our
mock config that we use for building live images assumes the
existence of a `yum.conf` section in the config it inherits
from.
This change is now stable for F30 and F31, so at least we don't
have to do any conditional shenanigans; we can just change to
'the new style' unconditionally and things should work OK.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
IoT is becoming a release-blocking edition for F32, so we should
be testing it for sure. We may add specific tests, but for now
let's run the install and base tests on it.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
On graphical flavors we are not at a console when this test runs.
We need to do root_console to get there, and also bypass_1691487
for ppc64le. Copied from base_selinux.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This is failing a lot lately. I've no idea why, but it's not
really part of the actual test and we don't need it for debugging
all the time, so let's drop it for now.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The text we used to get has been replaced with a spinner, which
is difficult and unreliable to match on. This match was only
here to make the test fail a bit faster if it was broken, so
let's just live without it.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
We're getting issues with KDE's crazy-slow alt-f2 behaviour
here. So try and be even more conservative.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This match keeps giving us problems, and now I look at the test
again, there is actually no need for it at all. Clicking it
doesn't do anything, and we already confirm that we're on the
right page at the next step, where we look for a log entry and
click on that - that will fail if we aren't actually on the
Logs page.
I don't remember what Cockpit used to look like when we first
put this line and needle in, presumably there's a reason we had
them, but they're clearly unnecessary now.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
wait_idle is finally removed upstream in recent git os-autoinst.
This replaces all remaining wait_idles with sleeps, except for
one which is removed (I'm hoping improvements to typing in the
last few years should mean it isn't necessary any more, if it
turns out to be, I'll put it back as a sleep).
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This failed a couple times in a row on KDE live, let's see if
wait_still_screen is better than sleep.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
We need a couple of new needles, plus the 'join domain' button
has disappeared from the front page due to the very inefficient
UI redesign, so we need to scroll down to find it.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Lately launching anaconda on the KDE live image seems pretty
unreliable and we're not sure why. My last attempt to fix it
doesn't seem to be working, here's another effort based on the
idea it might be caused by moving the mouse from the hidden
position to the icon and back again, let's try moving the mouse
close to the icon before we assert and click it...
Signed-off-by: Adam Williamson <awilliam@redhat.com>
We still have a 'apps_run_firefox_stop' needle tag which is for
the same thing as 'firefox_close_tabs'. That's dumb. Get rid of
it and only have the firefox_close_tabs tag and needles. Also
clean up some old firefox_close_tabs needles that haven't matched
for months and all the 'apps_run_firefox_stop' needles.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
KDE tests are quite frequently failing lately because anaconda
doesn't launch. I'm hoping this will help.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
autocloud is dying soon. So, we want to run the tests in openQA
instead. This adds a test module called 'autocloud' and a test
suite called 'cloud_autocloud' which basically replicate what
autocloud does (download a tarball full of tests and run each
one), and the necessary template bits to run it on Cloud_Base
qcow2 images.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
...otherwise the VNC client tests fail on aarch64 because we try
to apply the 'console=tty0' workaround for #1661288. Fortunately
we don't really need that for the VNC install test to work, so
let's just skip it. We can make this more sophisticated later if
it turns out to be necessary.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
os-autoinst has this code when doing `script_run` on a serial
terminal that does a `wait_serial` for '# ' before running the
command. For some reason when we switch to the anaconda root
console and run a couple of commands after finishing install on
ppc64, the first of these prompt checks times out, which means
the test sits there doing nothing for 90 seconds unnecessarily.
Let's try and avoid that by hacking the prompt check regex to
be empty.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
There is nothing inherently 'root'-y about these so it makes no
sense to prefix their names with 'root-'. And why change from
'console' to 'terminal' compared to the naming used in the
actual qemu command and the log files? It's just confusing.
Let's be consistent (except for using - instead of _ here...
but - is easier to type!)
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This adds tests for vncconnect, a variation on VNC install using
reverse VNC (where the client waits for the server to connect to
it). Very similar to the regular-VNC test, but we have to use
tigervnc as Boxes doesn't do reverse VNC.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This adds a test for QA:Testcase_Anaconda_User_Interface_VNC -
the VNC install test case. It's implemented as a server/client
pair, with the server booting from the Server DVD image with
`inst.vnc` and the client booting from the desktop base disk
image, setting up networking, then running Boxes to connect to
the server and run the install.
There are various little tweaks to test loading and logic to
handle this, mostly pretty clear. We also move the workaround
for 'spurious auth prompt appears on desktop after you switch
away to a VT and back' out of the desktop update test and into
the `desktop_vt` helper function, since now this test can hit
it as well. We enhance _graphical_wait_login to handle the boot
loader if needed (it has never needed to until now).
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This handles a case where KDE shows a notification saying
'PIM Maintenance (Finished)', like this:
https://openqa.fedoraproject.org/tests/477345#step/desktop_notifications/34
we need to click it away for the desktop_notification test to
pass. It also clarifies the difference between this notification
and the eternal 'akonadi_migration_agent is doing something'
popup in the needle names and comments. It also replaces the
'check_screen then assert_and_click if found' pattern in several
notifications-related places with the better 'check_screen then
click_lastmatch if found' pattern now available upstream.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
We don't know exactly what's going on with this bug yet, but as
a 15 second sleep seems to avoid the issue, let's do that for
now so we catch any future issues.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
See https://openqa.fedoraproject.org/tests/473215 - it failed
because we tried to click FINISH CONFIGURATION while it was
still animating downwards.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The way this works at present, with a check_screen and then an
assert_and_click, there's a window where the check has passed
so we're committed to the assert, but it takes a half second or
so for the assert to actually complete (checking for a needle
is a somewhat heavy operation). During that half second the
'new update!' notification can...and quite often does...appear.
Changing the assert_and_click to a click_lastmatch should (I
hope) tighten this window; click_lastmatch should fire faster
than assert_and_click so there'll be less of a window for the
update notification to appear and break stuff.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This PR adds the following:
* moves out the presetting procedures, so that two long terminal tests do not have
to run twice
* add methods for application to register when successfully started
* adds a test that checks if all required applications have registered
This reduces the coverage of the identification test a bit but
also *substantially* simplifies it. We run into a ton of problems
when we try to check the version and prerelease text on screens
where it appears on banners:
* The banners differ between variants
* The pre-release text is translated
* The banners have gradients so for RTL languages, even if some
text is untranslated (e.g. 'Fedora 31') it appears on a
different background color than on LTR languages
* The prerelease text is dark red; if it appears on a dark blue
area of the banner this can trigger an os-autoinst needle
comparison bug: https://progress.opensuse.org/issues/56822
All of this together means we wind up continually fighting these
checks and we have a whole forest of needles just for them, and
it doesn't seem worthwhile. So let's drop all the places where
we were checking version and prerelease on banners, and only
check them in two places where they appear on a grey background,
which avoids most of the problems (we just need one version
needle per release, and one prerelease needle per language).
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This PR fixes#124.
It uses the BUILD variable to get the build string from the
tested compose and uses it to compare correct values for Silverblue.
Originally, it used Workstation values that did not match the
Silveblue ones.
Delete bad yank.
One of the test cases we didn't yet automate is:
https://fedoraproject.org/wiki/QA:Testcase_Kickstart_File_Path_Ks_Cfg
Now we have a PXE test, it's actually a good opportunity to test
that at the same time. I don't usually like combining tests like
this but in this case it sort of makes sense as otherwise we'd
have to have a whole parallel PXE install just to test this one
other detail. So, instead of doing an interactive PXE install as
we did at first, let's tweak the test to include a kickstart in
the initramfs and run the install from that.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
* server-cockpit-updates tests that Cockpit can be used to update the system.
* server-cockpit-autoupdate tests that users can use dnf-automatic for system
updates.
* cockpit functions were removed from utils.pm and put into an extra library
for cockpit - cockpit.pm which all cockpit tests are now using.
Review cockpit.pm
Review autoupdate test.
Review the update test.
Fix typo in cockpit.pm
Add sleep.
Add missing command.
Delete an unused needle.
mock is changing the config files to be based on templates,
which breaks how we generate the live build mock config. This
should work with both the old and new config files.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This is causing all kinds of trouble, because when the test is
run on the Server DVD - with the 'orange to blue' gradient - the
prerelease note is dark red text on a dark blue background.
os-autoinst actually reduces the color depth of images/needles
and greyscales them before performing the match...but for this
dark red text on dark blue background, the result seems to be
that the text and background come out *the same grey*, so *any*
text will match the needle (even if it's completely different
text), as will *no text at all*. I've tried finessing around
this a few times but it just keeps happening, so for now I'm
just disabling the pre-release text check at this point. We still
have the check during _boot_to_anaconda, when the text appears
on a *grey* background and so isn't a problem. I'm not removing
the needles yet, until we hear back from upstream:
https://progress.opensuse.org/issues/56822
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Building images with SELinux in permissive mode worked fine in
F29 and F30, but it seems to be broken in F31. Releng seems to
have it set up so the build environment has SELinux disabled,
so let's do the same. Will file a bug for this, as bcl says it
should be considered a bug.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This is doing slightly less to exercise the launcher menus and see
whether icons appear or disappear, but it's much faster and more
reliable. We do still use menu launch for one app, just to check
the mechanism works in general.
This adds a whole wodge of stuff to support_server to make it
act as a PXE server, then adds a new test which boots from PXE
and so should hit the PXE server. We use the NFS install repo as
that can be relied on to work for a support_server install.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
We do this in quite a few places and need to do it in another,
so let's just have a function for it. It takes a file glob
so we can have it run on a different one for _live_build.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Split this out of install_default, because it really is not a
part of that test and we do not want that test to fail because
the desktop background is wrong. Make it its own test module
and test suite instead. Don't do it on Rawhide, because we
really can't assert anything worthwhile about Rawhide at the
moment at least (this means the test runs but is a no-op and
will always pass on Rawhide, unfortunately). Move the needles
to a more appropriate location (this has nothing to do with
anaconda) and use 'background' not 'wallpaper' naming (that's
the name we use elsewhere in the project, e.g. package names).
Also, run the test on updates, and add an F29 needle for this
purpose.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
cockpit test often seems to fail because the click on a services
entry to open the services detail screen just gets lost. Let's
wait longer before trying it, and retry once if it fails.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The user and root spokes were moved from during-install to pre-
install hub in Rawhide. This should cope with that, while still
working for older images.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
There are three places where we basically want to click away
pop-up update notifications and the buggy akonadi_migration_agent
notification if it's there, in KDE tests. Let's share this code
between them, and also let's record soft failures for the buggy
cases in the desktop_notifications test.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Current KDE seems to like showing us multiple update available
notifications. So the test must dismiss all of them. See:
https://bugzilla.redhat.com/show_bug.cgi?id=1730482
Signed-off-by: Adam Williamson <awilliam@redhat.com>
* Add test to check module defaults.
* Add whitelist download.
* Fix install test to include selected profile to be on the safe side.
* Add test into templates.
Cockpit 198 comes with a UI overhaul, so almost all needles
need an update.
The 'auditd' service is no longer on the first page. To make
this less fragile (at the cost of not testing that clicking on
a service actually opens the detail page *for that service*,
tweak the needles to just look for *any* running service, click
on it, and check we got to a 'details' page. We also redo the
existing needles for this design.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
I'm pretty sure I worked around the modifier bug in os-autoinst,
so this shouldn't be necessary any more. See:
https://bugzilla.redhat.com/show_bug.cgi?id=1727388
Signed-off-by: Adam Williamson <awilliam@redhat.com>
We can run these tests much like support_server - have a test
which boots from the support_server disk image but with the ISO
to be tested attached, then mounts it and runs the scripts.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
@lruzicka added this to the KDE app tests in 71c4e273, but there
was no need for a new needle as I'd already done the same thing
in the desktop updates tests; let's just use the same needle.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This shouldn't be needed any more (the referenced bug was fixed
in F28) and doesn't work in non-live installer.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This whole block where we do various things at the console after
install completes was becoming a real mess. I had secret hopes
of killing it entirely at some point, but...that doesn't look
like it's gonna happen this century. So let's make it better
instead. The conditionals were getting very nested and icky and
it was hard to see what was actually going on. This rationalizes
things so first we figure out all the things we might want to do
at a console, then if we don't have anything to do at a console
we go ahead and hit the reboot button; otherwise we go to the
console and do all the things we need to do, including rebooting
unless this is the memory check test.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This just doesn't work when running the test on a graphical
install because we're at a desktop when we try and do it, not
a text console. So let's only try to do it if there is no
DESKTOP. See e.g.:
https://openqa.fedoraproject.org/tests/408484#step/base_selinux/2
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The way KDE does update notifications has changed - it's now a
permanent pop-up notification. This is a bit awkward for our
logic; it's hard to define a needle that proves this pop-up is
the only notification. Instead, let's dismiss it, then open the
notification tray and assert that there aren't any others. But
we also retain the old behaviour (more or less) for testing old
releases.
The popup notification also blocks the 'refresh' needle in the
systray and so breaks the desktop update test, so we deal with
that too.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
So, turns out new os-autoinst does *not* still accept the old
argument style for assert_and_click...and old os-autoinst
doesn't accept the new one. This adds a wrapper that handles
both, so our tests can work with old and new os-autoinst. We can
drop this once both deployments are on newer os-autoinst.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
In the new os-autoinst I just sent to staging, the old style
doesn't work any more, breaks all tests. This style should also
work with the older os-autoinst on stable.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
'use password' now seems to be default when creating a user in
text mode, so we need to handle that.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This was necessary for debugging the FreeIPA 4.8 pre-release
update bug, so let's have it for all runs, just in case.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
It seems that when this problem happens now we get *two* auth
requests, so we need to handle that.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
File doesn't exist for ostree installs. No point failing if this
fails, we may as well just continue and see what happens.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
No purpose is being served by all update tests failing on this
bug, so let's try and work around it.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This is covering the NFSISO_variation test as per:
https://fedoraproject.org/wiki/QA:Testcase_install_repository_NFSISO_variation
Change support_server to use support_5 disk image in templates
supposed to be previously created by updated createhdds tool
required for new test.
Signed-off-by: Guy Menanteau <menantea@linux.vnet.ibm.com>
Since the KDE menus have transparency set, any time the wallpaper
changes the menus will look different, and the app tests may
fail. This sets the desktop wallpaper to black at the start of
the test suite to avoid this problem.
It seems that there was a lag in button drawing, which prevented
OpenQA from fully recognizing the krusader_welcome needle because
it was evaluated too soon.
I added "wait_still_screen 3" to make sure, there is time for a
button to be drawn correctly before assertion.
Change method from assert_script_run to script_run
Change to reboot from console.
Move code in submethod, use ifs to make sure everything runs.
Fix chroot rebooting problem.
In the last few weeks Boxes is showing an auth dialog on start.
I've filed a bug on this; let's have the test handle it as a
soft failure, since this isn't a fatal problem. Do this by
making an existing needle for this dialog a bit more generic
in name and using it.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The KDE and GNOME 'apps' tests for Firefox both invented their
own 'oh look Firefox is running' needle, even though we already
had one. The GNOME one was broken by the removal of the app
title bar in Firefox 66.
Instead of having three needles for the same thing, let's just
throw the 'apps' ones out and use the pre-existing one from
needles/firefox for all cases.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Add png files as a background for the needles.
Rewrite the test handling methods to make them ready for KDE
Add the ABRT startstop test.
Make changes and corrections to the abrt test.
Add test for Firewall.
Add test for DNF dragora.
Add wait time for dnfdragora test.
Correct syntax.
Add Language test.
Make some changes to the DNF dragora test.
Add Users test case.
Add needles for DBUS viewer test.
Add Dbusviewer test.
Add Mahjong test and needles for games.
Add Minesweeper tests.
Add Patience test.
Add test for Document Viewer.
add test for gwenview
Add test for koulourpaint.
Add test for Kruler
Add test for Kcolorchooser
Reneedle failing needle.
Add ktorrent tests.
Add tests for CPT editors.
Add test for Krfb
Fix names for those files.
Add test for Kget
Add Akregator test.
Add test for Konversation.
Make Konversation end really.
Add tests for Kmail
Add test for PIM exporter.
Add test for KTnef and Krdc.
Fix problems after test runs.
Make more tries.
Fix needle to be found better.
Fix more errors.
Add test for Falkon.
Add tests for browsers.
Add support for closing tabs into Firefox test.
Add tests for K3B
move needles to correct directory
Add Kaddressbook test.
Add Kontakt text.
Add test for korganizer.
Add menu office needle and correct konqueror needle.
Add test for calligra stage.
Add test for Calligra.
Add test for network connections.
Modify needle for kaddressbook to prevent failing.
Add test for system settings and fix others.
Add test for FMW.
Add test for Dolphin
Add test for Infocenter.
Add test for kparted.
Fix a wrong needle.
Test relnotes.
Fix some errors in tests.
Add test for Discover.
Add test for Ksysguard.
Add tests for Konsole.
Add tests for KDE wallet.
Add tests for several utilities.
Add Krusader test.
Finish utility tests.
Fix some errors.
Fix needle for spectacle.
Add wait time to let Dragora wait for network.
As of Fedora-30-20190316.n.1, gnome-abrt is back under the
Utilities menu and the 'Sundry' menu seems to have disappeared.
Change the test back to using utilities and drop the sundry
needle.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
It seems 3 secs was a bit tight for recent Branched and Rawhide,
test was failing when the screen was just a bit slow to update
for some reason.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Sigh. It seems sometimes the button isn't visible because GNOME
pops up a fricking notification that blocks it.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
As there is *always* a base repo error when booting a netinst
image, this check was giving us spurious failures when running
universal tests on a netinst (which we are currently doing on
ppc64 it seems). Tweak it a bit to hopefully avoid this.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
wait_idle was deprecated and effectively it just sleeps, so
replace these wait_idles with explicit sleeps - I can't think
of a better choice here as we just need to wait for Firefox to
calm down after various operations.
The wait_still_screens were always hitting the 30 second time
out because of the flashing cursor; setting the similarity level
required to lower than the default 47 should fix this.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
ctrl-t seems to be failing often in Firefox tests, and I can't
figure out why. Let's try clicking the 'new tab' button instead
as a workaround.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Just like the installer image build test, only...it builds a live
image. This involves reimplementing quite a chunk of the Koji
livemedia task. Ah, well. Also involves rethinking the flavor
names a bit here, these seem...better.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
We're getting a spurious "Authentication required" for a
background refresh of the system repos sometimes, depending on
whether it happens while we're at a tty. This is now accepted
by upstream, so let's handle it and make it a soft failure for
now.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
We quite often want to run the update tests on a Koji task (not
a Bodhi update) for some reason - usually to test a potential
fix for an issue, or at a maintainer's request to test a change
before it is merged upstream and officially sent out as an
update. Up till now I've always hacked up utils.pm on the
staging server by hand to do this, which is horrible. Together
with a commit to fedora_openqa, this should allow us to do it in
a nice, sane way via the CLI. It's mostly just tweaking the
"updates" repo setup in utils.pm as you'd expect, but there's a
bit of subtlety to it because of the installer tests that use
%ADVISORY% as a variable substitution in the disk image name;
you can't do something like `%ADVISORY or KOJITASK%`, sadly, so
I had to have almost-redundant variables ADVISORY, KOJITASK and
ADVISORY_OR_TASK (we could kinda just live with ADVISORY_OR_TASK
except I didn't want to drop ADVISORY as it's an unnecessary
change from previous behavior).
Signed-off-by: Adam Williamson <awilliam@redhat.com>
inst.addrepo was added in F29. In F28 it should just sorta not
do anything at all. It's not worth doing a big workaround to
make F28 graphically add the repo in this case; let's just allow
F28 to not actually add the repo. It's fine! This is fine.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
In https://bugzilla.redhat.com/show_bug.cgi?id=1669256 it became
obvious that there's a missing feature in the new installer test
for updates: the update is both used in the image build process
and built into the installer environment itself, but it is not
actually included in the installed package set. This can be a
problem if the update has a bug that manifests *only* at install
time if it is in the install transaction (which is exactly the
case there), because the test will not catch this, and nor will
any other test.
So this commit makes `support_server` set up the update repo and
serve it out via NFS when it's run in an update context, and
makes the actual update install test run parallel with it and
use that repository. This way the install should include the
package(s) from the update. (It also of course means the test
fails if an update breaks NFS or something like that, but hey,
we want to know that!)
A parallel commit for fedora_openqa is necessary to add the
required CURRREL setting for the updates-installer flavor.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This adds a test which builds a netinst image potentially with
the package(s) from the update, and uploads that image. It also
adds a test which runs a default install using that image. This
is intended to check whether the update breaks the creation or
use of install images; particularly this will let us test
anaconda etc. updates. We also update the minimal disk image
name, as we have to make it bigger to accommodate this test,
and making it bigger changes its name - the actual change to
the disk image itself is in createhdds. We also have to redo a
bunch of installer needles for F28 fonts, after I removed them
a month or so back...
Signed-off-by: Adam Williamson <awilliam@redhat.com>
There seems to be an issue in Rawhide ATM which can cause the
'beta nag' screen to take a very long time to appear. Bump the
timeout to avoid tests failing on this.
https://bugzilla.redhat.com/show_bug.cgi?id=1666112
Signed-off-by: Adam Williamson <awilliam@redhat.com>
We don't want the tests to fail on this now we know what the
bug is, really - we want to find if there are any subsequent
fails, and allow the post-install tests to run also. So, let's
make it a soft failure.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
I'm pretty sure we got all the bugs this was working around
fixed. Again, if not, we can put this back!
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The bug never got explicitly addressed, but it's very old and
anaconda has been substantially changed since. The workaround
sometimes triggers erroneously now (because the icon sometimes
goes black while the spoke is still in 'Checking storage
configuration...' state), which is awkward. I can't be 100% sure
the bug doesn't sometimes still happen, but if it does, we'll
notice fairly soon, and we can tweak this and put it back.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
It seems Rawhide auto-signing is working fine now: openQA claims
this needle has 'never' been matched (which really means 'not
for a long time'), and I can't find any test fail in the last
year which looks like it landed on this 'authenticate' screen
but the needle failed to match, or anything like that.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
https://openqa.stg.fedoraproject.org/tests/424393 is a failure
where the 'Download' [updates] button was already visible when
we went to the tab. We already checked whether an 'apply' button
is visible and skipped the 'refresh' click if so, but because
the 'download' button is a new thing, we weren't skipping the
'refresh' click if 'download' was already visible.
So in this case, even though we could already see 'download', we
went ahead and clicked 'refresh'...then *immediately* started
looking for 'download'. It seems that Software did not refresh
and remove the 'Download' button *immediately* when we pressed
'refresh' - it left the 'Download' button visible briefly, and
*in this brief window*, we clicked it. *Then* Software kinda
'noticed' we'd clicked 'Update', and it seems it just sort of
throws away our click on 'Download' at that point and does the
refresh.
So at that point, the test thinks it's clicked 'Download' and
expects to see 'Apply', but actually the 'Download' click got
more or less thrown away, so the test fails, sitting at the
'Download' button.
To solve this, let's just extend the existing check to skip the
'refresh' click if 'download' *or* 'apply' are already visible.
There is a sort of possibility here that we could wind up
downloading and installing some updates that existed and were
noticed *before* we did our python3-kickstart trick, but not
install the python3-kickstart update, and cause the test to fail
because of that, but that doesn't seem to have happened before
when we were seeing the 'update' button, so I think I'm not
going to borrow trouble. If it happens, we'll deal with it I
guess.
The comment talks only about KDE, but clearly it can be the case
that an automatic check makes the button visible on GNOME too,
so let's rewrite the comment too.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The fix for this bug was sent to all releases now, so we should
not need the workaround any more. Let's kill it.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This is breaking the memory_check tests. I just reproduced it
manually and the UI *does* come back to life if you wait some
time; let's see if we can work around the bug this way.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
If an update test fails before reaching advisory_post, we don't
generate the 'what update packages were installed' and 'were
any update packages *not* installed when they should have been'
logs, but these may well be useful for diagnosing the failure -
so let's also do the same stuff there. Only let's not do it all
twice.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
We hit an interesting case in update testing recently:
https://bodhi.fedoraproject.org/updates/FEDORA-2018-115068f60e
An earlier version of that update failed testing. When we dug
into it a bit, we found that the test was failing because an
earlier version of the `pki-server` package was installed than
the version that was in the update; when asked (as part of
FreeIPA deployment) to install it, dnf had noticed that there
were dependency issues with the version of the package from the
update, but it happened to be able to install the version from
the frozen 'stable' repo...so it just went ahead and did that.
In this case, the 'missed' package resulted in a test failure,
but it'd actually be possible for this to happen and the test
to complete; we really ought to notice when this happens, and
treat it as a test failure.
So what this attempts to do is: at the end of all update tests,
check for all installed packages with the same name as a package
from the update, and compare their full NEVR to the one of the
package from the update. If a package with the same name as one
of the update packages is installed, but does not appear to be
the *same NEVR*, we fail, and upload the lists of packages for
manual investigation as to what the heck's going on.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
From local experimentation, it still actually produces the
output, even though it prints the message about the order being
wrong and exits 1.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Weirdly, occasionally some update tests seem to fail because
the 'comm' util we use to produce the list of packages from the
update that were actually tested during the job doesn't think
one of the input files is in alphabetical order, even though we
sort them both when they're produced. I don't know if this is
possibly due to the definition of 'alphabetical order' changing
as part of the update, or what. But we really shouldn't *fail*
the test when this happens, as it's not part of the functional
test, we're just producing convenience data. So, let's handle
the command failing, and if it happens, upload the input files
so we can maybe figure out why it's unhappy...
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The previous commit would lead to the 'workaround' getting hit
incorrectly, and might have had some other issues...tweak it a
bit.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
GNOME Software 3.30.5 split the offline update process into two
separate 'download' and 'apply' phases. So we need to handle
clicking 'download' before 'apply', if that happens.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Somehow, recently, FreeIPA tests are running into Firefox not
quitting because it's showing a warning about closing multiple
tabs. (I think we didn't *get* multiple tabs before but now we
do, for some reason). So let's work around this by clicking
"Close tabs" if the warning appears.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Lately, we can't be sure the desktop will be on tty1 after we
do 'systemctl isolate graphical.target'. For recent Workstation
lives it actually shows up on tty2.
We could be 'clever' and switch to tty2 on F29+ Workstation
lives...but actually it seems like if we just don't do anything,
systemd switches us to the correct tty. So let's rely on that,
at least as long as it's working.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
At least one test (desktop_notifications_postinstall) boots from
the disk image uploaded by install_default_upload, and needs to
access the grub menu. On F29+ Workstation this is failing,
because the grub menu is now hidden by default, so when the test
boots, it never sees the bootloader screen, and fails.
I considered trying to teach it to hold down shift or hit f8 or
esc at the right time, but that seems like it might be hard. So
instead let's just try to disable the hidden menu when we're
about to upload the installed system image. This is kinda going
against the 'preserve natural system behaviour' principle we try
to use for openQA, but I think it's OK as we do have other tests
that will exercise the 'hidden boot menu' stuff to some extent.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
On F27 we don't get a 'Software is up to date' screen because
there's an upgrade available. Let's work with the refresh button
instead.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
We're not seeing *exactly* #1314991 any more, but we're seeing
something that looks quite similar: the first attempt to find
updates just doesn't find any. No error message, no updates. I
have reported a bug for this and am investigating it, in the
meantime, let's restore the workaround, elaborated a bit, and
looking for the 'Software is up to date' screen instead of the
error message.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
I rather suspect the *bug* is still basically present and it's
why this test often fails, but we no longer seem to see the
*error message* which lets us detect the bug happening. This
needle has not been hit by any test for six months. So let's
remove the workaround as it adds complexity.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
For some reason, in recent tests, switching to a console after
live install completes is taking a long time, and tests are
failing because we 'only' allow 10 seconds for the login prompt
to appear. This seems to indicate some kind of performance bug,
but we don't really want all liveinst tests to fail on in, this
is not primarily a performance testing framework. So let's
tweak the root_console / console_login bits a bit to allow a
configurable timeout for the login prompt to appear, and use
that to wait 30 secs instead of 10 in this case.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The FreeIPA upgrade test didn't actually check that FreeIPA is
actually running after the upgrade and reboot, it just kinda
assumed it is. Let's add a check to the start of the 'check'
test module that makes sure ipa.service actually comes up to
'active' state. This'll make it clearer when tests are failing
because FreeIPA didn't come up right after the upgrade. The
check will run on non-upgrade tests too, but that's fine.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This adds a new test intended to just check boot chain things
for updates. It doesn't run any test modules besides the stock
update ones, but sets a variable, ADVISORY_BOOT_TEST, which
causes _advisory_update to do some additional stuff after
installing the updates but before rebooting: it forces regen
of the initramfs and bootloader config, and reinstalls the
bootloader on BIOS (not UEFI as it's not relevant). If the
following boot fails, we probably have a bug somewhere.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
There's this annoying problem where the screen sometimes goes
messed up after ipa-server-uninstall. 'clear' doesn't seem to
really work to fix it up either. Let's try flipping between
ttys. I don't like this much as it's already a pain trying to
work out / remember what tty we might possibly be on at any
given time, but I think we're always on either 1 or 3 here, so
let's do ctrl-alt-f1 ctrl-alt-f3 to ensure at least one change
and wind up on tty3...
Signed-off-by: Adam Williamson <awilliam@redhat.com>
