I started out trying to fix os-release for the recent change to
add "Prerelease" tags to the VERSION and PRETTY_NAME fields, then
things spiralled. It got me thinking about the awkward DEVELOPMENT
variable we use, so I decided to get rid of it and refactor the
few things that use it. I refactored the anaconda prerelease tag
check, and wrote a new giant comment that gives details about
exactly how anaconda decides whether to show those tags, to give
context to our choices about when to expect them. This check now
uses a new LABEL variable the scheduler now sets. I also wound up
creating new UP1REL and UP2REL vars to define the 'source' release
for upgrade tests, separate from CURRREL and PREVREL, which are
now never lies - they really are the current stable and previous
stable release, even for update upgrade tests.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The old code waited after launching the terminal, the new code
doesn't, which led to a 'g' being swallowed in the first command
in https://openqa.fedoraproject.org/tests/592759 .
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This adds a new test that implementsQA:Testcase_desktop_login
on both GNOME and KDE.
While working on this, we realized that the "desktop_clean"
needles were really "app menu" needles, and for KDE, this was
a duplication with the new "system menu" needles, because on KDE
the app menu and the system menu are the same. So I (Adam)
started to de-duplicate that, but also realized that "app menu
button" is a much more accurate name for these needles, so I was
renaming the old desktop_clean needles to app_menu_button. That
led me to the realization that "check_desktop_clean" is itself a
dumb name, because we don't (at least, any more, way back in the
mists of time we may have done) do anything to check that the
desktop is "clean" - we're really just asserting that we're at a
desktop *at all*. While thinking *that* through, I *also* realized
that the whole "open the overview and look for the app grid icon"
workaround it did is no longer necessary, because GNOME doesn't
use a translucent top bar any more. That went away in GNOME 3.32,
which is in Fedora 30, our oldest supported release.
So I threw that away, renamed the function "check_desktop",
cleaned up all the needle naming and tagging, and also added an
app menu needle for GNOME in Japanese because we were missing
one (the Japanese tests have been using the "app grid icon"
workaround the whole time).
Remove a bunch of needles that have not been used for some time,
plus a few workarounds that are similarly stale.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
FreeIPA F31 -> F32 upgrade test is currently failing because
a new pki-core hit F31 stable but not F32 stable yet. It can't go
backwards on upgrade, that breaks stuff. The F32 update has been
pushed stable but just hasn't made mirrors yet as the last F32
nightly compose failed, so let's add it to the workarounds for
now.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Except one that's pushed stable but hasn't made repos yet (as the
last F32 nightly compose failed).
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Seems 5 seconds isn't long enough to wait here on aarch64, the
previous dialog hasn't always cleared by then. See e.g.
https://openqa.stg.fedoraproject.org/tests/753802
Signed-off-by: Adam Williamson <awilliam@redhat.com>
We seem to quite often get a failure in the blivet_lvmthin test
here which seems to be caused by trying to click 'OK' while the
'Device type' menu is still changing state or something. Let's
throw in a little delay.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This reverts commit adf3f91818.
The bug has been fixed in anaconda, so we can drop this, which
is good as it has timing issues producing false positives on
Rawhide...
This is pending stable, but looks like the update push won't
happen for a few hours, so I'm adding it as a workaround so we
can re-run the tests and get them to pass.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Rawhide seems to have developed a bug where a single disk is no
longer automatically selected as the install target when you
enter the INSTALLATION DESTINATION spoke. We need to work around
this (but register it as a soft failure).
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Inspired by openQA's 01-compile-check-all.t, this adds a perl
test which checks the syntax of main.pm and all lib and test
files, and hooks it up to CI. Requires os-autoinst and
perl-Test-Strict.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
It seems to take a long time sometimes for some reason. Can't
pin it down but it's causing test flakes, so let's just let it
be. It *may* happen when chrony adjusts the system clock just as
Firefox is starting, for some reason.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Otherwise the new cockpit main screen needle match fails, because
'machine' is not on the top part of the screen.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This block is kinda weird, but I don't want to fiddle with it
any more right now. I don't know why we do this check_screen
exactly like this.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
wait_idle is finally removed upstream in recent git os-autoinst.
This replaces all remaining wait_idles with sleeps, except for
one which is removed (I'm hoping improvements to typing in the
last few years should mean it isn't necessary any more, if it
turns out to be, I'll put it back as a sleep).
Signed-off-by: Adam Williamson <awilliam@redhat.com>
...because it comes out as 'cleqr'. Note, this may be fragile if
we start doing more stuff post-install, but for now I think it's
safe, I don't *think* we should ever hit this after running
`loadkeys us`.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
It seems like the situation where we need to pull an update from
updates-testing into all update tests to work around some known
issue is going to keep happening. So instead of constantly
adding and then entirely removing bespoke lines for each specific
workaround, let's have a permanent mechanism for doing this: a
hash with release numbers as keys, and arrayrefs of update IDs
as values, and a block to call `bodhi updates download` on the
appropriate array for the release under test. This way, to add
or remove a workaround you just update the hash. If we're at a
point where *no* workarounds are needed the %workarounds hash
can be made entirely empty (it must exist, though) and the code
will be a clean no-op.
The actual workaround here pulls in Lmod updates I just sent out
to work around this issue in one of the KDE update tests:
https://openqa.fedoraproject.org/tests/497160#step/base_update_cli/11
there's some code in Lmod that gets sourced in bash profiles
which breaks openQA's `validate_script_output` by blurping two
lines of informational output into the output of the script.
The update backports a change from upstream Lmod master that
sends that informational output to stderr instead of stdout.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The jss updates all went stable already. Now we have a problem
with SELinux, upower and container-selinux (we need a newer
selinux-policy to avoid upower failures in the services_start
test, but the first attempt to fix it caused the desktop_updates
test to start failing because container-selinux needed adapting
to changes in selinux-policy...let's just pull in the updates
with the latest versions of both to be safe), and one with NSS
that causes Firefox to give false certificate errors sometimes
(this is particularly affecting the FreeIPA browser test). As
usual these should be dropped once the updates go stable.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
On Rawhide Cloud_Base boots, there's some SSH key and network
information printed above the 'login:' prompt, so we can't
expect empty space there. Also tweak console_login() to clear
the screen after logging in, so the login prompt is cleared and
doesn't confuse things on subsequent runs (like it did first
time we tried this). And add a new user logged in needle, as it
seems after we clear the screen the tilde appears in a slightly
different position and the existing needle doesn't match.
https://openqa.fedoraproject.org/tests/489003#step/_console_wait_login/7
Signed-off-by: Adam Williamson <awilliam@redhat.com>
There is nothing inherently 'root'-y about these so it makes no
sense to prefix their names with 'root-'. And why change from
'console' to 'terminal' compared to the naming used in the
actual qemu command and the log files? It's just confusing.
Let's be consistent (except for using - instead of _ here...
but - is easier to type!)
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This adds a test for QA:Testcase_Anaconda_User_Interface_VNC -
the VNC install test case. It's implemented as a server/client
pair, with the server booting from the Server DVD image with
`inst.vnc` and the client booting from the desktop base disk
image, setting up networking, then running Boxes to connect to
the server and run the install.
There are various little tweaks to test loading and logic to
handle this, mostly pretty clear. We also move the workaround
for 'spurious auth prompt appears on desktop after you switch
away to a VT and back' out of the desktop update test and into
the `desktop_vt` helper function, since now this test can hit
it as well. We enhance _graphical_wait_login to handle the boot
loader if needed (it has never needed to until now).
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This handles a case where KDE shows a notification saying
'PIM Maintenance (Finished)', like this:
https://openqa.fedoraproject.org/tests/477345#step/desktop_notifications/34
we need to click it away for the desktop_notification test to
pass. It also clarifies the difference between this notification
and the eternal 'akonadi_migration_agent is doing something'
popup in the needle names and comments. It also replaces the
'check_screen then assert_and_click if found' pattern in several
notifications-related places with the better 'check_screen then
click_lastmatch if found' pattern now available upstream.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This PR adds the following:
* moves out the presetting procedures, so that two long terminal tests do not have
to run twice
* add methods for application to register when successfully started
* adds a test that checks if all required applications have registered
* server-cockpit-updates tests that Cockpit can be used to update the system.
* server-cockpit-autoupdate tests that users can use dnf-automatic for system
updates.
* cockpit functions were removed from utils.pm and put into an extra library
for cockpit - cockpit.pm which all cockpit tests are now using.
Review cockpit.pm
Review autoupdate test.
Review the update test.
Fix typo in cockpit.pm
Add sleep.
Add missing command.
Delete an unused needle.
This goes with a corresponding fedora_openqa commit that passes
ADVISORY_NVRS when scheduling update jobs. This is to address
https://pagure.io/fedora-qa/fedora_openqa/issue/78 so we can
publish spec-compliant and correct CI Messages: we will decide
what NVRs are in the update at scheduling time and always get
and test those NVRs, rather than the test downloading whatever
is in the update when it is run.
Some consequences: manual restarting or cloning of an update
test scheduled before this change lands will result in failure;
to do this you'll have to add the ADVISORY_NVRS value manually
with clone_job.pl, or simply reschedule the update with
fedora-openqa.
Manual restarting or cloning of an update test scheduled with
this change will always test the same NVRs. If the update has
changed and you want to re-test with the new packages in the
update, you must change ADVISORY_NVRS manually with clone_job,
or reschedule with fedora-openqa.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
reported in autoinst-log.txt
===
"my" variable $files masks earlier declaration in same scope at /var/lib/openqa/share/tests/fedora/lib/utils.pm line 352.
"my" variable $beta masks earlier declaration in same scope at /var/lib/openqa/share/tests/fedora/lib/utils.pm line 995.
===
Signed-off-by: Michel Normand <normand@linux.vnet.ibm.com>
We need another needle for French installs, and we need to use
ISO_URL for checking the ISO file name if ISO isn't set.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
On F31 update tests, desktop_vt is broken because the 'ps -C'
command is showing 'tty?' as the tty on which Xwayland/Xorg is
running. Let's try using loginctl as a workaround for this.
This ordering of commands should ensure the ps -C output takes
precedence when it's correct.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
A couple of tests failed in F31/Rawhide lately because quitting
Firefox (running on a bare X server) took more than 10 seconds
to get back to a console...not sure why, but it's not something
we ought to waste a lot of time on, let's just bump the timeout.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
We do this in quite a few places and need to do it in another,
so let's just have a function for it. It takes a file glob
so we can have it run on a different one for _live_build.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The log files are all under the ostree deploy root, the 'real'
system root has nothing useful. Try and find the deploy root
and prepend it to all the relevant commands if we're a CANNED
install.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
There are some situations in which ISO_URL is not set (e.g. a
manual POST to re-run tests after download has been done). If
there is no ISO_URL just skip the checksum check here.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
A broken lorax got into F29 stable and is causing live image
builds to fail in all F29 update tests. Work around that by
downloading a newer fixed package in _repo_setup_updates.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Restarting NetworkManager.service doesn't seem to apply our
newly-created static config in Rawhide any more (again!), since
NetworkManager 1.20.0-0.5.fc31 landed. This seems to work around
the bug.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Current tests sometimes seem to fail because X takes quite a
while to come up for some reason. I can't figure out why, and
this isn't properly a failure of the test (starting the X server
is an implementation detail), so let's just wait a bit longer
for now.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
There are three places where we basically want to click away
pop-up update notifications and the buggy akonadi_migration_agent
notification if it's there, in KDE tests. Let's share this code
between them, and also let's record soft failures for the buggy
cases in the desktop_notifications test.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
* Add test to check module defaults.
* Add whitelist download.
* Fix install test to include selected profile to be on the safe side.
* Add test into templates.
Restarting NetworkManager.service doesn't seem to apply our
newly-created static config in Rawhide any more, since
NetworkManager 1.20.0-0.3.fc31 landed. This seems to work around
the bug.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
type_very_safely commonly gets stuck waiting on a still screen
at the end because when you're typing, there's often a flashing
cursor, and the default similarity for wait_still_screen is 47
which is tight enough that a flashing cursor usually fails it.
So back it off to 45.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
We're not downloading all the packages from updates that contain
more than one builds ATM, which makes the test invalid and has
caused some false fails (and may even have caused false passes,
though I can't tell yet). Install a fixed Bodhi as a workaround
for this.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
So, turns out new os-autoinst does *not* still accept the old
argument style for assert_and_click...and old os-autoinst
doesn't accept the new one. This adds a wrapper that handles
both, so our tests can work with old and new os-autoinst. We can
drop this once both deployments are on newer os-autoinst.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
In the new os-autoinst I just sent to staging, the old style
doesn't work any more, breaks all tests. This style should also
work with the older os-autoinst on stable.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
'use password' now seems to be default when creating a user in
text mode, so we need to handle that.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
A BIND update broke a test, then went stable, so it's failing on
all updates right now. This subsequent update fixes the bug, so
let's pull it in here until it's pushed stable.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This was necessary for debugging the FreeIPA 4.8 pre-release
update bug, so let's have it for all runs, just in case.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Because we use check_screen not assert_screen here, the match
can actually fail, but the match_has_tag conditional can pass
on the *previous* match, if that happened to also be matching
on a console tag. We don't want that. Let's just make these
into assert_screens to avoid it; I don't think there's any path
where we're actually expecting this to work if those
check_screens didn't match.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Add png files as a background for the needles.
Rewrite the test handling methods to make them ready for KDE
Add the ABRT startstop test.
Make changes and corrections to the abrt test.
Add test for Firewall.
Add test for DNF dragora.
Add wait time for dnfdragora test.
Correct syntax.
Add Language test.
Make some changes to the DNF dragora test.
Add Users test case.
Add needles for DBUS viewer test.
Add Dbusviewer test.
Add Mahjong test and needles for games.
Add Minesweeper tests.
Add Patience test.
Add test for Document Viewer.
add test for gwenview
Add test for koulourpaint.
Add test for Kruler
Add test for Kcolorchooser
Reneedle failing needle.
Add ktorrent tests.
Add tests for CPT editors.
Add test for Krfb
Fix names for those files.
Add test for Kget
Add Akregator test.
Add test for Konversation.
Make Konversation end really.
Add tests for Kmail
Add test for PIM exporter.
Add test for KTnef and Krdc.
Fix problems after test runs.
Make more tries.
Fix needle to be found better.
Fix more errors.
Add test for Falkon.
Add tests for browsers.
Add support for closing tabs into Firefox test.
Add tests for K3B
move needles to correct directory
Add Kaddressbook test.
Add Kontakt text.
Add test for korganizer.
Add menu office needle and correct konqueror needle.
Add test for calligra stage.
Add test for Calligra.
Add test for network connections.
Modify needle for kaddressbook to prevent failing.
Add test for system settings and fix others.
Add test for FMW.
Add test for Dolphin
Add test for Infocenter.
Add test for kparted.
Fix a wrong needle.
Test relnotes.
Fix some errors in tests.
Add test for Discover.
Add test for Ksysguard.
Add tests for Konsole.
Add tests for KDE wallet.
Add tests for several utilities.
Add Krusader test.
Finish utility tests.
Fix some errors.
Fix needle for spectacle.
Add wait time to let Dragora wait for network.
This tweaks the update live and installer image build tests to
store the update repo on a separate disk. This should hopefully
avoid the tests failing due to insufficient disk space when the
update is huge (e.g. KDE or GNOME megaupdates).
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The initial implementation here has a problem if we spot the
'successful' screen briefly, then the system reboots normally,
reaches the bootloader and proceeds past it all within 10
seconds; in this case we'll never actually spot the bootloader
and do our stuff. This tweak should continue through the code
block immediately if the bootloader shows up during the ten
seconds, otherwise check again for the 'successful' screen and
reboot.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The f30 tag is currently a lot fresher than what's in the repos,
because we haven't had a compose for five days. This causes
problems in some F30 update tests because the update works fine
with what's in the `f30` tag but not what's on the mirrors. To
deal with this until we get a successful compose, let's just
enable the f30-build repo in the update test process...
Signed-off-by: Adam Williamson <awilliam@redhat.com>
We want to get accurate results from the rest of the upgrade
test and we know about this bug, so let's make it a soft fail
so we can see how the test is functioning otherwise.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Investigation suggests that the Firefox certificate validation
failure we sometimes get when loading the FreeIPA web UI in the
freeipa_webui test module is a race issue in Firefox itself:
https://bugzilla.mozilla.org/show_bug.cgi?id=1530429
refreshing the page seems to work around the problem, so let's
do that.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
We're getting an intermittent case where FreeIPA tests fail
because of a web server certificate issue. Click 'Advanced' in
Firefox when this happens so we can get a bit more info on the
problem.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
I've seen a few tests fail lately because Firefox suddenly
opened a tab with some "Privacy Study" in it and switched to it.
Per https://bugzilla.mozilla.org/show_bug.cgi?id=1529626 , this
should be the way to disable these...
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The previous attempt at this (in a20ea59) was wrong, because it
meant we didn't create the update repo at all, and we do need
the repo to exist in the support_server test every time (as the
point of the support server is to serve it out via NFS). We just
don't want to *use it on the support_server system itself* if
the releases don't match. So rejig things to do that properly.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Just like the installer image build test, only...it builds a live
image. This involves reimplementing quite a chunk of the Koji
livemedia task. Ah, well. Also involves rethinking the flavor
names a bit here, these seem...better.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
We quite often want to run the update tests on a Koji task (not
a Bodhi update) for some reason - usually to test a potential
fix for an issue, or at a maintainer's request to test a change
before it is merged upstream and officially sent out as an
update. Up till now I've always hacked up utils.pm on the
staging server by hand to do this, which is horrible. Together
with a commit to fedora_openqa, this should allow us to do it in
a nice, sane way via the CLI. It's mostly just tweaking the
"updates" repo setup in utils.pm as you'd expect, but there's a
bit of subtlety to it because of the installer tests that use
%ADVISORY% as a variable substitution in the disk image name;
you can't do something like `%ADVISORY or KOJITASK%`, sadly, so
I had to have almost-redundant variables ADVISORY, KOJITASK and
ADVISORY_OR_TASK (we could kinda just live with ADVISORY_OR_TASK
except I didn't want to drop ADVISORY as it's an unnecessary
change from previous behavior).
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This adds a test which builds a netinst image potentially with
the package(s) from the update, and uploads that image. It also
adds a test which runs a default install using that image. This
is intended to check whether the update breaks the creation or
use of install images; particularly this will let us test
anaconda etc. updates. We also update the minimal disk image
name, as we have to make it bigger to accommodate this test,
and making it bigger changes its name - the actual change to
the disk image itself is in createhdds. We also have to redo a
bunch of installer needles for F28 fonts, after I removed them
a month or so back...
Signed-off-by: Adam Williamson <awilliam@redhat.com>
If a test fails to the dracut shell, we currently don't do
anything useful. This should recognize when that happens, and
upload rdsosreport.txt.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The packages from this update seem to be breaking F28 update
tests for some reason; a later update has gone stable anyway, so
this is no longer needed.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
A bug showed up in Rawhide where, when you run startx in a tty,
when you exit that X session, the tty quits and returns to the
login prompt. This is a slightly sloppy workaround for that
problem.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
That clever-clever 'check the packages from the update were
installed' thing from yesterday breaks on kernel updates, as
they're installonly; after the update, the new version of the
package is installed, but the *old* version is too, and the way
I implemented the check, it treats that as a failure. Let's try
and handle this a somewhat-clever way (if this fails, I'm just
going to grep out lines with 'kernel' in them, as a *dumb* way).
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This should fix log collection when a French or Japanese test
fails before the test itself would have done this.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
If an update test fails before reaching advisory_post, we don't
generate the 'what update packages were installed' and 'were
any update packages *not* installed when they should have been'
logs, but these may well be useful for diagnosing the failure -
so let's also do the same stuff there. Only let's not do it all
twice.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
We hit an interesting case in update testing recently:
https://bodhi.fedoraproject.org/updates/FEDORA-2018-115068f60e
An earlier version of that update failed testing. When we dug
into it a bit, we found that the test was failing because an
earlier version of the `pki-server` package was installed than
the version that was in the update; when asked (as part of
FreeIPA deployment) to install it, dnf had noticed that there
were dependency issues with the version of the package from the
update, but it happened to be able to install the version from
the frozen 'stable' repo...so it just went ahead and did that.
In this case, the 'missed' package resulted in a test failure,
but it'd actually be possible for this to happen and the test
to complete; we really ought to notice when this happens, and
treat it as a test failure.
So what this attempts to do is: at the end of all update tests,
check for all installed packages with the same name as a package
from the update, and compare their full NEVR to the one of the
package from the update. If a package with the same name as one
of the update packages is installed, but does not appear to be
the *same NEVR*, we fail, and upload the lists of packages for
manual investigation as to what the heck's going on.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
There's really no point having separate error and error_report
needles. Just match on error_report as well as clicking on it.
Also add a new error_report needle for latest Rawhide fonts.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Sometimes we get a test failing because the SUT isn't connecting
to the network for some reason. In this case we never get any
logs, because `upload_logs` relies on being able to reacht at
least the worker host system via the network.
This attempts to detect when we can't ping the worker host, and
in that case, send some info out over the serial line instead.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
