Commit Graph

1238 Commits

Author SHA1 Message Date
Adam Williamson
e5728beada Add version 32 identification needle
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-02-14 16:53:31 -08:00
Adam Williamson
1eb4e3dca5 Add perl syntax check test, add it to CI
Inspired by openQA's 01-compile-check-all.t, this adds a perl
test which checks the syntax of main.pm and all lib and test
files, and hooks it up to CI. Requires os-autoinst and
perl-Test-Strict.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-02-13 15:28:09 -08:00
Adam Williamson
80ddb89f91 Fix firefox addon_success needle
I updated this last month without looking closely enough; one of
the matches was bogus. We now don't seem to get the 'added to
Firefox' popup any more, so we can't match on it, but I guess we
can match on the 'remove' button as an indicator that the add
worked.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-02-07 12:10:21 -08:00
Adam Williamson
66e53feb4d Update workarounds (drop update now stable, add F31 libdnf fix)
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-02-05 02:51:37 -08:00
Adam Williamson
214f2cc8eb Add fifloader tests, template schemas, update README
This adds a test suite for fifloader (renamed fifloader.py for
test sanity). It adds JSON Schema form schemas for both FIF and
upstream openQA template data, and has fifloader (optionally,
but by default) validate both input and output data against the
schemas. It also adds a tox.ini configured to run the fifloader
tests, use fifloader to validate the template files, and do diff
coverage and lint checks. It also adjusts the Zuul config to run
tox instead of the test job.

There are also some pylint cleanups, since the new tests run
pylint.

fifcheck, fifconverter and tojson.pm are removed, as they were
mainly only needed for one-time conversion of the old format
templates; now they are in the git history we can always recover
them if we need them.

Along with all this I updated the README a bit to explain some
of it (and explain FIF better), and to explicitly state that this
repo is GPLv2+ licensed, and added GPL headers to some of the
files.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-01-29 22:45:38 -08:00
Tristan Cacqueray
7b9a4306cd Add Zuul CI configuration 2020-01-27 13:34:31 +00:00
Adam Williamson
ceb147200a fifloader: Simplify generated job templates a bit
Again looking at the upstream template loader, we don't need to
pass a 'machine' hash with a 'name' key (we can just pass a
'machine_name' string), or a 'test_suite' hash with a 'name' key
(we can just pass 'test_suite_name' string), and we don't need
to pass a 'product' hash, we can just pass the product keys
directly in the job template hash. In fact the upstream loader
transforms 'machine' and 'test_suite' hash 'name' keys into the
strings and moves 'product' hash keys up into the top level hash
before POSTing to the API anyway.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-01-25 17:39:12 +01:00
Adam Williamson
07e57c638f Re-enable memory_check test (on Everything, with WS packages)
We were only running this test on the Workstation boot ISO...
then that was removed, and so we actually never ran this test at
all any more. We should run it on at least one image as check-
compose uses the output to check whether installer memory usage
has increased. I initially ran it on Workstation as that installs
the most packages; so let's run it on Everything but use the
Workstation package set.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-01-25 17:15:08 +01:00
Adam Williamson
80c54d5491 Add cockpit_updates and remote_logging tests to updates
Again, no reason not to run these on updates. Includes adding
oldcantarell versions of several needles for current cockpit,
as they're needed for the tests to pass. Also tweak a couple of
needles to avoid false matches (add more empty space).

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-01-24 21:03:41 +01:00
Adam Williamson
44d1dc3607 Add base_reboot_unmount and base_system_logging to update tests
No reason not to run these on updates as well. And it's much
easier with FIF!

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-01-24 20:50:39 +01:00
Adam Williamson
a348198cdc Correct except line in fifloader
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-01-24 19:48:22 +01:00
Adam Williamson
2c197d520c Add a whole intermediate template format ('FIF') and tools
I and @lruzicka (and I think @jskladan and @jsedlak and
@michelmno and everyone else who's ever touched it...) are being
gradually driven nuts by manually editing the test templates.
The bigger the files get the more awkward it is to keep them
straight and be sure we're doing it right. Upstream doesn't do
things the same way we do (they mostly edit in the web UI and
dump to file for the record), but we do still think making
changes in the repo and posting to the web UI is the right way
around to do it, we just wish the format was saner.

Upstream has actually recently introduced a YAML-based approach
to storing job templates which tries to condense things a bit,
and you can dump to that format with dump-templates --json, but
@lruzicka and I agree that that format is barely better for
hand editing in a text editor than the older one our templates
currently use.

So, this commit introduces...Fedora Intermediate Format (FIF) -
an alternative format for representing job templates - and some
tools for working with it. It also contains our existing
templates in this new format, and removes the old template files.
The format is documented in the docstrings of the tools, but
briefly, it keeps Machines, Products and TestSuites but improves
their format a bit (by turning dicts-of-lists into dicts-of-
dicts), and adds Profiles, which are combinations of Machines and
Products. TestSuites can indicate which Profiles they should be
run on.

The intermediate format converter (`fifconverter`) converts
existing template data (in JSON format; use tojson.pm to convert
our perl templates to JSON) to the intermediate format and
writes it out. As this was really intended only for one-time use
(the idea is that after one-time conversion, we will edit the
templates in the intermediate format from now on), its operation
is hardcoded and relies on specific filenames.

The intermediate format loader (`fifloader`) generates
JobTemplates from the TestSuites and Profiles, reverses the
quality-of-life improvements of the intermediate format, and
produces template data compatible with the upstream loader, then
can write it to disk and/or call the upstream loader directly.

The check script (`fifcheck`) runs existing template data through
both the converter and the loader, then checks that the result is
equivalent to the input. Again this was mostly written for one-
time use so is fairly rough and hard-coded, but I'm including it
in the commit so others can check the work and so on.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-01-24 15:21:23 +01:00
Adam Williamson
12b0fb04dc Update a ppc64le-only blivet needle for Cantarell 0.201
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-01-24 15:20:54 +01:00
Adam Williamson
776d31bd5b Update various Cockpit needles for 211
Note we need two variants of most of these, one for Rawhide with
Cantarell 0.201, one for F30/F31 with Cantarell 0.111.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-01-24 15:18:27 +01:00
Adam Williamson
f11a57f01a Drop match level on cockpit main needle
New cockpit hits 95%, let's just drop the required %age rather
than make a new needle.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-01-24 12:53:14 +01:00
Adam Williamson
b2139bf2dd Fully fix the reboot unmount test
OK, we need to define $self. And we're also gonna need to login
a bit harder later too.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-01-24 12:30:58 +01:00
Adam Williamson
9d985d8ebd Fix base_reboot_unmount on graphical installs
On graphical flavors we are not at a console when this test runs.
We need to do root_console to get there, and also bypass_1691487
for ppc64le. Copied from base_selinux.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-01-24 11:51:18 +01:00
Adam Williamson
8b28625ea2 Add a little script that converts the templates to JSON
For reasons! We are thinking about ways to improve the template
workflow, and this is useful for doing that.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-01-22 14:54:24 +01:00
Adam Williamson
0a4a787c03 Give Firefox longer to start
It seems to take a long time sometimes for some reason. Can't
pin it down but it's causing test flakes, so let's just let it
be. It *may* happen when chrony adjusts the system clock just as
Firefox is starting, for some reason.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-01-22 10:40:46 +01:00
Lukáš Růžička
23bae35d82 Create a test to test unmount errors during reboot (aka Reboot Testcase) 2020-01-21 16:55:43 +01:00
Lukáš Růžička
780a88f443 Fix indentation in templates. 2020-01-21 16:09:32 +01:00
Adam Williamson
c922226d7c Add a variant Cockpit join_complete needle
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-01-20 13:36:58 +01:00
Adam Williamson
49a540cc3d Update addon_success needle for new Firefox
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-01-20 13:36:28 +01:00
Adam Williamson
f8292bfeeb More Cantarell: Russian and French
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-01-18 23:56:11 +00:00
Adam Williamson
a653ded877 Next batch of Cantarell needles: Arabic install
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-01-17 15:32:30 +00:00
Adam Williamson
28642f4002 Move a stray app needle to the right place
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-01-17 15:19:17 +00:00
Adam Williamson
8612858b1f Second batch of Cantarell needles: GNOME apps tests
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-01-17 15:18:50 +00:00
Adam Williamson
21febe3592 First batch of updates for new version of cantarell
Still need to do the non-English languages. I'll do those
tomorrow.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-01-17 01:18:52 +00:00
Adam Williamson
d236c48152 Doh, fix stray quote mark from last commit
Where did I leave my brown paper bag...

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-01-16 18:57:17 +00:00
Adam Williamson
1098450fd6 Pull in F30 Cockpit 210 update as a workaround
Otherwise the new cockpit main screen needle match fails, because
'machine' is not on the top part of the screen.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-01-16 18:51:40 +00:00
Adam Williamson
13da4f2d26 Add another new Cockpit join complete needle
This one from ppc64le test. The difference between all the
recent needles here is the amount of white space between the
'Domain' and 'domain.local'. I don't want to change it to two
match areas as there's a chance we may get a false positive
match when the join actually failed (it may match on 'Domain'
in the right place, and 'domain.local' somewhere else just as
part of the system's hostname). I suppose if this keeps happening
we could try two match areas but include some white space to the
left of 'domain.local' to avoid matching on it as part of the
hostname...

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-01-15 15:22:34 +00:00
Adam Williamson
e9fa0b4bde Change type_safely in VNC tests to type_very_safely
We seem to have problems with the typing here on ppc64le, so
let's make it safer.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-01-15 15:13:44 +00:00
Adam Williamson
1e025c0b27 Tweak cockpit main needle to work on ppc64 too
We don't get a CPU model on ppc64, we get a machine ID everywhere
(I hope).

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-01-15 12:16:33 +00:00
Adam Williamson
e020b87d3a Drop the pseries-4.0 workaround for ppc64le
Should not be needed any more.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-01-10 14:45:09 -08:00
Adam Williamson
ba40ba3e4a Drop #1663050 workaround (fixed a while ago)
This block is kinda weird, but I don't want to fiddle with it
any more right now. I don't know why we do this check_screen
exactly like this.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-01-10 14:41:41 -08:00
Adam Williamson
0714143673 Drop a stray old comment and move a debug block up
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-01-10 14:34:58 -08:00
Adam Williamson
2aed12c6c1 Drop another old version check
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-01-10 14:28:49 -08:00
Adam Williamson
4903f7bfc3 Drop another old version check
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-01-10 14:27:51 -08:00
Adam Williamson
fec49f05ec Drop old rolekit code from database and freeipa tests
RIP rolekit

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-01-10 14:25:45 -08:00
Adam Williamson
fc93309b41 Drop _assert_and_click shim
We're on newer code on both deployments now, no longer needed.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-01-10 14:22:20 -08:00
Adam Williamson
dba4e8c57d Drop old version conditionals
We're well past 27 now...

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-01-10 14:20:44 -08:00
Adam Williamson
fcb06a874c Drop no-longer-needed update workarounds
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-01-10 14:18:54 -08:00
Adam Williamson
367433c298 Drop a debugging upload from server_remote_logging_server
This is failing a lot lately. I've no idea why, but it's not
really part of the actual test and we don't need it for debugging
all the time, so let's drop it for now.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-01-10 09:43:03 -08:00
Adam Williamson
35c863a312 Lower match level on another kernel.org patch needle
We're now getting 91-92% matches on this older needle.
¯\_(ツ)_/¯

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-01-10 08:25:55 -08:00
Adam Williamson
aa0edd263b Update Cockpit 'realm join complete' needle for new cockpit
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-01-08 17:28:49 -08:00
Adam Williamson
3998e9c86c Drop the 'cockpit join progress' needle match
The text we used to get has been replaced with a spinner, which
is difficult and unreliable to match on. This match was only
here to make the test fail a bit faster if it was broken, so
let's just live without it.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-01-08 15:32:29 -08:00
Adam Williamson
0926be0520 Drop similarity level on a flashing cursor wait_still_screen
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-01-08 15:31:32 -08:00
Adam Williamson
3067e0d223 Try and type safer in desktop_printing
We're getting issues with KDE's crazy-slow alt-f2 behaviour
here. So try and be even more conservative.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-01-07 08:09:12 -08:00
Adam Williamson
e3430e82ef Update Rhythmbox app needle
The icon changed, so the needle needs changing.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-01-06 12:32:44 -08:00
Adam Williamson
766d5f163e Update GDM input needles (an icon appeared in the text entry)
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-01-06 10:57:53 -08:00