Commit Graph

89 Commits

Author SHA1 Message Date
Adam Williamson
9deb95a876 Unify base tests across normal, Cloud and arm
We have variant versions of several of the base tests which exist
to account for differences in required variable settings to run
essentially the same test in different situations. By twiddling
the variables a little, including inventing a new variable
defined in the flavors and substituted into the test suites so
the same test suite can have a different START_AFTER_TEST when
run on different flavors, I think we can unite them all and make
this a lot simpler.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-10-30 15:27:25 -07:00
Adam Williamson
92d52c6ac6 Split VNC client steps in two for aarch64 timing issue
So, there's a problem with how we figure out the NetworkManager
connection to use in setup_tap_static: it expects the first
connection in the list to be the right one, but this is only
actually true so long as it's *active*. When we're in the tap
case, it's usually not going to actually *work* out of the box
on boot (or else we wouldn't need setup_tap_static at all...),
so some time after boot, NetworkManager gives up on it and marks
it as inactive. And after that, setup_tap_static won't work any
more.

I never noticed this as a problem before because usually we do
setup_tap_static before that point. But it seems in the vnc
client tests, on aarch64, desktop boot and login is slow enough
that by the time we switch to a VT and try to setup the network,
we're very close to that cutoff, and sometimes miss it.

This, I hope, avoids the problem by doing the network setup in
that test before we deal with the desktop login, then doing the
desktop login, then doing the actual VNC bits.

The alternative here would be to figure out a better way to do
setup_tap_static, but I can't.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-10-30 12:39:52 -07:00
Adam Williamson
c7a1b94c84 Enable aarch64 disk image testing, related fixes
This sets us up to test the release-blocking aarch64 disk images
(Minimal, Server and Workstation). It also allows for testing
armhfp disk images on aarch64 worker hosts (though my testing of
that isn't going too well so far), and fixes the initial-setup
handling for a change upstream ('use password' is now the default
so we don't need to choose it). We rewire disk image deployment
test loading to work through the generic loader code rather than
using ENTRYPOINT, as it allows us to more gracefully handle
graphical (Workstation) vs. console (Server, Minimal), moving
the code for handling console initial-setup to a helper function
just like the code for gnome-initial-setup and having _console_
wait_login call it when appropriate. We also tweak desktop_vt a
bit because now we need to switch from a console running as test
to a desktop, which breaks the assumption that the highest
numbered session of user test is the desktop...

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-10-30 12:39:49 -07:00
Adam Williamson
478b7eff9e Add initial CoreOS product and test templates
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-09-02 14:49:29 -07:00
Adam Williamson
170ef0733a Use nmcli for static network stuff, not ifcfg files
This should work even if the ifcfg plugin is not present (hi,
CoreOS) or 'predictable' (har) network names are on.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-09-02 14:49:29 -07:00
Adam Williamson
47b4b3af60 Bump disk_ks version to 4
We changed the FreeIPA kickstarts for 172.16 network.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-08-24 12:39:17 -07:00
Adam Williamson
232b224d22 Add 'with swap' tests, drop swap parts from other tests (#180)
In Fedora 33, we generally no longer include a disk-based swap
partition by default (instead swap-on-ZRAM is used, see
https://fedoraproject.org/wiki/Changes/SwapOnZRAM ). This tweaks
our tests to account for that. In tests that aren't to do with
swap at all, we stop including a swap partition in order to be
closer to the default layout. We replace the old _no_swap blivet
and custom tests with _with_swap tests that, as the name implies,
*explicitly include* a swap partition, and adjust the postinstall
test to check the disk swap partition is there.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-08-11 15:09:33 -07:00
Adam Williamson
aa41fe4e4e Automate QA:Testcase_Zezere_Ignition
This is a bit complex to automate, because we cannot really use
the production Zezere server (provision.fedoraproject.org) as
the test case shows, as we'd have to solve authentication and
we also don't really want to constantly keep registering new
hosts to it that are going to disappear and never be seen again.

So, instead we'll do it by setting up our *own* Zezere, and
provisioning our IoT system in that. We run two tests. The
'ignition' test is the actual IoT 'device'; all it really does
is boot up, sit around, and wait to be provisioned. The 'server'
test first sets up a Zezere server, then logs into it, adds an
ssh key, claims the IoT device, provisions it, and connects to
it to create a special file which tells the 'ignition' test
everything worked and it can close out.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-07-23 18:01:06 -07:00
Adam Williamson
72edbfe991 Use qemu host IP 172.16.2.2 not 10.0.2.2
This is to make the infra folks happy, apparently using 10.0.x.x
and 10.1.x.x is causing conflicts since our actual infra network
uses those ranges too.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-07-23 16:40:45 -07:00
Adam Williamson
3a28227155 Bump QEMURAM on install_updates_nfs too
It's hitting the 2GiB crash bug as well.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-07-17 16:25:52 -07:00
Adam Williamson
6a46dfc2a4 Try harder to apply that 3GiB RAM setting
Seems it may need the magic override...

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-07-17 12:44:54 -07:00
Adam Williamson
ee3c27ccb7 Bump netinst images to 3GiB RAM (#1856514)
Since systemd-246~rc1-1.fc33, network install images crash if
booted with only 2GiB of RAM. No-one seems to be fixing this
as a matter of urgency, so let's just bump the tests to 3GiB at
least for now.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-07-17 09:25:57 -07:00
Adam Williamson
18aa1fbb80 Temporarily remove no_swap tests
https://fedoraproject.org/wiki/Changes/SwapOnZRAM changes things
a lot here. No swap on disk is now the default case, making these
tests obsolete. We'll probably want to test various different
configurations around ZRAM swap instead - ZRAM swap *and* disk
swap, disk swap only, all swaps disabled - but that's a more
complex change that will need a ticket and a PR, so for now let's
just cut these tests.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-07-16 15:38:31 -07:00
Adam Williamson
919c88f48f Add QA:Testcase_Clevis test (TPM-based automatic decryption)
This adds a test that automates
https://fedoraproject.org/wiki/QA:Testcase_Clevis. It requires
os-autoinst-4.6-18.20200623git5038d8c or newer, and a worker
host in the 'tpm' class which is set up to have an instance of
swtpm running at /tmp/mytpmX , where X is the worker instance
number, for each worker. The Fedora infrastructure ansible
plays have been updated to handle this via an instantiated
systemd service, which other instances can also adopt.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-07-02 16:44:55 -07:00
Adam Williamson
8d64da5f8d base tests: run on Cloud, add one on aarch64, clean up priorities
This adds new variant test suites for several of the base tests
to run them on Cloud disk images. We can't re-use the existing
test suites because they have `START_AFTER_TEST` defined and
there's no viable way to undefine it (there are things you can
do to override the value, but you can't just unset it). Also
add a stray test we weren't running on aarch64, and correct some
priorities.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-06-17 16:55:01 -07:00
Paul Whalen
fd440bfd3a Add QA:Testcase_RpmOstree_Rebase for iot testing (#167)
Signed-off-by: Paul Whalen <pwhalen@fedoraproject.org>
2020-06-04 14:00:32 -04:00
Lukáš Růžička
8c40583ee6 Automate the IoT rpm-ostree package layering test case. 2020-06-02 10:41:31 +02:00
Lukáš Růžička
9ac658cdd0 Use correct image for both tests. 2020-06-01 13:57:27 +02:00
Lukáš Růžička
228cc43565 Move tests to another product and use another base image. 2020-06-01 13:14:35 +02:00
Lukáš Růžička
ce584750b4 Create test for IoT Greenboot.
Finish test and add necessary needles.

Create IOT needle directory and move needles in it.

Delete needle as required by the review.

Add to templates.
2020-05-29 16:25:46 +00:00
Adam Williamson
14b21866f2 Add QA:Testcase_Podman test and run it on IoT (#166)
This adds a pair of tests, one which does almost all the work
from the test case, the other just a client test to check that
we can connect to an HTTP server running in a container on the
host. We also have to bump the _console_wait_login timeout on
this path a bit as we're booting a disk image that was installed
with DHCP working, but we change the network setup so DHCP does
not work any more, and the system spends quite some time trying
to bring the network up on boot before eventually giving up and
proceeding.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-05-28 14:55:27 -07:00
Lukáš Růžička
5a319ca76c Fix an error in test distribution. 2020-05-28 13:32:45 -07:00
Lukáš Růžička
4d4b2d7739 Finish the post review changes. 2020-05-28 13:32:42 -07:00
Lukáš Růžička
5f1db40ebe Add to templates. 2020-05-27 15:54:09 -07:00
Adam Williamson
acc46464f8 Revise release variable handling, prerelease checks, os-release
I started out trying to fix os-release for the recent change to
add "Prerelease" tags to the VERSION and PRETTY_NAME fields, then
things spiralled. It got me thinking about the awkward DEVELOPMENT
variable we use, so I decided to get rid of it and refactor the
few things that use it. I refactored the anaconda prerelease tag
check, and wrote a new giant comment that gives details about
exactly how anaconda decides whether to show those tags, to give
context to our choices about when to expect them. This check now
uses a new LABEL variable the scheduler now sets. I also wound up
creating new UP1REL and UP2REL vars to define the 'source' release
for upgrade tests, separate from CURRREL and PREVREL, which are
now never lies - they really are the current stable and previous
stable release, even for update upgrade tests.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-05-07 15:42:08 -07:00
Lukáš Růžička
f3d6a9574c Add desktop login test, revise and rename check_desktop
This adds a new test that implementsQA:Testcase_desktop_login
on both GNOME and KDE.

While working on this, we realized that the "desktop_clean"
needles were really "app menu" needles, and for KDE, this was
a duplication with the new "system menu" needles, because on KDE
the app menu and the system menu are the same. So I (Adam)
started to de-duplicate that, but also realized that "app menu
button" is a much more accurate name for these needles, so I was
renaming the old desktop_clean needles to app_menu_button. That
led me to the realization that "check_desktop_clean" is itself a
dumb name, because we don't (at least, any more, way back in the
mists of time we may have done) do anything to check that the
desktop is "clean" - we're really just asserting that we're at a
desktop *at all*. While thinking *that* through, I *also* realized
that the whole "open the overview and look for the app grid icon"
workaround it did is no longer necessary, because GNOME doesn't
use a translucent top bar any more. That went away in GNOME 3.32,
which is in Fedora 30, our oldest supported release.

So I threw that away, renamed the function "check_desktop",
cleaned up all the needle naming and tagging, and also added an
app menu needle for GNOME in Japanese because we were missing
one (the Japanese tests have been using the "app grid icon"
workaround the whole time).
2020-04-17 17:27:04 -07:00
Lukáš Růžička
d2f53799c9 Disable Modularity Checkdefaults tests
It is not clear, what the requirements currently are, so let us
disable the test for the time's being to stop getting failures
all the time.
2020-03-16 10:12:16 +01:00
Adam Williamson
81f8b783e7 Fix profile for FreeIPA replication tests on aarch64
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-03-12 17:37:21 -07:00
Adam Williamson
42385ac397 Enable two-release FreeIPA upgrade test on aarch64 too
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-03-12 16:19:29 -07:00
Adam Williamson
c3d4b29219 Enable FreeIPA replica and upgrade tests on aarch64
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-03-12 16:17:09 -07:00
Adam Williamson
ea39133fcf Enable remote logging tests on aarch64
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-03-12 16:16:15 -07:00
Guy Menanteau
85b71c809f Increase HDD size to 13G for install_package_set_kde test
to avoid failure in "disk_guided_empty" module,
at least required for PowerPC

Signed-off-by: Guy Menanteau <menantea@linux.vnet.ibm.com>
2020-03-05 09:11:27 +01:00
Guy Menanteau
fbacfa8ec5 Increase HDD size from 13 to 15G for language tests
because failing in "disk_guided_empty" module
at least for PowerPC.

Signed-off-by: Guy Menanteau <menantea@linux.vnet.ibm.com>
2020-03-05 09:11:27 +01:00
Adam Williamson
1bb38cbb16 Set gic-version=max for aarch64 qemu 'machine' arg
This is needed for qemu to run on the new aarch64 worker hosts,
per https://bugzilla.redhat.com/show_bug.cgi?id=1661976 .

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-03-03 16:41:04 -08:00
Adam Williamson
2e6cf6453d Tweak IoT initial-setup handling (only on F31)
initial-setup doesn't appear on IoT F32+, so we shouldn't expect
it.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-02-20 08:38:50 -08:00
Adam Williamson
04a2c22273 Set disk size for memory_check test to 13GB
It needs more space to install Workstation.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-02-20 08:10:28 -08:00
Adam Williamson
0f8f6082eb Enable IoT testing with install and base tests
IoT is becoming a release-blocking edition for F32, so we should
be testing it for sure. We may add specific tests, but for now
let's run the install and base tests on it.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-02-17 08:36:55 -08:00
Adam Williamson
07e57c638f Re-enable memory_check test (on Everything, with WS packages)
We were only running this test on the Workstation boot ISO...
then that was removed, and so we actually never ran this test at
all any more. We should run it on at least one image as check-
compose uses the output to check whether installer memory usage
has increased. I initially ran it on Workstation as that installs
the most packages; so let's run it on Everything but use the
Workstation package set.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-01-25 17:15:08 +01:00
Adam Williamson
2c197d520c Add a whole intermediate template format ('FIF') and tools
I and @lruzicka (and I think @jskladan and @jsedlak and
@michelmno and everyone else who's ever touched it...) are being
gradually driven nuts by manually editing the test templates.
The bigger the files get the more awkward it is to keep them
straight and be sure we're doing it right. Upstream doesn't do
things the same way we do (they mostly edit in the web UI and
dump to file for the record), but we do still think making
changes in the repo and posting to the web UI is the right way
around to do it, we just wish the format was saner.

Upstream has actually recently introduced a YAML-based approach
to storing job templates which tries to condense things a bit,
and you can dump to that format with dump-templates --json, but
@lruzicka and I agree that that format is barely better for
hand editing in a text editor than the older one our templates
currently use.

So, this commit introduces...Fedora Intermediate Format (FIF) -
an alternative format for representing job templates - and some
tools for working with it. It also contains our existing
templates in this new format, and removes the old template files.
The format is documented in the docstrings of the tools, but
briefly, it keeps Machines, Products and TestSuites but improves
their format a bit (by turning dicts-of-lists into dicts-of-
dicts), and adds Profiles, which are combinations of Machines and
Products. TestSuites can indicate which Profiles they should be
run on.

The intermediate format converter (`fifconverter`) converts
existing template data (in JSON format; use tojson.pm to convert
our perl templates to JSON) to the intermediate format and
writes it out. As this was really intended only for one-time use
(the idea is that after one-time conversion, we will edit the
templates in the intermediate format from now on), its operation
is hardcoded and relies on specific filenames.

The intermediate format loader (`fifloader`) generates
JobTemplates from the TestSuites and Profiles, reverses the
quality-of-life improvements of the intermediate format, and
produces template data compatible with the upstream loader, then
can write it to disk and/or call the upstream loader directly.

The check script (`fifcheck`) runs existing template data through
both the converter and the loader, then checks that the result is
equivalent to the input. Again this was mostly written for one-
time use so is fairly rough and hard-coded, but I'm including it
in the commit so others can check the work and so on.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-01-24 15:21:23 +01:00