aba2814611
Summary: This adds tests for the Server_cockpit_default and cockpit_basic test cases. Some notes: I was initially thinking of combining these into a single test with multiple test modules and coming up with a system for doing wiki reporting based on individual test module status, but because we'll also want to do a cockpit FreeIPA enrol test, I decided against it. We don't really want to combine all three because then we would skip the cockpit tests whenever FreeIPA server deployment failed, which isn't ideal. So since we'll need a separate FreeIPA enrolment test anyway it doesn't really make sense to go to the trouble of designing a system for loading multiple postinstall tests (though I have an idea for that!) and a per-module wiki reporting system. This was the most minimal and hopefully reliable method for running Cockpit from a stock Server install that I could think of. An alternative approach would be to have, say, the most recent stable Workstation live as a 'stock' asset and have two tests, one which runs a stock Server install and just waits and another which boots the live image and accesses the cockpit running on the other box, but that seems a bit over-complex. It is not possible to have dependencies between tests for different ISOs, in case you were wondering about having a Workstation live test which runs parallel with a Server DVD test, we can't do that. One funny thing is the font that winds up getting used for the desktop, but I don't *think* that should be a problem. Picking needles was a bit tricky; any improvement suggestions are welcome. I'm hoping it turns out to be safe to rely on some dbus log messages being present; I think logging into Cockpit triggers activation of the realmd dbus interface, so there *should* always be some messages related to that. An alternative would just be to match on a sliver of the dark grey table header and the light grey row beneath it and assume that'll always be the first message (whatever the message is), but then we have to find some area of the message details screen which is always present for any message, and it just seems a tad more likely to result in false passes. Similary I'm making an assumption that auditd is always going to show up on the first page of the Services screen and the details screen will always show that 'loaded...enabled' text. Test Plan: Run the tests and see if they work! See https://openqa.stg.fedoraproject.org/tests/21373 and https://openqa.stg.fedoraproject.org/tests/21371 for my tests. Reviewers: garretraziel Reviewed By: garretraziel Subscribers: tflink Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D874 |
||
---|---|---|
lib | ||
needles | ||
tests | ||
.arcconfig | ||
.gitignore | ||
COPYING | ||
main.pm | ||
README.md | ||
templates | ||
VARIABLES.md |
openQA tests for the Fedora distribution
This repository contains tests and images for testing Fedora with openQA. For additional tools, Installation Guide and Docker images, see this repository.
Test development
See official documentation on basic concept, test development (including API specification), needles specification and supported variables for backend. See this example repo on how tests should be structured.
main.pm modular architecture
Since openQA uses only one entrypoint for all tests (main.pm), we have decided to utilize
this feature and make tests modular. It means that basic passing through main.pm (without any variables set)
results in most basic installation test executed. Developer can customize it with additional variables
(for example by setting PACKAGE_SET=minimal
to do installation only with minimal package set).
Fedora installation (and consequently main.pm) consists of several parts:
-
booting into Anaconda or booting live image and starting Anaconda
Since there isn't much variation between tests in this step, we have developed universal
_boot_to_anaconda.pm
test that is loaded automatically each time except whenENTRYPOINT
orUPGRADE
is set (see VARIABLES.md).To customize this step, you can set following variables:
GRUB
is appended to kernel line before boot. You can set for exampleinst.updates
here.- If
KICKSTART
is set, this part of installation ends here (program doesn't wait for Anaconda to appear). Note that you should setinst.ks
yourself by settingGRUB
variable. - If
LIVE
is set, program waits for desktop to appear and then clicks on "Install to Hard Drive" button.
-
customizing installation by interacting with Anaconda spokes
Most of the differences between tests take place in this part. If you want to add another installation test, you will probably put your variable checking and test loading here. All tests in this part should start on Anaconda's main hub and after they done its part, they should go back to Anaconda's main hub so that next test could be executed. In this phase, universal
_software_selection.pm
test is loaded that handles selecting what software to install.To customize this step, you can set following variables:
- Set
PACKAGE_SET
to install required package set on "Software selection spoke" - you have to provide correct needles with the name ofanaconda_${PACKAGE_SET}_highlighted
andanaconda_${PACKAGE_SET}_selected
. - Set
ENCRYPT_PASSWORD
to encrypt disk, value of this variable is used as an actual password.
- Set
-
installing Fedora and waiting for Fedora to reboot
After all customizations are finished,
_do_install_and_reboot.pm
test is automatically loaded. It starts installation, creates user and sets root password when required, waits for installation to finish and reboots into installed system. Only variables that control flow in this part are these:ROOT_PASSWORD
to set root password to this value.- When set,
USER_LOGIN
andUSER_PASSWORD
are used to create user in Anaconda.
-
post-install phase
After installation is finished and installed system is fully booted, you can run additional tests as checks that installed system has correct attributes - that correct file system is used, that RAID is used etc.
Make your test modular, so that it utilizes _boot_to_anaconda.pm
, _software_selection.pm
and
_do_install_and_reboot.pm
tests (that are loaded automatically). Break your test into smaller parts,
each dealing with one specific feature (e. g. partitioning, user creation...) and add their loading
into main.pm based on reasonable variable setting (so they can be used in other tests also).
Test inheritance
Your test can inherit from basetest
, fedorabase
, installedtest
or anacondatest
.
basetest
is basic class provided by os-autoinst - it has emptypost_fail_hook()
and doesn't set any flags.fedorabase
doesn't neither set flags nor does anything inpost_fail_hook()
, but it provides basic functions that will be useful during testing Fedora. It should be used when no other, more specific class can be used. It provides these functions:console_login()
handles logging in as a root/specified user into console. It requires TTY to be already displayed (handled by theroot_console()
method of subclasses). You can configure user and password by settinguser
andpassword
arguments. If you setcheck
argument to 1, this function dies if it fails to log in. Example usage:$self->console_login(user => "garret", password => "weakpassword");
logs in as usergarret
, with passwordweakpassword
.boot_to_login_screen()
handles booting from bootloader to login screen. It can take three optional arguments: first is the name of the login screen needle that should be displayed when system is booted, second is time how long still screen should be displayed until openQA decides that system is booted and third is timeout how long it should wait for still screen to appear. Example usage:$self->boot_to_login_screen("graphical_login", 30);
will wait until screen is not moving for 30 seconds and then checks, whethergraphical_login
needle is displayed.clone_host_resolv()
copies the contents of the host's/etc/resolv.conf
into the guest, overwriting any existing contents. This is mainly intended for use by openvswitch guests which need external connectivity.
anacondatest
should be used in tests where Anaconda is running. It uploads Anaconda logs (for exampleanaconda.log
orpackaging.log
) inpost_fail_hook()
. It also provides these convenient methods for Anaconda:root_console()
tries to login is as a root. It decides to what TTY to switch into and then callsconsole_login()
for root. If you setcheck
argument, it dies if it fails to log in. Example usage: after calling$self->root_console(check=>1);
, console should be shown with root logged in.select_disks()
handles disk selecting. It have one optional argument - number of disks to select. It should be run when main Anaconda hub is displayed. It enters disk selection spoke and then ensures that required number of disks are selected. Additionally, if$PARTITIONING
variable (set in Web UI) starts withcustom_
, it selects "custom partitioning" checkbox. Example usage: after calling$self->select_disks(2);
from Anaconda main hub, installation destination spoke will be displayed and two attached disks will be selected for installation.custom_scheme_select()
is used for setting custom partitioning scheme (such as LVM). It should be called when custom partitioning spoke is displayed. You have to pass it name of partitioning scheme and needleanaconda_part_scheme_$scheme
should exist. Example usage:$self->custom_scheme_select("btrfs");
usesanaconda_part_scheme_btrfs
to set partitioning scheme to Btrfs.custom_change_type()
is used to set different device types for specified partition (e. g. RAID). It should be called when custom partitioning spoke is displayed. You have to pass it type of partition and name of partition and needlesanaconda_part_select_$part
andanaconda_part_device_type_$type
should exist. Example usage:$self->custom_change_type("raid", "root");
usesanaconda_part_select_root
andanaconda_part_device_type_raid
needles to set RAID for root partition.custom_change_fs()
is used to set different file systems for specified partition. It should be called when custom partitioning spoke is displayed. You have to pass it filesystem name and name of partition and needlesanaconda_part_select_$part
andanaconda_part_fs_$fs
should exist. Example usage:$self->custom_change_fs("ext3", "root");
usesanaconda_part_select_root
andanaconda_part_fs_ext3
needles to set ext3 file system for root partition.custom_delete_part()
is used for deletion of previously added partitions in custom partitioning spoke. It should be called when custom partitioning spoke is displayed. You have to pass it partition name and needleanaconda_part_select_$part
should exist. Example usage:$self->custom_delete_part('swap');
usesanaconda_part_select_swap
to delete previously added swap partition.
installedtest
should be used in tests that are running on installed system (either in postinstall phase or in upgrade tests). It uploads/var/log
inpost_fail_hook()
. It provides these functions:root_console()
tries to login is as a root. It switches to TTY that is set as an argument (default is TTY1) and then callsconsole_login()
for root. If you setcheck
argument, it dies if it fails to log in. Example usage: running$self->root_console(tty=>2, check=>0);
results in TTY2 displayed with root logged in.check_release()
checks whether the installed release matches a given value. E.g.check_release(23)
checks whether the installed system is Fedora 23. The value can be 'Rawhide' or a Fedora release number; often you will want to useget_var('VERSION')
. Expects a console prompt to be active when it is called.
New test development workflow
- Select test from phabricator page.
- Put each part of your test as a separate file into
tests/
directory, reimplementingrun()
method andtest_flags()
method, inheriting from one of the classes mentioned above. - Set correct variables (so that all test parts you have made are executed) in WebUI -> Test suites.
- Link your newly created Test suite to medium type in WebUI -> Job groups.
- Run test (see openqa_fedora_tools repository).
- Create needles (images) by using interactive mode and needles editor in WebUI.
- Add new Job template and Test suite into
templates
file. - Add new Test suite and Test case into
conf_test_suites.py
file in openqa_fedora_tools repository. - Open differential request via phabricator, set openqa_fedora as a project and repository.
- Mark your test in phabricator page as done.
Language handling
Tests can run in different languages. To set the language which will be used for a test, set the LANGUAGE
variable for the test suite. The results of this will be:
- The value set will be typed into the language search box in anaconda.
- Any needle with at least one tag that starts with
LANGUAGE
will be unregistered unless it has the tagLANGUAGE-(LANGUAGE)
(where(LANGUAGE)
is the value set, forced to upper-case). - As a consequence, the chosen language will be selected at the anaconda Welcome screen.
It is very important, therefore, that needles have the correct tags. Any needle which is expected to match for
tests run in any language must have no LANGUAGE
tags. Other needles must have the appropriate tag(s)
for the languages they are expected to match. The safest option if you are unsure is to set no LANGUAGE
tag(s).
The only danger of this is that missing translations may not be caught.
Note that tags of the form ENV-INSTLANG-(anything)
are useless artefacts and should be removed. Due to
unfortunate design in openQA, any needle created in the web UI needle editor will have a ENV-INSTLANG-en_US
tag by default; this should be removed before submission.