os-autoinst-distri-rocky-mi.../README.md

157 lines
12 KiB
Markdown
Raw Normal View History

openQA tests for the Fedora distribution
========================================
This repository contains tests and images for testing [Fedora](https://getfedora.org/) with
[openQA](http://os-autoinst.github.io/openQA/). For additional tools, Installation Guide and
Docker images, see [this repository](https://bitbucket.org/rajcze/openqa_fedora_tools).
Test development
----------------
See official documentation [on basic concept](https://github.com/os-autoinst/openQA/blob/master/docs/GettingStarted.asciidoc),
[test development (including API specification)](https://github.com/os-autoinst/openQA/blob/master/docs/WritingTests.asciidoc),
[needles specification](https://github.com/os-autoinst/os-autoinst/blob/master/doc/needles.txt) and
[supported variables for backend](https://github.com/os-autoinst/os-autoinst/blob/master/doc/backend_vars.asciidoc). See
[this example repo](https://github.com/os-autoinst/os-autoinst-distri-example) on how tests should be structured.
### main.pm modular architecture
Since openQA uses only one entrypoint for all tests (main.pm), we have decided to utilize
this feature and make tests modular. It means that basic passing through main.pm (without any variables set)
results in most basic installation test executed. Developer can customize it with additional variables
(for example by setting `PACKAGE_SET=minimal` to do installation only with minimal package set).
Fedora installation (and consequently main.pm) consists of several parts:
1. booting into Anaconda or booting live image and starting Anaconda
Since there isn't much variation between tests in this step, we have developed universal `_boot_to_anaconda.pm`
test that is loaded automatically each time except when `ENTRYPOINT` or `UPGRADE` is set (see VARIABLES.md).
To customize this step, you can set following variables:
- `GRUB` is appended to kernel line before boot. You can set for example `inst.updates` here.
- If `KICKSTART` is set, this part of installation ends here (program doesn't wait for Anaconda to appear).
Note that you should set `inst.ks` yourself by setting `GRUB` variable.
- If `LIVE` is set, program waits for desktop to appear and then clicks on "Install to Hard Drive" button.
2. customizing installation by interacting with Anaconda spokes
Most of the differences between tests take place in this part. If you want to add another installation test,
you will probably put your variable checking and test loading here. All tests in this part should start on
Anaconda's main hub and after they done its part, they should go back to Anaconda's main hub so that next
test could be executed. In this phase, universal `_software_selection.pm` test is loaded that handles
selecting what software to install.
To customize this step, you can set following variables:
- Set `PACKAGE_SET` to install required package set on "Software selection spoke" - you have to provide correct needles with the
name of `anaconda_${PACKAGE_SET}_highlighted` and `anaconda_${PACKAGE_SET}_selected`.
- Set `ENCRYPT_PASSWORD` to encrypt disk, value of this variable is used as an actual password.
3. installing Fedora and waiting for Fedora to reboot
After all customizations are finished, `_do_install_and_reboot.pm` test is automatically loaded.
It starts installation, creates user and sets root password when required, waits for installation
to finish and reboots into installed system. Only variables that control flow in this part are these:
- `ROOT_PASSWORD` to set root password to this value.
- When set, `USER_LOGIN` and `USER_PASSWORD` are used to create user in Anaconda.
4. post-install phase
After installation is finished and installed system is fully booted, you can run additional tests
as checks that installed system has correct attributes - that correct file system is used, that
RAID is used etc.
Make your test modular, so that it utilizes `_boot_to_anaconda.pm`, `_software_selection.pm` and
`_do_install_and_reboot.pm` tests (that are loaded automatically). Break your test into smaller parts,
each dealing with one specific feature (e. g. partitioning, user creation...) and add their loading
into main.pm based on reasonable variable setting (so they can be used in other tests also).
### Test inheritance
Your test can inherit from `basetest`, `fedorabase`, `installedtest` or `anacondatest`.
- `basetest` is basic class provided by os-autoinst - it has empty `post_fail_hook()` and doesn't set any flags.
- `fedorabase` doesn't neither set flags nor does anything in `post_fail_hook()`, but it provides basic functions
that will be useful during testing Fedora. It should be used when no other, more specific class can be used. It provides
these functions:
- `console_login()` handles logging in as a root/specified user into console. It requires TTY to
be already displayed (handled by the `root_console()` method of subclasses). You can configure user and password
by setting `user` and `password` arguments. If you set `check` argument to 1, this function
dies if it fails to log in. Example usage: `$self->console_login(user => "garret", password => "weakpassword");`
logs in as user `garret`, with password `weakpassword`.
- `boot_to_login_screen()` handles booting from bootloader to login screen. It can take three optional arguments:
first is the name of the login screen needle that should be displayed when system is booted, second is time how
long still screen should be displayed until openQA decides that system is booted and third is timeout how long
it should wait for still screen to appear. Example usage: `$self->boot_to_login_screen("graphical_login", 30);`
will wait until screen is not moving for 30 seconds and then checks, whether `graphical_login` needle is displayed.
- `anacondatest` should be used in tests where Anaconda is running. It uploads Anaconda logs (for example
`anaconda.log` or `packaging.log`) in `post_fail_hook()`. It also provides these convenient methods for Anaconda:
- `root_console()` tries to login is as a root. It decides to what TTY to switch into and then calls `console_login()`
for root. If you set `check` argument, it dies if it fails to log in. Example usage:
after calling `$self->root_console(check=>1);`, console should be shown with root logged in.
- `select_disks()` handles disk selecting. It have one optional argument - number of disks to select. It should be
run when main Anaconda hub is displayed. It enters disk selection spoke and then ensures that required number of
disks are selected. Additionally, if `$PARTITIONING` variable (set in Web UI) starts with `custom_`, it selects
"custom partitioning" checkbox. Example usage: after calling `$self->select_disks(2);` from Anaconda main hub,
installation destination spoke will be displayed and two attached disks will be selected for installation.
- `custom_scheme_select()` is used for setting custom partitioning scheme (such as LVM). It should be called when
custom partitioning spoke is displayed. You have to pass it name of partitioning scheme and needle
`anaconda_part_scheme_$scheme` should exist. Example usage: `$self->custom_scheme_select("btrfs");` uses
`anaconda_part_scheme_btrfs` to set partitioning scheme to Btrfs.
- `custom_change_type()` is used to set different device types for specified partition (e. g. RAID). It should be
called when custom partitioning spoke is displayed. You have to pass it type of partition and name of partition
and needles `anaconda_part_select_$part` and `anaconda_part_device_type_$type` should exist. Example usage:
`$self->custom_change_type("raid", "root");` uses `anaconda_part_select_root` and `anaconda_part_device_type_raid`
needles to set RAID for root partition.
- `custom_change_fs()` is used to set different file systems for specified partition. It should be
called when custom partitioning spoke is displayed. You have to pass it filesystem name and name of partition
and needles `anaconda_part_select_$part` and `anaconda_part_fs_$fs` should exist. Example usage:
`$self->custom_change_fs("ext3", "root");` uses `anaconda_part_select_root` and `anaconda_part_fs_ext3` needles
to set ext3 file system for root partition.
- `custom_delete_part()` is used for deletion of previously added partitions in custom partitioning spoke. It should
be called when custom partitioning spoke is displayed. You have to pass it partition name and needle
`anaconda_part_select_$part` should exist. Example usage: `$self->custom_delete_part('swap');` uses
`anaconda_part_select_swap` to delete previously added swap partition.
- `installedtest` should be used in tests that are running on installed system (either in postinstall phase
or in upgrade tests). It uploads `/var/log` in `post_fail_hook()`. It provides these functions:
- `root_console()` tries to login is as a root. It switches to TTY that is set as an argument (default is TTY1)
and then calls `console_login()` for root. If you set `check` argument, it dies if it fails to log in.
Example usage: running `$self->root_console(tty=>2, check=>0);` results in TTY2 displayed with root logged
in.
convert upgrade tests to dnf-plugin-system-upgrade Summary: This is a first cut which more or less works for now. Issues: 1) We're not really testing the BUILD, here. All the test does is try and upgrade to the specified VERSION - so it'll be using the latest 'stable' for the given VERSION at the time the test runs. This isn't really that terrible, but especially for TC/RC validation, we might want to make things a bit more elaborate and set up the repo for the actual BUILD (and disable the main repos). 2) We'd actually need --nogpgcheck for non-Rawhide, at one specific point in the release cycle - after Branching but before Bodhi activation (which is when we can be sure all packages are signed). This won't matter until 24 branches, and maybe releng will have it fixed by then...if not, I'll tweak it. 3) We don't really test that the upgrade actually *happened* for desktop, at the moment - the only thing in the old test that really checked that was where we checked for the fedup boot menu entry, but that has no analog in dnf. What we should probably do is check that GUI login works, then switch to a console and check /etc/fedora-release just as the minimal test does. Test Plan: Run the tests. Note that creating the desktop disk image doesn't work ATM, so I can't verify the desktop test works, but the minimal one seems to (with D565). There'll be a matching diff for openqa_fedora_tools to update the test case names there. Reviewers: jskladan, garretraziel Reviewed By: jskladan, garretraziel Subscribers: tflink Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D567
2015-09-10 21:49:13 +00:00
- `check_release()` checks whether the installed release matches a given value. E.g. `check_release(23)`
checks whether the installed system is Fedora 23. The value can be 'Rawhide' or a Fedora release number;
often you will want to use `get_var('VERSION')`. Expects a console prompt to be active when it is called.
### New test development workflow
1. Select test from [phabricator page](https://phab.qadevel.cloud.fedoraproject.org/w/openqa/tests/).
2. Put each part of your test as a separate file into `tests/` directory, reimplementing `run()` method
and `test_flags()` method, inheriting from one of the classes mentioned above.
3. Set correct variables (so that all test parts you have made are executed) in [WebUI -> Test suites](https://localhost:8080/admin/test_suites).
4. Link your newly created Test suite to medium type in [WebUI -> Job groups](https://localhost:8080/admin/groups).
5. Run test (see [openqa_fedora_tools repository](https://bitbucket.org/rajcze/openqa_fedora_tools)).
6. Create needles (images) by using interactive mode and needles editor in WebUI.
7. Add new Job template and Test suite into `templates` file.
8. Add new Test suite and Test case into [`conf_test_suites.py`](https://bitbucket.org/rajcze/openqa_fedora_tools/src/develop/tools/openqa_trigger/conf_test_suites.py)
file in openqa_fedora_tools repository.
9. Open differential request via phabricator, set openqa_fedora as a project and repository.
10. Mark your test in [phabricator page](https://phab.qadevel.cloud.fedoraproject.org/w/openqa/tests/) as done.
add a french (encrypted) test Summary: this handles Non-English European Language Install. Basically it's a bunch of new screenshots for existing tag names, plus a bit of configurability in _boot_to_anaconda and tweaking some existing needles to do non-text matches. The weird 'half-the- icon' needles are for cases where there may or may not be a warning triangle but we want to click it either way (saves duplicating the needle). This also sets up a convention for tagging what languages a needle is appropriate for. If it's specifically appropriate for one or more languages, a tag ENV-LANGUAGE-(LANGUAGE) should be applied for each language, where (LANGUAGE) is the install language in upper-case ('LANGUAGE' variable, which should also be the string that will be typed into the language selection screen). If the needle ought to be used for *all* languages - i.e. it's not a text match, or any text in the match is known not to be translated - the tag ENV-INSTLANG-ALL should be applied. To back this, main.pm now unregisters all needles that are not tagged with either ENV-LANGUAGE-ALL or the tag for the language actually being used (if the LANGUAGE var is not set, we assume english). The point of this is to check the install is actually translated; if we allow all needles to match, the test would pass even if no translations appeared at all. Test Plan: Run all tests and make sure you get the expected results. You can schedule a run against 23 Beta TC1 to see the French test fails 'correctly' when translations are missing. Reviewers: jskladan, garretraziel Reviewed By: garretraziel Subscribers: tflink Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D577
2015-09-15 01:08:58 +00:00
### Language handling
Tests can run in different languages. To set the language which will be used for a test, set the `LANGUAGE`
variable for the test suite. The results of this will be:
1. The value set will be typed into the language search box in anaconda.
2. Any needle with at least one tag that starts with `LANGUAGE` will be unregistered unless it has the tag `LANGUAGE-(LANGUAGE)` (where `(LANGUAGE)` is the value set, forced to upper-case).
add a french (encrypted) test Summary: this handles Non-English European Language Install. Basically it's a bunch of new screenshots for existing tag names, plus a bit of configurability in _boot_to_anaconda and tweaking some existing needles to do non-text matches. The weird 'half-the- icon' needles are for cases where there may or may not be a warning triangle but we want to click it either way (saves duplicating the needle). This also sets up a convention for tagging what languages a needle is appropriate for. If it's specifically appropriate for one or more languages, a tag ENV-LANGUAGE-(LANGUAGE) should be applied for each language, where (LANGUAGE) is the install language in upper-case ('LANGUAGE' variable, which should also be the string that will be typed into the language selection screen). If the needle ought to be used for *all* languages - i.e. it's not a text match, or any text in the match is known not to be translated - the tag ENV-INSTLANG-ALL should be applied. To back this, main.pm now unregisters all needles that are not tagged with either ENV-LANGUAGE-ALL or the tag for the language actually being used (if the LANGUAGE var is not set, we assume english). The point of this is to check the install is actually translated; if we allow all needles to match, the test would pass even if no translations appeared at all. Test Plan: Run all tests and make sure you get the expected results. You can schedule a run against 23 Beta TC1 to see the French test fails 'correctly' when translations are missing. Reviewers: jskladan, garretraziel Reviewed By: garretraziel Subscribers: tflink Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D577
2015-09-15 01:08:58 +00:00
3. As a consequence, the chosen language will be selected at the anaconda Welcome screen.
It is very important, therefore, that needles have the correct tags. Any needle which is expected to match for
tests run in *any* language must have no `LANGUAGE` tags. Other needles must have the appropriate tag(s)
for the languages they are expected to match. The safest option if you are unsure is to set no `LANGUAGE` tag(s).
add a french (encrypted) test Summary: this handles Non-English European Language Install. Basically it's a bunch of new screenshots for existing tag names, plus a bit of configurability in _boot_to_anaconda and tweaking some existing needles to do non-text matches. The weird 'half-the- icon' needles are for cases where there may or may not be a warning triangle but we want to click it either way (saves duplicating the needle). This also sets up a convention for tagging what languages a needle is appropriate for. If it's specifically appropriate for one or more languages, a tag ENV-LANGUAGE-(LANGUAGE) should be applied for each language, where (LANGUAGE) is the install language in upper-case ('LANGUAGE' variable, which should also be the string that will be typed into the language selection screen). If the needle ought to be used for *all* languages - i.e. it's not a text match, or any text in the match is known not to be translated - the tag ENV-INSTLANG-ALL should be applied. To back this, main.pm now unregisters all needles that are not tagged with either ENV-LANGUAGE-ALL or the tag for the language actually being used (if the LANGUAGE var is not set, we assume english). The point of this is to check the install is actually translated; if we allow all needles to match, the test would pass even if no translations appeared at all. Test Plan: Run all tests and make sure you get the expected results. You can schedule a run against 23 Beta TC1 to see the French test fails 'correctly' when translations are missing. Reviewers: jskladan, garretraziel Reviewed By: garretraziel Subscribers: tflink Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D577
2015-09-15 01:08:58 +00:00
The only danger of this is that missing translations may not be caught.
Note that tags of the form `ENV-INSTLANG-(anything)` are useless artefacts and should be removed. Due to
unfortunate design in openQA, any needle created in the web UI needle editor will have a `ENV-INSTLANG-en_US`
tag by default; this should be removed before submission.