33ac181955
The reason we have all this horrible code to use the commented- out baseurl lines in the repo files instead of the metalinks that are usually used is a timing issue with the metalink system. As a protection against stale mirrors, the metalink system sends the package manager a list of mirrors *and a list of recent checksums for the repo metadata*. The package manager goes out and gets the metadata from the first mirror on the list, then checksums it; if the checksum isn't on the list of checksums it got from mirrormanager, it assumes that means the mirror is stale, and tries the next on the list instead. The problem is that MM's list of checksums is currently only updated once an hour (by a cron job). So we kept running into a problem where, when a test ran just after one of the repos had been regenerated, the infra mirror it's supposed to use would be rejected because the checksum wasn't on the list - but not because the mirror was stale, but because it was too fresh, it had got the new packages and metadata but mirrormanager's list of checksums hadn't been updated to include the checksum for the latest metadata. All this baseurl munging code was getting ridiculous, though, what with the tests getting more complicated and errors showing up in the actual repo files and stuff. It occurred to me that instead of using the baseurl we can just use the 'mirrorlist' system instead of 'metalink'. mirrorlist is the dumber, older system which just provides the package manager a list of mirrors and nothing else - the whole stale-mirror-detection-checksum thing does not happen with mirrorlists, the package manager just tries all the mirrors in order and uses the first that works. And happily, it's very easy to convert the metalink URLs into mirrorlist URLs, and it saves all that faffing around trying to fix up baseurls. Also, adjust upgrade_boot to do the s/metalink/mirrorlist/ substitution, so upgrade tests don't run into the timing issue in the steps before the main repo_setup run is done by upgrade_run, and adjust repo_setup_compose to sub this line out later. Signed-off-by: Adam Williamson <awilliam@redhat.com> |
||
---|---|---|
lib | ||
needles | ||
tests | ||
.arcconfig | ||
.gitignore | ||
COPYING | ||
main.pm | ||
README.md | ||
templates | ||
templates-updates | ||
VARIABLES.md |
openQA tests for the Fedora distribution
This repository contains tests and images for testing Fedora with openQA. The fedora_openqa library and CLI are used for scheduling tests, and createhdds is used for creating base disk images for the test. For openQA installation instructions, see the Fedora openQA wiki page.
Issues
Issues and pull requests are tracked in os-autoinst-distri-fedora Pagure. Pagure uses a Github-like pull request workflow, so if you're familiar with that, you can easily submit Pagure pull requests. If not, you can read up in the Pagure documentation.
Note that this repository does not use the 'gitflow' system, so the main development branch is master
: please branch from master
and submit diffs against it. This is not a Python repository and has no tests or linting.
Test development
See official documentation on:
- basic concept
- test development (including API specification)
- needles specification
- supported variables for backend.
See this example repo on how tests should be structured.
main.pm modular architecture
Since openQA uses only one entrypoint for all tests (main.pm), we have decided to utilize this feature and make tests modular. It means that basic passing through main.pm (without any variables set) results in most basic installation test executed. Developer can customize it with additional variables (for example by setting PACKAGE_SET=minimal
to do installation only with minimal package set).
Make your test modular, so that it utilizes _boot_to_anaconda.pm
, _software_selection.pm
and _do_install_and_reboot.pm
tests (that are loaded automatically). Break your test into smaller parts, each dealing with one specific feature (e. g. partitioning, user creation...) and add their loading into main.pm based on reasonable variable setting (so they can be used in other tests also).
Fedora installation (and consequently main.pm) consists of several parts:
Booting into Anaconda or booting live image and starting Anaconda
Since there isn't much variation between tests in this step, we have developed universal _boot_to_anaconda.pm
test that is loaded automatically each time except when ENTRYPOINT
or UPGRADE
is set (see VARIABLES.md).
To customize this step, you can set following variables:
GRUB
is appended to kernel line before boot. You can set for exampleinst.updates
here.- If
KICKSTART
is set, this part of installation ends here (program doesn't wait for Anaconda to appear). Note that you should setinst.ks
yourself by settingGRUB
variable. - If
LIVE
is set, program waits for desktop to appear and then clicks on "Install to Hard Drive" button.
Customizing installation by interacting with Anaconda spokes
Most of the differences between tests take place in this part. If you want to add another installation test, you will probably put your variable checking and test loading here. All tests in this part should start on Anaconda's main hub and after they done its part, they should go back to Anaconda's main hub so that next test could be executed. In this phase, universal _software_selection.pm
test is loaded that handles selecting what software to install.
To customize this step, you can set following variables:
- Set
PACKAGE_SET
to install required package set on "Software selection spoke" - you have to provide correct needles with the name ofanaconda_${PACKAGE_SET}_highlighted
andanaconda_${PACKAGE_SET}_selected
. - Set
ENCRYPT_PASSWORD
to encrypt disk, value of this variable is used as an actual password.
Installing Fedora and waiting for Fedora to reboot
After all customizations are finished, _do_install_and_reboot.pm
test is automatically loaded. It starts installation, creates user and sets root password when required, waits for installation to finish and reboots into installed system. Only variables that control flow in this part are these:
ROOT_PASSWORD
to set root password to this value.- When set,
USER_LOGIN
andUSER_PASSWORD
are used to create user in Anaconda.
Post-install phase
After installation is finished and installed system is fully booted, you can run additional tests as checks that installed system has correct attributes - that correct file system is used, that RAID is used etc.
Test inheritance
Your test can inherit from basetest
, installedtest
or anacondatest
. Each provides relevant methods that are documented in-line, so read the files (lib/anacondatest.pm
, lib/installedtest.pm
) for information on these.
basetest
: A base class provided by os-autoinst - it has emptypost_fail_hook()
and doesn't set any flags.anacondatest
: should be used in tests where Anaconda is running. It uploads Anaconda logs (for exampleanaconda.log
orpackaging.log
) inpost_fail_hook()
.installedtest
: should be used in tests that are running on installed system (either in postinstall phase or in upgrade tests).
There are also several modules that export utility functions, currently utils
, anaconda
, freeipa
, packagetest
and tapnet
. Your test can use
any of these modules and then directly call the functions they export. Again, the functions are documented in-line.
New test development workflow
- Put each part of your test as a separate file into
tests/
directory, reimplementingrun()
method andtest_flags()
method, inheriting from one of the classes mentioned above. - Set correct variables (so that all test parts you have made are executed) in WebUI -> Test suites.
- Link your newly created Test suite to medium type in WebUI -> Job groups.
- Run test (see openqa_fedora_tools repository).
- Create needles (images) by using interactive mode and needles editor in WebUI.
- Add new Job template and Test suite into
templates
file (andtemplates-updates
, if the test is applicable to the update testing workflow) - Add new Test suite and Test case into
conf_test_suites.py
file in fedora_openqa repository. - Open pull request for the os-autoinst-distri-fedora changes in Pagure. Pagure uses a Github-style workflow (summary: fork the project via the web interface, push your changes to a branch on your fork, then use the web interface to submit a pull request). See the Pagure documentation for more details.
- Open a pull request in fedora_openqa Pagure for any necessary fedora_openqa changes.
Language handling
Tests can run in different languages. To set the language which will be used for a test, set the LANGUAGE
variable for the test suite. The results of this will be:
- The value set will be typed into the language search box in anaconda.
- Any needle with at least one tag that starts with
LANGUAGE
will be unregistered unless it has the tagLANGUAGE-(LANGUAGE)
(where(LANGUAGE)
is the value set, forced to upper-case). - As a consequence, the chosen language will be selected at the anaconda Welcome screen.
It is very important, therefore, that needles have the correct tags. Any needle which is expected to match for tests run in any language must have no LANGUAGE
tags. Other needles must have the appropriate tag(s) for the languages they are expected to match. The safest option if you are unsure is to set no LANGUAGE
tag(s). The only danger of this is that missing translations may not be caught.
Note that tags of the form ENV-INSTLANG-(anything)
are useless artefacts and should be removed.