From cad839375185039f8e712344c252813c481d2ccf Mon Sep 17 00:00:00 2001 From: Alan Marshall Date: Wed, 3 May 2023 18:34:46 +0000 Subject: [PATCH 1/7] Upload files to 'docs/guidelines' Adds document openqa_manual_install.md giving a well tested procedure for manual install of openqa for Rockylinux together with a general introduction to the process to be followed. This will be useful for those who initially wish to go through the install steps manually rather than use the various install scripts that are available. This is a first time pr here so not sure if it will work. --- docs/guidelines/openqa_manual_install.md | 265 +++++++++++++++++++++++ 1 file changed, 265 insertions(+) create mode 100644 docs/guidelines/openqa_manual_install.md diff --git a/docs/guidelines/openqa_manual_install.md b/docs/guidelines/openqa_manual_install.md new file mode 100644 index 0000000..5eac670 --- /dev/null +++ b/docs/guidelines/openqa_manual_install.md @@ -0,0 +1,265 @@ +--- +title: Manual Install of OpenQA for rockylinux +author: Alan Marshall +revision_date: 2023-05-03 +rc: + prod: Rocky Linux + level: Draft +--- + + +#### Intended Audience +Those who wish to use the OpenQA automated testing system configured for Rocky Linux tests. To run your own automated tests in OpenQA, you need a PC or server with hardware-virtualisation and an up-to-date Fedora Linux. + +### Introduction +This guide explains the use of the OpenQA automated testing system to test various aspects of Rocky Linux releases either at the pre-release stage or thereafter. + +OpenQA is an automated test tool that makes it possible to test the whole installation process. It uses virtual machines which it sets up to reproduce the process, check the output (both serial console and GUI screen) in every step and send the necessary keystrokes and commands to proceed to the next step. OpenQA checks whether the system can be installed, whether it works properly, whether applications work and whether the system responds as expected to different installation options and commands. + +OpenQA can run numerous combinations of tests for every revision of the operating system, reporting the errors detected for each combination of hardware configuration, installation options and variant of the operating system. + +Upstream documentation is useful for reference but since it is a mixture of advice and instructions relating to openSUSE and Fedora which have substantial differences between them it is not always clear which are significant for Rocky. However, as an rpm based distribution, Rocky Linux use is closely related to the Fedora version. + +### WebUI +The web UI is a very useful feature of the OpenQA system since it provides an easily accessed view of the progress and details of OpenQA tests either on the local machine or remotely or both. It is intended to be intuitive and self-explanatory. Look out for the little blue help icons and click them for detailed help on specific sections. + +Some pages use queries to select what should be shown. The query parameters are generated on clickable links, for example starting from the index page or the group overview page clicking on single builds. On the query pages there can be UI elements to control the parameters, for example to look for older builds or show only failed jobs, or other settings. Additionally, the query parameters can be tweaked by hand if you want to provide a link to specific views. + +## Step-by-step Install Guide + +OpenQA can be installed only on a Fedora (or OpenSUSE) server or workstation. + +``` +# install Packages +# for openqa +sudo dnf install openqa openqa-httpd openqa-worker fedora-messaging python3-jsonschema +sudo dnf install perl-REST-Client.noarch + +# and for createhdds +sudo dnf install libguestfs-tools libguestfs-xfs python3-fedfind python3-libguestfs +sudo dnf install libvirt-daemon-config-network libvirt-python3 virt-install withlock + +# configure httpd: +cd /etc/httpd/conf.d/ +sudo cp openqa.conf.template openqa.conf +sudo cp openqa-ssl.conf.template openqa-ssl.conf +sudo setsebool -P httpd_can_network_connect 1 +sudo systemctl restart httpd + +# configure the web UI +sudo vi /etc/openqa/openqa.ini + +[global] +branding=plain +download_domains = rockylinux.org +[auth] +method = Fake + +sudo dnf install postgresql-server +sudo postgresql-setup --initdb + +# enable and start services +sudo systemctl enable postgresql --now +sudo systemctl enable httpd --now +sudo systemctl enable openqa-gru --now +sudo systemctl enable openqa-scheduler --now +sudo systemctl enable openqa-websockets --now +sudo systemctl enable openqa-webui --now +sudo systemctl enable fm-consumer@fedora_openqa_scheduler --now +sudo systemctl enable libvirtd --now +sudo setsebool -P httpd_can_network_connect 1 +sudo firewall-cmd --add-service=http --permanent +sudo firewall-cmd --reload +sudo systemctl restart httpd + +# to create API key in local web interface at http://localhost +# or on remote system http:// +# Click Login, then Manage API Keys, create a key and secret. + +# insert key and secret +sudo vi /etc/openqa/client.conf + +[localhost] +key = ... +secret = ... + +# create workers +sudo systemctl enable openqa-worker@1 --now +# then ...@2 ...etc as desired. Look in webui workers to check shown idle. +# as a rule of thumb, you can have about half the number of workers as cores + +# get Rocky tests +cd /var/lib/openqa/tests/ +sudo git clone https://github.com/rocky-linux/os-autoinst-distri-rocky.git rocky +sudo chown -R geekotest:geekotest rocky +cd rocky + +# when working in /var/lib/openqa nearly all commands need sudo so it is +# easier to su to root. If desired sudo per command can be used instead. +sudo su +git config --global --add safe.directory /var/lib/openqa/share/tests/rocky + +git checkout develop +# or whichever branch has the latest updates for your tests + +./fifloader.py -l -c templates.fif.json templates-updates.fif.json +git clone https://github.com/rocky-linux/createhdds.git ~/createhdds +mkdir -p /var/lib/openqa/share/factory/hdd/fixed + +# will need about 200GB disk space available for ongoing tests +cd /var/lib/openqa/factory/hdd/fixed + +# start a long running process that provides hdd image files for ongoing tests +~/createhdds/createhdds.py all + +# get Rocky iso files for testing +mkdir -p /var/lib/openqa/share/factory/iso/fixed +cd /var/lib/openqa/factory/iso/fixed + +curl -LOR https://dl.rockylinux.org/pub/rocky/9/isos/x86_64/Rocky-9.1-x86_64-boot.iso +curl -LOR https://dl.rockylinux.org/pub/rocky/9/isos/x86_64/Rocky-9.1-x86_64-minimal.iso +curl -LOR https://dl.rockylinux.org/pub/rocky/9/isos/x86_64/Rocky-9.1-x86_64-dvd.iso + +# post tests and view progress on webui +cd /var/lib/openqa/tests/rocky/ +openqa-cli api -X POST isos ISO=Rocky-9.1-x86_64-minimal.iso ARCH=x86_64 DISTRI=rocky FLAVOR=minimal-iso VERSION=9.1 BUILD="$(date +%Y%m%d.%H%M%S).0"-minimal +openqa-cli api -X POST isos ISO=Rocky-9.1-x86_64-boot.iso ARCH=x86_64 DISTRI=rocky FLAVOR=boot-iso VERSION=9.1 BUILD="$(date +%Y%m%d.%H%M%S).0"-boot + +``` +At this point your system is configured for single vm deployment and it can be used as such. Pause here if you wish to get some practice using openqa before progressing further. +Facilities for multi-vm testing which is substantially more complicated is beyond the scope of this document. + + +### Helper Scripts + +see: +https://github.com/rocky-linux/os-autoinst-distri-rocky/tree/develop/scripts + +cancel-build.sh is especially useful when you discover that you have initiated a large build and got it wrong... d'oh. + +### Using Templates + +#### Problem +A problem arises when testing an operating system, especially when doing continuous testing, that there is always a certain combination of jobs, each one with its own settings, that needs to be run for every revision. Those combinations can be different for different 'flavors' of the same revision, like running a different set of jobs for each architecture. This combinational problem can go one step further if OpenQA is being used for different kinds of tests, like running some simple pre-integration tests for some snapshots combined with more comprehensive post-integration tests for release candidates. + +This section describes how an instance of OpenQA *could* be configured using the options in the admin area of the webUI to automatically create all the required jobs for each revision of your operating system that needs to be tested. *If* you were starting from scratch (the difficult way), you would probably go through the following order: + +1. Define machines in 'Machines' menu +1. Define medium types (products) you have in 'Medium types' menu +1. Specify various collections of tests you want to run in the 'Test suites' menu +1. Define job groups in 'Job groups' menu for groups of tests +1. Select individual 'Job groups' and decide what combinations make sense and need to be tested + +If you followed the install guide above then the cloned Rocky tests from github.com/rocky-linux/os-autoinst-distri-rocky.git will have pre-configured the admin area of the webUI. You may find it useful when reading the following sections. + +Machines, mediums, test suites and job templates can all set various configuration variables. The job templates within the job groups define how the test suites, mediums and machines should be combined in various ways to produce individual 'jobs'. All the variables from the test suite, medium, machine and job template are combined and made available to the actual test code run by the 'job', along with variables specified as part of the job creation request. Certain variables also influence OpenQA’s and/or os-autoinst’s own behavior in terms of how it configures the environment for the job. + +#### Machines +You need to have at least one machine set up to be able to run any tests. Those machines represent virtual machine types that you want to test. To make tests actually happen, you have to have an 'OpenQA worker' connected that can fulfill those specifications. + +- Name. User defined string - only needed for operator to identify the machine configuration. +- Backend. What backend should be used for this machine. Recommended value is qemu as it is the most tested one, but other options (such as kvm2usb or vbox) are also possible. +- Variables Most machine variables influence os-autoinst’s behavior in terms of how the test machine is set up. A few important examples: + - QEMUCPU can be 'qemu32' or 'qemu64' and specifies the architecture of the virtual CPU + - QEMUCPUS is an integer that specifies the number of cores you wish for + + - USBBOOT when set to 1, the image will be loaded through an emulated USB stick. + +#### Medium Types (products) +A medium type (product) in OpenQA is a simple description without any definite meaning. It basically consists of a name and a set of variables that define or characterize this product in os-autoinst. + +Some example variables are: + +- ISO_MAXSIZE contains the maximum size of the product. There is a test that checks that the current size of the product is less or equal than this variable. +- DVD if it is set to 1, this indicates that the medium is a DVD. +- LIVECD if it is set to 1, this indicates that the medium is a live image (can be a CD or USB) +- GNOME this variable, if it is set to 1, indicates that it is a GNOME only distribution. +- RESCUECD is set to 1 for rescue CD images. + +#### Test Suites +A test suite consists of a name and a set of test variables that are used inside this particular test together with an optional description. The test variables can be used to parameterize the actual test code and influence the behaviour according to the settings. + +Some sample variables are: + +- DESKTOP possible values are 'kde' 'gnome' 'lxde' 'xfce' or 'textmode'. Used to indicate the desktop selected by the user during the test. +- ENCRYPT encrypt the home directory via YaST. +- HDDSIZEGB hard disk size in GB. +- HDD_1 path for the pre-created hard disk +- RAIDLEVEL RAID configuration variable + +#### Job Groups +The job groups are the place where the actual test scenarios are defined by the selection of the medium type, the test suite and machine together with a priority value. + +The priority value is used in the scheduler to choose the next job. If multiple jobs are scheduled and their requirements for running them are fulfilled the ones with a lower priority value are triggered. The id is the second sorting key: of two jobs with equal requirements and same priority value the one with lower id is triggered first. + +Job groups themselves can be created over the web UI as well as the REST API. Job groups can optionally be nested into categories. The display order of job groups and categories can be configured by drag-and-drop in the web UI. + +The scenario definitions within the job groups can be created and configured by different means: + +- A simple web UI wizard which is automatically shown for job groups when a new medium is added to the job group. +- An intuitive table within the web UI for adding additional test scenarios to existing media including the possibility to configure the priority values. +- The scripts OpenQA-load-templates and OpenQA-dump-templates to quickly dump and load the configuration from custom plain-text dump format files using the REST API. +- Using declarative schedule definitions in the YAML format using REST API routes or an online-editor within the web UI including a syntax checker. + +### Needles +Needles are very precise and the slightest deviation from the specified display will be detected. This means that every time there is a new release, very small changes occur in layout of displays resulting in many new or modified needles being required. There is always a significant amount of work needed by the Test Team to produce the automatic tests for a new version. + +### Glossary +The following terms are used within the context of OpenQA:- + +test module +* An individual test case in a single perl module (.pm) file, e.g. "sshxterm". If not further specified a test module is denoted with its "short name" equivalent to the filename including the test definition. The "full name" is composed of the test group (TBC), which itself is formed by the top-folder of the test module file, and the short name, e.g. "x11-sshxterm" (for x11/sshxterm.pm) + +test suite +1. A collection of test modules, e.g. "textmode". All test modules within one test suite are run serially +job +1. One run of individual test cases in a row denoted by a unique number for one instance of OpenQA, e.g. one installation with subsequent testing of applications within gnome + +test run +* Equivalent to job +test result +* The result of one job, e.g. "passed" with the details of each individual test module +test step +* the execution of one test module within a job + +distri +* A test distribution but also sometimes referring to a product (CAUTION: ambiguous, historically a "GNU/Linux distribution"), composed of multiple test modules in a folder structure that compose test suites, e.g. "rocky" (test distribution, short for "os-autoinst-distri-rocky") + +product +* The main "system under test" (SUT), e.g. "rocky", also called "Medium Types" in the web interface of OpenQA + +job group +* Equivalent to product, used in context of the webUI + +version +* One version of a product, don’t confuse with builds + +flavor +* Keyword for a specific variant of a product to distinguish differing variants, e.g. "dvd-iso" + +arch +* An architecture variant of a product, e.g. "x86_64" + +machine +* Additional variant of machine, e.g. used for "64bit", "uefi", etc. + +scenario +* A composition of ----@, e.g. "Rocky-9-dvd-x86_64-gnome@64bit" + +build +* Different versions of a product as tested, can be considered a "sub-version" of version, e.g. "Build1234"; CAUTION: ambiguity: either with the prefix "Build" included or not + +#### History (briefly) +Openqa started with OS-autoinst: automated testing of Operating Systems +The OS-autoinst project aims at providing a means to run fully automated tests, especially to run tests of basic and low-level operating system components such as bootloader, kernel, installer and upgrade, which can not easily be tested with other automated testing frameworks. However, it can just as well be used to test firefox and openoffice operation on top of a newly installed OS. +OpenQA is a test-scheduler and web-front for openSUSE and Fedora using OS-autoinst as a backend. +OpenQA originated at openSuse and was adopted by Fedora as the automated test system for their frequent distribution updates. Maintenance activity is fairly intense and is ongoing at various levels of users. OpenQA was adopted by Rocky Linux Test Team as the preferred automated testing system for the ongoing releases of it's distribution. +OpenQA is free software released under the GPLv2 license. + +#### Attribution +This guide is heavily inspired by the numerous upstream documents in which installation and usage of OS-autoinst and OpenQA are described. + +#### References +Since Rocky Linux use of OpenQA is drawn from upstream Fedora and hence openSUSE this document contains **many** passages which are edited versions of upstream documentation and that use is hereby gratefully acknowledged. As with many open source projects, we build on previous work. + + From 9edaaaf30f86bd0be2df58b33da1927da423b28e Mon Sep 17 00:00:00 2001 From: Alan Marshall Date: Fri, 5 May 2023 11:44:12 +0000 Subject: [PATCH 2/7] Update 'docs/guidelines/openqa_manual_install.md' Add hyperlink. --- docs/guidelines/openqa_manual_install.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/docs/guidelines/openqa_manual_install.md b/docs/guidelines/openqa_manual_install.md index 5eac670..81576ab 100644 --- a/docs/guidelines/openqa_manual_install.md +++ b/docs/guidelines/openqa_manual_install.md @@ -132,8 +132,7 @@ Facilities for multi-vm testing which is substantially more complicated is beyon ### Helper Scripts -see: -https://github.com/rocky-linux/os-autoinst-distri-rocky/tree/develop/scripts +[see here](https://github.com/rocky-linux/os-autoinst-distri-rocky/tree/develop/scripts) cancel-build.sh is especially useful when you discover that you have initiated a large build and got it wrong... d'oh. From cbbd6cbb90e3a95d4a7b260382bcd6f4792111dd Mon Sep 17 00:00:00 2001 From: AlanMarshall Date: Fri, 26 Apr 2024 10:17:43 +0100 Subject: [PATCH 3/7] Update Step-by-step section for Fedora 40 --- docs/guidelines/openqa_manual_install.md | 60 ++++++++++++++---------- 1 file changed, 34 insertions(+), 26 deletions(-) diff --git a/docs/guidelines/openqa_manual_install.md b/docs/guidelines/openqa_manual_install.md index 81576ab..29b756b 100644 --- a/docs/guidelines/openqa_manual_install.md +++ b/docs/guidelines/openqa_manual_install.md @@ -1,7 +1,7 @@ --- title: Manual Install of OpenQA for rockylinux author: Alan Marshall -revision_date: 2023-05-03 +revision_date: 2024-04-25 rc: prod: Rocky Linux level: Draft @@ -28,16 +28,17 @@ Some pages use queries to select what should be shown. The query parameters are ## Step-by-step Install Guide OpenQA can be installed only on a Fedora (or OpenSUSE) server or workstation. +Tested on Fedora 40 Server ``` # install Packages # for openqa -sudo dnf install openqa openqa-httpd openqa-worker fedora-messaging python3-jsonschema -sudo dnf install perl-REST-Client.noarch +sudo dnf install -y openqa openqa-httpd openqa-worker fedora-messaging python3-jsonschema +sudo dnf install -y perl-REST-Client.noarch # and for createhdds -sudo dnf install libguestfs-tools libguestfs-xfs python3-fedfind python3-libguestfs -sudo dnf install libvirt-daemon-config-network libvirt-python3 virt-install withlock +sudo dnf install -y libguestfs-tools libguestfs-xfs python3-fedfind python3-libguestfs +sudo dnf install -y libvirt libvirt-daemon-config-network libvirt-python3 virt-install withlock # configure httpd: cd /etc/httpd/conf.d/ @@ -47,8 +48,7 @@ sudo setsebool -P httpd_can_network_connect 1 sudo systemctl restart httpd # configure the web UI -sudo vi /etc/openqa/openqa.ini - +sudoedit /etc/openqa/openqa.ini [global] branding=plain download_domains = rockylinux.org @@ -77,7 +77,7 @@ sudo systemctl restart httpd # Click Login, then Manage API Keys, create a key and secret. # insert key and secret -sudo vi /etc/openqa/client.conf +sudoedit /etc/openqa/client.conf [localhost] key = ... @@ -94,40 +94,48 @@ sudo git clone https://github.com/rocky-linux/os-autoinst-distri-rocky.git rocky sudo chown -R geekotest:geekotest rocky cd rocky -# when working in /var/lib/openqa nearly all commands need sudo so it is -# easier to su to root. If desired sudo per command can be used instead. -sudo su -git config --global --add safe.directory /var/lib/openqa/share/tests/rocky +# when working in /var/lib/openqa nearly all commands need sudo. -git checkout develop +sudo git config --global --add safe.directory /var/lib/openqa/share/tests/rocky + +sudo git checkout develop # or whichever branch has the latest updates for your tests -./fifloader.py -l -c templates.fif.json templates-updates.fif.json -git clone https://github.com/rocky-linux/createhdds.git ~/createhdds -mkdir -p /var/lib/openqa/share/factory/hdd/fixed +sudo ./fifloader.py -l -c templates.fif.json +sudo git clone https://github.com/rocky-linux/createhdds.git ~/createhdds +sudo mkdir -p /var/lib/openqa/share/factory/hdd/fixed # will need about 200GB disk space available for ongoing tests cd /var/lib/openqa/factory/hdd/fixed # start a long running process that provides hdd image files for ongoing tests -~/createhdds/createhdds.py all +~/createhdds/createhdds.py -t -b stg all -# get Rocky iso files for testing -mkdir -p /var/lib/openqa/share/factory/iso/fixed +# get Rocky iso files for testing from staging repository +sudo mkdir -p /var/lib/openqa/share/factory/iso/fixed cd /var/lib/openqa/factory/iso/fixed -curl -LOR https://dl.rockylinux.org/pub/rocky/9/isos/x86_64/Rocky-9.1-x86_64-boot.iso -curl -LOR https://dl.rockylinux.org/pub/rocky/9/isos/x86_64/Rocky-9.1-x86_64-minimal.iso -curl -LOR https://dl.rockylinux.org/pub/rocky/9/isos/x86_64/Rocky-9.1-x86_64-dvd.iso +sudo curl -LOR https://dl.rockylinux.org/stg/rocky/9/isos/x86_64/Rocky-9.3-x86_64-boot.iso +sudo curl -LOR https://dl.rockylinux.org/stg/rocky/9/isos/x86_64/Rocky-9.3-x86_64-minimal.iso +sudo curl -LOR https://dl.rockylinux.org/stg/rocky/9/isos/x86_64/Rocky-9.3-x86_64-dvd.iso +sudo curl -LOR https://dl.rockylinux.org/stg/rocky/9/isos/x86_64/CHECKSUM + +sha256sum -c CHECKSUM + +# fix ownership, add to group, reboot +cd /var/lib/openqa/factory/ +sudo chown -R geekotest:geekotest ./ +sudo usermod -aG geekotest +sudo init 6 # post tests and view progress on webui cd /var/lib/openqa/tests/rocky/ -openqa-cli api -X POST isos ISO=Rocky-9.1-x86_64-minimal.iso ARCH=x86_64 DISTRI=rocky FLAVOR=minimal-iso VERSION=9.1 BUILD="$(date +%Y%m%d.%H%M%S).0"-minimal -openqa-cli api -X POST isos ISO=Rocky-9.1-x86_64-boot.iso ARCH=x86_64 DISTRI=rocky FLAVOR=boot-iso VERSION=9.1 BUILD="$(date +%Y%m%d.%H%M%S).0"-boot +sudo openqa-cli api -X POST isos ISO=Rocky-9.3-x86_64-minimal.iso ARCH=x86_64 DISTRI=rocky FLAVOR=minimal-iso VERSION=9.3 BUILD="$(date +%Y%m%d.%H%M%S).0"-minimal +sudo openqa-cli api -X POST isos ISO=Rocky-9.3-x86_64-boot.iso ARCH=x86_64 DISTRI=rocky FLAVOR=boot-iso VERSION=9.3 BUILD="$(date +%Y%m%d.%H%M%S).0"-boot ``` At this point your system is configured for single vm deployment and it can be used as such. Pause here if you wish to get some practice using openqa before progressing further. -Facilities for multi-vm testing which is substantially more complicated is beyond the scope of this document. +Installation of facilities for multi-vm testing which is substantially more complicated is beyond the (present) scope of this document. ### Helper Scripts @@ -139,7 +147,7 @@ cancel-build.sh is especially useful when you discover that you have initiated a ### Using Templates #### Problem -A problem arises when testing an operating system, especially when doing continuous testing, that there is always a certain combination of jobs, each one with its own settings, that needs to be run for every revision. Those combinations can be different for different 'flavors' of the same revision, like running a different set of jobs for each architecture. This combinational problem can go one step further if OpenQA is being used for different kinds of tests, like running some simple pre-integration tests for some snapshots combined with more comprehensive post-integration tests for release candidates. +One of the problems that arises when testing an operating system, especially when doing continuous testing, is that there is always a certain combination of jobs, each one with its own settings, that needs to be run for every revision. These combinations can be different for different 'flavors' of the same revision, like running a different set of jobs for each architecture. This combinational problem can go one step further if OpenQA is being used for different kinds of tests, like running some simple pre-integration tests for some snapshots combined with more comprehensive post-integration tests for release candidates. This section describes how an instance of OpenQA *could* be configured using the options in the admin area of the webUI to automatically create all the required jobs for each revision of your operating system that needs to be tested. *If* you were starting from scratch (the difficult way), you would probably go through the following order: From e263f02eaa5c4450712cba99499bd4441cef74fa Mon Sep 17 00:00:00 2001 From: AlanMarshall Date: Sat, 27 Apr 2024 13:57:38 +0100 Subject: [PATCH 4/7] Fix navigation --- docs/guidelines/.pages | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/guidelines/.pages b/docs/guidelines/.pages index 940ce1a..a4d4add 100644 --- a/docs/guidelines/.pages +++ b/docs/guidelines/.pages @@ -2,4 +2,5 @@ nav: - ... | index.md - Release Criteria & Status: release_criteria + - OpenQA Manual Install: openqa_manual_install.md ... From d4295d8d665670067f04632feb72265c8ed17f4c Mon Sep 17 00:00:00 2001 From: AlanMarshall Date: Sat, 27 Apr 2024 16:28:30 +0100 Subject: [PATCH 5/7] Numerous edits --- docs/guidelines/openqa_manual_install.md | 30 +++++++++++++----------- 1 file changed, 16 insertions(+), 14 deletions(-) diff --git a/docs/guidelines/openqa_manual_install.md b/docs/guidelines/openqa_manual_install.md index 29b756b..999c04e 100644 --- a/docs/guidelines/openqa_manual_install.md +++ b/docs/guidelines/openqa_manual_install.md @@ -1,7 +1,7 @@ --- title: Manual Install of OpenQA for rockylinux author: Alan Marshall -revision_date: 2024-04-25 +revision_date: 2024-04-27 rc: prod: Rocky Linux level: Draft @@ -9,26 +9,25 @@ rc: #### Intended Audience -Those who wish to use the OpenQA automated testing system configured for Rocky Linux tests. To run your own automated tests in OpenQA, you need a PC or server with hardware-virtualisation and an up-to-date Fedora Linux. +Those who wish to use the OpenQA automated testing system configured for Rocky Linux tests. If so, you will need a PC or server with hardware-virtualisation running an up-to-date Fedora Linux. ### Introduction This guide explains the use of the OpenQA automated testing system to test various aspects of Rocky Linux releases either at the pre-release stage or thereafter. OpenQA is an automated test tool that makes it possible to test the whole installation process. It uses virtual machines which it sets up to reproduce the process, check the output (both serial console and GUI screen) in every step and send the necessary keystrokes and commands to proceed to the next step. OpenQA checks whether the system can be installed, whether it works properly, whether applications work and whether the system responds as expected to different installation options and commands. +Rocky Linux OpenQA tests can be found in the [os-autoinst-distri-rocky](https://github.com/rocky-linux/os-autoinst-distri-rocky) repository + OpenQA can run numerous combinations of tests for every revision of the operating system, reporting the errors detected for each combination of hardware configuration, installation options and variant of the operating system. -Upstream documentation is useful for reference but since it is a mixture of advice and instructions relating to openSUSE and Fedora which have substantial differences between them it is not always clear which are significant for Rocky. However, as an rpm based distribution, Rocky Linux use is closely related to the Fedora version. - ### WebUI -The web UI is a very useful feature of the OpenQA system since it provides an easily accessed view of the progress and details of OpenQA tests either on the local machine or remotely or both. It is intended to be intuitive and self-explanatory. Look out for the little blue help icons and click them for detailed help on specific sections. +The web UI is a very useful feature of the OpenQA system since it provides an easily accessed view of the progress and details of OpenQA tests either on the local machine or remotely or both. It is intended to be intuitive and self-explanatory. Some pages use queries to select what should be shown. The query parameters are generated on clickable links, for example starting from the index page or the group overview page clicking on single builds. On the query pages there can be UI elements to control the parameters, for example to look for older builds or show only failed jobs, or other settings. Additionally, the query parameters can be tweaked by hand if you want to provide a link to specific views. ## Step-by-step Install Guide -OpenQA can be installed only on a Fedora (or OpenSUSE) server or workstation. -Tested on Fedora 40 Server +OpenQA can be installed only on a Fedora (or OpenSUSE) server or workstation. The following install procedure was tested on Fedora 40 Server. ``` # install Packages @@ -134,8 +133,8 @@ sudo openqa-cli api -X POST isos ISO=Rocky-9.3-x86_64-minimal.iso ARCH=x86_64 DI sudo openqa-cli api -X POST isos ISO=Rocky-9.3-x86_64-boot.iso ARCH=x86_64 DISTRI=rocky FLAVOR=boot-iso VERSION=9.3 BUILD="$(date +%Y%m%d.%H%M%S).0"-boot ``` -At this point your system is configured for single vm deployment and it can be used as such. Pause here if you wish to get some practice using openqa before progressing further. -Installation of facilities for multi-vm testing which is substantially more complicated is beyond the (present) scope of this document. +At this point your system is configured for single vm deployment and it can be used as such. Pause here if you wish to get some practice using openqa before progressing further (recommended). +Installation of facilities for multi-vm testing, which is substantially more complicated, will be described in this document in a later revision (watch this space). ### Helper Scripts @@ -147,7 +146,7 @@ cancel-build.sh is especially useful when you discover that you have initiated a ### Using Templates #### Problem -One of the problems that arises when testing an operating system, especially when doing continuous testing, is that there is always a certain combination of jobs, each one with its own settings, that needs to be run for every revision. These combinations can be different for different 'flavors' of the same revision, like running a different set of jobs for each architecture. This combinational problem can go one step further if OpenQA is being used for different kinds of tests, like running some simple pre-integration tests for some snapshots combined with more comprehensive post-integration tests for release candidates. +One of the problems that arises when testing an operating system, especially when doing continuous testing, is that there is always a certain combination of jobs, each one with its own settings, that needs to be run for every revision. These combinations can be different for different FLAVORs of the same revision, like running a different set of jobs for each architecture. This combinational problem can go one step further if OpenQA is being used for different kinds of tests, like running some simple pre-integration tests for some snapshots combined with more comprehensive post-integration tests for release candidates. This section describes how an instance of OpenQA *could* be configured using the options in the admin area of the webUI to automatically create all the required jobs for each revision of your operating system that needs to be tested. *If* you were starting from scratch (the difficult way), you would probably go through the following order: @@ -157,12 +156,12 @@ This section describes how an instance of OpenQA *could* be configured using the 1. Define job groups in 'Job groups' menu for groups of tests 1. Select individual 'Job groups' and decide what combinations make sense and need to be tested -If you followed the install guide above then the cloned Rocky tests from github.com/rocky-linux/os-autoinst-distri-rocky.git will have pre-configured the admin area of the webUI. You may find it useful when reading the following sections. +If you followed the install guide above then the cloned Rocky tests from [os-autoinst-distri-rocky](https://github.com/rocky-linux/os-autoinst-distri-rocky) will have pre-configured the admin area of the webUI. You may find it useful when reading the following sections. Machines, mediums, test suites and job templates can all set various configuration variables. The job templates within the job groups define how the test suites, mediums and machines should be combined in various ways to produce individual 'jobs'. All the variables from the test suite, medium, machine and job template are combined and made available to the actual test code run by the 'job', along with variables specified as part of the job creation request. Certain variables also influence OpenQA’s and/or os-autoinst’s own behavior in terms of how it configures the environment for the job. #### Machines -You need to have at least one machine set up to be able to run any tests. Those machines represent virtual machine types that you want to test. To make tests actually happen, you have to have an 'OpenQA worker' connected that can fulfill those specifications. +You need to have at least one machine set up to be able to run any tests. Those machines represent virtual machine types that you want to test. Realistically to make tests actually happen, you have to have a number of 'OpenQA workers' connected that can fulfill these specifications. - Name. User defined string - only needed for operator to identify the machine configuration. - Backend. What backend should be used for this machine. Recommended value is qemu as it is the most tested one, but other options (such as kvm2usb or vbox) are also possible. @@ -173,7 +172,7 @@ You need to have at least one machine set up to be able to run any tests. Those - USBBOOT when set to 1, the image will be loaded through an emulated USB stick. #### Medium Types (products) -A medium type (product) in OpenQA is a simple description without any definite meaning. It basically consists of a name and a set of variables that define or characterize this product in os-autoinst. +A medium type (product) in OpenQA is a simple description without any definite meaning. It basically consists of a name and a set of variables that define or characterise this product in os-autoinst. Some example variables are: @@ -184,7 +183,7 @@ Some example variables are: - RESCUECD is set to 1 for rescue CD images. #### Test Suites -A test suite consists of a name and a set of test variables that are used inside this particular test together with an optional description. The test variables can be used to parameterize the actual test code and influence the behaviour according to the settings. +A test suite consists of a name and a set of test variables that are used inside this particular test together with an optional description. The test variables can be used to parameterise the actual test code and influence the behaviour according to the settings. Some sample variables are: @@ -211,6 +210,9 @@ The scenario definitions within the job groups can be created and configured by ### Needles Needles are very precise and the slightest deviation from the specified display will be detected. This means that every time there is a new release, very small changes occur in layout of displays resulting in many new or modified needles being required. There is always a significant amount of work needed by the Test Team to produce the automatic tests for a new version. +### Upstream Documentation +[Upstream documentation](https://github.com/os-autoinst/openQA/blob/master/docs/Installing.asciidoc) is useful for reference but since it is a mixture of advice and instructions relating to openSUSE and Fedora which have substantial differences between them it is not always clear which are significant for Rocky. However, as an rpm based distribution, Rocky Linux use is loosely related to the [Fedora](https://fedoraproject.org/wiki/OpenQA) version. + ### Glossary The following terms are used within the context of OpenQA:- From 70021c8c5fca88403576af482ec562e50f3357d6 Mon Sep 17 00:00:00 2001 From: AlanMarshall Date: Sun, 28 Apr 2024 13:36:17 +0100 Subject: [PATCH 6/7] Final edits for issue --- docs/guidelines/openqa_manual_install.md | 47 ++++++++++++++++++------ 1 file changed, 36 insertions(+), 11 deletions(-) diff --git a/docs/guidelines/openqa_manual_install.md b/docs/guidelines/openqa_manual_install.md index 999c04e..298ee20 100644 --- a/docs/guidelines/openqa_manual_install.md +++ b/docs/guidelines/openqa_manual_install.md @@ -1,10 +1,11 @@ --- title: Manual Install of OpenQA for rockylinux author: Alan Marshall -revision_date: 2024-04-27 +version: v1.0 +revision_date: 2024-04-28 rc: prod: Rocky Linux - level: Draft + level: Issue --- @@ -27,7 +28,7 @@ Some pages use queries to select what should be shown. The query parameters are ## Step-by-step Install Guide -OpenQA can be installed only on a Fedora (or OpenSUSE) server or workstation. The following install procedure was tested on Fedora 40 Server. +OpenQA can be installed only on a Fedora (or OpenSUSE) server or workstation. The following install procedure was tested on Fedora 40 Server. You can use either a local terminal or an ssh login from another host on the lan. ``` # install Packages @@ -129,18 +130,35 @@ sudo init 6 # post tests and view progress on webui cd /var/lib/openqa/tests/rocky/ +sudo ./fifloader.py -c -l templates.fif.json sudo openqa-cli api -X POST isos ISO=Rocky-9.3-x86_64-minimal.iso ARCH=x86_64 DISTRI=rocky FLAVOR=minimal-iso VERSION=9.3 BUILD="$(date +%Y%m%d.%H%M%S).0"-minimal sudo openqa-cli api -X POST isos ISO=Rocky-9.3-x86_64-boot.iso ARCH=x86_64 DISTRI=rocky FLAVOR=boot-iso VERSION=9.3 BUILD="$(date +%Y%m%d.%H%M%S).0"-boot - +and for a full build (this will post 95 jobs) +sudo openqa-cli api -X POST isos ISO=Rocky-9.3-x86_64-dvd.iso ARCH=x86_64 DISTRI=rocky FLAVOR=dvd-iso VERSION=9.3 BUILD="$(date +%Y%m%d.%H%M%S).0"-dvd-iso +sudo openqa-cli api -X POST isos ISO=Rocky-9.3-x86_64-dvd.iso ARCH=x86_64 DISTRI=rocky FLAVOR=universal VERSION=9.3 BUILD="$(date +%Y%m%d.%H%M%S).0"-universal ``` -At this point your system is configured for single vm deployment and it can be used as such. Pause here if you wish to get some practice using openqa before progressing further (recommended). +You can watch progress of these tests on the webui on any browser on the same lan as the test host at + +```http:///tests``` + +If you click "login" in the top right corner you will be able to control tests from the webui. + +At this point the multi-vm tests will fail or be skipped. This is because at the moment your system is configured for single vm deployment and it can be used as such. Pause your installation here if you wish to get some practice using openqa before progressing further (recommended). + Installation of facilities for multi-vm testing, which is substantially more complicated, will be described in this document in a later revision (watch this space). -### Helper Scripts +### Helpers -[see here](https://github.com/rocky-linux/os-autoinst-distri-rocky/tree/develop/scripts) +#### Createhdds +Createhdds is used to prepare .img and .qcow2 files for some of the Rocky tests. If you ran the above procedure you will have noticed that it produces a number of files in /var/lib/openqa/factory/hdd/fixed determined by the files provided [here](https://github.com/rocky-linux/createhdds). +#### openqa-cli + +Tests are normally posted using openqa-cli as you have already used above. Test parameters are listed and explained [here](https://github.com/rocky-linux/os-autoinst-distri-rocky/blob/develop/VARIABLES.md) + +#### Scripts +see [here](https://github.com/rocky-linux/os-autoinst-distri-rocky/tree/develop/scripts) - cancel-build.sh is especially useful when you discover that you have initiated a large build and got it wrong... d'oh. ### Using Templates @@ -156,10 +174,12 @@ This section describes how an instance of OpenQA *could* be configured using the 1. Define job groups in 'Job groups' menu for groups of tests 1. Select individual 'Job groups' and decide what combinations make sense and need to be tested -If you followed the install guide above then the cloned Rocky tests from [os-autoinst-distri-rocky](https://github.com/rocky-linux/os-autoinst-distri-rocky) will have pre-configured the admin area of the webUI. You may find it useful when reading the following sections. +If you followed the install guide above then the cloned Rocky tests from [os-autoinst-distri-rocky](https://github.com/rocky-linux/os-autoinst-distri-rocky) will have pre-configured the admin area of the webUI. You may find it useful to consult when reading the following sections. Machines, mediums, test suites and job templates can all set various configuration variables. The job templates within the job groups define how the test suites, mediums and machines should be combined in various ways to produce individual 'jobs'. All the variables from the test suite, medium, machine and job template are combined and made available to the actual test code run by the 'job', along with variables specified as part of the job creation request. Certain variables also influence OpenQA’s and/or os-autoinst’s own behavior in terms of how it configures the environment for the job. +The configuration is set up from /var/lib/openqa/tests/rocky/templates.fif.json + #### Machines You need to have at least one machine set up to be able to run any tests. Those machines represent virtual machine types that you want to test. Realistically to make tests actually happen, you have to have a number of 'OpenQA workers' connected that can fulfill these specifications. @@ -210,6 +230,8 @@ The scenario definitions within the job groups can be created and configured by ### Needles Needles are very precise and the slightest deviation from the specified display will be detected. This means that every time there is a new release, very small changes occur in layout of displays resulting in many new or modified needles being required. There is always a significant amount of work needed by the Test Team to produce the automatic tests for a new version. +A very useful feature of the webui is the online needle editor. When a test fails for a missing needle, the needle editor can be activated by clicking the icon and a new needle can be created, usually by copying a similar needle together with the current screenshot. The needle files are saved in the /var/lib/openqa/tests/rocky/needles directory. + ### Upstream Documentation [Upstream documentation](https://github.com/os-autoinst/openQA/blob/master/docs/Installing.asciidoc) is useful for reference but since it is a mixture of advice and instructions relating to openSUSE and Fedora which have substantial differences between them it is not always clear which are significant for Rocky. However, as an rpm based distribution, Rocky Linux use is loosely related to the [Fedora](https://fedoraproject.org/wiki/OpenQA) version. @@ -258,17 +280,20 @@ scenario build * Different versions of a product as tested, can be considered a "sub-version" of version, e.g. "Build1234"; CAUTION: ambiguity: either with the prefix "Build" included or not -#### History (briefly) +### History (briefly) Openqa started with OS-autoinst: automated testing of Operating Systems The OS-autoinst project aims at providing a means to run fully automated tests, especially to run tests of basic and low-level operating system components such as bootloader, kernel, installer and upgrade, which can not easily be tested with other automated testing frameworks. However, it can just as well be used to test firefox and openoffice operation on top of a newly installed OS. OpenQA is a test-scheduler and web-front for openSUSE and Fedora using OS-autoinst as a backend. OpenQA originated at openSuse and was adopted by Fedora as the automated test system for their frequent distribution updates. Maintenance activity is fairly intense and is ongoing at various levels of users. OpenQA was adopted by Rocky Linux Test Team as the preferred automated testing system for the ongoing releases of it's distribution. OpenQA is free software released under the GPLv2 license. -#### Attribution +### Attribution This guide is heavily inspired by the numerous upstream documents in which installation and usage of OS-autoinst and OpenQA are described. -#### References +### References Since Rocky Linux use of OpenQA is drawn from upstream Fedora and hence openSUSE this document contains **many** passages which are edited versions of upstream documentation and that use is hereby gratefully acknowledged. As with many open source projects, we build on previous work. +### Revision History + +v1.0 - 2024/04/28 - First Issue From ad74a56029547f33103588d0d94c8e9cdc37e9e8 Mon Sep 17 00:00:00 2001 From: AlanMarshall Date: Tue, 30 Apr 2024 14:12:10 +0100 Subject: [PATCH 7/7] Numerous edits to implement review comments --- docs/guidelines/.pages | 2 +- docs/guidelines/openqa_manual_install.md | 163 ++++++++++++----------- 2 files changed, 83 insertions(+), 82 deletions(-) diff --git a/docs/guidelines/.pages b/docs/guidelines/.pages index a4d4add..68dce14 100644 --- a/docs/guidelines/.pages +++ b/docs/guidelines/.pages @@ -2,5 +2,5 @@ nav: - ... | index.md - Release Criteria & Status: release_criteria - - OpenQA Manual Install: openqa_manual_install.md + - openQA Manual Install: openqa_manual_install.md ... diff --git a/docs/guidelines/openqa_manual_install.md b/docs/guidelines/openqa_manual_install.md index 298ee20..af03e42 100644 --- a/docs/guidelines/openqa_manual_install.md +++ b/docs/guidelines/openqa_manual_install.md @@ -1,8 +1,8 @@ --- -title: Manual Install of OpenQA for rockylinux +title: Manual Install of openQA for rockylinux author: Alan Marshall version: v1.0 -revision_date: 2024-04-28 +revision_date: 2024-04-30 rc: prod: Rocky Linux level: Issue @@ -10,25 +10,25 @@ rc: #### Intended Audience -Those who wish to use the OpenQA automated testing system configured for Rocky Linux tests. If so, you will need a PC or server with hardware-virtualisation running an up-to-date Fedora Linux. +Those who wish to use the openQA automated testing system configured for Rocky Linux tests. If so, you will need a PC or server with hardware virtualisation running an up-to-date Fedora Linux. ### Introduction -This guide explains the use of the OpenQA automated testing system to test various aspects of Rocky Linux releases either at the pre-release stage or thereafter. +This guide explains the use of the openQA automated testing system to test various aspects of Rocky Linux releases either at the pre-release stage or thereafter. -OpenQA is an automated test tool that makes it possible to test the whole installation process. It uses virtual machines which it sets up to reproduce the process, check the output (both serial console and GUI screen) in every step and send the necessary keystrokes and commands to proceed to the next step. OpenQA checks whether the system can be installed, whether it works properly, whether applications work and whether the system responds as expected to different installation options and commands. +openQA is an automated test tool that makes it possible to test the whole installation process. It uses virtual machines which it sets up to reproduce the process, check the output (both serial console and GUI screen) in every step and send the necessary keystrokes and commands to proceed to the next step. openQA checks whether the system can be installed, whether it works properly, whether applications work and whether the system responds as expected to different installation options and commands. -Rocky Linux OpenQA tests can be found in the [os-autoinst-distri-rocky](https://github.com/rocky-linux/os-autoinst-distri-rocky) repository +Rocky Linux openQA tests can be found in the [os-autoinst-distri-rocky](https://github.com/rocky-linux/os-autoinst-distri-rocky) repository -OpenQA can run numerous combinations of tests for every revision of the operating system, reporting the errors detected for each combination of hardware configuration, installation options and variant of the operating system. +openQA can run numerous combinations of tests for every revision of the operating system, reporting the errors detected for each combination of hardware configuration, installation options and variant of the operating system. ### WebUI -The web UI is a very useful feature of the OpenQA system since it provides an easily accessed view of the progress and details of OpenQA tests either on the local machine or remotely or both. It is intended to be intuitive and self-explanatory. +The web UI is a very useful feature of the openQA system since it provides an easily accessed view of the progress and details of openQA tests either on the local machine or remotely or both. It is intended to be intuitive and self-explanatory. Some pages use queries to select what should be shown. The query parameters are generated on clickable links, for example starting from the index page or the group overview page clicking on single builds. On the query pages there can be UI elements to control the parameters, for example to look for older builds or show only failed jobs, or other settings. Additionally, the query parameters can be tweaked by hand if you want to provide a link to specific views. ## Step-by-step Install Guide -OpenQA can be installed only on a Fedora (or OpenSUSE) server or workstation. The following install procedure was tested on Fedora 40 Server. You can use either a local terminal or an ssh login from another host on the lan. +openQA can be installed only on a Fedora (or OpenSUSE) server or workstation. The following install procedure was tested on Fedora 40 Server. You can use either a local terminal or an ssh login from another host on the lan. ``` # install Packages @@ -137,13 +137,13 @@ and for a full build (this will post 95 jobs) sudo openqa-cli api -X POST isos ISO=Rocky-9.3-x86_64-dvd.iso ARCH=x86_64 DISTRI=rocky FLAVOR=dvd-iso VERSION=9.3 BUILD="$(date +%Y%m%d.%H%M%S).0"-dvd-iso sudo openqa-cli api -X POST isos ISO=Rocky-9.3-x86_64-dvd.iso ARCH=x86_64 DISTRI=rocky FLAVOR=universal VERSION=9.3 BUILD="$(date +%Y%m%d.%H%M%S).0"-universal ``` -You can watch progress of these tests on the webui on any browser on the same lan as the test host at +You can watch progress of these tests on the webui on any browser on the same lan as the test host at ```http:///tests``` If you click "login" in the top right corner you will be able to control tests from the webui. -At this point the multi-vm tests will fail or be skipped. This is because at the moment your system is configured for single vm deployment and it can be used as such. Pause your installation here if you wish to get some practice using openqa before progressing further (recommended). +At this point the multi-vm tests will fail or be skipped. This is because at the moment your system is configured for single vm deployment and it can be used as such. Pause your installation here if you wish to get some practice using openQA before progressing further (recommended). Installation of facilities for multi-vm testing, which is substantially more complicated, will be described in this document in a later revision (watch this space). @@ -151,22 +151,22 @@ Installation of facilities for multi-vm testing, which is substantially more com ### Helpers #### Createhdds -Createhdds is used to prepare .img and .qcow2 files for some of the Rocky tests. If you ran the above procedure you will have noticed that it produces a number of files in /var/lib/openqa/factory/hdd/fixed determined by the files provided [here](https://github.com/rocky-linux/createhdds). +```Createhdds``` is used to prepare ```.img``` and ```.qcow2``` files for some of the Rocky tests. If you ran the above procedure you will have noticed that it produces a number of files in ```/var/lib/openqa/factory/hdd/fixed``` determined by the files provided in [createhdds](https://github.com/rocky-linux/createhdds). #### openqa-cli -Tests are normally posted using openqa-cli as you have already used above. Test parameters are listed and explained [here](https://github.com/rocky-linux/os-autoinst-distri-rocky/blob/develop/VARIABLES.md) +Tests are normally posted using ```openqa-cli``` as you have already used above. Test parameters are listed and explained in the [openQA VARIABLES definition document](https://github.com/rocky-linux/os-autoinst-distri-rocky/blob/develop/VARIABLES.md) #### Scripts -see [here](https://github.com/rocky-linux/os-autoinst-distri-rocky/tree/develop/scripts) - -cancel-build.sh is especially useful when you discover that you have initiated a large build and got it wrong... d'oh. +[helper scripts](https://github.com/rocky-linux/os-autoinst-distri-rocky/tree/develop/scripts) - +```cancel-build.sh``` is especially useful when you discover that you have initiated a large build and got it wrong... d'oh. ### Using Templates -#### Problem -One of the problems that arises when testing an operating system, especially when doing continuous testing, is that there is always a certain combination of jobs, each one with its own settings, that needs to be run for every revision. These combinations can be different for different FLAVORs of the same revision, like running a different set of jobs for each architecture. This combinational problem can go one step further if OpenQA is being used for different kinds of tests, like running some simple pre-integration tests for some snapshots combined with more comprehensive post-integration tests for release candidates. +#### Challenge +One of the challenges that arises when testing an operating system, especially when doing continuous testing, is that there is always a certain combination of jobs, each one with its own settings, that needs to be run for every revision. These combinations can be different for different ```FLAVORs``` of the same revision, like running a different set of jobs for each architecture. This combinational problem can go one step further if openQA is being used for different kinds of tests, like running some simple pre-integration tests for some snapshots combined with more comprehensive post-integration tests for release candidates. -This section describes how an instance of OpenQA *could* be configured using the options in the admin area of the webUI to automatically create all the required jobs for each revision of your operating system that needs to be tested. *If* you were starting from scratch (the difficult way), you would probably go through the following order: +This section describes how an instance of openQA *could* be configured using the options in the admin area of the webUI to automatically create all the required jobs for each revision of your operating system that needs to be tested. *If* you were starting from scratch (the difficult way), you would probably go through the following order: 1. Define machines in 'Machines' menu 1. Define medium types (products) you have in 'Medium types' menu @@ -176,47 +176,48 @@ This section describes how an instance of OpenQA *could* be configured using the If you followed the install guide above then the cloned Rocky tests from [os-autoinst-distri-rocky](https://github.com/rocky-linux/os-autoinst-distri-rocky) will have pre-configured the admin area of the webUI. You may find it useful to consult when reading the following sections. -Machines, mediums, test suites and job templates can all set various configuration variables. The job templates within the job groups define how the test suites, mediums and machines should be combined in various ways to produce individual 'jobs'. All the variables from the test suite, medium, machine and job template are combined and made available to the actual test code run by the 'job', along with variables specified as part of the job creation request. Certain variables also influence OpenQA’s and/or os-autoinst’s own behavior in terms of how it configures the environment for the job. +Machines, mediums, test suites and job templates can all set various configuration variables. The job templates within the job groups define how the test suites, mediums and machines should be combined in various ways to produce individual 'jobs'. All the variables from the test suite, medium, machine and job template are combined and made available to the actual test code run by the 'job', along with variables specified as part of the job creation request. Certain variables also influence openQA’s and/or os-autoinst’s own behavior in terms of how it configures the environment for the job. -The configuration is set up from /var/lib/openqa/tests/rocky/templates.fif.json +The configuration is set up from ```/var/lib/openqa/tests/rocky/templates.fif.json``` #### Machines -You need to have at least one machine set up to be able to run any tests. Those machines represent virtual machine types that you want to test. Realistically to make tests actually happen, you have to have a number of 'OpenQA workers' connected that can fulfill these specifications. +You need to have at least one machine set up to be able to run any tests. These machines represent virtual machine types that you want to test. Realistically to make tests actually happen, you have to have a number of ```openQA workers``` connected that can fulfill these specifications. -- Name. User defined string - only needed for operator to identify the machine configuration. -- Backend. What backend should be used for this machine. Recommended value is qemu as it is the most tested one, but other options (such as kvm2usb or vbox) are also possible. -- Variables Most machine variables influence os-autoinst’s behavior in terms of how the test machine is set up. A few important examples: - - QEMUCPU can be 'qemu32' or 'qemu64' and specifies the architecture of the virtual CPU - - QEMUCPUS is an integer that specifies the number of cores you wish for +- ```Name``` User defined ```string``` - only needed for operator to identify the machine configuration. +- ```Backend``` What ```backend``` should be used for this ```machine``` Recommended value is ```qemu``` as it is the most tested one, but other options such as ```kvm2usb``` or ```vbox``` are also possible. +- ```Variables``` Most machine variables influence os-autoinst’s behavior in terms of how the test machine is set up. A few important examples: + - ```QEMUCPU``` can be ```qemu32``` or ```qemu64``` and specifies the architecture of the virtual CPU + - ```QEMUCPUS``` is an ```integer``` that specifies the number of cores you wish to be used - - USBBOOT when set to 1, the image will be loaded through an emulated USB stick. + - ```USBBOOT``` when set to ```1``` the image will be loaded through an emulated USB stick. -#### Medium Types (products) -A medium type (product) in OpenQA is a simple description without any definite meaning. It basically consists of a name and a set of variables that define or characterise this product in os-autoinst. +#### Medium Types +- ```product``` + - A medium type ```product``` in openQA is a simple description without any definite meaning. It basically consists of a ```name``` and a set of ```variables``` that define or characterise this product in os-autoinst. Some example variables are: -- ISO_MAXSIZE contains the maximum size of the product. There is a test that checks that the current size of the product is less or equal than this variable. -- DVD if it is set to 1, this indicates that the medium is a DVD. -- LIVECD if it is set to 1, this indicates that the medium is a live image (can be a CD or USB) -- GNOME this variable, if it is set to 1, indicates that it is a GNOME only distribution. -- RESCUECD is set to 1 for rescue CD images. +- ```ISO_MAXSIZE``` contains the maximum size of the ```product```. There is a test that checks that the current size of the product is less or equal than this variable. +- ```DVD``` if it is set to ```1``` this indicates that the medium is a DVD. +- ```LIVECD``` if it is set to ```1``` this indicates that the medium is a live image (can be a ```CD``` or ```USB```) +- ```GNOME``` this variable, if it is set to ```1``` indicates that it is a ```GNOME``` only distribution. +- ```RESCUECD``` is set to ```1``` for rescue CD images. #### Test Suites A test suite consists of a name and a set of test variables that are used inside this particular test together with an optional description. The test variables can be used to parameterise the actual test code and influence the behaviour according to the settings. Some sample variables are: -- DESKTOP possible values are 'kde' 'gnome' 'lxde' 'xfce' or 'textmode'. Used to indicate the desktop selected by the user during the test. -- ENCRYPT encrypt the home directory via YaST. -- HDDSIZEGB hard disk size in GB. -- HDD_1 path for the pre-created hard disk -- RAIDLEVEL RAID configuration variable +- ```DESKTOP``` possible values are ```kde``` ```gnome``` ```lxde``` ```xfce``` or ```textmode```. Used to indicate the desktop selected by the user during the test. +- ```ENCRYPT``` encrypt the home directory via ```YaST``` +- ```HDDSIZEGB``` hard disk size in GB. +- ```HDD_1``` path for the pre-created hard disk +- ```RAIDLEVEL RAID``` configuration variable #### Job Groups The job groups are the place where the actual test scenarios are defined by the selection of the medium type, the test suite and machine together with a priority value. -The priority value is used in the scheduler to choose the next job. If multiple jobs are scheduled and their requirements for running them are fulfilled the ones with a lower priority value are triggered. The id is the second sorting key: of two jobs with equal requirements and same priority value the one with lower id is triggered first. +The priority value is used in the scheduler to choose the next job. If multiple jobs are scheduled and their requirements for running them are fulfilled the ones with a lower priority value are triggered. The id is the second sorting key of two jobs with equal requirements and same priority value the one with lower id is triggered first. Job groups themselves can be created over the web UI as well as the REST API. Job groups can optionally be nested into categories. The display order of job groups and categories can be configured by drag-and-drop in the web UI. @@ -224,76 +225,76 @@ The scenario definitions within the job groups can be created and configured by - A simple web UI wizard which is automatically shown for job groups when a new medium is added to the job group. - An intuitive table within the web UI for adding additional test scenarios to existing media including the possibility to configure the priority values. -- The scripts OpenQA-load-templates and OpenQA-dump-templates to quickly dump and load the configuration from custom plain-text dump format files using the REST API. +- The scripts ```openQA-load-templates``` and ```openQA-dump-templates``` to quickly dump and load the configuration from custom plain-text dump format files using the REST API. - Using declarative schedule definitions in the YAML format using REST API routes or an online-editor within the web UI including a syntax checker. ### Needles Needles are very precise and the slightest deviation from the specified display will be detected. This means that every time there is a new release, very small changes occur in layout of displays resulting in many new or modified needles being required. There is always a significant amount of work needed by the Test Team to produce the automatic tests for a new version. -A very useful feature of the webui is the online needle editor. When a test fails for a missing needle, the needle editor can be activated by clicking the icon and a new needle can be created, usually by copying a similar needle together with the current screenshot. The needle files are saved in the /var/lib/openqa/tests/rocky/needles directory. +A very useful feature of the webui is the online needle editor. When a test fails for a missing needle, the needle editor can be activated by clicking the icon and a new needle can be created, usually by copying a similar needle together with the current screenshot. The needle files are saved in the ```/var/lib/openqa/tests/rocky/needles directory``` ### Upstream Documentation -[Upstream documentation](https://github.com/os-autoinst/openQA/blob/master/docs/Installing.asciidoc) is useful for reference but since it is a mixture of advice and instructions relating to openSUSE and Fedora which have substantial differences between them it is not always clear which are significant for Rocky. However, as an rpm based distribution, Rocky Linux use is loosely related to the [Fedora](https://fedoraproject.org/wiki/OpenQA) version. +[Starter Guide](http://open.qa/docs/) and [Upstream documentation](https://github.com/os-autoinst/openqa/blob/master/docs/Installing.asciidoc) are useful for reference but since they are a mixture of advice and instructions relating to openSUSE and Fedora which have substantial differences between them it is not always clear which are significant for Rocky. However, as an rpm based distribution, Rocky Linux use is loosely related to the [Fedora](https://fedoraproject.org/wiki/OpenQA) version. ### Glossary -The following terms are used within the context of OpenQA:- +The following terms are used within the context of openQA:- -test module -* An individual test case in a single perl module (.pm) file, e.g. "sshxterm". If not further specified a test module is denoted with its "short name" equivalent to the filename including the test definition. The "full name" is composed of the test group (TBC), which itself is formed by the top-folder of the test module file, and the short name, e.g. "x11-sshxterm" (for x11/sshxterm.pm) +- test module + - An individual test case in a single perl module ```.pm``` file, e.g. ```sshxterm``` If not further specified a test module is denoted with its short ```name``` equivalent to the filename including the test definition. The full ```name``` is composed of the test group, which itself is formed by the top-folder of the test module file, and the short name, e.g. ```x11-sshxterm``` (for ```x11/sshxterm.pm```) -test suite -1. A collection of test modules, e.g. "textmode". All test modules within one test suite are run serially -job -1. One run of individual test cases in a row denoted by a unique number for one instance of OpenQA, e.g. one installation with subsequent testing of applications within gnome +- test suite + - A collection of test modules, e.g. ```textmode``` All test modules within one test suite are run serially -test run -* Equivalent to job -test result -* The result of one job, e.g. "passed" with the details of each individual test module -test step -* the execution of one test module within a job + - One run of individual test cases in a row denoted by a unique number for one instance of openQA, e.g. one installation with subsequent testing of applications within ```gnome``` -distri -* A test distribution but also sometimes referring to a product (CAUTION: ambiguous, historically a "GNU/Linux distribution"), composed of multiple test modules in a folder structure that compose test suites, e.g. "rocky" (test distribution, short for "os-autoinst-distri-rocky") +- test run + - Equivalent to job +- test result + - The result of one job, e.g. ```passed``` with the details of each individual test module +- test step + - the execution of one test module within a job -product -* The main "system under test" (SUT), e.g. "rocky", also called "Medium Types" in the web interface of OpenQA +- distri + - A test distribution but also sometimes referring to a ```product``` (CAUTION: ambiguous, historically a "GNU/Linux distribution"), composed of multiple test modules in a folder structure that composes test suites, e.g. ```rocky``` (test distribution, short for ```os-autoinst-distri-rocky```) -job group -* Equivalent to product, used in context of the webUI +- product + - The main ```system under test (SUT)``` e.g. ```rocky``` also called ```Medium Types``` in the web interface of openQA -version -* One version of a product, don’t confuse with builds +- job group + - Equivalent to product, used in context of the webUI -flavor -* Keyword for a specific variant of a product to distinguish differing variants, e.g. "dvd-iso" +- version + - One version of a product, don’t confuse with ```build``` -arch -* An architecture variant of a product, e.g. "x86_64" +- flavor + - Keyword for a specific variant of a product to distinguish differing variants, e.g. ```dvd-iso``` -machine -* Additional variant of machine, e.g. used for "64bit", "uefi", etc. +- arch + - An architecture variant of a product, e.g. ```x86_64``` -scenario -* A composition of ----@, e.g. "Rocky-9-dvd-x86_64-gnome@64bit" +- machine + - Additional variant of machine, e.g. used for ```64bit``` ```bios``` ```uefi``` etc. -build -* Different versions of a product as tested, can be considered a "sub-version" of version, e.g. "Build1234"; CAUTION: ambiguity: either with the prefix "Build" included or not +- scenario + - A composition of ```----@``` e.g. ```Rocky-9-dvd-x86_64-gnome@64bit``` + +- build + - Different versions of a product as tested, can be considered a ```sub-version``` of ```version```, e.g. ```Build1234``` CAUTION: ambiguity: either with the prefix ```build``` included or not ### History (briefly) -Openqa started with OS-autoinst: automated testing of Operating Systems +openQA started with OS-autoinst: automated testing of Operating Systems The OS-autoinst project aims at providing a means to run fully automated tests, especially to run tests of basic and low-level operating system components such as bootloader, kernel, installer and upgrade, which can not easily be tested with other automated testing frameworks. However, it can just as well be used to test firefox and openoffice operation on top of a newly installed OS. -OpenQA is a test-scheduler and web-front for openSUSE and Fedora using OS-autoinst as a backend. -OpenQA originated at openSuse and was adopted by Fedora as the automated test system for their frequent distribution updates. Maintenance activity is fairly intense and is ongoing at various levels of users. OpenQA was adopted by Rocky Linux Test Team as the preferred automated testing system for the ongoing releases of it's distribution. -OpenQA is free software released under the GPLv2 license. +openQA is a test-scheduler and web-front for openSUSE and Fedora using OS-autoinst as a backend. +openQA originated at openSuse and was adopted by Fedora as the automated test system for their frequent distribution updates. Maintenance activity is fairly intense and is ongoing at various levels of users. openQA was adopted by Rocky Linux Test Team as the preferred automated testing system for the ongoing releases of it's distribution. +openQA is free software released under the GPLv2 license. ### Attribution -This guide is heavily inspired by the numerous upstream documents in which installation and usage of OS-autoinst and OpenQA are described. +This guide is heavily inspired by the numerous upstream documents in which installation and usage of OS-autoinst and openQA are described. ### References -Since Rocky Linux use of OpenQA is drawn from upstream Fedora and hence openSUSE this document contains **many** passages which are edited versions of upstream documentation and that use is hereby gratefully acknowledged. As with many open source projects, we build on previous work. +Since Rocky Linux use of openQA is drawn from upstream Fedora and hence openSUSE this document contains **many** passages which are edited versions of upstream documentation and that use is hereby gratefully acknowledged. As with many open source projects, we build on previous work. ### Revision History -v1.0 - 2024/04/28 - First Issue +v1.0 - 2024/04/30 - First Issue