diff --git a/docs/documentation/.pages b/docs/documentation/.pages index 06765ee..593beae 100644 --- a/docs/documentation/.pages +++ b/docs/documentation/.pages @@ -1,4 +1,6 @@ --- nav: - ... | index.md + - QA:Test Cases: qa_test_cases.md + - Wiki Development Guides: dev_guides ... diff --git a/docs/documentation/QA/.pages b/docs/documentation/QA/.pages new file mode 100644 index 0000000..840af76 --- /dev/null +++ b/docs/documentation/QA/.pages @@ -0,0 +1,2 @@ +--- +hide: true diff --git a/docs/documentation/QA/Testcase_Basic_Graphics_Mode.md b/docs/documentation/QA/Testcase_Basic_Graphics_Mode.md new file mode 100644 index 0000000..0658ab0 --- /dev/null +++ b/docs/documentation/QA/Testcase_Basic_Graphics_Mode.md @@ -0,0 +1,44 @@ +--- +title: QA:Testcase Basic Graphics Mode +author: Trevor Cooper +revision_date: 2022-06-23 +rc: + prod: Rocky Linux + ver: 8 + level: Final +--- + +!!! error "REFERENCED RELEASE CRITERIA IS OVERLY GENERAL AND UNTESTABLE" + The associated release criteria, [Release_Criteria#basic-graphics-mode-behaviors](8_release_criteria.md#basic-graphics-mode-behaviors), for this test case is overly general and **must** be modified to specific enough to be testable. + +!!! info "Associated release criterion" + This test case is associated with the [Release_Criteria#basic-graphics-mode-behaviors](8_release_criteria.md#basic-graphics-mode-behaviors) release criterion. If you are doing release validation testing, a failure of this test case may be a breach of that release criterion. + +## Description +This test case will verify that release-blocking installers function as intended using the generic video driver option (“basic graphics mode”) on supported systems and classes of hardware. + +{% include 'qa_testcase_supported_systems.md' %} + +## Setup +1. Obtain access to supported system and hardware class to be installed. +2. Prepare appropriate media for the selected ISO to be tested. + - Example: [QA:Testcase Media USB dd](Testcase_Media_USB_dd.md) + +## How to test +1. Boot the system from the prepared optical, USB media or virtual device attachment. + - Examples: [QA:Testcase Boot Methods Boot ISO](Testcase_Boot_Methods_Boot_Iso.md), [QA:Testcase Boot Methods DVD](Testcase_Boot_Methods_Dvd.md) +2. In the boot menu select the appropriate option to boot the installer. +3. In the installer select the appropriate option to intall in basic graphics mode. +4. Proceed with installation on the test system.
**Depending on installer choices this MAY destroy all the data on the test system.** + +!!! error "DATA LOSS" + If you choose to complete the installation of the test system any/all data on the system may be lost. Please do not install on a system whose contents you need to keep. + +## Expected Results +1. Selection of basic graphics mode in the Anaconda installer is possible. +2. Anaconda installer presents a usable graphical intallation environment. +3. System under test can be installed normally. +4. After reboot system boots into graphical environment. +5. After login user is able to operate the graphical environment. + +{% include 'qa_testcase_bottom.md' %} diff --git a/docs/documentation/QA/Testcase_Boot_Methods_Boot_Iso.md b/docs/documentation/QA/Testcase_Boot_Methods_Boot_Iso.md new file mode 100644 index 0000000..77f90f8 --- /dev/null +++ b/docs/documentation/QA/Testcase_Boot_Methods_Boot_Iso.md @@ -0,0 +1,28 @@ +--- +title: QA:Testcase Boot Methods Boot Iso +author: Trevor Cooper +revision_date: 2022-05-18 +rc: + prod: Rocky Linux + ver: 8 + level: Final +--- + +!!! info "Associated release criterion" + This test case is associated with the [Release_Criteria#initialization-requirements](9_release_criteria.md#initialization-requirements) release criterion. If you are doing release validation testing, a failure of this test case may be a breach of that release criterion. + +## Description +This is to verify that the Anaconda installer starts correctly when booting from the Rocky Linux boot.iso. + +## Setup +1. Prepare your system for booting the boot.iso image. This may involve writing the image to a USB key or burning it to an optical disk. Additionally, attaching the boot.iso to a virtual machine instance as a Virtual Optical Disk or mounting the boot.iso to server via baseboard management controller virtual media attach should be possible but is not expressly required. + +## How to test +1. Boot the system from the prepared optical, USB media or virtual device attachment. +2. In the boot menu select the appropriate option to boot the installer. + +## Expected Results +1. Graphical boot menu is displayed for users to select install options. Navigating the menu and selecting entries must work. If no option is selected, the installer should load after a reasonable timeout. +2. System boots into the Anaconda installer. + +{% include 'qa_testcase_bottom.md' %} diff --git a/docs/documentation/QA/Testcase_Boot_Methods_Dvd.md b/docs/documentation/QA/Testcase_Boot_Methods_Dvd.md new file mode 100644 index 0000000..f979aa7 --- /dev/null +++ b/docs/documentation/QA/Testcase_Boot_Methods_Dvd.md @@ -0,0 +1,28 @@ +--- +title: QA:Testcase Boot Methods DVD +author: Trevor Cooper +revision_date: 2022-05-18 +rc: + prod: Rocky Linux + ver: 8 + level: Final +--- + +!!! info "Associated release criterion" + This test case is associated with the [Release_Criteria#initialization-requirements](9_release_criteria.md#initialization-requirements) release criterion. If you are doing release validation testing, a failure of this test case may be a breach of that release criterion. + +## Description +This is to verify that the Anaconda installer starts correctly when booting from DVD.iso. + +## Setup +1. Prepare your system for booting the DVD.iso image. This may involve writing the image to a USB key or burning it to an optical disk of sufficient capacity. Additionally, attaching the DVD.iso to a virtual machine instance as a Virtual Optical Disk or mounting the DVD.iso to server via baseboard management controller virtual media attach should be possible but is not expressly required. + +## How to test +1. Boot the system from the prepared optical, USB media or virtual device attachment. +2. In the boot menu select the appropriate option to boot the installer. + +## Expected Results +1. Graphical boot menu is displayed for users to select install options. Navigating the menu and selecting entries must work. If no option is selected, the installer should load after a reasonable timeout. +2. System boots into the Anaconda installer. + +{% include 'qa_testcase_bottom.md' %} diff --git a/docs/documentation/QA/Testcase_Bootloader_Disk_Selection.md b/docs/documentation/QA/Testcase_Bootloader_Disk_Selection.md new file mode 100644 index 0000000..bbde0fa --- /dev/null +++ b/docs/documentation/QA/Testcase_Bootloader_Disk_Selection.md @@ -0,0 +1,34 @@ +--- +title: QA:Testcase Bootloader Disk Selection +author: Al Bowles +revision_date: 2022-07-08 +rc: + prod: Rocky Linux + ver: + - 8 + - 9 + level: Final +--- + +!!! info "Associated release criterion" + This test case is associated with the [Release_Criteria#Bootloader Disk Selection](9_release_criteria.md#bootloader-disk-selection) release criterion. If you are doing release validation testing, a failure of this test case may be a breach of that release criterion. + +## Description +This test case verifies that the user is able to select an alternative disk on which to install the bootloader. It also verifies that, if the user is so inclined, they may choose not to install a bootloader at all. + +{% include 'qa_data_loss_warning.md' %} + +## Setup +{% include 'qa_setup_boot_to_media.md' %} + +## How to test +1. In the Installation Destination spoke, select the disk(s) to install to, then click the "Full disk summary and bootl loader..." button at the bottom of the screen: ![Full disk summary and bootloader...](/assets/images/bootloader.png){ loading=lazy } +1. Click the checkbox next to the disk on which the bootloader is desired +1. Alternatively, uncheck the boot checkbox next to all disks to skip bootloader installation +1. Proceed with installation on the test system. + +## Expected Results +1. Installation should complete successfully. +1. Note that if no bootloader is installed, the system may not boot after installation is complete. This is expected. + +{% include 'qa_testcase_bottom.md' %} diff --git a/docs/documentation/QA/Testcase_Custom_Boot_Methods_Boot_Iso.md b/docs/documentation/QA/Testcase_Custom_Boot_Methods_Boot_Iso.md new file mode 100644 index 0000000..cb87435 --- /dev/null +++ b/docs/documentation/QA/Testcase_Custom_Boot_Methods_Boot_Iso.md @@ -0,0 +1,32 @@ +--- +title: QA:Testcase Custom Boot Methods Boot Iso +author: Trevor Cooper +revision_date: 2022-06-23 +rc: + prod: Rocky Linux + ver: 9 + level: Final +--- + +!!! info "Associated release criterion" + This test case is associated with the [Release_Criteria#vnc-graphics-mode-behaviors](9_release_criteria.md#vnc-graphics-mode-behaviors) release criterion. If you are doing release validation testing, a failure of this test case may be a breach of that release criterion. + +## Description +This is to verify that the Anaconda installer starts correctly when booting from the Rocky Linux boot.iso using a custom kernel command line. + +## Setup +1. Prepare your system for booting the boot.iso image. This may involve writing the image to a USB key or burning it to an optical disk. Additionally, attaching the boot.iso to a virtual machine instance as a Virtual Optical Disk or mounting the boot.iso to server via baseboard management controller virtual media attach should be possible but is not expressly required. + +## How to test +1. Boot the system from the prepared optical, USB media or virtual device attachment. +2. In the boot menu select the appropriate option to boot the installer. +3. Interrupt the normal boot and edit the kernel command line. +4. Add appropriate/required options to the kernel command line and resume booting into the installer. + - Example: For network install from an alternate repository add `--inst.url=http:///` and (optionally) `--inst.repo=AppStream,http:///` to the kernel command line. + - Example: For VNC install in **Direct Mode** add `--inst.vnc` to the kernel command line. For VNC install in **Connect Mode** add `--inst.vnc` and `--inst.vncserver=:` to the kernel command line. + +## Expected Results +1. Boot menu is displayed for users to select install options. Navigating the menu and selecting entries must work. Editing the boot command line must be possible. If no option is selected, the installer should load after a reasonable timeout. +2. System boots into the Anaconda installer and any command line options specified are utilized. + +{% include 'qa_testcase_bottom.md' %} diff --git a/docs/documentation/QA/Testcase_Debranding.md b/docs/documentation/QA/Testcase_Debranding.md new file mode 100644 index 0000000..264c434 --- /dev/null +++ b/docs/documentation/QA/Testcase_Debranding.md @@ -0,0 +1,200 @@ +--- +title: QA:Testcase Debranding +author: Trevor Cooper +revision_date: 2022-05-18 +rc: + prod: Rocky Linux + ver: 8 + level: Final +--- + +!!! info "Associated release criterion" + This test case is associated with the [Release_Criteria - Debranding](9_release_criteria.md#debranding) release criterion. If you are doing release validation testing, a failure of this test case may be a breach of that release criterion. + +## Description +The Rocky Linux [Release Engineering Team](/team/release_engineering/) builds and maintains tools to manage the debranding of packages received from the upstream vendor. They have published a comprehensive [debranding guide](/team/release_engineering/debranding/) and maintain a [list of packages](https://git.rockylinux.org/rocky/metadata/-/blob/main/patch.yml) requiring debranding patches. + +This testcase will verify that all packages available on released media that Rocky Linux Release Engineering has identified as requiring debranding are debranded successfully per their specification. + +## Setup +1. Obtain access to an environment with the `dnf`, and `koji` commands and access to [Rocky Linux Gitlab](https://git.rockylinux.org) and [Rocky Linux Koji](https://koji.rockylinux.org) +2. Download the ISO to be tested to test machine. +3. Configure `/etc/koji.conf` to access the [Rocky Linux Koji](https://koji.rockylinux.org). +4. Download a recent copy the [patch.yml](https://git.rockylinux.org/rocky/metadata/-/blob/main/patch.yml) from [Rocky Linux Gitlab](https://git.rockylinux.org). + +!!! info "patch.yml" + Packages listed in `patch.yml` are names of source RPMs. Binary RPMs containing content produced by building the patched source RPMs need to be validated. The easiest way to get the list of all possible binary RPMs for a particular package and arch is to ask obtain that information in koji. + +## How to test +1. Mount the ISO to be tested locally. + - Example: + ``` + $ mount -o loop Rocky-8.5-x86_64-dvd1.iso /media + ``` +2. Determine the path(s) to the `repodata` directory(ies) on the ISO. + - Example: + ``` + $ find /media -name repodata + ``` +3. For each package to be validated get the names of the `noarch` and `` specific packages created from it. + - Example: + ``` + $ koji --quiet latest-build --arch=x86_64 dist-rocky8-compose + $ koji --quiet latest-build --arch=noarch dist-rocky8-compose + ``` +4. Use `dnf` to obtain the paths to the binary packages requiring debranding. + - Example: + ``` + $ dnf download --urls --repofrompath BaseOS,/media/BaseOS --repo BaseOS \ + --repofrompath Minimal,/media/Minimal --repo Minimal \ + + ``` +5. Copy the `` from the media and examine it's metadata and/or contents to determine if it has obviously been patched. + - Example: + ``` + $ rpm -q --changelog -p | head | \ + grep "Release Engineering " -C2 | \ + grep -Eq "" + + $ rpm2cpio | + cpio --quiet --extract --to-stdout . | \ + grep -Eq "" + ``` + + !!! info "NOTE" + Note all debranding patches will patch files directly and leave very obvious traces, some patches don't even add changelog messages to use as an indicator that the package has been patched or debranded. Sometimes the only solution is to extract the binary package and examine the contents directly to find something to test. + +6. Unmount the ISO. + - Example: + ``` + $ umount /media + ``` + +## Expected Results +1. Packages tracked by Release Engineering as requiring debranding and published on installation media are, in fact, debranded per their specification. + +

Sample Output

+ +=== "Success" + + ``` + $ sudo mount -o loop Rocky-8.5-aarch64-minimal.iso /media + mount: /media: WARNING: device write-protected, mounted read-only. + + $ find /media -name repodata + /media/BaseOS/repodata + /media/Minimal/repodata + + $ curl -LOR https://git.rockylinux.org/rocky/metadata/-/raw/main/patch.yml + % Total % Received % Xferd Average Speed Time Time Time Current + Dload Upload Total Spent Left Speed + 100 3410 100 3410 0 0 20419 0 --:--:-- --:--:-- --:--:-- 20419 + + $ yq .debrand.all[] patch.yml | column -x -c 100 -o " " + abrt anaconda anaconda-user-help chrony + cloud-init cockpit crash dhcp + dnf firefox fwupd gcc + gcc-toolset-9-gcc gcc-toolset-10-gcc gcc-toolset-11-gcc gcc-toolset-12-gcc + gnome-settings-daemon grub2 httpd initial-setup + kernel kernel-rt libdnf libreoffice + libreport nginx opa-ff opa-fm + openscap pesign PackageKit python-pip + python3 redhat-rpm-config scap-security-guide shim + shim-unsigned-x64 shim-unsigned-aarch64 sos subscription-manager + systemd thunderbird WALinuxAgent + + $ ./yq .debrand.r8[] patch.yml | column -x -c 100 -o " " + dotnet3.0 fwupdate gnome-boxes libguestfs pcs plymouth + python2 + + NOTE: Only a single package will be shown in this Example. + + $ koji --quiet latest-build --arch=x86_64 dist-rocky8-compose sos + + $ koji --quiet latest-build --arch=noarch dist-rocky8-compose sos + sos-4.1-9.el8_5.rocky.3.noarch + sos-audit-4.1-9.el8_5.rocky.3.noarch + + $ dnf download --urls --repofrompath BaseOS,/media/BaseOS --repo BaseOS \ + --repofrompath Minimal,/media/Minimal --repo Minimal \ + sos sos-audit | grep -E "^file" + file:///media/BaseOS/Packages/s/sos-4.1-5.el8.noarch.rpm + file:///media/BaseOS/Packages/s/sos-audit-4.1-5.el8.noarch.rpm + + $ rpm -q --changelog -p /media/BaseOS/Packages/s/sos-4.1-5.el8.noarch.rpm | \ + head | grep "Release Engineering " -C2 + * Mon Oct 18 2021 Release Engineering - 4.1-5 + - Remove Red Hat branding from sos + $ echo $? + 0 + + $ rpm -q --changelog -p /media/BaseOS/Packages/s/sos-audit-4.1-5.el8.noarch.rpm | \ + head | grep "Release Engineering " -C2 + * Mon Oct 18 2021 Release Engineering - 4.1-5 + - Remove Red Hat branding from sos + $ echo $? + 0 + + $ umount /media + ``` + +=== "Failure" + + ``` + $ sudo mount -o loop Rocky-8.5-aarch64-minimal.iso /media + mount: /media: WARNING: device write-protected, mounted read-only. + + $ find /media -name repodata + /media/BaseOS/repodata + /media/Minimal/repodata + + $ curl -LOR https://git.rockylinux.org/rocky/metadata/-/raw/main/patch.yml + % Total % Received % Xferd Average Speed Time Time Time Current + Dload Upload Total Spent Left Speed + 100 3410 100 3410 0 0 20419 0 --:--:-- --:--:-- --:--:-- 20419 + + $ yq .debrand.all[] patch.yml | column -x -c 100 -o " " + abrt anaconda anaconda-user-help chrony + cloud-init cockpit crash dhcp + dnf firefox fwupd gcc + gcc-toolset-9-gcc gcc-toolset-10-gcc gcc-toolset-11-gcc gcc-toolset-12-gcc + gnome-settings-daemon grub2 httpd initial-setup + kernel kernel-rt libdnf libreoffice + libreport nginx opa-ff opa-fm + openscap pesign PackageKit python-pip + python3 redhat-rpm-config scap-security-guide shim + shim-unsigned-x64 shim-unsigned-aarch64 sos subscription-manager + systemd thunderbird WALinuxAgent + + $ ./yq .debrand.r8[] patch.yml | column -x -c 100 -o " " + dotnet3.0 fwupdate gnome-boxes libguestfs pcs plymouth + python2 + + NOTE: Only a single package will be shown in this Example. + + $ koji --quiet latest-build --arch=x86_64 dist-rocky8-compose sos + + $ koji --quiet latest-build --arch=noarch dist-rocky8-compose sos + sos-4.1-9.el8_5.rocky.3.noarch + sos-audit-4.1-9.el8_5.rocky.3.noarch + + $ dnf download --urls --repofrompath BaseOS,/media/BaseOS --repo BaseOS \ + --repofrompath Minimal,/media/Minimal --repo Minimal \ + sos sos-audit | grep -E "^file" + file:///media/BaseOS/Packages/s/sos-4.1-5.el8.noarch.rpm + file:///media/BaseOS/Packages/s/sos-audit-4.1-5.el8.noarch.rpm + + $ rpm -q --changelog -p /media/BaseOS/Packages/s/sos-4.1-5.el8.noarch.rpm | \ + head | grep "Release Engineering " -C2 + $ echo $? + 1 + + $ rpm -q --changelog -p /media/BaseOS/Packages/s/sos-audit-4.1-5.el8.noarch.rpm | \ + head | grep "Release Engineering " -C2 + $ echo $? + 1 + + $ umount /media + ``` + +{% include 'qa_testcase_bottom.md' %} diff --git a/docs/documentation/QA/Testcase_Disk_Layouts.md b/docs/documentation/QA/Testcase_Disk_Layouts.md new file mode 100644 index 0000000..0d20c8a --- /dev/null +++ b/docs/documentation/QA/Testcase_Disk_Layouts.md @@ -0,0 +1,61 @@ +--- +title: QA:Testcase Disk Layouts +author: Al Bowles +revision_date: 2022-07-08 +rc: + prod: Rocky Linux + ver: + - 8 + - 9 + level: Final +--- + +!!! info "Associated release criterion" + This test case is associated with the [Release_Criteria#Disk Layouts](9_release_criteria.md#disk-layouts) release criterion. If you are doing release validation testing, a failure of this test case may be a breach of that release criterion. + +## Description +This test case verifies successful installation to any supported partition layout using any file system or format combination. + +{% include 'qa_data_loss_warning.md' %} + +## Setup +{% include 'qa_setup_boot_to_media.md' %} + +## How to test +1. Select the Installation Destination spoke. +1. Select the volumes to which the operating system should be installed. +1. Select the Custom radio button under the Storage Configuration section, then click "Done". +1. For each volume, perform these steps: + 1. Choose the desired partitioning scheme from the dropdown menu. Supported options are Standard Partition, LVM, and LVM Thin Provisioning. + 1. Select the "Encrypt my data" checkbox to create an encrypted filesystem. + 1. Select the plus (+) button in the lower left hand corner to add a partition. + 1. Define the desired mount point and volume capacity, then click "Add mount point". + 1. Set the device type. Supported options are LVM, RAID, Standard Partition, and LVM Thin Provisioning. + 1. If device type was set to RAID, select the RAID level. Supported options are RAID0, RAID1, RAID4, RAID5, RAID6, and RAID10. + 1. Set the filesystem type. Supported options are BIOS Boot, ext2, ext3, ext4, swap, vfat, and xfs. + 1. In supported cases you may choose to disable formatting of existing partitions by unchecking the Reformat checkbox. +1. When all partitions have been created, click the blue Done button in the upper left corner. +1. Review the Summary of Changes dialog, then click Accept Changes. +1. Continue the installation as normal. + +## Expected Results +1. The installation should complete successfully and boot to the appropriate disk. +1. The specified filesystem type and partition scheme should be used. +1. If configured, software RAID should function as expected. + +## Testing with openQA +The following openQA test suites satisfy this release criteria: + +- `install_standard_partition_ext4` +- `install_custom_gui_standard_partition_ext4` +- `install_lvm_ext4` +- `install_custom_gui_lvm_ext4` +- `install_software_raid` +- `install_custom_gui_software_raid` +- `install_xfs` +- `install_custom_gui_xfs` +- `install_lvmthin` +- `install_multi` +- `install_multi_empty` + +{% include 'qa_testcase_bottom.md' %} diff --git a/docs/documentation/QA/Testcase_Firmware_RAID.md b/docs/documentation/QA/Testcase_Firmware_RAID.md new file mode 100644 index 0000000..c96c4c3 --- /dev/null +++ b/docs/documentation/QA/Testcase_Firmware_RAID.md @@ -0,0 +1,30 @@ +--- +title: QA:Testcase Firmware RAID +author: Al Bowles +revision_date: 2022-07-08 +rc: + prod: Rocky Linux + ver: + - 8 + - 9 + level: Final +--- + +!!! info "Associated release criterion" + This test case is associated with the [Release_Criteria#Firmware RAID](9_release_criteria.md#firmware-raid) release criterion. If you are doing release validation testing, a failure of this test case may be a breach of that release criterion. + +## Description +The installer must be able to detect and install to firmware RAID devices. Note that system-specific bugs do not count as blockers. It is likely that some hardware support might be broken or not available at all. DUDs (driver update disks) are not considered for this criteria. + +## Setup +1. Add steps for setup for this Testcase. + +## How to test +1. Do this first... +2. Then do this... + +## Expected Results +1. This is what you should see/verify. +2. You should also see/verify this. + +{% include 'qa_testcase_bottom.md' %} diff --git a/docs/documentation/QA/Testcase_Installation_Interfaces.md b/docs/documentation/QA/Testcase_Installation_Interfaces.md new file mode 100644 index 0000000..40b0054 --- /dev/null +++ b/docs/documentation/QA/Testcase_Installation_Interfaces.md @@ -0,0 +1,55 @@ +--- +title: QA:Testcase Installation Interfaces +author: Al Bowles +revision_date: 2022-07-08 +rc: + prod: Rocky Linux + ver: + - 8 + - 9 + level: Final +--- + +!!! info "Associated release criterion" + This test case is associated with the [Release_Criteria#Installation Interfaces](9_release_criteria.md#installation-interfaces) release criterion. If you are doing release validation testing, a failure of this test case may be a breach of that release criterion. + +## Description +This test case verifies that the installer can complete an installation using all Anaconda spokes. + +{% include 'qa_data_loss_warning.md' %} + +## Setup +{% include 'qa_setup_boot_to_media.md' %} + +## How to test + +1. Select a keyboard layout in the Keyboard Layout spoke +1. Set language support in the Language spoke +1. Set the system time and date in the Time and Date spoke + +1. Set a root password in the Root Password spoke +1. Create a user in the user creation spoke + +1. Select an installation source from the Installation Source spoke +1. Select a set of packages to install from the Package Selection spoke + +1. Set a disk to which the operating system should install in the Installation Destination spoke +1. Set the kdump state from the Kdump spoke +1. Configure the system's network and hostname from the Network and Hostname spoke +1. Select a security policy from the Security Policy spoke + +## Expected Results +1. The installation should complete and boot successfully. + +## Testing in openQA +The following openQA test suites satisfy this release criteria: + +- `install_arabic_language` OR `install_asian_language` + + +{% include 'qa_testcase_bottom.md' %} diff --git a/docs/documentation/QA/Testcase_Installer_Help.md b/docs/documentation/QA/Testcase_Installer_Help.md new file mode 100644 index 0000000..9cfe903 --- /dev/null +++ b/docs/documentation/QA/Testcase_Installer_Help.md @@ -0,0 +1,46 @@ +--- +title: QA:Testcase Installer Help +author: Al Bowles +revision_date: 2022-07-08 +rc: + prod: Rocky Linux + ver: + - 8 + - 9 + level: Final +--- + +!!! info "Associated release criterion" + This test case is associated with the [Release_Criteria#Installer Help](9_release_criteria.md#installer-help) release criterion. If you are doing release validation testing, a failure of this test case may be a breach of that release criterion. + +## Description +Any element in the installer which contains a “help” text must display the appropriate help documentation when selected. + +## Setup +{% include 'qa_setup_boot_to_media.md' %} + +## How to test +1. From the Anaconda Hub, click the Help button in the upper right hand corner. +1. Verify that you see the "Customizing your Installation" help page. +1. Verify that the "Configuring language and location settings" link displays a topically appropriate page. +1. Close the Help browser to return to the Anaconda Hub. +1. Verify that the Localization help page displays for the Keyboard, Language Support, and Time & Date spokes: + 1. Select the spoke, then click the Help button. + 1. Verify that you see the "Configuring localization options" page containing a functioning link to the "Configuring keyboard, language, and time and date settings" page. + 1. Close the Help browser (and click Done if necessary) to return to the Anaconda Hub. +1. Verify that the Help button in the Installation Source spoke displays the "Configuring installation source" page. +1. Verify that the Help button in the Software Selection spoke displays the "Configuring software selection" page. +1. Verify that the Help button in the Installation Destination spoke displays the "Configuring storage devices" page. +1. Verify that the Help button in the Network & Host Name spoke displays the "Configuring network and host name options" page. +1. Verify that the Help button in the Root Password spoke displays the "Configuring a root password" page. +1. Verify that the Help button in the User Creation spoke displays the "Creating a user account" page. + +## Expected Results +1. All links should work and display relevant content. + +## Testing in openQA +The following openQA test suites satisfy this release criteria: + +- `anaconda_help` + +{% include 'qa_testcase_bottom.md' %} diff --git a/docs/documentation/QA/Testcase_Installer_Translations.md b/docs/documentation/QA/Testcase_Installer_Translations.md new file mode 100644 index 0000000..a052160 --- /dev/null +++ b/docs/documentation/QA/Testcase_Installer_Translations.md @@ -0,0 +1,37 @@ +--- +title: QA:Testcase Installer Translations +author: Al Bowles +revision_date: 2022-07-08 +rc: + prod: Rocky Linux + ver: + - 8 + - 9 + level: Final +--- + +!!! info "Associated release criterion" + This test case is associated with the [Release_Criteria#Installer Translations](9_release_criteria.md#installer-translations) release criterion. If you are doing release validation testing, a failure of this test case may be a breach of that release criterion. + +## Description +The installer must correctly display all complete translations that are available for use. + +## Setup +{% include 'qa_setup_boot_to_media.md' %} + +## How to test +1. From the Language Selection spoke, select a language. + +## Expected Results +1. All spokes should display at least some of the content in the selected language. +2. It is expected to still see some content displayed in Latin characters even if a language that does not use Latin characters is selected. + +## Testing in openQA +The following openQA test suites satisfy this release criteria: + +- `install_asian_language` +- `install_arabic_language` +- `install_cyrillic_language` +- `install_european_language` + +{% include 'qa_testcase_bottom.md' %} diff --git a/docs/documentation/QA/Testcase_Kickstart_Installation.md b/docs/documentation/QA/Testcase_Kickstart_Installation.md new file mode 100644 index 0000000..1763890 --- /dev/null +++ b/docs/documentation/QA/Testcase_Kickstart_Installation.md @@ -0,0 +1,41 @@ +--- +title: QA:Testcase Kickstart Installation +author: Al Bowles +revision_date: 2022-07-08 +rc: + prod: Rocky Linux + ver: + - 8 + - 9 + level: Final +--- + +!!! info "Associated release criterion" + This test case is associated with the [Release_Criteria#Kickstart Installation](9_release_criteria.md#kickstart-installation) release criterion. If you are doing release validation testing, a failure of this test case may be a breach of that release criterion. + +## Description +This test case verifies that installations via both local and remote Kickstart configuration files are successful. + +{% include 'qa_data_loss_warning.md' %} + +## Setup +1. Copy a valid Kickstart file to a USB stick +1. Connect the USB stick to the test system +{% include 'qa_setup_boot_to_media.md' %} +1. Hit the Tab key to edit the boot command +1. Provide a local Kickstart file by supplying the GRUB boot option `inst.ks=file:/path/to/local.ks` or a remote Kickstart file by supplying the GRUB boot option `inst.ks=https://git.resf.org/testing/createhdds/raw/branch/rocky/server.ks`. + +## How to test +1. Continue booting the installer as normal. + +## Expected Results +1. The installation should complete and boot successfully, automatically populating the options specified in the Kickstart file. + +## Testing in openQA +The following openQA test suites satisfy this release criteria: + +- `install_kickstart_nfs` +- `server_realmd_join_kickstart` + + +{% include 'qa_testcase_bottom.md' %} diff --git a/docs/documentation/QA/Testcase_Media_File_Conflicts.md b/docs/documentation/QA/Testcase_Media_File_Conflicts.md new file mode 100644 index 0000000..aec8878 --- /dev/null +++ b/docs/documentation/QA/Testcase_Media_File_Conflicts.md @@ -0,0 +1,175 @@ +--- +title: QA:Testcase Media File Conflicts +author: Trevor Cooper +revision_date: 2022-05-18 +rc: + prod: Rocky Linux + ver: 8 + level: Final +--- + +!!! info "Associated release criterion" + This test case is associated with the [Release_Criteria#no-broken-packages](9_release_criteria.md#no-broken-packages) release criterion. If you are doing release validation testing, a failure of this test case may be a breach of that release criterion. + +## Description +This testcase will verify that the offline repository included on release blocking images will not contain any file conflicts between packages without explicit `Conflicts:` tag in the package metadata. + +## Setup +1. Obtain access to an environment with the `dnf` and `python3` commands. +2. Download the ISO to be tested to that machine. +3. Download the `potential_conflict.py` script provided by Rocky Linux Testing Team. + +## How to test +1. Mount the ISO to be tested locally. + - Example:
`mount -o loop Rocky-8.5-x86_64-minimal.iso /media` +2. Determine the path to the `repodata` directory(ies) on the ISO. + - Example:
`find /media -name repodata` +3. Run the `potential_conflict.py` script on the mounted ISO. + - Example:
`python3 /vagrant/scripts/potential_conflict.py --repofrompath BaseOS,/media/BaseOS --repoid BaseOS --repofrompath Minimal,/media/Minimal --repoid Minimal` +4. Unmount the ISO. + - Example:
`umount /media` + +## Expected Results +1. The `potential_conflict.py` script does not report any packages with non-declared conflicts. + +

Sample Output

+ +=== "Success" + + ``` + $ sudo mount -o loop Rocky-8.5-aarch64-minimal.iso /media + mount: /media: WARNING: device write-protected, mounted read-only. + + $ python3 /vagrant/scripts/potential_conflict.py \ + --repofrompath BaseOS,/media/BaseOS --repoid BaseOS \ + --repofrompath Minimal,/media/Minimal --repoid Minimal + + Added BaseOS repo from /media/BaseOS + Added Minimal repo from /media/Minimal + Getting complete filelist for: + file:///media/BaseOS + file:///media/Minimal + 168374 files found. + + Looking for duplicated filenames: + 524 duplicates found. + + Doing more advanced checks to see if these are real conflicts: + 10% complete ( 52/ 524, 1139/sec), 0 found - eta 0:00:00 + 35% complete ( 182/ 524, 1146/sec), 0 found - eta 0:00:00 + 45% complete ( 234/ 524, 1818/sec), 0 found - eta 0:00:00 + 50% complete ( 260/ 524, 592673/sec), 0 found - eta 0:00:00 + 55% complete ( 286/ 524, 778942/sec), 0 found - eta 0:00:00 + 60% complete ( 312/ 524, 801852/sec), 0 found - eta 0:00:00 + 79% complete ( 416/ 524, 234/sec), 0 found - eta 0:00:00 + 84% complete ( 442/ 524, 902/sec), 0 found - eta 0:00:00 + 89% complete ( 468/ 524, 935/sec), 0 found - eta 0:00:00 + 94% complete ( 494/ 524, 1616/sec), 0 found - eta 0:00:00 + 99% complete ( 520/ 524, 1114/sec), 0 found - eta 0:00:00 + + 0 file conflicts found. + 0 package conflicts found. + + == Package conflicts == + + == File conflicts, listed by conflicting packages == + + $ sudo umount /media + ``` + +=== "Failure" + + ``` + $ sudo mount -o loop Rocky-8.5-x86_64-dvd1.iso /media + mount: /media: WARNING: device write-protected, mounted read-only. + + + $ python3 /vagrant/scripts/potential_conflict.py \ + --repofrompath AppStream,/media/AppStream --repoid AppStream \ + --repofrompath BaseOS,/media/BaseOS --repoid BaseOS + + Added AppStream repo from /media/AppStream + Added BaseOS repo from /media/BaseOS + Getting complete filelist for: + file:///media/AppStream + file:///media/BaseOS + 851967 files found. + + Looking for duplicated filenames: + 101865 duplicates found. + + Doing more advanced checks to see if these are real conflicts: + 5% complete ( 5093/101865, 8713/sec), 0 found - eta 0:00:11 + 10% complete ( 10186/101865, 1787281/sec), 0 found - eta 0:00:05 + 15% complete ( 15279/101865, 2223312/sec), 0 found - eta 0:00:03 + 20% complete ( 20372/101865, 23614/sec), 0 found - eta 0:00:03 + 25% complete ( 25465/101865, 57188/sec), 0 found - eta 0:00:02 + 30% complete ( 30558/101865, 3831/sec), 0 found - eta 0:00:05 + 35% complete ( 35651/101865, 48455/sec), 0 found - eta 0:00:04 + 40% complete ( 40744/101865, 32067/sec), 0 found - eta 0:00:03 + 45% complete ( 45837/101865, 2136586/sec), 0 found - eta 0:00:03 + 50% complete ( 50930/101865, 72529/sec), 0 found - eta 0:00:02 + 55% complete ( 56023/101865, 176294/sec), 0 found - eta 0:00:02 + 60% complete ( 61116/101865, 68622/sec), 1 found - eta 0:00:01 + 65% complete ( 66209/101865, 155133/sec), 1 found - eta 0:00:01 + 70% complete ( 71302/101865, 13874/sec), 1 found - eta 0:00:01 + 75% complete ( 76395/101865, 10835/sec), 1 found - eta 0:00:01 + 80% complete ( 81488/101865, 27477/sec), 1 found - eta 0:00:00 + 85% complete ( 86581/101865, 9075/sec), 1 found - eta 0:00:00 + 90% complete ( 91674/101865, 14807/sec), 1 found - eta 0:00:00 + 95% complete ( 96767/101865, 197437/sec), 1 found - eta 0:00:00 + 100% complete (101860/101865, 38727/sec), 1 found - eta 0:00:00 + + 1 file conflicts found. + 11 package conflicts found. + + == Package conflicts == + mariadb-server-utils-3:10.3.28-1.module+el8.4.0+427+adf35707.x86_64 + mysql-server-8.0.26-1.module+el8.4.0+652+6de068a7.x86_64 + + python3-mod_wsgi-4.6.4-4.el8.x86_64 + python38-mod_wsgi-4.6.8-3.module+el8.4.0+570+c2eaf144.x86_64 + python39-mod_wsgi-4.7.1-4.module+el8.4.0+574+843c4898.x86_64 + + libcmpiCppImpl0-2.0.3-15.el8.i686 + tog-pegasus-libs-2:2.14.1-46.el8.i686 + + mariadb-connector-c-devel-3.1.11-2.el8_3.i686 + mariadb-connector-c-devel-3.1.11-2.el8_3.x86_64 + mariadb-devel-3:10.3.28-1.module+el8.4.0+427+adf35707.x86_64 + mysql-devel-8.0.26-1.module+el8.4.0+652+6de068a7.x86_64 + + mariadb-server-3:10.3.28-1.module+el8.4.0+427+adf35707.x86_64 + mysql-server-8.0.26-1.module+el8.4.0+652+6de068a7.x86_64 + + mariadb-test-3:10.3.28-1.module+el8.4.0+427+adf35707.x86_64 + mysql-test-8.0.26-1.module+el8.4.0+652+6de068a7.x86_64 + + mariadb-connector-c-devel-3.1.11-2.el8_3.i686 + mariadb-connector-c-devel-3.1.11-2.el8_3.x86_64 + mysql-devel-8.0.26-1.module+el8.4.0+652+6de068a7.x86_64 + + mariadb-devel-3:10.3.28-1.module+el8.4.0+427+adf35707.x86_64 + mysql-devel-8.0.26-1.module+el8.4.0+652+6de068a7.x86_64 + + mariadb-3:10.3.28-1.module+el8.4.0+427+adf35707.x86_64 + mysql-8.0.26-1.module+el8.4.0+652+6de068a7.x86_64 + + libcmpiCppImpl0-2.0.3-15.el8.x86_64 + tog-pegasus-libs-2:2.14.1-46.el8.x86_64 + + libev-libevent-devel-4.24-6.el8.i686 + libev-libevent-devel-4.24-6.el8.x86_64 + libevent-devel-2.1.8-5.el8.i686 + libevent-devel-2.1.8-5.el8.x86_64 + + + == File conflicts, listed by conflicting packages == + mariadb-server-3:10.3.28-1.module+el8.4.0+427+adf35707.x86_64 + mysql-test-8.0.26-1.module+el8.4.0+652+6de068a7.x86_64 + /usr/bin/mysqld_safe + + $ sudo umount /media + ``` + +{% include 'qa_testcase_bottom.md' %} diff --git a/docs/documentation/QA/Testcase_Media_Repoclosure.md b/docs/documentation/QA/Testcase_Media_Repoclosure.md new file mode 100644 index 0000000..d2be195 --- /dev/null +++ b/docs/documentation/QA/Testcase_Media_Repoclosure.md @@ -0,0 +1,86 @@ +--- +title: QA:Testcase Media Repoclosure +author: Trevor Cooper +revision_date: 2022-05-18 +rc: + prod: Rocky Linux + ver: 8 + level: Final +--- + +!!! info "Associated release criterion" + This test case is associated with the [Release_Criteria#no-broken-packages](9_release_criteria.md#no-broken-packages) release criterion. If you are doing release validation testing, a failure of this test case may be a breach of that release criterion. + +## Description +This testcase will verify that the offline repository included on release blocking images will not contain broken dependencies. + +## Setup +1. Obtain access to an environment with the `dnf repoclosure` command. +2. Download the ISO to be tested to that machine. + +## How to test +1. Mount the ISO to be tested locally. + - Example:
`mount -o loop Rocky-8.5-x86_64-minimal.iso /media` +2. Determine the path to the `repodata` directory(ies) on the ISO. + - Example:
`find /media -name repodata` +3. Run the `dnf repoclosure` command on the mounted ISO. + - Example:
`dnf --verbose repoclosure --repofrompath BaseOS,/media/BaseOS --repo BaseOS --repofrompath Minimal,/media/Minimal --repo Minimal` +4. Unmount the ISO. + - Example:
`umount /media` + +## Expected Results +1. The `dnf repoclosure` command does not generate any errors. + +

Sample Output

+ +=== "Success" + + ``` + $ sudo mount -o loop Rocky-8.5-x86_64-minimal.iso /media + mount: /media: WARNING: device write-protected, mounted read-only. + + [vagrant@localhost ~]$ dnf --refresh repoclosure \ + --repofrompath BaseOS,/media/BaseOS --repo BaseOS \ + --repofrompath Minimal,/media/Minimal --repo Minimal + Added BaseOS repo from /media/BaseOS + Added Minimal repo from /media/Minimal + BaseOS 102 MB/s | 2.6 MB 00:00 + Minimal 90 kB/s | 384 B 00:00 + + $ sudo umount /media + ``` + +=== "Failure" + + __NOTE: In this example the content of the `Rocky-8.5-x86_64-minimal.iso` was copied to `/tmp` then the BaseOS repository was modified to remove the `setup-2.12.2-6.el8.noarch.rpm` package and the repository metadata was regenerated.__ + + ``` + [vagrant@localhost ~]$ dnf --refresh repoclosure \ + --repofrompath BaseOS,/tmp/media/BaseOS --repo BaseOS \ + --repofrompath Minimal,/tmp/media/Minimal --repo Minimal + Added BaseOS repo from /tmp/media/BaseOS + Added Minimal repo from /tmp/media/Minimal + BaseOS 3.7 MB/s | 3.8 kB 00:00 + Minimal 3.7 MB/s | 3.8 kB 00:00 + package: basesystem-11-5.el8.noarch from BaseOS + unresolved deps: + setup + package: dump-1:0.4-0.36.b46.el8.x86_64 from BaseOS + unresolved deps: + setup + package: filesystem-3.8-6.el8.x86_64 from BaseOS + unresolved deps: + setup + package: initscripts-10.00.15-1.el8.x86_64 from BaseOS + unresolved deps: + setup + package: rpcbind-1.2.5-8.el8.x86_64 from BaseOS + unresolved deps: + setup + package: shadow-utils-2:4.6-14.el8.x86_64 from BaseOS + unresolved deps: + setup + Error: Repoclosure ended with unresolved dependencies. + ``` + +{% include 'qa_testcase_bottom.md' %} diff --git a/docs/documentation/QA/Testcase_Media_USB_dd.md b/docs/documentation/QA/Testcase_Media_USB_dd.md new file mode 100644 index 0000000..aede135 --- /dev/null +++ b/docs/documentation/QA/Testcase_Media_USB_dd.md @@ -0,0 +1,61 @@ +--- +title: QA:Testcase Media USB dd +author: Trevor Cooper +revision_date: 2022-05-18 +rc: + prod: Rocky Linux + ver: 8 + level: Final +--- + +!!! info "Associated release criterion" + This test case is associated with the [Release_Criteria#initialization-requirements](9_release_criteria.md#initialization-requirements) release criterion. If you are doing release validation testing, a failure of this test case may be a breach of that release criterion. + +## Description +This verifies that Rocky Linux ISO image can be written to USB media using `dd` command, and the resulting USB media successfully boots to the Anaconda Installer. + +!!! error "DATA LOSS" + Any data on the USB stick used for this test is likely to be destroyed. Please do not use a stick whose contents you need to keep. + +## Setup + +1. Provide a USB media device that is larger than the ISO image you wish to test and that it can be completely erased. +2. Provide a Linux (or other *nix system) that has the `dd` command available and an unoccupied USB port. +3. Download the Rocky Linux ISO image you wish to test onto the test system. + - Example command:
`curl -LOR https://dl.rockylinux.org/pub/rocky/8/isos/x86_64/Rocky-x86_64-boot.iso` +4. Download the `CHECKSUM` file that goes with the Rocky Linux ISO image that you wish to test. + - Example command:
`curl -LOR https://dl.rockylinux.org/pub/rocky/8/isos/x86_64/CHECKSUM` +5. Download the `CHECKSUM.sig` file that does with the `CHECKSUM` file. + - Example command:
`curl -LOR https://dl.rockylinux.org/pub/rocky/8/isos/x86_64/CHECKSUM.sig` +6. Download the Rocky Release Engineering GPG key. + - Example command:
`curl -LOR https://dl.rockylinux.org/pub/rocky/RPM-GPG-KEY-rockyofficial` + +## How to test + +1. Import the Rocky Release Engineering GPG key. + - Example command:
`gpg --import RPM-GPG-KEY-rockyofficial` +2. Verify the signature of the CHECKSUM file. + - Example command:
`gpg --verify-file CHECKSUM.sig` +3. Verify the CHECKSUM of the Rocky Linux ISO... + - Example command:
`shasum -a 256 --ignore-missing -c CHECKSUM` +4. Write the Rocky Linux ISO to the USB media using `dd`... + - Example command:
`dd if=Rocky-8.5-x86_64-boot.iso of=/dev/sdX bs=16M status=progress oflag=direct`
...where you replace `sdX` with the device identifier of your USB media.
**This will destroy all data on the disk.** +5. Boot the test system with the USB media. +6. In the boot menu select the appropriate option to boot the installer. +7. **[OPTIONAL]** Proceed with installation on the test system.
**Depending on installer choices this MAY destroy all the data on the test system.** + +## Expected Results + +1. The gpg signature on the `CHECKSUM` file is valid. +2. The `CHECKSUM` of the Rocky Linux ISO is valid. +3. The Rocky Linux ISO is written to the USB stick without errors. +4. The USB stick boots without errors. +5. The Anaconda Installer starts without errors. + +!!! error "DATA LOSS" + If you choose to complete the installation of the test system any/all data on the system may be lost. Please do not install on a system whose contents you need to keep. + +**[OPTIONALLY]**
+6. The installation finishes successfully and, if the minimal or DVD ISO were used, the package repository on the USB stick (not a network based repository) was used for the installation. + +{% include 'qa_testcase_bottom.md' %} diff --git a/docs/documentation/QA/Testcase_Minimal_Installation.md b/docs/documentation/QA/Testcase_Minimal_Installation.md new file mode 100644 index 0000000..705dee6 --- /dev/null +++ b/docs/documentation/QA/Testcase_Minimal_Installation.md @@ -0,0 +1,37 @@ +--- +title: QA:Testcase Minimal Installation +author: Al Bowles +revision_date: 2022-07-08 +rc: + prod: Rocky Linux + ver: + - 8 + - 9 + level: Final +--- + +!!! info "Associated release criterion" + This test case is associated with the [Release_Criteria#Minimal Installation](9_release_criteria.md#minimal-installation) release criterion. If you are doing release validation testing, a failure of this test case may be a breach of that release criterion. + +## Description +This test case verifies that a networked minimal installation is able to install the 'Minimal' package set. The installation should not require use of local packages to complete. + +{% include 'qa_data_loss_warning.md' %} + +## Setup +{% include 'qa_setup_boot_to_media.md' %} + +## How to test +1. From the Installation Source spoke, configure a remote repository source from [MirrorManager](https://mirrors.rockylinux.org) appropriate to the architecture under test. +1. From the Software Selection spoke, select the Minimal package set. +1. Complete the installation using desired parameters. + +## Expected Results +1. The installation should complete and boot successfully. + +## Testing in openQA +The following openQA test suites satisfy this release criteria: + +- `install_minimal` + +{% include 'qa_testcase_bottom.md' %} diff --git a/docs/documentation/QA/Testcase_Network_Attached_Storage.md b/docs/documentation/QA/Testcase_Network_Attached_Storage.md new file mode 100644 index 0000000..268c4e2 --- /dev/null +++ b/docs/documentation/QA/Testcase_Network_Attached_Storage.md @@ -0,0 +1,37 @@ +--- +title: QA:Testcase Network Attached Storage +author: Al Bowles +revision_date: 2022-07-08 +rc: + prod: Rocky Linux + ver: + - 8 + - 9 + level: Final +--- + +!!! info "Associated release criterion" + This test case is associated with the [Release_Criteria#Network Attached Storage](9_release_criteria.md#nas-network-attached-storage) release criterion. If you are doing release validation testing, a failure of this test case may be a breach of that release criterion. + +## Description +The installer must be able to detect and install to supported NAS devices (if possible and supported by the kernel). + +## Setup +1. Add steps for setup for this Testcase. + +## How to test +### NFS +install nfs-utils +sudo mount -t nfs nfs-server:/nfs/path /mnt +then a created a file echo 1 > /mnt/1 +verified it and permissions ls /mnt; cat /mnt/1 +then deleted it rm /mnt/1 +then unmounted sudo umount /mnt + +### iSCSI + +## Expected Results +1. This is what you should see/verify. +2. You should also see/verify this. + +{% include 'qa_testcase_bottom.md' %} diff --git a/docs/documentation/QA/Testcase_Packages_Installer_Sources.md b/docs/documentation/QA/Testcase_Packages_Installer_Sources.md new file mode 100644 index 0000000..35c3f5e --- /dev/null +++ b/docs/documentation/QA/Testcase_Packages_Installer_Sources.md @@ -0,0 +1,53 @@ +--- +title: QA:Testcase Packages and Installer Sources +author: Al Bowles +revision_date: 2022-07-08 +rc: + prod: Rocky Linux + ver: 8 + level: Final +--- + +!!! info "Associated release criterion" + This test case is associated with the [Release_Criteria#Packages and Installer Sources](9_release_criteria.md#packages-installer-sources) release criterion. If you are doing release validation testing, a failure of this test case may be a breach of that release criterion. + +## Description +This test case verifies that the installer can successfully install any of the supported package sets via any of the supported installer sources. + +The following package sets are supported for installs from local media: + +- server +- minimal + +The following package sets are only available from remote sources and require a network connection: + +- workstation +- graphical-server +- virtualization-host + +{% include 'qa_data_loss_warning.md' %} + +## Setup +{% include 'qa_setup_boot_to_media.md' %} + +## How to test +1. For local package installations it is not necessary to enable networking or specify a mirror. +1. For package installation from remote sources: + 1. From the Network and Hostname spoke, enable networking. + 1. From the Installation Source spoke, configure a remote software source, supplying an appropriate [mirror](https://mirrors.rockylinux.org) for the version and architecture under test. +1. Complete the installer and wait for the machine to reboot. + +## Expected Results +1. The installation should complete and boot successfully. +1. If a graphical package set was specified, the system should boot to a graphical login screen. + +## Testing in openQA +The following openQA test suites satisfy this release criteria, provided they pass the `_do_install_reboot` module at a minimum: + +- `install_packageset_server` +- `install_packageset_minimal` +- `install_packageset_workstation` +- `install_packageset_graphical-server` +- `install_packageset_virtualization-host` + +{% include 'qa_testcase_bottom.md' %} diff --git a/docs/documentation/QA/Testcase_Packages_No_Insights.md b/docs/documentation/QA/Testcase_Packages_No_Insights.md new file mode 100644 index 0000000..f4f115d --- /dev/null +++ b/docs/documentation/QA/Testcase_Packages_No_Insights.md @@ -0,0 +1,96 @@ +--- +title: QA:Testcase Packages No Insights +author: Trevor Cooper +revision_date: 2022-05-18 +rc: + prod: Rocky Linux + ver: 8 + level: Final +--- + +!!! info "Associated release criterion" + This test case is associated with the [Release_Criteria#repositories-must-match-upstream](9_release_criteria.md#repositories-must-match-upstream) release criterion. If you are doing release validation testing, a failure of this test case may be a breach of that release criterion. + +## Description +This test will verify that `insights-client` package is not declared be installed as part of a package group. + +## Setup +1. Obtain access to an environment with the `dnf` command. +2. Download the ISO to be tested to that machine. + +## How to test +1. Mount the ISO to be tested locally. +2. Determine the path to the `comps` file(s) on the ISO. +3. Verify that `insights-client` is not declared to be installed automatically. + - Example 1:
`find /media -name "*comps*.xml" -exec grep -H "insights-client" '{}' \;` + - Example 2:
`dnf --refresh --repofrompath BaseOS,/media/BaseOS --repo BaseOS --repofrompath AppStream,/media/AppStream --repo AppStream groupinfo base | grep -E ":|insights"` +4. Unmount the ISO. + +## Expected Results +1. `insights-client` is not declared to be installed by default. + +

Sample Output

+ +=== "Success" + + + !!! info "UPDATE SAMPLE" + NOTE: This example needs to be refreshed when the 8.6 ISO has been produced. As seen in the Failure section below the `Rocky-8.5-x86_64-dvd1.iso` includes the `insights-client` as part of the `base` group. The package should be included on the DVD ISO but should **not** be installed automatically. + + ``` + $ sudo mount -o loop Rocky-8.5-aarch64-minimal.iso /media + mount: /media: WARNING: device write-protected, mounted read-only. + + $ dnf --refresh --repofrompath BaseOS,/media/BaseOS --repo BaseOS --repofrompath Minimal,/media/Minimal --repo Minimal search insights-client + Added BaseOS repo from /media/BaseOS + Added Minimal repo from /media/Minimal + BaseOS 3.8 MB/s | 3.9 kB 00:00 + Minimal 3.7 MB/s | 3.8 kB 00:00 + No matches found. + + $ find /media -name "*comps*.xml" -exec grep -H "insights-client" '{}' \; + + $ dnf --refresh --repofrompath BaseOS,/media/BaseOS --repo BaseOS --repofrompath Minimal,/media/Minimal --repo Minimal groupinfo base | grep -E ":|insights" + BaseOS 3.8 MB/s | 3.9 kB 00:00 + Minimal 3.7 MB/s | 3.8 kB 00:00 + Group: Base + Description: The standard installation of Rocky Linux. + Mandatory Packages: + Default Packages: + Optional Packages: + + $ sudo umount /media + ``` + +=== "Failure" + + ``` + $ sudo mount -o loop Rocky-8.5-x86_64-dvd1.iso /media + mount: /media: WARNING: device write-protected, mounted read-only. + + $ dnf --refresh --repofrompath BaseOS,/media/BaseOS --repo BaseOS --repofrompath AppStream,/media/AppStream --repo AppStream search insights-client + Added BaseOS repo from /media/BaseOS + Added AppStream repo from /media/AppStream + BaseOS 3.8 MB/s | 3.9 kB 00:00 + AppStream 4.2 MB/s | 4.3 kB 00:00 + ================================= Name Exactly Matched: insights-client ================================== + insights-client.noarch : Uploads Insights information to Red Hat on a periodic basis + + $ find /media -name "*comps*.xml" -exec grep -H "insights-client" '{}' \; + /media/AppStream/repodata/a6742e1300e1c786af91656b152d3b98bb7aff598e650509381417970e1f1b7e-comps-AppStream.x86_64.xml: insights-client + /media/AppStream/repodata/a6742e1300e1c786af91656b152d3b98bb7aff598e650509381417970e1f1b7e-comps-AppStream.x86_64.xml: insights-client + + $ dnf --refresh --repofrompath BaseOS,/media/BaseOS --repo BaseOS --repofrompath AppStream,/media/AppStream --repo AppStream groupinfo base | grep -E ":|insights" + BaseOS 3.8 MB/s | 3.9 kB 00:00 + AppStream 4.2 MB/s | 4.3 kB 00:00 + Group: Base + Description: The standard installation of Rocky Linux. + Mandatory Packages: + Default Packages: + insights-client + Optional Packages: + + $ sudo umount /media + ``` + +{% include 'qa_testcase_bottom.md' %} diff --git a/docs/documentation/QA/Testcase_Packages_No_RHSM.md b/docs/documentation/QA/Testcase_Packages_No_RHSM.md new file mode 100644 index 0000000..b3dc123 --- /dev/null +++ b/docs/documentation/QA/Testcase_Packages_No_RHSM.md @@ -0,0 +1,63 @@ +--- +title: QA:Testcase Packages No RHSM +author: Trevor Cooper +revision_date: 2022-05-18 +rc: + prod: Rocky Linux + ver: 8 + level: Final +--- + +!!! info "Associated release criterion" + This test case is associated with the [Release_Criteria#repositories-must-match-upstream](9_release_criteria.md#repositories-must-match-upstream) release criterion. If you are doing release validation testing, a failure of this test case may be a breach of that release criterion. + +## Description +This test will verify that packages that are availble from upstream do not have hard requirements on `subscription-manager` (RHSM). + +## Setup +1. Obtain access to an environment with the `dnf` command. +2. Download the ISO to be tested to that machine. + +## How to test +1. Mount the ISO to be tested locally. +2. Obtain a list of packages that have `Requires:` for `subscription-manager` + - Example:
`package_list=($(dnf --refresh repoquery --repofrompath BaseOS,/media/BaseOS --repo BaseOS --repofrompath AppStream,/media/AppStream --repo AppStream --whatrequires subscription-manager 2>/dev/null| grep el8))` +3. Download the packages with explicity `Requires:` for `subscription-manager` + - Example:
`dnf --repofrompath BaseOS,/media/BaseOS --repo BaseOS --repofrompath AppStream,/media/AppStream --repo AppStream download "${package_list[@]}"` +4. Obtain the `SOURCEPKG` definition for the above packages + - Example:
`rpm -q --queryformat="%{NAME}|%{SOURCERPM}\n" subscription-manager*.rpm | column -s\| -t` +4. Unmount the ISO. + +## Expected Results +1. No packages have an explicit requirement for `subscription-manager`. + + +

Sample Output

+ +=== "Success" + + ``` + $ sudo mount -o loop Rocky-8.5-aarch64-minimal.iso /media + mount: /media: WARNING: device write-protected, mounted read-only. + + $ package_list=($(dnf --refresh repoquery --repofrompath BaseOS,/media/BaseOS --repo BaseOS --repofrompath AppStream,/media/AppStream --repo AppStream --whatrequires subscription-manager 2>/dev/null| grep el8)) + + $ dnf --repofrompath BaseOS,/media/BaseOS --repo BaseOS --repofrompath AppStream,/media/AppStream --repo AppStream download "${package_list[@]}" + Added BaseOS repo from /media/BaseOS + Added AppStream repo from /media/AppStream + Last metadata expiration check: 0:00:25 ago on Sun 24 Apr 2022 10:57:13 PM UTC. + + $ rpm -q --queryformat="%{NAME}|%{SOURCERPM}\n" subscription-manager*.rpm | column -s\| -t + subscription-manager-cockpit subscription-manager-1.28.21-3.el8.src.rpm + subscription-manager-migration subscription-manager-1.28.21-3.el8.src.rpm + subscription-manager-plugin-ostree subscription-manager-1.28.21-3.el8.src.rpm + + $ sudo umount /media + ``` + +=== "Failure" + + TBD + + +{% include 'qa_testcase_bottom.md' %} diff --git a/docs/documentation/QA/Testcase_Post_Application_Functionality.md b/docs/documentation/QA/Testcase_Post_Application_Functionality.md new file mode 100644 index 0000000..d327272 --- /dev/null +++ b/docs/documentation/QA/Testcase_Post_Application_Functionality.md @@ -0,0 +1,55 @@ +--- +title: QA:Testcase Application Functionality +author: Lukas Magauer +revision_date: 2022-05-31 +rc: + prod: Rocky Linux + ver: + - 8 + - 9 + level: Final +--- + +!!! info "Release relevance" + This Testcase applies the following versions of {{ rc.prod }}: {% for version in rc.ver %}{{ version }}{% if not loop.last %}, {% endif %}{% endfor %} + +!!! info "Associated release criterion" + This test case is associated with the [Release_Criteria#default-application-functionality-desktop-only](9_release_criteria.md#default-application-functionality-desktop-only) release criterion. If you are doing release validation testing, a failure of this test case may be a breach of that release criterion. + +!!! error "REFERENCED RELEASE CRITERIA IS OVERLY GENERAL AND UNTESTABLE" + The associated release criteria, [Release_Criteria#default-application-functionality-desktop-only](9_release_criteria.md#default-application-functionality-desktop-only), for this test case is overly general and **must** be modified to specific enough to be testable. + +## Description + +This testcase handles all applications, considered as core applications of the desktop environment GNOME or user facing commandline applications. + +The following tasks apply in general to all of the following applications: + +- Firefox +- Files (Nautilus) +- GNOME Software +- (Image Viewer) +- (Document Viewer) +- Gedit (Text Editor) +- Archive Manager +- GNOME Terminal (Terminal Emulator) +- Problem Reporter +- Help Viewer +- System Settings +- vim (Console Text Editor) + +## Setup + +Obtain access to a suitable system with either a Workstation or a Graphical Server installation. + +## How to test + +1. Check that the application starts without any errors +2. Further check that the context menus for the correct function +3. Open files to test the functionality of the individual applications + +## Expected Results + +Make sure that the individual applications behave as they should. + +{% include 'qa_testcase_bottom.md' %} diff --git a/docs/documentation/QA/Testcase_Post_Artwork_and_Assets.md b/docs/documentation/QA/Testcase_Post_Artwork_and_Assets.md new file mode 100644 index 0000000..3e3ac65 --- /dev/null +++ b/docs/documentation/QA/Testcase_Post_Artwork_and_Assets.md @@ -0,0 +1,44 @@ +--- +title: QA:Testcase Artwork and Assets +author: Lukas Magauer +revision_date: 2022-05-31 +rc: + prod: Rocky Linux + ver: + - 8 + - 9 + level: Final +--- + +!!! info "Release relevance" + This Testcase applies the following versions of {{ rc.prod }}: {% for version in rc.ver %}{{ version }}{% if not loop.last %}, {% endif %}{% endfor %} + +!!! info "Associated release criterion" + This test case is associated with the [Release_Criteria#artwork-and-assets-server-and-desktop](9_release_criteria.md#artwork-and-assets-server-and-desktop) release criterion. If you are doing release validation testing, a failure of this test case may be a breach of that release criterion. + +## Description + +There are several brand artworks and assets throughout the whole OS, this testcase will take care of checking, that these are actually in place, and don't produce any UI errors. This is exclusively for installations with the default desktop environment GDM and GNOME. + +## Setup + +1. Acquire access to either a baremetal machine or a VM host, to install a new machine +2. Prepare appropriate media for the selected ISO to be tested. + - Example: [QA:Testcase Media USB dd](Testcase_Media_USB_dd.md) + +## How to test + +1. While booting the image check, that the correct logo is visible in the loading screen before Anaconda comes up +2. Look at the Anaconda images in the [rocky-logos repo](https://github.com/rocky-linux/rocky-logos/tree/r8-fedora/anaconda) and check if all assets are correctly applied in Anaconda (they will generally be visible right away while going through the install process) +3. Install the system with either the Workstation install set or Graphical Server +4. While the OS does its first boot, check that the correct logo is visible in the loading screen before the boot login screen shows up +5. Check the logo and background of the boot login screen +6. After the login check the desktop background and further all available options in the settings menu for the desktop background +7. Lock the screen and check the background visible in the flyover +8. At last check the logo and background of the login screen + +## Expected Results + +The tests during the process could be successfully finished. + +{% include 'qa_testcase_bottom.md' %} diff --git a/docs/documentation/QA/Testcase_Post_GNOME_UI_Functionality.md b/docs/documentation/QA/Testcase_Post_GNOME_UI_Functionality.md new file mode 100644 index 0000000..13ac9a4 --- /dev/null +++ b/docs/documentation/QA/Testcase_Post_GNOME_UI_Functionality.md @@ -0,0 +1,45 @@ +--- +title: QA:Testcase GNOME UI Functionality +author: Lukas Magauer +revision_date: 2022-05-31 +rc: + prod: Rocky Linux + ver: + - 8 + - 9 + level: Final +--- + +!!! info "Release relevance" + This Testcase applies the following versions of {{ rc.prod }}: {% for version in rc.ver %}{{ version }}{% if not loop.last %}, {% endif %}{% endfor %} + +!!! error "REFERENCED RELEASE CRITERIA IS OVERLY GENERAL AND UNTESTABLE" + The associated release criteria, [Release_Criteria#default-panel-functionality-desktop-only](9_release_criteria.md#default-panel-functionality-desktop-only), for this test case is overly general and **must** be modified to specific enough to be testable. + +!!! info "Associated release criterion" + This test case is associated with the [Release_Criteria#default-panel-functionality-desktop-only](9_release_criteria.md#default-panel-functionality-desktop-only) release criterion. If you are doing release validation testing, a failure of this test case may be a breach of that release criterion. + +## Description + +This test collection takes care of the correct functionality of the GNOME UI. + +## Setup + +Obtain access to a suitable system with either a Workstation or a Graphical Server installation. + +## How to test + +1. Login to the Rocky Machine via the UI +2. Navigate through the GNOME UI + +## Expected Results + +1. After the login you should have landed on the desktop with the background and the top bar of GNOME visible +2. Clicking the the Activities button in the upper right should bring up the overview +3. Further there should be the favourite applications ribbon on the left +4. And after clicking the 9-dot-icon all applications should appear +5. Back on the desktop check the function of the system and clock panel on the upper right and middle + +It is also a good measure to do some more basic clicking through the GNOME UI, like opening applications, interacting with the application headerbar, moving applications to different desktops or changing settings in the System settings. + +{% include 'qa_testcase_bottom.md' %} diff --git a/docs/documentation/QA/Testcase_Post_Identity_Management.md b/docs/documentation/QA/Testcase_Post_Identity_Management.md new file mode 100644 index 0000000..eebe5a2 --- /dev/null +++ b/docs/documentation/QA/Testcase_Post_Identity_Management.md @@ -0,0 +1,68 @@ +--- +title: QA:Testcase Identity Management +author: Lukas Magauer +revision_date: 2022-10-08 +rc: + prod: Rocky Linux + ver: + - 8 + - 9 + level: Final +--- + +!!! info "Release relevance" + This Testcase applies the following versions of {{ rc.prod }}: {% for version in rc.ver %}{{ version }}{% if not loop.last %}, {% endif %}{% endfor %} + +!!! info "Associated release criterion" + This test case is associated with the [Release_Criteria#packages-and-module-installation](9_release_criteria.md#packages-and-module-installation) release criterion. If you are doing release validation testing, a failure of this test case may be a breach of that release criterion. + +## Description + +Setting up a IdM system (FreeIPA) and using it's functionality leverages not also a lot of the packages in the official repos, it also tests quite a lot of used functions a corporate environment. This installatation will host it's own dns server for more generic testing without relying on the individual infrastructure of the environment. + +## Requirements + +- A freshly provisioned system (no other functions are allowed on this system except running the IdM services) +- IPv4 network with unmanaged domain name (installer will check for dns servers) and unmanaged reverse dns network (in my case here 10.30.30.0/24 and ipa1.network) +- In the case of this writeup the external dns server has the domain `example.com`, this has to have a entry for `r8-ipa1-dev.example.com` (this could also be replaced by a entry in the `/etc/hosts` file if no external dns server should be involved) + +## Setup + +1. `dnf module enable idm:DL1` +2. `dnf module install idm:DL1/dns` +3. `ipa-server-install` + + - Do you want to configure integrated DNS (BIND)? [no]: yes + - Server host name [r8-ipa1-dev.example.com]: r8-ipa1-dev.example.com + - Please confirm the domain name [ipa1.network]: ipa1.network + - Please provide a realm name [IPA1.NETWORK]: IPA1.NETWORK + - Directory Manager password: `` + Password (confirm): `` + - IPA admin password: `` + Password (confirm): `` + - Please provide the IP address to be used for this host name: 10.30.30.1 + - Enter an additional IP address, or press Enter to skip: `leave empty` + - Do you want to configure DNS forwarders? [yes]: yes + - Do you want to configure these servers as DNS forwarders? [yes]: yes + - Enter an IP address for a DNS forwarder, or press Enter to skip: `leave empty` + - Do you want to search for missing reverse zones? [yes]: yes + - NetBIOS domain name [IPA1]: IPA1 + - Do you want to configure chrony with NTP server or pool address? [no]: yes + - Enter NTP source server addresses separated by comma, or press Enter to skip: `leave empty` + - Enter a NTP source pool address, or press Enter to skip: pool.ntp.org + - Continue to configure the system with these values? [no]: yes + +4. `firewall-cmd --add-service={freeipa-4,dns} --permanent` +5. `firewall-cmd --add-service={freeipa-4,dns}` + +## How to test + +1. Make sure Kerberos works, by running `kinit admin` and `klist` +2. Make sure the webfrontend is reachable and login works +3. Furthermore you can also attach another system (DNS + connecting via SSSD) + +## Expected Results + +After installation all services should be available and work correctly. + +{% include 'qa_testcase_bottom.md' %} diff --git a/docs/documentation/QA/Testcase_Post_Keyboard_Layout.md b/docs/documentation/QA/Testcase_Post_Keyboard_Layout.md new file mode 100644 index 0000000..52906ac --- /dev/null +++ b/docs/documentation/QA/Testcase_Post_Keyboard_Layout.md @@ -0,0 +1,60 @@ +--- +title: QA:Testcase Keyboard Layout +author: Lukas Magauer +revision_date: 2022-05-31 +rc: + prod: Rocky Linux + ver: + - 8 + - 9 + level: Final +--- + +!!! info "Release relevance" + This Testcase applies the following versions of {{ rc.prod }}: {% for version in rc.ver %}{{ version }}{% if not loop.last %}, {% endif %}{% endfor %} + +!!! info "Associated release criterion" + This test case is associated with the [Release_Criteria#keyboard-layout](9_release_criteria.md#keyboard-layout) release criterion. If you are doing release validation testing, a failure of this test case may be a breach of that release criterion. + +## Description + +As there are a lot of different keyboard layouts available, it is necessary to test if the keyboard functionality works without any issues throughout the system. + +## Setup + +- Obtain access to a few different system configurations, especially with and without UI, and not to forget with disk encryption. +- Acquire access to either a baremetal machine or a VM host, to install a new machine + - Prepare appropriate media for the selected ISO to be tested. + - Example: [QA:Testcase Media USB dd](Testcase_Media_USB_dd.md) + +## How to test + +### Installer + +1. Bootup the installer +2. Choose a language +3. Make sure that the keyboard layout got chosen correctly corresponding to the language setting +4. Change the keyboard layout if needed to test +5. Enter text all over Anaconda to make sure the keyboard layout works correctly with the chosen keyboard layout + +### Disk Encryption + +1. Setup a system with disk encryption +2. Check that the password for the disk encryption works on bootup with graphical UI +3. Check that the password for the disk encryption works on bootup with text mode + +### Text mode + +Check that the chosen keyboard layout works correctly on text mode. + +### GNOME and Application + +1. Check the login, that the keyboard layout works correctly on the graphical UI login screen +2. Also check that the GNOME UI works correctly with the chosen keyboard layout +3. And finally check some applications, that the keyboard works as expected + +## Expected Results + +The tests during the process could be successfully finished. + +{% include 'qa_testcase_bottom.md' %} diff --git a/docs/documentation/QA/Testcase_Post_Module_Streams.md b/docs/documentation/QA/Testcase_Post_Module_Streams.md new file mode 100644 index 0000000..3cbe1e2 --- /dev/null +++ b/docs/documentation/QA/Testcase_Post_Module_Streams.md @@ -0,0 +1,49 @@ +--- +title: QA:Testcase Module Streams +author: Lukas Magauer +revision_date: 2022-05-31 +rc: + prod: Rocky Linux + ver: + - 8 + - 9 + level: Final +--- + +!!! info "Release relevance" + This Testcase applies the following versions of {{ rc.prod }}: {% for version in rc.ver %}{{ version }}{% if not loop.last %}, {% endif %}{% endfor %} + +!!! info "Associated release criterion" + This test case is associated with the [Release_Criteria#packages-and-module-installation](9_release_criteria.md#packages-and-module-installation) release criterion. If you are doing release validation testing, a failure of this test case may be a breach of that release criterion. + +## Description + +This test case takes care of testing the module streams, that they are all installable, all available and working as expected. + +## Setup + +For this tests you will need to install every module stream on its own, so it's the best to setup a new system which gets snapshoted after the initial setup. After that it can be rolled back for every module install. + +It's enough to setup a system with the Minimal Install group. + +## How to test + +1. Login to the machine +2. Get a list of all module streams and compare it to the [stream list from RHEL](https://access.redhat.com/support/policy/updates/rhel-app-streams-life-cycle) and to the source in [Git source repo](https://git.rockylinux.org/rocky/rocky-module-defaults) +3. The easiest way to test all streams is to install the package groups in the individual streams, i.e. for postgresql: + +```bash +dnf module install postgresql +dnf module install postgresql:13 +dnf module install postgresql:13/client +``` + +Repeat step 3 as often as module streams and package groups are available. + +This could be automated with i.e. Ansible to do all the `install -> rollback -> install -> rollback -> ...` and emiting the output via Ansible. + +## Expected Results + +All module streams should be available and there shouldn't be any errors while installing any of the package groups of the individual streams. (some of the installs will show warnings though because they are incompatible with other streams) + +{% include 'qa_testcase_bottom.md' %} diff --git a/docs/documentation/QA/Testcase_Post_Multimonitor_Setup.md b/docs/documentation/QA/Testcase_Post_Multimonitor_Setup.md new file mode 100644 index 0000000..28cbd04 --- /dev/null +++ b/docs/documentation/QA/Testcase_Post_Multimonitor_Setup.md @@ -0,0 +1,36 @@ +--- +title: QA:Testcase Multimonitor Setup +author: Lukas Magauer +revision_date: 2022-05-31 +rc: + prod: Rocky Linux + ver: + - 8 + - 9 + level: Final +--- + +!!! info "Release relevance" + This Testcase applies the following versions of {{ rc.prod }}: {% for version in rc.ver %}{{ version }}{% if not loop.last %}, {% endif %}{% endfor %} + +!!! info "Associated release criterion" + This test case is associated with the [Release_Criteria#dual-monitor-setup-desktop-only](9_release_criteria.md#dual-monitor-setup-desktop-only) release criterion. If you are doing release validation testing, a failure of this test case may be a breach of that release criterion. + +## Description + +This test covers the check if GNOME behaves as it should in multi-monitor setups. + +## Setup + +You will need either a machine which can be reinstalled with multiple screens, or a virtualization software which is capable of providing multiple screens (like VMware Workstation ([Pro](https://www.vmware.com/products/workstation-pro/workstation-pro-evaluation.html) or [Player](https://www.vmware.com/products/workstation-player/workstation-player-evaluation.html)) or [VMware Fusion](https://www.vmware.com/products/fusion/fusion-evaluation.html), but there is also a [hack](https://communities.vmware.com/t5/VMware-vSphere-Discussions/ESXi-6-7-Multiple-Monitors-for-VMs/td-p/2748906) for [VMware ESXi](https://customerconnect.vmware.com/en/web/vmware/evalcenter?p=vsphere-eval-7)) + +## How to test + +1. Run installer with multiple screens connected and install with either the Workstation or Graphical Server group +2. Login to the machine after the finished install + +## Expected Results + +There shouldn't be any graphical glitches, or scaling issues through the install and the usage. + +{% include 'qa_testcase_bottom.md' %} diff --git a/docs/documentation/QA/Testcase_Post_Package_installs.md b/docs/documentation/QA/Testcase_Post_Package_installs.md new file mode 100644 index 0000000..fafe2bc --- /dev/null +++ b/docs/documentation/QA/Testcase_Post_Package_installs.md @@ -0,0 +1,55 @@ +--- +title: QA:Testcase Basic Package installs +author: Lukas Magauer +revision_date: 2022-05-31 +rc: + prod: Rocky Linux + ver: + - 8 + - 9 + level: Final +--- + +!!! info "Release relevance" + This Testcase applies the following versions of {{ rc.prod }}: {% for version in rc.ver %}{{ version }}{% if not loop.last %}, {% endif %}{% endfor %} + +!!! info "Associated release criterion" + This test case is associated with the [Release_Criteria#packages-and-module-installation](9_release_criteria.md#packages-and-module-installation) release criterion. If you are doing release validation testing, a failure of this test case may be a breach of that release criterion. + +!!! error "REFERENCED RELEASE CRITERIA IS OVERLY GENERAL AND UNTESTABLE" + The associated release criteria, [Release_Criteria#packages-and-module-installation](9_release_criteria.md#packages-and-module-installation), for this test case is overly general and **must** be modified to specific enough to be testable. + +## Description + +Installing several packages should work without any issues. + +Please also test these usecases (it's basically the fun of learning to install software, it's even good if it's done differently each other time): + +- httpd +- httpd with php and ssl +- nginx +- nginx with php and ssl +- mysql-server +- mysql-server with secure setup +- mariadb-server +- postgresql-server +- postgresql-server with secure setup +- compiling packages with: + - cmake + - g++ +- ipa-server +- ipa-server with dns + +## Setup + +Obtain access to a suitable system where any of the tested packages can be installed without any issues. + +## How to test + +1. Install a list of packages or usecases + +## Expected Results + +Installs work without any issues. + +{% include 'qa_testcase_bottom.md' %} diff --git a/docs/documentation/QA/Testcase_Post_SELinux_Errors_Desktop.md b/docs/documentation/QA/Testcase_Post_SELinux_Errors_Desktop.md new file mode 100644 index 0000000..8497c3e --- /dev/null +++ b/docs/documentation/QA/Testcase_Post_SELinux_Errors_Desktop.md @@ -0,0 +1,38 @@ +--- +title: QA:Testcase SELinux Errors on Desktop clients +author: Lukas Magauer +revision_date: 2022-05-31 +rc: + prod: Rocky Linux + ver: + - 8 + - 9 + level: Final +--- + +!!! info "Release relevance" + This Testcase applies the following versions of {{ rc.prod }}: {% for version in rc.ver %}{{ version }}{% if not loop.last %}, {% endif %}{% endfor %} + +!!! info "Associated release criterion" + This test case is associated with the [Release_Criteria#selinux-and-crash-notifications-desktop-only](9_release_criteria.md#selinux-and-crash-notifications-desktop-only) release criterion. If you are doing release validation testing, a failure of this test case may be a breach of that release criterion. + +## Description + +Basically running a Workstation or Graphical Server install for a longer amount of time, while using it and then checking if there were any SELinux audit messages. + +## Setup + +Obtain access to a suitable system with either a Workstation or a Graphical Server installation. + +## How to test + +1. Setup new machine or get access to a installed machine +2. Click through the system and various applications, to mimic user behavior +3. Leave the system running for a few more minutes, if possible hours + +## Expected Results + +1. Open the SETroubleshoot Application and invoke the error summarization. +2. There must not be shown any errors / denials + +{% include 'qa_testcase_bottom.md' %} diff --git a/docs/documentation/QA/Testcase_Post_SELinux_Errors_Server.md b/docs/documentation/QA/Testcase_Post_SELinux_Errors_Server.md new file mode 100644 index 0000000..d2a7c87 --- /dev/null +++ b/docs/documentation/QA/Testcase_Post_SELinux_Errors_Server.md @@ -0,0 +1,40 @@ +--- +title: QA:Testcase SELinux Errors on Server installations +author: Lukas Magauer +revision_date: 2022-05-31 +rc: + prod: Rocky Linux + ver: + - 8 + - 9 + level: Final +--- + +!!! info "Release relevance" + This Testcase applies the following versions of {{ rc.prod }}: {% for version in rc.ver %}{{ version }}{% if not loop.last %}, {% endif %}{% endfor %} + +!!! info "Associated release criterion" + This test case is associated with the [Release_Criteria#selinux-errors-server](9_release_criteria.md#selinux-errors-server) release criterion. If you are doing release validation testing, a failure of this test case may be a breach of that release criterion. + +## Description + +Basically running a text mode installation for a longer amount of time, while using it and then checking if there were any SELinux audit messages. + +## Setup + +Obtain access to a suitable system with one of the text mode installation base groups. + +It might also be beneficial to run this test with other than the core installation, but that is a long term test and a bit out of scope of this test. + +## How to test + +1. Setup new machine or get access to a installed machine +2. As this test is mostly about the stability of the core system it is mostly only needed to let the system run for a few minutes, if possible hours + +## Expected Results + +1. Install `sealert` with `dnf install setroubleshoot-server` +2. Run `sealert -a /var/log/audit/audit.log` +3. There must not be shown any errors / denials + +{% include 'qa_testcase_bottom.md' %} diff --git a/docs/documentation/QA/Testcase_Post_System_Services.md b/docs/documentation/QA/Testcase_Post_System_Services.md new file mode 100644 index 0000000..366ba87 --- /dev/null +++ b/docs/documentation/QA/Testcase_Post_System_Services.md @@ -0,0 +1,41 @@ +--- +title: QA:Testcase System Services +author: Lukas Magauer +revision_date: 2022-05-31 +rc: + prod: Rocky Linux + ver: + - 8 + - 9 + level: Final +--- + +!!! info "Release relevance" + This Testcase applies the following versions of {{ rc.prod }}: {% for version in rc.ver %}{{ version }}{% if not loop.last %}, {% endif %}{% endfor %} + +!!! info "Associated release criterion" + This test case is associated with the [Release_Criteria#system-services](9_release_criteria.md#system-services) release criterion. If you are doing release validation testing, a failure of this test case may be a breach of that release criterion. + +## Description + +This test covers the check, that all basic system service which are getting installed with the base groups are starting / running normally. + +## Setup + +1. Acquire access to either a baremetal machine or a VM host, to install a new machine +2. Prepare appropriate media for the selected ISO to be tested. + - Example: [QA:Testcase Media USB dd](Testcase_Media_USB_dd.md) + +## How to test + +Startup the system and check that all services are running without any failure: + +```bash +systemctl status +``` + +## Expected Results + +The tests during the process could be successfully finished. + +{% include 'qa_testcase_bottom.md' %} diff --git a/docs/documentation/QA/Testcase_Repo_Compare.md b/docs/documentation/QA/Testcase_Repo_Compare.md new file mode 100644 index 0000000..10751f1 --- /dev/null +++ b/docs/documentation/QA/Testcase_Repo_Compare.md @@ -0,0 +1,28 @@ +--- +title: QA:Testcase Media Repo Compare +author: Trevor Cooper +revision_date: 2022-05-18 +rc: + prod: Rocky Linux + ver: 8 + level: Final +--- + +!!! info "Associated release criterion" + This test case is associated with the [Release_Criteria#repositories-must-match-upstream](9_release_criteria.md#repositories-must-match-upstream) release criterion. If you are doing release validation testing, a failure of this test case may be a breach of that release criterion. + +## Description +This test case will verify that repositories and the packages within them match upstream as closely as possible. + +## Setup +1. Verify access to the Rocky Linux repocompare tooling. + +## How to test +1. Access [Rocky Linux repocompare website](https://repocompare.rockylinux.org/). +2. Verify similarity of Rocky Linux repositories with upstream content. + +## Expected Results +1. Rocky Linux repositories should match, as closely as possible, upstream repositories. +2. The content of Rocky Linux packages should match, as closely as possible, upstream repositories. + +{% include 'qa_testcase_bottom.md' %} diff --git a/docs/documentation/QA/Testcase_Storage_Volume_Resize.md b/docs/documentation/QA/Testcase_Storage_Volume_Resize.md new file mode 100644 index 0000000..3548f26 --- /dev/null +++ b/docs/documentation/QA/Testcase_Storage_Volume_Resize.md @@ -0,0 +1,54 @@ +--- +title: QA:Testcase Storage Volume Resize +author: Al Bowles +revision_date: 2022-07-08 +rc: + prod: Rocky Linux + ver: + - 8 + - 9 + level: Final +--- + +!!! info "Associated release criterion" + This test case is associated with the [Release_Criteria#Storage Volume Resize](9_release_criteria.md#storage-volume-resize) release criterion. If you are doing release validation testing, a failure of this test case may be a breach of that release criterion. + +## Description +This test case verifies that the installer will successfully resize or delete and overwrite existing partitions on storage volumes. + +{% include 'qa_data_loss_warning.md' %} + +## Setup +{% include 'qa_setup_boot_to_media.md' %} + +## How to test + +### Resize +1. From the Installation Destination spoke, in the Storage Configuration section, select the Custom radio button, then click Done. +1. Expand the list of available partitions by clicking the black arrow to the left of the release version and architecture. +1. Select the partition you wish to resize. Be sure to uncheck the Reformat checkbox if you wish to resize without reformatting the partition. +1. Click the Update Settings button to save your settings. +1. Click the + button to create a new partition off of the existing partition. Provide a mount point and desired capacity, then click Add Mount Point. +1. Repeat as necessary for additional partitions, or click Done to return to the Anaconda main hub. + +### Delete +1. From the Installation Destination spoke, in the Storage Configuration section, select the Automatic radio button, then click Done. +1. You should be presented with an "Installation Options" dialog, indicating the amount of disk space that is available for use and available to reclaim. +1. Click the Reclaim Space button. +1. Select a partition, then click the Delete button to delete the partition and reclaim the space. +1. Alternatively, click the Delete All button to delete all existing partitions. +1. When you have finished, click the Reclaim Space button to reclaim available free space. + +## Expected Results +1. The installation should complete and boot successfully. +1. Resized partitions should correctly reflect the desired size. This may be verified using the `lsblk` command. +1. Deleted partitions should no longer exist. + +## Testing in openQA +The following openQA test suites satisfy this release criteria: + +- `install_delete_partial` +- `install_delete_pata` +- `install_resize_lvm` + +{% include 'qa_testcase_bottom.md' %} diff --git a/docs/documentation/QA/Testcase_Template.md b/docs/documentation/QA/Testcase_Template.md new file mode 100644 index 0000000..20dbb50 --- /dev/null +++ b/docs/documentation/QA/Testcase_Template.md @@ -0,0 +1,28 @@ +--- +title: QA:Testcase Template +author: Trevor Cooper +revision_date: 2022-05-18 +rc: + prod: Rocky Linux + ver: 8 + level: Final +--- + +!!! info "Associated release criterion" + This test case is associated with the [Release_Criteria#TBD](9_release_criteria.md#TBD) release criterion. If you are doing release validation testing, a failure of this test case may be a breach of that release criterion. + +## Description +Add a short description here for this Testcase. + +## Setup +1. Add steps for setup for this Testcase. + +## How to test +1. Do this first... +2. Then do this... + +## Expected Results +1. This is what you should see/verify. +2. You should also see/verify this. + +{% include 'qa_testcase_bottom.md' %} diff --git a/docs/documentation/QA/Testcase_Update_Image.md b/docs/documentation/QA/Testcase_Update_Image.md new file mode 100644 index 0000000..c3af4f7 --- /dev/null +++ b/docs/documentation/QA/Testcase_Update_Image.md @@ -0,0 +1,51 @@ +--- +title: QA:Testcase Update Image +author: Al Bowles +revision_date: 2022-07-08 +rc: + prod: Rocky Linux + ver: 8 + level: Final +--- + +!!! info "Associated release criterion" + This test case is associated with the [Release_Criteria#Update Image](9_release_criteria.md#update-image) release criterion. If you are doing release validation testing, a failure of this test case may be a breach of that release criterion. + +## Description + +This test case verifies that an [update image]() can be loaded into Anaconda and applied during the install process. + +{% include 'qa_data_loss_warning.md' %} + +## Setup +{% include 'qa_setup_boot_to_media.md' %} +1. Hit the Tab key to edit the boot command + +## How to test + +1. Supply `inst.updates=https://fedorapeople.org/groups/qa/updates/updates-openqa.img` to the GRUB command line +1. Boot into the installer as usual. +1. In Anaconda, open the Installation Destination spoke. + +## Expected Results +1. Within the Installation Destination spoke, the selected install disk should have a pink background +=== "FAIL" + ![No update provided - **FAIL**](/assets/images/no_updates.png){ loading=lazy } + +=== "PASS" + ![Update provided - **PASS**](/assets/images/updates.png){ loading=lazy } + +1. If you cannot verify visually, check for the existence of `/tmp/updates`, which should contain updated source files if the update was successfully applied. Note that if the update image doesn't actually contain any source files, this directory will not be created. + + +## Testing with openQA +The following openQA test suites satisfy this release criteria: + +- `install_scsi_updates_img` + +## Additional References +[Red Hat Debug Boot Options](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/performing_an_advanced_rhel_installation/kickstart-and-advanced-boot-options_installing-rhel-as-an-experienced-user#debug-boot-options_kickstart-and-advanced-boot-options)
+[Fedora QA:Testcase Anaconda updates.img via URL](https://fedoraproject.org/wiki/QA:Testcase_Anaconda_updates.img_via_URL)
+[Fedora QA:Testcase Anaconda updates.img via local media](https://fedoraproject.org/wiki/QA:Testcase_Anaconda_updates.img_via_local_media)
+ +{% include 'qa_testcase_bottom.md' %} diff --git a/docs/documentation/QA/Testcase_VNC_Graphics_Mode.md b/docs/documentation/QA/Testcase_VNC_Graphics_Mode.md new file mode 100644 index 0000000..89ede76 --- /dev/null +++ b/docs/documentation/QA/Testcase_VNC_Graphics_Mode.md @@ -0,0 +1,46 @@ +--- +title: QA:Testcase VNC Graphics Mode +author: Trevor Cooper +revision_date: 2022-06-23 +rc: + prod: Rocky Linux + ver: 9 + level: Final +--- + +!!! info "Associated release criterion" + This test case is associated with the [Release_Criteria#vnc-graphics-mode-behaviors](9_release_criteria.md#vnc-graphics-mode-behaviors) release criterion. If you are doing release validation testing, a failure of this test case may be a breach of that release criterion. + +## Description +This test case will verify that release-blocking installers function as intended using the VNC installation method on supported systems and classes of hardware. + +{% include 'qa_testcase_supported_systems.md' %} + +## Setup +1. Obtain access to supported system and hardware class to be installed. +2. Prepare appropriate media for the selected ISO to be tested. + - Example: [QA:Testcase Media USB dd](Testcase_Media_USB_dd.md) +3. Obtain access to remote system with a VNC client installed to use for VNC connection. + + +!!! info "Suggested VNC Clients" + Both [`tigervnc`](https://tigervnc.org) and [`vinagre`](https://wiki.gnome.org/Apps/Vinagre) are VNC clients provided in Rocky Linux but any VNC client may be used. + +## How to test +1. Boot the system from the prepared optical, USB media or virtual device attachment. + - Examples: [QA:Testcase Custom Boot Methods Boot ISO](Testcase_Custom_Boot_Methods_Boot_Iso.md) +2. Interrupt the kernel boot and specify the appropriate VNC installation option on the boot command line. +3. Proceed with installation on the test system.
**Depending on installer choices this MAY destroy all the data on the test system.** +4. Depending on the choice or `direct` or `connect` mode connect inbound to the system under test or wait for it to connect to your listening VNC client. + +!!! error "DATA LOSS" + If you choose to complete the installation of the test system any/all data on the system may be lost. Please do not install on a system whose contents you need to keep. + +## Expected Results +1. Connection to (with direct mode) or from (in connect mode) to the Anaconda installer using VNC is possible. +2. Anaconda installer presents a usable graphical intallation environment. +3. System under test can be installed normally. +4. After reboot system boots into graphical environment. +5. After login user is able to operate the graphical environment. + +{% include 'qa_testcase_bottom.md' %} diff --git a/docs/documentation/dev_guides/.pages b/docs/documentation/dev_guides/.pages new file mode 100644 index 0000000..83d7e4b --- /dev/null +++ b/docs/documentation/dev_guides/.pages @@ -0,0 +1,8 @@ +--- +nav: + - Development Box Setup: wiki_development_boxes.md + - Git Commit Signing: commit_signing.md + - openQA - Access: openqa_access.md + - openQA - openqa-cli POST Examples: openqa_cli_post_examples.md + - openQA - openqa-clone-job Examples: openqa_clone_job_examples.md + - openQA - openqa-clone-custom-git-refspec Examples: openqa_clone_custom_git_refspec_examples.md diff --git a/docs/documentation/dev_guides/commit_signing.md b/docs/documentation/dev_guides/commit_signing.md new file mode 100644 index 0000000..3400cfe --- /dev/null +++ b/docs/documentation/dev_guides/commit_signing.md @@ -0,0 +1,112 @@ +--- +title: Signing Commits with GPG +author: Al Bowles +revision_date: 2022-06-13 +rc: + prod: Rocky Linux + ver: 8 + level: Final +--- +# Creating your primary keypair + +1. Initiate the keypair generation wizard + + gpg --full-generate-key --expert + +1. Select option `(9) ECC and ECC` for the key type +1. Select option `(1) Curve 25519` for the elliptic curve +1. Set a validity period of your choice, ideally less than 1 year +1. Specify real name and email address to associate with this keypair. The email address must match your verified Github email address or be set to `your-github-username@users.noreply.github.com`. +1. Type a passphrase (twice) + +# Create a signing keypair + +1. Add a signing subkey + + gpg --expert --edit-key my@email.addr + gpg> addkey + +1. Select option `(10) ECC (sign only)` for the key type +1. Select option `(1) Curve 25519` for the elliptic curve +1. Set a validity period of your choice, ideally less than 1 year +1. Accept the prompts and type a passphrase (twice) +1. Save and exit + + gpg> save + +# Create revocation certificate + + gpg --output my_email_addr.gpg-revocation-certificate --gen-revoke my@email.addr + +# Back up your keypair +Export the *primary keypair* (put these somewhere very safe along with revocation certificate) + + gpg --export-secret-keys --armor my@email.addr > my_email_addr.private.gpg-key + gpg --export --armor my@email.addr > my_email_addr.public.gpg-key + +# Remove the *primary keypair* from your keyring + +1. Export all subkeys from the new keypair to a file + + gpg --export-secret-subkeys my@email.addr > $HOME/.gnupg/subkeys + +1. Delete primary key from keyring - *BE SURE TO BACK UP YOUR PRIMARY KEYPAIR FIRST!* + + gpg --delete-secret-key my@email.addr + +1. Re-import the previously exported keys + + gpg --import $HOME/.gnupg/subkeys + +1. Look for `sec#` instead of `sec` in the output - pound sign means signing subkey is *not* in the keypair located in the keyring + + gpg --list-secret-keys $HOME/.gnupg/secring.gpg + +# Revoking a *signing keypair* + +Find the *primary keypair* and import it (preferably into an ephemeral system like a liveUSB) + + gpg --import /path/to/my_email_addr.public.gpg-key /path/to/my_email_addr.private.gpg-key + gpg --edit-key my@email.addr + gpg> revkey + [ passphrase twice ] + gpg> save + +# Renew an expired or expiring keypair + + gpg --edit-key my@email.addr + [select a key] + gpg> expire + [specify an expiration] + gpg> save + +# Create a single signed git commit + + git commit -S -m "my awesome signed commit" + +# Configure git to always sign commits with a specified key + + $ gpg --list-secret-keys --keyid-format=long # grab the fingerprint from the 'sec' line + git config [--global] commit.gpgsign true + git config [--global] user.signingkey DEADB33FBAD1D3A + +# Configure VSCode to sign commits + + # User or workspace setting + "git.enableCommitSigning": true + +# Upload your public key to a keyserver + + gpg --keyserver pgp.mit.edu --send-keys 0xDEADB33FBAD1D3A + +# Verify your key has been published + + gpg --keyserver pgp.mit.edu --search-key my@email.addr + +# References + +[OpenPGP Best Practices](https://riseup.net/en/security/message-security/openpgp/best-practices#key-configuration)
+[Github: Signing Commits](https://docs.github.com/en/enterprise-server@3.5/authentication/managing-commit-signature-verification/signing-commits)
+[Braincoke's Log: Create a GPG Key](https://blog.braincoke.fr/security/create-a-gpg-key/)
+[Creating the Perfect GPG Keypair](https://alexcabal.com/creating-the-perfect-gpg-keypair)
+[Digital Neanderthal: Generate GPG Keys With Curve Ed25519](https://www.digitalneanderthal.com/post/gpg/)
diff --git a/docs/documentation/dev_guides/openqa_access.md b/docs/documentation/dev_guides/openqa_access.md new file mode 100644 index 0000000..5bd2e64 --- /dev/null +++ b/docs/documentation/dev_guides/openqa_access.md @@ -0,0 +1,120 @@ +--- +title: openQA - Access +author: Trevor Cooper +revision_date: 2023-04-22 +rc: + prod: Rocky Linux + level: Final +--- + +## System Requirements + +To complete any of the examples below you will need access to a system providing the openQA client. Typically that will be a Fedora based system/container with the `openqa-client` package and it's (~239) dependencies installed. + +## Access Requirement + +### API `GET` access + +The {{ rc.prod }} openQA system allows unrestricted public access via it's web interface or using the `openqa-client` for `GET` operations against the API. + +### API `POST` access + +In order to use the openQA client to interact with the {{ rc.prod }} openQA system for `POST` operations the following are required: + +- an account in good standing in the [{{ rc.prod }} Account Services](https://accounts.rockylinux.org) system, +- authorization for API `POST` access from the {{ rc.prod }} Testing Team, and +- an [openQA API key](https://open.qa/docs/#_authentication) generated on the {{ rc.prod }} openQA system. + +## Configuring your openqa client + +Per the openqa client command help you can configure your client to use your API key in a number of ways. + +The following example shows how to configure your client by the most common method used. It's possible to configure +multiple openqa client API keys in this way. + +```bash +$ mkdir -p ~/.config/openqa + +$ vim ~/.config/openqa/client.conf + +$ cat ~/.config/openqa/client.conf +[localhost] +key = your_localhost_api_key +secret = your_localhost_api_secret +[openqa.rockylinux.org] +key = your_api_key +secret = your_api_secret +``` + +## Testing your openqa client installation + +```bash +$ openqa-cli api --host http://openqa.rockylinux.org --pretty jobs/1 +{ + "job" : { + "assets" : { + "iso" : [ + "Rocky-8.6-x86_64-boot.iso" + ] + }, + "assigned_worker_id" : 2, + "blocked_by_id" : null, + "children" : { + "Chained" : [], + "Directly chained" : [], + "Parallel" : [] + }, + "clone_id" : null, + "group" : "Rocky", + "group_id" : 2, + "has_parents" : 0, + "id" : 1, + "name" : "rocky-8.6-boot-iso-x86_64-Build-8.6-boot-iso--20221110.223812.0-install_default@64bit", + "parents" : { + "Chained" : [], + "Directly chained" : [], + "Parallel" : [] + }, + "parents_ok" : 1, + "priority" : 10, + "result" : "failed", + "settings" : { + "ARCH" : "x86_64", + "ARCH_BASE_MACHINE" : "64bit", + "BACKEND" : "qemu", + "BUILD" : "-8.6-boot-iso--20221110.223812.0", + "DESKTOP" : "gnome", + "DISTRI" : "rocky", + "FLAVOR" : "boot-iso", + "GRUB" : "ip=dhcp", + "HDDSIZEGB" : "15", + "ISO" : "Rocky-8.6-x86_64-boot.iso", + "MACHINE" : "64bit", + "NAME" : "00000001-rocky-8.6-boot-iso-x86_64-Build-8.6-boot-iso--20221110.223812.0-install_default@64bit", + "PACKAGE_SET" : "default", + "PART_TABLE_TYPE" : "mbr", + "POSTINSTALL" : "_collect_data", + "QEMUCPU" : "Nehalem", + "QEMUCPUS" : "2", + "QEMURAM" : "3072", + "QEMUVGA" : "virtio", + "QEMU_VIRTIO_RNG" : "1", + "TEST" : "install_default", + "TEST_SUITE_NAME" : "install_default", + "TEST_TARGET" : "ISO", + "VERSION" : "8.6", + "WORKER_CLASS" : "qemu_x86_64" + }, + "state" : "done", + "t_finished" : "2022-11-10T22:44:19", + "t_started" : "2022-11-10T22:38:12", + "test" : "install_default" + } +} +``` + +## References + +[openQA Documentation](https://open.qa/documentation/) + +{% include 'content_bottom.md' %} diff --git a/docs/documentation/dev_guides/openqa_cli_post_examples.md b/docs/documentation/dev_guides/openqa_cli_post_examples.md new file mode 100644 index 0000000..541a890 --- /dev/null +++ b/docs/documentation/dev_guides/openqa_cli_post_examples.md @@ -0,0 +1,92 @@ +--- +title: openQA - openqa-cli POST Examples +author: Trevor Cooper +revision_date: 2023-04-23 +rc: + prod: Rocky Linux + level: Final +--- + +# openqa-cli POST Examples + +This page will provide a brief overview of some basic `openqa-cli` `POST` examples. + +## System / Access Requirements + +To complete any of the examples please complete the API `POST` Access steps outlined in the [openQA - Access](openqa_access.md) document. + +## Basic POST + +A basic `POST` can be used for any of the default test suites for the various {{ rc.prod }} media that are made available. The following examples show some of these standard `POST`s that are commonly used by our team and will be used to demonstrate how some minor variations. + +### FLAVOR=boot-iso + +This first `POST` is the most basic, simply providing the minimal set of variables required to trigger the standard test suite for the {{ rc.prod }} 9.1 boot ISO on openqa workers for the `x86_64` architecture. All tests of the test suite are predetermined and configure on the openQA server. Since the boot ISO doesn't contain any packages this test suite is effectively a network install from standard {{ rc.prod }} repository servers and/or mirrors. + +```bash +$ openqa-cli api -X POST isos ISO=Rocky-9.1-x86_64-boot.iso ARCH=x86_64 \ + DISTRI=rocky FLAVOR=boot-iso VERSION=9.1 CURRREL=9 BUILD=20230409-Rocky-9.1-x86_64.0 +``` + +### FLAVOR=minimal-iso + +This `POST` demonstrates how a different media type, in this case the minimal ISO, for an alternate {{ rc.prod }} version, in this case {{ rc.prod }} 8.7, can be triggered. As can be seen by this and the previous `POST` the `BUILD` variable will typically be designate the date, version and architecture of the test suite. Since the minimal ISO contains all packages required to conduct a ***minimal*** install of {{ rc.prod }} that is the behavior of this test suite. + +```bash +$ openqa-cli api -X POST isos ISO=Rocky-8.7-x86_64-minimal.iso ARCH=x86_64 \ + DISTRI=rocky FLAVOR=minimal-iso VERSION=8.7 CURRREL=8 BUILD=20230409-Rocky-8.7-x86_64.0 +``` + +### FLAVOR=package-set + +This `POST` demonstrates specification of the final normal media type, the dvd ISO, along with what is called a `FLAVOR`, in this case `package-set` for the `x86_64` architecture and {{ rc.prod }} 9.1. Since the dvd ISO contains all of the packages available at release of a specific version or {{ rc.prod }} the `package-set` test suite will test installation of all primary installation types of {{ rc.prod }} not included in the `minimal-iso` test suite above. + +```bash +$ openqa-cli api -X POST isos ISO=Rocky-9.1-20221214.1-x86_64-dvd.iso ARCH=x86_64 \ + DISTRI=rocky FLAVOR=package-set VERSION=9.1 CURRREL=9 BUILD=20230409-Rocky-9.1-x86_64.0 +``` + +These three test suites provide for the minimal testing of all ISOs produced for a given release of {{ rc.prod }}. + +## Advanced POST + +In addition to the [Basic POSTs](#basic-post) described above there are additional default test suites that use the dvd ISO media and include substantially more test cases. Those include: + +- installing in graphical, text and serial console +- installation for standard BIOS and UEFI +- validation of the Anaconda help system +- disk layout variations including LVM, RAID, partition shrink and/or grow, iSCSI and LUKS +- PXE installation from various network sources +- installation in various languages + +Standard `POST`s for these test suites is very similar to the basic POSTs above and are shown below... + +### FLAVOR=dvd-iso + +```bash +$ openqa-cli api -X POST isos ISO=Rocky-9.1-20221214.1-x86_64-dvd.iso ARCH=x86_64 \ + DISTRI=rocky FLAVOR=dvd-iso VERSION=9.1 CURRREL=9 BUILD=20230409-Rocky-9.1-x86_64.0 +``` + +### FLAVOR=universal + +```bash +$ openqa-cli api -X POST isos ISO=Rocky-9.1-20221214.1-x86_64-dvd.iso ARCH=x86_64 \ + DISTRI=rocky FLAVOR=universal VERSION=9.1 CURRREL=9 BUILD=20230409-Rocky-9.1-x86_64.0 +``` + +## Collection of test suites by BUILD + +A feature of openQA is that for a given job group test suites which use the same `BUILD` identifier are collected into a single view in the web UI. + +![openQA Home View...](/assets/images/openqa_home_view.png){ loading=lazy } + +Thus, the examples show above which all use `BUILD=20230409-Rocky-9.1-x86_64.0` are all shown in a single view. Additionally, that view is accessible via a predictable URI, for example [`https://openqa.rockylinux.org/tests/overview?build=20230409-Rocky-9.1-x86_64.0`](https://openqa.rockylinux.org/tests/overview?build=20230409-Rocky-9.1-x86_64.0) as shown below... + +![openQA Build View...](/assets/images/openqa_build_view.png){ loading=lazy } + +## References + +[openQA Documentation](https://open.qa/documentation/) + +{% include 'content_bottom.md' %} diff --git a/docs/documentation/dev_guides/openqa_clone_custom_git_refspec_examples.md b/docs/documentation/dev_guides/openqa_clone_custom_git_refspec_examples.md new file mode 100644 index 0000000..c747401 --- /dev/null +++ b/docs/documentation/dev_guides/openqa_clone_custom_git_refspec_examples.md @@ -0,0 +1,481 @@ +--- +title: openQA - openqa-clone-custom-refspec Examples +author: Trevor Cooper +revision_date: 2023-04-23 +rc: + prod: Rocky Linux + level: Final +--- + +# openqa-clone-custom-git-refspec Examples + +This page will provide a brief overview of basic and advanced job cloning using the `openqa-clone-custom-git-refspec` command. + +At a high level `openqa-clone-custom-git-refspec` can be viewed as a mechanism to test PRs for openQA tests directly in a {{ rc.prod }} openQA instance with making changes to the default configuration. As such, it can support testing of PRs that change test code and needles as long as changes to `templates.fif.json` are not also required. A combination of `openqa-clone-custom-git-refspec` and `openqa-clone-job` (which is actually used by `openqa-clone-custom-git-refspec` under the hood) can be used for some cases where `POST` variables are pre-defined in `templates.fif.json`. + +## System / Access Requirements + +To complete any of the examples please complete the API `POST` Access steps outlined in the [openQA - Access](openqa_access.md) document. + +## Basic `openqa-clone-custom-git-refspec` + +The following example demonstrates the testing of an open Github pull request in the {{ rc.prod }} openQA production system. The PR only changes test code and does not supply updated needles for the test. + +### Github PR information + +***NOTE: The Github CLI tool (`gh`) is used to display PR information statically in this guide.*** + +```text +➜ os-autoinst-distri-rocky git:(develop) gh pr view 168 +Serial console install #168 +Merged • AlanMarshall wants to merge 1 commit into develop from serial_console • about 27 days ago ++5 -2 • No checks +Reviewers: akatch (Approved), tcooper (Approved), lumarel (Requested) +Labels: priority: medium, type: bug, test suite + + + Network is enabled by default at v9 so requires conditional code to handle multiple versions. + Tested for 9.1, 8.7 & 8.8: + + openqa-cli api -X POST isos ISO=Rocky-9.1-20221214.1-x86_64-dvd.iso ARCH=x86_64 DISTRI=rocky FLAVOR=universal + VERSION=9.1 BUILD=-"$(date +%Y%m%d.%H%M%S).0"-9.1-20221214.1-universal TEST=install_serial_console + openqa-cli api -X POST isos ISO=Rocky-8.7-x86_64-dvd1.iso ARCH=x86_64 DISTRI=rocky FLAVOR=universal VERSION=8.7 BUILD=- + "$(date +%Y%m%d.%H%M%S).0"-8.7-20221110-universal TEST=install_serial_console + openqa-cli api -X POST isos ISO=Rocky-8.8-x86_64-dvd1.iso ARCH=x86_64 DISTRI=rocky FLAVOR=universal VERSION=8.8 BUILD=- + "$(date +%Y%m%d.%H%M%S).0"-8.8-lookahead-universal TEST=install_serial_console + + Result: Tests pass. + Also confirm that all main hub check boxes are checked and user test created prior to start of installation. + Fixes Issue #102 + +View this pull request on GitHub: https://github.com/rocky-linux/os-autoinst-distri-rocky/pull/168 +``` + +Above is the information provided in the original PR and it includes tests performed in Alan's openQA development system. We can rerun failing tests in the {{ rc.prod }} openQA system after identifying an appropriate job ID for each Rocky Linux version we are testing. For this example the openQA WebUI was used to find appropriate test IDs to clone. + +### Run `openqa-clone-custom-git-refspec` in `--verbose --dry-run` mode + +In practice it is useful to run `openqa-clone-custom-git-refspec` in `--verbose` and `--dry-run` mode to observe it's behavior even for the Basic cases... + +```bash +$ openqa-clone-custom-git-refspec --verbose --dry-run \ + https://github.com/rocky-linux/os-autoinst-distri-rocky/pull/168 \ + https://openqa.rockylinux.org/tests/16080 2>&1 | tee pr-168 +``` + +***NOTE: The full output of `openqa-clone-custom-git-refspece` will not be shown here.*** + +```diff ++ shift ++ true ++ case "$1" in ++ dry_run=true ++ shift ++ true ++ case "$1" in ++ shift ++ break ++ job_list=https://openqa.rockylinux.org/tests/16080 ++ [[ -z '' ]] ++ first_arg=https://github.com/rocky-linux/os-autoinst-distri-rocky/pull/168 ++ [[ https://github.com/rocky-linux/os-autoinst-distri-rocky/pull/168 == *\p\u\l\l* ]] ++ pr_url=https://github.com/rocky-linux/os-autoinst-distri-rocky/pull/168 ++ target_repo_part=https://github.com/rocky-linux/os-autoinst-distri-rocky ++ pr=168 ++ pr=168 ++ [[ -z '' ]] ++ pr_url=https://api.github.com/repos/rocky-linux/os-autoinst-distri-rocky/pulls/168 +++ eval 'curl -s https://api.github.com/repos/rocky-linux/os-autoinst-distri-rocky/pulls/168' ++++ curl -s https://api.github.com/repos/rocky-linux/os-autoinst-distri-rocky/pulls/168 + +...... + +++ jq -r '.NEEDLES_DIR | select (.!=null)' ++ old_needledir= ++ local needles_dir= ++ needles_dir=rocky/needles ++ local repo_branch=AlanMarshall/os-autoinst-distri-rocky#serial_console ++ local test_suffix=@AlanMarshall/os-autoinst-distri-rocky#serial_console ++ local build=AlanMarshall/os-autoinst-distri-rocky#168 ++ local casedir=https://github.com/AlanMarshall/os-autoinst-distri-rocky.git#serial_console ++ local GROUP=0 ++ local dry_run=true ++ local scriptdir +++ dirname /usr/bin/openqa-clone-custom-git-refspec ++ scriptdir=/usr/bin ++ local 'cmd=true /usr/bin/openqa-clone-job --skip-chained-deps --parental-inheritance --within-instance "https://openqa.rockylinux.org" "15973" _GROUP="0" TEST+="@AlanMarshall/os-autoinst-distri-rocky#serial_console" BUILD="AlanMarshall/os-autoinst-distri-rocky#168" CASEDIR="https://github.com/AlanMarshall/os-autoinst-distri-rocky.git#serial_console" PRODUCTDIR="os-autoinst-distri-rocky" NEEDLES_DIR="rocky/needles"' ++ [[ 0 -ne 0 ]] ++ [[ -n '' ]] ++ eval 'true /usr/bin/openqa-clone-job --skip-chained-deps --parental-inheritance --within-instance "https://openqa.rockylinux.org" "15973" _GROUP="0" TEST+="@AlanMarshall/os-autoinst-distri-rocky#serial_console" BUILD="AlanMarshall/os-autoinst-distri-rocky#168" CASEDIR="https://github.com/AlanMarshall/os-autoinst-distri-rocky.git#serial_console" PRODUCTDIR="os-autoinst-distri-rocky" NEEDLES_DIR="rocky/needles"' +++ true /usr/bin/openqa-clone-job --skip-chained-deps --parental-inheritance --within-instance https://openqa.rockylinux.org 15973 _GROUP=0 TEST+=@AlanMarshall/os-autoinst-distri-rocky#serial_console BUILD=AlanMarshall/os-autoinst-distri-rocky#168 CASEDIR=https://github.com/AlanMarshall/os-autoinst-distri-rocky.git#serial_console PRODUCTDIR=os-autoinst-distri-rocky NEEDLES_DIR=rocky/needles +``` + +What can be seen from the complete `--dry-run` output for `openqa-clone-custom-git-refspec` is that both the job to be cloned and the PR to be used are inspected and a `openqa-clone-job` command is generated to be submitted to the openQA system the job is being cloned on. + +Without using `--dry-run` the final `openqa-clone-job` command shown above will be run causing the job of interest to be cloned with additional `POST` variables that will cause the repository/branch referenced in the PR to be cloned into the test directory with important files referenced in the cloned job. + +### Run `openqa-clone-custom-git-refspec` without `--verbose --dry-run` mode... + +```bash +$ openqa-clone-custom-git-refspec \ + https://github.com/rocky-linux/os-autoinst-distri-rocky/pull/168 \ + https://openqa.rockylinux.org/tests/16080 +Created job #16119: rocky-9.1-universal-x86_64-Build20230329-Rocky-9.1-x86_64.0-install_serial_console@64bit -> https://openqa.rockylinux.org/t16119 +``` + +### Cloned job information... + +```bash +$ openqa-cli api jobs/16119 --pretty +{ + "job" : { + "assets" : { + "iso" : [ + "Rocky-9.1-20221214.1-x86_64-dvd.iso" + ] + }, + "assigned_worker_id" : 5, + "blocked_by_id" : null, + "children" : { + "Chained" : [], + "Directly chained" : [], + "Parallel" : [] + }, + "clone_id" : 16121, + "group_id" : null, + "has_parents" : 0, + "id" : 16119, + "name" : "rocky-9.1-universal-x86_64-BuildAlanMarshall_os-autoinst-distri-rocky_168-install_serial_console@AlanMarshall_os-autoinst-distri-rocky_serial_console@64bit", + "parents" : { + "Chained" : [], + "Directly chained" : [], + "Parallel" : [] + }, + "parents_ok" : 1, + "priority" : 10, + "reason" : "isotovideo abort: isotovideo received signal TERM", + "result" : "user_restarted", + "settings" : { + "ANACONDA_TEXT" : "1", + "ARCH" : "x86_64", + "ARCH_BASE_MACHINE" : "64bit", + "BACKEND" : "qemu", + "BUILD" : "AlanMarshall\/os-autoinst-distri-rocky#168", + "CASEDIR" : "https:\/\/github.com\/AlanMarshall\/os-autoinst-distri-rocky.git#serial_console", + "CLONED_FROM" : "https:\/\/openqa.rockylinux.org\/tests\/15973", + "CURRREL" : "9", + "DISTRI" : "rocky", + "FLAVOR" : "universal", + "HDDSIZEGB" : "15", + "ISO" : "Rocky-9.1-20221214.1-x86_64-dvd.iso", + "LOCATION" : "https:\/\/download.rockylinux.org\/pub\/rocky\/9.1\/BaseOS", + "MACHINE" : "64bit", + "NAME" : "00016119-rocky-9.1-universal-x86_64-BuildAlanMarshall_os-autoinst-distri-rocky_168-install_serial_console@AlanMarshall_os-autoinst-distri-rocky_serial_console@64bit", + "NEEDLES_DIR" : "rocky\/needles", + "NICTYPE_USER_OPTIONS" : "net=172.16.2.0\/24", + "NO_UEFI_POST" : "1", + "PART_TABLE_TYPE" : "mbr", + "PRODUCTDIR" : "os-autoinst-distri-rocky", + "QEMUCPU" : "Nehalem", + "QEMUCPUS" : "2", + "QEMURAM" : "2048", + "QEMU_HOST_IP" : "172.16.2.2", + "QEMU_VIDEO_DEVICE" : "virtio-vga", + "QEMU_VIRTIO_RNG" : "1", + "SERIAL_CONSOLE" : "1", + "TEST" : "install_serial_console@AlanMarshall\/os-autoinst-distri-rocky#serial_console", + "TEST_SUITE_NAME" : "install_serial_console", + "TEST_TARGET" : "ISO", + "VERSION" : "9.1", + "VIRTIO_CONSOLE_NUM" : "2", + "WORKER_CLASS" : "qemu_x86_64", + "XRES" : "1024", + "YRES" : "768" + }, + "state" : "done", + "t_finished" : "2023-03-29T06:19:37", + "t_started" : "2023-03-29T06:12:26", + "test" : "install_serial_console@AlanMarshall\/os-autoinst-distri-rocky#serial_console" + } +} +``` + + +## Advanced `openqa-clone-custom-git-refspec` + +The following example demonstrates the testing of an open Github pull request in the {{ rc.prod }} openQA production system. The PR changes test code and supplies updated needles for the test. + +### Github PR information + +```text +➜ os-autoinst-distri-rocky git:(nazunalika/develop) gh pr view 162 + +Anaconda text install #162 +Open • AlanMarshall wants to merge 2 commits into develop from anaconda-txt • about 1 day ago ++30 -5 • No checks +Reviewers: akatch (Approved), lumarel (Requested), tcooper (Requested) +Labels: priority: medium, type: bug, test suite + + + Added new needle for text install. + Deleted redundant code. + Tested for 9.1, 8.7 & 8.8: + + openqa-cli api -X POST isos ISO=Rocky-9.1-20221214.1-x86_64-dvd.iso ARCH=x86_64 DISTRI=rocky FLAVOR=universal + VERSION=9.1 BUILD=-"$(date +%Y%m%d.%H%M%S).0"-9.1-20221214.1-universal TEST=install_anaconda_text + openqa-cli api -X POST isos ISO=Rocky-8.7-x86_64-dvd1.iso ARCH=x86_64 DISTRI=rocky FLAVOR=universal VERSION=8.7 BUILD=- + "$(date +%Y%m%d.%H%M%S).0"-8.7-20221110-universal TEST=install_anaconda_text + openqa-cli api -X POST isos ISO=Rocky-8.8-x86_64-dvd1.iso ARCH=x86_64 DISTRI=rocky FLAVOR=universal VERSION=8.8 BUILD=- + "$(date +%Y%m%d.%H%M%S).0"-8.8-lookahead-universal TEST=install_anaconda_text + + Result: Pass + Fixes Issue #145 + + +akatch approved (Member) • 18h • Newest comment + + All indicated tests pass. + + +View this pull request on GitHub: https://github.com/rocky-linux/os-autoinst-distri-rocky/pull/162 +``` + +### Run `openqa-clone-custom-git-refspec` in `--verbose --dry-run` mode + +```diff +$ openqa-clone-custom-git-refspec --verbose --dry-run https://github.com/rocky-linux/os-autoinst-distri-rocky/pull/162 https://openqa.rockylinux.org/tests/13371 ++ shift ++ true ++ case "$1" in ++ dry_run=true ++ shift ++ true ++ case "$1" in ++ shift ++ break ++ job_list=https://openqa.rockylinux.org/tests/13371 ++ [[ -z '' ]] ++ first_arg=https://github.com/rocky-linux/os-autoinst-distri-rocky/pull/162 ++ [[ https://github.com/rocky-linux/os-autoinst-distri-rocky/pull/162 == *\p\u\l\l* ]] ++ pr_url=https://github.com/rocky-linux/os-autoinst-distri-rocky/pull/162 ++ target_repo_part=https://github.com/rocky-linux/os-autoinst-distri-rocky + + +...... + +++ jq -r '.NEEDLES_DIR | select (.!=null)' ++ old_needledir= ++ local needles_dir= ++ needles_dir=rocky/needles ++ local repo_branch=AlanMarshall/os-autoinst-distri-rocky#anaconda-txt ++ local test_suffix=@AlanMarshall/os-autoinst-distri-rocky#anaconda-txt ++ local build=AlanMarshall/os-autoinst-distri-rocky#162 ++ local casedir=https://github.com/AlanMarshall/os-autoinst-distri-rocky.git#anaconda-txt ++ local GROUP=0 ++ local dry_run=true ++ local scriptdir +++ dirname /usr/bin/openqa-clone-custom-git-refspec ++ scriptdir=/usr/bin ++ local 'cmd=true /usr/bin/openqa-clone-job --skip-chained-deps --parental-inheritance --within-instance "https://openqa.rockylinux.org" "13371" _GROUP="0" TEST+="@AlanMarshall/os-autoinst-distri-rocky#anaconda-txt" BUILD="AlanMarshall/os-autoinst-distri-rocky#162" CASEDIR="https://github.com/AlanMarshall/os-autoinst-distri-rocky.git#anaconda-txt" PRODUCTDIR="os-autoinst-distri-rocky" NEEDLES_DIR="rocky/needles"' ++ [[ 0 -ne 0 ]] ++ [[ -n '' ]] ++ eval 'true /usr/bin/openqa-clone-job --skip-chained-deps --parental-inheritance --within-instance "https://openqa.rockylinux.org" "13371" _GROUP="0" TEST+="@AlanMarshall/os-autoinst-distri-rocky#anaconda-txt" BUILD="AlanMarshall/os-autoinst-distri-rocky#162" CASEDIR="https://github.com/AlanMarshall/os-autoinst-distri-rocky.git#anaconda-txt" PRODUCTDIR="os-autoinst-distri-rocky" NEEDLES_DIR="rocky/needles"' +++ true /usr/bin/openqa-clone-job --skip-chained-deps --parental-inheritance --within-instance https://openqa.rockylinux.org 13371 _GROUP=0 TEST+=@AlanMarshall/os-autoinst-distri-rocky#anaconda-txt BUILD=AlanMarshall/os-autoinst-distri-rocky#162 CASEDIR=https://github.com/AlanMarshall/os-autoinst-distri-rocky.git#anaconda-txt PRODUCTDIR=os-autoinst-distri-rocky NEEDLES_DIR=rocky/needles +``` + +This PR provides updated needles and the default behavior of `openqa-clone-custom-git-refspec` is to **not** provide an alternate location for `NEEDLES`. The `--verbose --dry-run` output needs to be modified to ensure the needles provided in the PR are used in the test. + +### Modify `--verbose --dry-run` output to point to needles in the PR... + +Use output to modify clone job... + +#### original + +```bash +$ /usr/bin/openqa-clone-job --skip-chained-deps --parental-inheritance --within-instance https://openqa.rockylinux.org \ + 13371 _GROUP=0 TEST+=@AlanMarshall/os-autoinst-distri-rocky#anaconda-txt \ + BUILD=AlanMarshall/os-autoinst-distri-rocky#162 CASEDIR=https://github.com/AlanMarshall/os-autoinst-distri-rocky.git#anaconda-txt \ + PRODUCTDIR=os-autoinst-distri-rocky +NEEDLES_DIR=rocky/needles +``` + +#### specify NEEDLES_DIR manually pointing at PR branch + +```bash +$ /usr/bin/openqa-clone-job --skip-chained-deps --parental-inheritance --within-instance https://openqa.rockylinux.org \ + 13371 _GROUP=0 TEST+=@AlanMarshall/os-autoinst-distri-rocky#anaconda-txt \ + BUILD=AlanMarshall/os-autoinst-distri-rocky#162 CASEDIR=https://github.com/AlanMarshall/os-autoinst-distri-rocky.git#anaconda-txt \ + PRODUCTDIR=os-autoinst-distri-rocky NEEDLES_DIR=https://github.com/AlanMarshall/os-autoinst-distri-rocky.git#anaconda-txt/needles +``` + +#### {{ rc.prod }} 9.1 + +```bash +$ /usr/bin/openqa-clone-job --skip-chained-deps --parental-inheritance --within-instance https://openqa.rockylinux.org \ + 13255 _GROUP=0 TEST+=@AlanMarshall/os-autoinst-distri-rocky#anaconda-txt \ + BUILD=AlanMarshall/os-autoinst-distri-rocky#162 CASEDIR=https://github.com/AlanMarshall/os-autoinst-distri-rocky.git#anaconda-txt \ + PRODUCTDIR=os-autoinst-distri-rocky NEEDLES_DIR=https://github.com/AlanMarshall/os-autoinst-distri-rocky.git#anaconda-txt/needles +Created job #14228: rocky-9.1-universal-x86_64-Build20230319-Rocky-9.1-x86_64.0-install_anaconda_text@64bit -> https://openqa.rockylinux.org/t14228 +``` + +```bash +$ openqa-cli api jobs/14228 --pretty +{ + "job" : { + "assets" : { + "iso" : [ + "Rocky-9.1-20221214.1-x86_64-dvd.iso" + ] + }, + "assigned_worker_id" : 9, + "blocked_by_id" : null, + "children" : { + "Chained" : [], + "Directly chained" : [], + "Parallel" : [] + }, + "clone_id" : null, + "group_id" : null, + "has_parents" : 0, + "id" : 14228, + "name" : "rocky-9.1-universal-x86_64-BuildAlanMarshall_os-autoinst-distri-rocky_162-install_anaconda_text@AlanMarshall_os-autoinst-distri-rocky_anaconda-txt@64bit", + "parents" : { + "Chained" : [], + "Directly chained" : [], + "Parallel" : [] + }, + "parents_ok" : 1, + "priority" : 0, + "result" : "passed", + "settings" : { + "ANACONDA_TEXT" : "1", + "ARCH" : "x86_64", + "ARCH_BASE_MACHINE" : "64bit", + "BACKEND" : "qemu", + "BUILD" : "AlanMarshall\/os-autoinst-distri-rocky#162", + "CASEDIR" : "https:\/\/github.com\/AlanMarshall\/os-autoinst-distri-rocky.git#anaconda-txt", + "CLONED_FROM" : "https:\/\/openqa.rockylinux.org\/tests\/13255", + "CURRREL" : "9", + "DISTRI" : "rocky", + "FLAVOR" : "universal", + "HDDSIZEGB" : "15", + "ISO" : "Rocky-9.1-20221214.1-x86_64-dvd.iso", + "LOCATION" : "https:\/\/dl.rockylinux.org\/pub\/rocky\/9.1", + "MACHINE" : "64bit", + "NAME" : "00014228-rocky-9.1-universal-x86_64-BuildAlanMarshall_os-autoinst-distri-rocky_162-install_anaconda_text@AlanMarshall_os-autoinst-distri-rocky_anaconda-txt@64bit", + "NEEDLES_DIR" : "https:\/\/github.com\/AlanMarshall\/os-autoinst-distri-rocky.git#anaconda-txt\/needles", + "NICTYPE_USER_OPTIONS" : "net=172.16.2.0\/24", + "PART_TABLE_TYPE" : "mbr", + "PRODUCTDIR" : "os-autoinst-distri-rocky", + "QEMUCPU" : "Nehalem", + "QEMUCPUS" : "2", + "QEMURAM" : "2048", + "QEMU_HOST_IP" : "172.16.2.2", + "QEMU_VIDEO_DEVICE" : "virtio-vga", + "QEMU_VIRTIO_RNG" : "1", + "TEST" : "install_anaconda_text@AlanMarshall\/os-autoinst-distri-rocky#anaconda-txt", + "TEST_SUITE_NAME" : "install_anaconda_text", + "TEST_TARGET" : "ISO", + "VERSION" : "9.1", + "WORKER_CLASS" : "qemu_x86_64", + "XRES" : "1024", + "YRES" : "768" + }, + "state" : "done", + "t_finished" : "2023-03-22T05:28:28", + "t_started" : "2023-03-22T05:07:09", + "test" : "install_anaconda_text@AlanMarshall\/os-autoinst-distri-rocky#anaconda-txt" + } +} +``` + +![openqa-clone-custome-git-refspec-job-14228 example...](/assets/images/openqa-clone-custom-git-refspec-job-14228.png){ loading=lazy } + +#### {{ rc.prod }} 8.7 + +```bash +$ /usr/bin/openqa-clone-job --skip-chained-deps --parental-inheritance --within-instance https://openqa.rockylinux.org \ + 13371 _GROUP=0 TEST+=@AlanMarshall/os-autoinst-distri-rocky#anaconda-txt \ + BUILD=AlanMarshall/os-autoinst-distri-rocky#162 CASEDIR=https://github.com/AlanMarshall/os-autoinst-distri-rocky.git#anaconda-txt \ + PRODUCTDIR=os-autoinst-distri-rocky NEEDLES_DIR=https://github.com/AlanMarshall/os-autoinst-distri-rocky.git#anaconda-txt/needles +Created job #14229: rocky-8.7-universal-x86_64-Build20230319-Rocky-8.7-x86_64.0-install_anaconda_text@64bit -> https://openqa.rockylinux.org/t14229 +``` + +```bash +$ openqa-cli api jobs/14229 --pretty +{ + "job" : { + "assets" : { + "iso" : [ + "Rocky-8.7-x86_64-dvd1.iso" + ] + }, + "assigned_worker_id" : 8, + "blocked_by_id" : null, + "children" : { + "Chained" : [], + "Directly chained" : [], + "Parallel" : [] + }, + "clone_id" : null, + "group_id" : null, + "has_parents" : 0, + "id" : 14229, + "name" : "rocky-8.7-universal-x86_64-BuildAlanMarshall_os-autoinst-distri-rocky_162-install_anaconda_text@AlanMarshall_os-autoinst-distri-rocky_anaconda-txt@64bit", + "parents" : { + "Chained" : [], + "Directly chained" : [], + "Parallel" : [] + }, + "parents_ok" : 1, + "priority" : 0, + "result" : "passed", + "settings" : { + "ANACONDA_TEXT" : "1", + "ARCH" : "x86_64", + "ARCH_BASE_MACHINE" : "64bit", + "BACKEND" : "qemu", + "BUILD" : "AlanMarshall\/os-autoinst-distri-rocky#162", + "CASEDIR" : "https:\/\/github.com\/AlanMarshall\/os-autoinst-distri-rocky.git#anaconda-txt", + "CLONED_FROM" : "https:\/\/openqa.rockylinux.org\/tests\/13371", + "CURRREL" : "8", + "DISTRI" : "rocky", + "FLAVOR" : "universal", + "HDDSIZEGB" : "15", + "ISO" : "Rocky-8.7-x86_64-dvd1.iso", + "LOCATION" : "https:\/\/dl.rockylinux.org\/pub\/rocky\/8.7", + "MACHINE" : "64bit", + "NAME" : "00014229-rocky-8.7-universal-x86_64-BuildAlanMarshall_os-autoinst-distri-rocky_162-install_anaconda_text@AlanMarshall_os-autoinst-distri-rocky_anaconda-txt@64bit", + "NEEDLES_DIR" : "https:\/\/github.com\/AlanMarshall\/os-autoinst-distri-rocky.git#anaconda-txt\/needles", + "NICTYPE_USER_OPTIONS" : "net=172.16.2.0\/24", + "PART_TABLE_TYPE" : "mbr", + "PRODUCTDIR" : "os-autoinst-distri-rocky", + "QEMUCPU" : "Nehalem", + "QEMUCPUS" : "2", + "QEMURAM" : "2048", + "QEMU_HOST_IP" : "172.16.2.2", + "QEMU_VIDEO_DEVICE" : "virtio-vga", + "QEMU_VIRTIO_RNG" : "1", + "TEST" : "install_anaconda_text@AlanMarshall\/os-autoinst-distri-rocky#anaconda-txt", + "TEST_SUITE_NAME" : "install_anaconda_text", + "TEST_TARGET" : "ISO", + "VERSION" : "8.7", + "WORKER_CLASS" : "qemu_x86_64", + "XRES" : "1024", + "YRES" : "768" + }, + "state" : "done", + "t_finished" : "2023-03-22T05:31:22", + "t_started" : "2023-03-22T05:10:46", + "test" : "install_anaconda_text@AlanMarshall\/os-autoinst-distri-rocky#anaconda-txt" + } +} +``` + +![openqa-clone-custome-git-refspec-job-14229 example...](/assets/images/openqa-clone-custom-git-refspec-job-14229.png){ loading=lazy } + +## References + +[openQA Documentation](http://open.qa/documentation/) + +{% include 'content_bottom.md' %} diff --git a/docs/documentation/dev_guides/openqa_clone_job_examples.md b/docs/documentation/dev_guides/openqa_clone_job_examples.md new file mode 100644 index 0000000..1308652 --- /dev/null +++ b/docs/documentation/dev_guides/openqa_clone_job_examples.md @@ -0,0 +1,244 @@ +--- +title: openQA - openqa-clone-job Examples +author: Trevor Cooper +revision_date: 2023-04-22 +rc: + prod: Rocky Linux + level: Final +--- + +# openqa-clone-job Examples + +This page will provide a brief overview of basic and advanced job cloning using the `openqa-clone-job` command. + +## System / Access Requirements + +To complete any of the examples please complete the API `POST` Access steps outlined in the [openQA - Access](openqa_access.md) document. + +## Basic `openqa-clone-job` + +### Querying openQA for a specific test or job + +First you might want to query the {{ rc.prod }} openQA system for the latest job ID for a specific job or test. The openQA client, hereafter refered to as `openqa-cli` will allow you to quickly do that via the API. Here is an example... + +```bash +$ openqa-cli api --host http://openqa.rockylinux.org jobs/overview groupid=0 distri=rocky version=9.1 test=install_default_upload latest=1 | jq '.' +[ + { + "id": 22735, + "name": "rocky-9.1-dvd-iso-x86_64-Build20230423-Rocky-9.1-x86_64.0-install_default_upload@64bit" + } +] +``` + +This basically says "give me the job id and name of the most recent `install_default_upload` test for {{ rc.prod }} 9.1". + +### Cloning a job "as-is" + +With that job id in hand you can now clone that job directly to your local openQA development system with... + +```bash +$ openqa-clone-job --from https://openqa.rockylinux.org --skip-download 22735 +Cloning children of rocky-9.1-dvd-iso-x86_64-Build20230423-Rocky-9.1-x86_64.0-install_default_upload@64bit +Created job #23: rocky-9.1-dvd-iso-x86_64-Build20230423-Rocky-9.1-x86_64.0-install_default_upload@64bit -> http://localhost/t23 +``` + +### Basic job overview + +Now you should have the same job running in your local instance... + +```bash +$ openqa-cli api jobs/overview +[{"id":23,"name":"rocky-9.1-dvd-iso-x86_64-Build20230423-Rocky-9.1-x86_64.0-install_default_upload@64bit"}] +``` + +### Basic job details + +```bash +$ openqa-cli api jobs/23 | jq '.' +{ + "job": { + "assets": { + "iso": [ + "Rocky-9.1-20221214.1-x86_64-dvd.iso" + ] + }, + "assigned_worker_id": 2, + "blocked_by_id": null, + "children": { + "Chained": [], + "Directly chained": [], + "Parallel": [] + }, + "clone_id": null, + "group": "Rocky", + "group_id": 2, + "has_parents": 0, + "id": 23, + "name": "rocky-9.1-dvd-iso-x86_64-Build20230423-Rocky-9.1-x86_64.0-install_default_upload@64bit", + "parents": { + "Chained": [], + "Directly chained": [], + "Parallel": [] + }, + "parents_ok": 1, + "priority": 50, + "result": "none", + "settings": { + "ARCH": "x86_64", + "ARCH_BASE_MACHINE": "64bit", + "BACKEND": "qemu", + "BUILD": "20230423-Rocky-9.1-x86_64.0", + "CLONED_FROM": "https://openqa.rockylinux.org/tests/22735", + "CURRREL": "9", + "DEPLOY_UPLOAD_TEST": "install_default_upload", + "DESKTOP": "gnome", + "DISTRI": "rocky", + "FLAVOR": "dvd-iso", + "HDDSIZEGB": "15", + "ISO": "Rocky-9.1-20221214.1-x86_64-dvd.iso", + "LOCATION": "https://download.rockylinux.org/pub/rocky/9.1/BaseOS", + "MACHINE": "64bit", + "NAME": "00000023-rocky-9.1-dvd-iso-x86_64-Build20230423-Rocky-9.1-x86_64.0-install_default_upload@64bit", + "NICTYPE_USER_OPTIONS": "net=172.16.2.0/24", + "PACKAGE_SET": "default", + "PART_TABLE_TYPE": "mbr", + "POSTINSTALL": "_collect_data", + "QEMUCPU": "Nehalem", + "QEMUCPUS": "2", + "QEMURAM": "2048", + "QEMU_HOST_IP": "172.16.2.2", + "QEMU_VIDEO_DEVICE": "virtio-vga", + "QEMU_VIRTIO_RNG": "1", + "STORE_HDD_1": "disk_dvd-iso_64bit.qcow2", + "TEST": "install_default_upload", + "TEST_SUITE_NAME": "install_default_upload", + "TEST_TARGET": "ISO", + "VERSION": "9.1", + "WORKER_CLASS": "qemu_x86_64", + "XRES": "1024", + "YRES": "768" + }, + "state": "running", + "t_finished": null, + "t_started": "2023-04-23T03:02:06", + "test": "install_default_upload" + } +} +``` + +**NOTE: In the above job information you can clearly see the job was cloned from `https://openqa.rockylinux.org/tests/22735`.** + +## Advanced `openqa-clone-job` + +You can, of course, perform more elaborate operations while cloneing a job either from your local instance or from the production instance. Typically, this might be done to modify some of the job POST variables in the cloned job while keeping all other variables unchanged. + +### Changing variable during clone + +Here is an example where the ISO is changed in the cloned job... + +```bash +$ openqa-clone-job --from https://openqa.rockylinux.org --skip-download 22735 ISO=Rocky-9.1-x86_64-dvd.iso +Cloning children of rocky-9.1-dvd-iso-x86_64-Build20230423-Rocky-9.1-x86_64.0-install_default_upload@64bit +Created job #24: rocky-9.1-dvd-iso-x86_64-Build20230423-Rocky-9.1-x86_64.0-install_default_upload@64bit -> http://localhost/t24 +``` + +### Job overview + +```bash +$ openqa-cli api jobs/overview +[{"id":24,"name":"rocky-9.1-dvd-iso-x86_64-Build20230423-Rocky-9.1-x86_64.0-install_default_upload@64bit"}] +``` + +### Job details + +```bash +$ openqa-cli api jobs/24 | jq '.' +{ + "job": { + "assets": { + "iso": [ + "Rocky-9.1-x86_64-dvd.iso" + ] + }, + "assigned_worker_id": 1, + "blocked_by_id": null, + "children": { + "Chained": [], + "Directly chained": [], + "Parallel": [] + }, + "clone_id": null, + "group": "Rocky", + "group_id": 2, + "has_parents": 0, + "id": 24, + "name": "rocky-9.1-dvd-iso-x86_64-Build20230423-Rocky-9.1-x86_64.0-install_default_upload@64bit", + "parents": { + "Chained": [], + "Directly chained": [], + "Parallel": [] + }, + "parents_ok": 1, + "priority": 50, + "result": "none", + "settings": { + "ARCH": "x86_64", + "ARCH_BASE_MACHINE": "64bit", + "BACKEND": "qemu", + "BUILD": "20230423-Rocky-9.1-x86_64.0", + "CLONED_FROM": "https://openqa.rockylinux.org/tests/22735", + "CURRREL": "9", + "DEPLOY_UPLOAD_TEST": "install_default_upload", + "DESKTOP": "gnome", + "DISTRI": "rocky", + "FLAVOR": "dvd-iso", + "HDDSIZEGB": "15", + "ISO": "Rocky-9.1-x86_64-dvd.iso", + "LOCATION": "https://download.rockylinux.org/pub/rocky/9.1/BaseOS", + "MACHINE": "64bit", + "NAME": "00000024-rocky-9.1-dvd-iso-x86_64-Build20230423-Rocky-9.1-x86_64.0-install_default_upload@64bit", + "NICTYPE_USER_OPTIONS": "net=172.16.2.0/24", + "PACKAGE_SET": "default", + "PART_TABLE_TYPE": "mbr", + "POSTINSTALL": "_collect_data", + "QEMUCPU": "Nehalem", + "QEMUCPUS": "2", + "QEMURAM": "2048", + "QEMU_HOST_IP": "172.16.2.2", + "QEMU_VIDEO_DEVICE": "virtio-vga", + "QEMU_VIRTIO_RNG": "1", + "STORE_HDD_1": "disk_dvd-iso_64bit.qcow2", + "TEST": "install_default_upload", + "TEST_SUITE_NAME": "install_default_upload", + "TEST_TARGET": "ISO", + "VERSION": "9.1", + "WORKER_CLASS": "qemu_x86_64", + "XRES": "1024", + "YRES": "768" + }, + "state": "running", + "t_finished": null, + "t_started": "2023-04-23T03:08:03", + "test": "install_default_upload" + } +} +``` + +## Difference between Basic and Advanced `openqa-clone-job` + +You should notice that the only substantive difference between the two cloned jobs is the ISO that is used to run the `install_default_upload` test... + +```bash +$ openqa-cli api jobs/23 | jq '.job.settings.ISO' +"Rocky-9.1-20221214.1-x86_64-dvd.iso" + +$ openqa-cli api jobs/24 | jq '.job.settings.ISO' +"Rocky-9.1-x86_64-dvd.iso" +``` + +## References + +[openQA Documentation](http://open.qa/documentation/) + +{% include 'content_bottom.md' %} diff --git a/docs/documentation/dev_guides/wiki_development_boxes.md b/docs/documentation/dev_guides/wiki_development_boxes.md new file mode 100644 index 0000000..66b23fc --- /dev/null +++ b/docs/documentation/dev_guides/wiki_development_boxes.md @@ -0,0 +1,53 @@ +--- +title: Create live devbox for wiki.rockylinux.org +author: + - Lukas Magauer + - Al Bowles +revision_date: 2023-04-29 +--- + +# How to create a live system to work on the documentation + +There are several ways how to setup your development environment, here are the currently used once by the testing team: + +## Vagrant + +For now here is the link to [Trevor's vagrant box setup](https://github.com/tcooper/rocky-linux-wikibox), this might be merged here in the future! + +## Manual setup for WSL and toolbox + +### WSL specific + +Create a WSL machine like [described here](https://docs.rockylinux.org/guides/interoperability/import_rocky_to_wsl), make sure to give it a name like `rocky-wiki`. + +### toolbox specific + +- Make sure you have `toolbox` installed on your system +- Create a toolbox `toolbox create rocky-wiki` (on Rocky Linux 8 and 9 machines this will create either a Rocky Linux 8 or 9 toolbox container) + +### Container setup for both + +- Run `dnf -y update` to update the system +- Run `dnf -y install git python39-pip` to install Python 3.9 and pip (Python 3.9 is default for Rocky Linux 9, the package is called `python3-pip` there) +- Run `python3.9 -m pip install -U pip` to update pip +- Clone the repo `git clone ` +- And get into the folder of the repo `cd ` +- Sometimes you will need to switch the branch here +- Install all the requirements of the repo `python3.9 -m pip install -r requirements.txt` +- If you just want to look at the output run `mkdocs serve 2>&1 | tee ./mkdocs.serve.log` + +To develop then, the easiest way is to use VS Code with the [Remote - WSL](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-wsl), where you have to open the repo in the container (click on the Remote symbol on the lower left, 'Open folder in WSL...'). + +For toolbox just place the repo inside your user profile and you will be able to access it with VS Code inside and outside of the toolbox container. + +And finally run `mkdocs serve 2>&1 | tee ./mkdocs.serve.log` in the terminal of this VS Code session. Then you are ready to start changing stuff! + +## Docker + +From the root of this repository on a machine with Docker installed, run + + docker-compose up + +When the container finishes starting up, you can access the documentation in your web browser at [http://localhost:8000](http://localhost:8000). + +{% include 'rc_content_bottom.md' %} diff --git a/docs/documentation/qa_test_cases.md b/docs/documentation/qa_test_cases.md new file mode 100644 index 0000000..dc65d42 --- /dev/null +++ b/docs/documentation/qa_test_cases.md @@ -0,0 +1,65 @@ +--- +title: QA:Test Cases +revision_date: 2022-06-23 +rc: + prod: Rocky Linux +--- + +This page lists all test cases in work and who is working on them... + +## Initialization Requirements + +| Requirement | Test Case | Assignee | Status | +| --------------------------------------------------- | ------------------------------------------------------------------------ | ----------------------- | --------------------------------------- | +| Release-blocking images must boot
[{{ rc.prod }} 8](8_release_criteria.md#release-blocking-images-must-boot) [{{ rc.prod }} 9](9_release_criteria.md#release-blocking-images-must-boot) | [QA:Testcase Boot Methods Boot ISO](Testcase_Boot_Methods_Boot_Iso.md) | @tcooper | template exists, openQA covered (ref) | +| Release-blocking images must boot
[{{ rc.prod }} 8](8_release_criteria.md#release-blocking-images-must-boot) [{{ rc.prod }} 9](9_release_criteria.md#release-blocking-images-must-boot) | [QA:Testcase Boot Methods DVD](Testcase_Boot_Methods_Dvd.md) | @tcooper | template exists, openQA covered (ref) | +| Basic Graphics Mode behaviors
[{{ rc.prod }} 8](8_release_criteria.md#basic-graphics-mode-behaviors) | [QA:Testcase Basic Graphics Mode](Testcase_Basic_Graphics_Mode.md) | @tcooper | openQA TestCase | +| VNC Graphics Mode behaviors
[{{ rc.prod }} 9](9_release_criteria.md#vnc-graphics-mode-behaviors) | [QA:Testcase VNC Graphics Mode](Testcase_VNC_Graphics_Mode.md) | @tcooper | openQA TestCase | +| No Broken Packages
[{{ rc.prod }} 8](8_release_criteria.md#no-broken-packages) [{{ rc.prod }} 9](9_release_criteria.md#no-broken-packages) | [QA:Testcase Media Repoclosure](Testcase_Media_Repoclosure.md)
[QA:Testcase Media File Conflicts](Testcase_Media_File_Conflicts.md) | @tcooper | manual using scripts or automated in CI | +| Repositories Must Match Upstream
[{{ rc.prod }} 8](8_release_criteria.md#repositories-must-match-upstream) [{{ rc.prod }} 9 ](9_release_criteria.md#repositories-must-match-upstream) | [QA:Testcase repocompare](Testcase_Repo_Compare.md) | @tcooper | manual using Skip's repocompare | +| Debranding
[{{ rc.prod }} 8](8_release_criteria.md#debranding) [{{ rc.prod }} 9](9_release_criteria.md#debranding) | [QA:Testcase Debranding Analysis](Testcase_Debranding.md) | @tcooper | manual using scripts or automated in CI | + + +## Installer Requirements + +| Requirement | Test Case | Assignee | Status | +| --------------------------------------------------- | ----------------------------------------------------------------------------------------- | ----------------------- | --------------------------------------- | +| Media Consistency Verification | [QA:Testcase Media USB dd](Testcase_Media_USB_dd.md)
[QA:Testcase Boot Methods Boot ISO](Testcase_Boot_Methods_Boot_Iso.md)
[QA:Testcase Boot Methods DVD](Testcase_Boot_Methods_Dvd.md) | @raktajino | | +| Packages and Installer Sources | [QA:Testcase Packages and Installer Sources](Testcase_Packages_Installer_Sources.md) | @raktajino | Implemented in openQA, document | +| NAS (Network Attached Storage) | [QA:Testcase Network Attached Storage](Testcase_Network_Attached_Storage.md) | @raktajino | | +| Installation Interfaces | [QA:Testcase Installation Interfaces](Testcase_Installation_Interfaces.md) | @raktajino | Implemented in openQA, document | +| Minimal Installation | [QA:Testcase Minimal Installation](Testcase_Minimal_Installation.md) | @raktajino | Implemented in openQA, document | +| Kickstart Installation | [QA:Testcase Kickstart Installation](Testcase_Kickstart_Installation.md) | @raktajino | Implemented in openQA, document | +| Disk Layouts | [QA:Testcase Disk Layouts](Testcase_Disk_Layouts.md) | @raktajino | Implemented in openQA, document | +| Firmware RAID | [QA:Testcase Firmware RAID](Testcase_Firmware_RAID.md) | @raktajino | | +| Bootloader Disk Selection | [QA:Testcase Bootloader Disk Selection](Testcase_Bootloader_Disk_Selection.md) | @raktajino | | +| Storage Volume Resize | [QA:Testcase Storage Volume Resize](Testcase_Storage_Volume_Resize.md) | @raktajino | Implemented in openQA, document | +| Update Image | [QA:Testcase Update Image](Testcase_Update_Image.md) | @raktajino | Implemented in openQA, document | +| Installer Help | [QA:Testcase Installer Help](Testcase_Installer_Help.md) | @raktajino | Implemented in openQA, document | +| Installer Translations | [QA:Testcase Installer Translations](Testcase_Installer_Translations.md) | @raktajino | Implemented in openQA, document | + + +## Cloud Image Requirements + +| Requirement | Test Case | Assignee | Status | +| --------------------------------------------------- | ------------------------------------------------------------------------ | ----------------------- | --------------------------------------- | +| Images Published to Cloud Providers | [QA:Testcase TBD](Testcase_Template.md) | @tbd | | + + +## Post-Installation Requirements + +| Requirement | Test Case | Assignee | Status | +|--------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------|----------|----------------------------------------------------------------------| +| System Services | [QA:Testcase System Services](Testcase_Post_System_Services.md) | @lumarel | manual guide documented or needs new openQA testcase | +| Keyboard Layout | [QA:Testcase Keyboard Layout](Testcase_Post_Keyboard_Layout.md) | @lumarel | implemented in openQA | +| SELinux Errors (Server) | [QA:Testcase SELinux Errors on Server](Testcase_Post_SELinux_Errors_Server.md) | @lumarel | implemented in openQA | +| SELinux and Crash Notifications (Desktop Only) | [QA:Testcase SELinux Errors on Desktop](Testcase_Post_SELinux_Errors_Desktop.md) | @lumarel | partly implemented in openQA | +| Default Application Functionality (Desktop Only) | [QA:Testcase Application Functionality](Testcase_Post_Application_Functionality.md) | @lumarel | manual guide documented | +| Default Panel Functionality (Desktop Only) | [QA:Testcase GNOME UI Functionality](Testcase_Post_GNOME_UI_Functionality.md) | @lumarel | implemented in openQA, additionally documented for manual inspection | +| Dual Monitor Setup (Desktop Only) | [QA:Testcase Multimonitor Setup](Testcase_Post_Multimonitor_Setup.md) | @lumarel | manual guide documented | +| Artwork and Assets (Server and Desktop) | [QA:Testcase Artwork and Assets](Testcase_Post_Artwork_and_Assets.md) | @lumarel | implemented in openQA, additionally documented for manual inspection | +| Packages and Module Installation | [QA:Testcase Basic Package installs](Testcase_Post_Package_installs.md)
[QA:Testcase Module Streams](Testcase_Post_Module_Streams.md) | @lumarel | partly implemented in openQA, manual guide documented | +| Identity Management (FreeIPA) | [QA:Testcase Identity Management](Testcase_Post_Identity_Management.md) | @lumarel | manual guide documented, PR open for openQA implementation | + + +{% include 'content_bottom.md' %} diff --git a/docs/guidelines/.pages b/docs/guidelines/.pages index 06765ee..940ce1a 100644 --- a/docs/guidelines/.pages +++ b/docs/guidelines/.pages @@ -1,4 +1,5 @@ --- nav: - ... | index.md + - Release Criteria & Status: release_criteria ... diff --git a/docs/guidelines/release_criteria/.pages b/docs/guidelines/release_criteria/.pages new file mode 100644 index 0000000..313559f --- /dev/null +++ b/docs/guidelines/release_criteria/.pages @@ -0,0 +1,5 @@ +--- +nav: + - ... | release_criteria.md + - Rocky Linux 8: r8 + - Rocky Linux 9: r9 diff --git a/docs/guidelines/release_criteria/.qa_testing_summary_template.md b/docs/guidelines/release_criteria/.qa_testing_summary_template.md new file mode 100644 index 0000000..de13a8f --- /dev/null +++ b/docs/guidelines/release_criteria/.qa_testing_summary_template.md @@ -0,0 +1,64 @@ +--- +title: Release Status Template +author: Trevor Cooper +revision_date: 2022-05-22 +rc: + prod: Rocky Linux + ver: 9 + level: Final +--- + +# {{ rc.prod }} {{ rc.ver }} QA and Testing Summary +Last updated: + +## Scope +This document will record a summary of all QA and Testing results for {{ rc.prod }} {{ rc.9 }} release. It is only a record of success and/or failure. Solution discussion should take place elsewhere. + +## Reference +- Please check RHEL 9 Release Notes **BEFORE** marking issue here as **FAIL**. + +## SOP +- Please include **PASS**, **FAIL**, **NOTABUG**, **INVESTIGATE** or **UPSTREAM** as appropriate in all entries. +- Please only provide brief summary. Details should go to Rocky Pastebin, links here is OK. +- Please leave your MM @handle on all items you have done or are working on so we can talk to you to get resolution. +- If the item you have reported is related to a QA:Testcase please mention it. +- If you think the item you have reported should be a QA:Testcase, even if it's not a current requirement, suggest a title and create an issue in the wiki repository so we can add it. + +## INVESTIGATE +This is a list of items that are being INVESTIGATEd further before being assigned a PASS, FAIL, NOTABUG or UPSTREAM status. +PLEASE add your MM handle if you are working on this item to minimize duplication of work. More than one handle is allowed but please communicate. + +- QA:Testcase Basic Graphics Mode - [INVESTIGATE] - @tcooper +- QA:Testcase Boot Methods Boot Iso - [INVESTIGATE] - @neil +- QA:Testcase Boot Methods DVD - [INVESTIGATE] - @neil +- QA:Testcase Debranding - [INVESTIGATE] - @tcooper +- QA:Testcase Media Consistency Verification - [INVESTIGATE] - @tcooper +- QA:Testcase Media File Conflicts - [INVESTIGATE] - @tcooper +- QA:Testcase Media Repoclosure - [INVESTIGATE] - @tcooper +- QA:Testcase Storage Volume Resize - [INVESTIGATE] - @raktajino +- QA:Testcase Update Image - [INVESTIGATE] - @raktajino +- QA:Testcase boot/install minimal x86_64 over DVD/Bluray - [INVESTIGATE] - @atomicturtle +- QA:Testcase_Mediacheck - [INVESTIGATE] - @tcooper + +## UPSTREAM +This is a list of items that have been verified to be replicated UPSTREAM in RHEL {{ rc.9 }} and/or are described clearly in the RHEL 9 Release Notes. + +- QA:Testcase_Some_Testcase - [UPSTREAM] - @your_mm_handle - + +## FAIL +This is a list of items that have been verified to FAIL the QA:Testcase. In addition to recording who did the test please indicate if the item is BLOCKING release or not. + +- QA:Testcase_Some_Testcase - [FAIL] - @your_mm_handle - + +## NOTABUG +This is a list of items that have been verified as less than optimal but are expected and NOTABUG. + +- QA:Testcase_Some_Testcase - [NOTABUG] - @your_mm_handle - + +## PASS +This is a list of item that have been verified as PASSing the QA:Testcase named (or proposed). + +- QA:Testcase_Some_Testcase - [PASS] - @your_mm_handle - + +## OTHER NOTABLE ITEMS + diff --git a/docs/guidelines/release_criteria/.ver_qa_testing_go_no_go_template.md b/docs/guidelines/release_criteria/.ver_qa_testing_go_no_go_template.md new file mode 100644 index 0000000..8590d13 --- /dev/null +++ b/docs/guidelines/release_criteria/.ver_qa_testing_go_no_go_template.md @@ -0,0 +1,89 @@ +--- +title: Rocky Linux TBD QA and Testing GO / NO GO Status +author: Trevor Cooper +revision_date: 2022-05-18 +rc: + prod: Rocky Linux + ver: TBD +--- + +# {{ rc.prod }} {{ rc.ver }} QA and Testing : GO / NO GO Status +This document summarizes the GO / NO GO Status for Release of Rocky Linux 8.6 from perspective of the QA / Testing efforts. It is based largely on the Release Criteria (https://wiki.rockylinux.org/team/testing/release_criteria/) as was started as an import of that document. If there are differences between the official Release Critieria document and this document the official Release Criteria document will override and this document shall be updated. + +As a reminder, the objective of a release (major or minor) is to provide a solid Enterprise Linux release that is suitable to: +- Meet the needs of end users +- Meet the needs of enterprises big or small + + +## SUMMARY + +| Category | Proportion | Remaining Items | +| ----------------- | ----------------- | --------------- | +| TO_CONFIRM | 0 / 29 (0%) | | +| PASS | 0 / 29 (0%) | | +| FAIL_NON_BLOCKING | 0 / 29 (0%) | | +| FAIL_BLOCKING | 0 / 29 (0%) | | + + +## SOP +In this document each requirement is described and status is specified in the title. + +Current choices are... + +### TO_CONFIRM +- this means the item may be INCOMPLETE, PASS, FAIL_NON_BLOCKING or FAIL_BLOCKING and must be verified + +### PASS +- this means that the release criteria has been met and is not a blocker + +### FAIL_NON_BLOCKING +- this means the release criter has not been met but is non-blocking + +### FAIL_BLOCKING +- this means the release criteria has not been met and is blocking + +In this document criteria status should include who completed the item and generally how it was complete. + +Examples... + +- PASS - (@tcooper, virt only, manual) +- PASS - (@lumarel, @raktajino, @tcooper, semi-automatic , openQA) + + +## Initialization Requirements + +- Release-blocking images must boot - PASS - (@neil, @atomicturtle) +- Optical Media Requirements - PASS - (@atomicturtle) +- Basic Graphics Mode behaviors - PASS - (@tcooper, virt only, manual) +- No Broken Packages - PASS - (@tcooper, scripted, manual) +- Repositories Must Match Upstream - TO_CONFIRM - (@tcooper, manual) +- Debranding - PASS - (@tcooper, scripted, manual) +- Media Consistency Verification - PASS - (@tcooper, scripted, manual) +- Packages and Installer Sources - PASS - (@lumarel, semi-automatic, openQA test) +- NAS (Network Attached Storage) - TO_CONFIRM - (@lumarel?, semi-automatic, openQA dual-host test) +- Installation Interfaces - PASS - (@lumarel, @raktajino, @tcooper, semi-automatic, openQA, @atomicturtle, manual?, SCAP) +- Minimal Installation - PASS - (@lumarel, @raktajino, @tcooper, semi-automatic , openQA) +- Kickstart Installation - PASS - (@label, @tcooper, manual, createhdds) +- Disk Layouts - PASS - (@lumarel, @raktajino, @tcooper, semi-automatic, openQA) +- Firmware RAID - TO_CONFIRM - (@tbd, missing hardware support?) +- Bootloader Disk Selection - PASS - (@raktajino, manual) +- Storage Volume Resize - PASS - (@lumarel, @raktajino, @tcooper, semi-automatic, openQA) +- Update Image - PASS - (@raktajino,@tcooper, semi-automatic, openQA) +- Installer Help - PASS - (@lumarel, @raktajino, @tcooper, semi-automatic, openQA) +- Installer Translations - PASS - (@lumarel, @raktajino, @tcooper, semi-automatic, openQA) + + +## Cloud Image Requirements +- Images Published to Cloud Providers - FAIL_NON_BLOCKING - (@neil) + + +## Post-Installation Requirements +- System Services - PASS - (@lumarel, semi-automatic, openQA) +- Keyboard Layout - PASS - (@lumarel, semi-automatic, openQA) +- SELinux Errors (Server) - PASS - (@lumarel, semi-automatic, openQA) +- SELinux and Crash Notifications (Desktop Only) - PASS - (@lumarel, semi-automatic, openQA) +- Default Application Functionality (Desktop Only) - PASS - (@lumarel, semi-automatic, openQA) +- Default Panel Functionality (Desktop Only) - PASS - (@lumarel, semi-automatic, openQA) +- Dual Monitor Setup (Desktop Only) - PASS - (@lumarel, semi-automatic, openQA) +- Artwork and Assets (Server and Desktop) - PASS - (@lumarel, semi-automatic, openQA) +- Packages and Module Installation - PASS - (@lumarel, semi-automatic, openQA) diff --git a/docs/guidelines/release_criteria/r8/.pages b/docs/guidelines/release_criteria/r8/.pages new file mode 100644 index 0000000..cff21a5 --- /dev/null +++ b/docs/guidelines/release_criteria/r8/.pages @@ -0,0 +1,6 @@ +--- +nav: + - ... | index.md + - Rocky Linux 8 Release Criteria: 8_release_criteria.md + - Rocky Linux 8.6 QA and Testing Summary: 8.6_qa_testing_summary.md + - Rocky Linux 8.6 GO / NO-GO Status: 8.6_qa_testing_go_no_go.md diff --git a/docs/guidelines/release_criteria/r8/8.6_qa_testing_go_no_go.md b/docs/guidelines/release_criteria/r8/8.6_qa_testing_go_no_go.md new file mode 100644 index 0000000..26ca71c --- /dev/null +++ b/docs/guidelines/release_criteria/r8/8.6_qa_testing_go_no_go.md @@ -0,0 +1,90 @@ +--- +title: Rocky Linux 8.6 QA and Testing GO / NO GO Status +author: Trevor Cooper +revision_date: 2022-05-18 +rc: + prod: Rocky Linux + ver: 8.6 + level: Final +--- + +# Rocky Linux 8.6 QA and Testing : GO / NO GO Status +This document summarizes the GO / NO GO Status for Release of Rocky Linux 8.6 from perspective of the QA / Testing efforts. It is based largely on the Release Criteria (https://wiki.rockylinux.org/team/testing/release_criteria/) as was started as an import of that document. If there are differences between the official Release Critieria document and this document the official Release Criteria document will override and this document shall be updated. + +As a reminder, the objective of a release (major or minor) is to provide a solid Enterprise Linux release that is suitable to: +- Meet the needs of end users +- Meet the needs of enterprises big or small + + +## SUMMARY + +| Category | Proportion | Remaining Items | +| --------------------- | ----------------- | --------------- | +| **TO_CONFIRM** | 3 / 29 (10%) | | +| **PASS** | 25 / 29 (86%) | | +| **FAIL_NON_BLOCKING** | 1 / 29 (3%) | cloud-images | +| FAIL_BLOCKING | 0 / 29 (0%) | | + + +## SOP +In this document each requirement is described and status is specified in the title. + +Current choices are... + +### TO_CONFIRM +- this means the item may be **[INCOMPLETE]**, **[PASS]**, **[FAIL_NON_BLOCKING[]** or **[FAIL_BLOCKING]** and must be verified + +### PASS +- this means that the release criteria has been met and is not a blocker + +### FAIL_NON_BLOCKING +- this means the release criter has not been met but is non-blocking + +### FAIL_BLOCKING +- this means the release criteria has not been met and is blocking + +In this document criteria status should include who completed the item and generally how it was complete. + +Examples... + +- **[PASS]** - (@tcooper, virt only, manual) +- **[PASS]** - (@lumarel, @raktajino, @tcooper, semi-automatic , openQA) + + +## Initialization Requirements + +- Release-blocking images must boot - **[PASS]** - (@neil, @atomicturtle) +- Optical Media Requirements - **[PASS]** - (@atomicturtle) +- Basic Graphics Mode behaviors - **[PASS]** - (@tcooper, virt only, manual) +- No Broken Packages - **[PASS]** - (@tcooper, scripted, manual) +- Repositories Must Match Upstream -**TO_CONFIRM**- (@tcooper, manual) +- Debranding - **[PASS]** - (@tcooper, scripted, manual) +- Media Consistency Verification - **[PASS]** - (@tcooper, scripted, manual) +- Packages and Installer Sources - **[PASS]** - (@lumarel, semi-automatic, openQA test) +- NAS (Network Attached Storage) -**TO_CONFIRM**- (@lumarel?, semi-automatic, openQA dual-host test) +- Installation Interfaces - **[PASS]** - (@lumarel, @raktajino, @tcooper, semi-automatic, openQA, @atomicturtle, manual?, SCAP) +- Minimal Installation - **[PASS]** - (@lumarel, @raktajino, @tcooper, semi-automatic , openQA) +- Kickstart Installation - **[PASS]** - (@label, @tcooper, manual, createhdds) +- Disk Layouts - **[PASS]** - (@lumarel, @raktajino, @tcooper, semi-automatic, openQA) +- Firmware RAID -**TO_CONFIRM**- (@tbd, missing hardware support?) +- Bootloader Disk Selection - **[PASS]** - (@raktajino, manual) +- Storage Volume Resize - **[PASS]** - (@lumarel, @raktajino, @tcooper, semi-automatic, openQA) +- Update Image - **[PASS]** - (@raktajino,@tcooper, semi-automatic, openQA) +- Installer Help - **[PASS]** - (@lumarel, @raktajino, @tcooper, semi-automatic, openQA) +- Installer Translations - **[PASS]** - (@lumarel, @raktajino, @tcooper, semi-automatic, openQA) + + +## Cloud Image Requirements +- Images Published to Cloud Providers - **FAIL_NON_BLOCKING**- (@neil) + + +## Post-Installation Requirements +- System Services - **[PASS]** - (@lumarel, semi-automatic, openQA) +- Keyboard Layout - **[PASS]** - (@lumarel, semi-automatic, openQA) +- SELinux Errors (Server) - **[PASS]** - (@lumarel, semi-automatic, openQA) +- SELinux and Crash Notifications (Desktop Only) - **[PASS]** - (@lumarel, semi-automatic, openQA) +- Default Application Functionality (Desktop Only) - **[PASS]** - (@lumarel, semi-automatic, openQA) +- Default Panel Functionality (Desktop Only) - **[PASS]** - (@lumarel, semi-automatic, openQA) +- Dual Monitor Setup (Desktop Only) - **[PASS]** - (@lumarel, semi-automatic, openQA) +- Artwork and Assets (Server and Desktop) - **[PASS]** - (@lumarel, semi-automatic, openQA) +- Packages and Module Installation - **[PASS]** - (@lumarel, semi-automatic, openQA) diff --git a/docs/guidelines/release_criteria/r8/8.6_qa_testing_summary.md b/docs/guidelines/release_criteria/r8/8.6_qa_testing_summary.md new file mode 100644 index 0000000..31b69ac --- /dev/null +++ b/docs/guidelines/release_criteria/r8/8.6_qa_testing_summary.md @@ -0,0 +1,121 @@ +--- +title: Rocky Linux 8.6 QA and Testing Summary +author: Trevor Cooper +revision_date: 2022-05-18 +rc: + prod: Rocky Linux + ver: 8.6 + level: Final +--- + +# Rocky Linux 8.6 QA and Testing Summary +**_Last updated: Fri May 13 17:36:41 UTC 2022_** + + +## Scope +This document will record a summary of all QA and Testing results for Rocky Linux 8.6 release. It is only a record of success and/or failure. Solution discussion should take place elsewhere. + + +## SOP +- Please include PASS, FAIL, NOTABUG, INVESTIGATE or UPSTREAM as appropriate in all entries. +- Please only provide brief summary. Details should go to Rocky Pastebin, links here is OK. +- Especially for an negative result please leave your MM @handle so we can talk to you to get resolution. +- If the item you are reported is related to a QA:Testcase please mention it. If it should be a QA:Testcase, even if it's not a current requirement, suggest a title and create an issue in the wiki repository so we can add it. + + +## INVESTIGATE +- errors in all tests in openQA - INVESTIGATE - see openQA section below. +- INVESTIGATE whether kdump issue affects qcows with encrypted partitions in createhdds. These are pre-reqs for openQA multi-machine tests. NOTE: This is addressed below in section - @tcooper +- KDE and XFCE Life images are broken - INVESTIGATE - @label + + +## UPSTREAM +- Anaconda error when specific steps get executed in the right order (configure network -> disable kdump -> select some os install group -> configure default storage -> configure the storage a second time, but this time with encryption enabled, confirmed via several openqa test suites and manual installation on ESXi) - retested in RC1_2 - UPSTREAM - @lumarel +- Issue repeated on RHEL8.6 - @atomicturtle +- Issue reported to RH https://bugzilla.redhat.com/show_bug.cgi?id=2085321 - @stack + + +## FAIL +- ESXi secureboot (x86_64) still FAILing, but expected - @lumarel + + +## NOTABUG +- Minimal: Selecting a SCAP profile with dependencies not available (aide, etc), selecting Ignore dependency during installation will crash anaconda at the final oscap check. NOTABUG, this is for documentation - @atomicturtle +- Minimal ISO: is missing the source for rsyslog again, and somehow also doesn't pull it in when installing, which means it is missing it after the install (doesn't happen for boot ISO and dvd1 ISO) - NOTABUG (per @label) - @lumarel +- Minimal ISO: if the server base environment is installed with the minimal iso and cockpit is enabled after the installation, the SELinux submenu shows an error "semanage: command not found" (doesn't happen with boot/dvd-iso) - also marked as expected - NOTABUG (per @label) - @lumarel + + +## Manual success reported in MM +- minimal install from minimal ISO fine - PASS - @Luna Jernberg +- workstation (x86_64) install with applications fine - retested in RC1_2 - PASS - @lumarel +- all repos are available with the exact naming as they are in the rocky-repos package (nfv needs fix for that) - retested in RC1_2 - PASS - @lumarel +- packer build for 8.6 worked flawlessly - retested by @neil in RC1_2 - PASS - @gmazrael +- security profiles look good in anaconda UI - PASS - @atomicturtle (need openQA testing) +- minimal and dvd recognized as Rocky Linux 8 in KVM - PASS - @atomicturtle +- CIS profiles confirmed good in lvl1/2 in anaconda - PASS - @atomicturtle +- DISA profiles confirmed good in anaconda - PASS - @atomicturtle +- DVD: libvirt correctly boots when Rocky Linux 8 profile is selected - PASS - @atomicturtle +- SELinux checks on Server (x86_64) (letting it run for an hour and run sealert -a /var/log/audit/audit.log) - everything okay - retested in RC1_2 - PASS - @lumarel +- SELinux checks on Desktop (x86_64) (start the GNOME SETroubleshooter after some minutes of running) - everything okay - retested in RC1_2 - PASS - @lumarel +- DVD: Anaconda manual network configuration, and PCI-DSS SCAP profile selected confirmed good - PASS - @atomicturtle +- QA:Testcase_Mediacheck - PASS for all x86_64 ISOs - @tcooper +- QA:Testcase_Mediacheck - PASS for all aarch64 ISOs - @tcooper +- QA:Testcase Media Repoclosure - PASS for minimal & dvd1 for x86_64 & aarch64 (confirms RelEng results) - @tcooper +- QA:Testcase Media File Conflicts - PASS for minimal for x86_64 & aarch64 (0 file conflicts found and 0 package conflicts found) - @tcooper +- QA:Testcase Basic Graphics Mode - PASS - verified manually for Rocky-8.6-x86_64-dvd1.iso in VirtualBox on macOS X - @tcooper +- DVD: Anaconda install with 3rd party repo, encrypted filesystem, HIPAA SCAP profile selected, confirmed good - PASS - @atomicturtle +- Upgrade tests on several test machines from 8.5 to 8.6, no issues no SELinux alerts - PASS - @lumarel +- All module streams except perl:5.32 and log4j:2 correctly have the dependencies set and packages look to be built correctly - PASS - @lumarel +- log4j module stream was broken, (should be able to hook against java-8 and 11) - got fixed now in RC1_2 - PASS - @lumarel +- Anything perl 5.32 (module stream) was broken - got fixed in RC1_2 - PASS - @nazunalika +- Greenbone appliance installation test (https://rpa.st/DQNA) - PASS - @atomicturtle +- QA:Testcase Debranding for RC2 content from koji (srpms, kernel-rt and pcs are not all on the dvd1) - 46/47 PASS , 1 FALSE PASS - https://rpa.st/raw/QK3A - @tcooper +- QA:Testcase Media Consistency Verification (not written yet) for all RC2 ISOs x86_64, aarch64 - PASS - @tcooper +- QA:Testcase Media File Conflicts - EXPECTED(per @label) for Rocky-8.6-x86_64-dvd1.iso (4 file conflicts found and 13 package conflicts found, these appear to be same as 8.5 conflict between mariadb and mysql packages/files, full results - https://rpa.st/raw/ZWPQ) - @tcooper +- QA:Testcase Media File Conflicts - EXPECTED(per @label) for Rocky-8.6-aarch64-dvd1.iso (modular dependency problems, 3 file conflicts found 4 package conflicts found, full results - https://rpa.st/raw/KOFQ) - @tcooper +- QA:Testcase Media File Conflicts for both x86_64 (https://rpa.st/raw/NLGA) and aarch64 (https://rpa.st/raw/4SFQ) are essentially unchanged and remain - EXPECTED(per @label) with RC1_2 ISOs. - @tcooper +- OpenQA tests @lumarel - there are errors from the test cases, but everything image and repo related looks good - PASS - @lumarel +- the dvd1 iso of aarch64 doesn't show an workstation base environment - it doesn't have an workstation environment - PASS - @lumarel +- Installs of aarch64 systems of all 3 isos look good and installs with all base environments work as expected from these - PASS - @lumarel +- Live Image Workstation and Workstation Lite looks good - PASS - @lumarel +- QA:Testcase Boot Methods Boot Iso - PASS - @neil +- QA:Testcase Boot Methods DVD - PASS - @neil +- QA: Testcase boot/install minimal x86_64 over DVD/Bluray (burned with fedora mediawriter) on G752 ASUS laptop - PASS - @atomicturtle +- Container images for x86_64 and aarch64 work as expected in Docker, Podman and WSL - PASS - @lumarel +- QA: Testcase Storage Volume Resize - PASS - @raktajino https://rpa.st/MQSA +- QA: Testcase Update Image - PASS - @raktajino (manually checked against Fedora's testcase (https://fedoraproject.org/wiki/QA:Testcase_Anaconda_updates.img_via_URL), needles also still match in openQA) + + +## openQA summary +- errors in all tests in openQA - INVESTIGATE +- RC1_1 run 1: + - @lumarel: https://rpa.st/CCPQ + - @raktajino: https://rpa.st/5RVA +- RC1_1 run 2 + - @lumarel: https://rpa.st/FWTQ + - @raktajino: +- RC1_2 run 1: + - @lumarel: https://rpa.st/EOGQ +- @raktajino:https://rpa.st/VHLQ + - RC1_2 run 2: +- @raktajino: https://rpa.st/DKCQ +- Upgrade F35 -> F36 needs postgresql-setup --upgrade to convert openqa databse to new format - @alangm +- Per discussion in Testing Team meeting we have 4-8 (ish) issues to fix in openQA (needles and code) to be able to complete all tests. @lumarel has created issues our openQA repo (https://github.com/rocky-linux/os-autoinst-distri-rocky) and we'll pick up and resolve ASAP. + + +##createhdds kickstart file test summary + +Test method: Used packer to build VM. Booted VM. Verified root login. Shutdown VM. + +- UEFI Testing: + - desktop.ks - PASS - Note: resulting image asks for EULA acceptance when booted due to `firstboot --enable` (unsure if that is desired behavior) + - desktopencrypt.ks - PASS - Note: resulting image asks for EULA acceptance when booted due to `firstboot --enable` (unsure if that is desired behavior) + - minimal-uefi.ks - PASS + - server.ks - PASS + - support.ks - PASS +- BIOS Testing: + - desktop.ks - PASS - Note: resulting image asks for EULA acceptance when booted due to `firstboot --enable` (unsure if that is desired behavior) + - desktopencrypt.ks - PASS - Note: resulting image asks for EULA acceptance when booted due to `firstboot --enable` (unsure if that is desired behavior) + - minimal.ks - PASS + - server.ks - PASS + - support.ks - PASS \ No newline at end of file diff --git a/docs/guidelines/release_criteria/r8/8_release_criteria.md b/docs/guidelines/release_criteria/r8/8_release_criteria.md new file mode 100644 index 0000000..59b4000 --- /dev/null +++ b/docs/guidelines/release_criteria/r8/8_release_criteria.md @@ -0,0 +1,277 @@ +--- +title: Rocky Linux 8 Release Criteria +author: + - Trevor Cooper + - Lukas Magauer +revision_date: 2022-05-07 +rc: + prod: Rocky Linux + ver: 8 + level: Final +--- + +# {{ rc.prod }} {{ rc.ver }} {{ rc.level }} Release Objectives + +The objective of a release (major or minor) is to provide a solid Enterprise Linux release that is suitable to: + +- Meet the needs of end users +- Meet the needs of enterprises big or small + +## {{ rc.prod }} {{ rc.ver }} {{ rc.level }} Release Requirements + +In order for {{ rc.prod }} to be released to the general public, a compose must be able to meet all the following criteria as provided in this document. This is allows the decision process to be straightforward and as clear as possible. This document only contains “hard requirement” items. Optional/nice to have items are not to be included in this list. + +There may cases where a requirement cannot be met but only in particular configurations. In these types of cases, the Release Engineering Team should use their judgement to determine whether or not the issue should be considered to block the release. They should consider the number of users likely to be affected by said issue, the severity of the case, if the issue can be avoided with ease (by both informed and uninformed users), and if the problem exists upstream in the current Red Hat Enterprise Linux that the release is based on. + +!!! info "Release-blocking Server" + ...means bugs as it pertains to server functionality can be considered to block a release. This applies to any packages that provide a service such as httpd, nginx, etc. All architectures apply. + +!!! info "Release-blocking Desktop" + ...means bugs as it pertains to desktop functionality (GNOME) can be considered to block a release. This applies to both x86_64 and aarch64. Additional desktops (as provided by EPEL or a SIG) are not considered blockers. + +!!! info "Release-blocking Image" + ...means bugs as it pertains to the images built that can block a release. This applies to the DVD, minimal, and boot images on all architectures. + +### Initialization Requirements + +#### Release-blocking images must boot + +Release-blocking installer images must boot when written to optical media or USB flash drive of appropriate sizes (if applicable) via officially supported methods. It is not the testing team’s responsibility to test optical media, but they can and report back. If a bug is found, it is considered a blocker. + +??? tldr "Optical Media Requirements" + Release-blocking images must boot when written to optical media of an appropriate size. Current size requirements are: boot.iso = 789M, minimal.iso = 2.0G and dvd.iso = 10G. + +??? tldr "Officially supported USB flash drive writing methods" + The following methods of writing USB flash drives are officially support: dd
+ The following methods of writing USB flash drives are _**not**_ supported: rufus + +??? tldr "References" + - Test cases: + - [QA:Testcase Boot Methods Boot ISO](Testcase_Boot_Methods_Boot_Iso.md) + - [QA:Testcase Boot Methods DVD](Testcase_Boot_Methods_Dvd.md) + - [QA:Testcase Media USB dd](Testcase_Media_USB_dd.md) + +#### Basic Graphics Mode behaviors +The generic video driver option (“basic graphics mode”) on all release-blocking installers must function as intended. This means launching the installer or desktop and attempting to use a generic driver. There must be no bugs that prevent the installer from being reached in this configuration on all systems and classes of hardware supported by the enterprise linux kernel. +??? tldr "References" + - Test cases: + - [QA:Testcase Basic Graphics Mode](Testcase_Basic_Graphics_Mode.md) + +#### No Broken Packages +Critical errors, such as undeclared conflicts, unresolved dependencies, or modules relying on packages from another stream will be considered an automatic blocker. There are potential exceptions to this (eg, freeradius cannot be installed on an older perl stream, this is a known issue upstream). +??? tldr "References" + - Test cases: + - [QA:Testcase Media Repoclosure](Testcase_Media_Repoclosure.md) + - [QA:Testcase Media File Conflicts](Testcase_Media_File_Conflicts.md) + +#### Repositories Must Match Upstream +Repositories and the packages within them should match upstream as closely as possible. Notable exceptions would be kmods, kpatch, or what is deemed as “spyware” like insights. Packages that are available from upstream should not have hard requirements on RHSM and packages that have it default built in should be patched out. +??? tldr "References" + - Test cases: + - [QA:Testcase Repo Compare](Testcase_Repo_Compare.md) + - [QA:Testcase Packages No Insights](Testcase_Packages_No_Insights.md) + - [QA:Testcase Packages No RHSM](Testcase_Packages_No_RHSM.md) + +#### Debranding +Assets and functionality that are Red Hat specific should not be included. If they are not patched out, it will be considered an automatic blocker. +??? tldr "References" + - Test cases: + - [QA:Testcase Debranding](Testcase_Debranding.md) + +### Installer Requirements + +#### Media Consistency Verification +This means that the installer’s mechanism for verifying the install medium is intact and must complete successfully, with the assumption that the medium was correctly written. It should return a failure message if this not the case. +??? tldr "References" + - Test cases: + - [QA:Testcase Media USB dd](Testcase_Media_USB_dd.md) + - [QA:Testcase Boot Methods Boot ISO](Testcase_Boot_Methods_Boot_Iso.md) + - [QA:Testcase Boot Methods DVD](Testcase_Boot_Methods_Dvd.md) + +#### Packages and Installer Sources +The installer must be able to use all supported local/remote packages and installer sources. +??? tldr "References" + - Test cases: + - [QA:Testcase Packages and Installer Sources](Testcase_Packages_Installer_Sources.md) + +#### NAS (Network Attached Storage)" +The installer must be able to detect and install to supported NAS devices (if possible and supported by the kernel). +??? tldr "References" + - Test cases: + - [QA:Testcase Network Attached Storage](Testcase_Network_Attached_Storage.md) + +#### Installation Interfaces +The installer must be able to complete an installation using all supported spokes. +??? tldr "References" + - Test cases: + - [QA:Testcase Installation Interfaces](Testcase_Installation_Interfaces.md) + +#### Minimal Installation +A minimal installation (via network) must be able to install the minimal package set. +??? tldr "References" + - Test cases: + - [QA:Testcase Minimal Installation](Testcase_Minimal_Installation.md) + +#### Kickstart Installation +A kickstart installation should succeed, whether from optical/USB media or via the network. +??? tldr "References" + - Test cases: + - [QA:Testcase Kickstart Installation](Testcase_Kickstart_Installation.md) + +#### Disk Layouts +The installer must be able to create and install to any workable partition layout using any file system or format combination offered or supported by the installer. File systems that are not supported by the EL kernel is not tested here (this means btrfs, zfs, both of wish are not supported). +??? tldr "References" + - Test cases: + - [QA:Testcase Disk Layouts](Testcase_Disk_Layouts.md) + +#### Firmware RAID +The installer must be able to detect and install to firmware RAID devices. Note that system-specific bugs do not count as blockers. It is likely that some hardware support might be broken or not available at all. DUDs (driver update disks) are not considered for this criteria. +??? tldr "References" + - Test cases: + - [QA:Testcase Firmware RAID](Testcase_Firmware_RAID.md) + +#### Bootloader Disk Selection +The installer must allow the user to choose which disk the bootloader will be installed to or, if the user so chooses, not to install a bootloader. +??? tldr "References" + - Test cases: + - [QA:Testcase Bootloader Disk Selection](Testcase_Bootloader_Disk_Selection.md) + +#### Storage Volume Resize +Any installer mechanism for resizing storage volumes must correctly attempt the requested operation. This means that if the installer offers a way to resize storage volumes, then it must use the correct resizing tool with the correct parameters. However, it does not require the installer to disallow resizing of unformatted or volumes with an unknown filesystem type. +??? tldr "References" + - Test cases: + - [QA:Testcase Storage Volume Resize](Testcase_Storage_Volume_Resize.md) + +#### Update Image +The installer must be able to use an installer update image retrieved from removable media or a remote package source. This includes DUDs (driver update disks). +??? tldr "References" + - Test cases: + - [QA:Testcase Update Image](Testcase_Update_Image.md) + +#### Installer Help +Any element in the installer which contains a “help” text must display the appropriate help documentation when selected. +??? tldr "References" + - Test cases: + - [QA:Testcase Installer Help](Testcase_Installer_Help.md) + +#### Installer Translations +The installer must correctly display all complete translations that are available for use. +??? tldr "References" + - Test cases: + - [QA:Testcase Installer Translations](Testcase_Installer_Translations.md) + +### Cloud Image Requirements +#### Images Published to Cloud Providers +Release-blocking cloud disk images must be published to appropriate cloud providers (such as Amazon) and they must successfully boot. This also applies to KVM based instances, such as x86 and aarch64 systems. +??? tldr "References" + - Test cases: + - [QA:Testcase TBD](Testcase_Template.md) + +### Post-Installation Requirements + +#### System Services + +All system services present after installation must start properly, with the exception of services that require hardware which is not present. Examples of such services would be: + +- sshd +- firewalld +- auditd +- chronyd + +??? tldr "References" + - Test cases: + - [QA:Testcase System Services](Testcase_Post_System_Services.md) + +#### Keyboard Layout + +If a particular keyboard layout has been configured for the system, that layout must be used: + +- When unlocking storage volumes (encrypted by LUKS) +- When logging in at a TTY console +- When logging in via GDM +- After logging into a GNOME desktop system, if the user does not have their own layout configuration set. + +??? tldr "References" + - Test cases: + - [QA:Testcase Keyboard Layout](Testcase_Post_Keyboard_Layout.md) + +#### SELinux Errors (Server) + +There must be no SELinux denial logs in /var/log/audit/audit.log + +??? tldr "References" + - Test cases: + - [QA:Testcase SELinux Errors on Server installations](Testcase_Post_SELinux_Errors_Server.md) + +#### SELinux and Crash Notifications (Desktop Only) + +There must be no SELinux denial notifications or crash notifications on boot, during installation, or during first login. + +??? tldr "References" + - Test cases: + - [QA:Testcase SELinux Errors on Desktop clients](Testcase_Post_SELinux_Errors_Desktop.md) + +#### Default Application Functionality (Desktop Only) + +Applications that can be launched within GNOME or on the command line must start successfully and withstand basic functionality tests. This includes: + +- Web browser +- File manager +- Package manager +- Image/Document Viewers +- Text editors (gedit, vim) +- Archive manager +- Terminal Emulator (GNOME Terminal) +- Problem Reporter +- Help Viewer +- System Settings + +??? tldr "References" + - Test cases: + - [QA:Testcase Application Functionality](Testcase_Post_Application_Functionality.md) + +#### Default Panel Functionality (Desktop Only) + +All elements of GNOME should function properly in regular use. + +??? tldr "References" + - Test cases: + - [QA:Testcase GNOME UI Functionality](Testcase_Post_GNOME_UI_Functionality.md) + +#### Dual Monitor Setup (Desktop Only) + +Computers using two monitors, the graphical output is correctly shown on both monitors. + +??? tldr "References" + - Test cases: + - [QA:Testcase Multimonitor Setup](Testcase_Post_Multimonitor_Setup.md) + +#### Artwork and Assets (Server and Desktop) + +Proposed final artwork (such as wallpapers and other assets) must be included. A wallpaper from this package should show up as a default for GDM and GNOME. + +??? tldr "References" + - Test cases: + - [QA:Testcase Artwork and Assets](Testcase_Post_Artwork_and_Assets.md) + +#### Packages and Module Installation + +Packages (non-module) should be able to be installed without conflicts or dependent on repositories outside of {{ rc.prod }}. + +- Default modules (as listed in dnf module list) should be installed without requiring them to be enabled. +- Module streams should be able to be switched and those packages should be able to be installed without errors or unresolved dependencies. + +??? tldr "References" + - Test cases: + - [QA:Testcase Basic Package installs](Testcase_Post_Package_installs.md) + - [QA:Testcase Module Streams](Testcase_Post_Module_Streams.md) + +#### Identity Management Server Setup + +It should be possible to setup a IdM server (FreeIPA), use it's functionality and connect clients. + +??? tldr "References" + - Test cases + - [QA:Testcase Identity Management](Testcase_Post_Identity_Management.md) + +{% include 'rc_content_bottom.md' %} diff --git a/docs/guidelines/release_criteria/r9/.pages b/docs/guidelines/release_criteria/r9/.pages new file mode 100644 index 0000000..a69df71 --- /dev/null +++ b/docs/guidelines/release_criteria/r9/.pages @@ -0,0 +1,6 @@ +--- +nav: + - ... | index.md + - Rocky Linux 9 Release Criteria: 9_release_criteria.md + - Rocky Linux 9.0 QA and Testing Summary: 9.0_qa_testing_summary.md + - Rocky Linux 9.0 GO / NO-GO Status: 9.0_qa_testing_go_no_go.md diff --git a/docs/guidelines/release_criteria/r9/9.0_qa_testing_go_no_go.md b/docs/guidelines/release_criteria/r9/9.0_qa_testing_go_no_go.md new file mode 100644 index 0000000..9a851c6 --- /dev/null +++ b/docs/guidelines/release_criteria/r9/9.0_qa_testing_go_no_go.md @@ -0,0 +1,89 @@ +--- +title: Rocky Linux 9.0 QA and Testing GO / NO GO Status +author: Trevor Cooper +revision_date: 2022-05-18 +rc: + prod: Rocky Linux + ver: 9 +--- + +# {{ rc.prod }} {{ rc.ver }} QA and Testing : GO / NO GO Status +This document summarizes the GO / NO GO Status for Release of Rocky Linux 9.0 from perspective of the QA / Testing efforts. It is based largely on the Release Criteria (https://wiki.rockylinux.org/team/testing/release_criteria/) as was started as an import of that document. If there are differences between the official Release Critieria document and this document the official Release Criteria document will override and this document shall be updated. + +As a reminder, the objective of a release (major or minor) is to provide a solid Enterprise Linux release that is suitable to: +- Meet the needs of end users +- Meet the needs of enterprises big or small + + +## SUMMARY + +| Category | Proportion | Remaining Items | +| ----------------- | ----------------- | --------------- | +| TO_CONFIRM | 1 / 29 (3%) | Firmware RAID | +| PASS | 28/ 29 (97%) | | +| FAIL_NON_BLOCKING | 0 / 29 (0%) | | +| FAIL_BLOCKING | 0 / 29 (0%) | | + + +## SOP +In this document each requirement is described and status is specified in the title. + +Current choices are... + +### TO_CONFIRM +- this means the item may be INCOMPLETE, PASS, FAIL_NON_BLOCKING or FAIL_BLOCKING and must be verified + +### PASS +- this means that the release criteria has been met and is not a blocker + +### FAIL_NON_BLOCKING +- this means the release criter has not been met but is non-blocking + +### FAIL_BLOCKING +- this means the release criteria has not been met and is blocking + +In this document criteria status should include who completed the item and generally how it was complete. + +Examples... + +- PASS - (@tcooper, virt only, manual) +- PASS - (@lumarel, @raktajino, @tcooper, semi-automatic , openQA) + + +## Initialization Requirements + +- Release-blocking images must boot - PASS - (@neil, @atomicturtle, @lumarel) +- Optical Media Requirements - PASS - (@neil) +- Basic Graphics Mode behaviors - PASS - (@tcooper, virt only, manual) +- No Broken Packages - PASS - (@tcooper, scripted, manual) +- Repositories Must Match Upstream - PASS - (@tcooper, manual) +- Debranding - PASS - (@tcooper, scripted, manual) +- Media Consistency Verification - PASS - (@tcooper, scripted, manual) +- Packages and Installer Sources - PASS - (@lumarel, semi-automatic, openQA test) +- NAS (Network Attached Storage) - PASS - (@stack, manual) +- Installation Interfaces - PASS - (@lumarel, @raktajino, @tcooper, semi-automatic, openQA, @atomicturtle, manual?, SCAP) +- Minimal Installation - PASS - (@lumarel, @raktajino, @tcooper, semi-automatic , openQA) +- Kickstart Installation - PASS - (@label, @tcooper, manual, createhdds) +- Disk Layouts - PASS - (@lumarel, @raktajino, @tcooper, semi-automatic, openQA) +- Firmware RAID - TO_CONFIRM - (@tbd, missing hardware support?) +- Bootloader Disk Selection - PASS - (@raktajino, manual) +- Storage Volume Resize - PASS - (@lumarel, @raktajino, @tcooper, semi-automatic, openQA) +- Update Image - PASS - (@raktajino,@tcooper, semi-automatic, openQA) +- Installer Help - PASS - (@lumarel, @raktajino, @tcooper, semi-automatic, openQA) +- Installer Translations - PASS - (@lumarel, @raktajino, @tcooper, semi-automatic, openQA) + + +## Cloud Image Requirements +- Images Published to Cloud Providers - PASS - (@neil) + + +## Post-Installation Requirements +- System Services - PASS - (@lumarel, semi-automatic, openQA) +- Keyboard Layout - PASS - (@lumarel, semi-automatic, openQA) +- SELinux Errors (Server) - PASS - (@lumarel, semi-automatic, openQA) +- SELinux and Crash Notifications (Desktop Only) - PASS - (@lumarel, semi-automatic, openQA) +- Default Application Functionality (Desktop Only) - PASS - (@lumarel, semi-automatic, openQA) +- Default Panel Functionality (Desktop Only) - PASS - (@lumarel, semi-automatic, openQA) +- Dual Monitor Setup (Desktop Only) - PASS - (@lumarel, semi-automatic, openQA) +- Artwork and Assets (Server and Desktop) - PASS - (@lumarel, semi-automatic, openQA) +- Packages and Module Installation - PASS - (@lumarel, semi-automatic, openQA) diff --git a/docs/guidelines/release_criteria/r9/9.0_qa_testing_summary.md b/docs/guidelines/release_criteria/r9/9.0_qa_testing_summary.md new file mode 100644 index 0000000..6e0e209 --- /dev/null +++ b/docs/guidelines/release_criteria/r9/9.0_qa_testing_summary.md @@ -0,0 +1,114 @@ +--- +title: Rocky Linux 9.0 QA and Testing Summary +author: Trevor Cooper +revision_date: 2022-05-18 +rc: + prod: Rocky Linux + ver: 9 + level: Final +--- + +# {{ rc.prod }} {{ rc.ver }} QA and Testing Summary +Last updated: + +## Scope +This document will record a summary of all QA and Testing results for {{ rc.prod }} {{ rc.9 }} release. It is only a record of success and/or failure. Solution discussion should take place elsewhere. + +## Reference +- Please check RHEL 9 Release Notes **BEFORE** marking issue here as **FAIL**. + +## SOP +- Please include **PASS**, **FAIL**, **NOTABUG**, **INVESTIGATE** or **UPSTREAM** as appropriate in all entries. +- Please only provide brief summary. Details should go to Rocky Pastebin, links here is OK. +- Please leave your MM @handle on all items you have done or are working on so we can talk to you to get resolution. +- If the item you have reported is related to a QA:Testcase please mention it. +- If you think the item you have reported should be a QA:Testcase, even if it's not a current requirement, suggest a title and create an issue in the wiki repository so we can add it. + +## INVESTIGATE +This is a list of items that are being INVESTIGATEd further before being assigned a PASS, FAIL, NOTABUG or UPSTREAM status. +PLEASE add your MM handle if you are working on this item to minimize duplication of work. More than one handle is allowed but please communicate. + +- QA:Testcase Firmware RAID - [INVESTIGATE] - @neil + +## UPSTREAM +This is a list of items that have been verified to be replicated UPSTREAM in RHEL {{ rc.9 }} and/or are described clearly in the RHEL 9 Release Notes. + +- QA:Testcase_Some_Testcase - [UPSTREAM] - @your_mm_handle - + +## FAIL +This is a list of items that have been verified to FAIL the QA:Testcase. In addition to recording who did the test please indicate if the item is BLOCKING release or not. + +- QA:Testcase_Some_Testcase - [FAIL] - @your_mm_handle - +- RC1 - The blue side banner background at the left side of the Anaconda installer display is not displaying correctly in virt-viewer. [minor BUG] - @alangm + +## NOTABUG +This is a list of items that have been verified as less than optimal but are expected and NOTABUG. + +- QA:Testcase Basic Graphics Mode - [DEPRICATED] - @tcooper + +## PASS +This is a list of item that have been verified as PASSing the QA:Testcase named (or proposed). + +- QA:Testcase_Debranding - [PASS] - @tcooper - preliminary pass based on Peridot yumrepofs +- RC1 - QA:Testcase System Services - [PASS] - @lumarel - tested with boot-iso and every base environment +- RC1 - Basic Installs - [PASS] - @lumarel - Installs of all base environments on x86_64 without additional optional package groups +- RC1 - QA:Testcase Packages and Installer Sources - [PASS] - @raktajino - tested against RC1-boot-iso in openQA, all package sets pass. +- RC1 - QA:Testcase GNOME UI Functionality - [PASS] - @lumarel - Nothing to mention +- RC1 - QA:Testcase SELinux Errors on Server - [PASS] - @lumarel - no errors +- RC1 - QA:Testcase SELinux Errors on Desktop- [PASS] - @lumarel - no errors +- RC1 - QA:Testcase Artwork and Assets - [PASS] - @lumarel - everything looks correct +- RC1 - QA:Testcase Minimal Installation - [PASS] - @raktajino - nothing to report +- RC1 - QA: Testcase Boot Methods Boot Iso - [PASS] - @alangm - minimal system, graphical server & basic workstation - nothing to report except as above ^^^ +- RC1 - Minimal install on QEMU/KVM passing - [PASS] - @boris - reboot takes slightly longer on the first reboot +- RC1 - Repo Health Check - [PASS] - @lumarel - sqlite database files for repo metadata is missing up to now, which will hinder some applications from syncing the sources - are implemented +- RC1 - Repo Health Check - [PASS] - @lumarel - metadata signing not implemented up to now - looks to be implemented as well now +- RC1 - Repo Health Check - [PASS] - @lumarel - Repos look good now excl. BaseOS of aarch64, ppc64le, s390x, where the boot image is missing, which is referenced in the .treeinfo +- RC1 - Redesign looks to be broken (Post-Install error, effects some of the install groups) - [PASS] - @lumarel +- RC1 - Workstation install with known pesign bug - [PASS] - @alangm - error window "Exit Installer" button froze Anaconda instead of exit so had to initiate shutdown from cockpit - @lumarel - fixed! +- RC1 - Building packages with mock - [PASS] - @hbjy - building works with the mock config from sokels' repo (will be upstreamed for release) +- RC1 - Install apache + mariadb-server - [PASS] - @sspencerwire - mariadb-secure-installation works, nextcloud install on top as well +- RC1 - Install works on macOS M1 - [PASS] - @netwaze, @gardo984, @jordan_c +- RC1 - Install on baremetal went well - [PASS] - @alangm - on a Dell T630 +- RC1 - Install went well - [PASS] - @kars88 - there was some flickering during the install +- RC1.1 dvd - STIG security - [PASS] - @atomicturtle - packages were not complete in RC1 dvd, but are now in RC1.1 +- RC1.1 - Basic installs work - [PASS] - @mkahric - Virtualbox on Windows (UEFI w/o SB), Workstation Player on Windows (UEFI w/ SB), Hyper-V (UEFI w/ SB) +- RC1.1 - docker installation with ansible works - [PASS] - @gardo984 +- RC1 - Repo Health Check - [PASS] - @lumarel - all repos (x86_64, aarch64, ppc64le, s390x) look good now, all metadata is healthy and all package checksums match +- RC1.1 - QA:Testcase Multimonitor Setup - [PASS] - @lumarel - looks good +- RC1.1 - QA:Testcase Artwork and Assets - [PASS] - @lumarel - no issues found +- RC1.1 - QA:Testcase GNOME UI Functionality - [PASS] - @lumarel - everything behaves as it should +- RC1.1 - QA:Testcase SELinux Errors on Desktop - [PASS] - @lumarel - After an hour of usage no errors +- RC1.1 - QA:Testcase Keyboard Layout - [PASS] - @lumarel - OpenQA tests and manual tests show normal behavior +- RC1.1 - QA:Testcase SELinux Errors on Server - [PASS] - @lumarel - after more than an hour of runtime with several installs and application setups (missing, Katello bootstrap, which once brought some SELinux errors) +- RC1.1 - QA:Testcase System Services - [PASS] - @lumarel - no errors +- RC1.1 - QA: Testcase Installation Interfaces - [PASS] - @raktajino - failures on the Welcome Tour screen because we haven't handled it yet. Everything else seems good. +- RC1.1 - QA:Testcase Installer Help - [PASS] - @raktajino - no errors +- RC1.1 - QA: Testcase Disk Layouts - [PASS] - @raktajino - Assorted post-install failures due to either mismatch between console login and GUI login or unhandled Welcome Tour. +- RC1.1 - QA: Testcase Kickstart Installation - [PASS] - @raktajino - Executed manually with virt-install. Nothing to report. +- RC1.1 - QA: Testcase Update Image - [PASS] - @raktajino - failure on _console_wait_login as the GUI login screen is displayed. Update image is applied as expected. +- RC1.1 - QA:Testcase VNC Graphics Mode - [PASS] - @tcooper +- RC1.1 - QA:Testcase Basic Package installs - [PASS] - @lumarel - All package sets and services tested which are currently on the blocking list +- RC1.1 - QA:Testcase Application Functionality - [PASS] - @lumarel - everything looks good +- RC1/RC1.1 - QA:Testcase_Mediacheck - [PASS] - @tcooper +- RC1/RC1.1 - QA:Testcase Media Consistency Verification - [PASS] - @tcooper +- RC1.1 - QA:Testcase Media Repoclosure - [PASS] - @tcooper +- RC1.1 - QA:Testcase Media File Conflicts - [PASS] - @tcooper +- RC1.1 - QA: Testcase Bootloader Disk Selection - [PASS] - @raktajino +- RC1.1 - QA: Testcase Installer Translations - [PASS] - @raktajino +- RC1.1 - QA: Testcase Network Attached Storage - [PASS] - @Stack +- QA: Testcase Boot Methods Boot ISO - [PASS] - all of us <3 +- QA: Testcase Boot Methods DVD - [PASS] - todos nosotros <3 +- QA:Testcase Cloud Images Published - [PASS] - @neil +- QA:Testcase Storage Volume Resize - [PASS] - @raktajino, @tcooper +- QA:Testcase Repocompare - [PASS] - @tcooper +- QA:Testcase Basic install boot final ISO - [PASS] - @lumarel +- QA:Testcase Basic install minimal final ISO - [PASS] - @lumarel +- QA:Testcase Basic installs dvd final ISO - [PASS] - @lumarel - install of every base environment + custom operating system with all options enabled +- OpenQA tests with final ISOs - [PASS] - @lumarel - boot-iso 2/2 | minimal-iso 2/2 | minimal 22/26 (failing are all false-positive) | server 21/26 (failing are all false-positive) | package-set 5/5 | workstation 17/26 (failing are all false-positive) | graphical-server 17/26 (failing are all false-positive) | universal 34/43 (failing are all false-positive) +- QA:Testcase Minimal Installation - [PASS] - @neil + +## OTHER NOTABLE ITEMS +- QA:Testcase Module Streams - @lumarel - 9.0 currently does not have any Stream metadata and only the base Streams +- Repo Health Check - @lumarel - yumrepofs is not sending content lengths, which make syncing with Katello impossible +- Missing package in package group additional-devel - @stack - dnf -y group install additional-devel: No match for group package "hmaccalc" - this is an old EL bug +- Hosting a nfs server in macOS, and using that in Virtualbox for OpenQA shares is a pain - @tcooper diff --git a/docs/guidelines/release_criteria/r9/9_release_criteria.md b/docs/guidelines/release_criteria/r9/9_release_criteria.md new file mode 100644 index 0000000..519d83e --- /dev/null +++ b/docs/guidelines/release_criteria/r9/9_release_criteria.md @@ -0,0 +1,277 @@ +--- +title: Rocky Linux 9 Release Criteria +author: + - Trevor Cooper + - Lukas Magauer +revision_date: 2022-04-01 +rc: + prod: Rocky Linux + ver: 9 + level: Final +--- + +# {{ rc.prod }} {{ rc.ver }} {{ rc.level }} Release Objectives + +The objective of a release (major or minor) is to provide a solid Enterprise Linux release that is suitable to: + +- Meet the needs of end users +- Meet the needs of enterprises big or small + +## {{ rc.prod }} {{ rc.ver }} {{ rc.level }} Release Requirements + +In order for {{ rc.prod }} to be released to the general public, a compose must be able to meet all the following criteria as provided in this document. This is allows the decision process to be straightforward and as clear as possible. This document only contains “hard requirement” items. Optional/nice to have items are not to be included in this list. + +There may cases where a requirement cannot be met but only in particular configurations. In these types of cases, the Release Engineering Team should use their judgement to determine whether or not the issue should be considered to block the release. They should consider the number of users likely to be affected by said issue, the severity of the case, if the issue can be avoided with ease (by both informed and uninformed users), and if the problem exists upstream in the current Red Hat Enterprise Linux that the release is based on. + +!!! info "Release-blocking Server" + ...means bugs as it pertains to server functionality can be considered to block a release. This applies to any packages that provide a service such as httpd, nginx, etc. All architectures apply. + +!!! info "Release-blocking Desktop" + ...means bugs as it pertains to desktop functionality (GNOME) can be considered to block a release. This applies to both x86_64 and aarch64. Additional desktops (as provided by EPEL or a SIG) are not considered blockers. + +!!! info "Release-blocking Image" + ...means bugs as it pertains to the images built that can block a release. This applies to the DVD, minimal, and boot images on all architectures. + +### Initialization Requirements + +#### Release-blocking images must boot + +Release-blocking installer images must boot when written to optical media or USB flash drive of appropriate sizes (if applicable) via officially supported methods. It is not the testing team’s responsibility to test optical media, but they can and report back. If a bug is found, it is considered a blocker. + +??? tldr "Optical Media Requirements" + Release-blocking images must boot when written to optical media of an appropriate size. Current size requirements are: boot.iso = 789M, minimal.iso = 2.0G and dvd.iso = 10G. + +??? tldr "Officially supported USB flash drive writing methods" + The following methods of writing USB flash drives are officially support: dd
+ The following methods of writing USB flash drives are _**not**_ supported: rufus + +??? tldr "References" + - Test cases: + - [QA:Testcase Boot Methods Boot ISO](Testcase_Boot_Methods_Boot_Iso.md) + - [QA:Testcase Boot Methods DVD](Testcase_Boot_Methods_Dvd.md) + - [QA:Testcase Media USB dd](Testcase_Media_USB_dd.md) + +#### VNC Graphics Mode behaviors +The VNC installation option on all release-blocking installers must function as intended. This means launching the installer or desktop and connecting with VNC to complete the installation. There must be no bugs that prevent the installer from being reached in this configuration on all systems and classes of hardware supported by the enterprise linux kernel. +??? tldr "References" + - Test cases: + - [QA:Testcase VNC Graphics Mode](Testcase_VNC_Graphics_Mode.md) + +#### No Broken Packages +Critical errors, such as undeclared conflicts, unresolved dependencies, or modules relying on packages from another stream will be considered an automatic blocker. There are potential exceptions to this (eg, freeradius cannot be installed on an older perl stream, this is a known issue upstream). +??? tldr "References" + - Test cases: + - [QA:Testcase Media Repoclosure](Testcase_Media_Repoclosure.md) + - [QA:Testcase Media File Conflicts](Testcase_Media_File_Conflicts.md) + +#### Repositories Must Match Upstream +Repositories and the packages within them should match upstream as closely as possible. Notable exceptions would be kmods, kpatch, or what is deemed as “spyware” like insights. Packages that are available from upstream should not have hard requirements on RHSM and packages that have it default built in should be patched out. +??? tldr "References" + - Test cases: + - [QA:Testcase Repo Compare](Testcase_Repo_Compare.md) + - [QA:Testcase Packages No Insights](Testcase_Packages_No_Insights.md) + - [QA:Testcase Packages No RHSM](Testcase_Packages_No_RHSM.md) + +#### Debranding +Assets and functionality that are Red Hat specific should not be included. If they are not patched out, it will be considered an automatic blocker. +??? tldr "References" + - Test cases: + - [QA:Testcase Debranding](Testcase_Debranding.md) + +### Installer Requirements + +#### Media Consistency Verification +This means that the installer’s mechanism for verifying the install medium is intact and must complete successfully, with the assumption that the medium was correctly written. It should return a failure message if this not the case. +??? tldr "References" + - Test cases: + - [QA:Testcase Media USB dd](Testcase_Media_USB_dd.md) + - [QA:Testcase Boot Methods Boot ISO](Testcase_Boot_Methods_Boot_Iso.md) + - [QA:Testcase Boot Methods DVD](Testcase_Boot_Methods_Dvd.md) + +#### Packages and Installer Sources +The installer must be able to use all supported local/remote packages and installer sources. +??? tldr "References" + - Test cases: + - [QA:Testcase Packages and Installer Sources](Testcase_Packages_Installer_Sources.md) + +#### NAS (Network Attached Storage)" +The installer must be able to detect and install to supported NAS devices (if possible and supported by the kernel). +??? tldr "References" + - Test cases: + - [QA:Testcase Network Attached Storage](Testcase_Network_Attached_Storage.md) + +#### Installation Interfaces +The installer must be able to complete an installation using all supported spokes. +??? tldr "References" + - Test cases: + - [QA:Testcase Installation Interfaces](Testcase_Installation_Interfaces.md) + +#### Minimal Installation +A minimal installation (via network) must be able to install the minimal package set. +??? tldr "References" + - Test cases: + - [QA:Testcase Minimal Installation](Testcase_Minimal_Installation.md) + +#### Kickstart Installation +A kickstart installation should succeed, whether from optical/USB media or via the network. +??? tldr "References" + - Test cases: + - [QA:Testcase Kickstart Installation](Testcase_Kickstart_Installation.md) + +#### Disk Layouts +The installer must be able to create and install to any workable partition layout using any file system or format combination offered or supported by the installer. File systems that are not supported by the EL kernel is not tested here (this means btrfs, zfs, both of wish are not supported). +??? tldr "References" + - Test cases: + - [QA:Testcase Disk Layouts](Testcase_Disk_Layouts.md) + +#### Firmware RAID +The installer must be able to detect and install to firmware RAID devices. Note that system-specific bugs do not count as blockers. It is likely that some hardware support might be broken or not available at all. DUDs (driver update disks) are not considered for this criteria. +??? tldr "References" + - Test cases: + - [QA:Testcase Firmware RAID](Testcase_Firmware_RAID.md) + +#### Bootloader Disk Selection +The installer must allow the user to choose which disk the bootloader will be installed to or, if the user so chooses, not to install a bootloader. +??? tldr "References" + - Test cases: + - [QA:Testcase Bootloader Disk Selection](Testcase_Bootloader_Disk_Selection.md) + +#### Storage Volume Resize +Any installer mechanism for resizing storage volumes must correctly attempt the requested operation. This means that if the installer offers a way to resize storage volumes, then it must use the correct resizing tool with the correct parameters. However, it does not require the installer to disallow resizing of unformatted or volumes with an unknown filesystem type. +??? tldr "References" + - Test cases: + - [QA:Testcase Storage Volume Resize](Testcase_Storage_Volume_Resize.md) + +#### Update Image +The installer must be able to use an installer update image retrieved from removable media or a remote package source. This includes DUDs (driver update disks). +??? tldr "References" + - Test cases: + - [QA:Testcase Update Image](Testcase_Update_Image.md) + +#### Installer Help +Any element in the installer which contains a “help” text must display the appropriate help documentation when selected. +??? tldr "References" + - Test cases: + - [QA:Testcase Installer Help](Testcase_Installer_Help.md) + +#### Installer Translations +The installer must correctly display all complete translations that are available for use. +??? tldr "References" + - Test cases: + - [QA:Testcase Installer Translations](Testcase_Installer_Translations.md) + +### Cloud Image Requirements +#### Images Published to Cloud Providers +Release-blocking cloud disk images must be published to appropriate cloud providers (such as Amazon) and they must successfully boot. This also applies to KVM based instances, such as x86 and aarch64 systems. +??? tldr "References" + - Test cases: + - [QA:Testcase TBD](Testcase_Template.md) + +### Post-Installation Requirements + +#### System Services + +All system services present after installation must start properly, with the exception of services that require hardware which is not present. Examples of such services would be: + +- sshd +- firewalld +- auditd +- chronyd + +??? tldr "References" + - Test cases: + - [QA:Testcase System Services](Testcase_Post_System_Services.md) + +#### Keyboard Layout + +If a particular keyboard layout has been configured for the system, that layout must be used: + +- When unlocking storage volumes (encrypted by LUKS) +- When logging in at a TTY console +- When logging in via GDM +- After logging into a GNOME desktop system, if the user does not have their own layout configuration set. + +??? tldr "References" + - Test cases: + - [QA:Testcase Keyboard Layout](Testcase_Post_Keyboard_Layout.md) + +#### SELinux Errors (Server) + +There must be no SELinux denial logs in /var/log/audit/audit.log + +??? tldr "References" + - Test cases: + - [QA:Testcase SELinux Errors on Server installations](Testcase_Post_SELinux_Errors_Server.md) + +#### SELinux and Crash Notifications (Desktop Only) + +There must be no SELinux denial notifications or crash notifications on boot, during installation, or during first login. + +??? tldr "References" + - Test cases: + - [QA:Testcase SELinux Errors on Desktop clients](Testcase_Post_SELinux_Errors_Desktop.md) + +#### Default Application Functionality (Desktop Only) + +Applications that can be launched within GNOME or on the command line must start successfully and withstand basic functionality tests. This includes: + +- Web browser +- File manager +- Package manager +- Image/Document Viewers +- Text editors (gedit, vim) +- Archive manager +- Terminal Emulator (GNOME Terminal) +- Problem Reporter +- Help Viewer +- System Settings + +??? tldr "References" + - Test cases: + - [QA:Testcase Application Functionality](Testcase_Post_Application_Functionality.md) + +#### Default Panel Functionality (Desktop Only) + +All elements of GNOME should function properly in regular use. + +??? tldr "References" + - Test cases: + - [QA:Testcase GNOME UI Functionality](Testcase_Post_GNOME_UI_Functionality.md) + +#### Dual Monitor Setup (Desktop Only) + +Computers using two monitors, the graphical output is correctly shown on both monitors. + +??? tldr "References" + - Test cases: + - [QA:Testcase Multimonitor Setup](Testcase_Post_Multimonitor_Setup.md) + +#### Artwork and Assets (Server and Desktop) + +Proposed final artwork (such as wallpapers and other assets) must be included. A wallpaper from this package should show up as a default for GDM and GNOME. + +??? tldr "References" + - Test cases: + - [QA:Testcase Artwork and Assets](Testcase_Post_Artwork_and_Assets.md) + +#### Packages and Module Installation + +Packages (non-module) should be able to be installed without conflicts or dependent on repositories outside of {{ rc.prod }}. + +- Default modules (as listed in dnf module list) should be installed without requiring them to be enabled. +- Module streams should be able to be switched and those packages should be able to be installed without errors or unresolved dependencies. + +??? tldr "References" + - Test cases: + - [QA:Testcase Basic Package installs](Testcase_Post_Package_installs.md) + - [QA:Testcase Module Streams](Testcase_Post_Module_Streams.md) + +#### Identity Management Server Setup + +It should be possible to setup a IdM server (FreeIPA), use it's functionality and connect clients. + +??? tldr "References" + - Test cases + - [QA:Testcase Identity Management](Testcase_Post_Identity_Management.md) + +{% include 'rc_content_bottom.md' %} diff --git a/docs/guidelines/release_criteria/release_criteria.md b/docs/guidelines/release_criteria/release_criteria.md new file mode 100644 index 0000000..f9401ad --- /dev/null +++ b/docs/guidelines/release_criteria/release_criteria.md @@ -0,0 +1,19 @@ +--- +title: Rocky Linux Release Criteria & Status +author: Trevor Cooper +revision_date: 2022-05-22 +rc: + prod: Rocky Linux +--- + +# {{ rc.prod }} Release Status + +The QA and Testing efforts during releases are tracked in online shared documents. After release the status track and go/no-go documents are published here. + +| Rocky Linux Version | Release Criteria | QA and Testing Status | QA and Testing GO / NO-GO Summary | Official Release Date | +| -------------------- | --------------------------------------- |--------------------------------------- | ---------------------------------------- | --------------------------- | +| {{ rc.prod }} 8.5 | not available | not available | not available | NOVEMBER 15, 2021 | +| {{ rc.prod }} 8.6 | [AVAILABLE](8.6_qa_testing_summary.md) | [AVAILABLE](8.6_qa_testing_summary.md) | [AVAILABLE](8.6_qa_testing_go_no_go.md) | MAY 16, 2022 | +| {{ rc.prod }} 9.0 | [AVAILABLE](9.0_qa_testing_summary.md) | [AVAILABLE](9.0_qa_testing_summary.md) | [AVAILABLE](9.0_qa_testing_go_no_go.md) | TBD | + +{% include 'content_bottom.md' %} diff --git a/docs/include/qa_content_example_only.md b/docs/include/qa_content_example_only.md new file mode 100644 index 0000000..fb14dd4 --- /dev/null +++ b/docs/include/qa_content_example_only.md @@ -0,0 +1,3 @@ + +!!! error "CONTENT EXAMPLE ONLY" + Content on this page *may be* copy-pasta from [Fedora Quality Assurance](https://fedoraproject.org/wiki/QA) documents and needs to be replaced and/or reviewed before publishing for applicability for Rocky Linux. diff --git a/docs/include/qa_data_loss_warning.md b/docs/include/qa_data_loss_warning.md new file mode 100644 index 0000000..cd3523f --- /dev/null +++ b/docs/include/qa_data_loss_warning.md @@ -0,0 +1,3 @@ +!!! error "DATA LOSS" + **Depending on installer choices this MAY destroy all the data on the test system.** + If you choose to complete the installation of the test system any/all data on the system may be lost. Please do not install on a system whose contents you need to keep. diff --git a/docs/include/qa_setup_boot_to_media.md b/docs/include/qa_setup_boot_to_media.md new file mode 100644 index 0000000..6338c77 --- /dev/null +++ b/docs/include/qa_setup_boot_to_media.md @@ -0,0 +1,6 @@ +1. Obtain access to supported system and hardware class to be installed. +1. Prepare appropriate media for the selected ISO to be tested. + - Example: [QA:Testcase Media USB dd](Testcase_Media_USB_dd.md) +1. Boot the system from the prepared optical, USB media or virtual device attachment. + - Examples: [QA:Testcase Boot Methods Boot ISO](Testcase_Boot_Methods_Boot_Iso.md), [QA:Testcase Boot Methods DVD](Testcase_Boot_Methods_Dvd.md) +1. In the boot menu select the appropriate option to boot the installer. diff --git a/docs/include/qa_testcase_bottom.md b/docs/include/qa_testcase_bottom.md new file mode 100644 index 0000000..b5301f6 --- /dev/null +++ b/docs/include/qa_testcase_bottom.md @@ -0,0 +1,19 @@ + +

Additional Information

+ +=== "Contact" + + If you have questions with respect to this content or to report concerns regarding the use or misuse content please do not hesitate to contact us at [testing@rockylinux.org](mailto:testing@rockylinux.org). + + +=== "Disclaimer" + + Rocky Linux and the Rocky Enterprise Software Foundation (RESF) does not make any express or implied warranties, including but not limited to the warranties of non-infringement of any third party intellectual property rights. RESF does not warrant that any pending trademark applications for trademarks of RESF will result in any granted trademark protection. RESF shall not be liable for any claims relating to user's activities falling within the scope of the permission and user hereby agrees to indemnify, defend and hold RESF and its contributors harmless against any such claim. + +=== "Attribution" + + This work is heavily inspired by the [Fedora Quality Assurance](https://fedoraproject.org/wiki/QA) documents which were made available under [Attribution-Share Alike 4.0 International](https://creativecommons.org/licenses/by-sa/4.0/) license unless otherwise noted. + +=== "License" + + This content is licensed under under [Attribution-Share Alike 4.0 International](https://creativecommons.org/licenses/by-sa/4.0/) license unless otherwise noted. diff --git a/docs/include/qa_testcase_supported_systems.md b/docs/include/qa_testcase_supported_systems.md new file mode 100644 index 0000000..4fe5a53 --- /dev/null +++ b/docs/include/qa_testcase_supported_systems.md @@ -0,0 +1,18 @@ + +

Supported Systems and Hardware Classes

+ +=== "x86_64" + + TBD + +=== "aarch64" + + TBD + +=== "ppc64" + + TBD + +=== "s309x" + + TBD diff --git a/docs/include/rc_content_bottom.md b/docs/include/rc_content_bottom.md new file mode 100644 index 0000000..467d5dc --- /dev/null +++ b/docs/include/rc_content_bottom.md @@ -0,0 +1,19 @@ + +

Additional Information

+ +=== "Contact" + + If you have questions with respect to this content or to report concerns regarding the use or misuse content please do not hesitate to contact us at [testing@rockylinux.org](mailto:testing@rockylinux.org). + + +=== "Disclaimer" + + Rocky Linux and the Rocky Enterprise Software Foundation (RESF) does not make any express or implied warranties, including but not limited to the warranties of non-infringement of any third party intellectual property rights. RESF does not warrant that any pending trademark applications for trademarks of RESF will result in any granted trademark protection. RESF shall not be liable for any claims relating to user's activities falling within the scope of the permission and user hereby agrees to indemnify, defend and hold RESF and its contributors harmless against any such claim. + +=== "Attribution" + + This work is heavily inspired by the [Fedora Release Requirements](https://fedoraproject.org/wiki/Fedora_Release_Criteria) documents which were made available under [Attribution-Share Alike 4.0 International](https://creativecommons.org/licenses/by-sa/4.0/) license unless otherwise noted. + +=== "License" + + This content is licensed under under [Attribution-Share Alike 4.0 International](https://creativecommons.org/licenses/by-sa/4.0/) license unless otherwise noted. diff --git a/docs/include/rc_content_example_only.md b/docs/include/rc_content_example_only.md new file mode 100644 index 0000000..5e8069b --- /dev/null +++ b/docs/include/rc_content_example_only.md @@ -0,0 +1,3 @@ + +!!! error "CONTENT EXAMPLE ONLY" + Content on this page *may be* copied from [Fedora Release Requirements](https://fedoraproject.org/wiki/Fedora_Release_Criteria) documentation and needs to be replaced and/or reviewed before publishing for applicability for Rocky Linux.