testing
/
wiki
generated from sig_core/wiki-template
10
0
Fork 3
Code Issues 1 Pull Requests Actions Packages Projects Releases Wiki Activity

Testing Team Additions / Modifications (#5) 

* update testing team primary page
* add mkdocs-macros-plugin to support page includes
* add release_criteria page
* add collapsible admonitions
* add QA/Testcase_Template
* add testcase examples
* hide the QA directory from navigation while building content
* add support for tabbed content
* Update index.md
* Update release_criteria.md
* Update rc_content_example_only.md
* Update Testcase_Boot_Methods_Boot_Iso.md
* Update Testcase_Boot_Methods_Dvd.md
* add mkdocs-autolinks-plugin
- autogenerate relative URLs for interpage links
* rewrite Fedora example text
* add templates and QA Test Case index
* add testcase for USB media creation and testing
* add testcase for basic-graphics-mode requirement
* add testcases for no-broken-packages requirement
* add testcases for repositories must match upstream requirement
* add command samples and sample output to no broken packages testcases

Co-authored-by: Alan Marshall <alangm1001@gmail.com>
Co-authored-by: Trevor Cooper <tcooper@rockylinux.org>

Signed-off-by: Neil Hanlon <neil@resf.org>
This commit is contained in:
Trevor Cooper 2022-04-25 14:02:08 -07:00 committed by GitHub
parent dc3cda9b5b
commit b7177ba1b1
16 changed files with 986 additions and 4 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
.pages Normal file
View File

@ -0,0 +1,4 @@
---
nav:
- Team Members: index.md
- Release Criteria: release_criteria.md

2
QA/.pages Normal file
View File

@ -0,0 +1,2 @@
---
hide: true

44
QA/Testcase_Basic_Graphics_Mode.md Normal file
View File

@ -0,0 +1,44 @@
---
title: QA:Testcase Basic Graphics Mode
author: Trevor Cooper
revision_date: 2022-04-24
rc:
prod: Rocky Linux
ver: 8
level: Final
---
!!! error "REFERENCED RELEASE CRITERIA IS OVERLY GENERAL AND UNTESTABLE"
The associated release criteria, [Release_Criteria#basic-graphics-mode-behaviors](../release_criteria.md#basic-graphics-mode-behaviors), for this test case is overly general and **must** be modified to specific enough to be testable.
!!! info "Associated release criterion"
This test case is associated with the [Release_Criteria#basic-graphics-mode-behaviors](../release_criteria.md#basic-graphics-mode-behaviors) release criterion. If you are doing release validation testing, a failure of this test case may be a breach of that release criterion.
## Description
This test case will verify that release-blocking installers function as intended using the generic video driver option (“basic graphics mode”) on supported systems and classes of hardware.
{% include 'testing/qa_testcase_supported_systems.md' %}
## Setup
1. Obtain access to supported system and hardware class to be installed.
2. Prepare appropriate media for the selected ISO to be tested.
- Example: [QA:Testcase Media USB dd](Testcase_Media_USB_dd.md)
## How to test
1. Boot the system from the prepared optical, USB media or virtual device attachment.
- Examples: [QA:Testcase Boot Methods Boot ISO](Testcase_Boot_Methods_Boot_Iso.md), [QA:Testcase Boot Methods DVD](Testcase_Boot_Methods_Dvd.md)
2. In the boot menu select the appropriate option to boot the installer.
3. In the installer select the appropriate option to intall in basic graphics mode.
4. Proceed with installation on the test system.<br>**Depending on installer choices this MAY destroy all the data on the test system.**
!!! error "DATA LOSS"
If you choose to complete the installation of the test system any/all data on the system may be lost. Please do not install on a system whose contents you need to keep.
## Expected Results
1. Selection of basic graphics mode in the Anaconda installer is possible.
2. Anaconda installer presents a usable graphical intallation environment.
3. System under test can be installed normally.
4. After reboot system boots into graphical environment.
5. After login user is able to operate the graphical environment.
{% include 'testing/qa_testcase_bottom.md' %}

28
QA/Testcase_Boot_Methods_Boot_Iso.md Normal file
View File

@ -0,0 +1,28 @@
---
title: QA:Testcase Boot Methods Boot Iso
author: Trevor Cooper
revision_date: 2021-04-21
rc:
prod: Rocky Linux
ver: 8
level: Final
---
!!! info "Associated release criterion"
This test case is associated with the [Release_Criteria#initialization-requirements](../release_criteria.md#initialization-requirements) release criterion. If you are doing release validation testing, a failure of this test case may be a breach of that release criterion.
## Description
This is to verify that the Anaconda installer starts correctly when booting from the Rocky Linux boot.iso.
## Setup
1. Prepare your system for booting the boot.iso image. This may involve writing the image to a USB key or burning it to an optical disk. Additionally, attaching the boot.iso to a virtual machine instance as a Virtual Optical Disk or mounting the boot.iso to server via baseboard management controller virtual media attach should be possible but is not expressly required.
## How to test
1. Boot the system from the prepared optical, USB media or virtual device attachment.
2. In the boot menu select the appropriate option to boot the installer.
## Expected Results
1. Graphical boot menu is displayed for users to select install options. Navigating the menu and selecting entries must work. If no option is selected, the installer should load after a reasonable timeout.
2. System boots into the Anaconda installer.
{% include 'testing/qa_testcase_bottom.md' %}

28
QA/Testcase_Boot_Methods_Dvd.md Normal file
View File

@ -0,0 +1,28 @@
---
title: QA:Testcase Boot Methods DVD
author: Trevor Cooper
revision_date: 2021-04-21
rc:
prod: Rocky Linux
ver: 8
level: Final
---
!!! info "Associated release criterion"
This test case is associated with the [Release_Criteria#initialization-requirements](../release_criteria.md#initialization-requirements) release criterion. If you are doing release validation testing, a failure of this test case may be a breach of that release criterion.
## Description
This is to verify that the Anaconda installer starts correctly when booting from DVD.iso.
## Setup
1. Prepare your system for booting the DVD.iso image. This may involve writing the image to a USB key or burning it to an optical disk of sufficient capacity. Additionally, attaching the DVD.iso to a virtual machine instance as a Virtual Optical Disk or mounting the DVD.iso to server via baseboard management controller virtual media attach should be possible but is not expressly required.
## How to test
1. Boot the system from the prepared optical, USB media or virtual device attachment.
2. In the boot menu select the appropriate option to boot the installer.
## Expected Results
1. Graphical boot menu is displayed for users to select install options. Navigating the menu and selecting entries must work. If no option is selected, the installer should load after a reasonable timeout.
2. System boots into the Anaconda installer.
{% include 'testing/qa_testcase_bottom.md' %}

28
QA/Testcase_Debranding.md Normal file
View File

@ -0,0 +1,28 @@
---
title: QA:Testcase Debranding
author: Trevor Cooper
revision_date: 2022-04-20
rc:
prod: Rocky Linux
ver: 8
level: Final
---
!!! info "Associated release criterion"
This test case is associated with the [Release_Criteria - Debranding](../release_criteria.md#debranding) release criterion. If you are doing release validation testing, a failure of this test case may be a breach of that release criterion.
## Description
Add a short description here for this Testcase.
## Setup
1. Add steps for setup for this Testcase.
## How to test
1. Do this first...
2. Then do this...
## Expected Results
1. This is what you should see/verify.
2. You should also see/verify this.
{% include 'testing/qa_testcase_bottom.md' %}

175
QA/Testcase_Media_File_Conflicts.md Normal file
View File

@ -0,0 +1,175 @@
---
title: QA:Testcase Media File Conflicts
author: Trevor Cooper
revision_date: 2022-04-24
rc:
prod: Rocky Linux
ver: 8
level: Final
---
!!! info "Associated release criterion"
This test case is associated with the [Release_Criteria#no-broken-packages](../release_criteria.md#no-broken-packages) release criterion. If you are doing release validation testing, a failure of this test case may be a breach of that release criterion.
## Description
This testcase will verify that the offline repository included on release blocking images will not contain any file conflicts between packages without explicit `Conflicts:` tag in the package metadata.
## Setup
1. Obtain access to an environment with the `dnf` and `python3` commands.
2. Download the ISO to be tested to that machine.
3. Download the `potential_conflict.py` script provided by Rocky Linux Testing Team.
## How to test
1. Mount the ISO to be tested locally.
- Example:<br>`mount -o loop Rocky-8.5-x86_64-minimal.iso /media`
2. Determine the path to the `repodata` directory(ies) on the ISO.
- Example:<br>`find /media -name repodata`
3. Run the `potential_conflict.py` script on the mounted ISO.
- Example:<br>`python3 /vagrant/scripts/potential_conflict.py --repofrompath BaseOS,/media/BaseOS --repoid BaseOS --repofrompath Minimal,/media/Minimal --repoid Minimal`
4. Unmount the ISO.
- Example:<br>`umount /media`
## Expected Results
1. The `potential_conflict.py` script does not report any packages with non-declared conflicts.
<h3>Sample Output</h3>
=== "Success"
```
$ sudo mount -o loop Rocky-8.5-aarch64-minimal.iso /media
mount: /media: WARNING: device write-protected, mounted read-only.
$ python3 /vagrant/scripts/potential_conflict.py \
--repofrompath BaseOS,/media/BaseOS --repoid BaseOS \
--repofrompath Minimal,/media/Minimal --repoid Minimal
Added BaseOS repo from /media/BaseOS
Added Minimal repo from /media/Minimal
Getting complete filelist for:
file:///media/BaseOS
file:///media/Minimal
168374 files found.
Looking for duplicated filenames:
524 duplicates found.
Doing more advanced checks to see if these are real conflicts:
10% complete ( 52/ 524, 1139/sec), 0 found - eta 0:00:00
35% complete ( 182/ 524, 1146/sec), 0 found - eta 0:00:00
45% complete ( 234/ 524, 1818/sec), 0 found - eta 0:00:00
50% complete ( 260/ 524, 592673/sec), 0 found - eta 0:00:00
55% complete ( 286/ 524, 778942/sec), 0 found - eta 0:00:00
60% complete ( 312/ 524, 801852/sec), 0 found - eta 0:00:00
79% complete ( 416/ 524, 234/sec), 0 found - eta 0:00:00
84% complete ( 442/ 524, 902/sec), 0 found - eta 0:00:00
89% complete ( 468/ 524, 935/sec), 0 found - eta 0:00:00
94% complete ( 494/ 524, 1616/sec), 0 found - eta 0:00:00
99% complete ( 520/ 524, 1114/sec), 0 found - eta 0:00:00
0 file conflicts found.
0 package conflicts found.
== Package conflicts ==
== File conflicts, listed by conflicting packages ==
$ sudo umount /media
```
=== "Failure"
```
$ sudo mount -o loop Rocky-8.5-x86_64-dvd1.iso /media
mount: /media: WARNING: device write-protected, mounted read-only.
$ python3 /vagrant/scripts/potential_conflict.py \
--repofrompath AppStream,/media/AppStream --repoid AppStream \
--repofrompath BaseOS,/media/BaseOS --repoid BaseOS
Added AppStream repo from /media/AppStream
Added BaseOS repo from /media/BaseOS
Getting complete filelist for:
file:///media/AppStream
file:///media/BaseOS
851967 files found.
Looking for duplicated filenames:
101865 duplicates found.
Doing more advanced checks to see if these are real conflicts:
5% complete ( 5093/101865, 8713/sec), 0 found - eta 0:00:11
10% complete ( 10186/101865, 1787281/sec), 0 found - eta 0:00:05
15% complete ( 15279/101865, 2223312/sec), 0 found - eta 0:00:03
20% complete ( 20372/101865, 23614/sec), 0 found - eta 0:00:03
25% complete ( 25465/101865, 57188/sec), 0 found - eta 0:00:02
30% complete ( 30558/101865, 3831/sec), 0 found - eta 0:00:05
35% complete ( 35651/101865, 48455/sec), 0 found - eta 0:00:04
40% complete ( 40744/101865, 32067/sec), 0 found - eta 0:00:03
45% complete ( 45837/101865, 2136586/sec), 0 found - eta 0:00:03
50% complete ( 50930/101865, 72529/sec), 0 found - eta 0:00:02
55% complete ( 56023/101865, 176294/sec), 0 found - eta 0:00:02
60% complete ( 61116/101865, 68622/sec), 1 found - eta 0:00:01
65% complete ( 66209/101865, 155133/sec), 1 found - eta 0:00:01
70% complete ( 71302/101865, 13874/sec), 1 found - eta 0:00:01
75% complete ( 76395/101865, 10835/sec), 1 found - eta 0:00:01
80% complete ( 81488/101865, 27477/sec), 1 found - eta 0:00:00
85% complete ( 86581/101865, 9075/sec), 1 found - eta 0:00:00
90% complete ( 91674/101865, 14807/sec), 1 found - eta 0:00:00
95% complete ( 96767/101865, 197437/sec), 1 found - eta 0:00:00
100% complete (101860/101865, 38727/sec), 1 found - eta 0:00:00
1 file conflicts found.
11 package conflicts found.
== Package conflicts ==
mariadb-server-utils-3:10.3.28-1.module+el8.4.0+427+adf35707.x86_64
mysql-server-8.0.26-1.module+el8.4.0+652+6de068a7.x86_64
python3-mod_wsgi-4.6.4-4.el8.x86_64
python38-mod_wsgi-4.6.8-3.module+el8.4.0+570+c2eaf144.x86_64
python39-mod_wsgi-4.7.1-4.module+el8.4.0+574+843c4898.x86_64
libcmpiCppImpl0-2.0.3-15.el8.i686
tog-pegasus-libs-2:2.14.1-46.el8.i686
mariadb-connector-c-devel-3.1.11-2.el8_3.i686
mariadb-connector-c-devel-3.1.11-2.el8_3.x86_64
mariadb-devel-3:10.3.28-1.module+el8.4.0+427+adf35707.x86_64
mysql-devel-8.0.26-1.module+el8.4.0+652+6de068a7.x86_64
mariadb-server-3:10.3.28-1.module+el8.4.0+427+adf35707.x86_64
mysql-server-8.0.26-1.module+el8.4.0+652+6de068a7.x86_64
mariadb-test-3:10.3.28-1.module+el8.4.0+427+adf35707.x86_64
mysql-test-8.0.26-1.module+el8.4.0+652+6de068a7.x86_64
mariadb-connector-c-devel-3.1.11-2.el8_3.i686
mariadb-connector-c-devel-3.1.11-2.el8_3.x86_64
mysql-devel-8.0.26-1.module+el8.4.0+652+6de068a7.x86_64
mariadb-devel-3:10.3.28-1.module+el8.4.0+427+adf35707.x86_64
mysql-devel-8.0.26-1.module+el8.4.0+652+6de068a7.x86_64
mariadb-3:10.3.28-1.module+el8.4.0+427+adf35707.x86_64
mysql-8.0.26-1.module+el8.4.0+652+6de068a7.x86_64
libcmpiCppImpl0-2.0.3-15.el8.x86_64
tog-pegasus-libs-2:2.14.1-46.el8.x86_64
libev-libevent-devel-4.24-6.el8.i686
libev-libevent-devel-4.24-6.el8.x86_64
libevent-devel-2.1.8-5.el8.i686
libevent-devel-2.1.8-5.el8.x86_64
== File conflicts, listed by conflicting packages ==
mariadb-server-3:10.3.28-1.module+el8.4.0+427+adf35707.x86_64
mysql-test-8.0.26-1.module+el8.4.0+652+6de068a7.x86_64
/usr/bin/mysqld_safe
$ sudo umount /media
```
{% include 'testing/qa_testcase_bottom.md' %}

86
QA/Testcase_Media_Repoclosure.md Normal file
View File

@ -0,0 +1,86 @@
---
title: QA:Testcase Media Repoclosure
author: Trevor Cooper
revision_date: 2022-04-24
rc:
prod: Rocky Linux
ver: 8
level: Final
---
!!! info "Associated release criterion"
This test case is associated with the [Release_Criteria#no-broken-packages](../release_criteria.md#no-broken-packages) release criterion. If you are doing release validation testing, a failure of this test case may be a breach of that release criterion.
## Description
This testcase will verify that the offline repository included on release blocking images will not contain broken dependencies.
## Setup
1. Obtain access to an environment with the `dnf repoclosure` command.
2. Download the ISO to be tested to that machine.
## How to test
1. Mount the ISO to be tested locally.
- Example:<br>`mount -o loop Rocky-8.5-x86_64-minimal.iso /media`
2. Determine the path to the `repodata` directory(ies) on the ISO.
- Example:<br>`find /media -name repodata`
3. Run the `dnf repoclosure` command on the mounted ISO.
- Example:<br>`dnf --verbose repoclosure --repofrompath BaseOS,/media/BaseOS --repo BaseOS --repofrompath Minimal,/media/Minimal --repo Minimal`
4. Unmount the ISO.
- Example:<br>`umount /media`
## Expected Results
1. The `dnf repoclosure` command does not generate any errors.
<h3>Sample Output</h3>
=== "Success"
```
$ sudo mount -o loop Rocky-8.5-x86_64-minimal.iso /media
mount: /media: WARNING: device write-protected, mounted read-only.
[vagrant@localhost ~]$ dnf --refresh repoclosure \
--repofrompath BaseOS,/media/BaseOS --repo BaseOS \
--repofrompath Minimal,/media/Minimal --repo Minimal
Added BaseOS repo from /media/BaseOS
Added Minimal repo from /media/Minimal
BaseOS 102 MB/s | 2.6 MB 00:00
Minimal 90 kB/s | 384 B 00:00
$ sudo umount /media
```
=== "Failure"
__NOTE: In this example the content of the `Rocky-8.5-x86_64-minimal.iso` was copied to `/tmp` then the BaseOS repository was modified to remove the `setup-2.12.2-6.el8.noarch.rpm` package and the repository metadata was regenerated.__
```
[vagrant@localhost ~]$ dnf --refresh repoclosure \
--repofrompath BaseOS,/tmp/media/BaseOS --repo BaseOS \
--repofrompath Minimal,/tmp/media/Minimal --repo Minimal
Added BaseOS repo from /tmp/media/BaseOS
Added Minimal repo from /tmp/media/Minimal
BaseOS 3.7 MB/s | 3.8 kB 00:00
Minimal 3.7 MB/s | 3.8 kB 00:00
package: basesystem-11-5.el8.noarch from BaseOS
unresolved deps:
setup
package: dump-1:0.4-0.36.b46.el8.x86_64 from BaseOS
unresolved deps:
setup
package: filesystem-3.8-6.el8.x86_64 from BaseOS
unresolved deps:
setup
package: initscripts-10.00.15-1.el8.x86_64 from BaseOS
unresolved deps:
setup
package: rpcbind-1.2.5-8.el8.x86_64 from BaseOS
unresolved deps:
setup
package: shadow-utils-2:4.6-14.el8.x86_64 from BaseOS
unresolved deps:
setup
Error: Repoclosure ended with unresolved dependencies.
```
{% include 'testing/qa_testcase_bottom.md' %}

58
QA/Testcase_Media_USB_dd.md Normal file
View File

@ -0,0 +1,58 @@
---
title: QA:Testcase Media USB dd
author: Trevor Cooper
revision_date: 2021-04-21
rc:
prod: Rocky Linux
ver: 8
level: Final
---
!!! info "Associated release criterion"
This test case is associated with the [Release_Criteria#initialization-requirements](../release_criteria.md#initialization-requirements) release criterion. If you are doing release validation testing, a failure of this test case may be a breach of that release criterion.
## Description
This verifies that Rocky Linux ISO image can be written to USB media using `dd` command, and the resulting USB media successfully boots to the Anaconda Installer.
!!! error "DATA LOSS"
Any data on the USB stick used for this test is likely to be destroyed. Please do not use a stick whose contents you need to keep.
## Setup
1. Provide a USB media device that is larger than the ISO image you wish to test and that it can be completely erased.
2. Provide a Linux (or other *nix system) that has the `dd` command available and an unoccupied USB port.
3. Download the Rocky Linux ISO image you wish to test onto the test system.
- Example command:<br>`curl -LOR http://dl.rockylinux.org/pub/rocky/8/isos/x86_64/Rocky-8.5-x86_64-boot.iso`
4. Download the `CHECKSUM` file that goes with the Rocky Linux ISO image that you wish to test.
- Example command:<br>`curl -LOR http://dl.rockylinux.org/pub/rocky/8/isos/x86_64/CHECKSUM`
5. Download the `CHECKSUM.sig` file that does with the `CHECKSUM` file.
- Example command:<br>`curl -LOR http://dl.rockylinux.org/pub/rocky/8/isos/x86_64/CHECKSUM.sig`
6. Download the Rocky Release Engineering GPG key.
- Example command:<br>`curl -LOR http://dl.rockylinux.org/pub/rocky/RPM-GPG-KEY-rockyofficial`
## How to test
1. Import the Rocky Release Engineering GPG key.
- Example command:<br>`gpg --import RPM-GPG-KEY-rockyofficial`
2. Verify the signature of the CHECKSUM file.
- Example command:<br>`gpg --verify-file CHECKSUM.sig`
3. Verify the CHECKSUM of the Rocky Linux ISO...
- Example command:<br>`shasum -a 256 --ignore-missing -c CHECKSUM`
4. Write the Rocky Linux ISO to the USB media using `dd`...
- Example command:<br>`dd if=Rocky-8.5-x86_64-boot.iso of=/dev/sdX bs=16M status=progress oflag=direct`<br>...where you replace `sdX` with the device identifier of your USB media.<br>**This will destroy all data on the disk.**
5. Boot the test system with the USB media.
6. In the boot menu select the appropriate option to boot the installer.
7. **[OPTIONAL]** Proceed with installation on the test system.<br>**Depending on installer choices this MAY destroy all the data on the test system.**
## Expected Results
1. The gpg signature on the `CHECKSUM` file is valid.
2. The `CHECKSUM` of the Rocky Linux ISO is valid.
3. The Rocky Linux ISO is written to the USB stick without errors.
4. The USB stick boots without errors.
5. The Anaconda Installer starts without errors.
!!! error "DATA LOSS"
If you choose to complete the installation of the test system any/all data on the system may be lost. Please do not install on a system whose contents you need to keep.
**[OPTIONALLY]**<br>
6. The installation finishes successfully and, if the minimal or DVD ISO were used, the package repository on the USB stick (not a network based repository) was used for the installation.
{% include 'testing/qa_testcase_bottom.md' %}

96
QA/Testcase_Packages_No_Insights.md Normal file
View File

@ -0,0 +1,96 @@
---
title: QA:Testcase Packages No Insights
author: Trevor Cooper
revision_date: 2022-04-24
rc:
prod: Rocky Linux
ver: 8
level: Final
---
!!! info "Associated release criterion"
This test case is associated with the [Release_Criteria#repositories-must-match-upstream](../release_criteria.md#repositories-must-match-upstream) release criterion. If you are doing release validation testing, a failure of this test case may be a breach of that release criterion.
## Description
This test will verify that `insights-client` package is not declared be installed as part of a package group.
## Setup
1. Obtain access to an environment with the `dnf` command.
2. Download the ISO to be tested to that machine.
## How to test
1. Mount the ISO to be tested locally.
2. Determine the path to the `comps` file(s) on the ISO.
3. Verify that `insights-client` is not declared to be installed automatically.
- Example 1:<br>`find /media -name "*comps*.xml" -exec grep -H "insights-client" '{}' \;`
- Example 2:<br>`dnf --refresh --repofrompath BaseOS,/media/BaseOS --repo BaseOS --repofrompath AppStream,/media/AppStream --repo AppStream groupinfo base | grep -E ":|insights"`
4. Unmount the ISO.
## Expected Results
1. `insights-client` is not declared to be installed by default.
<h3>Sample Output</h3>
=== "Success"
!!! info "UPDATE SAMPLE"
NOTE: This example needs to be refreshed when the 8.6 ISO has been produced. As seen in the Failure section below the `Rocky-8.5-x86_64-dvd1.iso` includes the `insights-client` as part of the `base` group. The package should be included on the DVD ISO but should **not** be installed automatically.
```
$ sudo mount -o loop Rocky-8.5-aarch64-minimal.iso /media
mount: /media: WARNING: device write-protected, mounted read-only.
$ dnf --refresh --repofrompath BaseOS,/media/BaseOS --repo BaseOS --repofrompath Minimal,/media/Minimal --repo Minimal search insights-client
Added BaseOS repo from /media/BaseOS
Added Minimal repo from /media/Minimal
BaseOS 3.8 MB/s | 3.9 kB 00:00
Minimal 3.7 MB/s | 3.8 kB 00:00
No matches found.
$ find /media -name "*comps*.xml" -exec grep -H "insights-client" '{}' \;
$ dnf --refresh --repofrompath BaseOS,/media/BaseOS --repo BaseOS --repofrompath Minimal,/media/Minimal --repo Minimal groupinfo base | grep -E ":|insights"
BaseOS 3.8 MB/s | 3.9 kB 00:00
Minimal 3.7 MB/s | 3.8 kB 00:00
Group: Base
Description: The standard installation of Rocky Linux.
Mandatory Packages:
Default Packages:
Optional Packages:
$ sudo umount /media
```
=== "Failure"
```
$ sudo mount -o loop Rocky-8.5-x86_64-dvd1.iso /media
mount: /media: WARNING: device write-protected, mounted read-only.
$ dnf --refresh --repofrompath BaseOS,/media/BaseOS --repo BaseOS --repofrompath AppStream,/media/AppStream --repo AppStream search insights-client
Added BaseOS repo from /media/BaseOS
Added AppStream repo from /media/AppStream
BaseOS 3.8 MB/s | 3.9 kB 00:00
AppStream 4.2 MB/s | 4.3 kB 00:00
================================= Name Exactly Matched: insights-client ==================================
insights-client.noarch : Uploads Insights information to Red Hat on a periodic basis
$ find /media -name "*comps*.xml" -exec grep -H "insights-client" '{}' \;
/media/AppStream/repodata/a6742e1300e1c786af91656b152d3b98bb7aff598e650509381417970e1f1b7e-comps-AppStream.x86_64.xml: <packagereq type="default">insights-client</packagereq>
/media/AppStream/repodata/a6742e1300e1c786af91656b152d3b98bb7aff598e650509381417970e1f1b7e-comps-AppStream.x86_64.xml: <packagereq type="default">insights-client</packagereq>
$ dnf --refresh --repofrompath BaseOS,/media/BaseOS --repo BaseOS --repofrompath AppStream,/media/AppStream --repo AppStream groupinfo base | grep -E ":|insights"
BaseOS 3.8 MB/s | 3.9 kB 00:00
AppStream 4.2 MB/s | 4.3 kB 00:00
Group: Base
Description: The standard installation of Rocky Linux.
Mandatory Packages:
Default Packages:
insights-client
Optional Packages:
$ sudo umount /media
```
{% include 'testing/qa_testcase_bottom.md' %}

63
QA/Testcase_Packages_No_RHSM.md Normal file
View File

@ -0,0 +1,63 @@
---
title: QA:Testcase Packages No RHSM
author: Trevor Cooper
revision_date: 2022-04-24
rc:
prod: Rocky Linux
ver: 8
level: Final
---
!!! info "Associated release criterion"
This test case is associated with the [Release_Criteria#repositories-must-match-upstream](../release_criteria.md#repositories-must-match-upstream) release criterion. If you are doing release validation testing, a failure of this test case may be a breach of that release criterion.
## Description
This test will verify that packages that are availble from upstream do not have hard requirements on `subscription-manager` (RHSM).
## Setup
1. Obtain access to an environment with the `dnf` command.
2. Download the ISO to be tested to that machine.
## How to test
1. Mount the ISO to be tested locally.
2. Obtain a list of packages that have `Requires:` for `subscription-manager`
- Example:<br>`package_list=($(dnf --refresh repoquery --repofrompath BaseOS,/media/BaseOS --repo BaseOS --repofrompath AppStream,/media/AppStream --repo AppStream --whatrequires subscription-manager 2>/dev/null| grep el8))`
3. Download the packages with explicity `Requires:` for `subscription-manager`
- Example:<br>`dnf --repofrompath BaseOS,/media/BaseOS --repo BaseOS --repofrompath AppStream,/media/AppStream --repo AppStream download "${package_list[@]}"`
4. Obtain the `SOURCEPKG` definition for the above packages
- Example:<br>`rpm -q --queryformat="%{NAME}|%{SOURCERPM}\n" subscription-manager*.rpm | column -s\| -t`
4. Unmount the ISO.
## Expected Results
1. No packages have an explicit requirement for `subscription-manager`.
<h3>Sample Output</h3>
=== "Success"
```
$ sudo mount -o loop Rocky-8.5-aarch64-minimal.iso /media
mount: /media: WARNING: device write-protected, mounted read-only.
$ package_list=($(dnf --refresh repoquery --repofrompath BaseOS,/media/BaseOS --repo BaseOS --repofrompath AppStream,/media/AppStream --repo AppStream --whatrequires subscription-manager 2>/dev/null| grep el8))
$ dnf --repofrompath BaseOS,/media/BaseOS --repo BaseOS --repofrompath AppStream,/media/AppStream --repo AppStream download "${package_list[@]}"
Added BaseOS repo from /media/BaseOS
Added AppStream repo from /media/AppStream
Last metadata expiration check: 0:00:25 ago on Sun 24 Apr 2022 10:57:13 PM UTC.
$ rpm -q --queryformat="%{NAME}|%{SOURCERPM}\n" subscription-manager*.rpm | column -s\| -t
subscription-manager-cockpit subscription-manager-1.28.21-3.el8.src.rpm
subscription-manager-migration subscription-manager-1.28.21-3.el8.src.rpm
subscription-manager-plugin-ostree subscription-manager-1.28.21-3.el8.src.rpm
$ sudo umount /media
```
=== "Failure"
TBD
{% include 'testing/qa_testcase_bottom.md' %}

28
QA/Testcase_Repo_Compare.md Normal file
View File

@ -0,0 +1,28 @@
---
title: QA:Testcase Media Repo Compare
author: Trevor Cooper
revision_date: 2022-04-24
rc:
prod: Rocky Linux
ver: 8
level: Final
---
!!! info "Associated release criterion"
This test case is associated with the [Release_Criteria#repositories-must-match-upstream](../release_criteria.md#repositories-must-match-upstream) release criterion. If you are doing release validation testing, a failure of this test case may be a breach of that release criterion.
## Description
This test case will verify that repositories and the packages within them match upstream as closely as possible.
## Setup
1. Verify access to the Rocky Linux repocompare tooling.
## How to test
1. Access [Rocky Linux repocompare website](https://repocompare.rockylinux.org/).
2. Verify similarity of Rocky Linux repositories with upstream content.
## Expected Results
1. Rocky Linux repositories should match, as closely as possible, upstream repositories.
2. The content of Rocky Linux packages should match, as closely as possible, upstream repositories.
{% include 'testing/qa_testcase_bottom.md' %}

28
QA/Testcase_Template.md Normal file
View File

@ -0,0 +1,28 @@
---
title: QA:Testcase Template
author: Trevor Cooper
revision_date: 2022-04-01
rc:
prod: Rocky Linux
ver: 8
level: Final
---
!!! info "Associated release criterion"
This test case is associated with the [Release_Criteria#TBD](../release_criteria.md#TBD) release criterion. If you are doing release validation testing, a failure of this test case may be a breach of that release criterion.
## Description
Add a short description here for this Testcase.
## Setup
1. Add steps for setup for this Testcase.
## How to test
1. Do this first...
2. Then do this...
## Expected Results
1. This is what you should see/verify.
2. You should also see/verify this.
{% include 'testing/qa_testcase_bottom.md' %}

61
QA/index.md Normal file
View File

@ -0,0 +1,61 @@
---
title: QA: Test Cases
revision_date: 2022-04-20
---
This page lists all test cases in work and who is working on them...
## Initialization Requirements
| Requirement | Test Case | Assignee | Status |
| --------------------------------------------------- | ------------------------------------------------------------------------ | ----------------------- | --------------------------------------- |
| [Release-blocking images must boot](release_criteria.md#release-blocking-images-must-boot) | [QA:Testcase Boot Methods Boot ISO](Testcase_Boot_Methods_Boot_Iso.md) | @tcooper | template exists, openQA covered (ref) |
| [Release-blocking images must boot](release_criteria.md#release-blocking-images-must-boot) | [QA:Testcase Boot Methods DVD](Testcase_Boot_Methods_Dvd.md) | @tcooper | template exists, openQA covered (ref) |
| [Basic Graphics Mode behaviors](release_criteria.md#basic-graphics-mode-behaviors) | [QA:Testcase Basic Graphics Mode](Testcase_Basic_Graphics_Mode.md) | @tcooper | manual and/or new openQA TestCase |
| [No Broken Packages](release_criteria.md#no-broken-packages) | [QA:Testcase Media Repoclosure](Testcase_Media_Repoclosure.md)<br>[QA:Testcase Media File Conflicts](Testcase_Media_File_Conflicts.md) | @tcooper | manual using scripts or automated in CI |
| [Repositories Must Match Upstream](release_criteria.md#repositories-must-match-upstream) | [QA:Testcase repocompare](Testcase_Repo_Compare.md) | @tcooper | manual using Skip's repocompare |
| [Debranding](release_criteria.md#debranding) | [QA:Testcase Debranding Analysis](Testcase_Debranding.md) | @tcooper | process TBD |
## Installer Requirements
| Requirement | Test Case | Assignee | Status |
| --------------------------------------------------- | ------------------------------------------------------------------------ | ----------------------- | --------------------------------------- |
| Media Consistency Verification | [QA:Testcase TBD](Testcase_Template.md) | @tbd | |
| Packages and Installer Sources | [QA:Testcase TBD](Testcase_Template.md) | @tbd | |
| NAS (Network Attached Storage) | [QA:Testcase TBD](Testcase_Template.md) | @tbd | |
| Installation Interfaces | [QA:Testcase TBD](Testcase_Template.md) | @tbd | |
| Minimal Installation | [QA:Testcase TBD](Testcase_Template.md) | @tbd | |
| Kickstart Installation | [QA:Testcase TBD](Testcase_Template.md) | @tbd | |
| Disk Layouts | [QA:Testcase TBD](Testcase_Template.md) | @tbd | |
| Firmware RAID | [QA:Testcase TBD](Testcase_Template.md) | @tbd | |
| Bootloader Disk Selection | [QA:Testcase TBD](Testcase_Template.md) | @tbd | |
| Storage Volume Resize | [QA:Testcase TBD](Testcase_Template.md) | @tbd | |
| Update Image | [QA:Testcase TBD](Testcase_Template.md) | @tbd | |
| Installer Help | [QA:Testcase TBD](Testcase_Template.md) | @tbd | |
| Installer Translations | [QA:Testcase TBD](Testcase_Template.md) | @tbd | |
## Cloud Image Requirements
| Requirement | Test Case | Assignee | Status |
| --------------------------------------------------- | ------------------------------------------------------------------------ | ----------------------- | --------------------------------------- |
| Images Published to Cloud Providers | [QA:Testcase TBD](Testcase_Template.md) | @tbd | |
## Post-Installation Requirements
| Requirement | Test Case | Assignee | Status |
| --------------------------------------------------- | ------------------------------------------------------------------------ | ----------------------- | --------------------------------------- |
| System Services | [QA:Testcase TBD](Testcase_Template.md) | @tbd | |
| Keyboard Layout | [QA:Testcase TBD](Testcase_Template.md) | @tbd | |
| SELinux Errors (Server) | [QA:Testcase TBD](Testcase_Template.md) | @tbd | |
| SELinux and Crash Notifications (Desktop Only) | [QA:Testcase TBD](Testcase_Template.md) | @tbd | |
| Default Application Functionality (Desktop Only) | [QA:Testcase TBD](Testcase_Template.md) | @tbd | |
| Default Panel Functionality (Desktop Only) | [QA:Testcase TBD](Testcase_Template.md) | @tbd | |
| Dual Monitor Setup (Desktop Only) | [QA:Testcase TBD](Testcase_Template.md) | @tbd | |
| Artwork and Assets (Server and Desktop) | [QA:Testcase TBD](Testcase_Template.md) | @tbd | |
| Packages and Module Installation | [QA:Testcase TBD](Testcase_Template.md) | @tbd | |
{% include 'content_bottom.md' %}

14
index.md
View File

@ -1,12 +1,18 @@
---
title: Testing
revision_date: 2022-04-01
---
This section is primarily for documentation for the Testing Team. The testing team handles testing and QA for Rocky Linux.
## Members
| Role | Name | Email | Mattermost Name | IRC Name |
|------------------------------------: |------------------------------- |--------------------- |--- | --- |
| Testing Lead | Trevor Cooper | tcooper@rockylinux.org | @tcooper | |
| Testing Lead | Chris Stackpole | stack@rockylinux.org | @stack | |
| Role | Name | Email | Mattermost Name | IRC Name |
| -------------- | --------------- | ----------------------- | ------------------ | -------- |
| Testing Lead | Trevor Cooper | tcooper@rockylinux.org | @tcooper | |
| Testing Lead | Chris Stackpole | stack@rockylinux.org | @stack | |
| Testing Team | Al | | @raktajino | |
| Testing Team | Lukas Magauer | | @lumarel | |
{% include 'content_bottom.md' %}

247
release_criteria.md Normal file
View File

@ -0,0 +1,247 @@
---
title: Rocky Linux 8 Release Criteria
author: Trevor Cooper
revision_date: 2022-04-01
rc:
prod: Rocky Linux
ver: 8
level: Final
---
# {{ rc.level }} Release Objectives
{% include 'testing/rc_content_example_only.md' %}
The objective of a release (major or minor) is to provide a solid Enterprise Linux release that is suitable to:
- Meet the needs of end users
- Meet the needs of enterprises big or small
## {{ rc.level }} Release Requirements
In order for {{ rc.prod }} to be released to the general public, a compose must be able to meet all the following criteria as provided in this document. This is allows the decision process to be straightforward and as clear as possible. This document only contains “hard requirement” items. Optional/nice to have items are not to be included in this list.
There may cases where a requirement cannot be met but only in particular configurations. In these types of cases, the Release Engineering Team should use their judgement to determine whether or not the issue should be considered to block the release. They should consider the number of users likely to be affected by said issue, the severity of the case, if the issue can be avoided with ease (by both informed and uninformed users), and if the problem exists upstream in the current Red Hat Enterprise Linux that the release is based on.
!!! info "Release-blocking Server"
...means bugs as it pertains to server functionality can be considered to block a release. This applies to any packages that provide a service such as httpd, nginx, etc. All architectures apply.
!!! info "Release-blocking Desktop"
...means bugs as it pertains to desktop functionality (GNOME) can be considered to block a release. This applies to both x86_64 and aarch64. Additional desktops (as provided by EPEL or a SIG) are not considered blockers.
!!! info "Release-blocking Image"
...means bugs as it pertains to the images built that can block a release. This applies to the DVD, minimal, and boot images on all architectures.
### Initialization Requirements
#### Release-blocking images must boot
Release-blocking installer images must boot when written to optical media or USB flash drive of appropriate sizes (if applicable) via officially supported methods. It is not the testing team’s responsibility to test optical media, but they can and report back. If a bug is found, it is considered a blocker.
??? tldr "Optical Media Requirements"
Release-blocking images must boot when written to optical media of an appropriate size. Current size requirements are: boot.iso = 789M, minimal.iso = 2.0G and dvd.iso = 10G.
??? tldr "Officially supported USB flash drive writing methods"
The following methods of writing USB flash drives are officially support: dd<br>
The following methods of writing USB flash drives are _**not**_ supported: rufus
??? tldr "References"
- Test cases:
- [QA:Testcase Boot Methods Boot ISO](Testcase_Boot_Methods_Boot_Iso.md)
- [QA:Testcase Boot Methods DVD](Testcase_Boot_Methods_Dvd.md)
- [QA:Testcase Media USB dd](Testcase_Media_USB_dd.md)
#### Basic Graphics Mode behaviors
The generic video driver option (“basic graphics mode”) on all release-blocking installers must function as intended. This means launching the installer or desktop and attempting to use a generic driver. There must be no bugs that prevent the installer from being reached in this configuration on all systems and classes of hardware supported by the enterprise linux kernel.
??? tldr "References"
- Test cases:
- [QA:Testcase Basic Graphics Mode](Testcase_Basic_Graphics_Mode.md)
#### No Broken Packages
Critical errors, such as undeclared conflicts, unresolved dependencies, or modules relying on packages from another stream will be considered an automatic blocker. There are potential exceptions to this (eg, freeradius cannot be installed on an older perl stream, this is a known issue upstream).
??? tldr "References"
- Test cases:
- [QA:Testcase Media Repoclosure](Testcase_Media_Repoclosure.md)
- [QA:Testcase Media File Conflicts](Testcase_Media_File_Conflicts.md)
#### Repositories Must Match Upstream
Repositories and the packages within them should match upstream as closely as possible. Notable exceptions would be kmods, kpatch, or what is deemed as “spyware” like insights. Packages that are available from upstream should not have hard requirements on RHSM and packages that have it default built in should be patched out.
??? tldr "References"
- Test cases:
- [QA:Testcase Repo Compare](Testcase_Repo_Compare.md)
- [QA:Testcase Packages No Insights](Testcase_Packages_No_Insights.md)
- [QA:Testcase Packages No RHSM](Testcase_Packages_No_RHSM.md)
#### Debranding
Assets and functionality that are Red Hat specific should not be included. If they are not patched out, it will be considered an automatic blocker.
??? tldr "References"
- Test cases:
- [QA:Testcase Debranding](Testcase_Debranding.md)
### Installer Requirements
#### Media Consistency Verification
This means that the installer’s mechanism for verifying the install medium is intact and must complete successfully, with the assumption that the medium was correctly written. It should return a failure message if this not the case.
??? tldr "References"
- Test cases:
- [QA:Testcase TBD](Testcase_Template.md)
#### Packages and Installer Sources
The installer must be able to use all supported local/remote packages and installer sources.
??? tldr "References"
- Test cases:
- [QA:Testcase TBD](Testcase_Template.md)
#### NAS (Network Attached Storage)"
The installer must be able to detect and install to supported NAS devices (if possible and supported by the kernel).
??? tldr "References"
- Test cases:
- [QA:Testcase TBD](Testcase_Template.md)
#### Installation Interfaces
The installer must be able to complete an installation using all supported spokes.
??? tldr "References"
- Test cases:
- [QA:Testcase TBD](Testcase_Template.md)
#### Minimal Installation
A minimal installation (via network) must be able to install the minimal package set.
??? tldr "References"
- Test cases:
- [QA:Testcase TBD](Testcase_Template.md)
#### Kickstart Installation
A kickstart installation should succeed, whether from optical/USB media or via the network.
??? tldr "References"
- Test cases:
- [QA:Testcase TBD](Testcase_Template.md)
#### Disk Layouts
The installer must be able to create and install to any workable partition layout using any file system or format combination offered or supported by the installer. File systems that are not supported by the EL kernel is not tested here (this means btrfs, zfs, both of wish are not supported).
??? tldr "References"
- Test cases:
- [QA:Testcase TBD](Testcase_Template.md)
#### Firmware RAID
The installer must be able to detect and install to firmware RAID devices. Note that system-specific bugs do not count as blockers. It is likely that some hardware support might be broken or not available at all. DUDs (driver update disks) are not considered for this criteria.
??? tldr "References"
- Test cases:
- [QA:Testcase TBD](Testcase_Template.md)
#### Bootloader Disk Selection
The installer must allow the user to choose which disk the bootloader will be installed to or, if the user so chooses, not to install a bootloader.
??? tldr "References"
- Test cases:
- [QA:Testcase TBD](Testcase_Template.md)
#### Storage Volume Resize
Any installer mechanism for resizing storage volumes must correctly attempt the requested operation. This means that if the installer offers a way to resize storage volumes, then it must use the correct resizing tool with the correct parameters. However, it does not require the installer to disallow resizing of unformatted or volumes with an unknown filesystem type.
??? tldr "References"
- Test cases:
- [QA:Testcase TBD](Testcase_Template.md)
#### Update Image
The installer must be able to use an installer update image retrieved from removable media or a remote package source. This includes DUDs (driver update disks).
??? tldr "References"
- Test cases:
- [QA:Testcase TBD](Testcase_Template.md)
#### Installer Help
Any element in the installer which contains a “help” text must display the appropriate help documentation when selected.
??? tldr "References"
- Test cases:
- [QA:Testcase TBD](Testcase_Template.md)
#### Installer Translations
The installer must correctly display all complete translations that are available for use.
??? tldr "References"
- Test cases:
- [QA:Testcase TBD](Testcase_Template.md)
### Cloud Image Requirements
#### Images Published to Cloud Providers
Release-blocking cloud disk images must be published to appropriate cloud providers (such as Amazon) and they must successfully boot. This also applies to KVM based instances, such as x86 and aarch64 systems.
??? tldr "References"
- Test cases:
- [QA:Testcase TBD](Testcase_Template.md)
### Post-Installation Requirements
#### System Services
All system services present after installation must start properly, with the exception of services that require hardware which is not present. Examples of such services would be:
- sshd
- firewalld
- auditd
- chronyd
??? tldr "References"
- Test cases:
- [QA:Testcase TBD](Testcase_Template.md)
#### Keyboard Layout
If a particular keyboard layout has been configured for the system, that layout must be used:
- When unlocking storage volumes (encrypted by LUKS)
- When logging in at a TTY console
- When logging in via GDM
- After logging into a GNOME desktop system, if the user does not have their own layout configuration set.
??? tldr "References"
- Test cases:
- [QA:Testcase TBD](Testcase_Template.md)
#### SELinux Errors (Server)
There must be no SELinux denial logs in /var/log/audit/audit.log
??? tldr "References"
- Test cases:
- [QA:Testcase TBD](Testcase_Template.md)
#### SELinux and Crash Notifications (Desktop Only)
There must be no SELinux denial notifications or crash notifications on boot, during installation, or during first login.
??? tldr "References"
- Test cases:
- [QA:Testcase TBD](Testcase_Template.md)
#### Default Application Functionality (Desktop Only)
Applications that can be launched within GNOME or on the command line must start successfully and withstand basic functionality tests. This includes:
- Web browser
- File manager
- Package manager
- Image/Document Viewers
- Text editors (gedit, vim)
- Archive manager
- Terminal Emulator (gnome terminal)
- Problem Reporter
- Help Viewer
- System Settings
??? tldr "References"
- Test cases:
- [QA:Testcase TBD](Testcase_Template.md)
#### Default Panel Functionality (Desktop Only)
All elements of GNOME should function properly in regular use.
??? tldr "References"
- Test cases:
- [QA:Testcase TBD](Testcase_Template.md)
#### Dual Monitor Setup (Desktop Only)
Computers using two monitors, the graphical output is correctly shown on both monitors.
??? tldr "References"
- Test cases:
- [QA:Testcase TBD](Testcase_Template.md)
#### Artwork and Assets (Server and Desktop)
Proposed final artwork (such as wallpapers and other assets) must be included. A wallpaper from this package should show up as a default for GDM and GNOME.
??? tldr "References"
- Test cases:
- [QA:Testcase TBD](Testcase_Template.md)
#### Packages and Module Installation
Packages (non-module) should be able to be installed without conflicts or dependent on repositories outside of {{ rc.prod }}.
- Default modules (as listed in dnf module list) should be installed without requiring them to be enabled.
- Module streams should be able to be switched and those packages should be able to be installed without errors or unresolved dependencies.
??? tldr "References"
- Test cases:
- [QA:Testcase TBD](Testcase_Template.md)
{% include 'testing/rc_content_bottom.md' %}