testing
/
wiki
generated from sig_core/wiki-template
10
0
Fork 3
Code Issues 1 Pull Requests Actions Packages Projects Releases Wiki Activity

add 8.5 QA:Testing Summary and Go/NoGo Status and templates

This commit is contained in:
Trevor Cooper 2022-05-18 23:26:40 -07:00
parent a61afd053c
commit ab736ee034
5 changed files with 373 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

67
release_criteria/.qa_testing_summary_template.md Normal file
View File

@ -0,0 +1,67 @@
---
title: Release Status Template
author: Trevor Cooper
revision_date: 2022-05-18
rc:
prod: Rocky Linux
ver: 9
level: Final
---
# {{ rc.prod }} {{ rc.ver }} QA and Testing Summary
Last updated: <to_fill_in_on_etherpad_when_saving_a_version>
## Scope
This document will record a summary of all QA and Testing results for {{ rc.prod }} {{ rc.9 }} release. It is only a record of success and/or failure. Solution discussion should take place elsewhere.
## Reference
- Please check RHEL 9 Release Notes **BEFORE** marking issue here as **FAIL**.
## SOP
- Please include **PASS**, **FAIL**, **NOTABUG**, **INVESTIGATE** or **UPSTREAM** as appropriate in all entries.
- Please only provide brief summary. Details should go to Rocky Pastebin, links here is OK.
- Please leave your MM @handle on all items you have done or are working on so we can talk to you to get resolution.
- If the item you have reported is related to a QA:Testcase please mention it.
- If you think the item you have reported should be a QA:Testcase, even if it's not a current requirement, suggest a title and create an issue in the wiki repository so we can add it.
## INVESTIGATE
This is a list of items that are being INVESTIGATEd further before being assigned a PASS, FAIL, NOTABUG or UPSTREAM status.
PLEASE add your MM handle if you are working on this item to minimize duplication of work. More than one handle is allowed but please communicate.
- QA:Testcase Basic Graphics Mode - [INVESTIGATE] - @tcooper
- QA:Testcase Boot Methods Boot Iso - [INVESTIGATE] - @neil
- QA:Testcase Boot Methods DVD - [INVESTIGATE] - @neil
- QA:Testcase Debranding - [INVESTIGATE] - @tcooper
- QA:Testcase Media Consistency Verification - [INVESTIGATE] - @tcooper
- QA:Testcase Media File Conflicts - [INVESTIGATE] - @tcooper
- QA:Testcase Media File Conflicts - [INVESTIGATE] - @tcooper
- QA:Testcase Media File Conflicts - [INVESTIGATE] - @tcooper
- QA:Testcase Media File Conflicts - [INVESTIGATE] - @tcooper
- QA:Testcase Media Repoclosure - [INVESTIGATE] - @tcooper
- QA:Testcase Storage Volume Resize - [INVESTIGATE] - @raktajino
- QA:Testcase Update Image - [INVESTIGATE] - @raktajino
- QA:Testcase boot/install minimal x86_64 over DVD/Bluray - [INVESTIGATE] - @atomicturtle
- QA:Testcase_Mediacheck - [INVESTIGATE] - @tcooper
## UPSTREAM
This is a list of items that have been verified to be replicated UPSTREAM in RHEL {{ rc.9 }} and/or are described clearly in the RHEL 9 Release Notes.
- QA:Testcase_Some_Testcase - [UPSTREAM] - @your_mm_handle - <brief_description>
## FAIL
This is a list of items that have been verified to FAIL the QA:Testcase. In addition to recording who did the test please indicate if the item is BLOCKING release or not.
- QA:Testcase_Some_Testcase - [FAIL] - @your_mm_handle - <brief_description>
## NOTABUG
This is a list of items that have been verified as less than optimal but are expected and NOTABUG.
- QA:Testcase_Some_Testcase - [NOTABUG] - @your_mm_handle - <brief_description>
## PASS
This is a list of item that have been verified as PASSing the QA:Testcase named (or proposed).
- QA:Testcase_Some_Testcase - [PASS] - @your_mm_handle - <brief_description>
## OTHER NOTABLE ITEMS

89
release_criteria/.ver_qa_testing_go_no_go_template.md Normal file
View File

@ -0,0 +1,89 @@
---
title: Rocky Linux TBD QA and Testing GO / NO GO Status
author: Trevor Cooper
revision_date: 2022-05-18
rc:
prod: Rocky Linux
ver: TBD
---
# {{ rc.prod }} {{ rc.ver }} QA and Testing : GO / NO GO Status
This document summarizes the GO / NO GO Status for Release of Rocky Linux 8.6 from perspective of the QA / Testing efforts. It is based largely on the Release Criteria (https://wiki.rockylinux.org/team/testing/release_criteria/) as was started as an import of that document. If there are differences between the official Release Critieria document and this document the official Release Criteria document will override and this document shall be updated.
As a reminder, the objective of a release (major or minor) is to provide a solid Enterprise Linux release that is suitable to:
- Meet the needs of end users
- Meet the needs of enterprises big or small
## SUMMARY
| Category | Proportion | Remaining Items |
| ----------------- | ----------------- | --------------- |
| TO_CONFIRM | 0 / 29 (0%) | |
| PASS | 0 / 29 (0%) | |
| FAIL_NON_BLOCKING | 0 / 29 (0%) | |
| FAIL_BLOCKING | 0 / 29 (0%) | |
## SOP
In this document each requirement is described and status is specified in the title.
Current choices are...
### TO_CONFIRM
- this means the item may be INCOMPLETE, PASS, FAIL_NON_BLOCKING or FAIL_BLOCKING and must be verified
### PASS
- this means that the release criteria has been met and is not a blocker
### FAIL_NON_BLOCKING
- this means the release criter has not been met but is non-blocking
### FAIL_BLOCKING
- this means the release criteria has not been met and is blocking
In this document criteria status should include who completed the item and generally how it was complete.
Examples...
- PASS - (@tcooper, virt only, manual)
- PASS - (@lumarel, @raktajino, @tcooper, semi-automatic , openQA)
## Initialization Requirements
- Release-blocking images must boot - PASS - (@neil, @atomicturtle)
- Optical Media Requirements - PASS - (@atomicturtle)
- Basic Graphics Mode behaviors - PASS - (@tcooper, virt only, manual)
- No Broken Packages - PASS - (@tcooper, scripted, manual)
- Repositories Must Match Upstream - TO_CONFIRM - (@tcooper, manual)
- Debranding - PASS - (@tcooper, scripted, manual)
- Media Consistency Verification - PASS - (@tcooper, scripted, manual)
- Packages and Installer Sources - PASS - (@lumarel, semi-automatic, openQA test)
- NAS (Network Attached Storage) - TO_CONFIRM - (@lumarel?, semi-automatic, openQA dual-host test)
- Installation Interfaces - PASS - (@lumarel, @raktajino, @tcooper, semi-automatic, openQA, @atomicturtle, manual?, SCAP)
- Minimal Installation - PASS - (@lumarel, @raktajino, @tcooper, semi-automatic , openQA)
- Kickstart Installation - PASS - (@label, @tcooper, manual, createhdds)
- Disk Layouts - PASS - (@lumarel, @raktajino, @tcooper, semi-automatic, openQA)
- Firmware RAID - TO_CONFIRM - (@tbd, missing hardware support?)
- Bootloader Disk Selection - PASS - (@raktajino, manual)
- Storage Volume Resize - PASS - (@lumarel, @raktajino, @tcooper, semi-automatic, openQA)
- Update Image - PASS - (@raktajino,@tcooper, semi-automatic, openQA)
- Installer Help - PASS - (@lumarel, @raktajino, @tcooper, semi-automatic, openQA)
- Installer Translations - PASS - (@lumarel, @raktajino, @tcooper, semi-automatic, openQA)
## Cloud Image Requirements
- Images Published to Cloud Providers - FAIL_NON_BLOCKING - (@neil)
## Post-Installation Requirements
- System Services - PASS - (@lumarel, semi-automatic, openQA)
- Keyboard Layout - PASS - (@lumarel, semi-automatic, openQA)
- SELinux Errors (Server) - PASS - (@lumarel, semi-automatic, openQA)
- SELinux and Crash Notifications (Desktop Only) - PASS - (@lumarel, semi-automatic, openQA)
- Default Application Functionality (Desktop Only) - PASS - (@lumarel, semi-automatic, openQA)
- Default Panel Functionality (Desktop Only) - PASS - (@lumarel, semi-automatic, openQA)
- Dual Monitor Setup (Desktop Only) - PASS - (@lumarel, semi-automatic, openQA)
- Artwork and Assets (Server and Desktop) - PASS - (@lumarel, semi-automatic, openQA)
- Packages and Module Installation - PASS - (@lumarel, semi-automatic, openQA)

6
release_criteria/r8/.pages Normal file
View File

@ -0,0 +1,6 @@
---
nav:
- ... | index.md
- Rocky Linux 8 Release Criteria: 8_release_criteria.md
- Rocky Linux 8.6 QA and Testing Summary: 8.6_qa_testing_summary.md
- Rocky Linux 8.6 GO / NO-GO Status: 8.6_qa_testing_go_no_go.md

90
release_criteria/r8/8.6_qa_testing_go_no_go.md Normal file
View File

@ -0,0 +1,90 @@
---
title: Rocky Linux 8.6 QA and Testing GO / NO GO Status
author: Trevor Cooper
revision_date: 2022-05-18
rc:
prod: Rocky Linux
ver: 8.6
level: Final
---
# Rocky Linux 8.6 QA and Testing : GO / NO GO Status
This document summarizes the GO / NO GO Status for Release of Rocky Linux 8.6 from perspective of the QA / Testing efforts. It is based largely on the Release Criteria (https://wiki.rockylinux.org/team/testing/release_criteria/) as was started as an import of that document. If there are differences between the official Release Critieria document and this document the official Release Criteria document will override and this document shall be updated.
As a reminder, the objective of a release (major or minor) is to provide a solid Enterprise Linux release that is suitable to:
- Meet the needs of end users
- Meet the needs of enterprises big or small
## SUMMARY
| Category | Proportion | Remaining Items |
| --------------------- | ----------------- | --------------- |
| **TO_CONFIRM** | 3 / 29 (10%) | |
| **PASS** | 25 / 29 (86%) | |
| **FAIL_NON_BLOCKING** | 1 / 29 (3%) | cloud-images |
| FAIL_BLOCKING | 0 / 29 (0%) | |
## SOP
In this document each requirement is described and status is specified in the title.
Current choices are...
### TO_CONFIRM
- this means the item may be **[INCOMPLETE]**, **[PASS]**, **[FAIL_NON_BLOCKING[]** or **[FAIL_BLOCKING]** and must be verified
### PASS
- this means that the release criteria has been met and is not a blocker
### FAIL_NON_BLOCKING
- this means the release criter has not been met but is non-blocking
### FAIL_BLOCKING
- this means the release criteria has not been met and is blocking
In this document criteria status should include who completed the item and generally how it was complete.
Examples...
- **[PASS]** - (@tcooper, virt only, manual)
- **[PASS]** - (@lumarel, @raktajino, @tcooper, semi-automatic , openQA)
## Initialization Requirements
- Release-blocking images must boot - **[PASS]** - (@neil, @atomicturtle)
- Optical Media Requirements - **[PASS]** - (@atomicturtle)
- Basic Graphics Mode behaviors - **[PASS]** - (@tcooper, virt only, manual)
- No Broken Packages - **[PASS]** - (@tcooper, scripted, manual)
- Repositories Must Match Upstream -**TO_CONFIRM**- (@tcooper, manual)
- Debranding - **[PASS]** - (@tcooper, scripted, manual)
- Media Consistency Verification - **[PASS]** - (@tcooper, scripted, manual)
- Packages and Installer Sources - **[PASS]** - (@lumarel, semi-automatic, openQA test)
- NAS (Network Attached Storage) -**TO_CONFIRM**- (@lumarel?, semi-automatic, openQA dual-host test)
- Installation Interfaces - **[PASS]** - (@lumarel, @raktajino, @tcooper, semi-automatic, openQA, @atomicturtle, manual?, SCAP)
- Minimal Installation - **[PASS]** - (@lumarel, @raktajino, @tcooper, semi-automatic , openQA)
- Kickstart Installation - **[PASS]** - (@label, @tcooper, manual, createhdds)
- Disk Layouts - **[PASS]** - (@lumarel, @raktajino, @tcooper, semi-automatic, openQA)
- Firmware RAID -**TO_CONFIRM**- (@tbd, missing hardware support?)
- Bootloader Disk Selection - **[PASS]** - (@raktajino, manual)
- Storage Volume Resize - **[PASS]** - (@lumarel, @raktajino, @tcooper, semi-automatic, openQA)
- Update Image - **[PASS]** - (@raktajino,@tcooper, semi-automatic, openQA)
- Installer Help - **[PASS]** - (@lumarel, @raktajino, @tcooper, semi-automatic, openQA)
- Installer Translations - **[PASS]** - (@lumarel, @raktajino, @tcooper, semi-automatic, openQA)
## Cloud Image Requirements
- Images Published to Cloud Providers - **FAIL_NON_BLOCKING**- (@neil)
## Post-Installation Requirements
- System Services - **[PASS]** - (@lumarel, semi-automatic, openQA)
- Keyboard Layout - **[PASS]** - (@lumarel, semi-automatic, openQA)
- SELinux Errors (Server) - **[PASS]** - (@lumarel, semi-automatic, openQA)
- SELinux and Crash Notifications (Desktop Only) - **[PASS]** - (@lumarel, semi-automatic, openQA)
- Default Application Functionality (Desktop Only) - **[PASS]** - (@lumarel, semi-automatic, openQA)
- Default Panel Functionality (Desktop Only) - **[PASS]** - (@lumarel, semi-automatic, openQA)
- Dual Monitor Setup (Desktop Only) - **[PASS]** - (@lumarel, semi-automatic, openQA)
- Artwork and Assets (Server and Desktop) - **[PASS]** - (@lumarel, semi-automatic, openQA)
- Packages and Module Installation - **[PASS]** - (@lumarel, semi-automatic, openQA)

121
release_criteria/r8/8.6_qa_testing_summary.md Normal file
View File

@ -0,0 +1,121 @@
---
title: Rocky Linux 8.6 QA and Testing Summary
author: Trevor Cooper
revision_date: 2022-05-18
rc:
prod: Rocky Linux
ver: 8.6
level: Final
---
# Rocky Linux 8.6 QA and Testing Summary
**_Last updated: Fri May 13 17:36:41 UTC 2022_**
## Scope
This document will record a summary of all QA and Testing results for Rocky Linux 8.6 release. It is only a record of success and/or failure. Solution discussion should take place elsewhere.
## SOP
- Please include PASS, FAIL, NOTABUG, INVESTIGATE or UPSTREAM as appropriate in all entries.
- Please only provide brief summary. Details should go to Rocky Pastebin, links here is OK.
- Especially for an negative result please leave your MM @handle so we can talk to you to get resolution.
- If the item you are reported is related to a QA:Testcase please mention it. If it should be a QA:Testcase, even if it's not a current requirement, suggest a title and create an issue in the wiki repository so we can add it.
## INVESTIGATE
- errors in all tests in openQA - INVESTIGATE - see openQA section below.
- INVESTIGATE whether kdump issue affects qcows with encrypted partitions in createhdds. These are pre-reqs for openQA multi-machine tests. NOTE: This is addressed below in section - @tcooper
- KDE and XFCE Life images are broken - INVESTIGATE - @label
## UPSTREAM
- Anaconda error when specific steps get executed in the right order (configure network -> disable kdump -> select some os install group -> configure default storage -> configure the storage a second time, but this time with encryption enabled, confirmed via several openqa test suites and manual installation on ESXi) - retested in RC1_2 - UPSTREAM - @lumarel
- Issue repeated on RHEL8.6 - @atomicturtle
- Issue reported to RH https://bugzilla.redhat.com/show_bug.cgi?id=2085321 - @stack
## FAIL
- ESXi secureboot (x86_64) still FAILing, but expected - @lumarel
## NOTABUG
- Minimal: Selecting a SCAP profile with dependencies not available (aide, etc), selecting Ignore dependency during installation will crash anaconda at the final oscap check. NOTABUG, this is for documentation - @atomicturtle
- Minimal ISO: is missing the source for rsyslog again, and somehow also doesn't pull it in when installing, which means it is missing it after the install (doesn't happen for boot ISO and dvd1 ISO) - NOTABUG (per @label) - @lumarel
- Minimal ISO: if the server base environment is installed with the minimal iso and cockpit is enabled after the installation, the SELinux submenu shows an error "semanage: command not found" (doesn't happen with boot/dvd-iso) - also marked as expected - NOTABUG (per @label) - @lumarel
## Manual success reported in MM
- minimal install from minimal ISO fine - PASS - @Luna Jernberg
- workstation (x86_64) install with applications fine - retested in RC1_2 - PASS - @lumarel
- all repos are available with the exact naming as they are in the rocky-repos package (nfv needs fix for that) - retested in RC1_2 - PASS - @lumarel
- packer build for 8.6 worked flawlessly - retested by @neil in RC1_2 - PASS - @gmazrael
- security profiles look good in anaconda UI - PASS - @atomicturtle (need openQA testing)
- minimal and dvd recognized as Rocky Linux 8 in KVM - PASS - @atomicturtle
- CIS profiles confirmed good in lvl1/2 in anaconda - PASS - @atomicturtle
- DISA profiles confirmed good in anaconda - PASS - @atomicturtle
- DVD: libvirt correctly boots when Rocky Linux 8 profile is selected - PASS - @atomicturtle
- SELinux checks on Server (x86_64) (letting it run for an hour and run sealert -a /var/log/audit/audit.log) - everything okay - retested in RC1_2 - PASS - @lumarel
- SELinux checks on Desktop (x86_64) (start the GNOME SETroubleshooter after some minutes of running) - everything okay - retested in RC1_2 - PASS - @lumarel
- DVD: Anaconda manual network configuration, and PCI-DSS SCAP profile selected confirmed good - PASS - @atomicturtle
- QA:Testcase_Mediacheck - PASS for all x86_64 ISOs - @tcooper
- QA:Testcase_Mediacheck - PASS for all aarch64 ISOs - @tcooper
- QA:Testcase Media Repoclosure - PASS for minimal & dvd1 for x86_64 & aarch64 (confirms RelEng results) - @tcooper
- QA:Testcase Media File Conflicts - PASS for minimal for x86_64 & aarch64 (0 file conflicts found and 0 package conflicts found) - @tcooper
- QA:Testcase Basic Graphics Mode - PASS - verified manually for Rocky-8.6-x86_64-dvd1.iso in VirtualBox on macOS X - @tcooper
- DVD: Anaconda install with 3rd party repo, encrypted filesystem, HIPAA SCAP profile selected, confirmed good - PASS - @atomicturtle
- Upgrade tests on several test machines from 8.5 to 8.6, no issues no SELinux alerts - PASS - @lumarel
- All module streams except perl:5.32 and log4j:2 correctly have the dependencies set and packages look to be built correctly - PASS - @lumarel
- log4j module stream was broken, (should be able to hook against java-8 and 11) - got fixed now in RC1_2 - PASS - @lumarel
- Anything perl 5.32 (module stream) was broken - got fixed in RC1_2 - PASS - @nazunalika
- Greenbone appliance installation test (https://rpa.st/DQNA) - PASS - @atomicturtle
- QA:Testcase Debranding for RC2 content from koji (srpms, kernel-rt and pcs are not all on the dvd1) - 46/47 PASS , 1 FALSE PASS - https://rpa.st/raw/QK3A - @tcooper
- QA:Testcase Media Consistency Verification (not written yet) for all RC2 ISOs x86_64, aarch64 - PASS - @tcooper
- QA:Testcase Media File Conflicts - EXPECTED(per @label) for Rocky-8.6-x86_64-dvd1.iso (4 file conflicts found and 13 package conflicts found, these appear to be same as 8.5 conflict between mariadb and mysql packages/files, full results - https://rpa.st/raw/ZWPQ) - @tcooper
- QA:Testcase Media File Conflicts - EXPECTED(per @label) for Rocky-8.6-aarch64-dvd1.iso (modular dependency problems, 3 file conflicts found 4 package conflicts found, full results - https://rpa.st/raw/KOFQ) - @tcooper
- QA:Testcase Media File Conflicts for both x86_64 (https://rpa.st/raw/NLGA) and aarch64 (https://rpa.st/raw/4SFQ) are essentially unchanged and remain - EXPECTED(per @label) with RC1_2 ISOs. - @tcooper
- OpenQA tests @lumarel - there are errors from the test cases, but everything image and repo related looks good - PASS - @lumarel
- the dvd1 iso of aarch64 doesn't show an workstation base environment - it doesn't have an workstation environment - PASS - @lumarel
- Installs of aarch64 systems of all 3 isos look good and installs with all base environments work as expected from these - PASS - @lumarel
- Live Image Workstation and Workstation Lite looks good - PASS - @lumarel
- QA:Testcase Boot Methods Boot Iso - PASS - @neil
- QA:Testcase Boot Methods DVD - PASS - @neil
- QA: Testcase boot/install minimal x86_64 over DVD/Bluray (burned with fedora mediawriter) on G752 ASUS laptop - PASS - @atomicturtle
- Container images for x86_64 and aarch64 work as expected in Docker, Podman and WSL - PASS - @lumarel
- QA: Testcase Storage Volume Resize - PASS - @raktajino https://rpa.st/MQSA
- QA: Testcase Update Image - PASS - @raktajino (manually checked against Fedora's testcase (https://fedoraproject.org/wiki/QA:Testcase_Anaconda_updates.img_via_URL), needles also still match in openQA)
## openQA summary
- errors in all tests in openQA - INVESTIGATE
- RC1_1 run 1:
- @lumarel: https://rpa.st/CCPQ
- @raktajino: https://rpa.st/5RVA
- RC1_1 run 2
- @lumarel: https://rpa.st/FWTQ
- @raktajino:
- RC1_2 run 1:
- @lumarel: https://rpa.st/EOGQ
- @raktajino:https://rpa.st/VHLQ
- RC1_2 run 2:
- @raktajino: https://rpa.st/DKCQ
- Upgrade F35 -> F36 needs postgresql-setup --upgrade to convert openqa databse to new format - @alangm
- Per discussion in Testing Team meeting we have 4-8 (ish) issues to fix in openQA (needles and code) to be able to complete all tests. @lumarel has created issues our openQA repo (https://github.com/rocky-linux/os-autoinst-distri-rocky) and we'll pick up and resolve ASAP.
##createhdds kickstart file test summary
Test method: Used packer to build VM. Booted VM. Verified root login. Shutdown VM.
- UEFI Testing:
- desktop.ks - PASS - Note: resulting image asks for EULA acceptance when booted due to `firstboot --enable` (unsure if that is desired behavior)
- desktopencrypt.ks - PASS - Note: resulting image asks for EULA acceptance when booted due to `firstboot --enable` (unsure if that is desired behavior)
- minimal-uefi.ks - PASS
- server.ks - PASS
- support.ks - PASS
- BIOS Testing:
- desktop.ks - PASS - Note: resulting image asks for EULA acceptance when booted due to `firstboot --enable` (unsure if that is desired behavior)
- desktopencrypt.ks - PASS - Note: resulting image asks for EULA acceptance when booted due to `firstboot --enable` (unsure if that is desired behavior)
- minimal.ks - PASS
- server.ks - PASS
- support.ks - PASS