diff --git a/release_criteria/.qa_testing_summary_template.md b/release_criteria/.qa_testing_summary_template.md new file mode 100644 index 0000000..28abe0b --- /dev/null +++ b/release_criteria/.qa_testing_summary_template.md @@ -0,0 +1,67 @@ +--- +title: Release Status Template +author: Trevor Cooper +revision_date: 2022-05-18 +rc: + prod: Rocky Linux + ver: 9 + level: Final +--- + +# {{ rc.prod }} {{ rc.ver }} QA and Testing Summary +Last updated: + +## Scope +This document will record a summary of all QA and Testing results for {{ rc.prod }} {{ rc.9 }} release. It is only a record of success and/or failure. Solution discussion should take place elsewhere. + +## Reference +- Please check RHEL 9 Release Notes **BEFORE** marking issue here as **FAIL**. + +## SOP +- Please include **PASS**, **FAIL**, **NOTABUG**, **INVESTIGATE** or **UPSTREAM** as appropriate in all entries. +- Please only provide brief summary. Details should go to Rocky Pastebin, links here is OK. +- Please leave your MM @handle on all items you have done or are working on so we can talk to you to get resolution. +- If the item you have reported is related to a QA:Testcase please mention it. +- If you think the item you have reported should be a QA:Testcase, even if it's not a current requirement, suggest a title and create an issue in the wiki repository so we can add it. + +## INVESTIGATE +This is a list of items that are being INVESTIGATEd further before being assigned a PASS, FAIL, NOTABUG or UPSTREAM status. +PLEASE add your MM handle if you are working on this item to minimize duplication of work. More than one handle is allowed but please communicate. + +- QA:Testcase Basic Graphics Mode - [INVESTIGATE] - @tcooper +- QA:Testcase Boot Methods Boot Iso - [INVESTIGATE] - @neil +- QA:Testcase Boot Methods DVD - [INVESTIGATE] - @neil +- QA:Testcase Debranding - [INVESTIGATE] - @tcooper +- QA:Testcase Media Consistency Verification - [INVESTIGATE] - @tcooper +- QA:Testcase Media File Conflicts - [INVESTIGATE] - @tcooper +- QA:Testcase Media File Conflicts - [INVESTIGATE] - @tcooper +- QA:Testcase Media File Conflicts - [INVESTIGATE] - @tcooper +- QA:Testcase Media File Conflicts - [INVESTIGATE] - @tcooper +- QA:Testcase Media Repoclosure - [INVESTIGATE] - @tcooper +- QA:Testcase Storage Volume Resize - [INVESTIGATE] - @raktajino +- QA:Testcase Update Image - [INVESTIGATE] - @raktajino +- QA:Testcase boot/install minimal x86_64 over DVD/Bluray - [INVESTIGATE] - @atomicturtle +- QA:Testcase_Mediacheck - [INVESTIGATE] - @tcooper + +## UPSTREAM +This is a list of items that have been verified to be replicated UPSTREAM in RHEL {{ rc.9 }} and/or are described clearly in the RHEL 9 Release Notes. + +- QA:Testcase_Some_Testcase - [UPSTREAM] - @your_mm_handle - + +## FAIL +This is a list of items that have been verified to FAIL the QA:Testcase. In addition to recording who did the test please indicate if the item is BLOCKING release or not. + +- QA:Testcase_Some_Testcase - [FAIL] - @your_mm_handle - + +## NOTABUG +This is a list of items that have been verified as less than optimal but are expected and NOTABUG. + +- QA:Testcase_Some_Testcase - [NOTABUG] - @your_mm_handle - + +## PASS +This is a list of item that have been verified as PASSing the QA:Testcase named (or proposed). + +- QA:Testcase_Some_Testcase - [PASS] - @your_mm_handle - + +## OTHER NOTABLE ITEMS + diff --git a/release_criteria/.ver_qa_testing_go_no_go_template.md b/release_criteria/.ver_qa_testing_go_no_go_template.md new file mode 100644 index 0000000..8590d13 --- /dev/null +++ b/release_criteria/.ver_qa_testing_go_no_go_template.md @@ -0,0 +1,89 @@ +--- +title: Rocky Linux TBD QA and Testing GO / NO GO Status +author: Trevor Cooper +revision_date: 2022-05-18 +rc: + prod: Rocky Linux + ver: TBD +--- + +# {{ rc.prod }} {{ rc.ver }} QA and Testing : GO / NO GO Status +This document summarizes the GO / NO GO Status for Release of Rocky Linux 8.6 from perspective of the QA / Testing efforts. It is based largely on the Release Criteria (https://wiki.rockylinux.org/team/testing/release_criteria/) as was started as an import of that document. If there are differences between the official Release Critieria document and this document the official Release Criteria document will override and this document shall be updated. + +As a reminder, the objective of a release (major or minor) is to provide a solid Enterprise Linux release that is suitable to: +- Meet the needs of end users +- Meet the needs of enterprises big or small + + +## SUMMARY + +| Category | Proportion | Remaining Items | +| ----------------- | ----------------- | --------------- | +| TO_CONFIRM | 0 / 29 (0%) | | +| PASS | 0 / 29 (0%) | | +| FAIL_NON_BLOCKING | 0 / 29 (0%) | | +| FAIL_BLOCKING | 0 / 29 (0%) | | + + +## SOP +In this document each requirement is described and status is specified in the title. + +Current choices are... + +### TO_CONFIRM +- this means the item may be INCOMPLETE, PASS, FAIL_NON_BLOCKING or FAIL_BLOCKING and must be verified + +### PASS +- this means that the release criteria has been met and is not a blocker + +### FAIL_NON_BLOCKING +- this means the release criter has not been met but is non-blocking + +### FAIL_BLOCKING +- this means the release criteria has not been met and is blocking + +In this document criteria status should include who completed the item and generally how it was complete. + +Examples... + +- PASS - (@tcooper, virt only, manual) +- PASS - (@lumarel, @raktajino, @tcooper, semi-automatic , openQA) + + +## Initialization Requirements + +- Release-blocking images must boot - PASS - (@neil, @atomicturtle) +- Optical Media Requirements - PASS - (@atomicturtle) +- Basic Graphics Mode behaviors - PASS - (@tcooper, virt only, manual) +- No Broken Packages - PASS - (@tcooper, scripted, manual) +- Repositories Must Match Upstream - TO_CONFIRM - (@tcooper, manual) +- Debranding - PASS - (@tcooper, scripted, manual) +- Media Consistency Verification - PASS - (@tcooper, scripted, manual) +- Packages and Installer Sources - PASS - (@lumarel, semi-automatic, openQA test) +- NAS (Network Attached Storage) - TO_CONFIRM - (@lumarel?, semi-automatic, openQA dual-host test) +- Installation Interfaces - PASS - (@lumarel, @raktajino, @tcooper, semi-automatic, openQA, @atomicturtle, manual?, SCAP) +- Minimal Installation - PASS - (@lumarel, @raktajino, @tcooper, semi-automatic , openQA) +- Kickstart Installation - PASS - (@label, @tcooper, manual, createhdds) +- Disk Layouts - PASS - (@lumarel, @raktajino, @tcooper, semi-automatic, openQA) +- Firmware RAID - TO_CONFIRM - (@tbd, missing hardware support?) +- Bootloader Disk Selection - PASS - (@raktajino, manual) +- Storage Volume Resize - PASS - (@lumarel, @raktajino, @tcooper, semi-automatic, openQA) +- Update Image - PASS - (@raktajino,@tcooper, semi-automatic, openQA) +- Installer Help - PASS - (@lumarel, @raktajino, @tcooper, semi-automatic, openQA) +- Installer Translations - PASS - (@lumarel, @raktajino, @tcooper, semi-automatic, openQA) + + +## Cloud Image Requirements +- Images Published to Cloud Providers - FAIL_NON_BLOCKING - (@neil) + + +## Post-Installation Requirements +- System Services - PASS - (@lumarel, semi-automatic, openQA) +- Keyboard Layout - PASS - (@lumarel, semi-automatic, openQA) +- SELinux Errors (Server) - PASS - (@lumarel, semi-automatic, openQA) +- SELinux and Crash Notifications (Desktop Only) - PASS - (@lumarel, semi-automatic, openQA) +- Default Application Functionality (Desktop Only) - PASS - (@lumarel, semi-automatic, openQA) +- Default Panel Functionality (Desktop Only) - PASS - (@lumarel, semi-automatic, openQA) +- Dual Monitor Setup (Desktop Only) - PASS - (@lumarel, semi-automatic, openQA) +- Artwork and Assets (Server and Desktop) - PASS - (@lumarel, semi-automatic, openQA) +- Packages and Module Installation - PASS - (@lumarel, semi-automatic, openQA) diff --git a/release_criteria/r8/.pages b/release_criteria/r8/.pages new file mode 100644 index 0000000..cff21a5 --- /dev/null +++ b/release_criteria/r8/.pages @@ -0,0 +1,6 @@ +--- +nav: + - ... | index.md + - Rocky Linux 8 Release Criteria: 8_release_criteria.md + - Rocky Linux 8.6 QA and Testing Summary: 8.6_qa_testing_summary.md + - Rocky Linux 8.6 GO / NO-GO Status: 8.6_qa_testing_go_no_go.md diff --git a/release_criteria/r8/8.6_qa_testing_go_no_go.md b/release_criteria/r8/8.6_qa_testing_go_no_go.md new file mode 100644 index 0000000..26ca71c --- /dev/null +++ b/release_criteria/r8/8.6_qa_testing_go_no_go.md @@ -0,0 +1,90 @@ +--- +title: Rocky Linux 8.6 QA and Testing GO / NO GO Status +author: Trevor Cooper +revision_date: 2022-05-18 +rc: + prod: Rocky Linux + ver: 8.6 + level: Final +--- + +# Rocky Linux 8.6 QA and Testing : GO / NO GO Status +This document summarizes the GO / NO GO Status for Release of Rocky Linux 8.6 from perspective of the QA / Testing efforts. It is based largely on the Release Criteria (https://wiki.rockylinux.org/team/testing/release_criteria/) as was started as an import of that document. If there are differences between the official Release Critieria document and this document the official Release Criteria document will override and this document shall be updated. + +As a reminder, the objective of a release (major or minor) is to provide a solid Enterprise Linux release that is suitable to: +- Meet the needs of end users +- Meet the needs of enterprises big or small + + +## SUMMARY + +| Category | Proportion | Remaining Items | +| --------------------- | ----------------- | --------------- | +| **TO_CONFIRM** | 3 / 29 (10%) | | +| **PASS** | 25 / 29 (86%) | | +| **FAIL_NON_BLOCKING** | 1 / 29 (3%) | cloud-images | +| FAIL_BLOCKING | 0 / 29 (0%) | | + + +## SOP +In this document each requirement is described and status is specified in the title. + +Current choices are... + +### TO_CONFIRM +- this means the item may be **[INCOMPLETE]**, **[PASS]**, **[FAIL_NON_BLOCKING[]** or **[FAIL_BLOCKING]** and must be verified + +### PASS +- this means that the release criteria has been met and is not a blocker + +### FAIL_NON_BLOCKING +- this means the release criter has not been met but is non-blocking + +### FAIL_BLOCKING +- this means the release criteria has not been met and is blocking + +In this document criteria status should include who completed the item and generally how it was complete. + +Examples... + +- **[PASS]** - (@tcooper, virt only, manual) +- **[PASS]** - (@lumarel, @raktajino, @tcooper, semi-automatic , openQA) + + +## Initialization Requirements + +- Release-blocking images must boot - **[PASS]** - (@neil, @atomicturtle) +- Optical Media Requirements - **[PASS]** - (@atomicturtle) +- Basic Graphics Mode behaviors - **[PASS]** - (@tcooper, virt only, manual) +- No Broken Packages - **[PASS]** - (@tcooper, scripted, manual) +- Repositories Must Match Upstream -**TO_CONFIRM**- (@tcooper, manual) +- Debranding - **[PASS]** - (@tcooper, scripted, manual) +- Media Consistency Verification - **[PASS]** - (@tcooper, scripted, manual) +- Packages and Installer Sources - **[PASS]** - (@lumarel, semi-automatic, openQA test) +- NAS (Network Attached Storage) -**TO_CONFIRM**- (@lumarel?, semi-automatic, openQA dual-host test) +- Installation Interfaces - **[PASS]** - (@lumarel, @raktajino, @tcooper, semi-automatic, openQA, @atomicturtle, manual?, SCAP) +- Minimal Installation - **[PASS]** - (@lumarel, @raktajino, @tcooper, semi-automatic , openQA) +- Kickstart Installation - **[PASS]** - (@label, @tcooper, manual, createhdds) +- Disk Layouts - **[PASS]** - (@lumarel, @raktajino, @tcooper, semi-automatic, openQA) +- Firmware RAID -**TO_CONFIRM**- (@tbd, missing hardware support?) +- Bootloader Disk Selection - **[PASS]** - (@raktajino, manual) +- Storage Volume Resize - **[PASS]** - (@lumarel, @raktajino, @tcooper, semi-automatic, openQA) +- Update Image - **[PASS]** - (@raktajino,@tcooper, semi-automatic, openQA) +- Installer Help - **[PASS]** - (@lumarel, @raktajino, @tcooper, semi-automatic, openQA) +- Installer Translations - **[PASS]** - (@lumarel, @raktajino, @tcooper, semi-automatic, openQA) + + +## Cloud Image Requirements +- Images Published to Cloud Providers - **FAIL_NON_BLOCKING**- (@neil) + + +## Post-Installation Requirements +- System Services - **[PASS]** - (@lumarel, semi-automatic, openQA) +- Keyboard Layout - **[PASS]** - (@lumarel, semi-automatic, openQA) +- SELinux Errors (Server) - **[PASS]** - (@lumarel, semi-automatic, openQA) +- SELinux and Crash Notifications (Desktop Only) - **[PASS]** - (@lumarel, semi-automatic, openQA) +- Default Application Functionality (Desktop Only) - **[PASS]** - (@lumarel, semi-automatic, openQA) +- Default Panel Functionality (Desktop Only) - **[PASS]** - (@lumarel, semi-automatic, openQA) +- Dual Monitor Setup (Desktop Only) - **[PASS]** - (@lumarel, semi-automatic, openQA) +- Artwork and Assets (Server and Desktop) - **[PASS]** - (@lumarel, semi-automatic, openQA) +- Packages and Module Installation - **[PASS]** - (@lumarel, semi-automatic, openQA) diff --git a/release_criteria/r8/8.6_qa_testing_summary.md b/release_criteria/r8/8.6_qa_testing_summary.md new file mode 100644 index 0000000..31b69ac --- /dev/null +++ b/release_criteria/r8/8.6_qa_testing_summary.md @@ -0,0 +1,121 @@ +--- +title: Rocky Linux 8.6 QA and Testing Summary +author: Trevor Cooper +revision_date: 2022-05-18 +rc: + prod: Rocky Linux + ver: 8.6 + level: Final +--- + +# Rocky Linux 8.6 QA and Testing Summary +**_Last updated: Fri May 13 17:36:41 UTC 2022_** + + +## Scope +This document will record a summary of all QA and Testing results for Rocky Linux 8.6 release. It is only a record of success and/or failure. Solution discussion should take place elsewhere. + + +## SOP +- Please include PASS, FAIL, NOTABUG, INVESTIGATE or UPSTREAM as appropriate in all entries. +- Please only provide brief summary. Details should go to Rocky Pastebin, links here is OK. +- Especially for an negative result please leave your MM @handle so we can talk to you to get resolution. +- If the item you are reported is related to a QA:Testcase please mention it. If it should be a QA:Testcase, even if it's not a current requirement, suggest a title and create an issue in the wiki repository so we can add it. + + +## INVESTIGATE +- errors in all tests in openQA - INVESTIGATE - see openQA section below. +- INVESTIGATE whether kdump issue affects qcows with encrypted partitions in createhdds. These are pre-reqs for openQA multi-machine tests. NOTE: This is addressed below in section - @tcooper +- KDE and XFCE Life images are broken - INVESTIGATE - @label + + +## UPSTREAM +- Anaconda error when specific steps get executed in the right order (configure network -> disable kdump -> select some os install group -> configure default storage -> configure the storage a second time, but this time with encryption enabled, confirmed via several openqa test suites and manual installation on ESXi) - retested in RC1_2 - UPSTREAM - @lumarel +- Issue repeated on RHEL8.6 - @atomicturtle +- Issue reported to RH https://bugzilla.redhat.com/show_bug.cgi?id=2085321 - @stack + + +## FAIL +- ESXi secureboot (x86_64) still FAILing, but expected - @lumarel + + +## NOTABUG +- Minimal: Selecting a SCAP profile with dependencies not available (aide, etc), selecting Ignore dependency during installation will crash anaconda at the final oscap check. NOTABUG, this is for documentation - @atomicturtle +- Minimal ISO: is missing the source for rsyslog again, and somehow also doesn't pull it in when installing, which means it is missing it after the install (doesn't happen for boot ISO and dvd1 ISO) - NOTABUG (per @label) - @lumarel +- Minimal ISO: if the server base environment is installed with the minimal iso and cockpit is enabled after the installation, the SELinux submenu shows an error "semanage: command not found" (doesn't happen with boot/dvd-iso) - also marked as expected - NOTABUG (per @label) - @lumarel + + +## Manual success reported in MM +- minimal install from minimal ISO fine - PASS - @Luna Jernberg +- workstation (x86_64) install with applications fine - retested in RC1_2 - PASS - @lumarel +- all repos are available with the exact naming as they are in the rocky-repos package (nfv needs fix for that) - retested in RC1_2 - PASS - @lumarel +- packer build for 8.6 worked flawlessly - retested by @neil in RC1_2 - PASS - @gmazrael +- security profiles look good in anaconda UI - PASS - @atomicturtle (need openQA testing) +- minimal and dvd recognized as Rocky Linux 8 in KVM - PASS - @atomicturtle +- CIS profiles confirmed good in lvl1/2 in anaconda - PASS - @atomicturtle +- DISA profiles confirmed good in anaconda - PASS - @atomicturtle +- DVD: libvirt correctly boots when Rocky Linux 8 profile is selected - PASS - @atomicturtle +- SELinux checks on Server (x86_64) (letting it run for an hour and run sealert -a /var/log/audit/audit.log) - everything okay - retested in RC1_2 - PASS - @lumarel +- SELinux checks on Desktop (x86_64) (start the GNOME SETroubleshooter after some minutes of running) - everything okay - retested in RC1_2 - PASS - @lumarel +- DVD: Anaconda manual network configuration, and PCI-DSS SCAP profile selected confirmed good - PASS - @atomicturtle +- QA:Testcase_Mediacheck - PASS for all x86_64 ISOs - @tcooper +- QA:Testcase_Mediacheck - PASS for all aarch64 ISOs - @tcooper +- QA:Testcase Media Repoclosure - PASS for minimal & dvd1 for x86_64 & aarch64 (confirms RelEng results) - @tcooper +- QA:Testcase Media File Conflicts - PASS for minimal for x86_64 & aarch64 (0 file conflicts found and 0 package conflicts found) - @tcooper +- QA:Testcase Basic Graphics Mode - PASS - verified manually for Rocky-8.6-x86_64-dvd1.iso in VirtualBox on macOS X - @tcooper +- DVD: Anaconda install with 3rd party repo, encrypted filesystem, HIPAA SCAP profile selected, confirmed good - PASS - @atomicturtle +- Upgrade tests on several test machines from 8.5 to 8.6, no issues no SELinux alerts - PASS - @lumarel +- All module streams except perl:5.32 and log4j:2 correctly have the dependencies set and packages look to be built correctly - PASS - @lumarel +- log4j module stream was broken, (should be able to hook against java-8 and 11) - got fixed now in RC1_2 - PASS - @lumarel +- Anything perl 5.32 (module stream) was broken - got fixed in RC1_2 - PASS - @nazunalika +- Greenbone appliance installation test (https://rpa.st/DQNA) - PASS - @atomicturtle +- QA:Testcase Debranding for RC2 content from koji (srpms, kernel-rt and pcs are not all on the dvd1) - 46/47 PASS , 1 FALSE PASS - https://rpa.st/raw/QK3A - @tcooper +- QA:Testcase Media Consistency Verification (not written yet) for all RC2 ISOs x86_64, aarch64 - PASS - @tcooper +- QA:Testcase Media File Conflicts - EXPECTED(per @label) for Rocky-8.6-x86_64-dvd1.iso (4 file conflicts found and 13 package conflicts found, these appear to be same as 8.5 conflict between mariadb and mysql packages/files, full results - https://rpa.st/raw/ZWPQ) - @tcooper +- QA:Testcase Media File Conflicts - EXPECTED(per @label) for Rocky-8.6-aarch64-dvd1.iso (modular dependency problems, 3 file conflicts found 4 package conflicts found, full results - https://rpa.st/raw/KOFQ) - @tcooper +- QA:Testcase Media File Conflicts for both x86_64 (https://rpa.st/raw/NLGA) and aarch64 (https://rpa.st/raw/4SFQ) are essentially unchanged and remain - EXPECTED(per @label) with RC1_2 ISOs. - @tcooper +- OpenQA tests @lumarel - there are errors from the test cases, but everything image and repo related looks good - PASS - @lumarel +- the dvd1 iso of aarch64 doesn't show an workstation base environment - it doesn't have an workstation environment - PASS - @lumarel +- Installs of aarch64 systems of all 3 isos look good and installs with all base environments work as expected from these - PASS - @lumarel +- Live Image Workstation and Workstation Lite looks good - PASS - @lumarel +- QA:Testcase Boot Methods Boot Iso - PASS - @neil +- QA:Testcase Boot Methods DVD - PASS - @neil +- QA: Testcase boot/install minimal x86_64 over DVD/Bluray (burned with fedora mediawriter) on G752 ASUS laptop - PASS - @atomicturtle +- Container images for x86_64 and aarch64 work as expected in Docker, Podman and WSL - PASS - @lumarel +- QA: Testcase Storage Volume Resize - PASS - @raktajino https://rpa.st/MQSA +- QA: Testcase Update Image - PASS - @raktajino (manually checked against Fedora's testcase (https://fedoraproject.org/wiki/QA:Testcase_Anaconda_updates.img_via_URL), needles also still match in openQA) + + +## openQA summary +- errors in all tests in openQA - INVESTIGATE +- RC1_1 run 1: + - @lumarel: https://rpa.st/CCPQ + - @raktajino: https://rpa.st/5RVA +- RC1_1 run 2 + - @lumarel: https://rpa.st/FWTQ + - @raktajino: +- RC1_2 run 1: + - @lumarel: https://rpa.st/EOGQ +- @raktajino:https://rpa.st/VHLQ + - RC1_2 run 2: +- @raktajino: https://rpa.st/DKCQ +- Upgrade F35 -> F36 needs postgresql-setup --upgrade to convert openqa databse to new format - @alangm +- Per discussion in Testing Team meeting we have 4-8 (ish) issues to fix in openQA (needles and code) to be able to complete all tests. @lumarel has created issues our openQA repo (https://github.com/rocky-linux/os-autoinst-distri-rocky) and we'll pick up and resolve ASAP. + + +##createhdds kickstart file test summary + +Test method: Used packer to build VM. Booted VM. Verified root login. Shutdown VM. + +- UEFI Testing: + - desktop.ks - PASS - Note: resulting image asks for EULA acceptance when booted due to `firstboot --enable` (unsure if that is desired behavior) + - desktopencrypt.ks - PASS - Note: resulting image asks for EULA acceptance when booted due to `firstboot --enable` (unsure if that is desired behavior) + - minimal-uefi.ks - PASS + - server.ks - PASS + - support.ks - PASS +- BIOS Testing: + - desktop.ks - PASS - Note: resulting image asks for EULA acceptance when booted due to `firstboot --enable` (unsure if that is desired behavior) + - desktopencrypt.ks - PASS - Note: resulting image asks for EULA acceptance when booted due to `firstboot --enable` (unsure if that is desired behavior) + - minimal.ks - PASS + - server.ks - PASS + - support.ks - PASS \ No newline at end of file