From 27dbac6eccf69d828dacba1a0acd7f181b6fa896 Mon Sep 17 00:00:00 2001 From: Stack Korora Date: Thu, 18 Aug 2022 18:01:39 -0500 Subject: [PATCH] Migration from Etherpad to wiki. --- .../r9/9.0_qa_testing_go_no_go.md | 14 ++-- release_criteria/r9/9.0_qa_testing_summary.md | 81 +++++++++++++++---- 2 files changed, 71 insertions(+), 24 deletions(-) diff --git a/release_criteria/r9/9.0_qa_testing_go_no_go.md b/release_criteria/r9/9.0_qa_testing_go_no_go.md index 726ea65..9a851c6 100644 --- a/release_criteria/r9/9.0_qa_testing_go_no_go.md +++ b/release_criteria/r9/9.0_qa_testing_go_no_go.md @@ -19,8 +19,8 @@ As a reminder, the objective of a release (major or minor) is to provide a solid | Category | Proportion | Remaining Items | | ----------------- | ----------------- | --------------- | -| TO_CONFIRM | 0 / 29 (0%) | | -| PASS | 0 / 29 (0%) | | +| TO_CONFIRM | 1 / 29 (3%) | Firmware RAID | +| PASS | 28/ 29 (97%) | | | FAIL_NON_BLOCKING | 0 / 29 (0%) | | | FAIL_BLOCKING | 0 / 29 (0%) | | @@ -52,15 +52,15 @@ Examples... ## Initialization Requirements -- Release-blocking images must boot - PASS - (@neil, @atomicturtle) -- Optical Media Requirements - PASS - (@atomicturtle) +- Release-blocking images must boot - PASS - (@neil, @atomicturtle, @lumarel) +- Optical Media Requirements - PASS - (@neil) - Basic Graphics Mode behaviors - PASS - (@tcooper, virt only, manual) - No Broken Packages - PASS - (@tcooper, scripted, manual) -- Repositories Must Match Upstream - TO_CONFIRM - (@tcooper, manual) +- Repositories Must Match Upstream - PASS - (@tcooper, manual) - Debranding - PASS - (@tcooper, scripted, manual) - Media Consistency Verification - PASS - (@tcooper, scripted, manual) - Packages and Installer Sources - PASS - (@lumarel, semi-automatic, openQA test) -- NAS (Network Attached Storage) - TO_CONFIRM - (@lumarel?, semi-automatic, openQA dual-host test) +- NAS (Network Attached Storage) - PASS - (@stack, manual) - Installation Interfaces - PASS - (@lumarel, @raktajino, @tcooper, semi-automatic, openQA, @atomicturtle, manual?, SCAP) - Minimal Installation - PASS - (@lumarel, @raktajino, @tcooper, semi-automatic , openQA) - Kickstart Installation - PASS - (@label, @tcooper, manual, createhdds) @@ -74,7 +74,7 @@ Examples... ## Cloud Image Requirements -- Images Published to Cloud Providers - FAIL_NON_BLOCKING - (@neil) +- Images Published to Cloud Providers - PASS - (@neil) ## Post-Installation Requirements diff --git a/release_criteria/r9/9.0_qa_testing_summary.md b/release_criteria/r9/9.0_qa_testing_summary.md index e6deff0..0e22f25 100644 --- a/release_criteria/r9/9.0_qa_testing_summary.md +++ b/release_criteria/r9/9.0_qa_testing_summary.md @@ -28,20 +28,7 @@ This document will record a summary of all QA and Testing results for {{ rc.prod This is a list of items that are being INVESTIGATEd further before being assigned a PASS, FAIL, NOTABUG or UPSTREAM status. PLEASE add your MM handle if you are working on this item to minimize duplication of work. More than one handle is allowed but please communicate. -- QA:Testcase Basic Graphics Mode - [INVESTIGATE] - @tcooper -- QA:Testcase Boot Methods Boot Iso - [INVESTIGATE] - @neil -- QA:Testcase Boot Methods DVD - [INVESTIGATE] - @neil -- QA:Testcase Debranding - [INVESTIGATE] - @tcooper -- QA:Testcase Media Consistency Verification - [INVESTIGATE] - @tcooper -- QA:Testcase Media File Conflicts - [INVESTIGATE] - @tcooper -- QA:Testcase Media File Conflicts - [INVESTIGATE] - @tcooper -- QA:Testcase Media File Conflicts - [INVESTIGATE] - @tcooper -- QA:Testcase Media File Conflicts - [INVESTIGATE] - @tcooper -- QA:Testcase Media Repoclosure - [INVESTIGATE] - @tcooper -- QA:Testcase Storage Volume Resize - [INVESTIGATE] - @raktajino -- QA:Testcase Update Image - [INVESTIGATE] - @raktajino -- QA:Testcase boot/install minimal x86_64 over DVD/Bluray - [INVESTIGATE] - @atomicturtle -- QA:Testcase_Mediacheck - [INVESTIGATE] - @tcooper +- QA:Testcase Firmware RAID - [INVESTIGATE] - @neil ## UPSTREAM This is a list of items that have been verified to be replicated UPSTREAM in RHEL {{ rc.9 }} and/or are described clearly in the RHEL 9 Release Notes. @@ -52,16 +39,76 @@ This is a list of items that have been verified to be replicated UPSTREAM in RHE This is a list of items that have been verified to FAIL the QA:Testcase. In addition to recording who did the test please indicate if the item is BLOCKING release or not. - QA:Testcase_Some_Testcase - [FAIL] - @your_mm_handle - +- RC1 - The blue side banner background at the left side of the Anaconda installer display is not displaying correctly in virt-viewer. [minor BUG] - @alangm ## NOTABUG This is a list of items that have been verified as less than optimal but are expected and NOTABUG. -- QA:Testcase_Some_Testcase - [NOTABUG] - @your_mm_handle - +- QA:Testcase Basic Graphics Mode - [DEPRICATED] - @tcooper ## PASS This is a list of item that have been verified as PASSing the QA:Testcase named (or proposed). -- QA:Testcase_Some_Testcase - [PASS] - @your_mm_handle - +- QA:Testcase_Debranding - [PASS] - @tcooper - preliminary pass based on Peridot yumrepofs - https://rpa.st/PZKA +- RC1 - QA:Testcase System Services - [PASS] - @lumarel - tested with boot-iso and every base environment +- RC1 - Basic Installs - [PASS] - @lumarel - Installs of all base environments on x86_64 without additional optional package groups +- RC1 - QA:Testcase Packages and Installer Sources - [PASS] - @raktajino - tested against RC1-boot-iso in openQA, all package sets pass. +- RC1 - QA:Testcase GNOME UI Functionality - [PASS] - @lumarel - Nothing to mention +- RC1 - QA:Testcase SELinux Errors on Server - [PASS] - @lumarel - no errors +- RC1 - QA:Testcase SELinux Errors on Desktop- [PASS] - @lumarel - no errors +- RC1 - QA:Testcase Artwork and Assets - [PASS] - @lumarel - everything looks correct +- RC1 - QA:Testcase Minimal Installation - [PASS] - @raktajino - nothing to report +- RC1 - QA: Testcase Boot Methods Boot Iso - [PASS] - @alangm - minimal system, graphical server & basic workstation - nothing to report except as above ^^^ +- RC1 - Minimal install on QEMU/KVM passing - [PASS] - @boris - reboot takes slightly longer on the first reboot +- RC1 - Repo Health Check - [PASS] - @lumarel - sqlite database files for repo metadata is missing up to now, which will hinder some applications from syncing the sources - are implemented +- RC1 - Repo Health Check - [PASS] - @lumarel - metadata signing not implemented up to now - looks to be implemented as well now +- RC1 - Repo Health Check - [PASS] - @lumarel - Repos look good now excl. BaseOS of aarch64, ppc64le, s390x, where the boot image is missing, which is referenced in the .treeinfo +- RC1 - Redesign looks to be broken (Post-Install error, effects some of the install groups) - [PASS] - @lumarel +- RC1 - Workstation install with known pesign bug - [PASS] - @alangm - error window "Exit Installer" button froze Anaconda instead of exit so had to initiate shutdown from cockpit - @lumarel - fixed! +- RC1 - Building packages with mock - [PASS] - @hbjy - building works with the mock config from sokels' repo (will be upstreamed for release) +- RC1 - Install apache + mariadb-server - [PASS] - @sspencerwire - mariadb-secure-installation works, nextcloud install on top as well +- RC1 - Install works on macOS M1 - [PASS] - @netwaze, @gardo984, @jordan_c +- RC1 - Install on baremetal went well - [PASS] - @alangm - on a Dell T630 +- RC1 - Install went well - [PASS] - @kars88 - there was some flickering during the install +- RC1.1 dvd - STIG security - [PASS] - @atomicturtle - packages were not complete in RC1 dvd, but are now in RC1.1 +- RC1.1 - Basic installs work - [PASS] - @mkahric - Virtualbox on Windows (UEFI w/o SB), Workstation Player on Windows (UEFI w/ SB), Hyper-V (UEFI w/ SB) +- RC1.1 - docker installation with ansible works - [PASS] - @gardo984 +- RC1 - Repo Health Check - [PASS] - @lumarel - all repos (x86_64, aarch64, ppc64le, s390x) look good now, all metadata is healthy and all package checksums match +- RC1.1 - QA:Testcase Multimonitor Setup - [PASS] - @lumarel - looks good +- RC1.1 - QA:Testcase Artwork and Assets - [PASS] - @lumarel - no issues found +- RC1.1 - QA:Testcase GNOME UI Functionality - [PASS] - @lumarel - everything behaves as it should +- RC1.1 - QA:Testcase SELinux Errors on Desktop - [PASS] - @lumarel - After an hour of usage no errors +- RC1.1 - QA:Testcase Keyboard Layout - [PASS] - @lumarel - OpenQA tests and manual tests show normal behavior +- RC1.1 - QA:Testcase SELinux Errors on Server - [PASS] - @lumarel - after more than an hour of runtime with several installs and application setups (missing, Katello bootstrap, which once brought some SELinux errors) +- RC1.1 - QA:Testcase System Services - [PASS] - @lumarel - no errors +- RC1.1 - QA: Testcase Installation Interfaces - [PASS] - @raktajino - failures on the Welcome Tour screen because we haven't handled it yet. Everything else seems good. +- RC1.1 - QA:Testcase Installer Help - [PASS] - @raktajino - no errors +- RC1.1 - QA: Testcase Disk Layouts - [PASS] - @raktajino - Assorted post-install failures due to either mismatch between console login and GUI login or unhandled Welcome Tour. +- RC1.1 - QA: Testcase Kickstart Installation - [PASS] - @raktajino - Executed manually with virt-install. Nothing to report. +- RC1.1 - QA: Testcase Update Image - [PASS] - @raktajino - failure on _console_wait_login as the GUI login screen is displayed. Update image is applied as expected. +- RC1.1 - QA:Testcase VNC Graphics Mode - [PASS] - @tcooper +- RC1.1 - QA:Testcase Basic Package installs - [PASS] - @lumarel - All package sets and services tested which are currently on the blocking list +- RC1.1 - QA:Testcase Application Functionality - [PASS] - @lumarel - everything looks good +- RC1/RC1.1 - QA:Testcase_Mediacheck - [PASS] - @tcooper +- RC1/RC1.1 - QA:Testcase Media Consistency Verification - [PASS] - @tcooper +- RC1.1 - QA:Testcase Media Repoclosure - [PASS] - @tcooper +- RC1.1 - QA:Testcase Media File Conflicts - [PASS] - @tcooper +- RC1.1 - QA: Testcase Bootloader Disk Selection - [PASS] - @raktajino +- RC1.1 - QA: Testcase Installer Translations - [PASS] - @raktajino +- RC1.1 - QA: Testcase Network Attached Storage - [PASS] - @Stack +- QA: Testcase Boot Methods Boot ISO - [PASS] - all of us <3 +- QA: Testcase Boot Methods DVD - [PASS] - todos nosotros <3 +- QA:Testcase Cloud Images Published - [PASS] - @neil +- QA:Testcase Storage Volume Resize - [PASS] - @raktajino, @tcooper +- QA:Testcase Repocompare - [PASS] - @tcooper +- QA:Testcase Basic install boot final ISO - [PASS] - @lumarel +- QA:Testcase Basic install minimal final ISO - [PASS] - @lumarel +- QA:Testcase Basic installs dvd final ISO - [PASS] - @lumarel - install of every base environment + custom operating system with all options enabled +- OpenQA tests with final ISOs - [PASS] - @lumarel - boot-iso 2/2 | minimal-iso 2/2 | minimal 22/26 (failing are all false-positive) | server 21/26 (failing are all false-positive) | package-set 5/5 | workstation 17/26 (failing are all false-positive) | graphical-server 17/26 (failing are all false-positive) | universal 34/43 (failing are all false-positive) +- QA:Testcase Minimal Installation - [PASS] - @neil ## OTHER NOTABLE ITEMS - +- QA:Testcase Module Streams - @lumarel - 9.0 currently does not have any Stream metadata and only the base Streams +- Repo Health Check - @lumarel - yumrepofs is not sending content lengths, which make syncing with Katello impossible +- Missing package in package group additional-devel - @stack - dnf -y group install additional-devel: No match for group package "hmaccalc" - this is an old EL bug +- Hosting a nfs server in macOS, and using that in Virtualbox for OpenQA shares is a pain - @tcooper