testing/wiki
10
0
Fork 3
generated from sig_core/wiki-template
Code Issues 1 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

Migration from Etherpad to wiki.

Browse Source
This commit is contained in:
Stack Korora 2022-08-18 18:01:39 -05:00
parent f2114830c6
commit 27dbac6ecc
2 changed files with 71 additions and 24 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

14
release_criteria/r9/9.0_qa_testing_go_no_go.md
View File

@ -19,8 +19,8 @@ As a reminder, the objective of a release (major or minor) is to provide a solid
| Category | Proportion | Remaining Items | | Category | Proportion | Remaining Items |
| ----------------- | ----------------- | --------------- | | ----------------- | ----------------- | --------------- |
| TO_CONFIRM | 0 / 29 (0%) | | | TO_CONFIRM | 1 / 29 (3%) | Firmware RAID |
| PASS | 0 / 29 (0%) | | | PASS | 28/ 29 (97%) | |
| FAIL_NON_BLOCKING | 0 / 29 (0%) | | | FAIL_NON_BLOCKING | 0 / 29 (0%) | |
| FAIL_BLOCKING | 0 / 29 (0%) | | | FAIL_BLOCKING | 0 / 29 (0%) | |
@ -52,15 +52,15 @@ Examples...
## Initialization Requirements ## Initialization Requirements
- Release-blocking images must boot - PASS - (@neil, @atomicturtle) - Release-blocking images must boot - PASS - (@neil, @atomicturtle, @lumarel)
- Optical Media Requirements - PASS - (@atomicturtle) - Optical Media Requirements - PASS - (@neil)
- Basic Graphics Mode behaviors - PASS - (@tcooper, virt only, manual) - Basic Graphics Mode behaviors - PASS - (@tcooper, virt only, manual)
- No Broken Packages - PASS - (@tcooper, scripted, manual) - No Broken Packages - PASS - (@tcooper, scripted, manual)
- Repositories Must Match Upstream - TO_CONFIRM - (@tcooper, manual) - Repositories Must Match Upstream - PASS - (@tcooper, manual)
- Debranding - PASS - (@tcooper, scripted, manual) - Debranding - PASS - (@tcooper, scripted, manual)
- Media Consistency Verification - PASS - (@tcooper, scripted, manual) - Media Consistency Verification - PASS - (@tcooper, scripted, manual)
- Packages and Installer Sources - PASS - (@lumarel, semi-automatic, openQA test) - Packages and Installer Sources - PASS - (@lumarel, semi-automatic, openQA test)
- NAS (Network Attached Storage) - TO_CONFIRM - (@lumarel?, semi-automatic, openQA dual-host test) - NAS (Network Attached Storage) - PASS - (@stack, manual)
- Installation Interfaces - PASS - (@lumarel, @raktajino, @tcooper, semi-automatic, openQA, @atomicturtle, manual?, SCAP) - Installation Interfaces - PASS - (@lumarel, @raktajino, @tcooper, semi-automatic, openQA, @atomicturtle, manual?, SCAP)
- Minimal Installation - PASS - (@lumarel, @raktajino, @tcooper, semi-automatic , openQA) - Minimal Installation - PASS - (@lumarel, @raktajino, @tcooper, semi-automatic , openQA)
- Kickstart Installation - PASS - (@label, @tcooper, manual, createhdds) - Kickstart Installation - PASS - (@label, @tcooper, manual, createhdds)
@ -74,7 +74,7 @@ Examples...
## Cloud Image Requirements ## Cloud Image Requirements
- Images Published to Cloud Providers - FAIL_NON_BLOCKING - (@neil) - Images Published to Cloud Providers - PASS - (@neil)
## Post-Installation Requirements ## Post-Installation Requirements

81
release_criteria/r9/9.0_qa_testing_summary.md
View File

@ -28,20 +28,7 @@ This document will record a summary of all QA and Testing results for {{ rc.prod
This is a list of items that are being INVESTIGATEd further before being assigned a PASS, FAIL, NOTABUG or UPSTREAM status. This is a list of items that are being INVESTIGATEd further before being assigned a PASS, FAIL, NOTABUG or UPSTREAM status.
PLEASE add your MM handle if you are working on this item to minimize duplication of work. More than one handle is allowed but please communicate. PLEASE add your MM handle if you are working on this item to minimize duplication of work. More than one handle is allowed but please communicate.
- QA:Testcase Basic Graphics Mode - [INVESTIGATE] - @tcooper - QA:Testcase Firmware RAID - [INVESTIGATE] - @neil
- QA:Testcase Boot Methods Boot Iso - [INVESTIGATE] - @neil
- QA:Testcase Boot Methods DVD - [INVESTIGATE] - @neil
- QA:Testcase Debranding - [INVESTIGATE] - @tcooper
- QA:Testcase Media Consistency Verification - [INVESTIGATE] - @tcooper
- QA:Testcase Media File Conflicts - [INVESTIGATE] - @tcooper
- QA:Testcase Media File Conflicts - [INVESTIGATE] - @tcooper
- QA:Testcase Media File Conflicts - [INVESTIGATE] - @tcooper
- QA:Testcase Media File Conflicts - [INVESTIGATE] - @tcooper
- QA:Testcase Media Repoclosure - [INVESTIGATE] - @tcooper
- QA:Testcase Storage Volume Resize - [INVESTIGATE] - @raktajino
- QA:Testcase Update Image - [INVESTIGATE] - @raktajino
- QA:Testcase boot/install minimal x86_64 over DVD/Bluray - [INVESTIGATE] - @atomicturtle
- QA:Testcase_Mediacheck - [INVESTIGATE] - @tcooper
## UPSTREAM ## UPSTREAM
This is a list of items that have been verified to be replicated UPSTREAM in RHEL {{ rc.9 }} and/or are described clearly in the RHEL 9 Release Notes. This is a list of items that have been verified to be replicated UPSTREAM in RHEL {{ rc.9 }} and/or are described clearly in the RHEL 9 Release Notes.
@ -52,16 +39,76 @@ This is a list of items that have been verified to be replicated UPSTREAM in RHE
This is a list of items that have been verified to FAIL the QA:Testcase. In addition to recording who did the test please indicate if the item is BLOCKING release or not. This is a list of items that have been verified to FAIL the QA:Testcase. In addition to recording who did the test please indicate if the item is BLOCKING release or not.
- QA:Testcase_Some_Testcase - [FAIL] - @your_mm_handle - <brief_description> - QA:Testcase_Some_Testcase - [FAIL] - @your_mm_handle - <brief_description>
- RC1 - The blue side banner background at the left side of the Anaconda installer display is not displaying correctly in virt-viewer. [minor BUG] - @alangm
## NOTABUG ## NOTABUG
This is a list of items that have been verified as less than optimal but are expected and NOTABUG. This is a list of items that have been verified as less than optimal but are expected and NOTABUG.
- QA:Testcase_Some_Testcase - [NOTABUG] - @your_mm_handle - <brief_description> - QA:Testcase Basic Graphics Mode - [DEPRICATED] - @tcooper
## PASS ## PASS
This is a list of item that have been verified as PASSing the QA:Testcase named (or proposed). This is a list of item that have been verified as PASSing the QA:Testcase named (or proposed).
- QA:Testcase_Some_Testcase - [PASS] - @your_mm_handle - <brief_description> - QA:Testcase_Debranding - [PASS] - @tcooper - preliminary pass based on Peridot yumrepofs - https://rpa.st/PZKA
- RC1 - QA:Testcase System Services - [PASS] - @lumarel - tested with boot-iso and every base environment
- RC1 - Basic Installs - [PASS] - @lumarel - Installs of all base environments on x86_64 without additional optional package groups
- RC1 - QA:Testcase Packages and Installer Sources - [PASS] - @raktajino - tested against RC1-boot-iso in openQA, all package sets pass.
- RC1 - QA:Testcase GNOME UI Functionality - [PASS] - @lumarel - Nothing to mention
- RC1 - QA:Testcase SELinux Errors on Server - [PASS] - @lumarel - no errors
- RC1 - QA:Testcase SELinux Errors on Desktop- [PASS] - @lumarel - no errors
- RC1 - QA:Testcase Artwork and Assets - [PASS] - @lumarel - everything looks correct
- RC1 - QA:Testcase Minimal Installation - [PASS] - @raktajino - nothing to report
- RC1 - QA: Testcase Boot Methods Boot Iso - [PASS] - @alangm - minimal system, graphical server & basic workstation - nothing to report except as above ^^^
- RC1 - Minimal install on QEMU/KVM passing - [PASS] - @boris - reboot takes slightly longer on the first reboot
- RC1 - Repo Health Check - [PASS] - @lumarel - sqlite database files for repo metadata is missing up to now, which will hinder some applications from syncing the sources - are implemented
- RC1 - Repo Health Check - [PASS] - @lumarel - metadata signing not implemented up to now - looks to be implemented as well now
- RC1 - Repo Health Check - [PASS] - @lumarel - Repos look good now excl. BaseOS of aarch64, ppc64le, s390x, where the boot image is missing, which is referenced in the .treeinfo
- RC1 - Redesign looks to be broken (Post-Install error, effects some of the install groups) - [PASS] - @lumarel
- RC1 - Workstation install with known pesign bug - [PASS] - @alangm - error window "Exit Installer" button froze Anaconda instead of exit so had to initiate shutdown from cockpit - @lumarel - fixed!
- RC1 - Building packages with mock - [PASS] - @hbjy - building works with the mock config from sokels' repo (will be upstreamed for release)
- RC1 - Install apache + mariadb-server - [PASS] - @sspencerwire - mariadb-secure-installation works, nextcloud install on top as well
- RC1 - Install works on macOS M1 - [PASS] - @netwaze, @gardo984, @jordan_c
- RC1 - Install on baremetal went well - [PASS] - @alangm - on a Dell T630
- RC1 - Install went well - [PASS] - @kars88 - there was some flickering during the install
- RC1.1 dvd - STIG security - [PASS] - @atomicturtle - packages were not complete in RC1 dvd, but are now in RC1.1
- RC1.1 - Basic installs work - [PASS] - @mkahric - Virtualbox on Windows (UEFI w/o SB), Workstation Player on Windows (UEFI w/ SB), Hyper-V (UEFI w/ SB)
- RC1.1 - docker installation with ansible works - [PASS] - @gardo984
- RC1 - Repo Health Check - [PASS] - @lumarel - all repos (x86_64, aarch64, ppc64le, s390x) look good now, all metadata is healthy and all package checksums match
- RC1.1 - QA:Testcase Multimonitor Setup - [PASS] - @lumarel - looks good
- RC1.1 - QA:Testcase Artwork and Assets - [PASS] - @lumarel - no issues found
- RC1.1 - QA:Testcase GNOME UI Functionality - [PASS] - @lumarel - everything behaves as it should
- RC1.1 - QA:Testcase SELinux Errors on Desktop - [PASS] - @lumarel - After an hour of usage no errors
- RC1.1 - QA:Testcase Keyboard Layout - [PASS] - @lumarel - OpenQA tests and manual tests show normal behavior
- RC1.1 - QA:Testcase SELinux Errors on Server - [PASS] - @lumarel - after more than an hour of runtime with several installs and application setups (missing, Katello bootstrap, which once brought some SELinux errors)
- RC1.1 - QA:Testcase System Services - [PASS] - @lumarel - no errors
- RC1.1 - QA: Testcase Installation Interfaces - [PASS] - @raktajino - failures on the Welcome Tour screen because we haven't handled it yet. Everything else seems good.
- RC1.1 - QA:Testcase Installer Help - [PASS] - @raktajino - no errors
- RC1.1 - QA: Testcase Disk Layouts - [PASS] - @raktajino - Assorted post-install failures due to either mismatch between console login and GUI login or unhandled Welcome Tour.
- RC1.1 - QA: Testcase Kickstart Installation - [PASS] - @raktajino - Executed manually with virt-install. Nothing to report.
- RC1.1 - QA: Testcase Update Image - [PASS] - @raktajino - failure on _console_wait_login as the GUI login screen is displayed. Update image is applied as expected.
- RC1.1 - QA:Testcase VNC Graphics Mode - [PASS] - @tcooper
- RC1.1 - QA:Testcase Basic Package installs - [PASS] - @lumarel - All package sets and services tested which are currently on the blocking list
- RC1.1 - QA:Testcase Application Functionality - [PASS] - @lumarel - everything looks good
- RC1/RC1.1 - QA:Testcase_Mediacheck - [PASS] - @tcooper
- RC1/RC1.1 - QA:Testcase Media Consistency Verification - [PASS] - @tcooper
- RC1.1 - QA:Testcase Media Repoclosure - [PASS] - @tcooper
- RC1.1 - QA:Testcase Media File Conflicts - [PASS] - @tcooper
- RC1.1 - QA: Testcase Bootloader Disk Selection - [PASS] - @raktajino
- RC1.1 - QA: Testcase Installer Translations - [PASS] - @raktajino
- RC1.1 - QA: Testcase Network Attached Storage - [PASS] - @Stack
- QA: Testcase Boot Methods Boot ISO - [PASS] - all of us <3
- QA: Testcase Boot Methods DVD - [PASS] - todos nosotros <3
- QA:Testcase Cloud Images Published - [PASS] - @neil
- QA:Testcase Storage Volume Resize - [PASS] - @raktajino, @tcooper
- QA:Testcase Repocompare - [PASS] - @tcooper
- QA:Testcase Basic install boot final ISO - [PASS] - @lumarel
- QA:Testcase Basic install minimal final ISO - [PASS] - @lumarel
- QA:Testcase Basic installs dvd final ISO - [PASS] - @lumarel - install of every base environment + custom operating system with all options enabled
- OpenQA tests with final ISOs - [PASS] - @lumarel - boot-iso 2/2 | minimal-iso 2/2 | minimal 22/26 (failing are all false-positive) | server 21/26 (failing are all false-positive) | package-set 5/5 | workstation 17/26 (failing are all false-positive) | graphical-server 17/26 (failing are all false-positive) | universal 34/43 (failing are all false-positive)
- QA:Testcase Minimal Installation - [PASS] - @neil
## OTHER NOTABLE ITEMS ## OTHER NOTABLE ITEMS
- QA:Testcase Module Streams - @lumarel - 9.0 currently does not have any Stream metadata and only the base Streams
- Repo Health Check - @lumarel - yumrepofs is not sending content lengths, which make syncing with Katello impossible
- Missing package in package group additional-devel - @stack - dnf -y group install additional-devel: No match for group package "hmaccalc" - this is an old EL bug
- Hosting a nfs server in macOS, and using that in Virtualbox for OpenQA shares is a pain - @tcooper