testing/wiki
10
0
Fork 3
generated from sig_core/wiki-template
Code Issues 1 Pull Requests 1 Actions Packages Projects Releases Wiki Activity
84e3e46ea2
wiki/release_criteria/r8/8.6_qa_testing_summary.md

8.8 KiB
Raw Blame History

title author revision_date rc
Rocky Linux 8.6 QA and Testing Summary Trevor Cooper 2022-05-18
prod ver level
Rocky Linux 8.6 Final

Rocky Linux 8.6 QA and Testing Summary

Last updated: Fri May 13 17:36:41 UTC 2022

Scope

This document will record a summary of all QA and Testing results for Rocky Linux 8.6 release. It is only a record of success and/or failure. Solution discussion should take place elsewhere.

SOP

  • Please include PASS, FAIL, NOTABUG, INVESTIGATE or UPSTREAM as appropriate in all entries.
  • Please only provide brief summary. Details should go to Rocky Pastebin, links here is OK.
  • Especially for an negative result please leave your MM @handle so we can talk to you to get resolution.
  • If the item you are reported is related to a QA:Testcase please mention it. If it should be a QA:Testcase, even if it's not a current requirement, suggest a title and create an issue in the wiki repository so we can add it.

INVESTIGATE

  • errors in all tests in openQA - INVESTIGATE - see openQA section below.
  • INVESTIGATE whether kdump issue affects qcows with encrypted partitions in createhdds. These are pre-reqs for openQA multi-machine tests. NOTE: This is addressed below in section - @tcooper
  • KDE and XFCE Life images are broken - INVESTIGATE - @label

UPSTREAM

  • Anaconda error when specific steps get executed in the right order (configure network -> disable kdump -> select some os install group -> configure default storage -> configure the storage a second time, but this time with encryption enabled, confirmed via several openqa test suites and manual installation on ESXi) - retested in RC1_2 - UPSTREAM - @lumarel
  • Issue repeated on RHEL8.6 - @atomicturtle
  • Issue reported to RH https://bugzilla.redhat.com/show_bug.cgi?id=2085321 - @stack

FAIL

  • ESXi secureboot (x86_64) still FAILing, but expected - @lumarel

NOTABUG

  • Minimal: Selecting a SCAP profile with dependencies not available (aide, etc), selecting Ignore dependency during installation will crash anaconda at the final oscap check. NOTABUG, this is for documentation - @atomicturtle
  • Minimal ISO: is missing the source for rsyslog again, and somehow also doesn't pull it in when installing, which means it is missing it after the install (doesn't happen for boot ISO and dvd1 ISO) - NOTABUG (per @label) - @lumarel
  • Minimal ISO: if the server base environment is installed with the minimal iso and cockpit is enabled after the installation, the SELinux submenu shows an error "semanage: command not found" (doesn't happen with boot/dvd-iso) - also marked as expected - NOTABUG (per @label) - @lumarel

Manual success reported in MM

  • minimal install from minimal ISO fine - PASS - @Luna Jernberg
  • workstation (x86_64) install with applications fine - retested in RC1_2 - PASS - @lumarel
  • all repos are available with the exact naming as they are in the rocky-repos package (nfv needs fix for that) - retested in RC1_2 - PASS - @lumarel
  • packer build for 8.6 worked flawlessly - retested by @neil in RC1_2 - PASS - @gmazrael
  • security profiles look good in anaconda UI - PASS - @atomicturtle (need openQA testing)
  • minimal and dvd recognized as Rocky Linux 8 in KVM - PASS - @atomicturtle
  • CIS profiles confirmed good in lvl1/2 in anaconda - PASS - @atomicturtle
  • DISA profiles confirmed good in anaconda - PASS - @atomicturtle
  • DVD: libvirt correctly boots when Rocky Linux 8 profile is selected - PASS - @atomicturtle
  • SELinux checks on Server (x86_64) (letting it run for an hour and run sealert -a /var/log/audit/audit.log) - everything okay - retested in RC1_2 - PASS - @lumarel
  • SELinux checks on Desktop (x86_64) (start the GNOME SETroubleshooter after some minutes of running) - everything okay - retested in RC1_2 - PASS - @lumarel
  • DVD: Anaconda manual network configuration, and PCI-DSS SCAP profile selected confirmed good - PASS - @atomicturtle
  • QA:Testcase_Mediacheck - PASS for all x86_64 ISOs - @tcooper
  • QA:Testcase_Mediacheck - PASS for all aarch64 ISOs - @tcooper
  • QA:Testcase Media Repoclosure - PASS for minimal & dvd1 for x86_64 & aarch64 (confirms RelEng results) - @tcooper
  • QA:Testcase Media File Conflicts - PASS for minimal for x86_64 & aarch64 (0 file conflicts found and 0 package conflicts found) - @tcooper
  • QA:Testcase Basic Graphics Mode - PASS - verified manually for Rocky-8.6-x86_64-dvd1.iso in VirtualBox on macOS X - @tcooper
  • DVD: Anaconda install with 3rd party repo, encrypted filesystem, HIPAA SCAP profile selected, confirmed good - PASS - @atomicturtle
  • Upgrade tests on several test machines from 8.5 to 8.6, no issues no SELinux alerts - PASS - @lumarel
  • All module streams except perl:5.32 and log4j:2 correctly have the dependencies set and packages look to be built correctly - PASS - @lumarel
  • log4j module stream was broken, (should be able to hook against java-8 and 11) - got fixed now in RC1_2 - PASS - @lumarel
  • Anything perl 5.32 (module stream) was broken - got fixed in RC1_2 - PASS - @nazunalika
  • Greenbone appliance installation test (https://rpa.st/DQNA) - PASS - @atomicturtle
  • QA:Testcase Debranding for RC2 content from koji (srpms, kernel-rt and pcs are not all on the dvd1) - 46/47 PASS , 1 FALSE PASS - https://rpa.st/raw/QK3A - @tcooper
  • QA:Testcase Media Consistency Verification (not written yet) for all RC2 ISOs x86_64, aarch64 - PASS - @tcooper
  • QA:Testcase Media File Conflicts - EXPECTED(per @label) for Rocky-8.6-x86_64-dvd1.iso (4 file conflicts found and 13 package conflicts found, these appear to be same as 8.5 conflict between mariadb and mysql packages/files, full results - https://rpa.st/raw/ZWPQ) - @tcooper
  • QA:Testcase Media File Conflicts - EXPECTED(per @label) for Rocky-8.6-aarch64-dvd1.iso (modular dependency problems, 3 file conflicts found 4 package conflicts found, full results - https://rpa.st/raw/KOFQ) - @tcooper
  • QA:Testcase Media File Conflicts for both x86_64 (https://rpa.st/raw/NLGA) and aarch64 (https://rpa.st/raw/4SFQ) are essentially unchanged and remain - EXPECTED(per @label) with RC1_2 ISOs. - @tcooper
  • OpenQA tests @lumarel - there are errors from the test cases, but everything image and repo related looks good - PASS - @lumarel
  • the dvd1 iso of aarch64 doesn't show an workstation base environment - it doesn't have an workstation environment - PASS - @lumarel
  • Installs of aarch64 systems of all 3 isos look good and installs with all base environments work as expected from these - PASS - @lumarel
  • Live Image Workstation and Workstation Lite looks good - PASS - @lumarel
  • QA:Testcase Boot Methods Boot Iso - PASS - @neil
  • QA:Testcase Boot Methods DVD - PASS - @neil
  • QA: Testcase boot/install minimal x86_64 over DVD/Bluray (burned with fedora mediawriter) on G752 ASUS laptop - PASS - @atomicturtle
  • Container images for x86_64 and aarch64 work as expected in Docker, Podman and WSL - PASS - @lumarel
  • QA: Testcase Storage Volume Resize - PASS - @raktajino https://rpa.st/MQSA
  • QA: Testcase Update Image - PASS - @raktajino (manually checked against Fedora's testcase (https://fedoraproject.org/wiki/QA:Testcase_Anaconda_updates.img_via_URL), needles also still match in openQA)

openQA summary

##createhdds kickstart file test summary

Test method: Used packer to build VM. Booted VM. Verified root login. Shutdown VM.

  • UEFI Testing:
    • desktop.ks - PASS - Note: resulting image asks for EULA acceptance when booted due to firstboot --enable (unsure if that is desired behavior)
    • desktopencrypt.ks - PASS - Note: resulting image asks for EULA acceptance when booted due to firstboot --enable (unsure if that is desired behavior)
    • minimal-uefi.ks - PASS
    • server.ks - PASS
    • support.ks - PASS
  • BIOS Testing:
    • desktop.ks - PASS - Note: resulting image asks for EULA acceptance when booted due to firstboot --enable (unsure if that is desired behavior)
    • desktopencrypt.ks - PASS - Note: resulting image asks for EULA acceptance when booted due to firstboot --enable (unsure if that is desired behavior)
    • minimal.ks - PASS
    • server.ks - PASS
    • support.ks - PASS