toolkit/func/README.md

Release Engineering Core Functionality Testing
==============================================

These are a set of scripts that are designed to test the core functionality
of a Rocky Linux system. They are designed to work on current versions of
Rocky and are used to test a system as a Release Engineering self-QA but
can be used by others for their own personal testing (under the assumption
that you just want to see what happens, we don't judge.

These tests *must* pass for a X.0 release to be considered "Core Validated".

* common  -> Functions that our scripts and tests may or may not use. Templates
             and other files should come here too under common/files and
             scripts that use them should reference them as `./common/files/...`
* core    -> Core functionality and testing. For example, packages and service
             functionality.
* lib     -> Library tests (these may be done elsewhere, such as openqa)
* log     -> Log output. This repository has example logs of running on Rocky
             Linux.
* modules -> Tests for module streams and their basic tests
* stacks  -> Software stacks, think like LAMP (may be done elsewhere, such as openqa)

How to Run
----------

There are two ways to run through the tests:

* By running `/bin/bash runtests.sh`
  * Runs all core tests
* By running `/bin/bash stacktests.sh`
  * Runs all stack tests (eg, lamp, ipa)
* By running `/bin/bash monotests.sh`
  * Supposed to runs all tests one by one to help identify failures as they happen (not functional)

Adding Tests
------------

So you want to add a few tests. Great! Before you add them, I want you to ask
yourself the following questions:

* Are my test(s) brand new?
* Are my test(s) actually for the "core" functionality of the system?
* Will my test(s) be going through a shellcheck?
* Were my tests running with SELinux enforcing?

If you've answered no to any of the above, the test may not be valid for this
project. If you are planning on changing a test or fixing a test to look or
work better, then a PR is more than welcome. Some things could definitely
use some touching up or improvements.

When creating tests, the below should be followed (at a minimum):

* Use functions from `./common/imports.sh`
* Global variables should be in `./common/exports.sh`
* Reusable files should be in `./common/files`
* Logging is enforced; use `r_log` where ever necessary
* Exits and status checks should be against `r_checkExitStatus`
* Place comments where `r_log` won't be descriptive enough
* With some exceptions, keep lines to a maximum of 80 characters
* Use fullpath to binaries when necessary
* Use shellcheck to verify the scripts are valid and compliant (some stuff that
  shellcheck reports could be false - Just use a comment to turn off that test
  for that particular line, but you need to ensure it's a false positive.)
* All filenames should start with a number and end with `.sh` (eg `00-foo.sh`)
* The executable bit should be set (except for scripts that are sourced)

**Note**: that if tests should be skipped, they should be placed into the
`skip.list` file so that way they won't run during the test phase. The file will
get a -x placed on it. Note that this is generally OK, since this repo will just
be cloned when being used anyway and won't be committed back. It is just
expected that all scripts are +x to begin with unless there's a valid reason.
There are a few tests we already have disabled because they're either not done
or they are acting strangely.

**Note**: If a package required additional modification (eg, dotnet) and it
it has a `.rocky` on the release tag, then it should be noted in the mods.list.
The same thing goes for the debrand list. Additionally, if certain patches
can change the output, it would be good to test for this (see `core/pkg_httpd`)
for an example.

Core Functionality
------------------

Everyone has their own idea of "core functionality." In the case of Release
Engineering, core functionality is simply us saying that with a basic
installation of Rocky Linux, we can run basic commands that any system admin,
developer, or casual user would run and expect to work on a regular basis.

Think about the software you probably use fairly regularly on any Linux system
that you've installed, ran, or are currently running. Now think about the
commands that you run day in, and day out. Now consider that what you're
running isn't niche and it's highly likely others use them too. If something
goes wrong with the build of your distribution, your tools might not work as
expected. Which is why the idea of doing basic testing of most, if not all of
the common stuff is a good thing to do.

While writing this, the things that come to mind are:

* archiving: zip, tar, gzip, etc
* file: head, tail, less, cat, diff, find, grep, vim, git
* network: ping, ip, ssh, wget, curl
* packaging: rpm, dnf
* system utilities: systemctl, top, sudo, ps
* web (packaging): httpd

Those are just off the top of my head. There's obviously a lot more, but with
that in mind, you now have the idea of what we're trying to accomplish with
this set of tests.

With that being said, there are obviously other tests being employed for things
that people may or may not use (LAMP stacks for example). It's not a core
function by any means, but it at least validates that a common thing or set of
things works as intended without extending the system or fixing the baseline
set of packages.

FAQ
---

### How do I know what some of these scripts do?
You can view the script and look at the various `r_log` lines or the comments
if they happen to be there. If you don't see a comment, look for an `r_log`.

### How do I disable a test?
A test can be disabled by running `chmod -x` on any given test. It's also
recommended to add it to `skip.list`

### Won't some of the tests have to change on (insert major release here)?
Yes and no. There are some tests will have to be altered to deal with it, but
the only way to really find out is to run the tests on a new major release
and see what happens.

### A test failed, what do I do?
Run a test manually to get the error. (Most) errors are not sent to the logs
as the logs are mainly to say if something was "PASSED", "FAILED", or "SKIPPED".

### A test isn't descriptive enough on r_log or comments, can I PR for that?
Absolutely - If you feel there is a gap, please fork and change what you feel
needs more information!

### Do I really need SELinux enforcing to run/add tests?
Yes.

### Why though?
Ensuring the tests work and operate under default conditions (firewall and 
selinux are up) helps those who use our distribution in environments where
security is important, actually work and function correctly.

With that said, There is no reason to disable integral security layers on your
system.

### Should EPEL be enabled?
No. The point is to test Rocky packages, not EPEL. There are also package
differences that will break (eg: nc -> nmap-ncat vs netcat).

### What about CRB or extras?
It may say it's a failure, but it will continue anyway.

Current Tree
------------
```
.
├── common
│   ├── exports.sh
│   ├── files
│   │   ├── correct-passwd
│   │   ├── correct-shadow
│   │   ├── dovecot-test-sasl
│   │   ├── hello.c
│   │   ├── hello.cpp
│   │   ├── incorrect-passwd
│   │   ├── incorrect-shadow
│   │   ├── lamp-sql
│   │   ├── lamp-sql-php
│   │   ├── malform-group
│   │   ├── malform-gshadow
│   │   ├── openssl-answers
│   │   ├── postfix-test-sasl
│   │   ├── postfix-test-tls
│   │   └── smb.conf
│   └── imports.sh
├── core
│   ├── pkg_acl
│   │   ├── 00-install-acl.sh
│   │   ├── 10-test-acl-functions.sh
│   │   └── README.md
│   ├── pkg_archive
│   │   ├── 00-install-formats.sh
│   │   ├── 10-bzip.sh
│   │   ├── 20-gzip-bin-test.sh
│   │   ├── 21-gzip-test.sh
│   │   ├── 22-gzexe.sh
│   │   ├── 23-zcmp-zdiff.sh
│   │   ├── 24-zforce.sh
│   │   ├── 25-zgrep.sh
│   │   ├── 25-zless.sh
│   │   ├── 26-zmore.sh
│   │   ├── 27-znew.sh
│   │   ├── 30-tar.sh
│   │   ├── 40-xzcmp-xzdiff.sh
│   │   ├── 40-zip.sh
│   │   ├── 50-lzop.sh
│   │   └── README.md
│   ├── pkg_attr
│   │   ├── 00-install-attr.sh
│   │   ├── 10-check-attr.sh
│   │   └── README.md
│   ├── pkg_auditd
│   │   ├── 00-install-auditd.sh
│   │   ├── 10-auditd-logs.sh
│   │   ├── 11-generate-events.sh
│   │   └── README.md
│   ├── pkg_bash
│   │   ├── 00-bash-version.sh
│   │   └── README.md
│   ├── pkg_bc
│   │   ├── 00-install-bc.sh
│   │   ├── 10-test-calculation.sh
│   │   └── README.md
│   ├── pkg_bind
│   │   ├── 00-install-bind.sh
│   │   ├── 10-test-lookup.sh
│   │   └── README.md
│   ├── pkg_coreutils
│   │   ├── 00-install-coreutils.sh
│   │   ├── 10-arch.sh
│   │   ├── 11-basename.sh
│   │   ├── 12-cat.sh
│   │   ├── 13-cut.sh
│   │   ├── 14-bool.sh
│   │   ├── 15-heads-tails.sh
│   │   ├── 16-pathchk.sh
│   │   ├── 17-readlink.sh
│   │   ├── 18-seq.sh
│   │   ├── 19-timeout.sh
│   │   ├── 20-hash.sh
│   │   ├── 21-touch-ls.sh
│   │   ├── 22-uniq.sh
│   │   ├── 23-wc.sh
│   │   ├── 24-yes.sh
│   │   └── README.md
│   ├── pkg_cpio
│   │   ├── 00-install-cpio.sh
│   │   ├── 10-cpio.sh
│   │   └── README.md
│   ├── pkg_cracklib
│   │   ├── 00-install-cracklib.sh
│   │   ├── 10-test-passwords.sh
│   │   └── README.md
│   ├── pkg_cron
│   │   ├── 00-install-cron.sh
│   │   ├── 10-dot-cron.sh
│   │   └── README.md
│   ├── pkg_curl
│   │   ├── 00-install-curl.sh
│   │   ├── 10-test-curl.sh
│   │   └── README.md
│   ├── pkg_diffutils
│   │   ├── 00-install-diff.sh
│   │   └── README.md
│   ├── pkg_dnf
│   │   ├── 10-remove-package.sh
│   │   └── README.md
│   ├── pkg_dovecot
│   │   ├── 00-install-dovecot.sh
│   │   ├── 01-configure-dovecot.sh
│   │   ├── 10-pop3-test.sh
│   │   ├── 11-imap-test.sh
│   │   ├── 12-dovecot-clean.sh
│   │   └── README.md
│   ├── pkg_file
│   │   ├── 00-install-file.sh
│   │   ├── 10-mime-check.sh
│   │   ├── 20-mime-image.sh
│   │   ├── 30-mime-symlink.sh
│   │   └── README.md
│   ├── pkg_findutils
│   │   ├── 00-install-findutils.sh
│   │   ├── 10-find.sh
│   │   └── README.md
│   ├── pkg_firefox
│   │   ├── 00-install-firefox.sh
│   │   ├── 10-check-firefox-start-page.sh
│   │   └── README.md
│   ├── pkg_firewalld
│   │   ├── 00-install-firewalld.sh
│   │   ├── 10-firewalld-check-rule.sh
│   │   └── README.md
│   ├── pkg_freeradius
│   │   ├── 00-install-freeradius.sh
│   │   ├── 10-test-freeradius.sh
│   │   └── README.md
│   ├── pkg_gcc
│   │   ├── 00-install-gcc.sh
│   │   ├── 10-gcc-build-simple.sh
│   │   ├── 11-gcc-build-cpp.sh
│   │   ├── 20-annobin-test-gcc.sh
│   │   ├── 21-annobin-test-gplusplus.sh
│   │   └── README.md
│   ├── pkg_git
│   │   ├── 00-install-git.sh
│   │   ├── 10-test-git.sh
│   │   ├── 11-test-clone-log.sh
│   │   └── README.md
│   ├── pkg_httpd
│   │   ├── 00-install-httpd.sh
│   │   ├── 10-httpd-branding.sh
│   │   ├── 20-test-basic-http.sh
│   │   ├── 21-test-basic-https.sh
│   │   ├── 30-test-basic-auth.sh
│   │   ├── 40-test-basic-vhost.sh
│   │   ├── 50-test-basic-php.sh
│   │   └── README.md
│   ├── pkg_kernel
│   │   ├── 10-test-kernel-keyring.sh
│   │   ├── 11-test-secure-boot.sh
│   │   ├── 12-test-debrand.sh
│   │   └── README.md
│   ├── pkg_lsb
│   │   ├── 00-install-lsb.sh
│   │   ├── 10-test-branding.sh
│   │   └── README.md
│   ├── pkg_lsof
│   │   ├── 00-install-lsof.sh
│   │   ├── 10-test-lsof.sh
│   │   └── README.md
│   ├── pkg_mdadm
│   │   ├── 00-install-mdadm.sh
│   │   └── 01-test-mdadm.sh
│   ├── pkg_network
│   │   ├── 00-install-packages.sh
│   │   ├── 10-tracepath.sh
│   │   ├── 11-traceroute.sh
│   │   ├── 12-mtr.sh
│   │   ├── 13-iptraf.sh
│   │   ├── 20-configure-bridge.sh
│   │   ├── 30-test-arpwatch.sh
│   │   ├── imports.sh
│   │   └── README.md
│   ├── pkg_nfs
│   │   ├── 00-install-nfs.sh
│   │   ├── 10-prepare-nfs-ro.sh
│   │   ├── 11-prepare-nfs-rw.sh
│   │   ├── 12-prepare-autofs.sh
│   │   └── README.md
│   ├── pkg_openssl
│   │   ├── 00-install-openssl.sh
│   │   ├── 10-test-openssl.sh
│   │   └── README.md
│   ├── pkg_perl
│   │   ├── 00-install-perl.sh
│   │   ├── 10-test-perl.sh
│   │   ├── 11-test-perl-script.sh
│   │   └── README.md
│   ├── pkg_postfix
│   │   ├── 00-install-postfix.sh
│   │   ├── 10-test-helo.sh
│   │   ├── 20-mta.sh
│   │   ├── 30-postfix-sasl.sh
│   │   ├── 40-postfix-tls.sh
│   │   └── README.md
│   ├── pkg_postgresql
│   │   ├── 00-install-postgresql.sh
│   │   ├── 01-configure-postgresql.sh
│   │   ├── 10-create-db.sh
│   │   ├── 11-create-user.sh
│   │   ├── 20-drop-db.sh
│   │   └── 21-drop-user.sh
│   ├── pkg_python
│   │   ├── 00-install-python.sh
│   │   ├── 10-test-python3.sh
│   │   └── README.md
│   ├── pkg_release
│   │   ├── 00-install-file.sh
│   │   ├── 10-name-sanity-check.sh
│   │   ├── 20-check-gpg-keys.sh
│   │   ├── 30-os-release.sh
│   │   ├── 40-system-release.sh
│   │   └── README.md
│   ├── pkg_rootfiles
│   │   ├── 00-install-rootfiles.sh
│   │   └── 10-test-rootfiles.sh
│   ├── pkg_rsyslog
│   │   ├── 00-install-rsyslog.sh
│   │   ├── 10-test-syslog.sh
│   │   └── README.md
│   ├── pkg_samba
│   │   ├── 00-install-samba.sh
│   │   ├── 10-test-samba.sh
│   │   └── README.md
│   ├── pkg_secureboot
│   │   ├── 10-test-grub-secureboot.sh
│   │   ├── 11-test-shim-certs.sh
│   │   └── README.md
│   ├── pkg_selinux
│   │   ├── 00-install-selinux-tools.sh
│   │   ├── 10-check-alerts.sh
│   │   └── 20-check-policy-mismatch.sh
│   ├── pkg_setup
│   │   ├── 00-test-shells.sh
│   │   ├── 10-test-group-file.sh
│   │   ├── 20-test-passwd-file.sh
│   │   └── README.md
│   ├── pkg_shadow-utils
│   │   ├── 00-install.sh
│   │   ├── 10-files-verify.sh
│   │   ├── 20-user-tests.sh
│   │   ├── 30-group-tests.sh
│   │   ├── 40-pw.sh
│   │   ├── 90-clean.sh
│   │   └── README.md
│   ├── pkg_snmp
│   │   ├── 00-install-snmp.sh
│   │   ├── 10-test-snmp-1.sh
│   │   ├── 11-test-snmp-2.sh
│   │   ├── 12-test-snmp-3.sh
│   │   └── README.md
│   ├── pkg_sqlite
│   │   ├── 00-install-sqlite.sh
│   │   ├── 10-sqlite-tables.sh
│   │   ├── 20-sqlite-dump.sh
│   │   └── README.md
│   ├── pkg_strace
│   │   ├── 00-install-strace.sh
│   │   ├── 10-test-strace.sh
│   │   └── README.md
│   ├── pkg_sysstat
│   │   ├── 00-install-sysstat.sh
│   │   ├── 10-iostat.sh
│   │   ├── 11-cpu.sh
│   │   ├── 12-cpu-io.sh
│   │   └── README.md
│   ├── pkg_systemd
│   │   ├── 00-systemd-list-services.sh
│   │   ├── 10-systemd-list-non-native-sevices.sh
│   │   ├── 11-systemd-service-status.sh
│   │   ├── 20-systemd-journald.sh
│   │   └── README.md
│   ├── pkg_tcpdump
│   │   └── README.md
│   ├── pkg_telnet
│   │   ├── 00-install-telnet.sh
│   │   └── 10-test-telnet.sh
│   ├── pkg_tftp-server
│   │   ├── 00-install-tftp.sh
│   │   ├── 01-configure-tftp.sh
│   │   ├── 10-get-test.sh
│   │   └── 11-put-test.sh
│   ├── pkg_vsftpd
│   │   ├── 00-install-vsftpd.sh
│   │   ├── 10-anonymous-vsftpd.sh
│   │   ├── 20-local-login.sh
│   │   ├── 30-cleanup.sh
│   │   └── README.md
│   ├── pkg_wget
│   │   ├── 00-install-wget.sh
│   │   ├── 10-test-wget.sh
│   │   └── README.md
│   └── pkg_which
│       ├── 00-install-which.sh
│       ├── 10-test-which.sh
│       └── README.md
├── debrand.list
├── lib
├── log
│   └── README.md
├── mods.list
├── modules
├── monotests.sh
├── README.md
├── runtests.sh
├── skip.list
├── stacks
│   ├── ipa
│   │   ├── 00-ipa-pregame.sh
│   │   ├── 10-install-ipa.sh
│   │   ├── 11-configure-ipa.sh
│   │   ├── 12-verify-ipa.sh
│   │   ├── 20-ipa-user.sh
│   │   ├── 21-ipa-service.sh
│   │   ├── 22-ipa-dns.sh
│   │   ├── 23-ipa-sudo.sh
│   │   ├── 50-cleanup-ipa.sh
│   │   └── README.md
│   └── lamp
│       ├── 00-install-lamp.sh
│       ├── 01-verification.sh
│       └── 10-test-lamp.sh
└── stacks.sh
```