os-autoinst-distri-rocky/tests
Adam Williamson 92d588f245 Add support for testing updates
Summary:
This adds an entirely new workflow for testing distribution
updates. The `ADVISORY` variable is introduced: when set,
`main.pm` will load an early post-install test that sets up
a repository containing the packages from the specified update,
runs `dnf -y update`, and reboots. A new templates file is
added, `templates-updates`, which adds two new flavors called
`updates-server` and `updates-workstation`, each containing
job templates for appropriate post-install tests. Scheduler is
expected to post `ADVISORY=(update ID) HDD_1=(base image)
FLAVOR=updates-(server|workstation)`, where (base image) is one
of the stable release base disk images produced by `createhdds`
and usually used for upgrade testing. This will result in the
appropriate job templates being loaded.

We rejig postinstall test loading and static network config a
bit so that this works for both the 'compose' and 'updates' test
flows: we have to ensure we bring up networking for the tap
tests before we try and install the updates, but still allow
later adjustment of the configuration. We take advantage of the
openQA feature that was added a few months back to run the same
module multiple times, so the `_advisory_update` module can
reboot after installing the updates and the modules that take
care of bootloader, encryption and login get run again. This
looks slightly wacky in the web UI, though - it doesn't show the
later runs of each module.

We also use the recently added feature to specify `+HDD_1` in
the test suites which use a disk image uploaded by an earlier
post-install test, so the test suite value will take priority
over the value POSTed by the scheduler for those tests, and we
will use the uploaded disk image (and not the clean base image
POSTed by the scheduler) for those tests.

My intent here is to enhance the scheduler, adding a consumer
which listens out for critpath updates, and runs this test flow
for each one, then reports the results to ResultsDB where Bodhi
could query and display them. We could also add a list of other
packages to have one or both sets of update tests run on it, I
guess.

Test Plan:
Try a post something like:
HDD_1=disk_f25_server_3_x86_64.img DISTRI=fedora VERSION=25
FLAVOR=updates-server ARCH=x86_64 BUILD=FEDORA-2017-376ae2b92c
ADVISORY=FEDORA-2017-376ae2b92c CURRREL=25 PREVREL=24

Pick an appropriate `ADVISORY` (ideally, one containing some
packages which might actually be involved in the tests), and
matching `FLAVOR` and `HDD_1`. The appropriate tests should run,
a repo with the update packages should be created and enabled
(and dnf update run), and the tests should work properly. Also
test a regular compose run to make sure I didn't break anything.

Reviewers: jskladan, jsedlak

Reviewed By: jsedlak

Subscribers: tflink

Differential Revision: https://phab.qa.fedoraproject.org/D1143
2017-02-22 11:33:32 -08:00
..
_advisory_update.pm Add support for testing updates 2017-02-22 11:33:32 -08:00
_anaconda_network_static.pm Add support for testing updates 2017-02-22 11:33:32 -08:00
_boot_to_anaconda.pm Move all remaining utility functions into exporter modules 2017-01-17 23:15:44 -08:00
_check_install_source.pm Move all remaining utility functions into exporter modules 2017-01-17 23:15:44 -08:00
_collect_data.pm Use systemctl --no-legend when collecting service info 2017-01-11 11:58:37 -08:00
_console_avc_crash.pm Move all remaining utility functions into exporter modules 2017-01-17 23:15:44 -08:00
_console_login.pm Move all remaining utility functions into exporter modules 2017-01-17 23:15:44 -08:00
_console_shutdown.pm wait longer for system to shutdown in _console_shutdown 2016-10-26 14:03:15 -07:00
_console_wait_login.pm Make log upload work when installed system hits emergency mode 2017-02-01 12:30:21 +01:00
_do_install_and_reboot.pm Move all remaining utility functions into exporter modules 2017-01-17 23:15:44 -08:00
_graphical_input.pm Move all remaining utility functions into exporter modules 2017-01-17 23:15:44 -08:00
_graphical_wait_login.pm Add support for testing updates 2017-02-22 11:33:32 -08:00
_memcheck.pm Add jobs to gather memory usage data 2017-01-16 09:30:14 -08:00
_post_network_static.pm Add support for testing updates 2017-02-22 11:33:32 -08:00
_software_selection.pm try to be safer when typing in X: slower, more checks 2016-09-12 10:24:30 -07:00
_support_server.pm Add support for testing updates 2017-02-22 11:33:32 -08:00
base_selinux.pm consolidate login waits, use postinstall not entrypoint for base 2016-09-27 11:48:15 -07:00
base_service_manipulation.pm Move all remaining utility functions into exporter modules 2017-01-17 23:15:44 -08:00
base_services_start.pm Simplify the services test fix 2017-01-11 16:03:09 -08:00
base_update_cli.pm consolidate login waits, use postinstall not entrypoint for base 2016-09-27 11:48:15 -07:00
database_client.pm Add support for testing updates 2017-02-22 11:33:32 -08:00
desktop_browser.pm Move all remaining utility functions into exporter modules 2017-01-17 23:15:44 -08:00
desktop_notifications.pm Move all remaining utility functions into exporter modules 2017-01-17 23:15:44 -08:00
desktop_terminal.pm Move all remaining utility functions into exporter modules 2017-01-17 23:15:44 -08:00
desktop_update_graphical.pm Move all remaining utility functions into exporter modules 2017-01-17 23:15:44 -08:00
disk_custom_btrfs.pm Move all remaining utility functions into exporter modules 2017-01-17 23:15:44 -08:00
disk_custom_ext3.pm Move all remaining utility functions into exporter modules 2017-01-17 23:15:44 -08:00
disk_custom_iscsi_postinstall.pm add an iscsi test, and a support_server test to support it 2016-06-09 08:43:46 -07:00
disk_custom_iscsi.pm Move all remaining utility functions into exporter modules 2017-01-17 23:15:44 -08:00
disk_custom_lvmthin.pm Move all remaining utility functions into exporter modules 2017-01-17 23:15:44 -08:00
disk_custom_no_swap.pm Move all remaining utility functions into exporter modules 2017-01-17 23:15:44 -08:00
disk_custom_software_raid_postinstall.pm use validate_script_output instead of typing and needles matching 2015-11-04 14:38:36 +01:00
disk_custom_software_raid.pm Move all remaining utility functions into exporter modules 2017-01-17 23:15:44 -08:00
disk_custom_xfs.pm Move all remaining utility functions into exporter modules 2017-01-17 23:15:44 -08:00
disk_guided_delete_all.pm Move all remaining utility functions into exporter modules 2017-01-17 23:15:44 -08:00
disk_guided_delete_partial_postinstall.pm use assert_script_run when possible 2016-01-12 09:27:14 -08:00
disk_guided_delete_partial.pm Move all remaining utility functions into exporter modules 2017-01-17 23:15:44 -08:00
disk_guided_empty.pm Move all remaining utility functions into exporter modules 2017-01-17 23:15:44 -08:00
disk_guided_encrypted_postinstall.pm add encrypted workstation upgrade tests (current and previous) 2016-07-08 08:56:57 -07:00
disk_guided_encrypted.pm Move all remaining utility functions into exporter modules 2017-01-17 23:15:44 -08:00
disk_guided_free_space_postinstall.pm use assert_script_run when possible 2016-01-12 09:27:14 -08:00
disk_guided_multi_empty_all_postinstall.pm use validate_script_output instead of typing and needles matching 2015-11-04 14:38:36 +01:00
disk_guided_multi_empty_all.pm Move all remaining utility functions into exporter modules 2017-01-17 23:15:44 -08:00
disk_guided_multi_postinstall.pm fix server_multi postinstall for disk being virtio 2016-01-27 01:44:20 -08:00
disk_guided_multi.pm Move all remaining utility functions into exporter modules 2017-01-17 23:15:44 -08:00
disk_guided_shrink_postinstall.pm use assert_script_run when possible 2016-01-12 09:27:14 -08:00
disk_guided_shrink.pm Move all remaining utility functions into exporter modules 2017-01-17 23:15:44 -08:00
firewall_configured.pm consolidate login waits, use postinstall not entrypoint for base 2016-09-27 11:48:15 -07:00
firewall_disabled.pm consolidate login waits, use postinstall not entrypoint for base 2016-09-27 11:48:15 -07:00
freeipa_client.pm Move all remaining utility functions into exporter modules 2017-01-17 23:15:44 -08:00
freeipa_password_change.pm Move all remaining utility functions into exporter modules 2017-01-17 23:15:44 -08:00
freeipa_webui.pm Move all remaining utility functions into exporter modules 2017-01-17 23:15:44 -08:00
install_arm_image_deployment.pm Move all remaining utility functions into exporter modules 2017-01-17 23:15:44 -08:00
install_source_graphical.pm Move all remaining utility functions into exporter modules 2017-01-17 23:15:44 -08:00
install_text.pm Move all remaining utility functions into exporter modules 2017-01-17 23:15:44 -08:00
realmd_join_cockpit.pm Add support for testing updates 2017-02-22 11:33:32 -08:00
realmd_join_sssd.pm Add support for testing updates 2017-02-22 11:33:32 -08:00
rescue_mode_encrypted.pm reintroduce rescue on UEFI 2016-12-13 09:18:34 +01:00
role_deploy_database_server.pm Add support for testing updates 2017-02-22 11:33:32 -08:00
role_deploy_domain_controller.pm Add support for testing updates 2017-02-22 11:33:32 -08:00
server_cockpit_basic.pm Move all remaining utility functions into exporter modules 2017-01-17 23:15:44 -08:00
server_cockpit_default.pm Move all remaining utility functions into exporter modules 2017-01-17 23:15:44 -08:00
server_firewall_default.pm consolidate login waits, use postinstall not entrypoint for base 2016-09-27 11:48:15 -07:00
uefi_postinstall.pm use assert_script_run when possible 2016-01-12 09:27:14 -08:00
upgrade_postinstall.pm Move all remaining utility functions into exporter modules 2017-01-17 23:15:44 -08:00
upgrade_preinstall.pm Move all remaining utility functions into exporter modules 2017-01-17 23:15:44 -08:00
upgrade_run.pm Move all remaining utility functions into exporter modules 2017-01-17 23:15:44 -08:00