os-autoinst-distri-rocky-mi.../lib/installedtest.pm

113 lines
4 KiB
Perl
Raw Normal View History

package installedtest;
create fedora base class, factor out console login Summary: Root console in anaconda got broken by RHBZ #1222413 - no shell on tty2. Decided to clean up console use in general as part of fixing it. This creates a class 'fedorabase' and has 'anacondalog' and 'fedoralog' both inherit from it. boot_to_login_screen is moved there (as it seems appropriate) and it has a new method, console_login, which basically handles 'get me a shell on a console': if we're already at one it returns, if not it'll type the user name and the password *if necessary* (sometimes it's not) and return once it sees a prompt. It takes a hash of named parameters for user, password and 'check', which is whether it should die if it fails to reach a console or not (some users don't want it to). anacondalog and fedoralog both get 'root_console' methods which do something appropriate and then call console_login; both have a hash of named parameters, anacondalog's version only bothers with 'check', while fedoralog's also accepts 'tty' to pick the tty to use. This also adjusts all things which try to get to a console prompt to use either root_console or console_login as appropriate. It also tweaks the needle tags a bit, drops some unneeded needles, and adds a new 'user console prompt' needle; we really just need two versions of the root prompt needle and two of the user prompt needle (one for <F23, one for F23+ - the console font changed in F23, and the @ character at least doesn't match between the two). I think we still need the <F23 case for upgrade tests, for now. Test Plan: Do a full test run and see that more tests succeed. I've done a run on happyassassin with a hack to workaround the SELinux issue for interactive installs, and the results look good. I also fiddled about a bit to test some different cases, like forcing a failure in a live test to test post_fail_hook (and hence root_console) in that scenario, and forcing failures after some console commands had been run to check that it DTRT when we've already reached a console, etc. Reviewers: jskladan, garretraziel Reviewed By: jskladan, garretraziel Subscribers: tflink Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D462
2015-07-22 18:24:40 +00:00
use base 'fedorabase';
# base class for tests that run on installed system
# should be used when with tests, where system is already installed, e. g all parts
# of upgrade tests, postinstall phases...
use testapi;
use main_common;
create fedora base class, factor out console login Summary: Root console in anaconda got broken by RHBZ #1222413 - no shell on tty2. Decided to clean up console use in general as part of fixing it. This creates a class 'fedorabase' and has 'anacondalog' and 'fedoralog' both inherit from it. boot_to_login_screen is moved there (as it seems appropriate) and it has a new method, console_login, which basically handles 'get me a shell on a console': if we're already at one it returns, if not it'll type the user name and the password *if necessary* (sometimes it's not) and return once it sees a prompt. It takes a hash of named parameters for user, password and 'check', which is whether it should die if it fails to reach a console or not (some users don't want it to). anacondalog and fedoralog both get 'root_console' methods which do something appropriate and then call console_login; both have a hash of named parameters, anacondalog's version only bothers with 'check', while fedoralog's also accepts 'tty' to pick the tty to use. This also adjusts all things which try to get to a console prompt to use either root_console or console_login as appropriate. It also tweaks the needle tags a bit, drops some unneeded needles, and adds a new 'user console prompt' needle; we really just need two versions of the root prompt needle and two of the user prompt needle (one for <F23, one for F23+ - the console font changed in F23, and the @ character at least doesn't match between the two). I think we still need the <F23 case for upgrade tests, for now. Test Plan: Do a full test run and see that more tests succeed. I've done a run on happyassassin with a hack to workaround the SELinux issue for interactive installs, and the results look good. I also fiddled about a bit to test some different cases, like forcing a failure in a live test to test post_fail_hook (and hence root_console) in that scenario, and forcing failures after some console commands had been run to check that it DTRT when we've already reached a console, etc. Reviewers: jskladan, garretraziel Reviewed By: jskladan, garretraziel Subscribers: tflink Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D462
2015-07-22 18:24:40 +00:00
sub root_console {
# Switch to a default or specified TTY and log in as root.
my $self = shift;
create fedora base class, factor out console login Summary: Root console in anaconda got broken by RHBZ #1222413 - no shell on tty2. Decided to clean up console use in general as part of fixing it. This creates a class 'fedorabase' and has 'anacondalog' and 'fedoralog' both inherit from it. boot_to_login_screen is moved there (as it seems appropriate) and it has a new method, console_login, which basically handles 'get me a shell on a console': if we're already at one it returns, if not it'll type the user name and the password *if necessary* (sometimes it's not) and return once it sees a prompt. It takes a hash of named parameters for user, password and 'check', which is whether it should die if it fails to reach a console or not (some users don't want it to). anacondalog and fedoralog both get 'root_console' methods which do something appropriate and then call console_login; both have a hash of named parameters, anacondalog's version only bothers with 'check', while fedoralog's also accepts 'tty' to pick the tty to use. This also adjusts all things which try to get to a console prompt to use either root_console or console_login as appropriate. It also tweaks the needle tags a bit, drops some unneeded needles, and adds a new 'user console prompt' needle; we really just need two versions of the root prompt needle and two of the user prompt needle (one for <F23, one for F23+ - the console font changed in F23, and the @ character at least doesn't match between the two). I think we still need the <F23 case for upgrade tests, for now. Test Plan: Do a full test run and see that more tests succeed. I've done a run on happyassassin with a hack to workaround the SELinux issue for interactive installs, and the results look good. I also fiddled about a bit to test some different cases, like forcing a failure in a live test to test post_fail_hook (and hence root_console) in that scenario, and forcing failures after some console commands had been run to check that it DTRT when we've already reached a console, etc. Reviewers: jskladan, garretraziel Reviewed By: jskladan, garretraziel Subscribers: tflink Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D462
2015-07-22 18:24:40 +00:00
my %args = (
tty => 1, # what TTY to login to
create fedora base class, factor out console login Summary: Root console in anaconda got broken by RHBZ #1222413 - no shell on tty2. Decided to clean up console use in general as part of fixing it. This creates a class 'fedorabase' and has 'anacondalog' and 'fedoralog' both inherit from it. boot_to_login_screen is moved there (as it seems appropriate) and it has a new method, console_login, which basically handles 'get me a shell on a console': if we're already at one it returns, if not it'll type the user name and the password *if necessary* (sometimes it's not) and return once it sees a prompt. It takes a hash of named parameters for user, password and 'check', which is whether it should die if it fails to reach a console or not (some users don't want it to). anacondalog and fedoralog both get 'root_console' methods which do something appropriate and then call console_login; both have a hash of named parameters, anacondalog's version only bothers with 'check', while fedoralog's also accepts 'tty' to pick the tty to use. This also adjusts all things which try to get to a console prompt to use either root_console or console_login as appropriate. It also tweaks the needle tags a bit, drops some unneeded needles, and adds a new 'user console prompt' needle; we really just need two versions of the root prompt needle and two of the user prompt needle (one for <F23, one for F23+ - the console font changed in F23, and the @ character at least doesn't match between the two). I think we still need the <F23 case for upgrade tests, for now. Test Plan: Do a full test run and see that more tests succeed. I've done a run on happyassassin with a hack to workaround the SELinux issue for interactive installs, and the results look good. I also fiddled about a bit to test some different cases, like forcing a failure in a live test to test post_fail_hook (and hence root_console) in that scenario, and forcing failures after some console commands had been run to check that it DTRT when we've already reached a console, etc. Reviewers: jskladan, garretraziel Reviewed By: jskladan, garretraziel Subscribers: tflink Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D462
2015-07-22 18:24:40 +00:00
@_);
create fedora base class, factor out console login Summary: Root console in anaconda got broken by RHBZ #1222413 - no shell on tty2. Decided to clean up console use in general as part of fixing it. This creates a class 'fedorabase' and has 'anacondalog' and 'fedoralog' both inherit from it. boot_to_login_screen is moved there (as it seems appropriate) and it has a new method, console_login, which basically handles 'get me a shell on a console': if we're already at one it returns, if not it'll type the user name and the password *if necessary* (sometimes it's not) and return once it sees a prompt. It takes a hash of named parameters for user, password and 'check', which is whether it should die if it fails to reach a console or not (some users don't want it to). anacondalog and fedoralog both get 'root_console' methods which do something appropriate and then call console_login; both have a hash of named parameters, anacondalog's version only bothers with 'check', while fedoralog's also accepts 'tty' to pick the tty to use. This also adjusts all things which try to get to a console prompt to use either root_console or console_login as appropriate. It also tweaks the needle tags a bit, drops some unneeded needles, and adds a new 'user console prompt' needle; we really just need two versions of the root prompt needle and two of the user prompt needle (one for <F23, one for F23+ - the console font changed in F23, and the @ character at least doesn't match between the two). I think we still need the <F23 case for upgrade tests, for now. Test Plan: Do a full test run and see that more tests succeed. I've done a run on happyassassin with a hack to workaround the SELinux issue for interactive installs, and the results look good. I also fiddled about a bit to test some different cases, like forcing a failure in a live test to test post_fail_hook (and hence root_console) in that scenario, and forcing failures after some console commands had been run to check that it DTRT when we've already reached a console, etc. Reviewers: jskladan, garretraziel Reviewed By: jskladan, garretraziel Subscribers: tflink Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D462
2015-07-22 18:24:40 +00:00
send_key "ctrl-alt-f$args{tty}";
console_login;
}
sub post_fail_hook {
my $self = shift;
$self->root_console(tty=>6);
# We can't rely on tar being in minimal installs
assert_script_run "dnf -y install tar", 180;
# Note: script_run returns the exit code, so the logic looks weird.
# We're testing that the directory exists and contains something.
unless (script_run 'test -n "$(ls -A /var/tmp/abrt)" && cd /var/tmp/abrt && tar czvf tmpabrt.tar.gz *') {
add a cockpit realmd FreeIPA join test Summary: This requires a few other changes: * turn clone_host_resolv into clone_host_file, letting you clone any given host file (cloning /etc/hosts seems to make both server deployment and client enrolment faster/more reliable) * allow loading of multiple POSTINSTALL tests (so we can share the freeipa_client_postinstall test). Note this is compatible, existing uses will work fine * move initial password change for the IPA test users into the server deployment test (so the client tests don't conflict over doing that) * add GRUB_POSTINSTALL, for specifying boot parameters for boot of the installed system, and make it work by tweaking _console_wait _login (doesn't work for _graphical_wait_login yet, as I didn't need that) * make the static networking config for tap tests into a library function so the tests can share it * handle ABRT problem dirs showing up in /var/spool/abrt as well as /var/tmp/abrt (because the enrol attempt hits #1330766 and the crash report shows up in /var/spool/abrt, don't ask me why the difference, I just work here) * specify the DNS servers from the worker host's resolv.conf as the forwarders for the FreeIPA server when deploying it; if we don't do this, rolekit defaults to using the root servers as forwarders(!) and thus we get the public, not phx2-appropriate, results for e.g. mirrors.fedoraproject.org, some of which the workers can't reach, so PackageKit package install always fails (boy, was it fun figuring THAT mess out) Even after all that, the test still doesn't actually pass, but I'm reasonably confident this is because it's hitting actual bugs, not because it's broken. It runs into #1330766 nearly every time (I think I saw *one* time the enrolment actually succeeded), and seems to run into a subsequent bug I hadn't seen before when trying to work around that by trying the join again (see https://bugzilla.redhat.com/show_bug.cgi?id=1330766#c37 ). Test Plan: Run the test, see what happens. If you're really lucky, it'll actually pass. But you'll probably run into #1330766#c37, I'm mostly posting for comment. You'll need a tap-capable openQA instance to test this. Reviewers: jskladan, garretraziel Reviewed By: garretraziel Subscribers: tflink Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D880
2016-06-07 20:00:39 +00:00
upload_logs "/var/tmp/abrt/tmpabrt.tar.gz";
}
unless (script_run 'test -n "$(ls -A /var/spool/abrt)" && cd /var/spool/abrt && tar czvf spoolabrt.tar.gz *') {
add a cockpit realmd FreeIPA join test Summary: This requires a few other changes: * turn clone_host_resolv into clone_host_file, letting you clone any given host file (cloning /etc/hosts seems to make both server deployment and client enrolment faster/more reliable) * allow loading of multiple POSTINSTALL tests (so we can share the freeipa_client_postinstall test). Note this is compatible, existing uses will work fine * move initial password change for the IPA test users into the server deployment test (so the client tests don't conflict over doing that) * add GRUB_POSTINSTALL, for specifying boot parameters for boot of the installed system, and make it work by tweaking _console_wait _login (doesn't work for _graphical_wait_login yet, as I didn't need that) * make the static networking config for tap tests into a library function so the tests can share it * handle ABRT problem dirs showing up in /var/spool/abrt as well as /var/tmp/abrt (because the enrol attempt hits #1330766 and the crash report shows up in /var/spool/abrt, don't ask me why the difference, I just work here) * specify the DNS servers from the worker host's resolv.conf as the forwarders for the FreeIPA server when deploying it; if we don't do this, rolekit defaults to using the root servers as forwarders(!) and thus we get the public, not phx2-appropriate, results for e.g. mirrors.fedoraproject.org, some of which the workers can't reach, so PackageKit package install always fails (boy, was it fun figuring THAT mess out) Even after all that, the test still doesn't actually pass, but I'm reasonably confident this is because it's hitting actual bugs, not because it's broken. It runs into #1330766 nearly every time (I think I saw *one* time the enrolment actually succeeded), and seems to run into a subsequent bug I hadn't seen before when trying to work around that by trying the join again (see https://bugzilla.redhat.com/show_bug.cgi?id=1330766#c37 ). Test Plan: Run the test, see what happens. If you're really lucky, it'll actually pass. But you'll probably run into #1330766#c37, I'm mostly posting for comment. You'll need a tap-capable openQA instance to test this. Reviewers: jskladan, garretraziel Reviewed By: garretraziel Subscribers: tflink Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D880
2016-06-07 20:00:39 +00:00
upload_logs "/var/spool/abrt/spoolabrt.tar.gz";
}
# Upload /var/log
add QA:Testcase_FreeIPA_password_change test Summary: again, added as a non-fatal module for realmd_join_cockpit as it's convenient to do it here. Also abstract a couple of ipa bits into a new exporter package in the style of SUSE's mm_network, rather than using ill-fitting class inheritance as we have before - we should probably convert our existing class based stuff to work this way. Also a few minor tweaks and clean-ups of the other tests: The path in console_login() where we detect login of a regular user when we want root or vice versa and log out was actually broken because it would 'wait' for the result of the 'exit' command, which obviously doesn't work (as it relies on running another command afterwards, and we're no longer at a shell). This commit no longer actually uses that path, but I spotted the bug with an earlier version of this which did, and we may as well keep the fix. /var/log/lastlog is an apparently-extremely-large sparse file. A couple of times it seemed to cause tar to run very slowly while creating the /var/log archive for upload on failure. It's no use for diagnosing bugs, so we may as well exclude it from the archive. I caught cockpit webUI login failing one time when testing the test, so threw in a wait_still_screen before starting to type the URL, as we have for the FreeIPA webUI. I also caught a timing issue with the openQA webUI policy add step; the test flips from the Users screen to the HBAC screen then clicks the 'add' button, but there's actually an identical 'add' button on *both* screens, so it could wind up trying to click the one on the Users screen instead, if the web UI took a few milliseconds to switch. So we throw in a needle match to make sure we're actually on the HBAC screen before clicking the button. We make the freeipa_webui test a 'milestone' so that if the new test fails, restoring to the last-known-good milestone doesn't take so long; it actually seems like openQA can get confused and try to cancel the test if restoring the milestone takes a *really* long time, and wind up with a zombie qemu process, which isn't good. This seems to avoid that happening. Test Plan: In the simple case, just run all the FreeIPA-related tests on Fedora 24 (as Rawhide is broken) and make sure they all work properly. To get a bit more advanced you can throw in an `assert_script_run 'false'` in either of the non-fatal tests to break it and make sure things go properly when that happens (the last milestone should be restored - which should be right after freeipa_webui, sitting at tty1 - and run properly; things are set up so each test starts with root logged in on tty1). Reviewers: jskladan, garretraziel Reviewed By: garretraziel Subscribers: tflink Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D935
2016-08-03 20:21:12 +00:00
# lastlog can mess up tar sometimes and it's not much use
unless (script_run "tar czvf /tmp/var_log.tar.gz --exclude='lastlog' /var/log") {
upload_logs "/tmp/var_log.tar.gz";
}
}
convert upgrade tests to dnf-plugin-system-upgrade Summary: This is a first cut which more or less works for now. Issues: 1) We're not really testing the BUILD, here. All the test does is try and upgrade to the specified VERSION - so it'll be using the latest 'stable' for the given VERSION at the time the test runs. This isn't really that terrible, but especially for TC/RC validation, we might want to make things a bit more elaborate and set up the repo for the actual BUILD (and disable the main repos). 2) We'd actually need --nogpgcheck for non-Rawhide, at one specific point in the release cycle - after Branching but before Bodhi activation (which is when we can be sure all packages are signed). This won't matter until 24 branches, and maybe releng will have it fixed by then...if not, I'll tweak it. 3) We don't really test that the upgrade actually *happened* for desktop, at the moment - the only thing in the old test that really checked that was where we checked for the fedup boot menu entry, but that has no analog in dnf. What we should probably do is check that GUI login works, then switch to a console and check /etc/fedora-release just as the minimal test does. Test Plan: Run the tests. Note that creating the desktop disk image doesn't work ATM, so I can't verify the desktop test works, but the minimal one seems to (with D565). There'll be a matching diff for openqa_fedora_tools to update the test case names there. Reviewers: jskladan, garretraziel Reviewed By: jskladan, garretraziel Subscribers: tflink Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D567
2015-09-10 21:49:13 +00:00
sub check_release {
# Checks whether the installed release matches a given value. E.g.
# `check_release(23)` checks whether the installed system is
# Fedora 23. The value can be 'Rawhide' or a Fedora release
# number; often you will want to use `get_var('VERSION')`. Expects
# a console prompt to be active when it is called.
convert upgrade tests to dnf-plugin-system-upgrade Summary: This is a first cut which more or less works for now. Issues: 1) We're not really testing the BUILD, here. All the test does is try and upgrade to the specified VERSION - so it'll be using the latest 'stable' for the given VERSION at the time the test runs. This isn't really that terrible, but especially for TC/RC validation, we might want to make things a bit more elaborate and set up the repo for the actual BUILD (and disable the main repos). 2) We'd actually need --nogpgcheck for non-Rawhide, at one specific point in the release cycle - after Branching but before Bodhi activation (which is when we can be sure all packages are signed). This won't matter until 24 branches, and maybe releng will have it fixed by then...if not, I'll tweak it. 3) We don't really test that the upgrade actually *happened* for desktop, at the moment - the only thing in the old test that really checked that was where we checked for the fedup boot menu entry, but that has no analog in dnf. What we should probably do is check that GUI login works, then switch to a console and check /etc/fedora-release just as the minimal test does. Test Plan: Run the tests. Note that creating the desktop disk image doesn't work ATM, so I can't verify the desktop test works, but the minimal one seems to (with D565). There'll be a matching diff for openqa_fedora_tools to update the test case names there. Reviewers: jskladan, garretraziel Reviewed By: jskladan, garretraziel Subscribers: tflink Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D567
2015-09-10 21:49:13 +00:00
my $self = shift;
my $release = shift;
my $check_command = "grep SUPPORT_PRODUCT_VERSION /usr/lib/os.release.d/os-release-fedora";
validate_script_output $check_command, sub { $_ =~ m/REDHAT_SUPPORT_PRODUCT_VERSION=$release/ };
convert upgrade tests to dnf-plugin-system-upgrade Summary: This is a first cut which more or less works for now. Issues: 1) We're not really testing the BUILD, here. All the test does is try and upgrade to the specified VERSION - so it'll be using the latest 'stable' for the given VERSION at the time the test runs. This isn't really that terrible, but especially for TC/RC validation, we might want to make things a bit more elaborate and set up the repo for the actual BUILD (and disable the main repos). 2) We'd actually need --nogpgcheck for non-Rawhide, at one specific point in the release cycle - after Branching but before Bodhi activation (which is when we can be sure all packages are signed). This won't matter until 24 branches, and maybe releng will have it fixed by then...if not, I'll tweak it. 3) We don't really test that the upgrade actually *happened* for desktop, at the moment - the only thing in the old test that really checked that was where we checked for the fedup boot menu entry, but that has no analog in dnf. What we should probably do is check that GUI login works, then switch to a console and check /etc/fedora-release just as the minimal test does. Test Plan: Run the tests. Note that creating the desktop disk image doesn't work ATM, so I can't verify the desktop test works, but the minimal one seems to (with D565). There'll be a matching diff for openqa_fedora_tools to update the test case names there. Reviewers: jskladan, garretraziel Reviewed By: jskladan, garretraziel Subscribers: tflink Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D567
2015-09-10 21:49:13 +00:00
}
sub menu_launch_type {
# Launch an application in a graphical environment, by opening a
# launcher, typing the specified string and hitting enter. Pass
# the string to be typed to launch whatever it is you want.
my $self = shift;
my $app = shift;
# super does not work on KDE, because fml
send_key 'alt-f1';
# srsly KDE y u so slo
wait_still_screen 3;
type_very_safely $app;
send_key 'ret';
}
add cockpit_default and cockpit_basic tests Summary: This adds tests for the Server_cockpit_default and cockpit_basic test cases. Some notes: I was initially thinking of combining these into a single test with multiple test modules and coming up with a system for doing wiki reporting based on individual test module status, but because we'll also want to do a cockpit FreeIPA enrol test, I decided against it. We don't really want to combine all three because then we would skip the cockpit tests whenever FreeIPA server deployment failed, which isn't ideal. So since we'll need a separate FreeIPA enrolment test anyway it doesn't really make sense to go to the trouble of designing a system for loading multiple postinstall tests (though I have an idea for that!) and a per-module wiki reporting system. This was the most minimal and hopefully reliable method for running Cockpit from a stock Server install that I could think of. An alternative approach would be to have, say, the most recent stable Workstation live as a 'stock' asset and have two tests, one which runs a stock Server install and just waits and another which boots the live image and accesses the cockpit running on the other box, but that seems a bit over-complex. It is not possible to have dependencies between tests for different ISOs, in case you were wondering about having a Workstation live test which runs parallel with a Server DVD test, we can't do that. One funny thing is the font that winds up getting used for the desktop, but I don't *think* that should be a problem. Picking needles was a bit tricky; any improvement suggestions are welcome. I'm hoping it turns out to be safe to rely on some dbus log messages being present; I think logging into Cockpit triggers activation of the realmd dbus interface, so there *should* always be some messages related to that. An alternative would just be to match on a sliver of the dark grey table header and the light grey row beneath it and assume that'll always be the first message (whatever the message is), but then we have to find some area of the message details screen which is always present for any message, and it just seems a tad more likely to result in false passes. Similary I'm making an assumption that auditd is always going to show up on the first page of the Services screen and the details screen will always show that 'loaded...enabled' text. Test Plan: Run the tests and see if they work! See https://openqa.stg.fedoraproject.org/tests/21373 and https://openqa.stg.fedoraproject.org/tests/21371 for my tests. Reviewers: garretraziel Reviewed By: garretraziel Subscribers: tflink Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D874
2016-06-01 16:05:33 +00:00
sub start_cockpit {
# Starting from a console, get to a browser with Cockpit (running
# on localhost) shown. If $login is truth-y, also log in. Assumes
# X and Firefox are installed.
add cockpit_default and cockpit_basic tests Summary: This adds tests for the Server_cockpit_default and cockpit_basic test cases. Some notes: I was initially thinking of combining these into a single test with multiple test modules and coming up with a system for doing wiki reporting based on individual test module status, but because we'll also want to do a cockpit FreeIPA enrol test, I decided against it. We don't really want to combine all three because then we would skip the cockpit tests whenever FreeIPA server deployment failed, which isn't ideal. So since we'll need a separate FreeIPA enrolment test anyway it doesn't really make sense to go to the trouble of designing a system for loading multiple postinstall tests (though I have an idea for that!) and a per-module wiki reporting system. This was the most minimal and hopefully reliable method for running Cockpit from a stock Server install that I could think of. An alternative approach would be to have, say, the most recent stable Workstation live as a 'stock' asset and have two tests, one which runs a stock Server install and just waits and another which boots the live image and accesses the cockpit running on the other box, but that seems a bit over-complex. It is not possible to have dependencies between tests for different ISOs, in case you were wondering about having a Workstation live test which runs parallel with a Server DVD test, we can't do that. One funny thing is the font that winds up getting used for the desktop, but I don't *think* that should be a problem. Picking needles was a bit tricky; any improvement suggestions are welcome. I'm hoping it turns out to be safe to rely on some dbus log messages being present; I think logging into Cockpit triggers activation of the realmd dbus interface, so there *should* always be some messages related to that. An alternative would just be to match on a sliver of the dark grey table header and the light grey row beneath it and assume that'll always be the first message (whatever the message is), but then we have to find some area of the message details screen which is always present for any message, and it just seems a tad more likely to result in false passes. Similary I'm making an assumption that auditd is always going to show up on the first page of the Services screen and the details screen will always show that 'loaded...enabled' text. Test Plan: Run the tests and see if they work! See https://openqa.stg.fedoraproject.org/tests/21373 and https://openqa.stg.fedoraproject.org/tests/21371 for my tests. Reviewers: garretraziel Reviewed By: garretraziel Subscribers: tflink Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D874
2016-06-01 16:05:33 +00:00
my $self = shift;
my $login = shift || 0;
# run firefox directly in X as root. never do this, kids!
type_string "startx /usr/bin/firefox -width 1024 -height 768 http://localhost:9090\n";
add cockpit_default and cockpit_basic tests Summary: This adds tests for the Server_cockpit_default and cockpit_basic test cases. Some notes: I was initially thinking of combining these into a single test with multiple test modules and coming up with a system for doing wiki reporting based on individual test module status, but because we'll also want to do a cockpit FreeIPA enrol test, I decided against it. We don't really want to combine all three because then we would skip the cockpit tests whenever FreeIPA server deployment failed, which isn't ideal. So since we'll need a separate FreeIPA enrolment test anyway it doesn't really make sense to go to the trouble of designing a system for loading multiple postinstall tests (though I have an idea for that!) and a per-module wiki reporting system. This was the most minimal and hopefully reliable method for running Cockpit from a stock Server install that I could think of. An alternative approach would be to have, say, the most recent stable Workstation live as a 'stock' asset and have two tests, one which runs a stock Server install and just waits and another which boots the live image and accesses the cockpit running on the other box, but that seems a bit over-complex. It is not possible to have dependencies between tests for different ISOs, in case you were wondering about having a Workstation live test which runs parallel with a Server DVD test, we can't do that. One funny thing is the font that winds up getting used for the desktop, but I don't *think* that should be a problem. Picking needles was a bit tricky; any improvement suggestions are welcome. I'm hoping it turns out to be safe to rely on some dbus log messages being present; I think logging into Cockpit triggers activation of the realmd dbus interface, so there *should* always be some messages related to that. An alternative would just be to match on a sliver of the dark grey table header and the light grey row beneath it and assume that'll always be the first message (whatever the message is), but then we have to find some area of the message details screen which is always present for any message, and it just seems a tad more likely to result in false passes. Similary I'm making an assumption that auditd is always going to show up on the first page of the Services screen and the details screen will always show that 'loaded...enabled' text. Test Plan: Run the tests and see if they work! See https://openqa.stg.fedoraproject.org/tests/21373 and https://openqa.stg.fedoraproject.org/tests/21371 for my tests. Reviewers: garretraziel Reviewed By: garretraziel Subscribers: tflink Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D874
2016-06-01 16:05:33 +00:00
assert_screen "cockpit_login";
wait_still_screen 5;
add cockpit_default and cockpit_basic tests Summary: This adds tests for the Server_cockpit_default and cockpit_basic test cases. Some notes: I was initially thinking of combining these into a single test with multiple test modules and coming up with a system for doing wiki reporting based on individual test module status, but because we'll also want to do a cockpit FreeIPA enrol test, I decided against it. We don't really want to combine all three because then we would skip the cockpit tests whenever FreeIPA server deployment failed, which isn't ideal. So since we'll need a separate FreeIPA enrolment test anyway it doesn't really make sense to go to the trouble of designing a system for loading multiple postinstall tests (though I have an idea for that!) and a per-module wiki reporting system. This was the most minimal and hopefully reliable method for running Cockpit from a stock Server install that I could think of. An alternative approach would be to have, say, the most recent stable Workstation live as a 'stock' asset and have two tests, one which runs a stock Server install and just waits and another which boots the live image and accesses the cockpit running on the other box, but that seems a bit over-complex. It is not possible to have dependencies between tests for different ISOs, in case you were wondering about having a Workstation live test which runs parallel with a Server DVD test, we can't do that. One funny thing is the font that winds up getting used for the desktop, but I don't *think* that should be a problem. Picking needles was a bit tricky; any improvement suggestions are welcome. I'm hoping it turns out to be safe to rely on some dbus log messages being present; I think logging into Cockpit triggers activation of the realmd dbus interface, so there *should* always be some messages related to that. An alternative would just be to match on a sliver of the dark grey table header and the light grey row beneath it and assume that'll always be the first message (whatever the message is), but then we have to find some area of the message details screen which is always present for any message, and it just seems a tad more likely to result in false passes. Similary I'm making an assumption that auditd is always going to show up on the first page of the Services screen and the details screen will always show that 'loaded...enabled' text. Test Plan: Run the tests and see if they work! See https://openqa.stg.fedoraproject.org/tests/21373 and https://openqa.stg.fedoraproject.org/tests/21371 for my tests. Reviewers: garretraziel Reviewed By: garretraziel Subscribers: tflink Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D874
2016-06-01 16:05:33 +00:00
if ($login) {
type_safely "root";
wait_screen_change { send_key "tab"; };
type_safely get_var("ROOT_PASSWORD", "weakpassword");
add cockpit_default and cockpit_basic tests Summary: This adds tests for the Server_cockpit_default and cockpit_basic test cases. Some notes: I was initially thinking of combining these into a single test with multiple test modules and coming up with a system for doing wiki reporting based on individual test module status, but because we'll also want to do a cockpit FreeIPA enrol test, I decided against it. We don't really want to combine all three because then we would skip the cockpit tests whenever FreeIPA server deployment failed, which isn't ideal. So since we'll need a separate FreeIPA enrolment test anyway it doesn't really make sense to go to the trouble of designing a system for loading multiple postinstall tests (though I have an idea for that!) and a per-module wiki reporting system. This was the most minimal and hopefully reliable method for running Cockpit from a stock Server install that I could think of. An alternative approach would be to have, say, the most recent stable Workstation live as a 'stock' asset and have two tests, one which runs a stock Server install and just waits and another which boots the live image and accesses the cockpit running on the other box, but that seems a bit over-complex. It is not possible to have dependencies between tests for different ISOs, in case you were wondering about having a Workstation live test which runs parallel with a Server DVD test, we can't do that. One funny thing is the font that winds up getting used for the desktop, but I don't *think* that should be a problem. Picking needles was a bit tricky; any improvement suggestions are welcome. I'm hoping it turns out to be safe to rely on some dbus log messages being present; I think logging into Cockpit triggers activation of the realmd dbus interface, so there *should* always be some messages related to that. An alternative would just be to match on a sliver of the dark grey table header and the light grey row beneath it and assume that'll always be the first message (whatever the message is), but then we have to find some area of the message details screen which is always present for any message, and it just seems a tad more likely to result in false passes. Similary I'm making an assumption that auditd is always going to show up on the first page of the Services screen and the details screen will always show that 'loaded...enabled' text. Test Plan: Run the tests and see if they work! See https://openqa.stg.fedoraproject.org/tests/21373 and https://openqa.stg.fedoraproject.org/tests/21371 for my tests. Reviewers: garretraziel Reviewed By: garretraziel Subscribers: tflink Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D874
2016-06-01 16:05:33 +00:00
send_key "ret";
assert_screen "cockpit_main";
# wait for any animation or other weirdness
# can't use wait_still_screen because of that damn graph
sleep 3;
add cockpit_default and cockpit_basic tests Summary: This adds tests for the Server_cockpit_default and cockpit_basic test cases. Some notes: I was initially thinking of combining these into a single test with multiple test modules and coming up with a system for doing wiki reporting based on individual test module status, but because we'll also want to do a cockpit FreeIPA enrol test, I decided against it. We don't really want to combine all three because then we would skip the cockpit tests whenever FreeIPA server deployment failed, which isn't ideal. So since we'll need a separate FreeIPA enrolment test anyway it doesn't really make sense to go to the trouble of designing a system for loading multiple postinstall tests (though I have an idea for that!) and a per-module wiki reporting system. This was the most minimal and hopefully reliable method for running Cockpit from a stock Server install that I could think of. An alternative approach would be to have, say, the most recent stable Workstation live as a 'stock' asset and have two tests, one which runs a stock Server install and just waits and another which boots the live image and accesses the cockpit running on the other box, but that seems a bit over-complex. It is not possible to have dependencies between tests for different ISOs, in case you were wondering about having a Workstation live test which runs parallel with a Server DVD test, we can't do that. One funny thing is the font that winds up getting used for the desktop, but I don't *think* that should be a problem. Picking needles was a bit tricky; any improvement suggestions are welcome. I'm hoping it turns out to be safe to rely on some dbus log messages being present; I think logging into Cockpit triggers activation of the realmd dbus interface, so there *should* always be some messages related to that. An alternative would just be to match on a sliver of the dark grey table header and the light grey row beneath it and assume that'll always be the first message (whatever the message is), but then we have to find some area of the message details screen which is always present for any message, and it just seems a tad more likely to result in false passes. Similary I'm making an assumption that auditd is always going to show up on the first page of the Services screen and the details screen will always show that 'loaded...enabled' text. Test Plan: Run the tests and see if they work! See https://openqa.stg.fedoraproject.org/tests/21373 and https://openqa.stg.fedoraproject.org/tests/21371 for my tests. Reviewers: garretraziel Reviewed By: garretraziel Subscribers: tflink Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D874
2016-06-01 16:05:33 +00:00
}
}
use compose repository (not master repo) for most tests Summary: we have a long-standing problem with all the tests that hit the repositories. The tests are triggered as soon as a compose completes. At this point in time, the compose is not synced to the mirrors, where the default 'fedora' repo definition looks; the sync happens after the compose completes, and there is also a metadata sync step that must happen after *that* before any operation that uses the 'fedora' repository definition will actually use the packages from the new compose. Thus all net install tests and tests that installed packages have been effectively testing the previous compose, not the current one. We have some thoughts about how to fix this 'properly' (such that the openQA tests wouldn't have to do anything special, but their 'fedora' repository would somehow reflect the compose under test), but none of them is in place right now or likely to happen in the short term, so in the mean time this should deal with most of the issues. With this change, everything but the default_install tests for the netinst images should use the compose-under-test's Everything tree instead of the 'fedora' repository, and thus should install and test the correct packages. This relies on a corresponding change to openqa_fedora_tools to set the LOCATION openQA setting (which is simply the base location of the compose under test). Test Plan: Do a full test run, check (as far as you can) tests run sensibly and use appropriate repositories. Reviewers: jskladan, garretraziel Reviewed By: garretraziel Subscribers: tflink Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D989
2016-09-01 15:22:59 +00:00
sub repo_setup {
# disable updates-testing and updates and use the compose location
# as the target for fedora and rawhide rather than mirrorlist, so
# tools see only packages from the compose under test
use compose repository (not master repo) for most tests Summary: we have a long-standing problem with all the tests that hit the repositories. The tests are triggered as soon as a compose completes. At this point in time, the compose is not synced to the mirrors, where the default 'fedora' repo definition looks; the sync happens after the compose completes, and there is also a metadata sync step that must happen after *that* before any operation that uses the 'fedora' repository definition will actually use the packages from the new compose. Thus all net install tests and tests that installed packages have been effectively testing the previous compose, not the current one. We have some thoughts about how to fix this 'properly' (such that the openQA tests wouldn't have to do anything special, but their 'fedora' repository would somehow reflect the compose under test), but none of them is in place right now or likely to happen in the short term, so in the mean time this should deal with most of the issues. With this change, everything but the default_install tests for the netinst images should use the compose-under-test's Everything tree instead of the 'fedora' repository, and thus should install and test the correct packages. This relies on a corresponding change to openqa_fedora_tools to set the LOCATION openQA setting (which is simply the base location of the compose under test). Test Plan: Do a full test run, check (as far as you can) tests run sensibly and use appropriate repositories. Reviewers: jskladan, garretraziel Reviewed By: garretraziel Subscribers: tflink Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D989
2016-09-01 15:22:59 +00:00
my $location = get_var("LOCATION");
assert_script_run 'dnf config-manager --set-disabled updates-testing updates';
use compose repository (not master repo) for most tests Summary: we have a long-standing problem with all the tests that hit the repositories. The tests are triggered as soon as a compose completes. At this point in time, the compose is not synced to the mirrors, where the default 'fedora' repo definition looks; the sync happens after the compose completes, and there is also a metadata sync step that must happen after *that* before any operation that uses the 'fedora' repository definition will actually use the packages from the new compose. Thus all net install tests and tests that installed packages have been effectively testing the previous compose, not the current one. We have some thoughts about how to fix this 'properly' (such that the openQA tests wouldn't have to do anything special, but their 'fedora' repository would somehow reflect the compose under test), but none of them is in place right now or likely to happen in the short term, so in the mean time this should deal with most of the issues. With this change, everything but the default_install tests for the netinst images should use the compose-under-test's Everything tree instead of the 'fedora' repository, and thus should install and test the correct packages. This relies on a corresponding change to openqa_fedora_tools to set the LOCATION openQA setting (which is simply the base location of the compose under test). Test Plan: Do a full test run, check (as far as you can) tests run sensibly and use appropriate repositories. Reviewers: jskladan, garretraziel Reviewed By: garretraziel Subscribers: tflink Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D989
2016-09-01 15:22:59 +00:00
# we use script_run here as the rawhide repo file won't always exist
# and we don't want to bother testing or predicting its existence;
# assert_script_run doesn't buy you much with sed anyway as it'll
# return 0 even if it replaced nothing
script_run "sed -i -e 's,^metalink,#metalink,g' -e 's,^#baseurl.*basearch,baseurl=${location}/Everything/\$basearch,g' /etc/yum.repos.d/{fedora,fedora-rawhide}.repo", 0;
script_run "cat /etc/yum.repos.d/{fedora,fedora-rawhide}.repo", 0;
use compose repository (not master repo) for most tests Summary: we have a long-standing problem with all the tests that hit the repositories. The tests are triggered as soon as a compose completes. At this point in time, the compose is not synced to the mirrors, where the default 'fedora' repo definition looks; the sync happens after the compose completes, and there is also a metadata sync step that must happen after *that* before any operation that uses the 'fedora' repository definition will actually use the packages from the new compose. Thus all net install tests and tests that installed packages have been effectively testing the previous compose, not the current one. We have some thoughts about how to fix this 'properly' (such that the openQA tests wouldn't have to do anything special, but their 'fedora' repository would somehow reflect the compose under test), but none of them is in place right now or likely to happen in the short term, so in the mean time this should deal with most of the issues. With this change, everything but the default_install tests for the netinst images should use the compose-under-test's Everything tree instead of the 'fedora' repository, and thus should install and test the correct packages. This relies on a corresponding change to openqa_fedora_tools to set the LOCATION openQA setting (which is simply the base location of the compose under test). Test Plan: Do a full test run, check (as far as you can) tests run sensibly and use appropriate repositories. Reviewers: jskladan, garretraziel Reviewed By: garretraziel Subscribers: tflink Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D989
2016-09-01 15:22:59 +00:00
}
1;
# vim: set sw=4 et: