now we're getting composes with the product name 'Fedora-Atomic'
this needle doesn't match them. Tweak it so it should match both
'Fedora' and 'Fedora-Atomic'.
Summary:
Set up the support server to provide DHCP/DNS functionality and
an NFS server, providing a kickstart. Add a kickstart test just
like the other root-user-crypted-net kickstart tests except it
gets the kickstart from the support server via NFS. Also add NFS
repository tests and a second support server for Server-dvd-iso
flavor: this test must run on that flavor to ensure that packages
are actually available. The support server just mounts the
attached 'DVD' and exports it via NFS.
Note we don't need to do anything clever to avoid IP conflicts
between the two support servers, because os-autoinst-openvswitch
ensures each worker group is on its own VLAN.
As part of adding the NFS repo tests, I did a bit of cleanup,
moving little things we were repeating a lot into anacondatest,
and sharing the 'check if the repo was used' logic between all
the tests (by making it into a test step that's loaded for all
of them). I also simplified the 'was repo used' checks a bit,
it seems silly to run a 'grep' command inside the VM then have
os-autoinst do a grep on the output (which is effectively what
we were doing before), instead we'll just use a single grep
within the VM, and clean up the messy quoting/escaping a bit.
Test Plan:
Run all tests - at least all repository tests - and
check they work (make sure the tests are actually still sane,
not just that they pass). I've done runs of all the repo tests
and they look good to me, but please double-check. I'm currently
re-running the whole 24-20160609.n.0 test on staging with these
changes.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D888
Summary:
this is following a SUSE model for tests where we need a server
end but don't want setting up the server to constitute a real
test in itself, we want it to be stable. The 'support_server'
test just boots a pre-built (by createhdds) disk image, sets up
networking, and runs the iSCSI server.
To run the iSCSI test we need to handle networking config in
anaconda (or we would need to set the support server up as a
DHCP server, which may be worth considering), so this adds that.
We also need to be able to specify the target device for a
volume in custom partitioning, so this adds that too.
Test Plan:
Build the necessary support server disk image (use
D883), then run the test and make sure it works. Also make sure
all other tests continue to work.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D884
Summary:
This requires a few other changes:
* turn clone_host_resolv into clone_host_file, letting you clone
any given host file (cloning /etc/hosts seems to make both
server deployment and client enrolment faster/more reliable)
* allow loading of multiple POSTINSTALL tests (so we can share
the freeipa_client_postinstall test). Note this is compatible,
existing uses will work fine
* move initial password change for the IPA test users into the
server deployment test (so the client tests don't conflict over
doing that)
* add GRUB_POSTINSTALL, for specifying boot parameters for boot of
the installed system, and make it work by tweaking _console_wait
_login (doesn't work for _graphical_wait_login yet, as I didn't
need that)
* make the static networking config for tap tests into a library
function so the tests can share it
* handle ABRT problem dirs showing up in /var/spool/abrt as well
as /var/tmp/abrt (because the enrol attempt hits #1330766 and
the crash report shows up in /var/spool/abrt, don't ask me why
the difference, I just work here)
* specify the DNS servers from the worker host's resolv.conf as
the forwarders for the FreeIPA server when deploying it; if we
don't do this, rolekit defaults to using the root servers as
forwarders(!) and thus we get the public, not phx2-appropriate,
results for e.g. mirrors.fedoraproject.org, some of which the
workers can't reach, so PackageKit package install always fails
(boy, was it fun figuring THAT mess out)
Even after all that, the test still doesn't actually pass, but
I'm reasonably confident this is because it's hitting actual bugs,
not because it's broken. It runs into #1330766 nearly every time
(I think I saw *one* time the enrolment actually succeeded), and
seems to run into a subsequent bug I hadn't seen before when
trying to work around that by trying the join again (see
https://bugzilla.redhat.com/show_bug.cgi?id=1330766#c37 ).
Test Plan:
Run the test, see what happens. If you're really lucky,
it'll actually pass. But you'll probably run into #1330766#c37,
I'm mostly posting for comment. You'll need a tap-capable openQA
instance to test this.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D880
we only 'double' the password for the user account, not for root
this is a trivial fix so just pushing it out to get cyrillic
test working at last (hopefully)
Summary:
This adds tests for the Server_cockpit_default and cockpit_basic
test cases. Some notes: I was initially thinking of combining
these into a single test with multiple test modules and coming
up with a system for doing wiki reporting based on individual
test module status, but because we'll also want to do a cockpit
FreeIPA enrol test, I decided against it. We don't really want
to combine all three because then we would skip the cockpit
tests whenever FreeIPA server deployment failed, which isn't
ideal. So since we'll need a separate FreeIPA enrolment test
anyway it doesn't really make sense to go to the trouble of
designing a system for loading multiple postinstall tests (though
I have an idea for that!) and a per-module wiki reporting system.
This was the most minimal and hopefully reliable method for
running Cockpit from a stock Server install that I could think
of. An alternative approach would be to have, say, the most
recent stable Workstation live as a 'stock' asset and have two
tests, one which runs a stock Server install and just waits and
another which boots the live image and accesses the cockpit
running on the other box, but that seems a bit over-complex. It
is not possible to have dependencies between tests for different
ISOs, in case you were wondering about having a Workstation live
test which runs parallel with a Server DVD test, we can't do
that. One funny thing is the font that winds up getting used for
the desktop, but I don't *think* that should be a problem.
Picking needles was a bit tricky; any improvement suggestions
are welcome. I'm hoping it turns out to be safe to rely on some
dbus log messages being present; I think logging into Cockpit
triggers activation of the realmd dbus interface, so there
*should* always be some messages related to that. An alternative
would just be to match on a sliver of the dark grey table header
and the light grey row beneath it and assume that'll always be
the first message (whatever the message is), but then we have to
find some area of the message details screen which is always
present for any message, and it just seems a tad more likely to
result in false passes. Similary I'm making an assumption that
auditd is always going to show up on the first page of the
Services screen and the details screen will always show that
'loaded...enabled' text.
Test Plan:
Run the tests and see if they work! See
https://openqa.stg.fedoraproject.org/tests/21373 and
https://openqa.stg.fedoraproject.org/tests/21371 for my tests.
Reviewers: garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D874
Summary:
The test here is a bit ugly, but it should work. Better ideas
welcome =)
Test Plan:
Run the test, check it works (and maybe hack it up
a bit and check it fails properly too, it worked first time for
me which is always suspicious)
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D870
Summary:
the Server DVD now just has 'Fedora Server' and 'Custom
Operating System' environments. Custom is basically minimal.
So we can use the DVD for 'universal' testing again, these
needles match the anaconda_minimal tags.
Test Plan:
Run the 'universal' tests on a DVD ISO with these
needles added, test that they work OK and use the 'Custom' env.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D844
this stuff is only appropriate when a switched keyboard layout
is in play. Left unconditional like this, it breaks the normal
encrypted install test case.
Summary:
We'll often be able to reach a root console and upload logs if
this test fails, and we want to do that so we can figure out
what's going on.
Test Plan:
Have the test fail and check that it manages to upload
logs.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D856
Summary:
Requires new needles and test suite and job template, plus a
few tweaks to handle 'switched' keyboard layouts (so we use the
switched layout in the username and password).
Test Plan:
Run the test and see that it...fails. But that's OK!
It's a genuine bug: RHBZ #1333998 . At least make sure it gets
to that point and no other tests have broken and all the needles
look sane.
Reviewers: garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D846
ARM actually doesn't have "install" test, but in install matrix,
there is test whether ARM disk boots into initial_setup. HDD is saved
after this test for Base tests.
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D853
Summary:
This requires a PR I sent upstream:
https://github.com/os-autoinst/os-autoinst/pull/490
This change is in os-autoinst -10. However, with older packages
it won't crash or anything, it'll just behave as before.
With the change, this allows log upload to fail, so if one of
the logs is missing, the hook doesn't immediately die and fail
to upload the rest of the logs. Various anaconda logs are not
always present: the DNF logs are not present for Atomic or live
installs, and the X.log and syslog are not present for live
installs. Adding a fail-tolerant mode to upstream upload_logs
seemed a better option than testing for the existence of each
log file prior to uploading it, or adding a bunch of GET_VAR
calls to try and figure out which log files 'should' exist.
Test Plan:
Run an Atomic or live install test that fails, and
check what logs get uploaded. You can just test a current
Rawhide Atomic installer ISO, as they're crashing right now.
Without this patch (and the os-autoinst update) the hook dies
when it tries to upload dnf.log, so the traceback and /var/tmp
archive don't get uploaded; with this patch all the present
logs should get uploaded. Compare:
https://openqa.fedoraproject.org/tests/14834https://openqa.stg.fedoraproject.org/tests/15371
(I tested this out on staging).
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D834
the updates-testing here was never meant to be permanent, it
was only added when the plugin was very new and we knew the
version in stable was busted. The test cases do not say to use
u-t, so let's not.
We've seen this test fail several times recently because of very
slow metadata downloads at this point, so let's give it longer
to run.
This should be a better fix for the problem of the install_done
needle matching in the sidebar gradient while the button is
transitioning. We just assert_screen first, then once we hit it,
we wait_still_screen for 3 seconds, then assert_and_click the
button.
the first param to assert_and_click is *the button number*, not
the wait time. So as soon as I set that for anything it was not
gonna work. So going back to the original approach but fixing
that, hope this works.
Summary:
I really just want to add the desktop_terminal test, but I think
this refactor is in order now. It splits up loading of the
various test phases (much as SUSE do it) and allows us to run
the post-install tests without the install tests, for e.g. I
tweaked things to allow the upgrade tests to use the existing
_wait_login tests for final login and combine the two upgrade
postinstall tests into one simple one.
This comes with a bit of a behaviour change to make graphical
wait login behave the same as console wait login: it will log
in unless USER_LOGIN is set to 'false'. Previously it only
logged in if both USER_LOGIN and USER_PASSWORD were set, which
I don't think ever happened in a graphical test, so we never
actually did a graphical login. The intent here is we should do
a login on the default_install tests. That's going a bit beyond
the test case, but it seems like a reasonable thing to test. We
can set USER_LOGIN to false if we don't want to do it.
Test Plan:
Do a full test run, make sure the new tests work and
no old tests break.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D839
Summary:
These require openQA tap networking to allow the server and
client boxes to communicate, and require masquerading (NAT) so
the server at least can reach a repository (dnf/rolekit really,
really do not want to work without a repo connection).
They use the 'parallel' test support to have the server deploy
run first while the client enrol test waits at the grub menu
until the server is done before it goes ahead.
This is all deployed and working on stg. The really tricky bit
was getting all the openvswitch and firewall config right in
ansible.
We *could* do the server deploy test as a follow-on from the
default install test to save the install, but then we'd have to
teach it to change the hostname and set up static networking
post-install. I'm not sure if it's worth doing that.
This requires the corresponding openqa_fedora_tools commit that
adds the hard disks (containing the kickstarts - it's possible
to get them from remote during install, but we have to set up
name resolution or hard code the IP of the server).
Test Plan:
Deploy this and the openqa_fedora_tools commit,
generate the disks, configure the networking (good luck! See
the docs in openqa_fedora_tools) and see if you can run the
tests. If you're using Docker, uh...sorry. You somehow need to
set things up so the workers can use tap interfaces that can
talk to each other and are NATed to the outside world. Have fun.
I can talk you through it on IRC...
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D831
Summary:
you can't use validate_script_output like this, it doesn't just
return a bool, it actually kills the test if the output doesn't
validate. So any time we didn't have anything in /var/tmp/abrt,
the post_fail_hook would die at that point and not upload the
contents of /var/log as we wanted it to.
Note that even script_output will kill the test if the return
code is >0. That's OK in this case, but always be careful with
these subroutines.
Test Plan:
Run a test that fails in post-install but doesn't
produce anything in /var/tmp/abrt (you may have to artificially
break something). Check that it now uploads /var/log. I noticed
this while working on FreeIPA tests...
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D830
Summary:
There's a weird problem that we hit occasionally only on the
Server image, see e.g.:
https://openqa.fedoraproject.org/tests/14300
I *think* what's happening (thanks hergertme from #gtk+ IRC for
help with this) is that when the 'install done!' text and the
reboot button appears, there's a short transition for the button
so it doesn't look *exactly* like the screenshot right away. If
openQA's needle match loop happens to kick in and check the
match right after the text and button appear, while the
transition is happening, it decides that some random area of the
gradient down the left hand side of the screen matches *better*
than the button, so it 'clicks' that, and we don't get a reboot.
If the openQA needle match loop happens to kick in a millisecond
or whatever later, after the button is done transitioning, the
button is now a 100% match and the bug doesn't happen. This only
happens on Server because the left-hand side gradient differs in
each flavor, and only the Server one just happens to have a spot
which is nearly identical to the needle area.
So to avoid that, let's include a bit of the grey space next to
the button. This is a bit more fragile against GTK+ changes to
button design or padding or whatever, but it will at least avoid
the match issue.
The other option would be to try and hit the button with tab and
enter or something, I guess.
Test Plan:
bit hard since this is an intermittent bug, I guess
run install_default on a Server image a bunch of times and see if
it always works.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D824
according to rdieter the runner dialog could be white or grey
background, we should match on either. One is probably a bug
but it's best to just work with it. Removing the February
version without the 'search' text.
