Commit graph

229 commits

Author SHA1 Message Date
Adam Williamson
09c264fe16 Add Workstation dvd-ostree flavor and tests
Summary:
As we're getting the Workstation dvd-ostree (OStree installer
image) built for Rawhide now, let's try testing it.

Test Plan:
Run the tests on a Rawhide compose that works and
has the image (e.g. 20170615.n.0). Check that new tests work
as expected and old tests are not adversely affected. A
corresponding diff for fedora_openqa will be coming to take
care of scheduling. Note that the tests will often soft fail
for now; this is intentional due to RHBZ#1193590, the bash
prompt for root is incorrect on ostree installs, so I have
added a needle that matches the incorrect prompt but which is
flagged as a workaround needle (so causing the test result to
be a soft fail).

Reviewers: jsedlak, jskladan

Reviewed By: jsedlak

Subscribers: tflink

Differential Revision: https://phab.qa.fedoraproject.org/D1211
2017-06-26 18:48:27 -07:00
Adam Williamson
ab085f4724 Correct name of realmd upgrade client test in job templates
Whoops.
2017-06-21 16:18:37 -07:00
Adam Williamson
a29875a2ff Upgrade tests: Run FreeIPA webUI and password change modules
Summary:
This just adds the FreeIPA web UI and password change
test modules to the FreeIPA upgrade test (client end). It's
useful to check out these features too. We don't need to
separate these into separate jobs, as we're not trying to
fill out different matrix checkboxes here, we just want to
know whether everything works.

Test Plan:
Run the test, see that the modules work properly.
I was actually expecting this to fail given the issues with
the upgrade on the server end, but it seems to pass.

Reviewers: jsedlak, jskladan

Reviewed By: jsedlak

Subscribers: tflink

Differential Revision: https://phab.qa.fedoraproject.org/D1207
2017-06-13 10:06:15 -07:00
Adam Williamson
df2c3cd906 Test upgrade of FreeIPA server and client deployment
Summary:
This adds an upgrade variant of the FreeIPA tests, with only
the simplest client enrolment (sssd) for now. The server test
starts from the N-1 release and deploys the domain controller
role. The client test similarly starts from the N-1 release
and, when the server is deployed, enrols as a domain client.
Then the server upgrades itself, while the client waits (as the
server is its name server). Then the client upgrades itself,
while the server does some self-checks. The server then waits
for the client to do its checks before decommissioning itself,
as usual. So, summary: *deployment* of both server and client
occurs on N-1, then both are upgraded, then the actual *checks*
occur on N.

In my testing, this all more or less works, except the role
decommission step fails. This failure seems to be a genuine one
so far as I can tell; I intend to file a bug for it soon.

Test Plan:
Run the new tests, check they work. Run the existing
FreeIPA tests (both the compose and the update variants), check
they both behave the same.

Reviewers: jsedlak, jskladan

Reviewed By: jsedlak

Subscribers: tflink

Differential Revision: https://phab.qa.fedoraproject.org/D1204
2017-06-02 12:17:07 -07:00
Jan Sedlák
48b99a2291 add base system logging test
Differential Revision: https://phab.qa.fedoraproject.org/D1202
2017-06-01 11:06:04 +02:00
Jan Sedlák
4114406668 add UEFI for blivet tests
Differential Revision: https://phab.qa.fedoraproject.org/D1201
2017-05-22 09:26:58 +02:00
Jan Sedlák
0b5f865c8f add custom btrfs partitioning test for blivet-gui
Differential Revision: https://phab.qa.fedoraproject.org/D1194
2017-05-19 13:58:16 +02:00
Jan Sedlák
ea5296b306 add postinstalls to custom partitioning tests
Differential Revision: https://phab.qa.fedoraproject.org/D1192
2017-05-09 09:11:34 +02:00
Jan Sedlák
140c5f0a42 Add custom partitioning tests for blivet
Differential Revision: https://phab.qa.fedoraproject.org/D1188
2017-04-24 14:23:35 +02:00
Adam Williamson
22e210aa21 Run Atomic Host installer test on UEFI as well as BIOS
Atomic team asked for this, so hey, why not.
2017-04-18 16:48:37 -07:00
Adam Williamson
1444c5030b Add tests with no user created during install
Summary:
This adds a new test suite, run for Workstation and KDE live
images, which does not create a user during install. It then
expects initial-setup (KDE) or gnome-initial-setup (Workstation)
to appear after install, creates a user, and proceeds with
normal boot.

Note the ARM image test already covers the initial-setup text
mode, and the ARM minimal image is the only case where that
actually matters (it's not included in Server).

Test Plan:
Run the new tests, check they work. Run all old
tests, check the changes didn't break them.

Reviewers: jsedlak, jskladan

Reviewed By: jsedlak

Subscribers: tflink

Differential Revision: https://phab.qa.fedoraproject.org/D1185
2017-04-05 09:43:26 -07:00
Adam Williamson
b6d4fd7d4c Don't create user when USER_LOGIN is false, but for KDE install
Summary:
For some reason, we have `USER_LOGIN` set to 'false' for the KDE
package set install test. I really don't know / remember why
that would be; I'd think we should create a user and log in as
that user to make sure it works properly when installing KDE
from the traditional installer. It's not strictly part of the
package set test, true, but still, seems worth doing.

Also, when `USER_LOGIN` is set to 'false' and the installer runs,
we create a user called 'false'. This doesn't seem like what we
wanted, so let's not do that. I dunno if there are any other
cases besides the KDE one that this commit changes, but still.

Test Plan:
Run the full test suite and look for weirdness, check
KDE package set test works as intended (now creates a user called
'test' and logs in as that user).

Reviewers: jsedlak, jskladan

Reviewed By: jsedlak

Subscribers: tflink

Differential Revision: https://phab.qa.fedoraproject.org/D1182
2017-03-29 09:30:16 -07:00
Jan Sedlák
d140e08463 test target on ARM is HDD_2
Differential Revision: https://phab.qa.fedoraproject.org/D1173
2017-03-22 10:38:12 +01:00
Adam Williamson
1925606a57 Use std not qxl graphics to workaround 2017-03-20 11:36:20 -07:00
Adam Williamson
115956f4ec Set a bus for the HDD in install_sata so it works with IDE CD
Explicitly specify the ahci0.0 bus for the HDD in install_sata.
This is needed to work if we are using CDMODEL=ide-cd (which we
need at present to work around a bug with SCSI CDs), and is a
good idea anyway to ensure the drive is actually connected to
the SATA bus (I dunno if it was before or not).
2017-03-09 13:50:19 -08:00
Adam Williamson
92d588f245 Add support for testing updates
Summary:
This adds an entirely new workflow for testing distribution
updates. The `ADVISORY` variable is introduced: when set,
`main.pm` will load an early post-install test that sets up
a repository containing the packages from the specified update,
runs `dnf -y update`, and reboots. A new templates file is
added, `templates-updates`, which adds two new flavors called
`updates-server` and `updates-workstation`, each containing
job templates for appropriate post-install tests. Scheduler is
expected to post `ADVISORY=(update ID) HDD_1=(base image)
FLAVOR=updates-(server|workstation)`, where (base image) is one
of the stable release base disk images produced by `createhdds`
and usually used for upgrade testing. This will result in the
appropriate job templates being loaded.

We rejig postinstall test loading and static network config a
bit so that this works for both the 'compose' and 'updates' test
flows: we have to ensure we bring up networking for the tap
tests before we try and install the updates, but still allow
later adjustment of the configuration. We take advantage of the
openQA feature that was added a few months back to run the same
module multiple times, so the `_advisory_update` module can
reboot after installing the updates and the modules that take
care of bootloader, encryption and login get run again. This
looks slightly wacky in the web UI, though - it doesn't show the
later runs of each module.

We also use the recently added feature to specify `+HDD_1` in
the test suites which use a disk image uploaded by an earlier
post-install test, so the test suite value will take priority
over the value POSTed by the scheduler for those tests, and we
will use the uploaded disk image (and not the clean base image
POSTed by the scheduler) for those tests.

My intent here is to enhance the scheduler, adding a consumer
which listens out for critpath updates, and runs this test flow
for each one, then reports the results to ResultsDB where Bodhi
could query and display them. We could also add a list of other
packages to have one or both sets of update tests run on it, I
guess.

Test Plan:
Try a post something like:
HDD_1=disk_f25_server_3_x86_64.img DISTRI=fedora VERSION=25
FLAVOR=updates-server ARCH=x86_64 BUILD=FEDORA-2017-376ae2b92c
ADVISORY=FEDORA-2017-376ae2b92c CURRREL=25 PREVREL=24

Pick an appropriate `ADVISORY` (ideally, one containing some
packages which might actually be involved in the tests), and
matching `FLAVOR` and `HDD_1`. The appropriate tests should run,
a repo with the update packages should be created and enabled
(and dnf update run), and the tests should work properly. Also
test a regular compose run to make sure I didn't break anything.

Reviewers: jskladan, jsedlak

Reviewed By: jsedlak

Subscribers: tflink

Differential Revision: https://phab.qa.fedoraproject.org/D1143
2017-02-22 11:33:32 -08:00
Adam Williamson
a5861ebc5d Tweak test priorities back in sync with wiki / criteria
The rule for test priorities is pretty simple. Ranges of
priority values map to the 'Milestone' by which the test must
be passing, per the release criteria. The priority for each
openQA test is the *highest* priority for any wiki test case /
criterion it covers.

0-20: critical smoke tests (higher than Alpha priority)
20-29: Alpha priority
30-39: Beta priority
40-49: Final priority
50+: Optional priority

Note that tests for non-release-blocking arches or images must
always be over 50; I've simply added 50 to the values for all
i386 tests in this change. Other than that, I just corrected a
few values which had got out of whack or were originally set
wrong.
2017-02-20 17:33:14 -08:00
Jan Sedlák
4a7a60a0a6 add TEST_TARGET to templates
Differential Revision: https://phab.qa.fedoraproject.org/D1109
2017-02-02 16:13:10 +01:00
Adam Williamson
062d9f8f5e Add jobs to gather memory usage data
Summary:
This adds a new test, memory_check, which just does a default
package set install with `inst.debug` parameter then uploads
the memory usage file (`/tmp/memory.dat`) at the end. We can
have check-compose use the data to analyze changes in memory
usage over time.

Test Plan:
Fire off the Workstation network install image tests
and make sure the memory usage test runs and works on all three
machines. This is live on staging already.

Reviewers: jskladan, garretraziel_but_actually_jsedlak_who_uses_stupid_nicknames

Reviewed By: garretraziel_but_actually_jsedlak_who_uses_stupid_nicknames

Subscribers: tflink

Differential Revision: https://phab.qa.fedoraproject.org/D1082
2017-01-16 09:30:14 -08:00
Jan Sedlák
a946b02e71 reintroduce SATA 2017-01-16 13:07:47 +01:00
Adam Williamson
c4f32ab5ad add an Asian (Japanese) language install test
Summary:
Include some basic testing of Japanese input, and split the
input testing (including Russian) into a separate module, since
it's not really part of 'login' testing.

Test Plan:
Run the test, and the Russian and French tests too to
make sure they didn't break. Tested on staging. Note the Japanese
test soft fails, intentionally, at present, as I discovered a bug
while working on it:
https://bugzilla.gnome.org/show_bug.cgi?id=776189

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D1072
2016-12-21 08:41:00 -08:00
Adam Williamson
8e70b4b8a8 switch to 'Nehalem' CPU model
Using 'host' is, for some reason, causing the problem where non-
minimal installs fail to boot. No idea why, but switching the
CPU model to Nehalem solves it.
2016-12-20 18:42:23 -08:00
Adam Williamson
0fa6138448 Have non-English tests do graphical install and login
Summary:
The non-English tests so far did not test that graphical login
worked as expected, which is a fairly large hole. With this
change, they should do a Workstation install and test login to
both GNOME and the console works as expected. KDE is not yet
tested.

As part of this we tweak the implementation of keyboard layout
switching in graphical environments to use a generic function
in main_common which can handle both anaconda and desktops
(just GNOME at present, but should extend easily to any desktop
with a known switcher key and a visible layout indicator),
replacing the anacondatest class method. I kinda don't like that
the test has to specifically tell the function when it's in
anaconda, but I don't think I want to start experimenting with
a global 'test phase' openQA variable or anything like that at
present.

Fixes T842.

Test Plan:
Run the French and Russian install tests and check
they work as expected. Also run an English Workstation install
if you like, and make sure that didn't break. This change is
live on staging ATM, seems to work fine.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Maniphest Tasks: T842

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D1071
2016-12-16 09:40:29 -08:00
Jan Sedlák
44cf1cd89c reintroduce rescue on UEFI
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D1060
2016-12-13 09:18:34 +01:00
Adam Williamson
65cadb11df Collect some data on freshly-installed systems after some tests
Summary:
I've been wanting to do this for a while; I think it'll let us
check for some significant changes between composes. This should
cause runs of a few test cases to collect and upload info on:

* installed packages
* free memory
* disk space
* active services
* 1 minute of CPU usage info (via top)

immediately after install and initial login. In some cases this
will be useful / interesting simply to look at directly, but
we can also have check-compose analyze the data and include
significant changes in its reports.

Test Plan:
Run affected tests, make sure the data collection
works.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D1046
2016-11-09 07:20:17 -08:00
Adam Williamson
7b31b8263e Force GNOME to notify updates, re-enable test on Workstation
Summary:
GNOME's update notification criteria are pretty braindead: it
fires the update check timer once on login then once every hour
thereafter, but only actually checks for and notifies of updates
once a day if it's after 6am(?!?!?!). So we have to do a bunch
of fiddling around to ensure we reliably get a notification.
Move the clock to 6am if it's earlier than that, and reset the
'last update check' timer to 48 hours ago, then log in to GNOME
after that.

Note: I thought this still wasn't fully reliable, but I've looked
into all the recent failures of either test on staging and
there's only one which was really 'no update notification came
up', and the logs clearly indicate PK did run an update check,
so I don't think that was a test bug (I think something went
wrong with the update check). The other failures are all 'GNOME
did something wacky', plus one case where the needle didn't quite
match because I think the match area is slightly too tall; I'll
fix that in a second.

Test Plan:
Run the tests on both KDE and GNOME and check they
work properly now (assuming nothing unrelated breaks, like KDE
crashing...)

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D1039
2016-10-27 16:23:59 -07:00
Adam Williamson
35568a50d7 Re-enable notifications tests for KDE
The notifications tests actually work pretty well on KDE, it's
only Workstation where they're busted. So let's keep them on
for KDE, for now at least.
2016-10-26 14:08:01 -07:00
Adam Williamson
02d4594cd2 disable notifications tests until I have time to fix them
These keep failing because they don't work right. I have it
partially fixed on a branch but not fully fixed yet, and it's
been this way for a few weeks already, so let's disable the
tests for now until I can actually complete the fix.
2016-10-26 13:52:22 -07:00
Jan Sedlák
7f93843dd6 Revert "add rescue mode test for UEFI" because it doesn't work and never would
This reverts commit 361a0ed150.
2016-10-19 10:48:14 +02:00
Jan Sedlák
ab316cfe48 move APPEND argument from scheduler to templates
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D1024
2016-10-17 12:23:06 +02:00
Jan Sedlák
361a0ed150 add rescue mode test for UEFI
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D1025
2016-10-14 10:28:05 +02:00
Adam Williamson
d9aa6b8ad6 update Atomic ostree installer image flavor name
Goes along with the commit to openqa_fedora_tools that makes
flavor generation go through fedfind's `correct_image` helper,
which changes the 'type' of ostree installer images from 'boot'
to 'dvd-ostree'. See https://pagure.io/pungi/issue/417 for
more background on this. Committing without review as it's an
urgent issue we have to fix right away.
2016-10-05 16:43:07 -07:00
Adam Williamson
e9ce14a891 consolidate login waits, use postinstall not entrypoint for base
Summary:
I started out wanting to fix an issue I noticed today where
graphical upgrade tests were failing because they didn't wait
for the graphical login screen properly; the test was sitting
at the 'full Fedora logo' state of plymouth for a long time,
so the current boot_to_login_screen's wait_still_screen was
triggered by it and the function wound up failing on the
assert_screen, because it was still some time before the real
login screen appeared.

So I tweaked the boot_to_login_screen implementation to work
slightly differently (look for a login screen match, *then* -
if we're dealing with a graphical login - wait_still_screen
to defeat the 'old GPU buffer showing login screen' problem
and assert the login screen again). But while working on it,
I figured we really should consolidate all the various places
that handle the bootloader -> login, we were doing it quite
differently in all sorts of different places. And as part of
that, I converted the base tests to use POSTINSTALL (and thus
go through the shared _wait_login tests) instead of handling
boot themselves. As part of *that*, I tweaked main.pm to not
require all POSTINSTALL tests have the _postinstall suffix on
their names, as it really doesn't make sense, and renamed the
tests.

Test Plan: Run all tests, see if they work.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D1015
2016-09-27 11:48:15 -07:00
Adam Williamson
a067d0655f add a desktop notifications test
Summary:
this more or less covers desktop_error_checks and desktop_
update_notification, though it can't really distinguish
between them easily. All we know is that if both the live and
postinstall versions of this test pass, both of those tests
pass. Any fails will have to be investigated manually.

Test Plan:
Run the tests for both KDE and Workstation, see
what happens. Workstation will fail for F25 and Rawhide at
present, due to SELinux/abrt notifications.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D1004
2016-09-23 16:03:13 -07:00
Adam Williamson
e8b20ec73f add a desktop_update_graphical test
Summary:
Very similar to the CLI update test, but using the desktops'
update applications. This is based off the CLI update test
branch as it uses the shared functions that branch introduced.
We do not use the fake update packages, as they don't really
do anything useful for these tests; for dnf they can help us
distinguish between issues with the dnf mechanism and issues
with the repos, but we can't really tell that in the graphical
case. So we only use the python3-kickstart package here.

Test Plan:
Run the test on both KDE and GNOME and ensure it
performs as intended. I've been testing it on staging, so you
can see it there.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D1010
2016-09-22 11:38:51 -07:00
Adam Williamson
44ec3d84c3 add a base_update_cli test
Summary:
this uses a couple of test repos with fake packages to test the
basic dnf mechanisms are working, then messes around with the
python3-kickstart package a bit to try and test the default repo
configuration is working, keys are in place and so on. We use
python3-kickstart because we should be able to rely on the copy
of that package in the 'stable' repo being installable (or else
the compose would have failed), but it shouldn't be vital to
the operation of the system.

Test Plan: Run the test, see if it works.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D1006
2016-09-22 10:57:12 -07:00
Jan Sedlák
c0b9bdb543 add anaconda rescue test on encrypted disk
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D995
2016-09-16 14:44:03 +02:00
Adam Williamson
1c5fffa859 add a database server role test (and client test)
Summary:
Pretty straightforward tests which deploy the database
server role and exercise it a bit.

Test Plan: Run the tests, check they work properly.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D991
2016-09-08 14:19:09 -07:00
Jan Sedlák
db95bccd52 add anaconda text UI test
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D980
2016-09-07 10:34:54 +02:00
Adam Williamson
ef689e75a9 use compose repository (not master repo) for most tests
Summary:
we have a long-standing problem with all the tests that hit
the repositories. The tests are triggered as soon as a compose
completes. At this point in time, the compose is not synced to
the mirrors, where the default 'fedora' repo definition looks;
the sync happens after the compose completes, and there is also
a metadata sync step that must happen after *that* before any
operation that uses the 'fedora' repository definition will
actually use the packages from the new compose. Thus all net
install tests and tests that installed packages have been
effectively testing the previous compose, not the current one.

We have some thoughts about how to fix this 'properly' (such
that the openQA tests wouldn't have to do anything special,
but their 'fedora' repository would somehow reflect the compose
under test), but none of them is in place right now or likely
to happen in the short term, so in the mean time this should
deal with most of the issues. With this change, everything but
the default_install tests for the netinst images should use
the compose-under-test's Everything tree instead of the 'fedora'
repository, and thus should install and test the correct
packages.

This relies on a corresponding change to openqa_fedora_tools
to set the LOCATION openQA setting (which is simply the base
location of the compose under test).

Test Plan:
Do a full test run, check (as far as you can) tests run sensibly
and use appropriate repositories.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D989
2016-09-01 08:22:59 -07:00
Jan Sedlák
0728e09c75 enable ENTRYPOINT to be more than one test, ensure shutdown when on ARM
This adds possiblity to specify more than one test in ENTRYPOINT
variable - test names should be separated with spaces and loading is done
as with POSTINSTALL.

Add _console_shutdown test to ARM image deployment.

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D973
2016-08-16 09:33:10 +02:00
Adam Williamson
962c350e91 templates: disk_ks is now version 3
Summary:
This goes with D968, which bumped the version of the disk_ks
image, so templates has to stay in sync.

Test Plan:
As per D968 (you'll need this diff as well or else
the tests will try to use the wrong disk image).

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D969
2016-08-15 17:01:18 -07:00
Adam Williamson
a8ddc002f8 add QA:Testcase_desktop_browser test
Summary:
pretty simple stuff here. The distinction between 'firefox' and
'browser' is that the 'browser' needles I expect would also be
correct for other default browsers, while the 'firefox' needles
are specific to Firefox. We need '-kde' variants of some Firefox
needles where interface text is included, because the font is
Cantarell in GNOME but whatever the default 'sans' font is in
KDE - I suppose we should really use -thatfontsname rather than
-kde, but I can't think what it's called...

I couldn't do the 'log in to FAS' bit of the test since we don't
really have a sane way to provide a password while not exposing
it to the public.

Test Plan:
Run the test, check it works - for both KDE and
Workstation.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D938
2016-08-03 13:22:29 -07:00
Adam Williamson
a901fce4ab add QA:Testcase_FreeIPA_password_change test
Summary:
again, added as a non-fatal module for realmd_join_cockpit as
it's convenient to do it here. Also abstract a couple of ipa
bits into a new exporter package in the style of SUSE's
mm_network, rather than using ill-fitting class inheritance as
we have before - we should probably convert our existing class
based stuff to work this way.

Also a few minor tweaks and clean-ups of the other tests:

The path in console_login() where we detect login of a regular
user when we want root or vice versa and log out was actually
broken because it would 'wait' for the result of the 'exit'
command, which obviously doesn't work (as it relies on running
another command afterwards, and we're no longer at a shell).
This commit no longer actually uses that path, but I spotted
the bug with an earlier version of this which did, and we may
as well keep the fix.

/var/log/lastlog is an apparently-extremely-large sparse file.
A couple of times it seemed to cause tar to run very slowly
while creating the /var/log archive for upload on failure. It's
no use for diagnosing bugs, so we may as well exclude it from
the archive.

I caught cockpit webUI login failing one time when testing the
test, so threw in a wait_still_screen before starting to type
the URL, as we have for the FreeIPA webUI.

I also caught a timing issue with the openQA webUI policy add
step; the test flips from the Users screen to the HBAC screen
then clicks the 'add' button, but there's actually an identical
'add' button on *both* screens, so it could wind up trying to
click the one on the Users screen instead, if the web UI took
a few milliseconds to switch. So we throw in a needle match to
make sure we're actually on the HBAC screen before clicking the
button.

We make the freeipa_webui test a 'milestone' so that if the
new test fails, restoring to the last-known-good milestone
doesn't take so long; it actually seems like openQA can get
confused and try to cancel the test if restoring the milestone
takes a *really* long time, and wind up with a zombie qemu
process, which isn't good. This seems to avoid that happening.

Test Plan:
In the simple case, just run all the FreeIPA-related
tests on Fedora 24 (as Rawhide is broken) and make sure they all
work properly. To get a bit more advanced you can throw in an
`assert_script_run 'false'` in either of the non-fatal tests to
break it and make sure things go properly when that happens (the
last milestone should be restored - which should be right after
freeipa_webui, sitting at tty1 - and run properly; things are
set up so each test starts with root logged in on tty1).

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D935
2016-08-03 13:21:12 -07:00
Adam Williamson
c6d8b54d58 add QA:Testcase_Anaconda_updates.img_via_installation_source
Summary:
we can test this quite easily by placing the standard openQA
updates image in the NFS repo used for the NFS repo install
tests. We just have to copy the contents of the ISO (instead of
directly exporting the ISO loop mount as an NFS share) so we
can add this extra file.

At first I planned to combine this with the NFS repo variation
test, but when you use a remote stage2 like this it changes repo
setup such that the packaging.log line we look for to verify
the remote repo was used does not show up, and there's enough
fuzziness in how anaconda-dracut fudges inst.repo and
inst.stage2 that it's probably a good idea to test them
separately anyhow.

Test Plan:
Run the new test and the other NFS tests, make sure
this one works and the others don't break.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D929
2016-08-03 13:19:18 -07:00
Adam Williamson
aacd01ea8b add encrypted workstation upgrade tests (current and previous)
Summary:
This requires us to handle decryption each time we reboot in
the upgrade process, so factor that little block out into the
base class so we don't have to keep pasting it. It's also a
bit tricky to integrate into the 'catch a boot loop' code we
have to deal with , but I think this should work. There
is a matching openqa_fedora_tools diff to generate the disk
image.

Test Plan:
Run the tests, check that they work, run the other
upgrade and encrypted install tests and check they still work
properly too.

Reviewers: garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D922
2016-07-08 08:56:57 -07:00
Adam Williamson
143fde4187 bump imgver in images now built with virt-install
Summary:
goes with the openqa_fedora_tools commit to switch from virt-
builder to virt-install. That bumps all the imgvers, so we must
update templates correspondingly.

Test Plan: As per D917.

Reviewers: garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D918
2016-07-04 09:32:27 -07:00
Adam Williamson
e24c377b01 add FreeIPA web UI testing
Summary:
as a new, non-fatal test step in the cockpit enrolment test,
because it kinda fits in there; we have an enrolled system with
a web browser *right there*. This will require making the wiki
reporting stuff slightly cleverer so we can say 'report a pass
for this wiki test instance if this test step passed', but that
should be possible. Making this non-fatal means the rest of the
cockpit enrolment test will go ahead even if the freeipa web UI
fails.

The 'check if we can log in' stuff is identical to freeipa_
client_postinstall except with different user names, so we could
potentially factor that out somehow, but I couldn't think of a
super clean way to do it so for now it's just copied.

Note this diff is on top of the freeipa-realmd branch which
is for D894, it's not on top of develop.

Test Plan:
Run the modified test and see if it works. No other
tests are modified, so they should be OK.

Reviewers: garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D895
2016-06-28 12:01:31 -07:00
Adam Williamson
5f5021530d add a realmd_join_sssd test, and use DHCP for it and cockpit
Summary:
This is a pretty straightforward IPA joining test. Since I
figured out how to set up a DHCP server for support_server,
let's do the same for the domain controller so we can simplify
these enrolment tests a bit.

We also extend the timeout on installing haveged on the server
a bit (2 minutes is a bit low when it hits a slow metadata
download), and drop an unnecessary clone_host_file from the
cockpit join test (it was only there for testing and the next
operation immediately overwrites it).

Test Plan:
Do a full server DVD test run, check the new test
works and none of the others broke.

Reviewers: garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D894
2016-06-28 12:00:13 -07:00
Adam Williamson
0da6652287 add NFS tests (and DHCP/DNS in the support server)
Summary:
Set up the support server to provide DHCP/DNS functionality and
an NFS server, providing a kickstart. Add a kickstart test just
like the other root-user-crypted-net kickstart tests except it
gets the kickstart from the support server via NFS. Also add NFS
repository tests and a second support server for Server-dvd-iso
flavor: this test must run on that flavor to ensure that packages
are actually available. The support server just mounts the
attached 'DVD' and exports it via NFS.

Note we don't need to do anything clever to avoid IP conflicts
between the two support servers, because os-autoinst-openvswitch
ensures each worker group is on its own VLAN.

As part of adding the NFS repo tests, I did a bit of cleanup,
moving little things we were repeating a lot into anacondatest,
and sharing the 'check if the repo was used' logic between all
the tests (by making it into a test step that's loaded for all
of them). I also simplified the 'was repo used' checks a bit,
it seems silly to run a 'grep' command inside the VM then have
os-autoinst do a grep on the output (which is effectively what
we were doing before), instead we'll just use a single grep
within the VM, and clean up the messy quoting/escaping a bit.

Test Plan:
Run all tests - at least all repository tests - and
check they work (make sure the tests are actually still sane,
not just that they pass). I've done runs of all the repo tests
and they look good to me, but please double-check. I'm currently
re-running the whole 24-20160609.n.0 test on staging with these
changes.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D888
2016-06-13 08:42:30 -07:00
Adam Williamson
7a8ae3a357 add an iscsi test, and a support_server test to support it
Summary:
this is following a SUSE model for tests where we need a server
end but don't want setting up the server to constitute a real
test in itself, we want it to be stable. The 'support_server'
test just boots a pre-built (by createhdds) disk image, sets up
networking, and runs the iSCSI server.

To run the iSCSI test we need to handle networking config in
anaconda (or we would need to set the support server up as a
DHCP server, which may be worth considering), so this adds that.
We also need to be able to specify the target device for a
volume in custom partitioning, so this adds that too.

Test Plan:
Build the necessary support server disk image (use
D883), then run the test and make sure it works. Also make sure
all other tests continue to work.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D884
2016-06-09 08:43:46 -07:00
Adam Williamson
66fc3cc7d4 add a cockpit realmd FreeIPA join test
Summary:
This requires a few other changes:

* turn clone_host_resolv into clone_host_file, letting you clone
  any given host file (cloning /etc/hosts seems to make both
  server deployment and client enrolment faster/more reliable)
* allow loading of multiple POSTINSTALL tests (so we can share
  the freeipa_client_postinstall test). Note this is compatible,
  existing uses will work fine
* move initial password change for the IPA test users into the
  server deployment test (so the client tests don't conflict over
  doing that)
* add GRUB_POSTINSTALL, for specifying boot parameters for boot of
  the installed system, and make it work by tweaking _console_wait
  _login (doesn't work for _graphical_wait_login yet, as I didn't
  need that)
* make the static networking config for tap tests into a library
  function so the tests can share it
* handle ABRT problem dirs showing up in /var/spool/abrt as well
  as /var/tmp/abrt (because the enrol attempt hits  and
  the crash report shows up in /var/spool/abrt, don't ask me why
  the difference, I just work here)
* specify the DNS servers from the worker host's resolv.conf as
  the forwarders for the FreeIPA server when deploying it; if we
  don't do this, rolekit defaults to using the root servers as
  forwarders(!) and thus we get the public, not phx2-appropriate,
  results for e.g. mirrors.fedoraproject.org, some of which the
  workers can't reach, so PackageKit package install always fails
  (boy, was it fun figuring THAT mess out)

Even after all that, the test still doesn't actually pass, but
I'm reasonably confident this is because it's hitting actual bugs,
not because it's broken. It runs into  nearly every time
(I think I saw *one* time the enrolment actually succeeded), and
seems to run into a subsequent bug I hadn't seen before when
trying to work around that by trying the join again (see
https://bugzilla.redhat.com/show_bug.cgi?id=1330766#c37 ).

Test Plan:
Run the test, see what happens. If you're really lucky,
it'll actually pass. But you'll probably run into #1330766#c37,
I'm mostly posting for comment. You'll need a tap-capable openQA
instance to test this.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D880
2016-06-07 13:00:39 -07:00
Adam Williamson
aba2814611 add cockpit_default and cockpit_basic tests
Summary:
This adds tests for the Server_cockpit_default and cockpit_basic
test cases. Some notes: I was initially thinking of combining
these into a single test with multiple test modules and coming
up with a system for doing wiki reporting based on individual
test module status, but because we'll also want to do a cockpit
FreeIPA enrol test, I decided against it. We don't really want
to combine all three because then we would skip the cockpit
tests whenever FreeIPA server deployment failed, which isn't
ideal. So since we'll need a separate FreeIPA enrolment test
anyway it doesn't really make sense to go to the trouble of
designing a system for loading multiple postinstall tests (though
I have an idea for that!) and a per-module wiki reporting system.

This was the most minimal and hopefully reliable method for
running Cockpit from a stock Server install that I could think
of. An alternative approach would be to have, say, the most
recent stable Workstation live as a 'stock' asset and have two
tests, one which runs a stock Server install and just waits and
another which boots the live image and accesses the cockpit
running on the other box, but that seems a bit over-complex. It
is not possible to have dependencies between tests for different
ISOs, in case you were wondering about having a Workstation live
test which runs parallel with a Server DVD test, we can't do
that. One funny thing is the font that winds up getting used for
the desktop, but I don't *think* that should be a problem.

Picking needles was a bit tricky; any improvement suggestions
are welcome. I'm hoping it turns out to be safe to rely on some
dbus log messages being present; I think logging into Cockpit
triggers activation of the realmd dbus interface, so there
*should* always be some messages related to that. An alternative
would just be to match on a sliver of the dark grey table header
and the light grey row beneath it and assume that'll always be
the first message (whatever the message is), but then we have to
find some area of the message details screen which is always
present for any message, and it just seems a tad more likely to
result in false passes. Similary I'm making an assumption that
auditd is always going to show up on the first page of the
Services screen and the details screen will always show that
'loaded...enabled' text.

Test Plan:
Run the tests and see if they work! See
https://openqa.stg.fedoraproject.org/tests/21373 and
https://openqa.stg.fedoraproject.org/tests/21371 for my tests.

Reviewers: garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D874
2016-06-01 09:05:33 -07:00
Adam Williamson
cf543adf72 add a server default firewall test
Summary:
The test here is a bit ugly, but it should work. Better ideas
welcome =)

Test Plan:
Run the test, check it works (and maybe hack it up
a bit and check it fails properly too, it worked first time for
me which is always suspicious)

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D870
2016-05-31 14:24:34 -07:00
Jan Sedlák
802c9298fd add base_services_start test for ARM
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D861
2016-05-27 12:36:15 +02:00
Adam Williamson
a1d240b135 add needles to treat 'Custom Operating System' as minimal
Summary:
the Server DVD now just has 'Fedora Server' and 'Custom
Operating System' environments. Custom is basically minimal.
So we can use the DVD for 'universal' testing again, these
needles match the anaconda_minimal tags.

Test Plan:
Run the 'universal' tests on a DVD ISO with these
needles added, test that they work OK and use the 'Custom' env.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D844
2016-05-20 12:56:44 -07:00
Adam Williamson
a4f3267534 add Russian install test
Summary:
Requires new needles and test suite and job template, plus a
few tweaks to handle 'switched' keyboard layouts (so we use the
switched layout in the username and password).

Test Plan:
Run the test and see that it...fails. But that's OK!
It's a genuine bug: RHBZ  . At least make sure it gets
to that point and no other tests have broken and all the needles
look sane.

Reviewers: garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D846
2016-05-20 07:52:55 -07:00
Jan Sedlák
dfc58f1b73 add ARM initial-setup test
ARM actually doesn't have "install" test, but in install matrix,
there is test whether ARM disk boots into initial_setup. HDD is saved
after this test for Base tests.

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D853
2016-05-18 14:04:45 +02:00
Adam Williamson
2df55efb49 add desktop_terminal test, refactor test loading a bit
Summary:
I really just want to add the desktop_terminal test, but I think
this refactor is in order now. It splits up loading of the
various test phases (much as SUSE do it) and allows us to run
the post-install tests without the install tests, for e.g. I
tweaked things to allow the upgrade tests to use the existing
_wait_login tests for final login and combine the two upgrade
postinstall tests into one simple one.

This comes with a bit of a behaviour change to make graphical
wait login behave the same as console wait login: it will log
in unless USER_LOGIN is set to 'false'. Previously it only
logged in if both USER_LOGIN and USER_PASSWORD were set, which
I don't think ever happened in a graphical test, so we never
actually did a graphical login. The intent here is we should do
a login on the default_install tests. That's going a bit beyond
the test case, but it seems like a reasonable thing to test. We
can set USER_LOGIN to false if we don't want to do it.

Test Plan:
Do a full test run, make sure the new tests work and
no old tests break.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D839
2016-05-05 16:39:47 -07:00
Adam Williamson
e5d169b67c set worker_class 'tap' for IPA tests
this will make them only-run on the tap-capable hosts in infra.
2016-05-05 14:42:16 -07:00
Adam Williamson
f59343403a add FreeIPA server role deploy and kickstart enrolment tests
Summary:
These require openQA tap networking to allow the server and
client boxes to communicate, and require masquerading (NAT) so
the server at least can reach a repository (dnf/rolekit really,
really do not want to work without a repo connection).

They use the 'parallel' test support to have the server deploy
run first while the client enrol test waits at the grub menu
until the server is done before it goes ahead.

This is all deployed and working on stg. The really tricky bit
was getting all the openvswitch and firewall config right in
ansible.

We *could* do the server deploy test as a follow-on from the
default install test to save the install, but then we'd have to
teach it to change the hostname and set up static networking
post-install. I'm not sure if it's worth doing that.

This requires the corresponding openqa_fedora_tools commit that
adds the hard disks (containing the kickstarts - it's possible
to get them from remote during install, but we have to set up
name resolution or hard code the IP of the server).

Test Plan:
Deploy this and the openqa_fedora_tools commit,
generate the disks, configure the networking (good luck! See
the docs in openqa_fedora_tools) and see if you can run the
tests. If you're using Docker, uh...sorry. You somehow need to
set things up so the workers can use tap interfaces that can
talk to each other and are NATed to the outside world. Have fun.
I can talk you through it on IRC...

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D831
2016-05-04 11:53:11 -07:00
Jan Sedlák
5afc97d0bf add kde and server upgrade tests to templates
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D822
2016-04-25 10:00:34 +02:00
Adam Williamson
df195a7853 rename BOOT_UPDATES_IMG_URL to TEST_UPDATES, add GRUBADD
Summary:
BOOT_UPDATES_IMG_URL is a pretty misleading name - it used to
be the actual URL, but now it's simply a boolean that decides
whether we look for the effect of the openQA updates image or
not. TEST_UPDATES seems clearer.

GRUBADD does the same thing as GRUB, on top of it. The point of
this is so we can add an option to the scheduler CLI that lets
you say 'run the normal tests, but with this updates image' -
so we can easily (albeit manually triggered) check the impact
of some anaconda change that needs testing. It should never be
set in the templates or the tests, it's there strictly for the
scheduler (whether that's fedora_openqa_schedule or literally a
person calling `client isos post`) to use as a kind of override.
The tests that test updates image loading will probably fail
when doing this, but all other tests should work as intended,
including ones that specify GRUB, becase the extra params will
just get added on top. That's why I invented a new var instead
of just letting the scheduler override GRUB's value when POST
ing.

Test Plan:
Check the rename didn't break anything (updates tests
still work). Run tests with GRUBADD param, make sure value is
correctly appended to cmdline both when GRUB is also specified
and when it is not.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D801
2016-04-08 13:21:29 -07:00
Adam Williamson
591b153238 revise updates.img test to work with new anaconda (T759)
Summary:
per details in T759, the 'unipony' updates image we use to test
the updates image features doesn't work with latest anaconda (f24
and Rawhide). I've built a new updates image which uses a neat
anaconda feature that allows you to override CSS with a file in
a special location; it sets the background for disk capacity
texts on the INSTALLATION DESTINATION spoke to be pink. This
lets us use a simple needle that just looks for a pink blob on
that spoke, on the basis that it's unlikely there'll ever be a
pink blob there for any other reason, so if there is one, the
updates image worked. There will be an accompanying tools diff
to change the updates disk image to use the new updates image.

Test Plan:
Do a test run and check the updates image tests pass
and no other tests are broken. You'll need to pull in the tools
diff and re-generate the updates disk image to check that test,
the scsi_updates_img test should work with just this diff.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D799
2016-04-01 08:00:47 -07:00
Adam Williamson
10783f7280 split install_default into upload/no-upload variants
Summary:
this should avoid unnecessary disk uploads and hopefully help
further reduce the incidence of weird failures in the chained
tests.

With this change we should only upload disk images for the cases
where we're actually going to run the chained tests: we won't
upload disk images for default_install runs on images we don't
run the chained tests for, or for the UEFI job for images we
*do* run the chained tests for.

We only actually need to run the current chained tests
for Server DVD, Workstation live and KDE live x86_64; there's
no need to run them for Everything boot, so we drop that.

Test Plan:
Do a full test run and make sure all tests run
properly and we now only upload disk images where we really need
to.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D794
2016-03-24 13:38:46 -07:00
Adam Williamson
3e435182dd add firewall kickstart tests (disabled and configured)
Summary:
these together test QA:Testcase_kickstart_firewall from the
Server matrix. I'll have to come up with some kinda way to
handle reporting that, might be tricky.

Couple of tweaks to overall test flow: tests can now specify
a POSTINSTALL variable which will load a post-install test
following a naming convention, and tests can specify USER_LOGIN
as 'false' to disable the 'log in as a user' step entirely. We
could easily adjust the kickstarts to create a user so the test
could log in as one, but it seems like an unnecessary step and
I liked the idea of allowing the user login to be skipped.

Test Plan:
Schedule 'universal' tests, check the new tests run
and pass or fail as they should, check no other test is broken
by the logic flow changes.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D792
2016-03-23 13:52:00 -07:00
Adam Williamson
ca60b9abb8 adjust openQA test suite names (install_foo not server_foo)
Summary:
We named a bunch of the	tests 'server_foo' back	when we	were
starting out and didn't	really know what we were doing.	They're
all really installation	tests, not 'server' tests. I actually
want to	start adding some Server tests now, so it seems	like a
good time to fix that mess. This standardizes on 'install_' as
a prefix for installation tests, converting all	the 'server_'
tests and fixing up a couple of	odd cases to use 'install_'.

Test Plan:
Apply along with the matching commit for tools, do
a full test run and report test, and make sure all tests run
with the new names and correct ResTups are generated for wiki
submission.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D790
2016-03-23 13:46:48 -07:00
Adam Williamson
4b73585a74 pungi4: Atomic variant name is gonna be Atomic 2016-02-23 12:36:02 -08:00
Adam Williamson
35735f21cd Pungi 4 conversion: handle Pungi-derived BUILD and FLAVOR
With the arrival of Pungi 4, the scheduler is no longer using
fedfind-provided BUILD and FLAVOR values, but ones derived from
Pungi properties. BUILD is now simply the Pungi compose_id.
FLAVOR is produced by joining the Pungi variant, type, and
format with '-' characters as the separators.

Pungi, unfortunately, does not treat 'Rawhide' as a release, it
synthesizes a release number for Rawhide composes and places
that in the compose ID. To cope with that, for now, the
scheduler will set RAWHIDE to '1' if the compose is a Rawhide
one. As we have to adapt all places where we parse the release
in any case, this commit consolidates them into a fedorabase
subroutine.

For the one place where we also used to parse the 'milestone'
from fedfind, there is a placeholder get_milestone subroutine
which currently returns an empty string, as I don't yet have a
good handle on how to draw the kinds of distinctions fedfind
mapped to 'milestone' from Pungi metadata.
2016-02-23 11:08:45 -08:00
Adam Williamson
936725c284 drop 'SATA' part of sata_multi - it doesn't work (T691)
Summary:
per T691, this has never actually tested SATA. With os-autoinst
4.2 it actually tests PATA, with os-autoinst 4.3 it breaks.
We can't find a way to make it do what we want with os-autoinst
4.3, so we're dropping the HDDMODEL bit entirely for now. We
will file a ticket upstream to see if this can be solved. We
keep the test itself because it's also the only test that hits
'guided_multi'; I'll post a matching diff for tools to change
the wiki reporting config to match.

Test Plan:
Schedule tests with os-autoinst 4.3 and see if this
one runs properly now.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D730
2016-01-22 12:23:51 -08:00
Adam Williamson
859a92a9c0 schedule base tests for kde_live
Summary:
we have a KDE column for the 'base' tests, so we should run
them on the kde_live flavor.

Test Plan:
Schedule a full test run, ensure all three base
tests are scheduled and run correctly for KDE live.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D720
2016-01-14 10:38:14 -08:00
Adam Williamson
63e03ecbdf add base_service_manipulation test
Summary:
Not much to say, pretty much just implements the test case using
some commands I dug up that give us handy 0/1 exit statuses.
The assert_script_run function (from testapi) simply runs a
command/script and passes or fails based on the exit status;
we use a handy bash-ism when we *want* the exit status to be 1.

Test Plan: Run the test and check that it passes (properly).

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D713
2016-01-11 12:30:24 -08:00
Adam Williamson
feac11d4d2 drop mistakenly added service manipulation test
I messed up the last commit and mistakenly included this test,
which I've been working on and is not yet reviewed. The commit
was only supposed to add base_services_start. I'll send a new
diff for service_manipulation.
2016-01-08 15:03:18 -08:00
Adam Williamson
242d2ca165 add a base_services_start test
Summary:
pretty simple, just make sure no services failed to start. We
may run into the rngd issue here, not sure, let's land it and
see!

Test Plan:
I guess run the test and see what happens? I haven't
actually tested this myself yet, so, yeah.

Reviewers: garretraziel, jskladan

Reviewed By: garretraziel, jskladan

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D710
2016-01-08 09:01:33 -08:00
Adam Williamson
7a64700eb2 use variables for release numbers in upgrade tests
Summary:
Along with the matching change to fedora-openqa-schedule to pass
these variables in when scheduling jobs, this avoids hardcoding
the release numbers for the upgrade tests (which means someone
has to remember to edit them every release). The new createhdds
similarly uses get_current_release() to decide what releases it
needs images for, so all this should hook up and work magically
without any human intervention required.

For clarity, the effect of the '_upgrade_' tests is "run an
upgrade from the 'current' Fedora release to whatever release
is being tested", and the effect of the '_upgrade_2_' tests is
"run an upgrade from the 'previous' Fedora release to whatever
release is being tested".

Test Plan:
Apply with D702, schedule upgrade tests, and make
sure the correct hard disk image filenames are used.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D703
2015-12-24 12:45:37 -08:00
Jan Sedlák
0d710e3ee9 add missing default_install tests to templates
This adds missing default_install tests to templates - we've missed
KDE live and Server DVD on UEFI for some reason and Workstation netinst was
missing. After this, we should have all non-optional tests from "Default
boot and install" section covered.

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D693
2015-12-22 09:23:25 +01:00
Adam Williamson
85c57b47e4 add a base_selinux test (follow-on from default_install)
Summary:
so here's our first attempt to use the 'carry on from a previous
test' stuff! This adds a base_selinux test that uses a disk
image from a previous default_install run, and adds jobtemplates
to run base_selinux for appropriate products: generic_boot
(for nightly tests), server_dvd, and workstation_live. Note that
you'll want to either update to the newest openQA package I just
built in COPR or create /var/lib/openqa/share/factory/tmp owned
by geekotest; openQA tries to use that directory as MOJO_TMPDIR
but in 4.2, if the directory doesn't exist, it doesn't create it,
and we wind up with the default MOJO_TMPDIR which is /tmp; when
the disk image is uploaded it creates a huge temp file in /tmp
and may well exhaust the available space as it's a tmpfs. I've
backported a recent upstream commit that tries to create the
directory if it doesn't exist, in 4.2-10.

It seems like openQA is smart enough to figure out the
dependencies correctly, so the 'base_selinux' test for each
product depends on the 'default_install' test for the same
product (not any of the other default_install runs) and will
use the hard disk image it produces.

Test Plan:
Do a full test run and make sure base_selinux tests
appear for appropriate products, depend on the correct default_
install test, the default_install test uploads the hard disk
image correctly, and the base_selinux test runs correctly. And
of course that nothing else broke in the process...

Reviewers: jskladan, garretraziel

Reviewed By: jskladan

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D699
2015-12-17 12:46:14 -08:00
Adam Williamson
76756a9005 templates: drop 'variables', move 'prio' to JobTemplates
Summary:
this is following a couple of upstream changes I noticeed while
playing with dump_templates. Upstream has completely got rid of
the 'variables' keys from all dicts, and when you run dump_
templates the output does not contain any 'variables' keys, so
it seems reasonable to ditch these entirely (I think it was some
old thing that got subsumed into 'settings'). Also, when you
run dump_templates the 'prio' values come out in JobTemplates,
not in TestSuites; if you look at load_templates there's a bit
marked "we have to migrate the prio from the TestSuite to the
JobTemplate" which does exactly what it sounds like, so it
seems a good idea to move these instead of relying on that to
always be there.

While I had to re-do all the priorities anyway I tried to clean
them up a bit. The idea is roughly this:

10 - 19: most-critical sanity tests (i.e. default_install)
20 - 29: Alpha tests
30 - 39: Beta tests
40 - 49: Final tests
50+    : Optional tests

within each group I ordered 64-bit first, UEFI second, and
32-bit last (usually just as x0 / x1 / x2).

Test Plan:
Check that the file is valid and loads correctly,
and that everything works more or less the same except the
order of tests run is a bit different.

Reviewers: jskladan, garretraziel

Reviewed By: jskladan, garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D692
2015-12-16 13:03:41 -08:00
Jan Sedlák
fd2ed44162 add KDE package set testcase
add KDE package set testcase - this only adds needles and
new testcase configuration, because with needles cleanup from D670,
it should work without change in code.

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D673
2015-12-08 11:49:38 +01:00
Jan Sedlák
6e801bcd52 remove ENV-DESKTOP- tag, set DESKTOP- tag instead where necessary
Let's use the same principle with DESKTOP tag as we use with
LANGUAGE tag. Where there are any GNOME-only (or KDE-only) needles, tag
them with `DESKTOP-gnome` and delete then when different `DESKTOP` value
is specified. This will help with `QA:Testcase_Package_Sets_KDE_Package_Install`
testcase - we will be able to use almost the same code and check that KDE
got installed properly.

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D670
2015-12-08 11:32:48 +01:00
Adam Williamson
7b085aff3a god DAMN it, perl
stupid commas.
2015-12-07 10:12:06 -08:00
Adam Williamson
9402860ec8 tweak upgrade tests again
damnit, that was still wrong. This should do the trick: we can
do 32-bit F23 upgrade tests now (virt-builder added the base
image we need), so re-add that, and put things in the best
order and fix the priorities.
2015-12-07 10:10:08 -08:00
Adam Williamson
cc963263e9 tweak upgrade tests a bit
Noticed some problems with the commit while testing it: we
weren't doing 32-bit minimal (to avoid having too many tests,
I think) so let's not do it for upgrade_2 for now, and also
the arch on one test was wrong.
2015-12-07 10:02:45 -08:00
Adam Williamson
4eb09e6cfb add f23 upgrade tests, rename f22 upgrade tests
Summary:
This bumps the existing upgrade tests to F23, and drops the
32-bit ones for now, as there is no 32-bit F23 base image for
virt-builder - RHBZ . It then adds new tests named
'upgrade_2_(etc)' and associates them with the F22 images. The
intent is that we should always have two sets of upgrade tests,
one for each of the currently-supported stable releases; when
we bump to testing F25, the 'upgrade' tests will be bumped to
F24 and the 'upgrade_2' tests to F23, and so on. There will be
a matching diff for openqa_fedora_tools.

Test Plan:
Execute a test run and make sure all the upgrade
tests run; of course you need to make sure you've built the
required disk images.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D679
2015-12-07 09:42:55 -08:00
Jan Sedlák
7240ce774f add custom partitioning xfs tests
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D663
2015-11-26 13:50:45 +01:00
Jan Sedlák
29eb92752b change MACHINE variables to have more compatible VMs, remove 32bit machine
As we saw with F23 testing, qemu32 isn't really supported CPU. Also,
we cannot be sure that `std` is really supported graphics. This changes MACHINE
variables to use `host` CPU with 64bit machines (moreover, this is the case in
BOS now). It also deletes 32bit machine and schedules 32bit tests on 64bit
instead. It also changes graphics to `qxl`. Even though we aren't using SPICE, qxl has
better support (and higher priority) and it seems to work OK with VNC. Fixes T637.

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D655
2015-11-25 12:37:55 +01:00
Jan Sedlák
8e2bcbdca1 add CDMODEL=scsi-cd when iSCSI is used
There is a bug (it is fixed in newest unstable version though) in openQA
4.2-1.2 that when you use iSCI HDD device, it fails to boot. Workaround is to
set CDMODEL=scsi-cd (and it won't hurt even when this gets fixed in stable).

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D642
2015-11-09 09:50:24 +01:00
Petr Schindler
079efe72e8 Adds variable PART_TABLE_TYPE to machines
PART_TABLE_TYPE variable says which type of partition table type
should be on attached HDDs.
Some tests with uefi have to use disks with gpt.
Tests are amended to use right disks.

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D623
2015-10-21 09:36:40 +02:00
Adam Williamson
800870dd0d schedule i686 and x86_64 default_install tests for server DVD
Summary: simple enough. scheduler should already have the necessary bits.

Test Plan:
Kick off a run and see if we get tests, and the results are
reported to the wiki (Final TC1 should work).

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D607
2015-10-06 15:27:27 -07:00
Adam Williamson
d4ddfe167a add a default_install test for cloud atomic installer image
Summary:
We have these 'atomic installer' images (so far just Cloud),
and maxamillion wanted to get them tested. Turns out it's
pretty trivial - they look much like other installs. Only
little wrinkle is they have a reduced hub (no repository
needles) like live images, but are not like live images in
any other way, so I rejigged the 'small hub needle filtering'
handling a bit.

There will be an accompanying diff for tools, and also some
changes in fedfind (these images are getting built nightly
for *current stable*, and it'd be good to test those).

Because we'd like to test the 22 nightlies, I had to add some
needles for 'olddpi' versions of a few screens. See 2e4c1c2 -
the 22 Atomic installer images still have the old GTK+ code
meaning they run at 96.09dpi. I only retook the necessary
needles for the default-install test, if we add any others we
made need to retake a few more needles.

Test Plan:
Schedule jobs for a compose with the atomic installer
image. You will need the matching openqa_fedora_tools diff and
the very latest git fedfind. Check the test for that image runs,
all other tests run as usual, excessive images are not
downloaded, and the atomic installer is not used for running
universal tests.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D595
2015-09-29 11:36:11 -07:00
Adam Williamson
fd177dd39e strip 32-bit test set a bit
Summary:
So test runs are getting very long on BOS, and we have UEFI
tests coming. Try to help out by reducing the 32-bit test load
a bit. I tried to strategically drop the tests that are least
likely to differ, e.g. different storage layouts (but not
filesystem/device types), kickstart delivery, and only doing
one upgrade test.

Test Plan:
Check the templates file loads and there are no
obvious errors. See if you agree with the tests I cut.

Reviewers: garretraziel, jskladan

Reviewed By: jskladan

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D580
2015-09-15 16:04:30 -07:00
Petr Schindler
338b4bf513 Adds uefi support to tests where it makes sense
What changed:
* There is a new needle for uefi bootloader.
* UEFI postinstallation phase
* UEFI tests and machine added to template
2015-09-15 11:04:01 +02:00
Adam Williamson
5d650b070d correct 32-bit kde live test entry in templates 2015-09-14 18:22:05 -07:00
Adam Williamson
b3aa968575 add a french (encrypted) test
Summary:
this handles Non-English European Language Install. Basically
it's a bunch of new screenshots for existing tag names, plus
a bit of configurability in _boot_to_anaconda and tweaking some
existing needles to do non-text matches. The weird 'half-the-
icon' needles are for cases where there may or may not be a
warning triangle but we want to click it either way (saves
duplicating the needle).

This also sets up a convention for tagging what languages a
needle is appropriate for. If it's specifically appropriate for
one or more languages, a tag ENV-LANGUAGE-(LANGUAGE) should be
applied for each language, where (LANGUAGE) is the install
language in upper-case ('LANGUAGE' variable, which should also
be the string that will be typed into the language selection
screen). If the needle ought to be used for *all* languages -
i.e. it's not a text match, or any text in the match is known
not to be translated - the tag ENV-INSTLANG-ALL should be
applied.

To back this, main.pm now unregisters all needles that are not
tagged with either ENV-LANGUAGE-ALL or the tag for the language
actually being used (if the LANGUAGE var is not set, we assume
english). The point of this is to check the install is actually
translated; if we allow all needles to match, the test would
pass even if no translations appeared at all.

Test Plan:
Run all tests and make sure you get the expected
results. You can schedule a run against 23 Beta TC1 to see the
French test fails 'correctly' when translations are missing.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D577
2015-09-14 18:08:58 -07:00
Jan Sedlák
7017486d43 add KDE live default install test
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D576
2015-09-14 08:52:37 +02:00
Adam Williamson
68acecb6d4 convert upgrade tests to dnf-plugin-system-upgrade
Summary:
This is a first cut which more or less works for now. Issues:

1) We're not really testing the BUILD, here. All the test does
is try and upgrade to the specified VERSION - so it'll be using
the latest 'stable' for the given VERSION at the time the test
runs. This isn't really that terrible, but especially for TC/RC
validation, we might want to make things a bit more elaborate
and set up the repo for the actual BUILD (and disable the main
repos).

2) We'd actually need --nogpgcheck for non-Rawhide, at one
specific point in the release cycle - after Branching but
before Bodhi activation (which is when we can be sure all
packages are signed). This won't matter until 24 branches, and
maybe releng will have it fixed by then...if not, I'll tweak
it.

3) We don't really test that the upgrade actually *happened*
for desktop, at the moment - the only thing in the old test
that really checked that was where we checked for the fedup
boot menu entry, but that has no analog in dnf. What we should
probably do is check that GUI login works, then switch to a
console and check /etc/fedora-release just as the minimal test
does.

Test Plan:
Run the tests. Note that creating the desktop disk
image doesn't work ATM, so I can't verify the desktop test
works, but the minimal one seems to (with D565). There'll be
a matching diff for openqa_fedora_tools to update the test
case names there.

Reviewers: jskladan, garretraziel

Reviewed By: jskladan, garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D567
2015-09-10 14:49:13 -07:00
Jan Sedlák
f8f242b7e0 add guided shrink test
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D561
2015-09-08 15:54:22 +02:00
Jan Sedlák
45ec69e504 separate 32bit and 64bit upgrade tests
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D535
2015-08-27 13:49:09 +02:00
Adam Williamson
95bc2657fb add a 'no swap partition' test
Summary:
since we did this live at Flock today, I figured I'd tidy it
up and submit it. This is an 'optional' test, but some people
do run this way so it'd be nice to have it. This adds another
little helper method in anacondatest.pm, for deleting partitions,
which works much like the others added in previous commits.

Test Plan: Schedule a test run, see if the test runs and works.

Reviewers: jskladan, garretraziel

Reviewed By: jskladan, garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D503
2015-08-19 17:41:41 -04:00
Jan Sedlák
3f8f37a348 use disk_f22 instead disk_f21 in templates
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D502
2015-08-17 17:40:06 +02:00
Jan Sedlák
f71faaa43d add updates.img via local media test
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D496
2015-08-17 17:36:40 +02:00
Adam Williamson
5d0afd612b add 32-bit tests to templates
Summary:
This duplicates basically the entire test suite for 32-bit. We
could choose to run only a few tests for 32-bit if we wanted,
but I figure we may as well do as much testing as we can. Only
a few of the results will actually wind up 'counting' separately
in the wiki, but we can always see the results for all the tests
in openQA itself.

Next we can duplicate the whole set again for UEFI!

Test Plan:
Schedule a full test run and make sure all the tests
run for both arches. Also check result submission works
correctly. Requires the corresponding change to tools (or
you can just use one of the trigger commands which lets you
specify arch with a parameter).

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D500
2015-08-15 21:10:14 -04:00
Adam Williamson
8fde224dc5 add ext3, btrfs and LVM thin tests, tweaks to custom methods
Summary:
This adds three new custom storage tests and some needles to
support them, and tweaks the custom storage methods a bit to
address some things that cropped up in writing the tests. A
new method is added for changing the filesystem, as that's
a distinct operation from changing the device type.

This also restores the previous behaviour of select_disks()
where it handled selecting custom partitioning when needed.
Turns out it's pretty common to use regex'es in perl! Who'd'a
thought.

A corresponding commit to add the tests to openqa_fedora_tools
is coming.

There's no post-install step for the tests yet; I'll try and
write those up and add them soon.

Test Plan:
Do a full run, including the new tests, on Alpha RC2 and check
all are scheduled correctly and run correctly. The LVM thinp
test is expected to fail as it catches a genuine bug.

Reviewers: jskladan, garretraziel

Reviewed By: garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D490
2015-08-10 11:01:12 -07:00
Jan Sedlák
335ce280f3 merge DISK_GUIDED and DISK_CUSTOM into PARTITIONING
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D486
2015-08-06 11:02:18 +02:00
Adam Williamson
248b7a9536 revise storage: better test loading, shared disk selection
Summary:
This contains several tweaks to storage handling. It adds a
method for disk selection which all the storage tests can
share. It sets up a more extensible approach for main.pm to
run the storage tests, instead of an ever-growing forest of
'else' clauses. Finally it sets up a couple of methods for
changing partitioning schemes on the custom part screen and
uses one of them in the software RAID test; the other will
be used for other custom storage tests.

This kills the two_disks needle. I could keep it and work
it into select_disks, but it doesn't fit naturally and I
really just don't see the point of the needle. The only thing
we lose is we don't check that anaconda actually sees two
disks in the 'attach two disks, only install to one' test
(that's server_sata_multi), but the other multi-disk tests
will serve to catch that case failing for some reason.

What I actually intended to do was add some more tests for
different custom part storage types, but it seemed a good
idea to do some of this cleanup so that can be implemented
efficiently. I'll have followups for that.

Test Plan:
Run all tests and ensure they work exactly as
before (not just that they still pass, but that the correct
test steps are actually scheduled in each case.)

Reviewers: garretraziel, jskladan

Reviewed By: garretraziel, jskladan

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D475
2015-07-31 01:31:27 -07:00
Garret Raziel
c8f02969f2 add fedup desktop test
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D451
2015-07-17 12:52:56 +02:00
Garret Raziel
1422d2c0e2 Add fedup_minimal test
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D358
2015-05-13 13:03:23 +02:00
Adam Williamson
eed0deb6c8 correct job templates added after 'live' merge
The flavor names changed and we started using the * version
wild card in the 'live' merge, so we need to correct the jobs
that were added to master in the meantime.
2015-03-19 14:11:17 -07:00
Adam Williamson
84a14cdb67 Support for live image testing
This requires adding products, flavors and needles and test
cases, and tweaking some existing ones to handle the
slightly different behaviour of live images in shared tests.

To handle the different main hub screens in live and non-live,
a less stringent needle is added which is unregistered for
non-live tests, so they don't match on it before they've
finished updating repository metadata.

There are a few small bugfix tweaks in this too, like some
delays in user creation to try and avoid intermittent failures
there.

A new root_logged_in needle is also included, to handle a new
console font in Rawhide - that has nothing strictly to do with
live testing, it just happened to come up while working on
this.
2015-03-18 14:28:03 -07:00
Garret Raziel
7c352d4b29 fix typo 2015-03-12 11:11:47 +01:00
Garret Raziel
591861586e add kickstart hdd testsuite 2015-03-12 10:58:04 +01:00
Garret Raziel
3863d840b0 add software RAID test suite 2015-03-06 10:36:25 +01:00
Garret Raziel
2038451b96 add multiple disks test 2015-03-05 13:57:47 +01:00
Garret Raziel
41d859cc7d add disk guided free space test 2015-03-05 11:44:02 +01:00
Garret Raziel
e7ae288abd remove hardcoded repo urls 2015-03-04 12:26:02 +01:00
Garret Raziel
7eb9eaa090 changes in user creation, http mirrors 2015-02-19 14:15:29 +01:00
Garret Raziel
4f8bc70d37 update templates 2015-02-13 10:58:24 +01:00
Josef Skladanka
07b75b1a30 fixed typo 2015-02-13 10:01:38 +01:00
Garret Raziel
a00cf9e530 New tests 2015-02-13 09:51:24 +01:00
Josef Skladanka
376ec0d81b Inst-repo HTTP variation 2015-02-04 14:45:37 +01:00
Josef Skladanka
475fcc0d19 Simplified the templates 2015-02-04 14:05:20 +01:00
Josef Skladanka
a4aec6b123 Fixed select-lang when geolocation pre-selects english 2015-02-03 18:11:29 +01:00
Josef Skladanka
df9ba3ba7f Updated templates 2015-01-27 16:35:10 +01:00
Josef Skladanka
8e6c972893 fixed disk guided multiselect 2015-01-27 16:24:12 +01:00
Garret Raziel
0c18688663 Added 'server_delete_pata' testsuite 2015-01-27 14:22:35 +01:00
Josef Skladanka
47e8c38dca Added few more tests 2015-01-27 13:35:27 +01:00
Josef Skladanka
780a9afbba Added 'server_sata_multi' testsuite
The testsuite covers guided installation on multiple devices.

TODO: change from VirtIO to SATA driver
2015-01-26 17:21:32 +01:00
Josef Skladanka
5264388dc5 Added 'server_simple' testsuite 2015-01-26 15:58:07 +01:00
Josef Skladanka
f17b37c292 Decoupled tools from tests 2015-01-26 14:43:01 +01:00
Renamed from fedora/templates (Browse further)