This option to the functional test runner outputs logs for each test
to ${LOGDIR}/${element}/${test_element}.log
Change-Id: I3bf5362bcda8acda1a03c1058728875a9a6c9711
Add a bionic test in replacement of trusty. We are already building
bionic images in the gate, so this seems like a good time to switch.
Change-Id: I20d4c25e9b79e7326c86767c36be8615ba0888a3
This adds support for a GPT label type to the partitioning code. This
is relatively straight-forward translation of the partition config
into a sgparted command-line and subsequent call.
A unit test is added based on a working GPT/EFI configuration and the
fedora-minimal functional test is updated to build a single-partition
GPT based using the new block-device-gpt override element. See notes
in the sample configuration files about partition requirements and
types.
Documentation has been updated.
Co-Authored-By: Marcin Juszkiewicz <marcin.juszkiewicz@linaro.org>
Change-Id: I6b819a8071389e7e4eb4874ff7750bd192695ff2
I747c2b8754effbc6ec82af3bf7543fd9599a6c14 removed the
centos-openstack-octata repos from the base image. We were previously
getting debootstrap from there, so this broke the centos-7 based jobs.
Enable EPEL for the install so we use that version of debootstrap.
Additionally we noticed that dpkg wasn't installed on the build host
either, but is actually required (calls were silently failing ... a
problem for another day). Add this.
Change-Id: I1e9db5130a05a004c3c877d1444d7097023c06e2
Change I008f8bbc9c8414ce948c601e3907e27764e15a52 has shown that we
build redhat images without the "semange" tool available, which comes
from the policycoreutils-python package (see also
I3f9e2c322d042a5dddba33451c0fc21a4d32a88a).
I403e7806ae10d5dd96d0727832f4da20e34b94c7 added some of the selinux
libraries to yum-minimal for ansible support, but not to others.
Given both these changes, it seems that selinux[-targeted],
libselinux[-python] and policycoreutils[-python] can reasonably
considered part of all base images. Move the selinux related packages
into redhat-common.
This also adds it explicitly to install_test_deps.sh. It was actually
being dragged in by the docker install, but is a required component
for building (should be in bindep, but not there with that yet).
Change-Id: Idd4ae71ee6deee84604823b6b5dc4a845f316e01
Related-Bug: #1707788
This was just to avoid our initial gate crisis, and has been put into
project-config with I45b4b181369032155f8908ee11641d2327586e6f
Change-Id: I3ab57b4455b39ccc3fa94ef1be2193fa7f082fb6
The image download tests have long been too unreliable for the gate.
We need to cache the base images similar to how devstack caches it's
testing images. Let's move them to non-voting jobs for the time
being.
This means that the gate jobs are now all based on "-minimal" and are
using infra mirrors. Unfortunately, there is still some unreliability
because we currently have issues with infra mirrors being very slow
after AFS updates, leading to job timeouts. But we're on the right
path...
Also, I noticed we don't have tests of the "ubuntu" image-download
based tests, which were tacitly being tested by apt-sources before we
moved that to -minimal. Add simple tests for these.
Change-Id: Ie33ee49656872467ef68d753210032156bb6b2cb
fedora/centos-minimal don't obey DIB_DISTRIBUTION_MIRROR currently. I
don't really want them too -- we want to be able to separate the
mirrors used during the build process from those embedded into the
final image. Add DIB_YUM_MINIMAL_BOOTSTRAP_REPOS which is a directory
with repo files to use during the install.
This introduces setup-gate-mirrors.sh which is intended to setup
repo/sources/whatever files in the openstack gate that point to the
local region mirror. It pulls the info from the mirror_info.sh script
on each CI node.
The openstack-ci-mirrors element is updated to export these variables.
elements are updated to depend on it. Tests are restored
Change-Id: I7604fc4d41cb1483be16b8d628a24e8fc764f515
This adds "openstack-ci-mirrors" element which performs various
settings to get builds using local mirrors. As a first step, we
convert ubuntu-minimal jobs
The main trick is that since infra mirrors are created with rerepo
they are not signed (they are recreated, not cloned, and not signing
is seen as a feature in that it deters external use). So we need to
instruct debootstrap to ignore signing and also turn it off for
in-chroot apt. Other than that, the existing DIB_DISTRIBUTION_MIRROR
works to redirect installs.
Remove "restricted" as it's not mirrored, and I don't think we want it
in here by default.
(I think DIB_DISTRIBUTION_MIRROR is a bit of an anti-pattern, because
it leaves the mirrors in the final image -- just because you use them
to build, doesn't mean you want them at runtime). But we don't need
to fix that now, and we don't use any created images.)
This pauses fedora testing until the next change, which moves to using
local mirrors for testing on fedora/centos
Change-Id: I778bd05a1e615c27edf1c9f0a1409119a6b3a850
The gate is currently extremley unstable, and these two issues are
causing most of the problems. We need to commit them atomically so we
can get anything moving again
---
The gate is very unstable downloading the ubuntu tarballs from
upstream at the moment. Move this to ubuntu-minimal which, in a later
change will source files from our local mirror.
We need a caching mechanism for these large files to avoid this
instability. This is future work for the various image-based jobs.
---
Move debian to default skip lists
I don't know if it's mirrors being worked hard for the Stretch
release, but this is constantly failing the gate. I will move this to
the -nv extras job
I am working on having the voting job use local mirrors for
everything. Unfortunately debian infra mirrors don't have stretch yet
and we need to do some fiddling to get "stable" available. Once we
have all this, we can consider making it voting again.
Change-Id: Iaf7b3888ef06c7aef63cbf76a94b33f96bc9c5c2
This adds a devstack-inspired output filter to standardise
timestamping.
Currently, python tools timestamp always (timestamp setup in
logging_config.py) but all the surrounding bash does not.
We have extra timestamps added in run_functests.sh for our own
purposes to get the bash timestamps; but this ends up giving us
double-timestamps for the python bits. Additionally, callers such as
nodepool capture our output and put their own timestamps on it, and
again have the double-timestamps.
This uses a lightly modified outfilter.py from devstack to standardise
this.
All output is run through this filter, which will timestamp it. I
have removed the places where we double-timestamp -- logging_config.py
and the prefix in dib-run-parts.
An env option is added to turn timestamps off completely (does not
seem worth taking up a command-line option for). For callers like
nodepool, they can set this and will just have their own timestamps as
they collect the lines.
Since all logging is going through outfilter, it's easy to add a
--logfile option. I think this will be quite handy; personally I'm
always redirecting dib runs to files for debugging.
I've also added a "quiet" option. I think this could be useful in
run_tests.sh if we were to start logging the output of each test to
individual files. This would be much easier to deal with than the
very large log files we get (especially if we wanted to turn on
parallel running...)
Change-Id: I202e1cb200bde17f6d7770cf1e2710bbf4cca64c
Add a -k flag that disables deleting of of ${destdir} for tests. This
should allow examination of the resulting images if required.
Change-Id: I107c33e70100b21495a807f10762d3b6babe9bfe
Ubuntu 12.04 LTS reached its regular End of Life on April 28, 2017.
Depends-On: I5e145095a10db112bb27516bfe652d2cdc052a61
Change-Id: I64af4c5183d77a75dcd062895d19b0a1330c8da8
On SUSE family distros the squash-tools are simply part of the
main package called "squashfs", so install that one instead. Without
this change bindep on SUSE hosts fails with:
ERROR: These requested packages were not installed:
squashfs-tools
Also adjust install_test_deps.sh to install required packages
on an openSUSE host.
Change-Id: I61dcd5314e78dbb1fb31e723799374edd456da99
Currently openSUSE 42.3 has entered feature freeze mode
so it is a good point in time to verify that 42.3 builds
are working successfully. Also test opensuse-minimal for
platforms that support it (need working zypper package)
Change-Id: I4c613e1e68cb7375c29d544bbf70b5da9bf21414
Instead, either use the bash built-in of type to ensure it exists. Since
which is an external dep, things can fail oddly in a constrained
environment.
Also add a dib-lint test for this.
Change-Id: I645029f5b5bfe1198c89ce10fd3246be8636e8af
Signed-off-by: Jesse Keating <omgjlk@us.ibm.com>
I think this has gone missing on test nodes during unpuppeting. Add,
as the gentoo job grabs some bz2 tarballs for extraction.
Change-Id: Icb82930702124f7bf112b03a469d90d4163d398b
If the path is missing, unmount_dir currently exits with an error which
unintentionally aborts cleanup efforts early. This change makes
unmount_dir idempotent by exiting successfully if a directory doesn't
exist.
Change-Id: I1491b4344e8569ecb2833f44baee445a89a39d61
We somewhat discussed skipping qcow2 generation previously in
I9372e195913798a851c96e62eee89029e067baa1. As recent issues with PPC
testing have shown, we are not actually testing the "vm" element and
hence the bootloader path in the functional tests.
I don't think we need to test this on every element; it overlaps
somewhat with the testing done by the nodepool jobs which build full
images and boot them. I also didn't want to introduce a separate run
for this. Thus it seems valuable to at least have one element
enhanced to do this installation and conversion in our default tests
for basic sanity.
This disables qcow generation by default, as per the other change, but
allows an element to drop a file that will override the output
formats. The Xenial element is modified to produce a qcow2 using
this, and also introduces a dependency on the "vm" element so it tries
to install the bootloader.
We now exit if the .qcow2 fails to build as well.
Change-Id: I1a6acefe52f8c696c39b2d592fdc7ae32a87e6fe
Change ec7f56c1b2 add added unit tests
in diskimage_builder/tests/functional (this is probably misnamed).
These are found and run by testr just as part of the normal "setup.py
test" run. Don't run them as part of the functional tests (this
breaks "-h"/"-l" because it installs a virtualenv and runs tests, and
also is incorrect in the gate where it's creating a nested virtualenv
underneath the testing virtualenv).
Change-Id: I9908e080042d3026a198ba89eb653c6eff376d22
During the creation of a disk image (e.g. for a VM), there is the need
to create, setup, configure and afterwards detach some kind of storage
where the newly installed OS can be copied to or directly installed
in.
This patch implements partitioning handling.
Change-Id: I0ca6a4ae3a2684d473b44e5f332ee4225ee30f8c
Signed-off-by: Andreas Florath <andreas@florath.net>
DIB automatically includes it's base elements in the ELEMENT_PATH;
this double-set means the base elements are in the path twice, leading
to some tools that walk the element list (package-installs-squash, for
example) to do things twice.
Change-Id: Icf631427601cfd2abed436711cedb9e002fff8f9
Because we run this image in openstack-infra, we want to increase our
test coverage to help avoid potential breaks to our CI systems.
Change-Id: I26405e3f7465654075278ec35b5e0da1338bb45e
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
Since we still run these 3 version of ubuntu-minimal elements in
openstack-infra, also run functional testing for them.
Trusty and xenial will be in voting gate, precise added as skipped for
non-voting.
Add the default skip/run status to the "-l" output just to confirm
this too.
Change-Id: Icfbfd0cb7d9acae824972474b77e2fe0486c4f69
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
The squashfs format brings a couple of advantages over the other
formats. Image is often an order of magnitude smaller and it can
be used natively, either as an initrd, either with loop mount.
Change-Id: If72940b0c4dafb2504c52dd0429a8eb3f8305751
We now support tgz (tar.gz) as an output format.
Change-Id: Iadec92f2f96c3f904f28bd49f87ffc7d48ef7bd7
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
openSUSE 13.1 was discontinued on Feb 3rd, 2016, so defaulting
to it doesn't make sense (see https://en.opensuse.org/Lifetime).
Leap 42.2 is the most current release that is supported by
disk-image-builder and being tested in a 3rd party ci.
Enable functests for it to ensure we're not regressing again.
Moved to non-voting gate first.
Depends-On: Iff495b3cd0b6c3558c44cf4883651eca67b572d6
Change-Id: Iae6cd34a5853f1e309861c554d94d8595cbd9993
Centos6 is no longer reasonable to expect to function with openstack,
and infra does not host a mirror for it.
Change-Id: I95ccf5840807fee73d6e78d596c82709e476bb3a
It has always been a weird thing that dib is a python package, but
is totally driven by the disk-image-create script. It creates this
strange division that is hard to explain.
This moves disk-image-create to a regular python entry-point
Currently, this simply exec()s the original disk-image-create script.
However, we now have a (private) interface between disk-image-create
written in python and the driver shell script. Here's some things we
could do, for example:
* Argument parsing is generally nicer in Python, and then end result
is mostly just setting environment variables to flag different things
in the shell script. I could see us moving the argument-parsing into
diskimage_builder.disk_image_create:main() and just setting things in
os.environ before the exec()).
* I7092e1845942f249175933d67ab121188f3511fd sets IMAGE_ELEMENT_YAML in
disk-image-create by calling-back to element-info. We can just call
element_dependencies.find_all_elements() in here an export is to
os.environ before disk-image-create starts.
* remove need for ramdisk-image-create symlink by just exporting
IS_RAMDISK based on sys.argv[1] value
* you could even unit test some of this :)
Change-Id: I69ca3d26fede0506a6353c077c69f735c8d84d28
Currently we have all our elements and library files in a top-level
directory and install them into
<root>/share/diskimage-builder/[elements|lib] (where root is either /
or the root of a virtualenv).
The problem with this is that editable/development installs (pip -e)
do *not* install data_files. Thus we have no canonical location to
look for elements -- leading to the various odd things we do such as a
whole bunch of guessing at the top of disk-image-create and having a
special test-loader in tests/test_elements.py so we can run python
unit tests on those elements that have it.
data_files is really the wrong thing to use for what are essentially
assets of the program. data_files install works well for things like
config-files, init.d files or dropping documentation files.
By moving the elements under the diskimage_builder package, we always
know where they are relative to where we import from. In fact,
pkg_resources has an api for this which we wrap in the new
diskimage_builder/paths.py helper [1].
We use this helper to find the correct path in the couple of places we
need to find the base-elements dir, and for the paths to import the
library shell functions.
Elements such as svc-map and pkg-map include python unit-tests, which
we do not need tests/test_elements.py to special-case load any more.
They just get found automatically by the normal subunit loader.
I have a follow-on change (I69ca3d26fede0506a6353c077c69f735c8d84d28)
to move disk-image-create to a regular python entry-point.
Unfortunately, this has to move to work with setuptools. You'd think
a symlink under diskimage_builder/[elements|lib] would work, but it
doesn't.
[1] this API handles stuff like getting files out of .zip archive
modules, which we don't do. Essentially for us it's returning
__file__.
Change-Id: I5e3e3c97f385b1a4ff2031a161a55b231895df5b
We're getting double time-stamps in the console log of upstream jobs.
Move the logging of a prefix datestamp into a "-t" option to retain
the status quo prior to Id9ea5131f0026c292ca6453ba2c80fe12c47f808 (we
could, of course, do it the other way and turn if off in the jobs, but
since we didn't have it before...)
While poking, make the time-stamp consistent and always prefixed if -t
is turned on.
Also, it seems the parallel options got a bit of sync with what got
merged. Add "-j" documentation and remove unused "p" option.
Change-Id: Ic7c2ebeca3f9d5784cac59505b6e6181151f5805
Running the functional tests is time consuming. This patch adds the
option `-j <job count>` to the tests/run_functests.sh: when given the
test run in parallel up the <job count> jobs.
When using this, be sure to have enough resources (CPUs, RAM and HD
space) on the host.
In addition there was the need to change two things:
o Global /tmp/dib-test-should-fail was move to temporary build
directory of each execution.
o Because the logs might now interleave, each log line has now a
prefix of the name of the testcase.
[In my environment running functests sequential takes 15+ minutes,
running them parallel takes less than 6 minutes.]
Change-Id: Id9ea5131f0026c292ca6453ba2c80fe12c47f808
Signed-off-by: Andreas Florath <andreas@florath.net>
This patch solves three issues with Debian packaging / apt:
o When building 'testing' only default apt sources is
included - backports, updates and security are skipped because they
do not exists.
o The default release for Debian was `unstable`: this is now fixed to
`stable`.
o Starting a Debian Stretch VM that was build with diskimage-builder
does not work, because some mandatory packages are missing.
This patch fixes this problem: it adds the mandatory packages and
the test case.
Change-Id: If49b5b162c4da1e074e9b19324839bc59d87dc57
Signed-off-by: Andreas Florath <andreas@florath.net>
