While trying to get docker image pre-caching to work we couldn't get a
docker daeomon to run within the chrooted environment. However we got
docker running with the help of bwrap outside of the chrooted
environment. The only option so far for this is the block-device.d
phase. But this has the problem that it runs after the image size has
been calculated. This leads to broken builds if the docker images
being pulled are big.
This can be solved by adding a post-root.d phase that runs outside the
chroot but before the image size calculation.
Change-Id: I36c2a81e2d9f5069f18ce5b0d52c5f1c7212c3ae
In exploring Gentoo caching, it was realised that we have no way to
bind mount the cache into the finalised image for the finalise.d
phases.
By adding a pre-finalise.d phase that runs outside the chroot, we can
mount outside things into the hierarchy at $TMP_BUILD_DIR/mnt which
are then seen by the in-chroot finalise.d phase.
This is similar to the pre-install phase
Change-Id: I9d782994843383ddf90f62c40498af9925fd9558
Some minor things after looking at these parts.
The dib-run-parts element doesn't do any of the copying any more, so
these comments are wrong.
The reason for the multiple mounts in the bind mount was non-obvious
to modern eyes (as util-linux has handled this for some time).
Formatting fix for the rst
Change-Id: Idb4c9ff32c49aced2c68a5c905bf7a8b2832a5a2
Make clear to where in the chroot the contents of
`$TMP_HOOKS_PATH` will be available.
Change-Id: I4b9d20f7ec1c317eb61da44bfd05242dd45263c4
Co-Authored-By: Elyezer Rezende <erezende@redhat.com>
dib_[environment|args] manifest files are currently generated by the
base element and then moved by the manifest element.
This creates too many corner cases -- if you don't include the base
element (we are trying to empty it ATM) you don't get the env/args
saved at all; if you include base but don't include the manifest
element they're saved to /etc, but if you do have the manifest element
they're moved to the manifest dir.
Move generation of these into the manifest element directly and update
the documentation to reflect this. In practice this doesn't change
things, because the "manifests" element gets pulled in via deps for
most builds.
Change-Id: I3f23037058137d166b29f0b70fd1a02c22c07fc8
Signed-off-by: Andreas Florath <andreas@florath.net>
Because environment files are sourced into the current environment,
they shouldn't be setting global settings like tracing else they
affect every preceeding import. This is quite confusing when only
half your imports are traced in the logs, because it was either turned
on, or off, by a preceeding environment import.
There is a corresponding dib-run-parts change in
I29f7df1514aeb988222d1094e8269eddb485c2a0 that will greatly increase
debugability for environment files by deliberately logging what files
are sourced and consistently turning on tracing around their import.
This isn't strictly necessary (since dib-run-parts with the prior
change will just turn tracing off after import anyway) but it's a
decent cleanup for consistency. A bare-minimum dib-lint check is
added. Documentation is updated.
Change-Id: I10f68be0642835a04af7e5a2bc101502f61e5357
Block device handling can be somewhat complex - especially
when taking things like md, lvm or encryption into account.
This patch factors out the creation and deletion of the local
loop image device handling into a python library.
The main propose of this patch is to implement the needed
infrastructure. Based on this, more advanced functions can be added.
Example: (advanced) partitioning, LVM, handling different boot
scenarios (BIOS, UEFI, ...), possibility of handling multiple images
(local loop image, iSCSI, physical hard disk, ...), handling of
different filesystems for different partitions / LVs.
Change-Id: Ib626b36a00f8a5dc3dbde8df3e2619a2438eaaf1
Signed-off-by: Andreas Florath <andreas@florath.net>
These new variables are a list of elements chosen for the build along
with their full paths. For Python elements, IMAGE_ELEMENT_YAML is a
YAML formatted list that can be easily parsed. For bash elements,
"get_image_element_array" will produce an associative-array of the
same (working around lack of array export in Bash).
This list is intended for consumption of elements who need to copy
files from other elements, such as pkg-map and svc-map. As discussed
in I2a29861c67de2d25c595cb35d850e92807d26ac6, this list has already
been pruned and had overrides processed, so it is safe to simply walk
over this list with no further processing.
Since we're presenting the element list in a couple of different ways,
we combine it all into the element-info script. It will output an
eval-able string that declares the appropriate variables.
I've added some inline documentation so they still appear in grep.
The documentation is updated with examples, and moved to a more
appropriate location as a sub-section of the element sytle guide.
To test this out, use the associative-array in generate_hooks, where
we can now find the element's directory without searching.
Change-Id: Ibbd07d082ec827441def2d3f6240df3efdc6eae3
Running the functional tests is time consuming. This patch adds the
option `-j <job count>` to the tests/run_functests.sh: when given the
test run in parallel up the <job count> jobs.
When using this, be sure to have enough resources (CPUs, RAM and HD
space) on the host.
In addition there was the need to change two things:
o Global /tmp/dib-test-should-fail was move to temporary build
directory of each execution.
o Because the logs might now interleave, each log line has now a
prefix of the name of the testcase.
[In my environment running functests sequential takes 15+ minutes,
running them parallel takes less than 6 minutes.]
Change-Id: Id9ea5131f0026c292ca6453ba2c80fe12c47f808
Signed-off-by: Andreas Florath <andreas@florath.net>
As motivation for this; we have had two breakouts of dib in recent
memory. One was a failure to unmount through symlinks in the core
code (I335316019ef948758392b03e91f9869102a472b9) and the other was
removing host keys on the build-system
(Ib01d71ff9415a0ae04d963f6e380aab9ac2260ce).
For the most part, dib runs unprivileged. Bits of the core code are
hopefully well tested (modulo bugs like the first one!). We give free
reign inside the chroot (although there is still some potential there
for adverse external affects via bind mounts). Where we could be a
bit safer (and could have prevented at least the second of these
breakouts) is with some better checking that the "sudo" calls
*outside* the chroot at least looked sane.
This adds a basic check that we're using chroot or image paths when
calling sudo in those parts of elements that run *outside* the chroot.
Various files are updated to accomodate this check; mostly by just
ignoring it for existing code (I have not audited these calls).
Nobody is pretending this type of checking makes dib magically safe,
or removes the issues with it needing to do things as root during the
build. But this can help find egregious errors like the key removal.
Change-Id: I161a5aea1d29dcdc7236f70d372c53246ec73749
Add documentation to our developer guide about not creating executables
before or after 10/90 in the upstream element's phase directories.
Change-Id: I93ab70f37da0d81f8683a76fd3b341b761ea04e9
This simplifies and enhances the functional-test runner script for
much better interactive behaviour and to give us the ability to better
choose what is running in CI.
Firstly, I have split the image-output testing into a separate script.
This is not actually part of the functional testing of elements and is
both logically and functionally different. It currently does not run
in upstream CI because we don't have docker in the images. I have
nothing against it, but it can be it's own thing.
run_functests.sh is overhauled to have a useful interactive interface,
e.g.
---
$ ./run_functests.sh -h
run_functests.sh [-h] [-l] <test> <test> ...
-h : show this help
-l : list available tests
<test> : functional test to run
Special test 'all' will run all tests
$ ./run_functests.sh -l
The available functional tests are:
apt-sources/test-sources
debian/build-succeeds
fedora/build-succeeds
fedora/build-succeeds-f21
ironic-agent/build-succeeds-fedora
---
As described there, you can run a single test, a number of tests, the
default tests (as CI will do) or all tests. Running all tests is too
much for regular CI, but currently the only way to stop a low priority
test running, or temporarily pause is to remove it completely --
clearly sub-optimal (see I93c2990472e88ab3e5ff14db56b4ff1b4dd965ef).
There is nothing complicated about this, and to further simplify I
have merged the runner functions back into run_functests.sh which
remains a very modest ~150 lines, with most of that being argument
sanity. With that and the image-format cleanup, we can remove the
indirection of the 3 small library files.
For consistency, I have renamed the "dib_functions_test" (that tests
things from the dib functions library) with a run_* prefix.
Because the default list is the same as the current functional tests
run, this does not modify the status-quo. I plan to modify this,
however, to run fedora-minimal & centos-minimal tests in a future
change, as these are required to be stable for openstack ci.
Documentation is updated, and a README.rst is added in the tests
directory for discoverability.
Change-Id: I86d208bd34ff09a29fdb916a4e7ef740c7f65af8
Now 'tox -efunc' can be invoked to run all functional tests in
the 'venv' tox environment. Also `tox -efunc element-name` can be
used to run function tests for one element (e.g. ironic-agent).
Change-Id: Ia685d1b2a7deef2f8b98876ac09792134dd30f2f
Augment the developing_elements.rst by taking advantage of Sphinx
markup. Most of the doc used to be in /README.md and thus did not
leverage on Sphinx.
Use inline codeblock to denote variables, files, command: ``foo``
Phase Subdirectories:
List phase names in the preliminary introduction
Get rid of lists in favor of definitions
Highlight whether the phase runs in or outside the chroot
Input parameters are now lists
Use definition lists in Dependencies and Ramdisk sections.
Link to elements README when they exist.
Testing Element: split into two subsections: 'shell' and 'python'.
Use "sourcecode:: sh" for the couple examples at the very top and very
bottom of the document.
Change-Id: I2421f76ec452cac243ccb2208f88c7d320ffedd3
The README.rst has a lot of information that has been duplicated in the
Sphinx maintained documentation (3600330).
Remove dupes from README.rst
Point to http://docs.openstack.org/developer/diskimage-builder/
Change summary:
=====================+======================================
README.md | Sphinx document
section |
=====================+======================================
Installation | installation.rst
---------------------+--------------------------------------
Invocation | invocation.rst
---------------------+--------------------------------------
Requirements | installation.rst Speedups
---------------------+--------------------------------------
Caches/offline | caches.rst + changes from 849e9cb2
| fix some markup
---------------------+--------------------------------------
Install Types | install_types.rst
---------------------+--------------------------------------
Writing an element | developing_elements.rst + fe7823a2
| `Testing element` from b9b6640f
| `3rd party elements' from f1e7bf3a
---------------------+--------------------------------------
Existing elements | elements.rst
---------------------+--------------------------------------
What tools are there | components.rst
---------------------+--------------------------------------
Design | design.rst
---------------------+--------------------------------------
Change-Id: I578daa8e3a8d876b3ee3c9a748d7c8aa2bf7a0b7
jenkin => jenkins
on ci.md
documention => documentation
on developer/caches.html
typicallly => typically
on developer/developing_elements.html
Closes-Bug: #1476993
Change-Id: Ie40205debad5dbc6074e65672e0f3ebeaee5b08e
Having data files in the phase subdirs is an easy source of confusion
in reviews (especially when the data file is a script) and theres really
no reason to be putting data files there at this point. Lets make a
convention out of not doing this.
Change-Id: I99571a2a49e14e8c709af20f6d13d662ac745eb4
Adding a test function which allows us to use elements to perform
element-specific tests. In order for this to work sanely, also adding
some configuration to our break system so we can assert on negative
tests.
Also adding a test for apt-sources to verify this code actually works.
Change-Id: I378a74255010eca192f5766b653f8a42404be5ea
Add a small documentation paragraph about the operating system elements,
what they are required to provide (and thus what other elements can rely
on).
This makes DISTRO_NAME a prime-class variable, which can now be assumed
to always exists (it was de-facto required so far).
Change-Id: Iffbc69de0516b58bfde48e87cd73073428d66b05
Set the pbr option 'warnerrors' to make build_sphinx turns warnings into
error. Fix all warnings.
`tox -edocs` will thus abort whenever someone introduce a new error.
Change-Id: Id6d09768a241866e1fdc1a1e2bf90336f5c5087d
Our docs are very developer focused. Lets create a separate user guide
to help new users get started.
Change-Id: I8a03920e6d3306dd0405177875ea55ccb4b40fea
2015-04-01 19:51:08 +00:00
Renamed from doc/source/developing_elements.rst (Browse further)
