Move dib-run-parts from dib-utils into diskimage-builder directly.
For calling outside the chroot, we provide a standard entry-point
script. However, as noted in the warning comment, the underlying
script is still copied directly into the chroot by the dib-run-parts
element. I believe this to be the KISS approach.
This removes the dependency on dib-utils. We have discussed this
previously and nobody seemed to think retiring dib-utils was going to
be an issue.
This also updates the documentation to not mention dib-utils, or using
disk-image-create via $PATH setup, but rather gives instructions on
installing from pip with a virtualenv.
Change-Id: Ic1e22ba498d2c368da7d72e2e2b70ff34324feb8
Currently we have all our elements and library files in a top-level
directory and install them into
<root>/share/diskimage-builder/[elements|lib] (where root is either /
or the root of a virtualenv).
The problem with this is that editable/development installs (pip -e)
do *not* install data_files. Thus we have no canonical location to
look for elements -- leading to the various odd things we do such as a
whole bunch of guessing at the top of disk-image-create and having a
special test-loader in tests/test_elements.py so we can run python
unit tests on those elements that have it.
data_files is really the wrong thing to use for what are essentially
assets of the program. data_files install works well for things like
config-files, init.d files or dropping documentation files.
By moving the elements under the diskimage_builder package, we always
know where they are relative to where we import from. In fact,
pkg_resources has an api for this which we wrap in the new
diskimage_builder/paths.py helper [1].
We use this helper to find the correct path in the couple of places we
need to find the base-elements dir, and for the paths to import the
library shell functions.
Elements such as svc-map and pkg-map include python unit-tests, which
we do not need tests/test_elements.py to special-case load any more.
They just get found automatically by the normal subunit loader.
I have a follow-on change (I69ca3d26fede0506a6353c077c69f735c8d84d28)
to move disk-image-create to a regular python entry-point.
Unfortunately, this has to move to work with setuptools. You'd think
a symlink under diskimage_builder/[elements|lib] would work, but it
doesn't.
[1] this API handles stuff like getting files out of .zip archive
modules, which we don't do. Essentially for us it's returning
__file__.
Change-Id: I5e3e3c97f385b1a4ff2031a161a55b231895df5b
Because environment files are sourced into the current environment,
they shouldn't be setting global settings like tracing else they
affect every preceeding import. This is quite confusing when only
half your imports are traced in the logs, because it was either turned
on, or off, by a preceeding environment import.
There is a corresponding dib-run-parts change in
I29f7df1514aeb988222d1094e8269eddb485c2a0 that will greatly increase
debugability for environment files by deliberately logging what files
are sourced and consistently turning on tracing around their import.
This isn't strictly necessary (since dib-run-parts with the prior
change will just turn tracing off after import anyway) but it's a
decent cleanup for consistency. A bare-minimum dib-lint check is
added. Documentation is updated.
Change-Id: I10f68be0642835a04af7e5a2bc101502f61e5357
We currently have 'user guide' and 'developer documentation'. Lets
rename to 'developer guide' for consistency.
Change-Id: I834ea313bc34275ef33e8c49a1689dff41892015
By default sphinx uses localtoc which means 'show TOC for this page'.
Using a global table of contents on the sidebar is much more user
friendly wih our docs structure.
Change-Id: I215732d3848b4b75d9171bdbaaf2ff2e4dcc01f0
The table of contents for our developer guide does not show due to the
fact that it is past the first sub-header.
Change-Id: I8459a4949e3e4822b0a3cd4f163475d2c60b0f2e
Move element-info from a wrapper script to a standard entry-point
console_script.
Update the documentation to explain how to run it for development. I
don't think we should support the idea that you can check-out the code
and run ./bin/disk-image-create -- it has dependencies (dib-utils,
etc) and needs to be run from a virtualenv (this is what CI in the
gate does). A follow-up can clean-up some of the path munging stuff
we have for this in disk-image-create.
Change-Id: Ic0c03995667f320a27ac30441279f3e6abb6bca8
Block device handling can be somewhat complex - especially
when taking things like md, lvm or encryption into account.
This patch factors out the creation and deletion of the local
loop image device handling into a python library.
The main propose of this patch is to implement the needed
infrastructure. Based on this, more advanced functions can be added.
Example: (advanced) partitioning, LVM, handling different boot
scenarios (BIOS, UEFI, ...), possibility of handling multiple images
(local loop image, iSCSI, physical hard disk, ...), handling of
different filesystems for different partitions / LVs.
Change-Id: Ib626b36a00f8a5dc3dbde8df3e2619a2438eaaf1
Signed-off-by: Andreas Florath <andreas@florath.net>
These new variables are a list of elements chosen for the build along
with their full paths. For Python elements, IMAGE_ELEMENT_YAML is a
YAML formatted list that can be easily parsed. For bash elements,
"get_image_element_array" will produce an associative-array of the
same (working around lack of array export in Bash).
This list is intended for consumption of elements who need to copy
files from other elements, such as pkg-map and svc-map. As discussed
in I2a29861c67de2d25c595cb35d850e92807d26ac6, this list has already
been pruned and had overrides processed, so it is safe to simply walk
over this list with no further processing.
Since we're presenting the element list in a couple of different ways,
we combine it all into the element-info script. It will output an
eval-able string that declares the appropriate variables.
I've added some inline documentation so they still appear in grep.
The documentation is updated with examples, and moved to a more
appropriate location as a sub-section of the element sytle guide.
To test this out, use the associative-array in generate_hooks, where
we can now find the element's directory without searching.
Change-Id: Ibbd07d082ec827441def2d3f6240df3efdc6eae3
This is a re-factor of element_dependencies to achieve two things --
centralising override policy and storing path names.
Firstly we want to make the override policy for elements completely
explicit. Currently, elements that wish to copy parts of other
elements walk ELEMENTS_PATH themselves and look for elements in
IMAGE_ELEMENT. How they handle duplicate elements can differ, leading
to inconsistent behaviour.
We introduce logic in element-info to find elements in each of the
directories in ELEMENT_PATHS in *reverse* order -- that is to say,
earlier entries in the paths will overwrite later ones.
For example
ELEMENT_PATHS=foo:bar:baz
will mean that "foo/element" will override "baz/element", since "foo"
is first. This should be sane to anyone familiar with $PATH.
Documentation is clarified around this point and a test-case is added.
The second thing is that we want to keep the complete path of the
elements we have chosen. We want the aforementioned elements that
walk the element list to use these canonical paths to pickup files;
this way they don't need to make local decisions about element
overrides, but can simply iterate a list and copy/merge files if they
exist.
A follow-on change (I7092e1845942f249175933d67ab121188f3511fd) will
expose this data in a separate variable that can be parsed by elements
(a further follow-on I0a64b45e9f2cfa28e84b2859d76b065a6c4590f0
modifies the elements to use this information). Thus this does not
change the status-quo -- elements that are walking ELEMENTS_PATH
themselves and can/will continue doing that.
Change-Id: I2a29861c67de2d25c595cb35d850e92807d26ac6
Running the functional tests is time consuming. This patch adds the
option `-j <job count>` to the tests/run_functests.sh: when given the
test run in parallel up the <job count> jobs.
When using this, be sure to have enough resources (CPUs, RAM and HD
space) on the host.
In addition there was the need to change two things:
o Global /tmp/dib-test-should-fail was move to temporary build
directory of each execution.
o Because the logs might now interleave, each log line has now a
prefix of the name of the testcase.
[In my environment running functests sequential takes 15+ minutes,
running them parallel takes less than 6 minutes.]
Change-Id: Id9ea5131f0026c292ca6453ba2c80fe12c47f808
Signed-off-by: Andreas Florath <andreas@florath.net>
During the creation of a disk image (e.g. for a VM), there is the need
to create, setup, configure and afterwards detach some kind of storage
where the newly installed OS can be copied to or directly installed
in.
Change-Id: I0a43e247fb9e258e3983db35362f627416983773
Depends-On: I7bd7e9fa94635621590f72702107e218155fef2a
Signed-off-by: Andreas Florath <andreas@florath.net>
There has been some confusion about what this environment variable
controls, and it isnt very clear in the docs.
Change-Id: Id21b3c5ce361c4d1121eb7015020235b4c0a2f36
Currently we do not have a dib-specific specs repository. Technically,
we are part of the tripleo-specs repository but dib-core does not imply
tripleo-specs core. To fix this and to encourage the use of specs lets
create a specs process that lives right in tree.
Change-Id: I7bd7e9fa94635621590f72702107e218155fef2a
Currently, running sphinx_build fails for us because we depend on
diskimage-builder in our sphinx conf.py. This causes doc generation
on sites like rtfd to fail unless they install the diskimage-builder
module beforehand. We can, alternatively, import pbr directly and not
require the module as part of doc generation.
Change-Id: I41f222ff9c67950fc30841935a6a603f5718395e
I noticed we have no way easy way to get to the release notes. I
updated the front-page text to be a simpler introduction/overview and
provided links to the relnotes and source directly.
Change-Id: I5e339baf2921752ca3d409d82e0cbfc856ead1f8
As motivation for this; we have had two breakouts of dib in recent
memory. One was a failure to unmount through symlinks in the core
code (I335316019ef948758392b03e91f9869102a472b9) and the other was
removing host keys on the build-system
(Ib01d71ff9415a0ae04d963f6e380aab9ac2260ce).
For the most part, dib runs unprivileged. Bits of the core code are
hopefully well tested (modulo bugs like the first one!). We give free
reign inside the chroot (although there is still some potential there
for adverse external affects via bind mounts). Where we could be a
bit safer (and could have prevented at least the second of these
breakouts) is with some better checking that the "sudo" calls
*outside* the chroot at least looked sane.
This adds a basic check that we're using chroot or image paths when
calling sudo in those parts of elements that run *outside* the chroot.
Various files are updated to accomodate this check; mostly by just
ignoring it for existing code (I have not audited these calls).
Nobody is pretending this type of checking makes dib magically safe,
or removes the issues with it needing to do things as root during the
build. But this can help find egregious errors like the key removal.
Change-Id: I161a5aea1d29dcdc7236f70d372c53246ec73749
Prior to this, no user documentation of dib-lint existed, which
meant users had to read the dib-lint code itself to figure out
how it worked. This changes adds documentation on using dib-lint
and the checks it currently supports.
Change-Id: I285c5cc680dd9fbd9bd3f667ef102be14e248114
Add documentation to our developer guide about not creating executables
before or after 10/90 in the upstream element's phase directories.
Change-Id: I93ab70f37da0d81f8683a76fd3b341b761ea04e9
This cuts the image size down alot, esspecially if there were lots of
small file deletes.
The fstrim utility is in the util-linux package and should be on
most all systems. fstrim also works with XFS, ext4, btrfs, etc
prodiving the kernel is new enough.
A reduction of 25% or more in size is common.
Change-Id: I269b4416be450369616f9b8e030f84c30e329804
This reverts commit 5184d02a7c.
The decision was made to go with fstrim because it is faster and more
universal that zerofree. The related-id has the patchset that implements
fstrim.
Related-Id: I269b4416be450369616f9b8e030f84c30e329804
Change-Id: If40cf2fc0ecd8686768cbfeac9ecee90907674e7
In the common case of not specifying a size, we are already running
"du" over the image to figure out how big it is. Leverage that by
saving it's output and displaying a pruned list of big files when
requested.
We add a flag to show a summarised option (files >10MiB) and another
to show full output, should you wish that level of detail.
"Invocation" documentation is updated (and formatted a little better
while we're here).
Change-Id: I255800790a62fed1c82fcd311f1cc29c9867766d
If the image has an ext filesystem and the zerofree utility is present
on the build system then run zerofree. This should make images as
compressable as possible which is a nice feature when building
compressed qcow2 images.
Change-Id: Ia6062c291f7a3f58b85a4f408ecb3d0574c65d53
The quickstart should be the first bit of developer documentation, not
the last. Also add in a short blurb for the developer docs so we dont
have two doc titles back to back.
Change-Id: Icb5683b8eb22e759fefb1cb2252ed445dea5f7dd
