Block device handling can be somewhat complex - especially
when taking things like md, lvm or encryption into account.
This patch factors out the creation and deletion of the local
loop image device handling into a python library.
The main propose of this patch is to implement the needed
infrastructure. Based on this, more advanced functions can be added.
Example: (advanced) partitioning, LVM, handling different boot
scenarios (BIOS, UEFI, ...), possibility of handling multiple images
(local loop image, iSCSI, physical hard disk, ...), handling of
different filesystems for different partitions / LVs.
Change-Id: Ib626b36a00f8a5dc3dbde8df3e2619a2438eaaf1
Signed-off-by: Andreas Florath <andreas@florath.net>
Since the ironic-agent element builds the ramdisk and extracts the
kernel itself, there's no need to actually generate an image at the
end of the process. Previously the unnecessary image was being
deleted, but this wastes a bunch of time compressing and converting
the image. It's better to just not create the image at all.
This change adds a noop element called no-final-image that
disk-image-create looks for in the element list and, if found, will
cause it to skip the final image generation. This is more flexible
than the previous ironic-agent-specific method that would have
required changes to disk-image-create for every element that wanted
to behave similarly.
Note that this cannot be done using an environment variable, because
element environments.d entries do not propagate out to
disk-image-create. It also doesn't make sense as a user option
because it should be set by the element author, not the user.
Change-Id: I168feb18f0d578b3babbe4784d3ef75e755e1ebd
Under some systems this leads to an error if the oder of parameters
does not comply exactly with the way it is specified.
Change-Id: Ie1ff871dfffecaf95e7ac467b18543561aaa0ceb
The ironic-agent element doesn't care about the final qcow2/raw/
whatever image the disk-image-create command normally creates, so
previously it was deleting it at the end of the process. This is
a pretty significant waste of time when building those images, and
instead we can just skip creating the image when building
ironic-agent.
Change-Id: If48f575e795a823c777891f193ebf8bd943aa296
Allow file test to return all possible mime-types and ensure anything
that matches the python file type is parsed by flake8 instead of
relying on the first match returned.
Closes-Bug: #1585688
Change-Id: Iba31f1853537fe9234ab6f83d66f13dc1c578abb
For something fairly simple, I went back-and-forward with this a bit.
Firstly, I realise calling readlink constantly sucks. Due to the way
we call dib and source various files, you end up with the source-file
from "caller" being usually a very ugly path including levels of "../"
indirection. Cleaning this up to something canonical is the only sane
way to present it.
Because we evaluate _ps4() from a sub-shell in the PS4 string, there's
no way for it to do something like build a global in-memory cache in
an associative array or similar. It could write out a temp file or
some other side-band method, but the overheads of managing this don't
seem any different to just calling readlink. If anyone can think of a
bash-hack around this that doesn't involve a fork() I'm interested.
We could potentially strip some of the leading paths in the assumption
you know what they are; but it gets complex when things are split
across /usr/bin & /usr/lib and external elements, etc. I thought
about arbitrarily shortening it (e.g. just take last 20 characters)
which gives you enough of an idea of the file, but looks a bit ugly.
Or we could just leave the file-name out all together and assume the
function name is unique enough; this also seemed a bit ugly.
Obviously it's a matter of taste in the output. It is certainly
wider, but it also adds a lot of information. It also makes it fairly
clear where there are things we can make less verbose,
e.g. I1e39822f218dc0322e2490a770f3dc867a55802c disables tracing in
run-parts which is just noise. There's a few other frequently used
loops that we could disable tracing for by default to benefit
signal:noise.
tl;dr : take a look at the logs. I think it is a step in the right
direction of making the logs more usable for debugging.
Change-Id: I8054a3050415fcb527baeb7012bf133e5c864bf3
We have some test cases which attempt to build docker images, therefore
we need docker.
Fix a few bugs that showed up when we run docker tests - we need to
docker rm with sudo and docker images don't always have a /tmp so check
before unmounting it.
Change-Id: I147d0ef3f2ea83f35bac568214573a6bde0b1967
As motivation for this; we have had two breakouts of dib in recent
memory. One was a failure to unmount through symlinks in the core
code (I335316019ef948758392b03e91f9869102a472b9) and the other was
removing host keys on the build-system
(Ib01d71ff9415a0ae04d963f6e380aab9ac2260ce).
For the most part, dib runs unprivileged. Bits of the core code are
hopefully well tested (modulo bugs like the first one!). We give free
reign inside the chroot (although there is still some potential there
for adverse external affects via bind mounts). Where we could be a
bit safer (and could have prevented at least the second of these
breakouts) is with some better checking that the "sudo" calls
*outside* the chroot at least looked sane.
This adds a basic check that we're using chroot or image paths when
calling sudo in those parts of elements that run *outside* the chroot.
Various files are updated to accomodate this check; mostly by just
ignoring it for existing code (I have not audited these calls).
Nobody is pretending this type of checking makes dib magically safe,
or removes the issues with it needing to do things as root during the
build. But this can help find egregious errors like the key removal.
Change-Id: I161a5aea1d29dcdc7236f70d372c53246ec73749
It turns out that invalid JSON can be valid YAML ... thus if you mess
up a pkg-map file that still works as a YAML file dib-lint will let it
pass, but when pkg-map later tries to open it as a JSON file, it
fails.
Parse each type separately to catch these problems.
Change-Id: Ib3985e7d1599ed6bf3b7a73b786a53177b71fae0
It's hard to tell if dib-lint is working as it outputs nothing. Add
some minimal output strings at some key points.
Change-Id: Id11cc9ecb8d5215d6fc8d8ef3584bfeeba53ff13
It's better to report all of the failures in one shot, so we should
make sure a flake8 failure doesn't immediately end the dib-lint
run, and instead just sets the error flag like the other checks.
Change-Id: Ib13fc71bb12a6565888bdd89f33fc6ada89f8d8c
This was not well tested. Build the argument into a variable which
can be eval()ed to produce the final output.
Add the flag so we test this during functional tests. Add "-x" to dib
invocations so we can more easily debug failures.
Change-Id: Ifdc82627c520379b4124ccb9a4c2fe806c52c75c
We don't want the output of "du" run on the image spammed into the
logs with "set -x". Swizzle it off around the sensitive commands.
Change-Id: I687e77275f9a49e7934211835aba8610e88cdca6
If you check logs like [1] it's literally thousands of lines of the
same thing over-and-over as the git caching happens. It is basically
all just noise unless you're debugging it specifically. Up this to
tracing level 2 ("-x -x") to see it. Add a note in the help about
multiple flags, which has always been intended but not documented.
Image builds should continue to run with single "-x", but we could
probably greatly increase signal:noise ratio in the logs with a little
more judicial use of this to turn down some of the very noisy &
repetitive parts.
[1] anything in http://nodepool.openstack.org/
Change-Id: I91c5e55814ba9840769357261d203f4850e2eba6
This cuts the image size down alot, esspecially if there were lots of
small file deletes.
The fstrim utility is in the util-linux package and should be on
most all systems. fstrim also works with XFS, ext4, btrfs, etc
prodiving the kernel is new enough.
A reduction of 25% or more in size is common.
Change-Id: I269b4416be450369616f9b8e030f84c30e329804
In the common case of not specifying a size, we are already running
"du" over the image to figure out how big it is. Leverage that by
saving it's output and displaying a pruned list of big files when
requested.
We add a flag to show a summarised option (files >10MiB) and another
to show full output, should you wish that level of detail.
"Invocation" documentation is updated (and formatted a little better
while we're here).
Change-Id: I255800790a62fed1c82fcd311f1cc29c9867766d
Being able to discover DIB's version from the command itself is
convenient. This patch adds a --version option to the disk-image-create
command, failing gracefully if diskimage-builder is not installed.
This adds an explicit dependency on pbr to the requirements since this
is required to run diskimage_builder/version.py outside of a test
environment.
This patch consciously chooses to only provide the long-form option
and no '-v' to allow for the future possibility that a '-v' might
indicate '--verbose' in the future.
Change-Id: I9fc084774d6c7a39a944b07680b3eb8be8e34f9c
This checks the profile, if it has hardened in it's name it needs xattr support
unfortunately xattr support cannot yet be relied on everywhere, so it needs to
be disabled for hardened profile builds to correctly pax-mark.
Change-Id: I7fb855249a9e6c9b6497ab5061b4ea3c014f5081
Closes-Bug: 1537177
Our dib-lint checking is only considering scripts with #!/bin/bash.
While there's nothing really wrong with some other shebang line like
"#!/usr/bin/env bash" let's keep things consistent.
We can use the same regex match to reduce a few forks in the main
checking.
Also a minor cleanup to the file matching
Change-Id: I609721b2671e704ea26075dad7e5b39a8b858f6b
This patch fixes the calculation of the resultant image size
when building an image with diskimage-builder on ext4 a
filesystem.
Prior to this, using the '--image-size 2' (2GB) setting would
generate an image that would not boot under a 2GB nova flavor.
Change-Id: I7a753bdef84c6300ccea73ae4a92bf330dcd77cb
Closes-Bug: #1513622
Patch adds support for using decimal values for $DIB_IMAGE_SIZE.
This allows for creating images that are <1G in size.
Change-Id: I945644a8e77fecfb0b83efa282dc00bb29514e0b
Closes-Bug: #1366909
Some of the elements-deps in the project-config repo have a blank line
at the end, which throws out the ordering. Strip blank lines, like
comment lines, before processing.
As an additional help, show a side-by-side diff of what is provided
versus what is expected when showing an error about sorting.
Change-Id: I007851ee01d6853ad992ce4437331e8bd79bbfce
When dib-lint complains about wrong indents, it doesn't give you any
indication where the problem is. This repeats the grep on failure,
outputting the line and line-number.
As a bonus, skip *.orig files from merges
Change-Id: Ifbbdf854ea19191f66e9823468dbc0afc2f93e1f
Look for files .yaml and pkg-map configurations, and try to load them
either as json or yaml. This way, invalid ones can be detected before
they are committed unnoticed.
Also, exclude .yaml files from being searched while checking bash and
python scripts.
Change-Id: I2478837cfe66929ae1b0d7dd96e049773a35e11c
As described in the comments, inspect the installation to see if we
have been installed with "pip -e" and, if so, make sure we reference
the scripts from the source location rather than the
system-installations.
Update the documentation with a terse but helpful quick-start to show
an easy way to start developing a change using this.
Closes-Bug: #1491035
Change-Id: I0460061b834a2b854175f8c9be2be8d38c540c9d
App containers are a format used by rocket and are specified at the
following url:
https://github.com/appc/spec/
Change-Id: I8ac24f0194c4bf53dffd6c47e0587bc413101698
We can already produce tarballs, which is the input format docker import
expects. This makes it trivial to add docker as an output format.
Co-Authored-By: Dan Prince <dprince@redhat.com>
Change-Id: Ib60db3b717d33d4cf3181d70fe0ffbfa86fd5d02
Pass an explicit template to mktemp since that is required on Mac OS X.
Hardcoded to /tmp since I did not want to bother validating TMPDIR which
is user supplied.
Explicit define a repeating group which causes a grep error:
repetition-operator operand invalid
Change-Id: Ib23cc15db7f7dfdd611280884e46342562d545df
Split the cleanup_dirs function in two, i.e. cleanup of the build dir
and the image dir, and use the former to cleanup the temporary build
subdirs after their unmount, before the conversion to other disk
formats; they are not needed anyway at that point, and allows to save
disk space during the conversion phase.
Change-Id: Ie30d7e6033613d6979148423326ae7e17a7342e7
I noticed dib-lint was running on my editor backup files and checking
python files (covered by flake8) and RST files
Change-Id: Ide2e8a4c1aa8d619bc4b0326a2cfc5e63c74475f
