Commit Graph

115 Commits

Author SHA1 Message Date
Ben Nemec
88bf264fbb Don't create an ironic-agent image just to delete it
The ironic-agent element doesn't care about the final qcow2/raw/
whatever image the disk-image-create command normally creates, so
previously it was deleting it at the end of the process.  This is
a pretty significant waste of time when building those images, and
instead we can just skip creating the image when building
ironic-agent.

Change-Id: If48f575e795a823c777891f193ebf8bd943aa296
2016-07-08 15:56:26 -05:00
Ian Wienand
34efed027a Add PS4 to show file/function/line in debug output
For something fairly simple, I went back-and-forward with this a bit.

Firstly, I realise calling readlink constantly sucks.  Due to the way
we call dib and source various files, you end up with the source-file
from "caller" being usually a very ugly path including levels of "../"
indirection.  Cleaning this up to something canonical is the only sane
way to present it.

Because we evaluate _ps4() from a sub-shell in the PS4 string, there's
no way for it to do something like build a global in-memory cache in
an associative array or similar.  It could write out a temp file or
some other side-band method, but the overheads of managing this don't
seem any different to just calling readlink.  If anyone can think of a
bash-hack around this that doesn't involve a fork() I'm interested.

We could potentially strip some of the leading paths in the assumption
you know what they are; but it gets complex when things are split
across /usr/bin & /usr/lib and external elements, etc.  I thought
about arbitrarily shortening it (e.g. just take last 20 characters)
which gives you enough of an idea of the file, but looks a bit ugly.
Or we could just leave the file-name out all together and assume the
function name is unique enough; this also seemed a bit ugly.

Obviously it's a matter of taste in the output.  It is certainly
wider, but it also adds a lot of information.  It also makes it fairly
clear where there are things we can make less verbose,
e.g. I1e39822f218dc0322e2490a770f3dc867a55802c disables tracing in
run-parts which is just noise.  There's a few other frequently used
loops that we could disable tracing for by default to benefit
signal:noise.

tl;dr : take a look at the logs.  I think it is a step in the right
direction of making the logs more usable for debugging.

Change-Id: I8054a3050415fcb527baeb7012bf133e5c864bf3
2016-05-18 09:23:15 +10:00
Jenkins
05382d10b9 Merge "Fix disk usage report" 2016-04-18 19:36:40 +00:00
Jenkins
a6dd8d6b5a Merge "Turn down tracing for source-repo cache" 2016-04-11 06:12:49 +00:00
Ian Wienand
a7afe652d6 Fix disk usage report
This was not well tested.  Build the argument into a variable which
can be eval()ed to produce the final output.

Add the flag so we test this during functional tests.  Add "-x" to dib
invocations so we can more easily debug failures.

Change-Id: Ifdc82627c520379b4124ccb9a4c2fe806c52c75c
2016-04-08 07:07:00 +10:00
Ian Wienand
2f214ff3e8 Turn of tracing around du invocations
We don't want the output of "du" run on the image spammed into the
logs with "set -x".  Swizzle it off around the sensitive commands.

Change-Id: I687e77275f9a49e7934211835aba8610e88cdca6
2016-04-07 08:11:39 +10:00
Jenkins
404ca1b944 Merge "Add image size report" 2016-04-02 15:45:42 +00:00
Ian Wienand
8c837409b8 Turn down tracing for source-repo cache
If you check logs like [1] it's literally thousands of lines of the
same thing over-and-over as the git caching happens.  It is basically
all just noise unless you're debugging it specifically.  Up this to
tracing level 2 ("-x -x") to see it.  Add a note in the help about
multiple flags, which has always been intended but not documented.

Image builds should continue to run with single "-x", but we could
probably greatly increase signal:noise ratio in the logs with a little
more judicial use of this to turn down some of the very noisy &
repetitive parts.

[1] anything in http://nodepool.openstack.org/

Change-Id: I91c5e55814ba9840769357261d203f4850e2eba6
2016-03-22 09:57:30 +11:00
SamYaple
5b6716cee8 Use fstrim to prep the block device
This cuts the image size down alot, esspecially if there were lots of
small file deletes.

The fstrim utility is in the util-linux package and should be on
most all systems. fstrim also works with XFS, ext4, btrfs, etc
prodiving the kernel is new enough.

A reduction of 25% or more in size is common.

Change-Id: I269b4416be450369616f9b8e030f84c30e329804
2016-03-13 16:24:59 +00:00
Ian Wienand
fa3c5e3056 Add image size report
In the common case of not specifying a size, we are already running
"du" over the image to figure out how big it is.  Leverage that by
saving it's output and displaying a pruned list of big files when
requested.

We add a flag to show a summarised option (files >10MiB) and another
to show full output, should you wish that level of detail.

"Invocation" documentation is updated (and formatted a little better
while we're here).

Change-Id: I255800790a62fed1c82fcd311f1cc29c9867766d
2016-03-08 13:58:40 +11:00
Colleen Murphy
76a8d3704b Add --version option to disk-image-create
Being able to discover DIB's version from the command itself is
convenient. This patch adds a --version option to the disk-image-create
command, failing gracefully if diskimage-builder is not installed.

This adds an explicit dependency on pbr to the requirements since this
is required to run diskimage_builder/version.py outside of a test
environment.

This patch consciously chooses to only provide the long-form option
and no '-v' to allow for the future possibility that a '-v' might
indicate '--verbose' in the future.

Change-Id: I9fc084774d6c7a39a944b07680b3eb8be8e34f9c
2016-02-29 16:13:01 -08:00
Jenkins
6f6a096d94 Merge "Handle install with pip -e" 2016-02-12 20:28:51 +00:00
Matthew Thode
01fce7b70c
Fix Gentoo hardened support
This checks the profile, if it has hardened in it's name it needs xattr support
unfortunately xattr support cannot yet be relied on everywhere, so it needs to
be disabled for hardened profile builds to correctly pax-mark.

Change-Id: I7fb855249a9e6c9b6497ab5061b4ea3c014f5081
Closes-Bug: 1537177
2016-02-01 20:56:37 -06:00
Jenkins
c8c14edfe3 Merge "Support building ACIs" 2015-12-18 13:53:38 +00:00
Michael Johnson
590dce8dcc Fix diskimage-builder image size
This patch fixes the calculation of the resultant image size
when building an image with diskimage-builder on ext4 a
filesystem.

Prior to this, using the '--image-size 2' (2GB) setting would
generate an image that would not boot under a 2GB nova flavor.

Change-Id: I7a753bdef84c6300ccea73ae4a92bf330dcd77cb
Closes-Bug: #1513622
2015-11-09 16:20:18 +00:00
Jenkins
4d986ce3a5 Merge "Enable decimal value for $DIB_IMAGE_SIZE." 2015-11-02 20:13:22 +00:00
yogananth subramanian
f84b910f15 Enable decimal value for $DIB_IMAGE_SIZE.
Patch adds support for using decimal values for $DIB_IMAGE_SIZE.
This allows for creating images that are <1G in size.

Change-Id: I945644a8e77fecfb0b83efa282dc00bb29514e0b
Closes-Bug: #1366909
2015-10-13 15:51:27 +00:00
Ian Wienand
ab5ed610e4 Handle install with pip -e
As described in the comments, inspect the installation to see if we
have been installed with "pip -e" and, if so, make sure we reference
the scripts from the source location rather than the
system-installations.

Update the documentation with a terse but helpful quick-start to show
an easy way to start developing a change using this.

Closes-Bug: #1491035
Change-Id: I0460061b834a2b854175f8c9be2be8d38c540c9d
2015-09-10 16:55:39 +10:00
Gregory Haynes
eb6a260a5a Support building ACIs
App containers are a format used by rocket and are specified at the
following url:
https://github.com/appc/spec/

Change-Id: I8ac24f0194c4bf53dffd6c47e0587bc413101698
2015-07-22 09:30:09 +00:00
Jenkins
d748aa17d2 Merge "Add docker output support" 2015-07-19 11:40:43 +00:00
Monty Taylor
1187f9d5df Add docker output support
We can already produce tarballs, which is the input format docker import
expects. This makes it trivial to add docker as an output format.

Co-Authored-By: Dan Prince <dprince@redhat.com>

Change-Id: Ib60db3b717d33d4cf3181d70fe0ffbfa86fd5d02
2015-07-19 10:20:39 +00:00
Pino Toscano
987a832351 Cleanup the build directories earlier
Split the cleanup_dirs function in two, i.e. cleanup of the build dir
and the image dir, and use the former to cleanup the temporary build
subdirs after their unmount, before the conversion to other disk
formats; they are not needed anyway at that point, and allows to save
disk space during the conversion phase.

Change-Id: Ie30d7e6033613d6979148423326ae7e17a7342e7
2015-06-04 10:46:18 +02:00
James Slagle
82b328bb13 Fix $DIB_DEFAULT_INSTALLTYPE export statement
$DIB_DEFAULT_INSTALLTYPE is propogated into the chroot by writing out an
export statement to a file under environment.d. However, that export
statement does not account for other elements that may choose to define
$DIB_DEFAULT_INSTALLTYPE earlier on in their own environment.d files.

Change-Id: I5865b1859c080c328251e8dce072c0634ef75c16
2015-05-01 16:37:56 -04:00
Gregory Haynes
8266e14adc Add functional smoke test for disk-image-create
We do not have any testing inside DIB for testing disk-image-create
logic. Lets do some smoke testing for all our supported image formats.

Also adding a run_functests.sh so we can extend this later without editing
the jenkins job.

Change-Id: Ie491e27f00bde54f73af6b47c9696ec04d973b14
2015-04-24 16:37:29 +00:00
Jenkins
44d6d5cd88 Merge "Support VHD output format" 2015-04-22 05:31:04 +00:00
Gregory Haynes
361a7a60b9 Fix image size to fit filesystem journal
Our logic to determine needed image size doesnt fully account for the
in-filesystem journal. This only shows up when creating images that are
very small relative to the FS journal size.

Change-Id: Ic3c2bcd31ec4fee6bcd9f67767842eb3fbe20d3a
2015-04-20 23:26:12 +00:00
Jenkins
e9ac9c9823 Merge "Open MKFS_OPTS for extension in disk-image-create" 2015-04-20 19:08:44 +00:00
Gregory Haynes
7937fb5392 Support VHD output format
This useful for building images for xen.

Change-Id: Ib716bd7fc1ed979968ea92f46f1644038e40df66
2015-04-14 13:39:18 -04:00
Gregory Haynes
7347577784 Correctly handle raw type ordering
We need to run raw outputting last, and were incorrectly detecting that
image type.

Change-Id: Idfbe46c34e783046b940d3eb28c846a0cf9ac0a6
2015-04-01 20:28:44 +00:00
Ethan Gafford
199196e399 Open MKFS_OPTS for extension in disk-image-create
At present, MKFS_OPTS is closed for modification. The ability
to extend the set of MKFS_OPTS adds a great deal of power for
knowledgeable end-users. (And in some specific circumstances,
it is vital to success, as in the case of building RHEL/CentOS
6 images from RHEL/CentOS 7 hosts, in which case -O ^64bit is
required in order for the image to boot.)

Change-Id: I714e86a5a413779e63f598fbbb5a79d23cf6d8c3
2015-03-26 14:36:08 -04:00
Gregory Haynes
2d79e9d395 Short circuit qemu-img convert for raw images
We currently use qemu-img convert with a raw source and dest when
building raw images. We can just mv the file for increased speed.

Change-Id: I3da095cb9ecad7224a121a434a9fb204132bf6df
2015-03-25 22:50:42 +00:00
Ian Wienand
36b59c001c Standarise tracing for scripts
There is a wide variety of tracing options through the various shell
scripts.  Some use "set -eux", others explicity set xtrace and others
do nothing.  There is a "-x" option to bin/disk-image-create but it
doesn't flow down to the many scripts it calls.

This adds a global integer variable set by disk-image-create
DIB_DEBUG_TRACE.  All scripts have a stanza added to detect this and
turn on tracing.  Any other tracing methods are rolled into this.  So
the standard header is

---
if [ "${DIB_DEBUG_TRACE:-0}" -gt 0 ]; then
    set -x
fi
set -eu
set -o pipefail
---

Multiple -x options can be specified to dib-create-image, which
increases the value of DIB_DEBUG_TRACE.  If script authors feel their
script should only trace at higher levels, they should modify the
"-gt" value.  If they feel it should trace by default, they can modify
the default value also.

Changes to pachset 16 : scripts which currently trace themselves by
default have retained this behaviour with DIB_DEBUG_TRACE defaulting
to "1".  This was done by running [1] on patch set 15.  See the thread
beginning at [2]

dib-lint is also updated to look for the variable being matched.

[1] https://gist.github.com/ianw/71bbda9e6acc74ccd0fd
[2] http://lists.openstack.org/pipermail/openstack-dev/2014-November/051575.html

Change-Id: I6c5a962260741dcf6f89da9a33b96372a719b7b0
2015-02-12 10:41:32 +11:00
Michael Kerrin
6337e6fe17 Fix issue with leaking /tmp/image.* directories
When building the ramdisk we don't cleanup the temporary
directories after ourselves. This leaves /tmp/image.* directories
mounted and /tmp/image.* directories on the system.

Also the ramdisk-functions duplicate, from what I can see,
the cleanup function from common-functions. So when a job
is killed off it ends up leaving /tmp/image.* directories
on the system.

Change-Id: I2d73aabd0eb176027b4e7368580db08902e2b6ab
2015-01-28 12:07:07 +00:00
Gregory Haynes
c94880aa2f Fail helpfully if uuidgen is missing
If uuidgen is not installed we currently fail when performing tune2fs.

Change-Id: I4062eb1d62322c455dac5b83b856f454da211320
2014-12-15 15:42:35 -08:00
Ben Nemec
ff2a89215e Improve --ramdisk-element docs
Unlike most of the options to disk-image-create, this one didn't
document its default value, and it also didn't have any context for
why one might use it.

Change-Id: I57ea529cf387ab8ec8497d35be19ac64198df47a
Related-Bug: 1397073
2014-12-01 10:21:13 -06:00
Frederic Lepied
ccd7811dc0 disk-image-create: fix test when no argument is given.
Avoid a warning when multiple arguments are passed to disk-image-create
like:

bin/disk-image-create: 146: [: aa: unexpected operator

Change-Id: I619d7bf2e4b56885d629019b80b249bc80b8f43e
2014-11-25 22:28:01 +01:00
Gregory Haynes
9bb7ace03c Fail helpfully if no elements are specified
Instead of charging ahead and failing because we lack dib-run-parts we
can fail in this case with a useful error message.

Change-Id: Ie10176b5a7019a0385796984ebbaf7742876925a
2014-11-21 10:31:11 -08:00
Yuriy Zveryanskyy
676b1b738d Add element for building ramdisk with ironic-python-agent
This element creates kernel and ramdisk files based on Fedora,
example:
disk-image-create -a i386 -o test fedora ironic-agent disable-selinux

Change-Id: Ifa133d1680b81cb87d32a405aa7d7b40fe91f835
2014-11-13 19:30:44 +02:00
Dan Prince
022cb2c1c2 Support installing packages by default
Adds new disk-image-create --install-type option which
can be used to enable alternate install types. The
default install type is 'source'.

The motivation is to eliminate elements like
enable-package-installs which require coupling and also
don't work with elements in multiple element repositories.

This patch does not prevent you from using the previous
DIB_INSTALLTYPE_ variables to customize the install type,
rather it just changes the default so you don't have to
set it so often when using things like packages.

Change-Id: Icee98440fc2251728f2dca30e7c4789a0fd89b93
2014-10-28 15:05:53 -04:00
Jenkins
47502b8042 Merge "export DIB_ROOT_LABEL to make it global" 2014-10-28 15:51:19 +00:00
Ghe Rivero
48ce0fde51 export DIB_ROOT_LABEL to make it global
It is needed by the debian element in the root.d phase. (enviroment.d variables are not available after the base image is built)

Change-Id: Idb8f9795a619687569bcb0de774c87333d11b292
Closes-Bug: #1384103
2014-10-23 07:38:21 +00:00
Ramakrishnan G
13aed64e97 iso element to build bootable ISO images
This commit adds a new element named 'iso' to build a bootable
ISO image for the kernel/ramdisk emitted by the 'baremetal' or
'ramdisk' element.

Change-Id: I89d175a29e2d0bc64b47fe527f0d0f6875f6849a
2014-10-21 14:37:57 +05:30
Jenkins
bb07de7fe9 Merge "Enable dracut deploy ramdisks" 2014-10-20 11:03:54 +00:00
Ben Nemec
eed30adc48 Enable dracut deploy ramdisks
The element builds dracut from source on Ubuntu because the
Ubuntu dracut package is broken and very old, so it can't be
installed properly and causes a number of other issues that
are fixed by using a newer version of Dracut.

This initial version should work in virtualized environments.
Further validation of its suitability for real baremetal
deployments will need to be done in the future, but this should
be sufficient to enable that work.

Regarding Dracut specifically, in order to limit the changes
needed in the existing scripts this element continues to use a
cut down version of the /init script that we were building for the
existing ramdisk.  However, instead of running it as pid 0 it is
run as a Dracut pre-mount hook.  This allows Dracut to set up all
of the hardware and system bits, while falling early enough in the
Dracut sequence to complete the deployment before Dracut would try
to boot off the hard disk.

bp tripleo-juno-dracut-ramdisks
Change-Id: I144c8993fe040169f440bd4f7a428fdbe3d745cf
2014-10-17 21:44:09 -05:00
Jenkins
2b60bea961 Merge "Allow for multiple image outputs from raw source" 2014-10-16 21:56:28 +00:00
Ghe Rivero
873de01491 Remove first-boot.d support
After being deprecated two releases ago, finally remove any reference
for the support of first-boot.d

Change-Id: I08d67404ef48cad61db3b18fb86e970abfa5d2b6
2014-10-16 09:48:14 +00:00
Clark Boylan
ae928057bd Allow for multiple image outputs from raw source
When uploading images to multiple clouds it is possible that the same
image will be needed in multiple formats to accomodate hypervisors
across clouds. Update disk-image-create's -t flag to take a list of
desired output image formats so that a single disk-image-create can
output all of the desired image formats.

Change-Id: If121b2342ae888855ba435aa3189f039e985b812
2014-10-13 09:16:42 -07:00
Ben Nemec
dde3d24213 Allow custom rootfs labels
Per [1], our current root label of cloudimg-root does not work with
XFS because XFS only allows 12 character labels.  This change
addresses that by allowing a custom rootfs label to be specified
in the call to disk-image-create.  There should be no backwards
compatibility concerns as the default label is unchanged.  Any
external elements dealing with the label would need to be updated
to support this new feature, but should continue to work as before
as long as a custom label is not specified.

[1]: https://bugzilla.redhat.com/show_bug.cgi?id=1139584

Change-Id: I596104d1a63b5dc6549e8460a1ae3da00165ef04
2014-09-26 17:44:08 -05:00
Jenkins
7d0e7fe5ad Merge "Centralize handling of /lost+found" 2014-08-28 16:33:12 +00:00
Pino Toscano
267e96f52c Centralize handling of /lost+found
Just remove globally /lost+found after the root and install phases, so
distribution elements don't need to do that by themselves.

Change-Id: Ic783e613bd8794aefd3f40c9a7c308d14cd04b8d
2014-08-04 11:48:21 +02:00