Commit Graph

175 Commits

Author SHA1 Message Date
SamYaple
0c32784663 Revert "Zerofree the image if possible"
This reverts commit 5184d02a7c.

The decision was made to go with fstrim because it is faster and more
universal that zerofree. The related-id has the patchset that implements
fstrim.

Related-Id: I269b4416be450369616f9b8e030f84c30e329804
Change-Id: If40cf2fc0ecd8686768cbfeac9ecee90907674e7
2016-03-10 20:38:01 +00:00
Clark Boylan
5184d02a7c Zerofree the image if possible
If the image has an ext filesystem and the zerofree utility is present
on the build system then run zerofree. This should make images as
compressable as possible which is a nice feature when building
compressed qcow2 images.

Change-Id: Ia6062c291f7a3f58b85a4f408ecb3d0574c65d53
2016-03-07 17:45:14 -08:00
Jenkins
6f6a096d94 Merge "Handle install with pip -e" 2016-02-12 20:28:51 +00:00
Jenkins
b13512a7f4 Merge "Only match #!/bin/bash in scripts" 2016-02-10 08:35:09 +00:00
Matthew Thode
01fce7b70c
Fix Gentoo hardened support
This checks the profile, if it has hardened in it's name it needs xattr support
unfortunately xattr support cannot yet be relied on everywhere, so it needs to
be disabled for hardened profile builds to correctly pax-mark.

Change-Id: I7fb855249a9e6c9b6497ab5061b4ea3c014f5081
Closes-Bug: 1537177
2016-02-01 20:56:37 -06:00
Ian Wienand
d8abe72537 Only match #!/bin/bash in scripts
Our dib-lint checking is only considering scripts with #!/bin/bash.
While there's nothing really wrong with some other shebang line like
"#!/usr/bin/env bash" let's keep things consistent.

We can use the same regex match to reduce a few forks in the main
checking.

Also a minor cleanup to the file matching

Change-Id: I609721b2671e704ea26075dad7e5b39a8b858f6b
2016-01-29 15:49:10 +11:00
Jenkins
c8c14edfe3 Merge "Support building ACIs" 2015-12-18 13:53:38 +00:00
Jenkins
3412ced2f0 Merge "dib-lint: ignore blank lines in element ordering" 2015-11-10 00:32:04 +00:00
Michael Johnson
590dce8dcc Fix diskimage-builder image size
This patch fixes the calculation of the resultant image size
when building an image with diskimage-builder on ext4 a
filesystem.

Prior to this, using the '--image-size 2' (2GB) setting would
generate an image that would not boot under a 2GB nova flavor.

Change-Id: I7a753bdef84c6300ccea73ae4a92bf330dcd77cb
Closes-Bug: #1513622
2015-11-09 16:20:18 +00:00
Jenkins
4d986ce3a5 Merge "Enable decimal value for $DIB_IMAGE_SIZE." 2015-11-02 20:13:22 +00:00
yogananth subramanian
f84b910f15 Enable decimal value for $DIB_IMAGE_SIZE.
Patch adds support for using decimal values for $DIB_IMAGE_SIZE.
This allows for creating images that are <1G in size.

Change-Id: I945644a8e77fecfb0b83efa282dc00bb29514e0b
Closes-Bug: #1366909
2015-10-13 15:51:27 +00:00
Ian Wienand
9189e24f6d dib-lint: ignore blank lines in element ordering
Some of the elements-deps in the project-config repo have a blank line
at the end, which throws out the ordering.  Strip blank lines, like
comment lines, before processing.

As an additional help, show a side-by-side diff of what is provided
versus what is expected when showing an error about sorting.

Change-Id: I007851ee01d6853ad992ce4437331e8bd79bbfce
2015-10-06 14:02:10 +11:00
Ian Wienand
e678910166 Output failing lines when dib-lint finds wrong indents
When dib-lint complains about wrong indents, it doesn't give you any
indication where the problem is.  This repeats the grep on failure,
outputting the line and line-number.

As a bonus, skip *.orig files from merges

Change-Id: Ifbbdf854ea19191f66e9823468dbc0afc2f93e1f
2015-09-24 21:03:36 +10:00
Pino Toscano
aeab45ad7f dib-lint: validate json/yaml files
Look for files .yaml and pkg-map configurations, and try to load them
either as json or yaml. This way, invalid ones can be detected before
they are committed unnoticed.

Also, exclude .yaml files from being searched while checking bash and
python scripts.

Change-Id: I2478837cfe66929ae1b0d7dd96e049773a35e11c
2015-09-15 12:48:23 +02:00
Ian Wienand
ab5ed610e4 Handle install with pip -e
As described in the comments, inspect the installation to see if we
have been installed with "pip -e" and, if so, make sure we reference
the scripts from the source location rather than the
system-installations.

Update the documentation with a terse but helpful quick-start to show
an easy way to start developing a change using this.

Closes-Bug: #1491035
Change-Id: I0460061b834a2b854175f8c9be2be8d38c540c9d
2015-09-10 16:55:39 +10:00
Gregory Haynes
eb6a260a5a Support building ACIs
App containers are a format used by rocket and are specified at the
following url:
https://github.com/appc/spec/

Change-Id: I8ac24f0194c4bf53dffd6c47e0587bc413101698
2015-07-22 09:30:09 +00:00
Jenkins
d748aa17d2 Merge "Add docker output support" 2015-07-19 11:40:43 +00:00
Monty Taylor
1187f9d5df Add docker output support
We can already produce tarballs, which is the input format docker import
expects. This makes it trivial to add docker as an output format.

Co-Authored-By: Dan Prince <dprince@redhat.com>

Change-Id: Ib60db3b717d33d4cf3181d70fe0ffbfa86fd5d02
2015-07-19 10:20:39 +00:00
Jenkins
cdf4467a8a Merge "Cleanup the build directories earlier" 2015-07-14 11:50:53 +00:00
Jenkins
7aab6c63c2 Merge "Remove deprecated disk-image-get-kernel" 2015-07-07 01:37:51 +00:00
Gregory Haynes
8cf37064a5 Remove deprecated disk-image-get-kernel
This script has been deprecated in favor of the baremetal element for
some time. Lets remove it.

Change-Id: Idd7db766a039ba8e74f0e98e74eb27246e753f4b
2015-07-06 16:44:07 +00:00
Antoine Musso
35584efa90 dib-lint: make it work on Mac OS X
Pass an explicit template to mktemp since that is required on Mac OS X.
Hardcoded to /tmp since I did not want to bother validating TMPDIR which
is user supplied.

Explicit define a repeating group which causes a grep error:
    repetition-operator operand invalid

Change-Id: Ib23cc15db7f7dfdd611280884e46342562d545df
2015-06-18 11:12:01 +02:00
Pino Toscano
987a832351 Cleanup the build directories earlier
Split the cleanup_dirs function in two, i.e. cleanup of the build dir
and the image dir, and use the former to cleanup the temporary build
subdirs after their unmount, before the conversion to other disk
formats; they are not needed anyway at that point, and allows to save
disk space during the conversion phase.

Change-Id: Ie30d7e6033613d6979148423326ae7e17a7342e7
2015-06-04 10:46:18 +02:00
Ian Wienand
ab05a8b7e2 Skip backups and other non-relevant files for dib-lint
I noticed dib-lint was running on my editor backup files and checking
python files (covered by flake8) and RST files

Change-Id: Ide2e8a4c1aa8d619bc4b0326a2cfc5e63c74475f
2015-05-29 15:39:46 +10:00
James Slagle
82b328bb13 Fix $DIB_DEFAULT_INSTALLTYPE export statement
$DIB_DEFAULT_INSTALLTYPE is propogated into the chroot by writing out an
export statement to a file under environment.d. However, that export
statement does not account for other elements that may choose to define
$DIB_DEFAULT_INSTALLTYPE earlier on in their own environment.d files.

Change-Id: I5865b1859c080c328251e8dce072c0634ef75c16
2015-05-01 16:37:56 -04:00
Gregory Haynes
8266e14adc Add functional smoke test for disk-image-create
We do not have any testing inside DIB for testing disk-image-create
logic. Lets do some smoke testing for all our supported image formats.

Also adding a run_functests.sh so we can extend this later without editing
the jenkins job.

Change-Id: Ie491e27f00bde54f73af6b47c9696ec04d973b14
2015-04-24 16:37:29 +00:00
Jenkins
44d6d5cd88 Merge "Support VHD output format" 2015-04-22 05:31:04 +00:00
Gregory Haynes
361a7a60b9 Fix image size to fit filesystem journal
Our logic to determine needed image size doesnt fully account for the
in-filesystem journal. This only shows up when creating images that are
very small relative to the FS journal size.

Change-Id: Ic3c2bcd31ec4fee6bcd9f67767842eb3fbe20d3a
2015-04-20 23:26:12 +00:00
Jenkins
e9ac9c9823 Merge "Open MKFS_OPTS for extension in disk-image-create" 2015-04-20 19:08:44 +00:00
Gregory Haynes
7937fb5392 Support VHD output format
This useful for building images for xen.

Change-Id: Ib716bd7fc1ed979968ea92f46f1644038e40df66
2015-04-14 13:39:18 -04:00
Jenkins
65ad6377a6 Merge "No markdown docs for elements" 2015-04-09 13:59:32 +00:00
Gregory Haynes
cc308464c6 No markdown docs for elements
Sphinx does not support markdown, therefore all our element docs should
not be in this format.

Change-Id: I6fceb5c2c218e94a463f13d6d9050aea485e6c31
2015-04-02 23:55:19 +00:00
Gregory Haynes
7347577784 Correctly handle raw type ordering
We need to run raw outputting last, and were incorrectly detecting that
image type.

Change-Id: Idfbe46c34e783046b940d3eb28c846a0cf9ac0a6
2015-04-01 20:28:44 +00:00
Ethan Gafford
199196e399 Open MKFS_OPTS for extension in disk-image-create
At present, MKFS_OPTS is closed for modification. The ability
to extend the set of MKFS_OPTS adds a great deal of power for
knowledgeable end-users. (And in some specific circumstances,
it is vital to success, as in the case of building RHEL/CentOS
6 images from RHEL/CentOS 7 hosts, in which case -O ^64bit is
required in order for the image to boot.)

Change-Id: I714e86a5a413779e63f598fbbb5a79d23cf6d8c3
2015-03-26 14:36:08 -04:00
Gregory Haynes
2d79e9d395 Short circuit qemu-img convert for raw images
We currently use qemu-img convert with a raw source and dest when
building raw images. We can just mv the file for increased speed.

Change-Id: I3da095cb9ecad7224a121a434a9fb204132bf6df
2015-03-25 22:50:42 +00:00
Ian Wienand
36b59c001c Standarise tracing for scripts
There is a wide variety of tracing options through the various shell
scripts.  Some use "set -eux", others explicity set xtrace and others
do nothing.  There is a "-x" option to bin/disk-image-create but it
doesn't flow down to the many scripts it calls.

This adds a global integer variable set by disk-image-create
DIB_DEBUG_TRACE.  All scripts have a stanza added to detect this and
turn on tracing.  Any other tracing methods are rolled into this.  So
the standard header is

---
if [ "${DIB_DEBUG_TRACE:-0}" -gt 0 ]; then
    set -x
fi
set -eu
set -o pipefail
---

Multiple -x options can be specified to dib-create-image, which
increases the value of DIB_DEBUG_TRACE.  If script authors feel their
script should only trace at higher levels, they should modify the
"-gt" value.  If they feel it should trace by default, they can modify
the default value also.

Changes to pachset 16 : scripts which currently trace themselves by
default have retained this behaviour with DIB_DEBUG_TRACE defaulting
to "1".  This was done by running [1] on patch set 15.  See the thread
beginning at [2]

dib-lint is also updated to look for the variable being matched.

[1] https://gist.github.com/ianw/71bbda9e6acc74ccd0fd
[2] http://lists.openstack.org/pipermail/openstack-dev/2014-November/051575.html

Change-Id: I6c5a962260741dcf6f89da9a33b96372a719b7b0
2015-02-12 10:41:32 +11:00
Michael Kerrin
6337e6fe17 Fix issue with leaking /tmp/image.* directories
When building the ramdisk we don't cleanup the temporary
directories after ourselves. This leaves /tmp/image.* directories
mounted and /tmp/image.* directories on the system.

Also the ramdisk-functions duplicate, from what I can see,
the cleanup function from common-functions. So when a job
is killed off it ends up leaving /tmp/image.* directories
on the system.

Change-Id: I2d73aabd0eb176027b4e7368580db08902e2b6ab
2015-01-28 12:07:07 +00:00
Jenkins
9981f94828 Merge "Continue past dependency ordering diffs." 2015-01-08 10:20:16 +00:00
Gregory Haynes
c94880aa2f Fail helpfully if uuidgen is missing
If uuidgen is not installed we currently fail when performing tune2fs.

Change-Id: I4062eb1d62322c455dac5b83b856f454da211320
2014-12-15 15:42:35 -08:00
Chris Jones
f4a2eb6ef3 Continue past dependency ordering diffs.
We now guard against diff returning non-zero (which it does when it
discovers changes). This allows dib-lint runs to continue past the point
of displaying a dependency ordering nit.

Change-Id: Ib7801597400d6d87146181935922c8832baf5aaa
2014-12-12 18:36:55 +00:00
Jenkins
2dfb931d95 Merge "Use env python" 2014-12-11 21:26:03 +00:00
Abel Lopez
fc862f460c Use env python
Change the shebang from /usr/bin/python to /usr/bin/env python
Some users, especially on older systems don't use the python
supplied with the OS. This has an impact with module loading if
the python in /usr/bin is 2.6, and user wants to use 2.7
in /usr/local/bin, for example

Change-Id: Ib31b369a94c33fb7508f0539c2b2f14177e507e0
2014-12-08 13:42:16 -08:00
Ben Nemec
ff2a89215e Improve --ramdisk-element docs
Unlike most of the options to disk-image-create, this one didn't
document its default value, and it also didn't have any context for
why one might use it.

Change-Id: I57ea529cf387ab8ec8497d35be19ac64198df47a
Related-Bug: 1397073
2014-12-01 10:21:13 -06:00
Frederic Lepied
ccd7811dc0 disk-image-create: fix test when no argument is given.
Avoid a warning when multiple arguments are passed to disk-image-create
like:

bin/disk-image-create: 146: [: aa: unexpected operator

Change-Id: I619d7bf2e4b56885d629019b80b249bc80b8f43e
2014-11-25 22:28:01 +01:00
Jenkins
9fae41c898 Merge "Fail helpfully if no elements are specified" 2014-11-25 12:29:06 +00:00
Gregory Haynes
9bb7ace03c Fail helpfully if no elements are specified
Instead of charging ahead and failing because we lack dib-run-parts we
can fail in this case with a useful error message.

Change-Id: Ie10176b5a7019a0385796984ebbaf7742876925a
2014-11-21 10:31:11 -08:00
Gregory Haynes
fd0708286c Check python with flake8 instead of dib-lint
We check python files with dib-lint rather than flake8 which have
conflicting opinions. This means weve been (forcibly) writing non pep8
python.

Also fixing pep8 issues so tests pass.

Change-Id: Idc9db40334f6e15738a7802c06697270df68741c
2014-11-21 09:04:52 -08:00
Yuriy Zveryanskyy
676b1b738d Add element for building ramdisk with ironic-python-agent
This element creates kernel and ramdisk files based on Fedora,
example:
disk-image-create -a i386 -o test fedora ironic-agent disable-selinux

Change-Id: Ifa133d1680b81cb87d32a405aa7d7b40fe91f835
2014-11-13 19:30:44 +02:00
Jenkins
ce67ce87ea Merge "Support installing packages by default" 2014-11-13 10:27:56 +00:00
Ryan Brady
6fedbc58f9 Fix indent exclusion
The current dib-lint script has an exclusion for indent but it is
erroneously used as a label around the tab indention check.  This
patch renames the tab indention exclusion and adds the indent
exclusion where it belongs.

Change-Id: I701595e7a08f4764a45c6701d67cbb931c3d47f3
2014-11-11 21:06:25 -05:00