We do not have any testing inside DIB for testing disk-image-create
logic. Lets do some smoke testing for all our supported image formats.
Also adding a run_functests.sh so we can extend this later without editing
the jenkins job.
Change-Id: Ie491e27f00bde54f73af6b47c9696ec04d973b14
Our logic to determine needed image size doesnt fully account for the
in-filesystem journal. This only shows up when creating images that are
very small relative to the FS journal size.
Change-Id: Ic3c2bcd31ec4fee6bcd9f67767842eb3fbe20d3a
At present, MKFS_OPTS is closed for modification. The ability
to extend the set of MKFS_OPTS adds a great deal of power for
knowledgeable end-users. (And in some specific circumstances,
it is vital to success, as in the case of building RHEL/CentOS
6 images from RHEL/CentOS 7 hosts, in which case -O ^64bit is
required in order for the image to boot.)
Change-Id: I714e86a5a413779e63f598fbbb5a79d23cf6d8c3
We currently use qemu-img convert with a raw source and dest when
building raw images. We can just mv the file for increased speed.
Change-Id: I3da095cb9ecad7224a121a434a9fb204132bf6df
There is a wide variety of tracing options through the various shell
scripts. Some use "set -eux", others explicity set xtrace and others
do nothing. There is a "-x" option to bin/disk-image-create but it
doesn't flow down to the many scripts it calls.
This adds a global integer variable set by disk-image-create
DIB_DEBUG_TRACE. All scripts have a stanza added to detect this and
turn on tracing. Any other tracing methods are rolled into this. So
the standard header is
---
if [ "${DIB_DEBUG_TRACE:-0}" -gt 0 ]; then
set -x
fi
set -eu
set -o pipefail
---
Multiple -x options can be specified to dib-create-image, which
increases the value of DIB_DEBUG_TRACE. If script authors feel their
script should only trace at higher levels, they should modify the
"-gt" value. If they feel it should trace by default, they can modify
the default value also.
Changes to pachset 16 : scripts which currently trace themselves by
default have retained this behaviour with DIB_DEBUG_TRACE defaulting
to "1". This was done by running [1] on patch set 15. See the thread
beginning at [2]
dib-lint is also updated to look for the variable being matched.
[1] https://gist.github.com/ianw/71bbda9e6acc74ccd0fd
[2] http://lists.openstack.org/pipermail/openstack-dev/2014-November/051575.html
Change-Id: I6c5a962260741dcf6f89da9a33b96372a719b7b0
When building the ramdisk we don't cleanup the temporary
directories after ourselves. This leaves /tmp/image.* directories
mounted and /tmp/image.* directories on the system.
Also the ramdisk-functions duplicate, from what I can see,
the cleanup function from common-functions. So when a job
is killed off it ends up leaving /tmp/image.* directories
on the system.
Change-Id: I2d73aabd0eb176027b4e7368580db08902e2b6ab
We now guard against diff returning non-zero (which it does when it
discovers changes). This allows dib-lint runs to continue past the point
of displaying a dependency ordering nit.
Change-Id: Ib7801597400d6d87146181935922c8832baf5aaa
Change the shebang from /usr/bin/python to /usr/bin/env python
Some users, especially on older systems don't use the python
supplied with the OS. This has an impact with module loading if
the python in /usr/bin is 2.6, and user wants to use 2.7
in /usr/local/bin, for example
Change-Id: Ib31b369a94c33fb7508f0539c2b2f14177e507e0
Unlike most of the options to disk-image-create, this one didn't
document its default value, and it also didn't have any context for
why one might use it.
Change-Id: I57ea529cf387ab8ec8497d35be19ac64198df47a
Related-Bug: 1397073
Avoid a warning when multiple arguments are passed to disk-image-create
like:
bin/disk-image-create: 146: [: aa: unexpected operator
Change-Id: I619d7bf2e4b56885d629019b80b249bc80b8f43e
Instead of charging ahead and failing because we lack dib-run-parts we
can fail in this case with a useful error message.
Change-Id: Ie10176b5a7019a0385796984ebbaf7742876925a
We check python files with dib-lint rather than flake8 which have
conflicting opinions. This means weve been (forcibly) writing non pep8
python.
Also fixing pep8 issues so tests pass.
Change-Id: Idc9db40334f6e15738a7802c06697270df68741c
This element creates kernel and ramdisk files based on Fedora,
example:
disk-image-create -a i386 -o test fedora ironic-agent disable-selinux
Change-Id: Ifa133d1680b81cb87d32a405aa7d7b40fe91f835
The current dib-lint script has an exclusion for indent but it is
erroneously used as a label around the tab indention check. This
patch renames the tab indention exclusion and adds the indent
exclusion where it belongs.
Change-Id: I701595e7a08f4764a45c6701d67cbb931c3d47f3
Adds new disk-image-create --install-type option which
can be used to enable alternate install types. The
default install type is 'source'.
The motivation is to eliminate elements like
enable-package-installs which require coupling and also
don't work with elements in multiple element repositories.
This patch does not prevent you from using the previous
DIB_INSTALLTYPE_ variables to customize the install type,
rather it just changes the default so you don't have to
set it so often when using things like packages.
Change-Id: Icee98440fc2251728f2dca30e7c4789a0fd89b93
It is needed by the debian element in the root.d phase. (enviroment.d variables are not available after the base image is built)
Change-Id: Idb8f9795a619687569bcb0de774c87333d11b292
Closes-Bug: #1384103
This commit adds a new element named 'iso' to build a bootable
ISO image for the kernel/ramdisk emitted by the 'baremetal' or
'ramdisk' element.
Change-Id: I89d175a29e2d0bc64b47fe527f0d0f6875f6849a
The element builds dracut from source on Ubuntu because the
Ubuntu dracut package is broken and very old, so it can't be
installed properly and causes a number of other issues that
are fixed by using a newer version of Dracut.
This initial version should work in virtualized environments.
Further validation of its suitability for real baremetal
deployments will need to be done in the future, but this should
be sufficient to enable that work.
Regarding Dracut specifically, in order to limit the changes
needed in the existing scripts this element continues to use a
cut down version of the /init script that we were building for the
existing ramdisk. However, instead of running it as pid 0 it is
run as a Dracut pre-mount hook. This allows Dracut to set up all
of the hardware and system bits, while falling early enough in the
Dracut sequence to complete the deployment before Dracut would try
to boot off the hard disk.
bp tripleo-juno-dracut-ramdisks
Change-Id: I144c8993fe040169f440bd4f7a428fdbe3d745cf
After being deprecated two releases ago, finally remove any reference
for the support of first-boot.d
Change-Id: I08d67404ef48cad61db3b18fb86e970abfa5d2b6
When uploading images to multiple clouds it is possible that the same
image will be needed in multiple formats to accomodate hypervisors
across clouds. Update disk-image-create's -t flag to take a list of
desired output image formats so that a single disk-image-create can
output all of the desired image formats.
Change-Id: If121b2342ae888855ba435aa3189f039e985b812
Per [1], our current root label of cloudimg-root does not work with
XFS because XFS only allows 12 character labels. This change
addresses that by allowing a custom rootfs label to be specified
in the call to disk-image-create. There should be no backwards
compatibility concerns as the default label is unchanged. Any
external elements dealing with the label would need to be updated
to support this new feature, but should continue to work as before
as long as a custom label is not specified.
[1]: https://bugzilla.redhat.com/show_bug.cgi?id=1139584
Change-Id: I596104d1a63b5dc6549e8460a1ae3da00165ef04
Now that dib-run-parts has been moved to the dib-utils project, we
need to update diskimage-builder to use it instead of the version
directly in diskimage-builder.
This change removes the old copy of the dib-run-parts script in
the element, adds dib-utils as a dependency of diskimage-builder,
and updates the uses of dib-run-parts to correctly handle the fact
that it is now external to the project.
Requires I0be1f876d0e4a7d38e0d5c6010a552a8ebb158a4
Change-Id: Ia0a0df7784a14c49b5c47ac0b03e6c2602c84b3b
Just remove globally /lost+found after the root and install phases, so
distribution elements don't need to do that by themselves.
Change-Id: Ic783e613bd8794aefd3f40c9a7c308d14cd04b8d
This patch adds exclusions for the indentation and newline
rules in dib-lint. The indentation exclusion is needed for
files using heredoc and wanting to preserve indentation.
The newline exclusion is added here for consisently treating
all dib-lint rules the same.
Change-Id: Ic8a2164eba3cf99e55473846e3a5fe663a3c1095
This will be useful for adopting dib-lint in other projects, in
particular tripleo-image-elements. It allows some dib-lint rules
to be used even if the project can't pass all of the checks. The
failing checks can then be fixed one at a time and removed from the
exclusions list.
For consistency, this change reads the new exclusions from tox.ini
in the same way as flake8. To use the exclusion mechanism, the
the following section can be added to tox.ini:
[dib-lint]
ignore = sete setu setpipefail
Change-Id: I6d8024e7613890e935ceb5e50d0d507bd554d8dd
Add a --options flag that allows disk-image-create to send custom
options to qemu-img calls. It is useful for forcing compat into
qemu-img created images.
Usage:
disk-image-create --qemu-img-options compat=0.10 base ubuntu
Change-Id: I73ee2c86abc115220d51ffebbbe9ce3c4d92f188
Closes-Bug: #1329746
The script to find best kernel from image (used by vm
and baremetal element) is duplicated and is not in sync
with each other. Moving the code to img-functions as a
function will reduce duplicated code and make it reusable
in future.
Since img-functions is not accessible in chroot env,
kernel selection is being moved from finalise.d to
cleanup.d in the vm element.
Change-Id: I8fbccc13a2c61a5191ef9ea5d2a8302a3e43b000
Add support for d-i-b to create tarballs, the primary consumer for
tarball images would be linux containers.
Change-Id: I27d67401f3e4415226a4a51e1dde46f739c0220a
Currently when dib-lint finds a problem it does something like:
echo "ERROR: Problem found"
rc=1
This is repetitive and error-prone since it's easy to forget to set
rc to actually fail the check. This change makes those two steps
a single function call.
Change-Id: I40b5bf39348a69add1f955c49f310e3bda21be0e
With this change, dib-lint raises an error if when finding an element
using space indentation that is not multiple of 4.
Co-Authored-By: Jon-Paul Sullivan <jonpaul.sullivan@hp.com>
Change-Id: I470e1fdfc38a3f3c7ba5644c5103f2a9ef073005
This will hopefully catch bugs where they happen rather than
allowing scripts to continue on and fail later.
Change-Id: Idacd9274415b21db285a198dafff19b1d19a4a68
With this change, dib-lint ensure elements do not use tab
indentation. The following files are checked:
- executable file
- .md file
Change-Id: I071262ff9f6599548f869f5439ee127f64eeb46f
If scripts are not set -e then errors can be ignored, causing more
confusing failures later.
Also adds an exclusion comment to the ramdisk init script since we
don't want that to exit on failure.
Change-Id: Idf43993bd10b1ef16c1d3b0d9df8d0ad94c46458
There are certain scripts (such as the ramdisk init script) that
need to ignore linting rules for one reason or another. This adds
support for exclusions via a comment in the file like:
"# dib-lint: disable=executable sete"
There should be no ", but I need those to prevent git from treating
that as a comment. This syntax is similar to the pylint exclusion
mechanism.
Exclusion support is added to the executable check, but not
the alphabetical ordering one because I can't imagine a reason we
would need to disable that, and I don't know that comments are
supported in those files anyway.
Change-Id: I9ecfb47269841dc75a005855455ac26ad2cbc642
The phrase is no longer needed as of August 23, 2000 with Nicaragua's
joining of the Berne Convention.
Additionally, in at least one instance,
elements/cache-url/bin/cache-url, its existence in the file between
Copyright lines is just weird and feels misleading, even though it is
not.
Remove all of the lines, because sanity.
Change-Id: I24fd76c2b4f66b8036010b5079db39ead729abee
We've started to require this in reviews, so we should really have
automation in place to catch it right away.
Change-Id: I43fd90647acba400cea11c665fb587856514b0ee
This will provide a place to put checks that catch common errors
in elements. To start, this just checks that files starting with
a shebang are chmod +x so they can actually be run.
Change-Id: I4116a8f38f7bdfc5866764354c459fad8ca18e92
save_image is used to copy kernel and ramdisks out of the image, which
we will sometimes want to keep the source, and sometimes not. However
for the main image itself, the temp copy is never kept, so use mv
rather than cp and avoid the excess IO.
Change-Id: I5a9f0d69ffee3e6b872a8927537ac17f02f5aa4d
A recent change[1] relies on IMAGE_NAME being available when
98-source-repositories was run. However IMAGE_NAME was previously only
exported if the -o option had been used.
[1] I7dbe9e163ad38a418cf2869a81e720de2c27dfb1
Change-Id: Ife3d94fd45a2e0e0948c17414c369e0e6e040442
In fb246a02eb we introduced an
ext4 option to allow root filesystems to be resized up to 1PB.
This appears to cause an ext4 resize2fs bug in some images.
When the issue occurs an image will hit a kernel bug when
cloud-init runs the resize2fs command during first boot:
kernel BUG at fs/ext4/resize.c:409!
In this commit we add a new option for max-online-resize
which can be used if a really large root partition is
desirable.
The root cause of all this is really a design problem in
DIB/TripleO at the moment in that we shouldn't have to worry about the
max size of the root file system when creating our images. Ideally we'd
just mkfs on the root file system itself. Much more efficient,
avoids this problem altogether...
I think the best thing to do today to avoid this is make setting
max-online-resize an option in DIB. This will allow us to stick
to the (well tested) ext4 defaults for most cases, and if someone has
need for a large root filesystem they can easily bump the setting. This
may be temporary until we either fix the design... or the ext4 fix is
released.
Change-Id: I371f62555d2753cec48790c8fd811c4342af925c
Closes-bug: #1280709
Fixes issues with option parsing. The --min-tmpfs and --image-cache
options where previously broken.
Closes-bug: 1282077
Change-Id: I40c62b16d854335902d1b6b5ceab7f0e1992623a
Rather than using a script to mount the image using nbd to extract the
kernel and ramdisk, make a new element called baremetal, which contains
a cleanup.d script that will copy them out to <image name>.{vmlinuz,initrd}.
Closes-Bug: 1224669
Change-Id: I8f3569aa12148d18b1c8242b6fbbd8857894b26f
With smaller base images, such as Debian, the padding value used to
size the root filesystem does not leave enough room for grub which is
installed after the filesystem is created.
Change-Id: Ic2ab9e2efc9bf1b02f802ee47a36e3fff9c3512e
Occasionally cloud-init fails to resize the disk on first boot, this is
occuring because the filesystem has only 10% free. Nudging the disk size
up a bit more should give us a little more headroom so this is less
likely to occur.
I havn't been able to find out what value ensures we wont hit this
problem but I can say for sure this fixes all casses of the problem I
have seen.
Change-Id: Ib23f3a654151338ad91839e49b323b65b4054245
disk-image-get-kernel is pretty noisy and you see e.g.:
$> load-image overcloud-compute.qcow2
XXX -d '/tmp/image.lWGCgPoj' -o 'tmp' -i '/home/stack/overcloud-compute.qcow2' --
Extracting kernel + ramdisk from /home/stack/overcloud-compute.qcow2 and writing them to /tmp/image.lWGCgPoj
nbd 17554 0
nbd 17554 0
basename: missing operand
Try 'basename --help' for more information.
/dev/nbd0 disconnected
tmp-vmlinuz,tmp-initrd
Clean all this up so we just get:
$> load-image overcloud-compute.qcow2
Extracting kernel + ramdisk from /home/stack/overcloud-compute.qcow2 to tmp-vmlinuz and tmp-initrd in /tmp/image.g6b0lG88
Change-Id: I8971ec0bbcd87157b07fc17254c56bb9f9f2a597
Converts our existing default root element code to be just a check
which exits with a failure message if no root/distribution element
is found.
Change-Id: I954a6abfd7871d5807b1a171a03fa98932410cff
Or environment variables like OS_USERNAME OS_PASSWORD will be stored
in /etc/dib_environment also. It's potentially unsafe.
Change-Id: I3a65d5bc0e4469a071db9ac23ebc82196a3f3feb
Adds an option for --image-size, which sets the DIB_IMAGE_SIZE
environment variable. Having the cli option makes disk-image-create
more consistent in that parameters are able to be specified via a flag
on the command line or an environment variable. Also, you may not be
able to remember all the environment variables that you can tweak, yet
--help will tell you about the cli parameters. Preserves the old
behavior as well where if DIB_IMAGE_SIZE is set in your environment, it
will be honored.
Change-Id: I195c9144a80ce7b8bd5809b57f2bed71a2cbdf26
This commit refactors the python code in diskimage_builder slightly. This is
in preparation for a larger review that adds more functionality to the python
code, namely the ability to apply elements to the current system as opposed to
a chroot. Further, the refactorings can stand on their own for better clarity.
They include:
- renaming elements.py to element_dependencies.py. Adds clarity about the
purpose of this module.
- updating other code for this rename.
- move tests into a tests submodule.
Change-Id: I5519cc52398e442b24e33802bae42070d64b0c1d
Commit 0210be22ae introduced the
possibility of basename being called on an empty string. Because
that can happen in normal operation (such as an x86_64 kernel,
in which case the PAE check will find nothing), don't allow the
basename error to kill the script. Also ignore failures in the
second basename call so the proper error message is echoed.
Change-Id: I38a18af09ab24fda9c98cbf3ace8fd7acc6faef5
disk-image-get-kernel picks the wrong version of kernel and initramfs
if there's a PAE and non-PAE version present. Only affects i386 images.
Change-Id: I06e08fdf038988759b620f549261499cb0a69b34
Closes-Bug: #1240873
the problem is that the journal isn't large enough to allow online
resizing. Solution is straight forward. So the file system can be
resized successfully to disk size specified in flavor.
Fixes bug #1233008
Change-Id: Ie84fb8aea8d334706574d1a8006ec9eaee5bb5be
DIB_IMAGE_CACHE will be a user override for the location where images
are cached. Default location is ~/.cache/image-create
Change-Id: I3e9b9f970864d555c9ec9436344b53f6d3d66dfa
For example if there are following kernels in undercloud/overcloud
image:
/boot/vmlinuz-3.9.5-301.fc19.i686.PAE
/boot/vmlinuz-3.10.10-200.fc19.i686.PAE
then disk-image-get-kernel picks vmlinuz-3.9.5. It should use the
newest one.
Change-Id: I7bbf06705e85370d66c7dd8a5d4f8d6c93b21c0c
Fixes: bug #1224365
In some scenarios, the required space in the tmpfs partition can be
larger (or smaller) than the default one, producing errors due to
the lack of enough space (or performance penalties for not using
tmpfs).
Using --min-tmpfs <size>, we can hint the working set size we'll need
and let dib choose to avoid or use tmpfs.
Change-Id: I7d5fe498302a100c8555ae542268e14b21f3a0c5
When compressing an image, this is done in the same dir where the raw
image resides, doubling the amount of space needed (scarce when
using tmpfs), and then it's moved to the .cache folder in disk.
Combining these two functions, we reduce the amount of space needed
in the tmpfs partition (when in use), and the compressed image is
created directly on the .cache folder disk, so there is no need to
move the compressed image after the process into disk.
Change-Id: I451d24bdd6fa0983414244135dff5e96c0549833
A user running di-b several times while developing an element may not
want to drop to a shell in all cases but may only want to do so if one
of their in target hooks failed.
This patch gives them the ability to do so, If break=after-error is set
then a user will be provided a in target shell taking over from where the
last failed command left off.
Change-Id: Ia2f7ac4c21b64b971f87f4ae9cb867981b13eb5e
(Based on review https://review.openstack.org/#/c/36009)
Scripts test for existence of ../share/diskimage-builder and
fall-back to ../ if not found. This allows scripts to run unmodified
from a packaged installation or a local archive/repository.
Change-Id: I0cf4c1fdb8e42ec284c56860cb15818632b93b9e
- Ensures /sbin and friends are in $PATH when invoked (without this,
various sudo invocations fail in exciting ways).
- Use dib-run-parts in lib/common-functions instead of run-parts
(neither SLES nor openSUSE ship run-parts).
- Ensure dib-run-parts doesn't descend into subdirectories (same
behaviour as run-parts).
- Move dib-run-parts from root.d to bin (cleaner, consistent with
other elements with separate bin scripts).
- Tested by building Ubuntu image on openSUSE 12.3.
- Note: this doesn't add support for creating SUSE images, it just
lets you run disk-image-create on SUSE-based distros.
Change-Id: I906c6bc3cf51cdf2c4415adeae1ca250faac25e1
I missed the getopt parameter and forgot defaults are imported after
option processing. Untested code is broken code!
Change-Id: I133a691909d38e834c204950276a57f4884fc4ed
Complex image builds can download hundreds of MB of data from the
internet with many separate lookups. It would be nice to allow users
to ask for a fast build where those lookups are entirely avoided,
using locally cached resources (where possible). This new interface
allows users to signal to elements that they wish to operate without
updating cached resources, which will in turn allow us to avoid
checking for stale data at all.
As part of this I've also documented where we cache data, so that
things like the ccache cache dir and image cache files are not a
surprise to users.
Change-Id: I27f5de6ceaa4e9c6390721b7c434fe0908df84f5
Ramdisks are now built inside a chroot which is built by the normal
image build process. Doing so improves our independence of the
precise state of the build host.
This fixes bug 1194055.
Change-Id: Ibc254fbb9e7b404b5f38c1b35bcde8a4136e8e28
