Commit Graph

92 Commits

Author SHA1 Message Date
Jenkins
59bb62d1ee Merge "Refactor: remove unused functions" 2016-06-07 01:01:13 +00:00
Andreas Florath
a6a5c385eb Refactor: rename temporary directories
The temporary directories for image creation and building the OS both
started with 'image' as their names followed by some random
characters.  During debugging this is annoying, because on first sight
it is not clear, where which files are stored.

This patch renames them to dib_build.XXXXXXXX and dib_image.XXXXXXXX.

This patch introduces no user-visible change:
the temporary directories are only used during the
run of disk-image-builder.

Change-Id: I249cdb7750fe9a746b375b462789cd9b82681a2e
Signed-off-by: Andreas Florath <andreas@florath.net>
2016-06-03 07:21:03 +02:00
Andreas Florath
b365d3c833 Refactor: remove unused functions
There were a couple of functions which were unused:
ensure_nbd, map_nbd, unmount_qcow_image, mount_qcow_image, ensure_sudo
Because some of them use 'trap ... EXIT' this hinders introducing a
separate exit phase - therefore they are removed.

(It would also be impossible to use them in the current setup, because
they overwrite the 'trap ... EXIT' of the disk-image-creates 'main'.)

Change-Id: If932a557dca9aea4864154ad6c4f286373d6dd7c
Signed-off-by: Andreas Florath <andreas@florath.net>
2016-06-02 21:06:12 +02:00
Ben Kero
9a00bf144f Revert "Revert "Properly fail/trap in eval_run_d"" and fix PIPESTATUS
This reverts commit f07e33a2e9.

This change reverts the revert while fixing the underlying issue --
$PIPESTATUS needed to be encapsulated in ${}s

Change-Id: I1df06ffa7aecf4ea4b8e187dc756e9fc779786bc
2016-05-16 16:25:45 -07:00
Derek Higgins
f07e33a2e9 Revert "Properly fail/trap in eval_run_d"
This reverts commit 0d1d6bec7c.
This patch breaks tripleo-ci (the instack.qcow2 images is failing
to build) and was merged without passing CI.

Closes-Bug: #1582115
Change-Id: Ic4725ad0689c937fb4c8c792e1eaff5f4ea9ada9
2016-05-16 13:16:12 +01:00
Gregory Haynes
0d1d6bec7c Properly fail/trap in eval_run_d
In phases which are called from eval_run_d (block-device.d) we do not
listen to exit 1's nor do we allow break=after-error. This is because
the run_d function is called in a subshell in order to grab its output.

This also turns on pipefail in the main disk-image-create script.

Change-Id: I88ab2e7104148437eabfe6880e3a1e5ebbb2c15d
2016-05-16 01:37:02 +00:00
Ian Wienand
53c57d75c7 Cleanup unmount_dir function
Cleanup this function to work with a symlinked directory.  Document
it's behaviour more exactly, and add a simple unit-test for it (not
run by default, due to doing things like mounting and unmounting
system dirs on a live system, which doesn't seem safe for CI.  But it
is useful for developers ensuring sanity).

Change-Id: I335316019ef948758392b03e91f9869102a472b9
2016-01-28 14:47:43 +11:00
Gregory Haynes
966f12dc4c Properly account for pipefail during cleanup
Due to a bug in how we were running some of our phases we were not
detecting some nonzero exit's. When this is fixed, dib fails early during
cleanup (leaving some resources attatched) due to not propertly
accounting for pipefail.

Change-Id: Icc0b35acbe035cac12a9291e2d07b6c690c3a6ad
2015-12-31 19:43:55 +00:00
James Arendt
12cb94a75d Fix unmount/remove race in cleanup_build_dir
With a slow file system, umount can return 0 and the immediately
following remove can fail with a "Device or resource busy" error.
This happened in DevStack in disk-image-create where unmount_image
is followed by an immediate cleanup_build_dir.

Solution is to apply same logic from bug 1332521 to allow the
remove to retry on failure (up to 5s) in case the umount has not
completed.

Change-Id: I3337e2b4ad0111e77f79dc179439cdfea8ebdeda
Closes-Bug: #1527721
2015-12-17 20:55:02 -08:00
Ian Wienand
494a833987 Add #!/bin/bash to library functions
Currently when these files are opened your editor doesn't know what to
do with them.  Add #!/bin/bash to library functions so that editors,
diff-tools, etc can do syntax highlighting.

There are other ways to skin this cat, such as renaming to ".sh",
adding -* style editor flags, etc.  We had this discussion in DevStack
too, and came to the conclusion the simplest thing that works for
everyone is to just put the #! at the top.

Change-Id: I4cf64321e14844696139f5d40e4d719436390b35
2015-09-16 13:54:07 +10:00
Jenkins
cdf4467a8a Merge "Cleanup the build directories earlier" 2015-07-14 11:50:53 +00:00
Antoine Musso
9bd84d4044 Only chown tmp dirs when they are a tmpfs mount
Temp dirs are created with mktemp and thus belong to the user. There
is no need to chown them unless we used `mount -t tmpfs`.

Move chown under the tmpfs_check conditional.

Change-Id: I37efe18ced3a06d461364dc5cb20600f1527e995
2015-06-24 13:58:31 +00:00
Dan Prince
96b01a7d2a Revert "Remove unused map_nbd function"
This reverts commit ea4a823810.

This function was actually still in use in lib/common-functions
and removing it causes the disk-image-get-kernel /bin
to fail entirely.

Change-Id: Icddb3ca369922a6ea915af8b1b62c434cb1bdf28
Closes-bug: 1464031
2015-06-10 18:37:08 -04:00
Pino Toscano
987a832351 Cleanup the build directories earlier
Split the cleanup_dirs function in two, i.e. cleanup of the build dir
and the image dir, and use the former to cleanup the temporary build
subdirs after their unmount, before the conversion to other disk
formats; they are not needed anyway at that point, and allows to save
disk space during the conversion phase.

Change-Id: Ie30d7e6033613d6979148423326ae7e17a7342e7
2015-06-04 10:46:18 +02:00
Tristan Cacqueray
356404f229 Follow symlink for elements
This allow custom elements to be added with symlink. Without -follow
a symlinked element is valid but scripts in *.d directory aren't used.

Change-Id: If50b7d9c3b1f6fe278c28488146709efe5cf065f
Closes-Bug: 1461124
2015-06-02 15:10:32 +00:00
Ben Nemec
1b92a11268 Don't log tmpfs message during cleanup
By that point in the build it isn't generally useful, and it causes
confusion when builds fail because people think that's the error.

Change-Id: I26dee4ac0947b71a4a065ef6c5a18103e7df6667
2015-06-01 10:48:48 -05:00
Dougal Matthews
a6751c53a0 Make it clear that tmpfs is optional
Given this is often the final output, it can look like an error occured.
Changing the wording makes this clearer.

Change-Id: I70f157054e3120cffee6fa5241b1ffe0b7bfa650
2015-05-27 12:49:38 +01:00
Jenkins
9ba196394c Merge "Remove unused map_nbd function" 2015-05-06 02:50:17 +00:00
Gregory Haynes
ea4a823810 Remove unused map_nbd function
This is a leftover from when we used nbd. We no longer do and this
function is not used.

Change-Id: If037bdd0100ef27401ddd5931ec3db7017302675
2015-04-15 20:20:12 +00:00
Gregory Haynes
f06e3fdb01 Dont try to unmount if were not using tmpfs
I regularly see users report that their build fails because this unmount
line reports an error. Even though we dont bail here because of the ||
true, as a user it is hard to distinguish this from an error.

Change-Id: Ic43f4fb24c53c58329fdf501bba6ba14024ec2aa
2015-03-24 18:30:55 +00:00
Jenkins
df7ae94df2 Merge "Corrected element-info usage" 2015-02-03 15:50:06 +00:00
jodewey
8fc027227a Corrected element-info usage
Deprecated the `--expand-dependencies` flag from `element-info` usage.
The flag was required and not optional.  We can rely on argparse to exit non-0
when the required positional argument is not provided.

Change-Id: Iaf8eb962eb600760974bc33c30b809a07a23278e
Closes-Bug: 1265649
2015-01-29 13:43:48 -08:00
Michael Kerrin
6337e6fe17 Fix issue with leaking /tmp/image.* directories
When building the ramdisk we don't cleanup the temporary
directories after ourselves. This leaves /tmp/image.* directories
mounted and /tmp/image.* directories on the system.

Also the ramdisk-functions duplicate, from what I can see,
the cleanup function from common-functions. So when a job
is killed off it ends up leaving /tmp/image.* directories
on the system.

Change-Id: I2d73aabd0eb176027b4e7368580db08902e2b6ab
2015-01-28 12:07:07 +00:00
Jenkins
34c273d8df Merge "Make diskimage-builder work in Docker." 2014-10-24 20:13:45 +00:00
Jenkins
2ea4f6ec8a Merge "Avoid overwritting of hooks" 2014-10-20 11:04:07 +00:00
Jenkins
bb07de7fe9 Merge "Enable dracut deploy ramdisks" 2014-10-20 11:03:54 +00:00
Ben Nemec
eed30adc48 Enable dracut deploy ramdisks
The element builds dracut from source on Ubuntu because the
Ubuntu dracut package is broken and very old, so it can't be
installed properly and causes a number of other issues that
are fixed by using a newer version of Dracut.

This initial version should work in virtualized environments.
Further validation of its suitability for real baremetal
deployments will need to be done in the future, but this should
be sufficient to enable that work.

Regarding Dracut specifically, in order to limit the changes
needed in the existing scripts this element continues to use a
cut down version of the /init script that we were building for the
existing ramdisk.  However, instead of running it as pid 0 it is
run as a Dracut pre-mount hook.  This allows Dracut to set up all
of the hardware and system bits, while falling early enough in the
Dracut sequence to complete the deployment before Dracut would try
to boot off the hard disk.

bp tripleo-juno-dracut-ramdisks
Change-Id: I144c8993fe040169f440bd4f7a428fdbe3d745cf
2014-10-17 21:44:09 -05:00
Ghe Rivero
88271757a5 Avoid overwritting of hooks
Until now there was a possibility for two elements to install hooks
with the same name, so one of them was overwritten. Change logic to
copy the hooks and fail in case one with the same name exists.

Change-Id: Ic2c46835b27c9319f7a889ffd0ccf3f5ccc1f0cd
Closes-Bug: 1251952
2014-10-17 08:16:46 +00:00
Jenkins
2b60bea961 Merge "Allow for multiple image outputs from raw source" 2014-10-16 21:56:28 +00:00
Jenkins
5b71da8899 Merge "Remove first-boot.d support" 2014-10-16 18:36:59 +00:00
Ghe Rivero
873de01491 Remove first-boot.d support
After being deprecated two releases ago, finally remove any reference
for the support of first-boot.d

Change-Id: I08d67404ef48cad61db3b18fb86e970abfa5d2b6
2014-10-16 09:48:14 +00:00
Clark Boylan
ae928057bd Allow for multiple image outputs from raw source
When uploading images to multiple clouds it is possible that the same
image will be needed in multiple formats to accomodate hypervisors
across clouds. Update disk-image-create's -t flag to take a list of
desired output image formats so that a single disk-image-create can
output all of the desired image formats.

Change-Id: If121b2342ae888855ba435aa3189f039e985b812
2014-10-13 09:16:42 -07:00
Gregory Haynes
f06493bcd6 Preserve exit value when leaving cleanup trap
If we entered the cleanup trap due to exit with an error code we should
exit dib with an error code.

Change-Id: Iee1a05668b3239113fb91a2da0d9a66d7de4db6b
2014-10-10 17:04:50 -07:00
Clint Byrum
c781115784 Do not try to detach non-existant loopback devices
A user reported symptoms where the losetup line used to detach the
loopback device was failing in tar mode. We don't need to detach a
device that does not exist.

Change-Id: I807996e16199288927b49b4f300ae9b461cb8fe7
Closes-Bug: #1378033
2014-10-08 10:37:48 -07:00
Christian Berendt
6192bb78fb Use $((EXPRESSION)) instead of $[EXPRESSION]
$((EXPRESSION)) is POSIX-conform and should be preferred.

Change-Id: Ibd0524194aeda40961e202b6761dbaed2ea27dfa
2014-09-30 14:52:46 +02:00
Victor Lowther
38b14df72c Make diskimage-builder work in Docker.
When running inside a Docker container, we cannot rely on devices in
/dev/mapper to be automagically created by udev, because we probably
don't have a udev at all.  To work around this, run dmsetup mknodes
after every kpartx run.

Change-Id: If7e30579224ce54c5ed26d08974d8293c144719a
2014-09-15 06:43:53 -05:00
Ben Nemec
644598941c Use dib-run-parts from dib-utils
Now that dib-run-parts has been moved to the dib-utils project, we
need to update diskimage-builder to use it instead of the version
directly in diskimage-builder.

This change removes the old copy of the dib-run-parts script in
the element, adds dib-utils as a dependency of diskimage-builder,
and updates the uses of dib-run-parts to correctly handle the fact
that it is now external to the project.

Requires I0be1f876d0e4a7d38e0d5c6010a552a8ebb158a4

Change-Id: Ia0a0df7784a14c49b5c47ac0b03e6c2602c84b3b
2014-08-26 16:44:18 -05:00
Robert Collins
4319730357 Add a ramdisk-install.d hook path.
We need to be able to do install.d like things for ramdisks
themselves, but install.d runs outside the ramdisk context - and its
likely to break peoples brains if we mangle the two together - so this
adds a new hook point, ramdisk-install, specifically for installing
things into the ramdisk.

Change-Id: I37d1660309cda6e28bd0b316b08f61db4e080613
2014-07-16 19:23:50 +12:00
Jenkins
348a281795 Merge "Allow overwriting old images" 2014-07-15 14:52:06 +00:00
Steve Kowalik
84b2ac45de Allow overwriting old images
The two duplicated functions, save_image and finish_image will move
an existing image out of the way if it exists, but it isn't
configurable. Check an environment variable is 0 before doing so.

Switch save_image to just calling finish_image, rather than
duplicating its code exactly.

Change-Id: I26a5a8fa4b6e853c9440bffab195b0bc3728be40
2014-07-08 14:08:10 +10:00
AzherKhan
d06a1caafb Correction: if then statement code style.
Corrected the if; then statement to follow bash convention.

Change-Id: Ia3f80520bb9446c4feb5ce36dccffa2b1c1d72c4
2014-07-07 03:43:15 -07:00
Jenkins
bedd50f80d Merge "fail at startup with no operating-system element" 2014-06-13 14:36:59 +00:00
Jenkins
fa9a561819 Merge "Rename old image file instead of rewrite it" 2014-06-12 22:19:15 +00:00
Gonéri Le Bouder
1011adf370 fail at startup with no operating-system element
Raise an error if there is no element with “operating-system”
in the element-provides file.

Change-Id: I2242537abc4c610252984c32c286bc6eb25de672
2014-06-12 11:19:20 +02:00
iberezovskiy
b740c50c76 Rename old image file instead of rewrite it
If we have the old image file with same name as new image file
we shouldn't rewrite it, just need to rename old.
Uses date number in old image name.

Fixes TODO in lib/common-functions

Change-Id: Ica58a0f4435217912bddf950322299d2ed2889f3
2014-06-06 12:59:28 +04:00
James Slagle
813b4bc242 Check return code of element-info
Ensure the return code of element-info is actually checked by moving the
export IMAGE_ELEMENT to the subsequent line.

A recent change (Ic0a061995e2ae708c95a535cba6237bff58f7d93) exported the
IMAGE_ELEMENT variable. This caused the return code of element-info to not
actually be checked, since the export command is now checked instead and
almost always will return 0.

This caused a misleading error message when using diskimage-builder with an
unknown element since the script didn't exit when it should have:
$ bin/disk-image-create -a amd64 foobar vm
Building elements: base  foobar vm
ERROR: Element 'foobar' not found in 'bin/../elements'
Expanded element dependencies to:
Building in /var/tmp/image.DuIDY1lW
Please include at least one distribution root element.

Change-Id: I3d9c02bf0b1a4f320280185dd3fab0697d2c92f2
2014-05-30 15:41:05 -04:00
Dan Prince
c55539b142 Add pkg-map element.
Adds a new pkg-map element which can be used to install
packages based on an in element 'pkg-map' JSON config
file format.

As part of this change we expose DIB's IMAGE_ELEMENT variable
so that we can have access to the element names which are being
installed in a clean manner.

This change is intended to decouple elements from DIB
and allow new elements to support multiple distributions
(with various package naming schemes) without having to
constantly maintain DIB's various bin/map-packages files.

Change-Id: Ic0a061995e2ae708c95a535cba6237bff58f7d93
2014-05-28 12:21:44 -04:00
Jenkins
dcb5494e04 Merge "Build raw image in separate tmpfs" 2014-05-08 23:15:27 +00:00
Dan Prince
956acd5f59 Eliminate 'tr' in favor of inline bash.
Updates the lib/common-functions generate_hooks() function
to use bash instead of exec'ing tr.

Change-Id: Ie32d3ce5c7c43affd0031a568d57763e36209603
2014-05-02 13:10:33 -04:00
Derek Higgins
3fada2566a Build raw image in separate tmpfs
If the tmpfs being used to hold the image filesystem is close to being
full then housing the raw disk image in the same place can be
problematic as it tends to grow faster then the filesystem shrinks when
data is being moved into it.

Putting them both into separate tmpfs's will allow there to be an
overlap where they jointly use more then the size limit for a singe tmpfs.

Change-Id: Ia17ca357d3b865d70a2d3e13e0479d008ca5f924
Closes-Bug: #1289582
2014-05-01 11:46:29 +01:00