Commit Graph

80 Commits

Author SHA1 Message Date
Antoine Musso
9bd84d4044 Only chown tmp dirs when they are a tmpfs mount
Temp dirs are created with mktemp and thus belong to the user. There
is no need to chown them unless we used `mount -t tmpfs`.

Move chown under the tmpfs_check conditional.

Change-Id: I37efe18ced3a06d461364dc5cb20600f1527e995
2015-06-24 13:58:31 +00:00
Dan Prince
96b01a7d2a Revert "Remove unused map_nbd function"
This reverts commit ea4a823810.

This function was actually still in use in lib/common-functions
and removing it causes the disk-image-get-kernel /bin
to fail entirely.

Change-Id: Icddb3ca369922a6ea915af8b1b62c434cb1bdf28
Closes-bug: 1464031
2015-06-10 18:37:08 -04:00
Tristan Cacqueray
356404f229 Follow symlink for elements
This allow custom elements to be added with symlink. Without -follow
a symlinked element is valid but scripts in *.d directory aren't used.

Change-Id: If50b7d9c3b1f6fe278c28488146709efe5cf065f
Closes-Bug: 1461124
2015-06-02 15:10:32 +00:00
Ben Nemec
1b92a11268 Don't log tmpfs message during cleanup
By that point in the build it isn't generally useful, and it causes
confusion when builds fail because people think that's the error.

Change-Id: I26dee4ac0947b71a4a065ef6c5a18103e7df6667
2015-06-01 10:48:48 -05:00
Dougal Matthews
a6751c53a0 Make it clear that tmpfs is optional
Given this is often the final output, it can look like an error occured.
Changing the wording makes this clearer.

Change-Id: I70f157054e3120cffee6fa5241b1ffe0b7bfa650
2015-05-27 12:49:38 +01:00
Jenkins
9ba196394c Merge "Remove unused map_nbd function" 2015-05-06 02:50:17 +00:00
Gregory Haynes
ea4a823810 Remove unused map_nbd function
This is a leftover from when we used nbd. We no longer do and this
function is not used.

Change-Id: If037bdd0100ef27401ddd5931ec3db7017302675
2015-04-15 20:20:12 +00:00
Gregory Haynes
f06e3fdb01 Dont try to unmount if were not using tmpfs
I regularly see users report that their build fails because this unmount
line reports an error. Even though we dont bail here because of the ||
true, as a user it is hard to distinguish this from an error.

Change-Id: Ic43f4fb24c53c58329fdf501bba6ba14024ec2aa
2015-03-24 18:30:55 +00:00
Jenkins
df7ae94df2 Merge "Corrected element-info usage" 2015-02-03 15:50:06 +00:00
jodewey
8fc027227a Corrected element-info usage
Deprecated the `--expand-dependencies` flag from `element-info` usage.
The flag was required and not optional.  We can rely on argparse to exit non-0
when the required positional argument is not provided.

Change-Id: Iaf8eb962eb600760974bc33c30b809a07a23278e
Closes-Bug: 1265649
2015-01-29 13:43:48 -08:00
Michael Kerrin
6337e6fe17 Fix issue with leaking /tmp/image.* directories
When building the ramdisk we don't cleanup the temporary
directories after ourselves. This leaves /tmp/image.* directories
mounted and /tmp/image.* directories on the system.

Also the ramdisk-functions duplicate, from what I can see,
the cleanup function from common-functions. So when a job
is killed off it ends up leaving /tmp/image.* directories
on the system.

Change-Id: I2d73aabd0eb176027b4e7368580db08902e2b6ab
2015-01-28 12:07:07 +00:00
Jenkins
34c273d8df Merge "Make diskimage-builder work in Docker." 2014-10-24 20:13:45 +00:00
Jenkins
2ea4f6ec8a Merge "Avoid overwritting of hooks" 2014-10-20 11:04:07 +00:00
Jenkins
bb07de7fe9 Merge "Enable dracut deploy ramdisks" 2014-10-20 11:03:54 +00:00
Ben Nemec
eed30adc48 Enable dracut deploy ramdisks
The element builds dracut from source on Ubuntu because the
Ubuntu dracut package is broken and very old, so it can't be
installed properly and causes a number of other issues that
are fixed by using a newer version of Dracut.

This initial version should work in virtualized environments.
Further validation of its suitability for real baremetal
deployments will need to be done in the future, but this should
be sufficient to enable that work.

Regarding Dracut specifically, in order to limit the changes
needed in the existing scripts this element continues to use a
cut down version of the /init script that we were building for the
existing ramdisk.  However, instead of running it as pid 0 it is
run as a Dracut pre-mount hook.  This allows Dracut to set up all
of the hardware and system bits, while falling early enough in the
Dracut sequence to complete the deployment before Dracut would try
to boot off the hard disk.

bp tripleo-juno-dracut-ramdisks
Change-Id: I144c8993fe040169f440bd4f7a428fdbe3d745cf
2014-10-17 21:44:09 -05:00
Ghe Rivero
88271757a5 Avoid overwritting of hooks
Until now there was a possibility for two elements to install hooks
with the same name, so one of them was overwritten. Change logic to
copy the hooks and fail in case one with the same name exists.

Change-Id: Ic2c46835b27c9319f7a889ffd0ccf3f5ccc1f0cd
Closes-Bug: 1251952
2014-10-17 08:16:46 +00:00
Jenkins
2b60bea961 Merge "Allow for multiple image outputs from raw source" 2014-10-16 21:56:28 +00:00
Jenkins
5b71da8899 Merge "Remove first-boot.d support" 2014-10-16 18:36:59 +00:00
Ghe Rivero
873de01491 Remove first-boot.d support
After being deprecated two releases ago, finally remove any reference
for the support of first-boot.d

Change-Id: I08d67404ef48cad61db3b18fb86e970abfa5d2b6
2014-10-16 09:48:14 +00:00
Clark Boylan
ae928057bd Allow for multiple image outputs from raw source
When uploading images to multiple clouds it is possible that the same
image will be needed in multiple formats to accomodate hypervisors
across clouds. Update disk-image-create's -t flag to take a list of
desired output image formats so that a single disk-image-create can
output all of the desired image formats.

Change-Id: If121b2342ae888855ba435aa3189f039e985b812
2014-10-13 09:16:42 -07:00
Gregory Haynes
f06493bcd6 Preserve exit value when leaving cleanup trap
If we entered the cleanup trap due to exit with an error code we should
exit dib with an error code.

Change-Id: Iee1a05668b3239113fb91a2da0d9a66d7de4db6b
2014-10-10 17:04:50 -07:00
Clint Byrum
c781115784 Do not try to detach non-existant loopback devices
A user reported symptoms where the losetup line used to detach the
loopback device was failing in tar mode. We don't need to detach a
device that does not exist.

Change-Id: I807996e16199288927b49b4f300ae9b461cb8fe7
Closes-Bug: #1378033
2014-10-08 10:37:48 -07:00
Christian Berendt
6192bb78fb Use $((EXPRESSION)) instead of $[EXPRESSION]
$((EXPRESSION)) is POSIX-conform and should be preferred.

Change-Id: Ibd0524194aeda40961e202b6761dbaed2ea27dfa
2014-09-30 14:52:46 +02:00
Victor Lowther
38b14df72c Make diskimage-builder work in Docker.
When running inside a Docker container, we cannot rely on devices in
/dev/mapper to be automagically created by udev, because we probably
don't have a udev at all.  To work around this, run dmsetup mknodes
after every kpartx run.

Change-Id: If7e30579224ce54c5ed26d08974d8293c144719a
2014-09-15 06:43:53 -05:00
Ben Nemec
644598941c Use dib-run-parts from dib-utils
Now that dib-run-parts has been moved to the dib-utils project, we
need to update diskimage-builder to use it instead of the version
directly in diskimage-builder.

This change removes the old copy of the dib-run-parts script in
the element, adds dib-utils as a dependency of diskimage-builder,
and updates the uses of dib-run-parts to correctly handle the fact
that it is now external to the project.

Requires I0be1f876d0e4a7d38e0d5c6010a552a8ebb158a4

Change-Id: Ia0a0df7784a14c49b5c47ac0b03e6c2602c84b3b
2014-08-26 16:44:18 -05:00
Robert Collins
4319730357 Add a ramdisk-install.d hook path.
We need to be able to do install.d like things for ramdisks
themselves, but install.d runs outside the ramdisk context - and its
likely to break peoples brains if we mangle the two together - so this
adds a new hook point, ramdisk-install, specifically for installing
things into the ramdisk.

Change-Id: I37d1660309cda6e28bd0b316b08f61db4e080613
2014-07-16 19:23:50 +12:00
Jenkins
348a281795 Merge "Allow overwriting old images" 2014-07-15 14:52:06 +00:00
Steve Kowalik
84b2ac45de Allow overwriting old images
The two duplicated functions, save_image and finish_image will move
an existing image out of the way if it exists, but it isn't
configurable. Check an environment variable is 0 before doing so.

Switch save_image to just calling finish_image, rather than
duplicating its code exactly.

Change-Id: I26a5a8fa4b6e853c9440bffab195b0bc3728be40
2014-07-08 14:08:10 +10:00
AzherKhan
d06a1caafb Correction: if then statement code style.
Corrected the if; then statement to follow bash convention.

Change-Id: Ia3f80520bb9446c4feb5ce36dccffa2b1c1d72c4
2014-07-07 03:43:15 -07:00
Jenkins
bedd50f80d Merge "fail at startup with no operating-system element" 2014-06-13 14:36:59 +00:00
Jenkins
fa9a561819 Merge "Rename old image file instead of rewrite it" 2014-06-12 22:19:15 +00:00
Gonéri Le Bouder
1011adf370 fail at startup with no operating-system element
Raise an error if there is no element with “operating-system”
in the element-provides file.

Change-Id: I2242537abc4c610252984c32c286bc6eb25de672
2014-06-12 11:19:20 +02:00
iberezovskiy
b740c50c76 Rename old image file instead of rewrite it
If we have the old image file with same name as new image file
we shouldn't rewrite it, just need to rename old.
Uses date number in old image name.

Fixes TODO in lib/common-functions

Change-Id: Ica58a0f4435217912bddf950322299d2ed2889f3
2014-06-06 12:59:28 +04:00
James Slagle
813b4bc242 Check return code of element-info
Ensure the return code of element-info is actually checked by moving the
export IMAGE_ELEMENT to the subsequent line.

A recent change (Ic0a061995e2ae708c95a535cba6237bff58f7d93) exported the
IMAGE_ELEMENT variable. This caused the return code of element-info to not
actually be checked, since the export command is now checked instead and
almost always will return 0.

This caused a misleading error message when using diskimage-builder with an
unknown element since the script didn't exit when it should have:
$ bin/disk-image-create -a amd64 foobar vm
Building elements: base  foobar vm
ERROR: Element 'foobar' not found in 'bin/../elements'
Expanded element dependencies to:
Building in /var/tmp/image.DuIDY1lW
Please include at least one distribution root element.

Change-Id: I3d9c02bf0b1a4f320280185dd3fab0697d2c92f2
2014-05-30 15:41:05 -04:00
Dan Prince
c55539b142 Add pkg-map element.
Adds a new pkg-map element which can be used to install
packages based on an in element 'pkg-map' JSON config
file format.

As part of this change we expose DIB's IMAGE_ELEMENT variable
so that we can have access to the element names which are being
installed in a clean manner.

This change is intended to decouple elements from DIB
and allow new elements to support multiple distributions
(with various package naming schemes) without having to
constantly maintain DIB's various bin/map-packages files.

Change-Id: Ic0a061995e2ae708c95a535cba6237bff58f7d93
2014-05-28 12:21:44 -04:00
Jenkins
dcb5494e04 Merge "Build raw image in separate tmpfs" 2014-05-08 23:15:27 +00:00
Dan Prince
956acd5f59 Eliminate 'tr' in favor of inline bash.
Updates the lib/common-functions generate_hooks() function
to use bash instead of exec'ing tr.

Change-Id: Ie32d3ce5c7c43affd0031a568d57763e36209603
2014-05-02 13:10:33 -04:00
Derek Higgins
3fada2566a Build raw image in separate tmpfs
If the tmpfs being used to hold the image filesystem is close to being
full then housing the raw disk image in the same place can be
problematic as it tends to grow faster then the filesystem shrinks when
data is being moved into it.

Putting them both into separate tmpfs's will allow there to be an
overlap where they jointly use more then the size limit for a singe tmpfs.

Change-Id: Ia17ca357d3b865d70a2d3e13e0479d008ca5f924
Closes-Bug: #1289582
2014-05-01 11:46:29 +01:00
Ben Nemec
381ff6ab1d Fix set -eu and pipefail failures
Fixes problems found by set -eu and pipefail, including:
-Many unset variables
-Commands that can fail under normal circumstances, which breaks
with set -e.  This change swallows those expected errors to allow
our existing error code to handle them.
-The dkms element was not finding Fedora kernel versions correctly.
This may be an issue for other distros too, but since Fedora was
working fine without this functionality I only changed it to print
a warning message rather than failing the build when it happens.
-The ramdisk init script will not be set -eu because if it fails
the result is a kernel panic, which can be tricky to debug.
However, in testing with set -e a few failing commands were found
and have been fixed in this patch.

Change-Id: I44cf98dfc80cfcaec54b88cc83be80a3dbf2cec3
2014-04-15 20:53:15 -05:00
Derek Higgins
85dadd0d90 Revert "Only create a tmpfs big enough for DIB_MIN_TMPFS"
This didn't pass CI and is now causing failures as it moves us back to a
4G tmpfs.

Change-Id: Ie92fd909ad21216b1fda55c21457893df38e882f
2014-03-26 10:07:41 +00:00
Jenkins
bfba0ccd03 Merge "Remove an excess cp of disk images." 2014-03-26 08:38:52 +00:00
Robert Collins
dc93feee69 Remove an excess cp of disk images.
save_image is used to copy kernel and ramdisks out of the image, which
we will sometimes want to keep the source, and sometimes not. However
for the main image itself, the temp copy is never kept, so use mv
  rather than cp and avoid the excess IO.

Change-Id: I5a9f0d69ffee3e6b872a8927537ac17f02f5aa4d
2014-03-25 17:20:37 +00:00
Clint Byrum
e63625a7e1 Only create a tmpfs big enough for DIB_MIN_TMPFS
This will help to ensure images that would cause bug #1289582 would
still fail even on systems that have more RAM. Also this will ensure we
only reserve enough tmpfs space for this build.

Change-Id: Icece64e691d126a5fc89262897630fdbef1ffdf1
2014-03-07 12:52:15 -08:00
JUN JIE NAN
6b7cf26689 Fixed device or resource busy issue in EXIT trap
cleanup for ramdisk failed to umount TMP_BUILD_DIR with device or
resource busy error.

The patch unmounts all the mountpoints under TMP_BUILD_DIR and
detaches loop devices associated with TMP_IMAGE_PATH.

The unmounts are applied with both force(-f) and lazy(-l)
options. Force option is only for NFS mounts, it's kept here since no
harm for lazy option.

Change-Id: I84035e6a003d8135186b2fda3facbd2c37967529
2014-01-03 16:53:38 +08:00
Mark McLoughlin
af8b7f05a1 Quieten disk-image-get-kernel
disk-image-get-kernel is pretty noisy and you see e.g.:

  $> load-image overcloud-compute.qcow2
  XXX  -d '/tmp/image.lWGCgPoj' -o 'tmp' -i '/home/stack/overcloud-compute.qcow2' --
  Extracting kernel + ramdisk from /home/stack/overcloud-compute.qcow2 and writing them to /tmp/image.lWGCgPoj
  nbd                    17554  0
  nbd                    17554  0
  basename: missing operand
  Try 'basename --help' for more information.
  /dev/nbd0 disconnected
  tmp-vmlinuz,tmp-initrd

Clean all this up so we just get:

  $> load-image overcloud-compute.qcow2
  Extracting kernel + ramdisk from /home/stack/overcloud-compute.qcow2 to tmp-vmlinuz and tmp-initrd in /tmp/image.g6b0lG88

Change-Id: I8971ec0bbcd87157b07fc17254c56bb9f9f2a597
2013-12-09 13:39:29 +00:00
Dan Prince
255387a288 Drop default distribution root element support
Converts our existing default root element code to be just a check
which exits with a failure message if no root/distribution element
is found.

Change-Id: I954a6abfd7871d5807b1a171a03fa98932410cff
2013-12-02 11:01:29 -05:00
Tim Serong
43c0082aa7 Move /tmp/ccache setup to base element
Commit c7d80dd (Cleanup mount points automatically) removed the unmount
of $TMP_MOUNT_PATH/tmp/ccache in run_d_in_target() and moved the
"rm /tmp/ccache" to elements/base/finalise.d/02-remove-ccache.  There
are two problems with this:

1) Not unmounting at the end of run_d_in_target() results in tmp/ccache
   being bind mounted muliple times on top of itself (three times, if you
   just run `disk-image-create base`).  It is eventually unmounted, but
   somehow the auto unmount code is confused, and tries to unmount it
   one more time than it was mounted, which results in an error like
   "umount: /tmp/image.THQkZxQa/mnt/tmp/ccache: not mounted".
   This doesn't actually break anything, but it's a little messy.

2) "rm /tmp/ccache" in elements/base/finalise.d/02-remove-ccache never
   succeeds in removing /tmp/ccache, because that hook is invoked by
   run_d_in_target(), *while* /tmp/ccache is mounted.

This present commit solves the above by moving the ccache setup glue out
of img-functions and into the base element's root.d.  This has the
following implications:

1) lib/img-functions is a little cleaner.

2) /tmp/ccache is available in the chroot during the root, extra-data,
   pre-install, install and post-install stages.  It is not available
   during block-device, finalise and cleanup stages as it will have been
   automatically unmounted by then.

3) /tmp/ccache won't be setup if you're building an image that doesn't
   include the base element.

Change-Id: Ief4c0a6f4ec622db6c6f652776215684178d8943
2013-10-10 16:27:09 +11:00
Tim Serong
0a5d3a9446 Mount /dev/pts in chroot
Mounting /dev/pts fixes mildly irritating errors during image creation,
i.e.: "Can not write log, openpty() failed (/dev/pts not mounted?)"
emitted by apt-get.

Change-Id: I4106bc584f1b29ce9283a20064bb5ae447f17fbe
2013-10-08 21:41:18 +11:00
Clint Byrum
d07921eca5 Deprecate first-boot.d
Print a message and pause the build for 10 seconds to ensure interactive
users see the message.

Fixes bug #1212080

Change-Id: Ia388a54892c479e428b0ed7b8c70d64d65010e21
2013-09-27 09:42:51 -07:00
Ghe Rivero
fefc28a918 Add option --min-tmpfs <size> to disk-image-create
In some scenarios, the required space in the tmpfs partition can be
larger (or smaller) than the default one, producing errors due to
the lack of enough space (or performance penalties for not using
tmpfs).

Using --min-tmpfs <size>, we can hint the working set size we'll need
and let dib choose to avoid or use tmpfs.

Change-Id: I7d5fe498302a100c8555ae542268e14b21f3a0c5
2013-08-28 11:14:06 +00:00