Commit Graph

85 Commits

Author SHA1 Message Date
Andreas Florath
b365d3c833 Refactor: remove unused functions
There were a couple of functions which were unused:
ensure_nbd, map_nbd, unmount_qcow_image, mount_qcow_image, ensure_sudo
Because some of them use 'trap ... EXIT' this hinders introducing a
separate exit phase - therefore they are removed.

(It would also be impossible to use them in the current setup, because
they overwrite the 'trap ... EXIT' of the disk-image-creates 'main'.)

Change-Id: If932a557dca9aea4864154ad6c4f286373d6dd7c
Signed-off-by: Andreas Florath <andreas@florath.net>
2016-06-02 21:06:12 +02:00
SamYaple
5b6716cee8 Use fstrim to prep the block device
This cuts the image size down alot, esspecially if there were lots of
small file deletes.

The fstrim utility is in the util-linux package and should be on
most all systems. fstrim also works with XFS, ext4, btrfs, etc
prodiving the kernel is new enough.

A reduction of 25% or more in size is common.

Change-Id: I269b4416be450369616f9b8e030f84c30e329804
2016-03-13 16:24:59 +00:00
SamYaple
0c32784663 Revert "Zerofree the image if possible"
This reverts commit 5184d02a7c.

The decision was made to go with fstrim because it is faster and more
universal that zerofree. The related-id has the patchset that implements
fstrim.

Related-Id: I269b4416be450369616f9b8e030f84c30e329804
Change-Id: If40cf2fc0ecd8686768cbfeac9ecee90907674e7
2016-03-10 20:38:01 +00:00
Clark Boylan
5184d02a7c Zerofree the image if possible
If the image has an ext filesystem and the zerofree utility is present
on the build system then run zerofree. This should make images as
compressable as possible which is a nice feature when building
compressed qcow2 images.

Change-Id: Ia6062c291f7a3f58b85a4f408ecb3d0574c65d53
2016-03-07 17:45:14 -08:00
Matthew Thode
176ae7bff6
Initial add of gentoo support for diskimage-builder
uses upstream's stage4 images, includes all the needed bells and
whistles for openstack on kvm.

Change-Id: Ibca43173c30c2a74a73a2e2d9dd6d6d832c62694
Closes-Bug: 1530911
2016-01-12 19:15:00 -06:00
Ian Wienand
494a833987 Add #!/bin/bash to library functions
Currently when these files are opened your editor doesn't know what to
do with them.  Add #!/bin/bash to library functions so that editors,
diff-tools, etc can do syntax highlighting.

There are other ways to skin this cat, such as renaming to ".sh",
adding -* style editor flags, etc.  We had this discussion in DevStack
too, and came to the conclusion the simplest thing that works for
everyone is to just put the #! at the top.

Change-Id: I4cf64321e14844696139f5d40e4d719436390b35
2015-09-16 13:54:07 +10:00
Pino Toscano
987a832351 Cleanup the build directories earlier
Split the cleanup_dirs function in two, i.e. cleanup of the build dir
and the image dir, and use the former to cleanup the temporary build
subdirs after their unmount, before the conversion to other disk
formats; they are not needed anyway at that point, and allows to save
disk space during the conversion phase.

Change-Id: Ie30d7e6033613d6979148423326ae7e17a7342e7
2015-06-04 10:46:18 +02:00
Jenkins
38a3d98099 Merge "Cleanup /tmp in the guest" 2015-05-28 23:38:04 +00:00
Pino Toscano
37572a548c Cleanup /tmp in the guest
Clean all the content in the /tmp directory of the guest, leaving the
directory itself (usually has special attributes).

Modern distributions usually either setup a tmpfs on /tmp, or clean it
at every boot, so the leftovers will be just few bytes in the generated
image.
Regarding other distributions, a clean /tmp at their first boot will
surely not be a bad idea anyway.

Change-Id: I2b0f8864bc4909542d924f5bd9296dca5d0189f2
2015-05-20 11:24:17 +02:00
Gregory Haynes
b9b6640fa7 Initial element tests
Adding a test function which allows us to use elements to perform
element-specific tests. In order for this to work sanely, also adding
some configuration to our break system so we can assert on negative
tests.

Also adding a test for apt-sources to verify this code actually works.

Change-Id: I378a74255010eca192f5766b653f8a42404be5ea
2015-05-17 02:07:40 +00:00
Jenkins
b2c2c47dc1 Merge "Actually set a sane PATH for inside chroot" 2015-05-05 01:53:11 +00:00
Jenkins
f8302171f5 Merge "Add element ubuntu-signed to provide signed kernel" 2015-04-28 11:02:44 +00:00
Gregory Haynes
f2a893b74c Actually set a sane PATH for inside chroot
In I084aff7e449f5de811a6169ec90e352ada7da439 we attemped to address a
bug for systems which dont have a path that works well for inside a
chroot. Turns out there were multiple issues:
 * The PATH we were setting was after we attempted to call sh.
 * PATH was being set in a sibling process to the command being run.
 * PATH was not being exported so it was not effecting child processes.

Using env to set a sane path before we attempt to run our commands
addresses these issues.

Change-Id: I4285f8048465ee5c2490116447d32033007bd185
2015-04-28 00:02:15 +00:00
Jenkins
f0736e7d97 Merge "Append full path to img-functions:run_in_target" 2015-04-22 19:06:12 +00:00
Jenkins
44d6d5cd88 Merge "Support VHD output format" 2015-04-22 05:31:04 +00:00
Jenkins
212da2f149 Merge "Try 5 times for rmdir command call after umount" 2015-04-21 21:22:19 +00:00
Ben Kero
4a2939c207 Append full path to img-functions:run_in_target
The build host might not necessarily have a complete set of dirs
required in its PATH. It will likely be better to statically code a
complete(ish) set of PATH entries inside the function.

Some distros (ex: archlinux) are lacking some dirs commonly found in a
PATH. If this is used as a host OS, it will transfer this incomplete
PATH into the chroot, where other guests (ubuntu, centos) will fail to
find basic binaries due to the lack of a /bin entry.

Change-Id: I084aff7e449f5de811a6169ec90e352ada7da439
2015-04-16 11:19:44 -07:00
Gregory Haynes
7937fb5392 Support VHD output format
This useful for building images for xen.

Change-Id: Ib716bd7fc1ed979968ea92f46f1644038e40df66
2015-04-14 13:39:18 -04:00
Shivanand Tendulker
06e3d7c767 Add element ubuntu-signed to provide signed kernel
ubuntu-signed element would install 'linux-signed-image-generic' that
provides signed kernel that can be used for deploy in UEFI secure boot mode.

Package 'linux-signed-image-generic' ships signed kernel with extension
'.efi.signed' (Ex. '/boot/vmlinuz-3.13.0-49-generic.efi.signed').

The kernel modules directory for signed kernel and unsigned kernel is same.
It is without 'efi.signed' extension to its name. This is different from normal
practice of directory naming in '/lib/modules' (Ex. For signed kernel
'vmlinuz-3.13.0-49-generic.efi.signed', modules directory is
'/lib/modules/3.13.0-49-generic').
This needed some changes in '/lib/ramdisk-functions' and 'ramdisk' element to
copy kernel modules.

The signed kernel package contains both signed and unsigned kernel. The
unsiged kernel is without extension '.efi.signed' (Ex.
'/boot/vmlinuz-3.13.0-49-generic'). This required change into
'/lib/img-functions' and 'baremetal' element to pick up signed kernel version
when this element is used.

Closes-Bug: 1443076
Change-Id: I60061cbea847b47fa752b9463cfd387e8e7f0635
2015-04-12 11:36:17 -07:00
Haomeng, Wang
cc9870ec11 Try 5 times for rmdir command call after umount
Sometime, we will get "Device or resource busy" during the mount point
deleting, umount return 0, but the resouce is busy for a while, so need
to add sleep interval to wait resource free, then we can delete it.
Change-Id: Idaa219d12e847824960eec8907739add5d619d1a
Closes-Bug: 1332521
2015-04-07 02:01:29 +00:00
Gregory Haynes
2d79e9d395 Short circuit qemu-img convert for raw images
We currently use qemu-img convert with a raw source and dest when
building raw images. We can just mv the file for increased speed.

Change-Id: I3da095cb9ecad7224a121a434a9fb204132bf6df
2015-03-25 22:50:42 +00:00
yogananth subramanian
57705694e7 ramdisk-image-create: add support for vmlinux file
ramdisk-image-create assumes the presence of vmlinuz file in OS image.
But ubuntu PPC LE images have only vmlinux file. This patch adds
support for using vmlinux file.

Change-Id: I5dd0b8ceb46b73be57d4c15b39b96f99b524fa3f
Closes-Bug: 1413362
2015-01-22 02:10:39 +05:30
Jenkins
2b60bea961 Merge "Allow for multiple image outputs from raw source" 2014-10-16 21:56:28 +00:00
Jenkins
212c6814aa Merge "Force empty $TMPDIR inside the chroot" 2014-10-16 20:04:50 +00:00
Jenkins
5b71da8899 Merge "Remove first-boot.d support" 2014-10-16 18:36:59 +00:00
Ghe Rivero
873de01491 Remove first-boot.d support
After being deprecated two releases ago, finally remove any reference
for the support of first-boot.d

Change-Id: I08d67404ef48cad61db3b18fb86e970abfa5d2b6
2014-10-16 09:48:14 +00:00
Ghe Rivero
d3c89d85b6 Force empty $TMPDIR inside the chroot
There are some situations (use of libpam-tmpdir) where the $TMPDIR is
defined per user, so any sudo call will define an specific $TMPDIR for
the root user which doesn't exist inside the chroot.

Change-Id: I866651fed520007506c7be83837d3791ed9f2235
Closes-Bug: #1330290
2014-10-16 09:28:53 +00:00
Jenkins
16b0e5e856 Merge "Preserve exit value when leaving cleanup trap" 2014-10-14 21:05:14 +00:00
Clark Boylan
ae928057bd Allow for multiple image outputs from raw source
When uploading images to multiple clouds it is possible that the same
image will be needed in multiple formats to accomodate hypervisors
across clouds. Update disk-image-create's -t flag to take a list of
desired output image formats so that a single disk-image-create can
output all of the desired image formats.

Change-Id: If121b2342ae888855ba435aa3189f039e985b812
2014-10-13 09:16:42 -07:00
Steve Kowalik
c6f3c1aae1 Echo that qemu-img convert is running
qemu-img convert can take a few minutes to complete, or far longer
on heavily loaded systems. Notify the user that the image is
undergoing conversion to save them just seeing END PROFILING.

Change-Id: I1ad61ce5ed011b721d48e2d12fb42ef4aa5cd1f6
2014-10-13 15:01:49 +11:00
Gregory Haynes
f06493bcd6 Preserve exit value when leaving cleanup trap
If we entered the cleanup trap due to exit with an error code we should
exit dib with an error code.

Change-Id: Iee1a05668b3239113fb91a2da0d9a66d7de4db6b
2014-10-10 17:04:50 -07:00
Juerg Haefliger
3af14a5bad Handle non-existing *generic kernel and initrd
Finding the Debian boot kernel and initrd is collapsed into a
single statement so that the script doesn't bail out when
'set -o pipefail' is set and the *generic kernel and initrd are
absent.

Change-Id: Ifd616818f7387e6f102636c7cf186be6097f1d5c
Closes-Bug: 1335016
2014-09-16 11:57:01 +02:00
Dirk Mueller
d4198bde24 Replace backticks with $()
It seems this is generally preferred in shell
code to what I heard from reviewers.

Change-Id: If61813bd3c4bc61d0282232c99f8011e776eba8b
2014-09-05 19:00:06 +02:00
Dirk Mueller
58f5a543d4 Fix openSUSE kernel/initrd detection
For some of the scripts, the unversioned name is not
good enough. so determine the link target of "vmlinuz" and
"initrd" to get the full versioned path.

Change-Id: I52f9b5435fafaf7925e2332058a9e41b26478f1c
2014-09-05 18:59:03 +02:00
AzherKhan
d06a1caafb Correction: if then statement code style.
Corrected the if; then statement to follow bash convention.

Change-Id: Ia3f80520bb9446c4feb5ce36dccffa2b1c1d72c4
2014-07-07 03:43:15 -07:00
Jenkins
d92cbb7b1e Merge "Disk-image-create should allow sending compat flags to qemu-img" 2014-06-28 00:14:07 +00:00
Yolanda Robla
fabbb8da58 Disk-image-create should allow sending compat flags to qemu-img
Add a --options flag that allows disk-image-create to send custom
options to qemu-img calls. It is useful for forcing compat into
qemu-img created images.

Usage:
    disk-image-create --qemu-img-options compat=0.10 base ubuntu

Change-Id: I73ee2c86abc115220d51ffebbbe9ce3c4d92f188
Closes-Bug: #1329746
2014-06-25 12:38:06 +02:00
Om Kumar
bccffc8bfd Refactor code to select boot kernel
The script to find best kernel from image (used by vm
and baremetal element) is duplicated and is not in sync
with each other. Moving the code to img-functions as a
function will reduce duplicated code and make it reusable
in future.

Since img-functions is not accessible in chroot env,
kernel selection is being moved from finalise.d to
cleanup.d in the vm element.

Change-Id: I8fbccc13a2c61a5191ef9ea5d2a8302a3e43b000
2014-06-13 11:46:39 +05:30
Monty Taylor
0c08692e75 Respect inmutable resolv.conf in the image
In some instances, the content of the image may want to convey specific
resolv.conf settings. For instance, if unbound is installed, it's also
desirable to set resolv.conf to point to it.

Short circuit diskimage-builder's friendliness around resolv.conf files
by not dying in a fire if it finds the file to be immutable.

Change-Id: I88632fb79289681dcf95d32f39a0ad658ba39a60
2014-04-25 16:35:44 -07:00
Robert Collins
dc93feee69 Remove an excess cp of disk images.
save_image is used to copy kernel and ramdisks out of the image, which
we will sometimes want to keep the source, and sometimes not. However
for the main image itself, the temp copy is never kept, so use mv
  rather than cp and avoid the excess IO.

Change-Id: I5a9f0d69ffee3e6b872a8927537ac17f02f5aa4d
2014-03-25 17:20:37 +00:00
JUN JIE NAN
893211f19a Refactor unmount_image with unmount_dir
To replace the similiar umount logic in unmout image.

Change-Id: Ia2105a48bbed00ba2571b442e216fccfbcc17f01
2014-01-03 16:55:17 +08:00
Mark McLoughlin
af8b7f05a1 Quieten disk-image-get-kernel
disk-image-get-kernel is pretty noisy and you see e.g.:

  $> load-image overcloud-compute.qcow2
  XXX  -d '/tmp/image.lWGCgPoj' -o 'tmp' -i '/home/stack/overcloud-compute.qcow2' --
  Extracting kernel + ramdisk from /home/stack/overcloud-compute.qcow2 and writing them to /tmp/image.lWGCgPoj
  nbd                    17554  0
  nbd                    17554  0
  basename: missing operand
  Try 'basename --help' for more information.
  /dev/nbd0 disconnected
  tmp-vmlinuz,tmp-initrd

Clean all this up so we just get:

  $> load-image overcloud-compute.qcow2
  Extracting kernel + ramdisk from /home/stack/overcloud-compute.qcow2 to tmp-vmlinuz and tmp-initrd in /tmp/image.g6b0lG88

Change-Id: I8971ec0bbcd87157b07fc17254c56bb9f9f2a597
2013-12-09 13:39:29 +00:00
Tim Serong
43c0082aa7 Move /tmp/ccache setup to base element
Commit c7d80dd (Cleanup mount points automatically) removed the unmount
of $TMP_MOUNT_PATH/tmp/ccache in run_d_in_target() and moved the
"rm /tmp/ccache" to elements/base/finalise.d/02-remove-ccache.  There
are two problems with this:

1) Not unmounting at the end of run_d_in_target() results in tmp/ccache
   being bind mounted muliple times on top of itself (three times, if you
   just run `disk-image-create base`).  It is eventually unmounted, but
   somehow the auto unmount code is confused, and tries to unmount it
   one more time than it was mounted, which results in an error like
   "umount: /tmp/image.THQkZxQa/mnt/tmp/ccache: not mounted".
   This doesn't actually break anything, but it's a little messy.

2) "rm /tmp/ccache" in elements/base/finalise.d/02-remove-ccache never
   succeeds in removing /tmp/ccache, because that hook is invoked by
   run_d_in_target(), *while* /tmp/ccache is mounted.

This present commit solves the above by moving the ccache setup glue out
of img-functions and into the base element's root.d.  This has the
following implications:

1) lib/img-functions is a little cleaner.

2) /tmp/ccache is available in the chroot during the root, extra-data,
   pre-install, install and post-install stages.  It is not available
   during block-device, finalise and cleanup stages as it will have been
   automatically unmounted by then.

3) /tmp/ccache won't be setup if you're building an image that doesn't
   include the base element.

Change-Id: Ief4c0a6f4ec622db6c6f652776215684178d8943
2013-10-10 16:27:09 +11:00
Clint Byrum
e29cc0c10c Use lazy umount to avoid race problems with dev
A problem with unmounting the dev filesystem in Ubuntu images caused
the umount of the /dev bind mount to fail, which left it there to be
removed during the mv -t step, causing the build host's /dev to be
wiped out.

The lazy umount will detach it from the filesystem hierarchy and then
clean up the mount reference later.

Change-Id: I8f8cea857c445fb0b4fd02bc063722fb1553c947
2013-09-18 14:09:24 -07:00
Ghe Rivero
a495079695 Delete -new image once copied
Before, this was created on /tmp/image.XXX so was removed at the
end of the script.

Change-Id: I8ceb97626d389c5bcb66fa3058f74388009ea677
2013-09-05 09:19:15 +00:00
Ghe Rivero
dbfca7a816 Combine compress and save image into one function
When compressing an image, this is done in the same dir where the raw
image resides, doubling the amount of space needed (scarce when
using tmpfs), and then it's moved to the .cache folder in disk.

Combining these two functions, we reduce the amount of space needed
in the tmpfs partition (when in use), and the compressed image is
created directly on the .cache folder disk, so there is no need to
move the compressed image after the process into disk.

Change-Id: I451d24bdd6fa0983414244135dff5e96c0549833
2013-08-28 10:51:40 +00:00
Chris Alfonso
1dea2f3524 Call sync before unmounting keeps the mount from being busy.
here is a bug associated with this patch because there is a
workaround included. Calling sync to enable the unmounting...
https://bugs.launchpad.net/diskimage-builder/+bug/1214388

Change-Id: I9d5a1f092fd8f3ae1c14de03dd516203baab4da3
2013-08-21 09:10:35 -04:00
Robert Collins
d6d941a8f1 Make $HOME in the chroot be reasonable.
With sudo -E $HOME is inherited, which is nearly always wrong.

Change-Id: Ic77b9c2892c19a2fc08a07d35f76da79a346ebac
2013-08-10 01:16:02 +12:00
Robert Collins
c7d80ddeed Cleanup mount points automatically.
Manually listing the mount points that need cleanup is not
maintainable and makes it hard to write elements that use bind
mounting to inject resources into the build environment.

Change-Id: I7c9ade444f4ebe42552f8e321f257a7ec0a674ef
2013-08-09 22:53:32 +12:00
Derek Higgins
4651913e55 Add a new break on error
A user running di-b several times while developing an element may not
want to drop to a shell in all cases but may only want to do so if one
of their in target hooks failed.

This patch gives them the ability to do so, If break=after-error is set
then a user will be provided a in target shell taking over from where the
last failed command left off.

Change-Id: Ia2f7ac4c21b64b971f87f4ae9cb867981b13eb5e
2013-08-01 17:08:54 +01:00