Commit Graph

54 Commits

Author SHA1 Message Date
Juerg Haefliger
3af14a5bad Handle non-existing *generic kernel and initrd
Finding the Debian boot kernel and initrd is collapsed into a
single statement so that the script doesn't bail out when
'set -o pipefail' is set and the *generic kernel and initrd are
absent.

Change-Id: Ifd616818f7387e6f102636c7cf186be6097f1d5c
Closes-Bug: 1335016
2014-09-16 11:57:01 +02:00
Dirk Mueller
d4198bde24 Replace backticks with $()
It seems this is generally preferred in shell
code to what I heard from reviewers.

Change-Id: If61813bd3c4bc61d0282232c99f8011e776eba8b
2014-09-05 19:00:06 +02:00
Dirk Mueller
58f5a543d4 Fix openSUSE kernel/initrd detection
For some of the scripts, the unversioned name is not
good enough. so determine the link target of "vmlinuz" and
"initrd" to get the full versioned path.

Change-Id: I52f9b5435fafaf7925e2332058a9e41b26478f1c
2014-09-05 18:59:03 +02:00
AzherKhan
d06a1caafb Correction: if then statement code style.
Corrected the if; then statement to follow bash convention.

Change-Id: Ia3f80520bb9446c4feb5ce36dccffa2b1c1d72c4
2014-07-07 03:43:15 -07:00
Jenkins
d92cbb7b1e Merge "Disk-image-create should allow sending compat flags to qemu-img" 2014-06-28 00:14:07 +00:00
Yolanda Robla
fabbb8da58 Disk-image-create should allow sending compat flags to qemu-img
Add a --options flag that allows disk-image-create to send custom
options to qemu-img calls. It is useful for forcing compat into
qemu-img created images.

Usage:
    disk-image-create --qemu-img-options compat=0.10 base ubuntu

Change-Id: I73ee2c86abc115220d51ffebbbe9ce3c4d92f188
Closes-Bug: #1329746
2014-06-25 12:38:06 +02:00
Om Kumar
bccffc8bfd Refactor code to select boot kernel
The script to find best kernel from image (used by vm
and baremetal element) is duplicated and is not in sync
with each other. Moving the code to img-functions as a
function will reduce duplicated code and make it reusable
in future.

Since img-functions is not accessible in chroot env,
kernel selection is being moved from finalise.d to
cleanup.d in the vm element.

Change-Id: I8fbccc13a2c61a5191ef9ea5d2a8302a3e43b000
2014-06-13 11:46:39 +05:30
Monty Taylor
0c08692e75 Respect inmutable resolv.conf in the image
In some instances, the content of the image may want to convey specific
resolv.conf settings. For instance, if unbound is installed, it's also
desirable to set resolv.conf to point to it.

Short circuit diskimage-builder's friendliness around resolv.conf files
by not dying in a fire if it finds the file to be immutable.

Change-Id: I88632fb79289681dcf95d32f39a0ad658ba39a60
2014-04-25 16:35:44 -07:00
Robert Collins
dc93feee69 Remove an excess cp of disk images.
save_image is used to copy kernel and ramdisks out of the image, which
we will sometimes want to keep the source, and sometimes not. However
for the main image itself, the temp copy is never kept, so use mv
  rather than cp and avoid the excess IO.

Change-Id: I5a9f0d69ffee3e6b872a8927537ac17f02f5aa4d
2014-03-25 17:20:37 +00:00
JUN JIE NAN
893211f19a Refactor unmount_image with unmount_dir
To replace the similiar umount logic in unmout image.

Change-Id: Ia2105a48bbed00ba2571b442e216fccfbcc17f01
2014-01-03 16:55:17 +08:00
Mark McLoughlin
af8b7f05a1 Quieten disk-image-get-kernel
disk-image-get-kernel is pretty noisy and you see e.g.:

  $> load-image overcloud-compute.qcow2
  XXX  -d '/tmp/image.lWGCgPoj' -o 'tmp' -i '/home/stack/overcloud-compute.qcow2' --
  Extracting kernel + ramdisk from /home/stack/overcloud-compute.qcow2 and writing them to /tmp/image.lWGCgPoj
  nbd                    17554  0
  nbd                    17554  0
  basename: missing operand
  Try 'basename --help' for more information.
  /dev/nbd0 disconnected
  tmp-vmlinuz,tmp-initrd

Clean all this up so we just get:

  $> load-image overcloud-compute.qcow2
  Extracting kernel + ramdisk from /home/stack/overcloud-compute.qcow2 to tmp-vmlinuz and tmp-initrd in /tmp/image.g6b0lG88

Change-Id: I8971ec0bbcd87157b07fc17254c56bb9f9f2a597
2013-12-09 13:39:29 +00:00
Tim Serong
43c0082aa7 Move /tmp/ccache setup to base element
Commit c7d80dd (Cleanup mount points automatically) removed the unmount
of $TMP_MOUNT_PATH/tmp/ccache in run_d_in_target() and moved the
"rm /tmp/ccache" to elements/base/finalise.d/02-remove-ccache.  There
are two problems with this:

1) Not unmounting at the end of run_d_in_target() results in tmp/ccache
   being bind mounted muliple times on top of itself (three times, if you
   just run `disk-image-create base`).  It is eventually unmounted, but
   somehow the auto unmount code is confused, and tries to unmount it
   one more time than it was mounted, which results in an error like
   "umount: /tmp/image.THQkZxQa/mnt/tmp/ccache: not mounted".
   This doesn't actually break anything, but it's a little messy.

2) "rm /tmp/ccache" in elements/base/finalise.d/02-remove-ccache never
   succeeds in removing /tmp/ccache, because that hook is invoked by
   run_d_in_target(), *while* /tmp/ccache is mounted.

This present commit solves the above by moving the ccache setup glue out
of img-functions and into the base element's root.d.  This has the
following implications:

1) lib/img-functions is a little cleaner.

2) /tmp/ccache is available in the chroot during the root, extra-data,
   pre-install, install and post-install stages.  It is not available
   during block-device, finalise and cleanup stages as it will have been
   automatically unmounted by then.

3) /tmp/ccache won't be setup if you're building an image that doesn't
   include the base element.

Change-Id: Ief4c0a6f4ec622db6c6f652776215684178d8943
2013-10-10 16:27:09 +11:00
Clint Byrum
e29cc0c10c Use lazy umount to avoid race problems with dev
A problem with unmounting the dev filesystem in Ubuntu images caused
the umount of the /dev bind mount to fail, which left it there to be
removed during the mv -t step, causing the build host's /dev to be
wiped out.

The lazy umount will detach it from the filesystem hierarchy and then
clean up the mount reference later.

Change-Id: I8f8cea857c445fb0b4fd02bc063722fb1553c947
2013-09-18 14:09:24 -07:00
Ghe Rivero
a495079695 Delete -new image once copied
Before, this was created on /tmp/image.XXX so was removed at the
end of the script.

Change-Id: I8ceb97626d389c5bcb66fa3058f74388009ea677
2013-09-05 09:19:15 +00:00
Ghe Rivero
dbfca7a816 Combine compress and save image into one function
When compressing an image, this is done in the same dir where the raw
image resides, doubling the amount of space needed (scarce when
using tmpfs), and then it's moved to the .cache folder in disk.

Combining these two functions, we reduce the amount of space needed
in the tmpfs partition (when in use), and the compressed image is
created directly on the .cache folder disk, so there is no need to
move the compressed image after the process into disk.

Change-Id: I451d24bdd6fa0983414244135dff5e96c0549833
2013-08-28 10:51:40 +00:00
Chris Alfonso
1dea2f3524 Call sync before unmounting keeps the mount from being busy.
here is a bug associated with this patch because there is a
workaround included. Calling sync to enable the unmounting...
https://bugs.launchpad.net/diskimage-builder/+bug/1214388

Change-Id: I9d5a1f092fd8f3ae1c14de03dd516203baab4da3
2013-08-21 09:10:35 -04:00
Robert Collins
d6d941a8f1 Make $HOME in the chroot be reasonable.
With sudo -E $HOME is inherited, which is nearly always wrong.

Change-Id: Ic77b9c2892c19a2fc08a07d35f76da79a346ebac
2013-08-10 01:16:02 +12:00
Robert Collins
c7d80ddeed Cleanup mount points automatically.
Manually listing the mount points that need cleanup is not
maintainable and makes it hard to write elements that use bind
mounting to inject resources into the build environment.

Change-Id: I7c9ade444f4ebe42552f8e321f257a7ec0a674ef
2013-08-09 22:53:32 +12:00
Derek Higgins
4651913e55 Add a new break on error
A user running di-b several times while developing an element may not
want to drop to a shell in all cases but may only want to do so if one
of their in target hooks failed.

This patch gives them the ability to do so, If break=after-error is set
then a user will be provided a in target shell taking over from where the
last failed command left off.

Change-Id: Ia2f7ac4c21b64b971f87f4ae9cb867981b13eb5e
2013-08-01 17:08:54 +01:00
Chris Jones
8d1ce9c0c3 Build ramdisks in an image chroot.
Ramdisks are now built inside a chroot which is built by the normal
image build process. Doing so improves our independence of the
precise state of the build host.

This fixes bug 1194055.

Change-Id: Ibc254fbb9e7b404b5f38c1b35bcde8a4136e8e28
2013-07-12 11:09:35 +01:00
Clint Byrum
f6cc6bb185 Use ccache to cache all compiles between builds.
Many python extensions are built from source each time an image is
built. Repeating these is wasteful, so ccache is employed to eliminate
that waste. A cache dir outside the chroot is used to speed up
subsequent image builds.

Change-Id: Ib73563ddbe5f3be7454bfc54ab91cedb559a1304
2013-07-09 18:18:19 -07:00
Robert Collins
a969be49a2 Move functions to common-functions for reuse.
Change-Id: I12a5687d94df678d7f7e8c0b075ee4aba1082156
2013-06-25 17:36:37 +01:00
Lucas Alvares Gomes
1435f6e48d Make the finalise_base function less Ubuntu-ish.
The finalise_base function was creating a broken link at /etc/resolv.conf
when building a non ubuntu image.

Change-Id: I974d43fd6db365dbff9c85195e3b72950f3409da
2013-06-18 12:51:39 +01:00
Dan Prince
1fa27e745d Stop apt-get installing qemu-img.
This is simply a prerequisite and we should not try to
apt-get install it inline.

Remove this code and exit with an error if qemu-img is
not found.

Change-Id: I52d47a95d24d4bfe801ba664200604bd385dc67f
2013-06-11 10:03:10 -04:00
Lucas Alvares Gomes
02ee161e81 Refactor the first boot routine.
Setting up a routine to run at first time the system boot may vary from
distribution to distribution. This patch will remove the logic from the
diskimage-builder code and put it into the elements.

The base element will now deploy a dib-first-boot script and each OS
element will ensure that this script runs at rc.local time. By doing
that we will put the OS specific stuff in the OS elements and remove
the embedded code out of the img-function file.

Change-Id: I24c5d1b1185de5693f145347fe912245f1ba7dfe
2013-05-16 15:50:10 +01:00
Steve Baker
c6da246b73 Fix for running /etc/rc.local on Fedora
With systemd rc.local lives in /etc/rc.d and is only run
if the rc-local service is enabled.

This change creates a /etc/rc.d/rc.local symlink, and modifies img-functions
prepare_first_boot to tolerate an initially missing /etc/rc.local.

Fixes bug #1179674

Change-Id: I7e6102b62224950c1de5d48205dc4feff60f9389
2013-05-14 11:48:58 +12:00
Lucas Alvares Gomes
e537e046d1 Force the inclusion of /usr/local/bin in PATH.
Sudo compiled with the --with-secure-path option enabled (default on
Ubuntu, Fedora and potentially other distros) will cause sudo to ignore
any changes made to the PATH variable in the user's environment (even when
the -E option is used). Also, some distros might not have /usr/local/bin
set in their sudoers secure_path causing the diskimage-builder to fail
when building an image using such distros as host. This patch will force
the inclusion of the /usr/local/bin in the PATH enviroment variable
inside the chroot.

Change-Id: Idb7156d8ff124bfe685cc721337d9bb21f49dc9e
Fixes: bug #1175980
2013-05-08 23:32:47 +01:00
Chris Krelle
76d8d98a60 Remove img-functions from ramdisk-image-create.
Ramdisk-image-create sources both img-functions and ramdisk-functions
causing duplicate functions. To correct this situation this patch
removes the import of img-functions from ramdisk-image-create.
It moves cleanup_dirs and run_d functions from img-functions to
common-functions to allow this.

fixes bug 1175427
Authored-by: Chris Krelle <nobodycam@gmail.com>

Change-Id: I971d019b0d92b06d9de661663b49c433e823ea42
2013-05-08 07:39:30 -07:00
Clint Byrum
d9a2211d14 Only use tmpfs if build machine has 4GB+ RAM.
For machines with low RAM (or no access to /proc/meminfo) the builder
will still run, but will just build inside the filesystem that is
hosting /tmp. This will result in a slower build (especially if there
are a lot of .deb packages installed).

Fixes bug #1175453

Change-Id: I79f2672058c11e377548820df0ab4fad8f47ffdc
2013-05-03 06:55:53 -07:00
Robert Collins
d119c28f49 Revert "Improve Fedora build host support."
This reverts commit 591059ffd3.

Change-Id: I8ad5bde2eabe12c6575d51e7443d9b46521c0928
2013-05-03 08:16:43 +12:00
Jenkins
55993863fc Merge "Improve Fedora build host support." 2013-05-02 19:30:43 +00:00
Lucas Alvares Gomes
591059ffd3 Improve Fedora build host support.
Sudo command (even using -E) resets $PATH for security reasons, Fedora
and other distos might have to explicitly add /usr/local/bin to $PATH
in order to run the commands in the chroot environment.

Change-Id: I95068ecf1e7437152e11c6d6789f59bcfd6ae9cc
2013-05-01 09:40:15 +01:00
Robert Collins
cb62bae9b8 Build images using loopdev instead of qemu-nbd.
Qemu-nbd does not perform well with older versions of qemu due to
the lack of writeback caching mode. It also only builds qcow2 images
and there is a desire for raw image support. Finally, qemu-nbd makes
it very difficult to build images concurrently due to the somewhat
opaque nature of how it selects a /dev/nbd# device. losetup, on
the other hand, makes this process very straight forward.

Change-Id: I309fad8af4fd1e8d1720c17b65e1897a76d5e897
Co-Author: Clint Byrum <clint@fewbar.com>
2013-04-30 08:56:12 -07:00
Ghe Rivero
a05629b9f4 Support multiple outputs disk formats
Using the IMAGE_TYPE env variable, you can select which ouput disk format you want for your image.
Only qcow2 format supports compression.

Change-Id: Ia01d9079c354dbd3cfcf870d86f97ee57bd9d479
2013-02-21 09:26:44 +01:00
Tim Miller
5544da4605 first-boot.d scripts log to a file:
Log first-boot.d script output to a file, to ease troubleshooting,
since the first-boot scripts are removed from the system after
running, even in case of failure.

Change-Id: I07d5c782dd918a804e054bf08efce89ed9442e0c
2013-02-15 10:04:50 -08:00
Clint Byrum
91c30f6ab2 Use dib-run-parts for running scripts in target
Also modified dib-run-parts to apply a more workable solution for
filtering out unwanted files such as editor backups and VCS.

The script is installed in its own element, depended on by the OS
specific ubuntu element. This is  because the ubuntu element (and
later other OS's) are responsible for populating the root filesystem.
If we try to install this in base, the root filesystem will look to
be populated already and we will skip automatically choosing ubuntu.

Change-Id: I017646748c1a8360299106289b57d976d45875a8
2013-02-14 10:48:10 -08:00
Robert Collins
7f77729ec5 Move the dpkg specific stuff to a dpkg element rather than being hardcoded.
This includes the install-packages implementation for dpkg, apt http proxy
config, daemon blocking and unblocking.

Change-Id: I8f159021d2b223d7003cec067de3aa605ad06974
2013-02-14 11:16:12 +13:00
Robert Collins
3860af4be5 Change run-parts usage to be compatible with Fedora.
Change-Id: I7a4e1252b0ced940cd4d8fa2fbd096c016933954
2013-02-14 11:16:12 +13:00
Tim Miller
56e50ee5cf Consolidate common OS installation into a script:
Move common openstack service installation operations
into a new script `os-svc-install`, which simplifies
the elements for openstack services.

Change-Id: Ied8ac3278e7fe8af76e24748ec4e598a84afa03c
2013-02-13 11:06:00 -08:00
Robert Collins
8220c00095 Bad code landed, causing ubuntu to be always landed.
Change-Id: Ia77073563fde70140169dbbc5a7448aad54295e2
2013-02-13 13:58:35 +13:00
Robert Collins
1ee3a01447 Move initial root contents into a hook.
This is a necessary but not complete step towards supporting Fedora and Suse
distributions. Further work is needed (e.g. to quiesce daemons on
installation).

Change-Id: If3ea6093d41a21de755db52328226b84b5a3ede6
2013-02-11 17:01:36 +13:00
Ghe Rivero
11b4a5501d Create install-packages as a binary.
Only supports Debian family distribution so far.

Change-Id: I9fa0b833c34a614fe498c6c6ae599733a69c1309
2013-01-31 00:08:58 +01:00
Chris Jones
0f73578721 Allow manual installation of packages.
Rather than force creation of an element for a single package install,
allow people to do this from the command line.

Change-Id: I63e2e7e50c4a7dbb8a8e198581dfadce91773621
2012-12-21 17:46:44 +00:00
Robert Collins
d03825b504 Add the ability to break into a shell during builds.
Change-Id: I87af952d892f8622e4c916085fc896c735a35438
2012-12-14 20:17:29 +13:00
Robert Collins
012116cad6 Move everything 'common' into base, making it avoidable for tests.
Change-Id: I87edd00f6e0edcd609856192c2c98604ef3274f0
2012-12-14 13:41:57 +13:00
Robert Collins
e8a8e99a08 Make it possible to not recompress the qcow2 image at the end.
Change-Id: Icca922853ee583dcccb6d3970d080fbec11edac7
2012-12-14 13:25:18 +13:00
Ghe Rivero
258185b7a2 Added true to avoid exit on error
disk-image-create just died without any warning

Change-Id: Ia5da4188d64bb22540b57d01184b91faa59149e7
2012-12-13 18:32:34 +01:00
Monty Taylor
1eff4a436e Rename flavour to element.
Flavour is overloaded in openstack due to it being used by nova. Element
seems to have the same feeling of combinability without using a term already
in active use in the openstack community.

Change-Id: Ia4c028d4062a8f69c66665821c94dd4bcdf06031
2012-12-05 14:04:58 -08:00
Darragh Bailey
e5cda1f731 Use system resolv.conf file when available
Where possible use the system configured resolv.conf file
to to support resolution of internal servers such as proxies.
Use previous hardcoded google dns nameserver as fallback

Change-Id: Ibc1e8a3b96e733a2f7fbcdd4b2c5d3e8926424c4
2012-12-03 15:18:09 +00:00
Ghe Rivero
38dc7680c9 Jenkins image fixed
- Removed http_proxy option
- Added sudoers entry to cp first-bot.d
- Fixed typo on 11-jenkins-plugins
- Restored trap exit on run-parts
2012-11-27 18:37:57 +01:00