Commit Graph

237 Commits

Author SHA1 Message Date
Yolanda Robla
fabbb8da58 Disk-image-create should allow sending compat flags to qemu-img
Add a --options flag that allows disk-image-create to send custom
options to qemu-img calls. It is useful for forcing compat into
qemu-img created images.

Usage:
    disk-image-create --qemu-img-options compat=0.10 base ubuntu

Change-Id: I73ee2c86abc115220d51ffebbbe9ce3c4d92f188
Closes-Bug: #1329746
2014-06-25 12:38:06 +02:00
Jenkins
20879b2ec9 Merge "avoid failure if /lib/firmware doesn't exist" 2014-06-24 16:46:20 +00:00
Jenkins
c0501291c9 Merge "Refactor code to select boot kernel" 2014-06-14 04:01:45 +00:00
Gonéri Le Bouder
131fb8c216 avoid failure if /lib/firmware doesn't exist
On Debian, /lib/firmware is provided by optional packages like
firmware-iwlwifi or firmware-linux-free. That's why this directory
may not exist.
This change ensure the directory exist before trying to copy it
content.

Change-Id: Icf9eedf4148141c92e9171f16ae744b88a8d8519
2014-06-14 00:05:27 +02:00
Jenkins
bedd50f80d Merge "fail at startup with no operating-system element" 2014-06-13 14:36:59 +00:00
Om Kumar
bccffc8bfd Refactor code to select boot kernel
The script to find best kernel from image (used by vm
and baremetal element) is duplicated and is not in sync
with each other. Moving the code to img-functions as a
function will reduce duplicated code and make it reusable
in future.

Since img-functions is not accessible in chroot env,
kernel selection is being moved from finalise.d to
cleanup.d in the vm element.

Change-Id: I8fbccc13a2c61a5191ef9ea5d2a8302a3e43b000
2014-06-13 11:46:39 +05:30
Jenkins
fa9a561819 Merge "Rename old image file instead of rewrite it" 2014-06-12 22:19:15 +00:00
Gonéri Le Bouder
1011adf370 fail at startup with no operating-system element
Raise an error if there is no element with “operating-system”
in the element-provides file.

Change-Id: I2242537abc4c610252984c32c286bc6eb25de672
2014-06-12 11:19:20 +02:00
iberezovskiy
b740c50c76 Rename old image file instead of rewrite it
If we have the old image file with same name as new image file
we shouldn't rewrite it, just need to rename old.
Uses date number in old image name.

Fixes TODO in lib/common-functions

Change-Id: Ica58a0f4435217912bddf950322299d2ed2889f3
2014-06-06 12:59:28 +04:00
James Slagle
813b4bc242 Check return code of element-info
Ensure the return code of element-info is actually checked by moving the
export IMAGE_ELEMENT to the subsequent line.

A recent change (Ic0a061995e2ae708c95a535cba6237bff58f7d93) exported the
IMAGE_ELEMENT variable. This caused the return code of element-info to not
actually be checked, since the export command is now checked instead and
almost always will return 0.

This caused a misleading error message when using diskimage-builder with an
unknown element since the script didn't exit when it should have:
$ bin/disk-image-create -a amd64 foobar vm
Building elements: base  foobar vm
ERROR: Element 'foobar' not found in 'bin/../elements'
Expanded element dependencies to:
Building in /var/tmp/image.DuIDY1lW
Please include at least one distribution root element.

Change-Id: I3d9c02bf0b1a4f320280185dd3fab0697d2c92f2
2014-05-30 15:41:05 -04:00
Dan Prince
c55539b142 Add pkg-map element.
Adds a new pkg-map element which can be used to install
packages based on an in element 'pkg-map' JSON config
file format.

As part of this change we expose DIB's IMAGE_ELEMENT variable
so that we can have access to the element names which are being
installed in a clean manner.

This change is intended to decouple elements from DIB
and allow new elements to support multiple distributions
(with various package naming schemes) without having to
constantly maintain DIB's various bin/map-packages files.

Change-Id: Ic0a061995e2ae708c95a535cba6237bff58f7d93
2014-05-28 12:21:44 -04:00
Jenkins
dcb5494e04 Merge "Build raw image in separate tmpfs" 2014-05-08 23:15:27 +00:00
Dan Prince
956acd5f59 Eliminate 'tr' in favor of inline bash.
Updates the lib/common-functions generate_hooks() function
to use bash instead of exec'ing tr.

Change-Id: Ie32d3ce5c7c43affd0031a568d57763e36209603
2014-05-02 13:10:33 -04:00
Derek Higgins
3fada2566a Build raw image in separate tmpfs
If the tmpfs being used to hold the image filesystem is close to being
full then housing the raw disk image in the same place can be
problematic as it tends to grow faster then the filesystem shrinks when
data is being moved into it.

Putting them both into separate tmpfs's will allow there to be an
overlap where they jointly use more then the size limit for a singe tmpfs.

Change-Id: Ia17ca357d3b865d70a2d3e13e0479d008ca5f924
Closes-Bug: #1289582
2014-05-01 11:46:29 +01:00
Monty Taylor
0c08692e75 Respect inmutable resolv.conf in the image
In some instances, the content of the image may want to convey specific
resolv.conf settings. For instance, if unbound is installed, it's also
desirable to set resolv.conf to point to it.

Short circuit diskimage-builder's friendliness around resolv.conf files
by not dying in a fire if it finds the file to be immutable.

Change-Id: I88632fb79289681dcf95d32f39a0ad658ba39a60
2014-04-25 16:35:44 -07:00
Ben Nemec
381ff6ab1d Fix set -eu and pipefail failures
Fixes problems found by set -eu and pipefail, including:
-Many unset variables
-Commands that can fail under normal circumstances, which breaks
with set -e.  This change swallows those expected errors to allow
our existing error code to handle them.
-The dkms element was not finding Fedora kernel versions correctly.
This may be an issue for other distros too, but since Fedora was
working fine without this functionality I only changed it to print
a warning message rather than failing the build when it happens.
-The ramdisk init script will not be set -eu because if it fails
the result is a kernel panic, which can be tricky to debug.
However, in testing with set -e a few failing commands were found
and have been fixed in this patch.

Change-Id: I44cf98dfc80cfcaec54b88cc83be80a3dbf2cec3
2014-04-15 20:53:15 -05:00
Derek Higgins
85dadd0d90 Revert "Only create a tmpfs big enough for DIB_MIN_TMPFS"
This didn't pass CI and is now causing failures as it moves us back to a
4G tmpfs.

Change-Id: Ie92fd909ad21216b1fda55c21457893df38e882f
2014-03-26 10:07:41 +00:00
Jenkins
bfba0ccd03 Merge "Remove an excess cp of disk images." 2014-03-26 08:38:52 +00:00
Robert Collins
dc93feee69 Remove an excess cp of disk images.
save_image is used to copy kernel and ramdisks out of the image, which
we will sometimes want to keep the source, and sometimes not. However
for the main image itself, the temp copy is never kept, so use mv
  rather than cp and avoid the excess IO.

Change-Id: I5a9f0d69ffee3e6b872a8927537ac17f02f5aa4d
2014-03-25 17:20:37 +00:00
Clint Byrum
e63625a7e1 Only create a tmpfs big enough for DIB_MIN_TMPFS
This will help to ensure images that would cause bug #1289582 would
still fail even on systems that have more RAM. Also this will ensure we
only reserve enough tmpfs space for this build.

Change-Id: Icece64e691d126a5fc89262897630fdbef1ffdf1
2014-03-07 12:52:15 -08:00
Ralf Haferkamp
05d8d4ee43 Include /lib64 into the deploy ramdisk on openSUSE
Change-Id: I88262f6793e1ed3e7f061888ce36b2b134e19507
2014-01-27 09:29:12 +01:00
Ralf Haferkamp
9dcd13d45c Make copy_required_libs() more robust
On some systems ldd gives a slightly different output for VDSOs. It doesn't
contain a '=>'. E.g.:
   linux-vdso.so.1 (0x00007fff2f4a6000)
instead of:
   linux-vdso.so.1 => (0x00007fff2f4a6000)
This patch simply skips all entries in the ldd output which don't expand to a
path name to workaround that.

Change-Id: Ie37637890b775b36bb31af4e586e61131bd80fa8
2014-01-22 21:54:26 +01:00
JUN JIE NAN
893211f19a Refactor unmount_image with unmount_dir
To replace the similiar umount logic in unmout image.

Change-Id: Ia2105a48bbed00ba2571b442e216fccfbcc17f01
2014-01-03 16:55:17 +08:00
JUN JIE NAN
6b7cf26689 Fixed device or resource busy issue in EXIT trap
cleanup for ramdisk failed to umount TMP_BUILD_DIR with device or
resource busy error.

The patch unmounts all the mountpoints under TMP_BUILD_DIR and
detaches loop devices associated with TMP_IMAGE_PATH.

The unmounts are applied with both force(-f) and lazy(-l)
options. Force option is only for NFS mounts, it's kept here since no
harm for lazy option.

Change-Id: I84035e6a003d8135186b2fda3facbd2c37967529
2014-01-03 16:53:38 +08:00
James Slagle
56f4ef0851 Make sure sbin paths are in $PATH
The sbin paths (/sbin, /usr/sbin, /usr/local/sbin) need to be in $PATH.
Since $PATH is set in the build environment from the host, if the host
does not have these paths set, they need to be added to $PATH.  In
particular, they must be set for apt-get which calls out to ldconfig,
start-stop-daemon, etc.

The previous check was insufficient because if you had /usr/sbin or
/usr/local/sbin in your $PATH, the addition to $PATH was not done since
the "if" statement evaluated to true.

Change-Id: I52b41d2eb9794e7ee0ae600215cb3ceca191582f
2013-12-16 14:42:59 -05:00
Mark McLoughlin
af8b7f05a1 Quieten disk-image-get-kernel
disk-image-get-kernel is pretty noisy and you see e.g.:

  $> load-image overcloud-compute.qcow2
  XXX  -d '/tmp/image.lWGCgPoj' -o 'tmp' -i '/home/stack/overcloud-compute.qcow2' --
  Extracting kernel + ramdisk from /home/stack/overcloud-compute.qcow2 and writing them to /tmp/image.lWGCgPoj
  nbd                    17554  0
  nbd                    17554  0
  basename: missing operand
  Try 'basename --help' for more information.
  /dev/nbd0 disconnected
  tmp-vmlinuz,tmp-initrd

Clean all this up so we just get:

  $> load-image overcloud-compute.qcow2
  Extracting kernel + ramdisk from /home/stack/overcloud-compute.qcow2 to tmp-vmlinuz and tmp-initrd in /tmp/image.g6b0lG88

Change-Id: I8971ec0bbcd87157b07fc17254c56bb9f9f2a597
2013-12-09 13:39:29 +00:00
JUN JIE NAN
a85f4d11f7 Fix no busybox symlinks issue on rhel
busybox on rhel has no `--list' option, which cause populate_busybox
can not create busybox symlinks. So the ramdisk created will not work
totally.

Fixes bug #1257669

Change-Id: I4f905673bb577867c99e0ccc0756c56a292474f5
2013-12-05 15:25:46 +08:00
Dan Prince
255387a288 Drop default distribution root element support
Converts our existing default root element code to be just a check
which exits with a failure message if no root/distribution element
is found.

Change-Id: I954a6abfd7871d5807b1a171a03fa98932410cff
2013-12-02 11:01:29 -05:00
Jan Provaznik
f6f4e808d1 Remove BUSYBOX variable from ramdisk-defaults
ramdisk-defaults file is sourced before chroot to an image being created
is executed. Busybox path should be set inside chroot instead.

Change-Id: Ic0e1d3093ca9d489dd96b14157e93b336cf0baee
2013-11-11 14:36:15 +01:00
Chris Jones
0ff3b94c86 Remove framebuffer video drivers from ramdisks.
We now forcibly remove all framebuffer video drivers from ramdisks, as
part of our ongoing effort to keep every kernel we boot, in text mode,
so it can be more easily intercepted by LOM hardware, nova console
logging, etc.

Fixes-Bug: #1178409

Change-Id: Ia72211c97c03c3f60eb9a1483297fe587de51f0b
2013-10-17 12:14:01 +01:00
Tim Serong
43c0082aa7 Move /tmp/ccache setup to base element
Commit c7d80dd (Cleanup mount points automatically) removed the unmount
of $TMP_MOUNT_PATH/tmp/ccache in run_d_in_target() and moved the
"rm /tmp/ccache" to elements/base/finalise.d/02-remove-ccache.  There
are two problems with this:

1) Not unmounting at the end of run_d_in_target() results in tmp/ccache
   being bind mounted muliple times on top of itself (three times, if you
   just run `disk-image-create base`).  It is eventually unmounted, but
   somehow the auto unmount code is confused, and tries to unmount it
   one more time than it was mounted, which results in an error like
   "umount: /tmp/image.THQkZxQa/mnt/tmp/ccache: not mounted".
   This doesn't actually break anything, but it's a little messy.

2) "rm /tmp/ccache" in elements/base/finalise.d/02-remove-ccache never
   succeeds in removing /tmp/ccache, because that hook is invoked by
   run_d_in_target(), *while* /tmp/ccache is mounted.

This present commit solves the above by moving the ccache setup glue out
of img-functions and into the base element's root.d.  This has the
following implications:

1) lib/img-functions is a little cleaner.

2) /tmp/ccache is available in the chroot during the root, extra-data,
   pre-install, install and post-install stages.  It is not available
   during block-device, finalise and cleanup stages as it will have been
   automatically unmounted by then.

3) /tmp/ccache won't be setup if you're building an image that doesn't
   include the base element.

Change-Id: Ief4c0a6f4ec622db6c6f652776215684178d8943
2013-10-10 16:27:09 +11:00
Tim Serong
0a5d3a9446 Mount /dev/pts in chroot
Mounting /dev/pts fixes mildly irritating errors during image creation,
i.e.: "Can not write log, openpty() failed (/dev/pts not mounted?)"
emitted by apt-get.

Change-Id: I4106bc584f1b29ce9283a20064bb5ae447f17fbe
2013-10-08 21:41:18 +11:00
Clint Byrum
d07921eca5 Deprecate first-boot.d
Print a message and pause the build for 10 seconds to ensure interactive
users see the message.

Fixes bug #1212080

Change-Id: Ia388a54892c479e428b0ed7b8c70d64d65010e21
2013-09-27 09:42:51 -07:00
Clint Byrum
e29cc0c10c Use lazy umount to avoid race problems with dev
A problem with unmounting the dev filesystem in Ubuntu images caused
the umount of the /dev bind mount to fail, which left it there to be
removed during the mv -t step, causing the build host's /dev to be
wiped out.

The lazy umount will detach it from the filesystem hierarchy and then
clean up the mount reference later.

Change-Id: I8f8cea857c445fb0b4fd02bc063722fb1553c947
2013-09-18 14:09:24 -07:00
Ghe Rivero
a495079695 Delete -new image once copied
Before, this was created on /tmp/image.XXX so was removed at the
end of the script.

Change-Id: I8ceb97626d389c5bcb66fa3058f74388009ea677
2013-09-05 09:19:15 +00:00
Jenkins
771f3ec141 Merge "Add option --min-tmpfs <size> to disk-image-create" 2013-08-28 15:40:31 +00:00
Ghe Rivero
fefc28a918 Add option --min-tmpfs <size> to disk-image-create
In some scenarios, the required space in the tmpfs partition can be
larger (or smaller) than the default one, producing errors due to
the lack of enough space (or performance penalties for not using
tmpfs).

Using --min-tmpfs <size>, we can hint the working set size we'll need
and let dib choose to avoid or use tmpfs.

Change-Id: I7d5fe498302a100c8555ae542268e14b21f3a0c5
2013-08-28 11:14:06 +00:00
Ghe Rivero
dbfca7a816 Combine compress and save image into one function
When compressing an image, this is done in the same dir where the raw
image resides, doubling the amount of space needed (scarce when
using tmpfs), and then it's moved to the .cache folder in disk.

Combining these two functions, we reduce the amount of space needed
in the tmpfs partition (when in use), and the compressed image is
created directly on the .cache folder disk, so there is no need to
move the compressed image after the process into disk.

Change-Id: I451d24bdd6fa0983414244135dff5e96c0549833
2013-08-28 10:51:40 +00:00
Chris Alfonso
1dea2f3524 Call sync before unmounting keeps the mount from being busy.
here is a bug associated with this patch because there is a
workaround included. Calling sync to enable the unmounting...
https://bugs.launchpad.net/diskimage-builder/+bug/1214388

Change-Id: I9d5a1f092fd8f3ae1c14de03dd516203baab4da3
2013-08-21 09:10:35 -04:00
Chris Alfonso
35364749de Remove a device mapping, then let the loop device get removed
Change-Id: Ie1eb621ff56688208421d0115547f127a3c7fd56
2013-08-20 16:58:25 -04:00
Robert Collins
d6d941a8f1 Make $HOME in the chroot be reasonable.
With sudo -E $HOME is inherited, which is nearly always wrong.

Change-Id: Ic77b9c2892c19a2fc08a07d35f76da79a346ebac
2013-08-10 01:16:02 +12:00
Robert Collins
c7d80ddeed Cleanup mount points automatically.
Manually listing the mount points that need cleanup is not
maintainable and makes it hard to write elements that use bind
mounting to inject resources into the build environment.

Change-Id: I7c9ade444f4ebe42552f8e321f257a7ec0a674ef
2013-08-09 22:53:32 +12:00
Chris Jones
357c2de681 Remove explicit sudoers requirement.
Most of our build process runs as root and the overhead of explicitly
tracking sudo commands in the shipped sudoers.d file, plus the
operational overhead of keeping the sudoers.d file updated on build
hosts over time, is now considered to be unnecessary.
Instead, we now document that the build process will expect sudo
commands to work and the operator must make this happen (either by
running as root, via sudo -E, or allowing passwordless sudo for the
build user).

Change-Id: If2628e7d0c7efde0fb99c84a217523ba29b1b38d
2013-08-06 16:08:35 +12:00
Derek Higgins
4651913e55 Add a new break on error
A user running di-b several times while developing an element may not
want to drop to a shell in all cases but may only want to do so if one
of their in target hooks failed.

This patch gives them the ability to do so, If break=after-error is set
then a user will be provided a in target shell taking over from where the
last failed command left off.

Change-Id: Ia2f7ac4c21b64b971f87f4ae9cb867981b13eb5e
2013-08-01 17:08:54 +01:00
Jenkins
4284936084 Merge "Ensure $TMP_BUILD_DIR is actually created." 2013-08-01 00:41:52 +00:00
Jenkins
9233ca4056 Merge "Enable running disk-image-create on SUSE Linux" 2013-08-01 00:36:36 +00:00
James Slagle
de6efe2b2c Ensure $TMP_BUILD_DIR is actually created.
Previously if mktemp failed in function mk_build_dir, die was not
called.  The check for the exit code was actually checking that the
previous export statement completed successfully, not the call to
mktemp.

Change-Id: I477a2ce75c87c8167883ce7aa342e93c40770e29
2013-07-31 14:26:05 -04:00
Jeff Peeler
87fb533979 Modify relative paths of lib, elements for packaging
(Based on review https://review.openstack.org/#/c/36009)

Scripts test for existence of ../share/diskimage-builder and
fall-back to ../ if not found. This allows scripts to run unmodified
from a packaged installation or a local archive/repository.

Change-Id: I0cf4c1fdb8e42ec284c56860cb15818632b93b9e
2013-07-31 13:15:33 +12:00
Jenkins
1146fe63fc Merge "Add needed symlink for Fedora deployment ramdisk." 2013-07-28 19:58:59 +00:00
James Slagle
92fa96b2f6 Add needed symlink for Fedora deployment ramdisk.
Symlink /usr/lib -> ../lib for Fedora deployment ramdisk.  This
symlink is needed for systemd-udevd.  Otherwise, the network device is
not loaded and initialized when the deployment image is booted.  This
logic was previously contained in
elements/deploy/root.d/50-redhat-prepare-fs, but now that ramdisk
images are built in a chroot, the change needs to be moved so that it
is applied to the ramdisk build.

Change-Id: Icea43230126956ccf5fb8a6a96ca706b75d5c32f
2013-07-26 11:09:29 -04:00
Derek Higgins
e79968f682 Only remove $TMP_BUILD_DIR on cleanup
During ramdisk cleanup if for some reason the filesystems mounted
in $TMP_BUILD_DIR were not removed this recursive delete ended up
deleting parts of the host filesystem that were mounted (in particular
most of the contents of /dev got removed.

This commit doesn't deal with the reason the filesystem didn't unmounted
but ensures the consequences are not as severe.

Fixes Bug 1202612

Change-Id: Id8eba0753c2fe76c79ae1d952ce690d26e33e3ed
2013-07-26 11:03:42 +01:00
Tim Serong
ee5ae03d58 Enable running disk-image-create on SUSE Linux
- Ensures /sbin and friends are in $PATH when invoked (without this,
  various sudo invocations fail in exciting ways).
- Use dib-run-parts in lib/common-functions instead of run-parts
  (neither SLES nor openSUSE ship run-parts).
- Ensure dib-run-parts doesn't descend into subdirectories (same
  behaviour as run-parts).
- Move dib-run-parts from root.d to bin (cleaner, consistent with
  other elements with separate bin scripts).
- Tested by building Ubuntu image on openSUSE 12.3.
- Note: this doesn't add support for creating SUSE images, it just
  lets you run disk-image-create on SUSE-based distros.

Change-Id: I906c6bc3cf51cdf2c4415adeae1ca250faac25e1
2013-07-24 20:16:31 +10:00
Robert Collins
f7f6cb45ee Fix the DIB_OFFLINE setting to actually work.
I missed the getopt parameter and forgot defaults are imported after
option processing. Untested code is broken code!

Change-Id: I133a691909d38e834c204950276a57f4884fc4ed
2013-07-18 13:19:25 +12:00
Robert Collins
b2f1d4e2af Document an interface for offline operation.
Complex image builds can download hundreds of MB of data from the
internet with many separate lookups. It would be nice to allow users
to ask for a fast build where those lookups are entirely avoided,
using locally cached resources (where possible). This new interface
allows users to signal to elements that they wish to operate without
updating cached resources, which will in turn allow us to avoid
checking for stale data at all.

As part of this I've also documented where we cache data, so that
things like the ccache cache dir and image cache files are not a
surprise to users.

Change-Id: I27f5de6ceaa4e9c6390721b7c434fe0908df84f5
2013-07-18 09:17:01 +12:00
Chris Jones
8d1ce9c0c3 Build ramdisks in an image chroot.
Ramdisks are now built inside a chroot which is built by the normal
image build process. Doing so improves our independence of the
precise state of the build host.

This fixes bug 1194055.

Change-Id: Ibc254fbb9e7b404b5f38c1b35bcde8a4136e8e28
2013-07-12 11:09:35 +01:00
Clint Byrum
f6cc6bb185 Use ccache to cache all compiles between builds.
Many python extensions are built from source each time an image is
built. Repeating these is wasteful, so ccache is employed to eliminate
that waste. A cache dir outside the chroot is used to speed up
subsequent image builds.

Change-Id: Ib73563ddbe5f3be7454bfc54ab91cedb559a1304
2013-07-09 18:18:19 -07:00
Richard Su
ad0fe876df Fix 50-firmware.rules no such file on Fedora 19
systemd on Fedora 19 has made /lib/udev/rules.d/50-firmware.rules
optional:
http://lists.freedesktop.org/archives/systemd-devel/2013-March/009708.html

Firmware is now handled by the in-kernel loader.

Added a file exist check for Fedora 19 and any other distros that
may have dropped the rules file. Backwards compatible with existing
distros.

Fixes: bug #1196409

Change-Id: I1d2acf3f88543736d75eb2e8766e83a3da194638
2013-07-03 14:10:33 -07:00
Robert Collins
a969be49a2 Move functions to common-functions for reuse.
Change-Id: I12a5687d94df678d7f7e8c0b075ee4aba1082156
2013-06-25 17:36:37 +01:00
Robert Collins
0b1ff7d4b0 Factor out element processing.
This reduces code duplication for hygiene, no functional changes.

Change-Id: I10c96da4f07bcd1e5aabfcfbd115ecaef7bb65a1
2013-06-24 16:30:25 +01:00
Robert Collins
9b01f2b129 Consolidate more ramdisk and disk-image code.
RAMDISK_ELEMENT is a spurious difference, to merge the code we need
only semantic difference to exist.

Change-Id: I7d38b2457b6fc838d1a536406bb3c34c00f2b1f9
2013-06-24 20:53:19 +12:00
Jenkins
e5e5845335 Merge "Reduce duplication between ramdisk and disk image codepaths." 2013-06-19 22:20:14 +00:00
Robert Collins
2c47b700be Reduce duplication between ramdisk and disk image codepaths.
Change-Id: Ie9e2d60ffe94f971953d52b7b5bce9c1a86fb1b5
2013-06-19 21:26:29 +12:00
Lucas Alvares Gomes
1435f6e48d Make the finalise_base function less Ubuntu-ish.
The finalise_base function was creating a broken link at /etc/resolv.conf
when building a non ubuntu image.

Change-Id: I974d43fd6db365dbff9c85195e3b72950f3409da
2013-06-18 12:51:39 +01:00
Dan Prince
1fa27e745d Stop apt-get installing qemu-img.
This is simply a prerequisite and we should not try to
apt-get install it inline.

Remove this code and exit with an error if qemu-img is
not found.

Change-Id: I52d47a95d24d4bfe801ba664200604bd385dc67f
2013-06-11 10:03:10 -04:00
Lucas Alvares Gomes
cd27dff29a Add fedora support for ramdisk-image-create.
This patch allows the creation of ramdisks from a fedora system.

Change-Id: Ic3b393af45ae1b85d015d2ceb5601ea723457da6
2013-05-31 15:37:31 +01:00
Lucas Alvares Gomes
02ee161e81 Refactor the first boot routine.
Setting up a routine to run at first time the system boot may vary from
distribution to distribution. This patch will remove the logic from the
diskimage-builder code and put it into the elements.

The base element will now deploy a dib-first-boot script and each OS
element will ensure that this script runs at rc.local time. By doing
that we will put the OS specific stuff in the OS elements and remove
the embedded code out of the img-function file.

Change-Id: I24c5d1b1185de5693f145347fe912245f1ba7dfe
2013-05-16 15:50:10 +01:00
Jenkins
7e0fe78cf2 Merge "Fix for running /etc/rc.local on Fedora" 2013-05-15 22:35:19 +00:00
Steve Baker
c6da246b73 Fix for running /etc/rc.local on Fedora
With systemd rc.local lives in /etc/rc.d and is only run
if the rc-local service is enabled.

This change creates a /etc/rc.d/rc.local symlink, and modifies img-functions
prepare_first_boot to tolerate an initially missing /etc/rc.local.

Fixes bug #1179674

Change-Id: I7e6102b62224950c1de5d48205dc4feff60f9389
2013-05-14 11:48:58 +12:00
Jenkins
ffdf0ba7ff Merge "Honor $DIB_IMAGE_SIZE." 2013-05-13 17:56:17 +00:00
Tim Miller
9983245249 Honor $DIB_IMAGE_SIZE.
The recent change to use loopdev instead of nbd stopped
honoring the DIB_IMAGE_SIZE setting.

This change adds it back, by resizing the image to
DIB_IMAGE_SIZE GB, if specified. If unspecified, it resizes
the image to (actual usage + 20%), as it did before this change.

Change-Id: I69afd9584e644ddacc948619100f153d3d8713a4
2013-05-13 10:20:53 -07:00
Clint Byrum
20661e8d80 Retry losetup -d for up to 10 seconds.
When partition tables are added to loopback devices, this can set off a
chain of udev hooks that may still be holding the loopback open. Failing
to detach loopback devices was the reason we were seeing leaked tmpfs
volumes.

Fixes bug #1178091

Change-Id: I836d6e2bbce824951dd4786e3ef28273ea18ee73
2013-05-09 09:34:53 -07:00
Lucas Alvares Gomes
e537e046d1 Force the inclusion of /usr/local/bin in PATH.
Sudo compiled with the --with-secure-path option enabled (default on
Ubuntu, Fedora and potentially other distros) will cause sudo to ignore
any changes made to the PATH variable in the user's environment (even when
the -E option is used). Also, some distros might not have /usr/local/bin
set in their sudoers secure_path causing the diskimage-builder to fail
when building an image using such distros as host. This patch will force
the inclusion of the /usr/local/bin in the PATH enviroment variable
inside the chroot.

Change-Id: Idb7156d8ff124bfe685cc721337d9bb21f49dc9e
Fixes: bug #1175980
2013-05-08 23:32:47 +01:00
Robert Collins
d5d775e9b4 Tweak the moved cleanup_dirs to match the original.
Change-Id: I5cb240aa82a5d9ccea539927d355a1f07e728381
2013-05-09 08:32:36 +12:00
Jenkins
c26bd8d655 Merge "Remove img-functions from ramdisk-image-create." 2013-05-08 19:51:32 +00:00
Chris Krelle
76d8d98a60 Remove img-functions from ramdisk-image-create.
Ramdisk-image-create sources both img-functions and ramdisk-functions
causing duplicate functions. To correct this situation this patch
removes the import of img-functions from ramdisk-image-create.
It moves cleanup_dirs and run_d functions from img-functions to
common-functions to allow this.

fixes bug 1175427
Authored-by: Chris Krelle <nobodycam@gmail.com>

Change-Id: I971d019b0d92b06d9de661663b49c433e823ea42
2013-05-08 07:39:30 -07:00
Ghe Rivero
a72d3c1af5 Ignore errors when ldd'ing static binaries.
When trying to get the library dependencies from a static program
it fails and die.
This scapes the error and continue with the execution.

Change-Id: Id3463f2dd77a182ce4f9d9d28165d35f17806892
2013-05-08 14:30:22 +02:00
Clint Byrum
d9a2211d14 Only use tmpfs if build machine has 4GB+ RAM.
For machines with low RAM (or no access to /proc/meminfo) the builder
will still run, but will just build inside the filesystem that is
hosting /tmp. This will result in a slower build (especially if there
are a lot of .deb packages installed).

Fixes bug #1175453

Change-Id: I79f2672058c11e377548820df0ab4fad8f47ffdc
2013-05-03 06:55:53 -07:00
Jenkins
f13570d322 Merge "Fix architecture filter." 2013-05-02 20:37:58 +00:00
Robert Collins
d119c28f49 Revert "Improve Fedora build host support."
This reverts commit 591059ffd3.

Change-Id: I8ad5bde2eabe12c6575d51e7443d9b46521c0928
2013-05-03 08:16:43 +12:00
Chris Jones
488c462dcd Fix architecture filter.
If we are using uname to collect the name of the architecture, we need
to filter its output to make it consistent with dpkg. This is now
happening in the right place to avoid a spurious warning on i386 builds.

Change-Id: Iae80dbfb757fc068155730686841c742acb7a6a2
2013-05-02 20:58:48 +01:00
Jenkins
55993863fc Merge "Improve Fedora build host support." 2013-05-02 19:30:43 +00:00
Lucas Alvares Gomes
2cd44bd77c Fix missing export of $ARCH
So that elements can have access to the variable.

Change-Id: I9a63160ef891e3a3a3c6c90856e2539250c8b807
2013-05-01 15:13:24 +01:00
Lucas Alvares Gomes
591059ffd3 Improve Fedora build host support.
Sudo command (even using -E) resets $PATH for security reasons, Fedora
and other distos might have to explicitly add /usr/local/bin to $PATH
in order to run the commands in the chroot environment.

Change-Id: I95068ecf1e7437152e11c6d6789f59bcfd6ae9cc
2013-05-01 09:40:15 +01:00
Robert Collins
e27ac025fc Use a different approach to solving the ramdisk-image-create failure.
Change-Id: I2861927d1989bde1137c253c8b3a48b245c65f19
2013-05-01 15:03:44 +12:00
Tim Miller
7ef33c51db Source img-functions into common-functions.
A recent commit added a call to function cleanup_dirs to
the common-functions file, but this function is not in scope
for all code paths.

The problem can be reproduced like so:
  bin/ramdisk-image-create deploy -o /tmp/foo

Which results in the error:
  bin/../lib/common-functions: line 35: cleanup_dirs: command not found

This change causes cleanup_dirs to be in scope when calling ramdisk-image-create.

Change-Id: Iaf01beb9dd42a9d810d8aed1aa4ce19a5854c09b
2013-04-30 19:36:37 -07:00
Jenkins
39152c1b30 Merge "Improve Fedora build host support: architectures." 2013-05-01 01:52:10 +00:00
Robert Collins
cb62bae9b8 Build images using loopdev instead of qemu-nbd.
Qemu-nbd does not perform well with older versions of qemu due to
the lack of writeback caching mode. It also only builds qcow2 images
and there is a desire for raw image support. Finally, qemu-nbd makes
it very difficult to build images concurrently due to the somewhat
opaque nature of how it selects a /dev/nbd# device. losetup, on
the other hand, makes this process very straight forward.

Change-Id: I309fad8af4fd1e8d1720c17b65e1897a76d5e897
Co-Author: Clint Byrum <clint@fewbar.com>
2013-04-30 08:56:12 -07:00
Chris Jones
be48317fc3 Improve Fedora build host support: architectures.
Instead of blindly calling dpkg, we will now perform an appropriate test
and fall back on uname with some subsequent massaging of uname's output
on amd64 and ARM build hosts.

Based on a patch from Lucas Alvares Gomes

Change-Id: I363addd9f8b4d748d50b69f4509ce627aa5676aa
2013-04-25 11:04:02 +01:00
Devananda van der Veen
8a22115c6d Allow build dir to be changed from /tmp
Adds an env var TMP_DIR which will override mktemp's default location.

Fixes bug 1168432

Change-Id: I60c63389bb2a639f948e7e96bf9aaea0ea73e27d
2013-04-12 07:43:53 -07:00
Chris Jones
bd1bae7c58 Introduce the DIB_ namespace for build-time config.
Document the use of DIB_ as a prefix on environmental variables that can
be overridden at build time.

Introduce the first such variable by migrating IMAGE_SIZE to
DIB_IMAGE_SIZE.

Change-Id: Ie36b734991b913a23f37f2add47d470d7c1576e4
2013-04-12 09:50:34 +01:00
Robert Collins
24c89932be Always include the in-tree elements directory.
Change-Id: I5a79fbf0568f5755dd14824ed147093387170b3f
2013-04-08 10:26:44 +12:00
Ghe Rivero
4f8040c7a4 Fix elements_path default path.
Change-Id: I0cf94813bd23edac8e25a55bbe58db6f126a2294
2013-02-27 11:11:26 +01:00
Ghe Rivero
308eee2827 Use multiple locations for elements dir.
Using ELEMENTS_DIR env variable, you can specify multiple dirs containint your elements.
It must be a ":" separated elements list.

When an element is seen in one of those lists, it will be chosen and will continue with the next element.

Change-Id: I18eca27d943139cd6ca1ebd232b419e502d7b048
2013-02-27 10:48:55 +01:00
Jenkins
619153bdf4 Merge "Support multiple outputs disk formats" 2013-02-27 00:02:26 +00:00
Devananda van der Veen
c5203262d7 Fix ramdisk-image-create
ELEMENTS_DIR env var was not being set by lib/ramdisk-defaults.
This caused ramdisk-image-create to fail.

Change-Id: I25ec43feddb8ef546dba83a77b8488fee98d4a17
2013-02-25 16:29:28 -08:00
Ghe Rivero
a05629b9f4 Support multiple outputs disk formats
Using the IMAGE_TYPE env variable, you can select which ouput disk format you want for your image.
Only qcow2 format supports compression.

Change-Id: Ia01d9079c354dbd3cfcf870d86f97ee57bd9d479
2013-02-21 09:26:44 +01:00
Tim Miller
5544da4605 first-boot.d scripts log to a file:
Log first-boot.d script output to a file, to ease troubleshooting,
since the first-boot scripts are removed from the system after
running, even in case of failure.

Change-Id: I07d5c782dd918a804e054bf08efce89ed9442e0c
2013-02-15 10:04:50 -08:00
Jenkins
7ae4fb5c1c Merge "Make populate_libs() more generic." 2013-02-15 02:19:05 +00:00
Clint Byrum
91c30f6ab2 Use dib-run-parts for running scripts in target
Also modified dib-run-parts to apply a more workable solution for
filtering out unwanted files such as editor backups and VCS.

The script is installed in its own element, depended on by the OS
specific ubuntu element. This is  because the ubuntu element (and
later other OS's) are responsible for populating the root filesystem.
If we try to install this in base, the root filesystem will look to
be populated already and we will skip automatically choosing ubuntu.

Change-Id: I017646748c1a8360299106289b57d976d45875a8
2013-02-14 10:48:10 -08:00
Arata Notsu
826c9c7935 Make populate_libs() more generic.
It can handle "path to path" output of ldd, such as
    /a/b/c.so => /x/y/z.so (0x123456789)

Change-Id: I68d1096694835818c9129212a02796e1ce1c181d
2013-02-14 20:50:27 +09:00
Robert Collins
7f77729ec5 Move the dpkg specific stuff to a dpkg element rather than being hardcoded.
This includes the install-packages implementation for dpkg, apt http proxy
config, daemon blocking and unblocking.

Change-Id: I8f159021d2b223d7003cec067de3aa605ad06974
2013-02-14 11:16:12 +13:00
Robert Collins
3860af4be5 Change run-parts usage to be compatible with Fedora.
Change-Id: I7a4e1252b0ced940cd4d8fa2fbd096c016933954
2013-02-14 11:16:12 +13:00
Tim Miller
56e50ee5cf Consolidate common OS installation into a script:
Move common openstack service installation operations
into a new script `os-svc-install`, which simplifies
the elements for openstack services.

Change-Id: Ied8ac3278e7fe8af76e24748ec4e598a84afa03c
2013-02-13 11:06:00 -08:00
Robert Collins
8220c00095 Bad code landed, causing ubuntu to be always landed.
Change-Id: Ia77073563fde70140169dbbc5a7448aad54295e2
2013-02-13 13:58:35 +13:00
Arata Notsu
9386a35e47 ramdisk-image-create shows a failed binary dependency
This patch fixes a problem: if a binary is not found, the script
exits without a message informs what is needed (because of 'set -e').

Change-Id: I00b8917918f0e6eddf8506d6548432077eb4bf14
2013-02-12 17:07:09 +09:00
Robert Collins
8f9d433917 The default ARCH was broken due to $ARCH not being exported.
Change-Id: I84f2b60c391d90ed285bd30669260c8483a3c76f
2013-02-12 19:23:34 +13:00
Robert Collins
1ee3a01447 Move initial root contents into a hook.
This is a necessary but not complete step towards supporting Fedora and Suse
distributions. Further work is needed (e.g. to quiesce daemons on
installation).

Change-Id: If3ea6093d41a21de755db52328226b84b5a3ede6
2013-02-11 17:01:36 +13:00
Robert Collins
3d95bf81c5 Export the ELEMENTS_DIR so that dependencies work.
Change-Id: I544b0762f8a13a11564cf02c5fdbe3cb521f63b3
2013-02-05 14:39:38 +13:00
Tim Miller
4688804bbc Improve error message for missing element.
Change-Id: I80beae77874e36d7f47880b030c225100b102e95
2013-01-30 16:07:02 -08:00
Ghe Rivero
11b4a5501d Create install-packages as a binary.
Only supports Debian family distribution so far.

Change-Id: I9fa0b833c34a614fe498c6c6ae599733a69c1309
2013-01-31 00:08:58 +01:00
Chris Jones
0f73578721 Allow manual installation of packages.
Rather than force creation of an element for a single package install,
allow people to do this from the command line.

Change-Id: I63e2e7e50c4a7dbb8a8e198581dfadce91773621
2012-12-21 17:46:44 +00:00
Robert Collins
d03825b504 Add the ability to break into a shell during builds.
Change-Id: I87af952d892f8622e4c916085fc896c735a35438
2012-12-14 20:17:29 +13:00
Robert Collins
012116cad6 Move everything 'common' into base, making it avoidable for tests.
Change-Id: I87edd00f6e0edcd609856192c2c98604ef3274f0
2012-12-14 13:41:57 +13:00
Robert Collins
e8a8e99a08 Make it possible to not recompress the qcow2 image at the end.
Change-Id: Icca922853ee583dcccb6d3970d080fbec11edac7
2012-12-14 13:25:18 +13:00
Ghe Rivero
258185b7a2 Added true to avoid exit on error
disk-image-create just died without any warning

Change-Id: Ia5da4188d64bb22540b57d01184b91faa59149e7
2012-12-13 18:32:34 +01:00
Arata Notsu
1d3ca33ce7 Add copyright to lib/ramdisk-*
Change-Id: I6b48205858f3bc6e55b3dd6efc3e038dd6c42127
2012-12-11 18:14:19 +09:00
Monty Taylor
1eff4a436e Rename flavour to element.
Flavour is overloaded in openstack due to it being used by nova. Element
seems to have the same feeling of combinability without using a term already
in active use in the openstack community.

Change-Id: Ia4c028d4062a8f69c66665821c94dd4bcdf06031
2012-12-05 14:04:58 -08:00
Jenkins
7ccf5282ed Merge "Use system resolv.conf file when available" 2012-12-03 21:38:54 +00:00
Darragh Bailey
e5cda1f731 Use system resolv.conf file when available
Where possible use the system configured resolv.conf file
to to support resolution of internal servers such as proxies.
Use previous hardcoded google dns nameserver as fallback

Change-Id: Ibc1e8a3b96e733a2f7fbcdd4b2c5d3e8926424c4
2012-12-03 15:18:09 +00:00
Arata Notsu
0b81a3e0e8 Remove udev stuff from the source tree
Pick udev stuff from the running evironment instead.

Change-Id: I90a004d0a2b83d3451e73ae351cf474a7bfd62ee
2012-11-29 12:32:06 +09:00
Ghe Rivero
38dc7680c9 Jenkins image fixed
- Removed http_proxy option
- Added sudoers entry to cp first-bot.d
- Fixed typo on 11-jenkins-plugins
- Restored trap exit on run-parts
2012-11-27 18:37:57 +01:00
Ghe Rivero
a3f9f12a4a Untrap EXIT before run-parts 2012-11-26 19:05:12 +01:00
Devananda van der Veen
d935fc0e07 be smarter about mounting qcow images
some images have partition tables, some don't
2012-11-21 19:00:53 -08:00
NoBodyCam
2b879aea6a adding dhclient support 2012-11-20 14:04:40 -08:00
Devananda van der Veen
d0d372ad78 mount_qcow_image should mount p1 2012-11-20 05:21:53 -08:00
NoBodyCam
f7fdec55a8 added lsmod to default busybox links 2012-11-19 16:01:40 -08:00
Devananda van der Veen
e5fcb27790 add some tooling that baremetal-devstack wants
- add bin/disk-image-get-kernel to extract vmlinuz/initrd from .qcow2
- add corresponding methods to lib/common-functions
  - map_nbd
  - (un)mount_qcow_image
2012-11-18 07:20:27 -08:00
chriskrelle
37253564d3 move tgtd & tgtadm to bin-deps of deploy and remove from lib/ramdisk-functions 2012-11-16 16:30:11 -08:00
Robert Collins
4b7cdb4fdc Fix copyrights for HP work. 2012-11-15 16:20:32 +13:00
Chris Jones
f7091cb2fc Add support for flavours to ship udev rules.d files and port over the mellanox variant to use this 2012-11-12 16:57:13 +00:00
Robert Collins
9ecbe14d9c Default to 2G in size - 1G is just too small. 2012-11-12 20:12:46 +13:00
Robert Collins
4c3c7ea9ba Fix handling of parameters that are not set by any flavour for eval_run_d. 2012-11-12 13:45:57 +13:00
Robert Collins
40e954735c Make it possible to set a size from within a flavour, and use that for devstack. 2012-11-12 11:47:26 +13:00
Chris Jones
13d65413e1 Remove some whitespace 2012-11-09 22:16:08 +00:00
Chris Jones
6fb8b19b7c Update ramdisk building to support init hooks for flavours 2012-11-09 22:15:05 +00:00
Chris Jones
eb584f14ed Port old baremetal-mkinitrd.sh to the new image creation standards 2012-11-10 10:50:15 +13:00
Robert Collins
5cc195e391 Bring across disk image code. 2012-11-10 00:04:13 +13:00