Commit Graph

90 Commits

Author SHA1 Message Date
James Slagle
de6efe2b2c Ensure $TMP_BUILD_DIR is actually created.
Previously if mktemp failed in function mk_build_dir, die was not
called.  The check for the exit code was actually checking that the
previous export statement completed successfully, not the call to
mktemp.

Change-Id: I477a2ce75c87c8167883ce7aa342e93c40770e29
2013-07-31 14:26:05 -04:00
Jeff Peeler
87fb533979 Modify relative paths of lib, elements for packaging
(Based on review https://review.openstack.org/#/c/36009)

Scripts test for existence of ../share/diskimage-builder and
fall-back to ../ if not found. This allows scripts to run unmodified
from a packaged installation or a local archive/repository.

Change-Id: I0cf4c1fdb8e42ec284c56860cb15818632b93b9e
2013-07-31 13:15:33 +12:00
Jenkins
1146fe63fc Merge "Add needed symlink for Fedora deployment ramdisk." 2013-07-28 19:58:59 +00:00
James Slagle
92fa96b2f6 Add needed symlink for Fedora deployment ramdisk.
Symlink /usr/lib -> ../lib for Fedora deployment ramdisk.  This
symlink is needed for systemd-udevd.  Otherwise, the network device is
not loaded and initialized when the deployment image is booted.  This
logic was previously contained in
elements/deploy/root.d/50-redhat-prepare-fs, but now that ramdisk
images are built in a chroot, the change needs to be moved so that it
is applied to the ramdisk build.

Change-Id: Icea43230126956ccf5fb8a6a96ca706b75d5c32f
2013-07-26 11:09:29 -04:00
Derek Higgins
e79968f682 Only remove $TMP_BUILD_DIR on cleanup
During ramdisk cleanup if for some reason the filesystems mounted
in $TMP_BUILD_DIR were not removed this recursive delete ended up
deleting parts of the host filesystem that were mounted (in particular
most of the contents of /dev got removed.

This commit doesn't deal with the reason the filesystem didn't unmounted
but ensures the consequences are not as severe.

Fixes Bug 1202612

Change-Id: Id8eba0753c2fe76c79ae1d952ce690d26e33e3ed
2013-07-26 11:03:42 +01:00
Robert Collins
f7f6cb45ee Fix the DIB_OFFLINE setting to actually work.
I missed the getopt parameter and forgot defaults are imported after
option processing. Untested code is broken code!

Change-Id: I133a691909d38e834c204950276a57f4884fc4ed
2013-07-18 13:19:25 +12:00
Robert Collins
b2f1d4e2af Document an interface for offline operation.
Complex image builds can download hundreds of MB of data from the
internet with many separate lookups. It would be nice to allow users
to ask for a fast build where those lookups are entirely avoided,
using locally cached resources (where possible). This new interface
allows users to signal to elements that they wish to operate without
updating cached resources, which will in turn allow us to avoid
checking for stale data at all.

As part of this I've also documented where we cache data, so that
things like the ccache cache dir and image cache files are not a
surprise to users.

Change-Id: I27f5de6ceaa4e9c6390721b7c434fe0908df84f5
2013-07-18 09:17:01 +12:00
Chris Jones
8d1ce9c0c3 Build ramdisks in an image chroot.
Ramdisks are now built inside a chroot which is built by the normal
image build process. Doing so improves our independence of the
precise state of the build host.

This fixes bug 1194055.

Change-Id: Ibc254fbb9e7b404b5f38c1b35bcde8a4136e8e28
2013-07-12 11:09:35 +01:00
Clint Byrum
f6cc6bb185 Use ccache to cache all compiles between builds.
Many python extensions are built from source each time an image is
built. Repeating these is wasteful, so ccache is employed to eliminate
that waste. A cache dir outside the chroot is used to speed up
subsequent image builds.

Change-Id: Ib73563ddbe5f3be7454bfc54ab91cedb559a1304
2013-07-09 18:18:19 -07:00
Richard Su
ad0fe876df Fix 50-firmware.rules no such file on Fedora 19
systemd on Fedora 19 has made /lib/udev/rules.d/50-firmware.rules
optional:
http://lists.freedesktop.org/archives/systemd-devel/2013-March/009708.html

Firmware is now handled by the in-kernel loader.

Added a file exist check for Fedora 19 and any other distros that
may have dropped the rules file. Backwards compatible with existing
distros.

Fixes: bug #1196409

Change-Id: I1d2acf3f88543736d75eb2e8766e83a3da194638
2013-07-03 14:10:33 -07:00
Robert Collins
a969be49a2 Move functions to common-functions for reuse.
Change-Id: I12a5687d94df678d7f7e8c0b075ee4aba1082156
2013-06-25 17:36:37 +01:00
Robert Collins
0b1ff7d4b0 Factor out element processing.
This reduces code duplication for hygiene, no functional changes.

Change-Id: I10c96da4f07bcd1e5aabfcfbd115ecaef7bb65a1
2013-06-24 16:30:25 +01:00
Robert Collins
9b01f2b129 Consolidate more ramdisk and disk-image code.
RAMDISK_ELEMENT is a spurious difference, to merge the code we need
only semantic difference to exist.

Change-Id: I7d38b2457b6fc838d1a536406bb3c34c00f2b1f9
2013-06-24 20:53:19 +12:00
Jenkins
e5e5845335 Merge "Reduce duplication between ramdisk and disk image codepaths." 2013-06-19 22:20:14 +00:00
Robert Collins
2c47b700be Reduce duplication between ramdisk and disk image codepaths.
Change-Id: Ie9e2d60ffe94f971953d52b7b5bce9c1a86fb1b5
2013-06-19 21:26:29 +12:00
Lucas Alvares Gomes
1435f6e48d Make the finalise_base function less Ubuntu-ish.
The finalise_base function was creating a broken link at /etc/resolv.conf
when building a non ubuntu image.

Change-Id: I974d43fd6db365dbff9c85195e3b72950f3409da
2013-06-18 12:51:39 +01:00
Dan Prince
1fa27e745d Stop apt-get installing qemu-img.
This is simply a prerequisite and we should not try to
apt-get install it inline.

Remove this code and exit with an error if qemu-img is
not found.

Change-Id: I52d47a95d24d4bfe801ba664200604bd385dc67f
2013-06-11 10:03:10 -04:00
Lucas Alvares Gomes
cd27dff29a Add fedora support for ramdisk-image-create.
This patch allows the creation of ramdisks from a fedora system.

Change-Id: Ic3b393af45ae1b85d015d2ceb5601ea723457da6
2013-05-31 15:37:31 +01:00
Lucas Alvares Gomes
02ee161e81 Refactor the first boot routine.
Setting up a routine to run at first time the system boot may vary from
distribution to distribution. This patch will remove the logic from the
diskimage-builder code and put it into the elements.

The base element will now deploy a dib-first-boot script and each OS
element will ensure that this script runs at rc.local time. By doing
that we will put the OS specific stuff in the OS elements and remove
the embedded code out of the img-function file.

Change-Id: I24c5d1b1185de5693f145347fe912245f1ba7dfe
2013-05-16 15:50:10 +01:00
Jenkins
7e0fe78cf2 Merge "Fix for running /etc/rc.local on Fedora" 2013-05-15 22:35:19 +00:00
Steve Baker
c6da246b73 Fix for running /etc/rc.local on Fedora
With systemd rc.local lives in /etc/rc.d and is only run
if the rc-local service is enabled.

This change creates a /etc/rc.d/rc.local symlink, and modifies img-functions
prepare_first_boot to tolerate an initially missing /etc/rc.local.

Fixes bug #1179674

Change-Id: I7e6102b62224950c1de5d48205dc4feff60f9389
2013-05-14 11:48:58 +12:00
Jenkins
ffdf0ba7ff Merge "Honor $DIB_IMAGE_SIZE." 2013-05-13 17:56:17 +00:00
Tim Miller
9983245249 Honor $DIB_IMAGE_SIZE.
The recent change to use loopdev instead of nbd stopped
honoring the DIB_IMAGE_SIZE setting.

This change adds it back, by resizing the image to
DIB_IMAGE_SIZE GB, if specified. If unspecified, it resizes
the image to (actual usage + 20%), as it did before this change.

Change-Id: I69afd9584e644ddacc948619100f153d3d8713a4
2013-05-13 10:20:53 -07:00
Clint Byrum
20661e8d80 Retry losetup -d for up to 10 seconds.
When partition tables are added to loopback devices, this can set off a
chain of udev hooks that may still be holding the loopback open. Failing
to detach loopback devices was the reason we were seeing leaked tmpfs
volumes.

Fixes bug #1178091

Change-Id: I836d6e2bbce824951dd4786e3ef28273ea18ee73
2013-05-09 09:34:53 -07:00
Lucas Alvares Gomes
e537e046d1 Force the inclusion of /usr/local/bin in PATH.
Sudo compiled with the --with-secure-path option enabled (default on
Ubuntu, Fedora and potentially other distros) will cause sudo to ignore
any changes made to the PATH variable in the user's environment (even when
the -E option is used). Also, some distros might not have /usr/local/bin
set in their sudoers secure_path causing the diskimage-builder to fail
when building an image using such distros as host. This patch will force
the inclusion of the /usr/local/bin in the PATH enviroment variable
inside the chroot.

Change-Id: Idb7156d8ff124bfe685cc721337d9bb21f49dc9e
Fixes: bug #1175980
2013-05-08 23:32:47 +01:00
Robert Collins
d5d775e9b4 Tweak the moved cleanup_dirs to match the original.
Change-Id: I5cb240aa82a5d9ccea539927d355a1f07e728381
2013-05-09 08:32:36 +12:00
Jenkins
c26bd8d655 Merge "Remove img-functions from ramdisk-image-create." 2013-05-08 19:51:32 +00:00
Chris Krelle
76d8d98a60 Remove img-functions from ramdisk-image-create.
Ramdisk-image-create sources both img-functions and ramdisk-functions
causing duplicate functions. To correct this situation this patch
removes the import of img-functions from ramdisk-image-create.
It moves cleanup_dirs and run_d functions from img-functions to
common-functions to allow this.

fixes bug 1175427
Authored-by: Chris Krelle <nobodycam@gmail.com>

Change-Id: I971d019b0d92b06d9de661663b49c433e823ea42
2013-05-08 07:39:30 -07:00
Ghe Rivero
a72d3c1af5 Ignore errors when ldd'ing static binaries.
When trying to get the library dependencies from a static program
it fails and die.
This scapes the error and continue with the execution.

Change-Id: Id3463f2dd77a182ce4f9d9d28165d35f17806892
2013-05-08 14:30:22 +02:00
Clint Byrum
d9a2211d14 Only use tmpfs if build machine has 4GB+ RAM.
For machines with low RAM (or no access to /proc/meminfo) the builder
will still run, but will just build inside the filesystem that is
hosting /tmp. This will result in a slower build (especially if there
are a lot of .deb packages installed).

Fixes bug #1175453

Change-Id: I79f2672058c11e377548820df0ab4fad8f47ffdc
2013-05-03 06:55:53 -07:00
Jenkins
f13570d322 Merge "Fix architecture filter." 2013-05-02 20:37:58 +00:00
Robert Collins
d119c28f49 Revert "Improve Fedora build host support."
This reverts commit 591059ffd3.

Change-Id: I8ad5bde2eabe12c6575d51e7443d9b46521c0928
2013-05-03 08:16:43 +12:00
Chris Jones
488c462dcd Fix architecture filter.
If we are using uname to collect the name of the architecture, we need
to filter its output to make it consistent with dpkg. This is now
happening in the right place to avoid a spurious warning on i386 builds.

Change-Id: Iae80dbfb757fc068155730686841c742acb7a6a2
2013-05-02 20:58:48 +01:00
Jenkins
55993863fc Merge "Improve Fedora build host support." 2013-05-02 19:30:43 +00:00
Lucas Alvares Gomes
2cd44bd77c Fix missing export of $ARCH
So that elements can have access to the variable.

Change-Id: I9a63160ef891e3a3a3c6c90856e2539250c8b807
2013-05-01 15:13:24 +01:00
Lucas Alvares Gomes
591059ffd3 Improve Fedora build host support.
Sudo command (even using -E) resets $PATH for security reasons, Fedora
and other distos might have to explicitly add /usr/local/bin to $PATH
in order to run the commands in the chroot environment.

Change-Id: I95068ecf1e7437152e11c6d6789f59bcfd6ae9cc
2013-05-01 09:40:15 +01:00
Robert Collins
e27ac025fc Use a different approach to solving the ramdisk-image-create failure.
Change-Id: I2861927d1989bde1137c253c8b3a48b245c65f19
2013-05-01 15:03:44 +12:00
Tim Miller
7ef33c51db Source img-functions into common-functions.
A recent commit added a call to function cleanup_dirs to
the common-functions file, but this function is not in scope
for all code paths.

The problem can be reproduced like so:
  bin/ramdisk-image-create deploy -o /tmp/foo

Which results in the error:
  bin/../lib/common-functions: line 35: cleanup_dirs: command not found

This change causes cleanup_dirs to be in scope when calling ramdisk-image-create.

Change-Id: Iaf01beb9dd42a9d810d8aed1aa4ce19a5854c09b
2013-04-30 19:36:37 -07:00
Jenkins
39152c1b30 Merge "Improve Fedora build host support: architectures." 2013-05-01 01:52:10 +00:00
Robert Collins
cb62bae9b8 Build images using loopdev instead of qemu-nbd.
Qemu-nbd does not perform well with older versions of qemu due to
the lack of writeback caching mode. It also only builds qcow2 images
and there is a desire for raw image support. Finally, qemu-nbd makes
it very difficult to build images concurrently due to the somewhat
opaque nature of how it selects a /dev/nbd# device. losetup, on
the other hand, makes this process very straight forward.

Change-Id: I309fad8af4fd1e8d1720c17b65e1897a76d5e897
Co-Author: Clint Byrum <clint@fewbar.com>
2013-04-30 08:56:12 -07:00
Chris Jones
be48317fc3 Improve Fedora build host support: architectures.
Instead of blindly calling dpkg, we will now perform an appropriate test
and fall back on uname with some subsequent massaging of uname's output
on amd64 and ARM build hosts.

Based on a patch from Lucas Alvares Gomes

Change-Id: I363addd9f8b4d748d50b69f4509ce627aa5676aa
2013-04-25 11:04:02 +01:00
Devananda van der Veen
8a22115c6d Allow build dir to be changed from /tmp
Adds an env var TMP_DIR which will override mktemp's default location.

Fixes bug 1168432

Change-Id: I60c63389bb2a639f948e7e96bf9aaea0ea73e27d
2013-04-12 07:43:53 -07:00
Chris Jones
bd1bae7c58 Introduce the DIB_ namespace for build-time config.
Document the use of DIB_ as a prefix on environmental variables that can
be overridden at build time.

Introduce the first such variable by migrating IMAGE_SIZE to
DIB_IMAGE_SIZE.

Change-Id: Ie36b734991b913a23f37f2add47d470d7c1576e4
2013-04-12 09:50:34 +01:00
Robert Collins
24c89932be Always include the in-tree elements directory.
Change-Id: I5a79fbf0568f5755dd14824ed147093387170b3f
2013-04-08 10:26:44 +12:00
Ghe Rivero
4f8040c7a4 Fix elements_path default path.
Change-Id: I0cf94813bd23edac8e25a55bbe58db6f126a2294
2013-02-27 11:11:26 +01:00
Ghe Rivero
308eee2827 Use multiple locations for elements dir.
Using ELEMENTS_DIR env variable, you can specify multiple dirs containint your elements.
It must be a ":" separated elements list.

When an element is seen in one of those lists, it will be chosen and will continue with the next element.

Change-Id: I18eca27d943139cd6ca1ebd232b419e502d7b048
2013-02-27 10:48:55 +01:00
Jenkins
619153bdf4 Merge "Support multiple outputs disk formats" 2013-02-27 00:02:26 +00:00
Devananda van der Veen
c5203262d7 Fix ramdisk-image-create
ELEMENTS_DIR env var was not being set by lib/ramdisk-defaults.
This caused ramdisk-image-create to fail.

Change-Id: I25ec43feddb8ef546dba83a77b8488fee98d4a17
2013-02-25 16:29:28 -08:00
Ghe Rivero
a05629b9f4 Support multiple outputs disk formats
Using the IMAGE_TYPE env variable, you can select which ouput disk format you want for your image.
Only qcow2 format supports compression.

Change-Id: Ia01d9079c354dbd3cfcf870d86f97ee57bd9d479
2013-02-21 09:26:44 +01:00
Tim Miller
5544da4605 first-boot.d scripts log to a file:
Log first-boot.d script output to a file, to ease troubleshooting,
since the first-boot scripts are removed from the system after
running, even in case of failure.

Change-Id: I07d5c782dd918a804e054bf08efce89ed9442e0c
2013-02-15 10:04:50 -08:00