Commit Graph

152 Commits

Author SHA1 Message Date
Robert Collins
b2f1d4e2af Document an interface for offline operation.
Complex image builds can download hundreds of MB of data from the
internet with many separate lookups. It would be nice to allow users
to ask for a fast build where those lookups are entirely avoided,
using locally cached resources (where possible). This new interface
allows users to signal to elements that they wish to operate without
updating cached resources, which will in turn allow us to avoid
checking for stale data at all.

As part of this I've also documented where we cache data, so that
things like the ccache cache dir and image cache files are not a
surprise to users.

Change-Id: I27f5de6ceaa4e9c6390721b7c434fe0908df84f5
2013-07-18 09:17:01 +12:00
Chris Jones
8d1ce9c0c3 Build ramdisks in an image chroot.
Ramdisks are now built inside a chroot which is built by the normal
image build process. Doing so improves our independence of the
precise state of the build host.

This fixes bug 1194055.

Change-Id: Ibc254fbb9e7b404b5f38c1b35bcde8a4136e8e28
2013-07-12 11:09:35 +01:00
Robert Collins
a969be49a2 Move functions to common-functions for reuse.
Change-Id: I12a5687d94df678d7f7e8c0b075ee4aba1082156
2013-06-25 17:36:37 +01:00
Robert Collins
0b1ff7d4b0 Factor out element processing.
This reduces code duplication for hygiene, no functional changes.

Change-Id: I10c96da4f07bcd1e5aabfcfbd115ecaef7bb65a1
2013-06-24 16:30:25 +01:00
Robert Collins
9b01f2b129 Consolidate more ramdisk and disk-image code.
RAMDISK_ELEMENT is a spurious difference, to merge the code we need
only semantic difference to exist.

Change-Id: I7d38b2457b6fc838d1a536406bb3c34c00f2b1f9
2013-06-24 20:53:19 +12:00
Robert Collins
2c47b700be Reduce duplication between ramdisk and disk image codepaths.
Change-Id: Ie9e2d60ffe94f971953d52b7b5bce9c1a86fb1b5
2013-06-19 21:26:29 +12:00
Lucas Alvares Gomes
c28cb1b5d8 Add RedHat support for disk-image-get-kernel.
RedHat and Debian based systems have different binary names.

Change-Id: I5162204759fbc4ff167d1584294c9936ab7df4c8
2013-06-05 17:12:09 +01:00
Robert Collins
1f0bc8f19b Increase the size heuristic for images.
Bug 1183597 reported that we create them too narrowly today.

Change-Id: Ia863cf2fb19c813c51f94dd0b8e210a75dccd945
2013-05-24 10:32:10 +12:00
Robert Collins
c139b0cb46 The ext4 resize fix was faulty - fix it.
Fix bug 1183251 by replacing an erroneous reference to OPTS with
MKFS_OPTS.

Change-Id: I741005d574715b8b237d9789398dbaba29df60e3
2013-05-24 10:06:33 +12:00
Lucas Alvares Gomes
85b5ef0c8d Move the ensure_nbd function call.
Move the ensure_nbd function call from the disk-image-create to the
disk-image-get-kernel. The image creation process does not use qemu-nbd
anymore.

Change-Id: I8ff2ad54f322a9e33d52408fa4e253a7b45900c7
2013-05-20 11:59:21 +01:00
Robert Collins
fb246a02eb Ensure that the ext4 fs can be rebuilt up to 1PB in size.
Change-Id: I16c5fe4eb0a9951d44745d6a46450f70d33b3481
2013-05-18 13:08:06 +12:00
Jenkins
ffdf0ba7ff Merge "Honor $DIB_IMAGE_SIZE." 2013-05-13 17:56:17 +00:00
Tim Miller
9983245249 Honor $DIB_IMAGE_SIZE.
The recent change to use loopdev instead of nbd stopped
honoring the DIB_IMAGE_SIZE setting.

This change adds it back, by resizing the image to
DIB_IMAGE_SIZE GB, if specified. If unspecified, it resizes
the image to (actual usage + 20%), as it did before this change.

Change-Id: I69afd9584e644ddacc948619100f153d3d8713a4
2013-05-13 10:20:53 -07:00
Clint Byrum
20661e8d80 Retry losetup -d for up to 10 seconds.
When partition tables are added to loopback devices, this can set off a
chain of udev hooks that may still be holding the loopback open. Failing
to detach loopback devices was the reason we were seeing leaked tmpfs
volumes.

Fixes bug #1178091

Change-Id: I836d6e2bbce824951dd4786e3ef28273ea18ee73
2013-05-09 09:34:53 -07:00
Chris Krelle
76d8d98a60 Remove img-functions from ramdisk-image-create.
Ramdisk-image-create sources both img-functions and ramdisk-functions
causing duplicate functions. To correct this situation this patch
removes the import of img-functions from ramdisk-image-create.
It moves cleanup_dirs and run_d functions from img-functions to
common-functions to allow this.

fixes bug 1175427
Authored-by: Chris Krelle <nobodycam@gmail.com>

Change-Id: I971d019b0d92b06d9de661663b49c433e823ea42
2013-05-08 07:39:30 -07:00
Clint Byrum
d9a2211d14 Only use tmpfs if build machine has 4GB+ RAM.
For machines with low RAM (or no access to /proc/meminfo) the builder
will still run, but will just build inside the filesystem that is
hosting /tmp. This will result in a slower build (especially if there
are a lot of .deb packages installed).

Fixes bug #1175453

Change-Id: I79f2672058c11e377548820df0ab4fad8f47ffdc
2013-05-03 06:55:53 -07:00
Robert Collins
d119c28f49 Revert "Improve Fedora build host support."
This reverts commit 591059ffd3.

Change-Id: I8ad5bde2eabe12c6575d51e7443d9b46521c0928
2013-05-03 08:16:43 +12:00
Jenkins
55993863fc Merge "Improve Fedora build host support." 2013-05-02 19:30:43 +00:00
Lucas Alvares Gomes
591059ffd3 Improve Fedora build host support.
Sudo command (even using -E) resets $PATH for security reasons, Fedora
and other distos might have to explicitly add /usr/local/bin to $PATH
in order to run the commands in the chroot environment.

Change-Id: I95068ecf1e7437152e11c6d6789f59bcfd6ae9cc
2013-05-01 09:40:15 +01:00
Robert Collins
e27ac025fc Use a different approach to solving the ramdisk-image-create failure.
Change-Id: I2861927d1989bde1137c253c8b3a48b245c65f19
2013-05-01 15:03:44 +12:00
Robert Collins
cb62bae9b8 Build images using loopdev instead of qemu-nbd.
Qemu-nbd does not perform well with older versions of qemu due to
the lack of writeback caching mode. It also only builds qcow2 images
and there is a desire for raw image support. Finally, qemu-nbd makes
it very difficult to build images concurrently due to the somewhat
opaque nature of how it selects a /dev/nbd# device. losetup, on
the other hand, makes this process very straight forward.

Change-Id: I309fad8af4fd1e8d1720c17b65e1897a76d5e897
Co-Author: Clint Byrum <clint@fewbar.com>
2013-04-30 08:56:12 -07:00
Lucas Alvares Gomes
45173ccdb5 Adds support for post-install scripts.
Post install scripts are useful because they can perform tasks you want
to handle after the OS/application install but before the first boot
of the image. e.g: Clean the cache left by the package manager reducing
the size of the image.

Change-Id: I03e77f602192bbdce29c02999d1b57fac8051ddc
Fixes: Bug #1145786
2013-04-17 09:47:51 +01:00
Lucas Alvares Gomes
b20e133714 Grab the next available /dev/nbdX.
Change-Id: I6a2c8e58cdab889d747e870d0009f51bba0ff148
2013-04-12 17:49:23 +01:00
Chris Jones
bd1bae7c58 Introduce the DIB_ namespace for build-time config.
Document the use of DIB_ as a prefix on environmental variables that can
be overridden at build time.

Introduce the first such variable by migrating IMAGE_SIZE to
DIB_IMAGE_SIZE.

Change-Id: Ie36b734991b913a23f37f2add47d470d7c1576e4
2013-04-12 09:50:34 +01:00
Robert Collins
5ada82a59c Add armhf support.
ARM doesn't have a generic Linux image due to the soc-specific nature of Linux
kernels today, so we drop the manual installation of that package, replacing it
with a dist-upgrade instead. This involved tweaks to the dpkg and fedora
install-package scripts.

Change-Id: I97924b80ca87781307e1087b9fe4b18215770e84
2013-04-09 08:17:22 +12:00
Chris Jones
0cfecd1024 Store build-time settings
This will write two files in /etc/ that contain the environment and
command line arguments used in the creation of an image. This should
assist with later efforts to repeat the creation of an image.

Change-Id: Icdbe6693380bed6c406feee10d2cb1a88a992932
2013-04-04 20:39:11 +01:00
Chris Jones
e34e75ff22 Remove duplicate file extensions.
Gracefully remove duplicate file extensions on the output filename

Change-Id: Id4f8b0d75b587c238c89d6ca7a06fab6a57cd627
2013-03-15 16:32:57 +00:00
Ghe Rivero
308eee2827 Use multiple locations for elements dir.
Using ELEMENTS_DIR env variable, you can specify multiple dirs containint your elements.
It must be a ":" separated elements list.

When an element is seen in one of those lists, it will be chosen and will continue with the next element.

Change-Id: I18eca27d943139cd6ca1ebd232b419e502d7b048
2013-02-27 10:48:55 +01:00
Robert Collins
9afddcf266 New element that uses a fedora cloud image as the base.
Change-Id: I7d83bb2b359e7a8c3858eca04c96e35cf4e1fe9e
2013-02-15 09:33:29 +13:00
Robert Collins
7f77729ec5 Move the dpkg specific stuff to a dpkg element rather than being hardcoded.
This includes the install-packages implementation for dpkg, apt http proxy
config, daemon blocking and unblocking.

Change-Id: I8f159021d2b223d7003cec067de3aa605ad06974
2013-02-14 11:16:12 +13:00
Tim Miller
56e50ee5cf Consolidate common OS installation into a script:
Move common openstack service installation operations
into a new script `os-svc-install`, which simplifies
the elements for openstack services.

Change-Id: Ied8ac3278e7fe8af76e24748ec4e598a84afa03c
2013-02-13 11:06:00 -08:00
Jenkins
750e4c3e06 Merge "Correct misuse of return in ramdisk-image-create" 2013-02-12 23:05:07 +00:00
Jenkins
4008475bfc Merge "Fix unnecessarily creating a temporary directory" 2013-02-12 23:04:56 +00:00
Chris Jones
8f96a286d7 Add default element selection
This change includes the 'base' element by default and adds a
command line switch to prevent this behaviour. This should reduce
confusion since this element is almost always necessary.

Change-Id: Ib4a2ef53a76b119b764751fa21058f4e2cb37741
2013-02-12 09:29:19 +00:00
Arata Notsu
aa6f5f8005 Correct misuse of return in ramdisk-image-create
Replaced return with exit.

Change-Id: Iee26177cdb58af67b6919da12456debfe5940105
2013-02-12 17:43:35 +09:00
Arata Notsu
5c13dc1d75 Fix unnecessarily creating a temporary directory
mk_build_dir was executed twice in ramdisk-image-create.

Change-Id: I93cf83775fb5e7ba3e6b84643f723568a2420305
2013-02-12 17:29:52 +09:00
Robert Collins
1ee3a01447 Move initial root contents into a hook.
This is a necessary but not complete step towards supporting Fedora and Suse
distributions. Further work is needed (e.g. to quiesce daemons on
installation).

Change-Id: If3ea6093d41a21de755db52328226b84b5a3ede6
2013-02-11 17:01:36 +13:00
Chris Jones
997165ff11 Add option to clear environment
This adds an option to disk-image-create that allows it to re-exec with
an empty environment if required.

Change-Id: I9e51aa07b903e18cdd0ed5f953800307f9899b5d
2013-01-30 18:24:24 +00:00
Chris Jones
15dbc42d72 Switch locale to C
Remove the explicit installation of an English language pack and instead
just default LANG to C. This settls Perl's noisy warnings and also stops
any side effects of different language build hosts.

Note that this is set right at the start of the build process, so it
should be entirely possible to override if needs be.

Change-Id: Id3b31162d4198fa02dc5a4d11168e57dbcd14a5d
2013-01-28 18:08:50 +00:00
Clint Byrum
c8c33e3bb1 Add a simple implementation of element dependency
This adds a new optional file to the root of elements. The
file lists dependencies which will be added to the list
requested by the user during disk image creation.

Change-Id: Id71c3b333563604bbbaf90f9cf40e24fa9738fc8
2013-01-23 16:04:20 -08:00
Chris Jones
0f73578721 Allow manual installation of packages.
Rather than force creation of an element for a single package install,
allow people to do this from the command line.

Change-Id: I63e2e7e50c4a7dbb8a8e198581dfadce91773621
2012-12-21 17:46:44 +00:00
Chris Jones
219aa52958 Fix disk-image-create's getopt error handling:
We originally checked the return code of getopt, but subsequently
a debugging echo has been inserted and now we are checking the
return code of "echo", which is very rarely going to be useful.

Change-Id: I054fecaf6576d5e9d42b07159bba7d50335bf6cc
2012-12-17 17:00:13 +00:00
Robert Collins
e8a8e99a08 Make it possible to not recompress the qcow2 image at the end.
Change-Id: Icca922853ee583dcccb6d3970d080fbec11edac7
2012-12-14 13:25:18 +13:00
Monty Taylor
1eff4a436e Rename flavour to element.
Flavour is overloaded in openstack due to it being used by nova. Element
seems to have the same feeling of combinability without using a term already
in active use in the openstack community.

Change-Id: Ia4c028d4062a8f69c66665821c94dd4bcdf06031
2012-12-05 14:04:58 -08:00
Arata Notsu
0b81a3e0e8 Remove udev stuff from the source tree
Pick udev stuff from the running evironment instead.

Change-Id: I90a004d0a2b83d3451e73ae351cf474a7bfd62ee
2012-11-29 12:32:06 +09:00
Devananda van der Veen
fb426153f4 remove force-xtrace in disk-image-get-kernel 2012-11-18 08:17:54 -08:00
Devananda van der Veen
e5fcb27790 add some tooling that baremetal-devstack wants
- add bin/disk-image-get-kernel to extract vmlinuz/initrd from .qcow2
- add corresponding methods to lib/common-functions
  - map_nbd
  - (un)mount_qcow_image
2012-11-18 07:20:27 -08:00
Robert Collins
4b7cdb4fdc Fix copyrights for HP work. 2012-11-15 16:20:32 +13:00
Chris Jones
f7091cb2fc Add support for flavours to ship udev rules.d files and port over the mellanox variant to use this 2012-11-12 16:57:13 +00:00
Robert Collins
40e954735c Make it possible to set a size from within a flavour, and use that for devstack. 2012-11-12 11:47:26 +13:00
Chris Jones
eb584f14ed Port old baremetal-mkinitrd.sh to the new image creation standards 2012-11-10 10:50:15 +13:00
Robert Collins
5cc195e391 Bring across disk image code. 2012-11-10 00:04:13 +13:00