Commit Graph

231 Commits

Author SHA1 Message Date
Jenkins
9fae41c898 Merge "Fail helpfully if no elements are specified" 2014-11-25 12:29:06 +00:00
Gregory Haynes
9bb7ace03c Fail helpfully if no elements are specified
Instead of charging ahead and failing because we lack dib-run-parts we
can fail in this case with a useful error message.

Change-Id: Ie10176b5a7019a0385796984ebbaf7742876925a
2014-11-21 10:31:11 -08:00
Gregory Haynes
fd0708286c Check python with flake8 instead of dib-lint
We check python files with dib-lint rather than flake8 which have
conflicting opinions. This means weve been (forcibly) writing non pep8
python.

Also fixing pep8 issues so tests pass.

Change-Id: Idc9db40334f6e15738a7802c06697270df68741c
2014-11-21 09:04:52 -08:00
Yuriy Zveryanskyy
676b1b738d Add element for building ramdisk with ironic-python-agent
This element creates kernel and ramdisk files based on Fedora,
example:
disk-image-create -a i386 -o test fedora ironic-agent disable-selinux

Change-Id: Ifa133d1680b81cb87d32a405aa7d7b40fe91f835
2014-11-13 19:30:44 +02:00
Jenkins
ce67ce87ea Merge "Support installing packages by default" 2014-11-13 10:27:56 +00:00
Ryan Brady
6fedbc58f9 Fix indent exclusion
The current dib-lint script has an exclusion for indent but it is
erroneously used as a label around the tab indention check.  This
patch renames the tab indention exclusion and adds the indent
exclusion where it belongs.

Change-Id: I701595e7a08f4764a45c6701d67cbb931c3d47f3
2014-11-11 21:06:25 -05:00
Dan Prince
022cb2c1c2 Support installing packages by default
Adds new disk-image-create --install-type option which
can be used to enable alternate install types. The
default install type is 'source'.

The motivation is to eliminate elements like
enable-package-installs which require coupling and also
don't work with elements in multiple element repositories.

This patch does not prevent you from using the previous
DIB_INSTALLTYPE_ variables to customize the install type,
rather it just changes the default so you don't have to
set it so often when using things like packages.

Change-Id: Icee98440fc2251728f2dca30e7c4789a0fd89b93
2014-10-28 15:05:53 -04:00
Jenkins
47502b8042 Merge "export DIB_ROOT_LABEL to make it global" 2014-10-28 15:51:19 +00:00
Ghe Rivero
48ce0fde51 export DIB_ROOT_LABEL to make it global
It is needed by the debian element in the root.d phase. (enviroment.d variables are not available after the base image is built)

Change-Id: Idb8f9795a619687569bcb0de774c87333d11b292
Closes-Bug: #1384103
2014-10-23 07:38:21 +00:00
Ramakrishnan G
13aed64e97 iso element to build bootable ISO images
This commit adds a new element named 'iso' to build a bootable
ISO image for the kernel/ramdisk emitted by the 'baremetal' or
'ramdisk' element.

Change-Id: I89d175a29e2d0bc64b47fe527f0d0f6875f6849a
2014-10-21 14:37:57 +05:30
Jenkins
bb07de7fe9 Merge "Enable dracut deploy ramdisks" 2014-10-20 11:03:54 +00:00
Ben Nemec
eed30adc48 Enable dracut deploy ramdisks
The element builds dracut from source on Ubuntu because the
Ubuntu dracut package is broken and very old, so it can't be
installed properly and causes a number of other issues that
are fixed by using a newer version of Dracut.

This initial version should work in virtualized environments.
Further validation of its suitability for real baremetal
deployments will need to be done in the future, but this should
be sufficient to enable that work.

Regarding Dracut specifically, in order to limit the changes
needed in the existing scripts this element continues to use a
cut down version of the /init script that we were building for the
existing ramdisk.  However, instead of running it as pid 0 it is
run as a Dracut pre-mount hook.  This allows Dracut to set up all
of the hardware and system bits, while falling early enough in the
Dracut sequence to complete the deployment before Dracut would try
to boot off the hard disk.

bp tripleo-juno-dracut-ramdisks
Change-Id: I144c8993fe040169f440bd4f7a428fdbe3d745cf
2014-10-17 21:44:09 -05:00
Jenkins
2b60bea961 Merge "Allow for multiple image outputs from raw source" 2014-10-16 21:56:28 +00:00
Ghe Rivero
873de01491 Remove first-boot.d support
After being deprecated two releases ago, finally remove any reference
for the support of first-boot.d

Change-Id: I08d67404ef48cad61db3b18fb86e970abfa5d2b6
2014-10-16 09:48:14 +00:00
Clark Boylan
ae928057bd Allow for multiple image outputs from raw source
When uploading images to multiple clouds it is possible that the same
image will be needed in multiple formats to accomodate hypervisors
across clouds. Update disk-image-create's -t flag to take a list of
desired output image formats so that a single disk-image-create can
output all of the desired image formats.

Change-Id: If121b2342ae888855ba435aa3189f039e985b812
2014-10-13 09:16:42 -07:00
Ben Nemec
dde3d24213 Allow custom rootfs labels
Per [1], our current root label of cloudimg-root does not work with
XFS because XFS only allows 12 character labels.  This change
addresses that by allowing a custom rootfs label to be specified
in the call to disk-image-create.  There should be no backwards
compatibility concerns as the default label is unchanged.  Any
external elements dealing with the label would need to be updated
to support this new feature, but should continue to work as before
as long as a custom label is not specified.

[1]: https://bugzilla.redhat.com/show_bug.cgi?id=1139584

Change-Id: I596104d1a63b5dc6549e8460a1ae3da00165ef04
2014-09-26 17:44:08 -05:00
Jenkins
6518876a5f Merge "Use dib-run-parts from dib-utils" 2014-09-17 14:31:49 +00:00
Jenkins
7d0e7fe5ad Merge "Centralize handling of /lost+found" 2014-08-28 16:33:12 +00:00
Jenkins
d5caf4418d Merge "Add dib-lint exclusions" 2014-08-28 07:53:56 +00:00
Ben Nemec
644598941c Use dib-run-parts from dib-utils
Now that dib-run-parts has been moved to the dib-utils project, we
need to update diskimage-builder to use it instead of the version
directly in diskimage-builder.

This change removes the old copy of the dib-run-parts script in
the element, adds dib-utils as a dependency of diskimage-builder,
and updates the uses of dib-run-parts to correctly handle the fact
that it is now external to the project.

Requires I0be1f876d0e4a7d38e0d5c6010a552a8ebb158a4

Change-Id: Ia0a0df7784a14c49b5c47ac0b03e6c2602c84b3b
2014-08-26 16:44:18 -05:00
Pino Toscano
267e96f52c Centralize handling of /lost+found
Just remove globally /lost+found after the root and install phases, so
distribution elements don't need to do that by themselves.

Change-Id: Ic783e613bd8794aefd3f40c9a7c308d14cd04b8d
2014-08-04 11:48:21 +02:00
iberezovskiy
a8d961e6ec Create cache directory
Creating of cache directory was missed in disk-image-create script

Change-Id: I4e5901bb455712d6403362a6aac48eaa93f62e29
2014-07-29 18:45:22 +04:00
Ryan Brady
c1985ea772 Add dib-lint exclusions
This patch adds exclusions for the indentation and newline
rules in dib-lint.  The indentation exclusion is needed for
files using heredoc and wanting to preserve indentation.
The newline exclusion is added here for consisently treating
all dib-lint rules the same.

Change-Id: Ic8a2164eba3cf99e55473846e3a5fe663a3c1095
2014-07-24 13:50:48 -04:00
JUNJIE NAN
d65b37fb03 Use readlink to get script path
So we can support usage:
ln -s ~/source/.../disk-image-create ~/bin/

Change-Id: Id54b613a247679b98cef3e9a6f37cd93155d029e
2014-07-10 15:41:41 +00:00
Jenkins
4e97def3d5 Merge "Correction: if then statement code style." 2014-07-10 07:38:49 +00:00
AzherKhan
d06a1caafb Correction: if then statement code style.
Corrected the if; then statement to follow bash convention.

Change-Id: Ia3f80520bb9446c4feb5ce36dccffa2b1c1d72c4
2014-07-07 03:43:15 -07:00
Ben Nemec
541a4b2270 Add global exclusions to dib-lint
This will be useful for adopting dib-lint in other projects, in
particular tripleo-image-elements.  It allows some dib-lint rules
to be used even if the project can't pass all of the checks.  The
failing checks can then be fixed one at a time and removed from the
exclusions list.

For consistency, this change reads the new exclusions from tox.ini
in the same way as flake8.  To use the exclusion mechanism, the
the following section can be added to tox.ini:

  [dib-lint]
  ignore = sete setu setpipefail

Change-Id: I6d8024e7613890e935ceb5e50d0d507bd554d8dd
2014-07-03 11:50:04 -05:00
Jenkins
d92cbb7b1e Merge "Disk-image-create should allow sending compat flags to qemu-img" 2014-06-28 00:14:07 +00:00
Yolanda Robla
fabbb8da58 Disk-image-create should allow sending compat flags to qemu-img
Add a --options flag that allows disk-image-create to send custom
options to qemu-img calls. It is useful for forcing compat into
qemu-img created images.

Usage:
    disk-image-create --qemu-img-options compat=0.10 base ubuntu

Change-Id: I73ee2c86abc115220d51ffebbbe9ce3c4d92f188
Closes-Bug: #1329746
2014-06-25 12:38:06 +02:00
Jenkins
5dfb4b5c93 Merge "Factor out error behavior in dib-lint" 2014-06-23 09:38:04 +00:00
Om Kumar
bccffc8bfd Refactor code to select boot kernel
The script to find best kernel from image (used by vm
and baremetal element) is duplicated and is not in sync
with each other. Moving the code to img-functions as a
function will reduce duplicated code and make it reusable
in future.

Since img-functions is not accessible in chroot env,
kernel selection is being moved from finalise.d to
cleanup.d in the vm element.

Change-Id: I8fbccc13a2c61a5191ef9ea5d2a8302a3e43b000
2014-06-13 11:46:39 +05:30
Derek Higgins
e7d105a751 Add tar as an output type
Add support for d-i-b to create tarballs, the primary consumer for
tarball images would be linux containers.

Change-Id: I27d67401f3e4415226a4a51e1dde46f739c0220a
2014-06-11 10:48:50 +01:00
Ben Nemec
e824b43cbd Factor out error behavior in dib-lint
Currently when dib-lint finds a problem it does something like:

 echo "ERROR: Problem found"
 rc=1

This is repetitive and error-prone since it's easy to forget to set
rc to actually fail the check.  This change makes those two steps
a single function call.

Change-Id: I40b5bf39348a69add1f955c49f310e3bda21be0e
2014-05-28 11:04:27 -05:00
Jenkins
43827916d5 Merge "dib-lint: ensure file finish with a new line" 2014-05-28 00:41:03 +00:00
Ben Nemec
2830ef4854 Check for set -o pipefail
Like the other set lint checks, this will hopefully catch errors
in element scripts sooner.

Change-Id: Ib1600938f6ffed657ff2950ef54bbcbdb5a0db08
2014-05-23 15:57:27 -05:00
Jenkins
273d364fc4 Merge "Ensure scripts are set -u" 2014-05-22 03:03:28 +00:00
Gonéri Le Bouder
0e516cc8d1 indent: search for !=4 spaces indentation
With this change, dib-lint raises an error if when finding an element
using space indentation that is not multiple of 4.

Co-Authored-By: Jon-Paul Sullivan <jonpaul.sullivan@hp.com>

Change-Id: I470e1fdfc38a3f3c7ba5644c5103f2a9ef073005
2014-05-10 11:17:39 +02:00
Ben Nemec
e0650572f2 Ensure scripts are set -u
This will hopefully catch bugs where they happen rather than
allowing scripts to continue on and fail later.

Change-Id: Idacd9274415b21db285a198dafff19b1d19a4a68
2014-05-06 15:51:07 -05:00
Gonéri Le Bouder
bd69d32806 dib-lint: check for tab indent in files
With this change, dib-lint ensure elements do not use tab
indentation. The following files are checked:

- executable file
- .md file

Change-Id: I071262ff9f6599548f869f5439ee127f64eeb46f
2014-05-06 13:54:38 +02:00
Gonéri Le Bouder
27e7aaefa6 dib-lint: ensure file finish with a new line
git and the patch command expect file to end with a new line.

Change-Id: I783583b2a5a703007bf206ba56a1caa6fe15973e
2014-05-06 13:44:58 +02:00
Ben Nemec
79ab95b16e Make sure all scripts are set -e
If scripts are not set -e then errors can be ignored, causing more
confusing failures later.

Also adds an exclusion comment to the ramdisk init script since we
don't want that to exit on failure.

Change-Id: Idf43993bd10b1ef16c1d3b0d9df8d0ad94c46458
2014-05-02 17:18:54 -05:00
Ben Nemec
825b1964bb Allow excluding tests from certain files
There are certain scripts (such as the ramdisk init script) that
need to ignore linting rules for one reason or another.  This adds
support for exclusions via a comment in the file like:

"# dib-lint: disable=executable sete"

There should be no ", but I need those to prevent git from treating
that as a comment.  This syntax is similar to the pylint exclusion
mechanism.

Exclusion support is added to the executable check, but not
the alphabetical ordering one because I can't imagine a reason we
would need to disable that, and I don't know that comments are
supported in those files anyway.

Change-Id: I9ecfb47269841dc75a005855455ac26ad2cbc642
2014-05-02 17:18:54 -05:00
Monty Taylor
1c07d4ba1a Remove All Rights Reserved
The phrase is no longer needed as of August 23, 2000 with Nicaragua's
joining of the Berne Convention.

Additionally, in at least one instance,
elements/cache-url/bin/cache-url, its existence in the file between
Copyright lines is just weird and feels misleading, even though it is
not.

Remove all of the lines, because sanity.

Change-Id: I24fd76c2b4f66b8036010b5079db39ead729abee
2014-04-25 11:25:10 -07:00
Ben Nemec
e6e5076698 Enforce alphabetical ordering of element-deps
We've started to require this in reviews, so we should really have
automation in place to catch it right away.

Change-Id: I43fd90647acba400cea11c665fb587856514b0ee
2014-03-26 15:23:18 -05:00
Ben Nemec
f961d745e3 Add dib-lint script
This will provide a place to put checks that catch common errors
in elements.  To start, this just checks that files starting with
a shebang are chmod +x so they can actually be run.

Change-Id: I4116a8f38f7bdfc5866764354c459fad8ca18e92
2014-03-26 15:23:05 -05:00
Jenkins
bfba0ccd03 Merge "Remove an excess cp of disk images." 2014-03-26 08:38:52 +00:00
Robert Collins
dc93feee69 Remove an excess cp of disk images.
save_image is used to copy kernel and ramdisks out of the image, which
we will sometimes want to keep the source, and sometimes not. However
for the main image itself, the temp copy is never kept, so use mv
  rather than cp and avoid the excess IO.

Change-Id: I5a9f0d69ffee3e6b872a8927537ac17f02f5aa4d
2014-03-25 17:20:37 +00:00
James Slagle
3953fd5d2c Always export IMAGE_NAME
A recent change[1] relies on IMAGE_NAME being available when
98-source-repositories was run. However IMAGE_NAME was previously only
exported if the -o option had been used.

[1] I7dbe9e163ad38a418cf2869a81e720de2c27dfb1

Change-Id: Ife3d94fd45a2e0e0948c17414c369e0e6e040442
2014-03-13 16:35:36 -04:00
Jenkins
66ae41756b Merge "Pep8/Pyflakes fixing" 2014-02-20 04:28:41 +00:00
Dan Prince
ea555476cc Make max-online-resize an option.
In fb246a02eb we introduced an
ext4 option to allow root filesystems to be resized up to 1PB.
This appears to cause an ext4 resize2fs bug in some images.

When the issue occurs an image will hit a kernel bug when
cloud-init runs the resize2fs command during first boot:

  kernel BUG at fs/ext4/resize.c:409!

In this commit we add a new option for max-online-resize
which can be used if a really large root partition is
desirable.

The root cause of all this is really a design problem in
DIB/TripleO at the moment in that we shouldn't have to worry about the
max size of the root file system when creating our images.  Ideally we'd
just mkfs on the root file system itself. Much more efficient,
avoids this problem altogether...

I think the best thing to do today to avoid this is make setting
max-online-resize an option in DIB. This will allow us to stick
to the (well tested) ext4 defaults for most cases, and if someone has
need for a large root filesystem they can easily bump the setting. This
may be temporary until we either fix the design... or the ext4 fix is
released.

Change-Id: I371f62555d2753cec48790c8fd811c4342af925c
Closes-bug: #1280709
2014-02-19 08:37:56 -05:00
Dan Prince
21c1a6205b Correct DIB long option parsing
Fixes issues with option parsing. The --min-tmpfs and --image-cache
options where previously broken.

Closes-bug: 1282077
Change-Id: I40c62b16d854335902d1b6b5ceab7f0e1992623a
2014-02-19 08:33:08 -05:00
Dirk Mueller
0ab2a1300f Pep8/Pyflakes fixing
Just avoiding the error popups in the editor that way.

Change-Id: Ic0cb903dfb0e9eff6b12d004da720919f6f1a009
2014-02-17 22:37:34 +01:00
Dirk Mueller
7dcbfba5a4 Fix kernel extraction on openSUSE
Change-Id: I4ceaff4e6a42fe8dcfe7447f128d7943889ffc3a
2014-02-06 23:46:28 +01:00
Steve Kowalik
58c755cf4c Create a new baremetal element
Rather than using a script to mount the image using nbd to extract the
kernel and ramdisk, make a new element called baremetal, which contains
a cleanup.d script that will copy them out to <image name>.{vmlinuz,initrd}.

Closes-Bug: 1224669
Change-Id: I8f3569aa12148d18b1c8242b6fbbd8857894b26f
2014-01-21 16:48:20 +11:00
Clint Byrum
d2cb979ddf Increase padding to allow for smaller images
With smaller base images, such as Debian, the padding value used to
size the root filesystem does not leave enough room for grub which is
installed after the filesystem is created.

Change-Id: Ic2ab9e2efc9bf1b02f802ee47a36e3fff9c3512e
2014-01-11 18:06:33 -08:00
Jenkins
37f3c4fe90 Merge "Quieten disk-image-get-kernel" 2013-12-12 16:31:54 +00:00
Derek Higgins
d5b30d126e Increase the size heuristic for images
Occasionally cloud-init fails to resize the disk on first boot, this is
occuring because the filesystem has only 10% free. Nudging the disk size
up a bit more should give us a little more headroom so this is less
likely to occur.

I havn't been able to find out what value ensures we wont hit this
problem but I can say for sure this fixes all casses of the problem I
have seen.

Change-Id: Ib23f3a654151338ad91839e49b323b65b4054245
2013-12-11 14:38:24 +00:00
Mark McLoughlin
af8b7f05a1 Quieten disk-image-get-kernel
disk-image-get-kernel is pretty noisy and you see e.g.:

  $> load-image overcloud-compute.qcow2
  XXX  -d '/tmp/image.lWGCgPoj' -o 'tmp' -i '/home/stack/overcloud-compute.qcow2' --
  Extracting kernel + ramdisk from /home/stack/overcloud-compute.qcow2 and writing them to /tmp/image.lWGCgPoj
  nbd                    17554  0
  nbd                    17554  0
  basename: missing operand
  Try 'basename --help' for more information.
  /dev/nbd0 disconnected
  tmp-vmlinuz,tmp-initrd

Clean all this up so we just get:

  $> load-image overcloud-compute.qcow2
  Extracting kernel + ramdisk from /home/stack/overcloud-compute.qcow2 to tmp-vmlinuz and tmp-initrd in /tmp/image.g6b0lG88

Change-Id: I8971ec0bbcd87157b07fc17254c56bb9f9f2a597
2013-12-09 13:39:29 +00:00
JUN JIE NAN
65a3063da3 Add examples for ramdisk-image-create
The examples to create cfn-tools image do not make sense for
ramdisk-image-create.

Change-Id: Iff50898119a1f37f41e52e6215c6bbeca003dfff
2013-12-05 18:10:28 +08:00
Dan Prince
255387a288 Drop default distribution root element support
Converts our existing default root element code to be just a check
which exits with a failure message if no root/distribution element
is found.

Change-Id: I954a6abfd7871d5807b1a171a03fa98932410cff
2013-12-02 11:01:29 -05:00
JUN JIE NAN
7c2d1b2295 Store DIB_* only env variables
Or environment variables like OS_USERNAME OS_PASSWORD will be stored
in /etc/dib_environment also. It's potentially unsafe.

Change-Id: I3a65d5bc0e4469a071db9ac23ebc82196a3f3feb
2013-12-02 09:26:35 +08:00
James Slagle
844daff526 Add option --image-size.
Adds an option for --image-size, which sets the DIB_IMAGE_SIZE
environment variable.  Having the cli option makes disk-image-create
more consistent in that parameters are able to be specified via a flag
on the command line or an environment variable.  Also, you may not be
able to remember all the environment variables that you can tweak, yet
--help will tell you about the cli parameters.  Preserves the old
behavior as well where if DIB_IMAGE_SIZE is set in your environment, it
will be honored.

Change-Id: I195c9144a80ce7b8bd5809b57f2bed71a2cbdf26
2013-11-25 15:28:18 -05:00
Jenkins
35830cc57c Merge "Usage message enhancement" 2013-11-14 09:38:10 +00:00
JUN JIE NAN
6d6634e804 Usage message enhancement
Add --help, add a simple usage and two examples

Change-Id: Iddf07ec1935f337000bd8ffb3d7c1fcc65753e8d
2013-11-14 09:18:45 +08:00
James Slagle
dee69b1810 Python code refactorings.
This commit refactors the python code in diskimage_builder slightly.  This is
in preparation for a larger review that adds more functionality to the python
code, namely the ability to apply elements to the current system as opposed to
a chroot.  Further, the refactorings can stand on their own for better clarity.
They include:
 - renaming elements.py to element_dependencies.py.  Adds clarity about the
   purpose of this module.
 - updating other code for this rename.
 - move tests into a tests submodule.

Change-Id: I5519cc52398e442b24e33802bae42070d64b0c1d
2013-11-12 10:47:57 -05:00
Ben Nemec
237fd64b47 Ignore basename failures
Commit 0210be22ae introduced the
possibility of basename being called on an empty string.  Because
that can happen in normal operation (such as an x86_64 kernel,
in which case the PAE check will find nothing), don't allow the
basename error to kill the script.  Also ignore failures in the
second basename call so the proper error message is echoed.

Change-Id: I38a18af09ab24fda9c98cbf3ace8fd7acc6faef5
2013-10-17 22:36:53 +00:00
Lucas Alvares Gomes
0210be22ae Fix disk-image-get-kernel for redhat
disk-image-get-kernel picks the wrong version of kernel and initramfs
if there's a PAE and non-PAE version present. Only affects i386 images.

Change-Id: I06e08fdf038988759b620f549261499cb0a69b34
Closes-Bug: #1240873
2013-10-17 12:20:04 +01:00
JUN JIE NAN
39cbbd6980 Increase journal size to 64 M for ext4 file system
the problem is that the journal isn't large enough to allow online
resizing. Solution is straight forward. So the file system can be
resized successfully to disk size specified in flavor.

Fixes bug #1233008

Change-Id: Ie84fb8aea8d334706574d1a8006ec9eaee5bb5be
2013-10-05 16:53:53 +08:00
Matthew Farrellee
1d2a85d5e4 Add DIB_IMAGE_CACHE
DIB_IMAGE_CACHE will be a user override for the location where images
are cached. Default location is ~/.cache/image-create

Change-Id: I3e9b9f970864d555c9ec9436344b53f6d3d66dfa
2013-09-19 14:34:05 -04:00
Jan Provaznik
0eaddbd214 Fixes files ordering when choosing newest image
For example if there are following kernels in undercloud/overcloud
image:
/boot/vmlinuz-3.9.5-301.fc19.i686.PAE
/boot/vmlinuz-3.10.10-200.fc19.i686.PAE
then disk-image-get-kernel picks vmlinuz-3.9.5. It should use the
newest one.

Change-Id: I7bbf06705e85370d66c7dd8a5d4f8d6c93b21c0c
Fixes: bug #1224365
2013-09-12 11:26:57 +02:00
Derek Higgins
e8c73cde49 Set raw image size to be multiple of 64k
This works around a bug in qemu-img causing it to sometimes error if
compressing a raw image that isn't a multiple of 64k (the qcow2
cluster size).
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1180021
http://lists.gnu.org/archive/html/qemu-devel/2013-04/msg02520.html

Change-Id: I88f8bb5c6bf3606ef5856b085a284af6b8858dbb
2013-09-08 21:36:29 +01:00
Jenkins
771f3ec141 Merge "Add option --min-tmpfs <size> to disk-image-create" 2013-08-28 15:40:31 +00:00
Ghe Rivero
fefc28a918 Add option --min-tmpfs <size> to disk-image-create
In some scenarios, the required space in the tmpfs partition can be
larger (or smaller) than the default one, producing errors due to
the lack of enough space (or performance penalties for not using
tmpfs).

Using --min-tmpfs <size>, we can hint the working set size we'll need
and let dib choose to avoid or use tmpfs.

Change-Id: I7d5fe498302a100c8555ae542268e14b21f3a0c5
2013-08-28 11:14:06 +00:00
Ghe Rivero
dbfca7a816 Combine compress and save image into one function
When compressing an image, this is done in the same dir where the raw
image resides, doubling the amount of space needed (scarce when
using tmpfs), and then it's moved to the .cache folder in disk.

Combining these two functions, we reduce the amount of space needed
in the tmpfs partition (when in use), and the compressed image is
created directly on the .cache folder disk, so there is no need to
move the compressed image after the process into disk.

Change-Id: I451d24bdd6fa0983414244135dff5e96c0549833
2013-08-28 10:51:40 +00:00
Derek Higgins
4651913e55 Add a new break on error
A user running di-b several times while developing an element may not
want to drop to a shell in all cases but may only want to do so if one
of their in target hooks failed.

This patch gives them the ability to do so, If break=after-error is set
then a user will be provided a in target shell taking over from where the
last failed command left off.

Change-Id: Ia2f7ac4c21b64b971f87f4ae9cb867981b13eb5e
2013-08-01 17:08:54 +01:00
Jenkins
9233ca4056 Merge "Enable running disk-image-create on SUSE Linux" 2013-08-01 00:36:36 +00:00
Jeff Peeler
87fb533979 Modify relative paths of lib, elements for packaging
(Based on review https://review.openstack.org/#/c/36009)

Scripts test for existence of ../share/diskimage-builder and
fall-back to ../ if not found. This allows scripts to run unmodified
from a packaged installation or a local archive/repository.

Change-Id: I0cf4c1fdb8e42ec284c56860cb15818632b93b9e
2013-07-31 13:15:33 +12:00
Tim Serong
ee5ae03d58 Enable running disk-image-create on SUSE Linux
- Ensures /sbin and friends are in $PATH when invoked (without this,
  various sudo invocations fail in exciting ways).
- Use dib-run-parts in lib/common-functions instead of run-parts
  (neither SLES nor openSUSE ship run-parts).
- Ensure dib-run-parts doesn't descend into subdirectories (same
  behaviour as run-parts).
- Move dib-run-parts from root.d to bin (cleaner, consistent with
  other elements with separate bin scripts).
- Tested by building Ubuntu image on openSUSE 12.3.
- Note: this doesn't add support for creating SUSE images, it just
  lets you run disk-image-create on SUSE-based distros.

Change-Id: I906c6bc3cf51cdf2c4415adeae1ca250faac25e1
2013-07-24 20:16:31 +10:00
Robert Collins
f7f6cb45ee Fix the DIB_OFFLINE setting to actually work.
I missed the getopt parameter and forgot defaults are imported after
option processing. Untested code is broken code!

Change-Id: I133a691909d38e834c204950276a57f4884fc4ed
2013-07-18 13:19:25 +12:00
Robert Collins
b2f1d4e2af Document an interface for offline operation.
Complex image builds can download hundreds of MB of data from the
internet with many separate lookups. It would be nice to allow users
to ask for a fast build where those lookups are entirely avoided,
using locally cached resources (where possible). This new interface
allows users to signal to elements that they wish to operate without
updating cached resources, which will in turn allow us to avoid
checking for stale data at all.

As part of this I've also documented where we cache data, so that
things like the ccache cache dir and image cache files are not a
surprise to users.

Change-Id: I27f5de6ceaa4e9c6390721b7c434fe0908df84f5
2013-07-18 09:17:01 +12:00
Chris Jones
8d1ce9c0c3 Build ramdisks in an image chroot.
Ramdisks are now built inside a chroot which is built by the normal
image build process. Doing so improves our independence of the
precise state of the build host.

This fixes bug 1194055.

Change-Id: Ibc254fbb9e7b404b5f38c1b35bcde8a4136e8e28
2013-07-12 11:09:35 +01:00
Robert Collins
a969be49a2 Move functions to common-functions for reuse.
Change-Id: I12a5687d94df678d7f7e8c0b075ee4aba1082156
2013-06-25 17:36:37 +01:00
Robert Collins
0b1ff7d4b0 Factor out element processing.
This reduces code duplication for hygiene, no functional changes.

Change-Id: I10c96da4f07bcd1e5aabfcfbd115ecaef7bb65a1
2013-06-24 16:30:25 +01:00
Robert Collins
9b01f2b129 Consolidate more ramdisk and disk-image code.
RAMDISK_ELEMENT is a spurious difference, to merge the code we need
only semantic difference to exist.

Change-Id: I7d38b2457b6fc838d1a536406bb3c34c00f2b1f9
2013-06-24 20:53:19 +12:00
Robert Collins
2c47b700be Reduce duplication between ramdisk and disk image codepaths.
Change-Id: Ie9e2d60ffe94f971953d52b7b5bce9c1a86fb1b5
2013-06-19 21:26:29 +12:00
Lucas Alvares Gomes
c28cb1b5d8 Add RedHat support for disk-image-get-kernel.
RedHat and Debian based systems have different binary names.

Change-Id: I5162204759fbc4ff167d1584294c9936ab7df4c8
2013-06-05 17:12:09 +01:00
Robert Collins
1f0bc8f19b Increase the size heuristic for images.
Bug 1183597 reported that we create them too narrowly today.

Change-Id: Ia863cf2fb19c813c51f94dd0b8e210a75dccd945
2013-05-24 10:32:10 +12:00
Robert Collins
c139b0cb46 The ext4 resize fix was faulty - fix it.
Fix bug 1183251 by replacing an erroneous reference to OPTS with
MKFS_OPTS.

Change-Id: I741005d574715b8b237d9789398dbaba29df60e3
2013-05-24 10:06:33 +12:00
Lucas Alvares Gomes
85b5ef0c8d Move the ensure_nbd function call.
Move the ensure_nbd function call from the disk-image-create to the
disk-image-get-kernel. The image creation process does not use qemu-nbd
anymore.

Change-Id: I8ff2ad54f322a9e33d52408fa4e253a7b45900c7
2013-05-20 11:59:21 +01:00
Robert Collins
fb246a02eb Ensure that the ext4 fs can be rebuilt up to 1PB in size.
Change-Id: I16c5fe4eb0a9951d44745d6a46450f70d33b3481
2013-05-18 13:08:06 +12:00
Jenkins
ffdf0ba7ff Merge "Honor $DIB_IMAGE_SIZE." 2013-05-13 17:56:17 +00:00
Tim Miller
9983245249 Honor $DIB_IMAGE_SIZE.
The recent change to use loopdev instead of nbd stopped
honoring the DIB_IMAGE_SIZE setting.

This change adds it back, by resizing the image to
DIB_IMAGE_SIZE GB, if specified. If unspecified, it resizes
the image to (actual usage + 20%), as it did before this change.

Change-Id: I69afd9584e644ddacc948619100f153d3d8713a4
2013-05-13 10:20:53 -07:00
Clint Byrum
20661e8d80 Retry losetup -d for up to 10 seconds.
When partition tables are added to loopback devices, this can set off a
chain of udev hooks that may still be holding the loopback open. Failing
to detach loopback devices was the reason we were seeing leaked tmpfs
volumes.

Fixes bug #1178091

Change-Id: I836d6e2bbce824951dd4786e3ef28273ea18ee73
2013-05-09 09:34:53 -07:00
Chris Krelle
76d8d98a60 Remove img-functions from ramdisk-image-create.
Ramdisk-image-create sources both img-functions and ramdisk-functions
causing duplicate functions. To correct this situation this patch
removes the import of img-functions from ramdisk-image-create.
It moves cleanup_dirs and run_d functions from img-functions to
common-functions to allow this.

fixes bug 1175427
Authored-by: Chris Krelle <nobodycam@gmail.com>

Change-Id: I971d019b0d92b06d9de661663b49c433e823ea42
2013-05-08 07:39:30 -07:00
Clint Byrum
d9a2211d14 Only use tmpfs if build machine has 4GB+ RAM.
For machines with low RAM (or no access to /proc/meminfo) the builder
will still run, but will just build inside the filesystem that is
hosting /tmp. This will result in a slower build (especially if there
are a lot of .deb packages installed).

Fixes bug #1175453

Change-Id: I79f2672058c11e377548820df0ab4fad8f47ffdc
2013-05-03 06:55:53 -07:00
Robert Collins
d119c28f49 Revert "Improve Fedora build host support."
This reverts commit 591059ffd3.

Change-Id: I8ad5bde2eabe12c6575d51e7443d9b46521c0928
2013-05-03 08:16:43 +12:00
Jenkins
55993863fc Merge "Improve Fedora build host support." 2013-05-02 19:30:43 +00:00
Lucas Alvares Gomes
591059ffd3 Improve Fedora build host support.
Sudo command (even using -E) resets $PATH for security reasons, Fedora
and other distos might have to explicitly add /usr/local/bin to $PATH
in order to run the commands in the chroot environment.

Change-Id: I95068ecf1e7437152e11c6d6789f59bcfd6ae9cc
2013-05-01 09:40:15 +01:00
Robert Collins
e27ac025fc Use a different approach to solving the ramdisk-image-create failure.
Change-Id: I2861927d1989bde1137c253c8b3a48b245c65f19
2013-05-01 15:03:44 +12:00
Robert Collins
cb62bae9b8 Build images using loopdev instead of qemu-nbd.
Qemu-nbd does not perform well with older versions of qemu due to
the lack of writeback caching mode. It also only builds qcow2 images
and there is a desire for raw image support. Finally, qemu-nbd makes
it very difficult to build images concurrently due to the somewhat
opaque nature of how it selects a /dev/nbd# device. losetup, on
the other hand, makes this process very straight forward.

Change-Id: I309fad8af4fd1e8d1720c17b65e1897a76d5e897
Co-Author: Clint Byrum <clint@fewbar.com>
2013-04-30 08:56:12 -07:00
Lucas Alvares Gomes
45173ccdb5 Adds support for post-install scripts.
Post install scripts are useful because they can perform tasks you want
to handle after the OS/application install but before the first boot
of the image. e.g: Clean the cache left by the package manager reducing
the size of the image.

Change-Id: I03e77f602192bbdce29c02999d1b57fac8051ddc
Fixes: Bug #1145786
2013-04-17 09:47:51 +01:00
Lucas Alvares Gomes
b20e133714 Grab the next available /dev/nbdX.
Change-Id: I6a2c8e58cdab889d747e870d0009f51bba0ff148
2013-04-12 17:49:23 +01:00
Chris Jones
bd1bae7c58 Introduce the DIB_ namespace for build-time config.
Document the use of DIB_ as a prefix on environmental variables that can
be overridden at build time.

Introduce the first such variable by migrating IMAGE_SIZE to
DIB_IMAGE_SIZE.

Change-Id: Ie36b734991b913a23f37f2add47d470d7c1576e4
2013-04-12 09:50:34 +01:00
Robert Collins
5ada82a59c Add armhf support.
ARM doesn't have a generic Linux image due to the soc-specific nature of Linux
kernels today, so we drop the manual installation of that package, replacing it
with a dist-upgrade instead. This involved tweaks to the dpkg and fedora
install-package scripts.

Change-Id: I97924b80ca87781307e1087b9fe4b18215770e84
2013-04-09 08:17:22 +12:00
Chris Jones
0cfecd1024 Store build-time settings
This will write two files in /etc/ that contain the environment and
command line arguments used in the creation of an image. This should
assist with later efforts to repeat the creation of an image.

Change-Id: Icdbe6693380bed6c406feee10d2cb1a88a992932
2013-04-04 20:39:11 +01:00
Chris Jones
e34e75ff22 Remove duplicate file extensions.
Gracefully remove duplicate file extensions on the output filename

Change-Id: Id4f8b0d75b587c238c89d6ca7a06fab6a57cd627
2013-03-15 16:32:57 +00:00
Ghe Rivero
308eee2827 Use multiple locations for elements dir.
Using ELEMENTS_DIR env variable, you can specify multiple dirs containint your elements.
It must be a ":" separated elements list.

When an element is seen in one of those lists, it will be chosen and will continue with the next element.

Change-Id: I18eca27d943139cd6ca1ebd232b419e502d7b048
2013-02-27 10:48:55 +01:00
Robert Collins
9afddcf266 New element that uses a fedora cloud image as the base.
Change-Id: I7d83bb2b359e7a8c3858eca04c96e35cf4e1fe9e
2013-02-15 09:33:29 +13:00
Robert Collins
7f77729ec5 Move the dpkg specific stuff to a dpkg element rather than being hardcoded.
This includes the install-packages implementation for dpkg, apt http proxy
config, daemon blocking and unblocking.

Change-Id: I8f159021d2b223d7003cec067de3aa605ad06974
2013-02-14 11:16:12 +13:00
Tim Miller
56e50ee5cf Consolidate common OS installation into a script:
Move common openstack service installation operations
into a new script `os-svc-install`, which simplifies
the elements for openstack services.

Change-Id: Ied8ac3278e7fe8af76e24748ec4e598a84afa03c
2013-02-13 11:06:00 -08:00
Jenkins
750e4c3e06 Merge "Correct misuse of return in ramdisk-image-create" 2013-02-12 23:05:07 +00:00
Jenkins
4008475bfc Merge "Fix unnecessarily creating a temporary directory" 2013-02-12 23:04:56 +00:00
Chris Jones
8f96a286d7 Add default element selection
This change includes the 'base' element by default and adds a
command line switch to prevent this behaviour. This should reduce
confusion since this element is almost always necessary.

Change-Id: Ib4a2ef53a76b119b764751fa21058f4e2cb37741
2013-02-12 09:29:19 +00:00
Arata Notsu
aa6f5f8005 Correct misuse of return in ramdisk-image-create
Replaced return with exit.

Change-Id: Iee26177cdb58af67b6919da12456debfe5940105
2013-02-12 17:43:35 +09:00
Arata Notsu
5c13dc1d75 Fix unnecessarily creating a temporary directory
mk_build_dir was executed twice in ramdisk-image-create.

Change-Id: I93cf83775fb5e7ba3e6b84643f723568a2420305
2013-02-12 17:29:52 +09:00
Robert Collins
1ee3a01447 Move initial root contents into a hook.
This is a necessary but not complete step towards supporting Fedora and Suse
distributions. Further work is needed (e.g. to quiesce daemons on
installation).

Change-Id: If3ea6093d41a21de755db52328226b84b5a3ede6
2013-02-11 17:01:36 +13:00
Chris Jones
997165ff11 Add option to clear environment
This adds an option to disk-image-create that allows it to re-exec with
an empty environment if required.

Change-Id: I9e51aa07b903e18cdd0ed5f953800307f9899b5d
2013-01-30 18:24:24 +00:00
Chris Jones
15dbc42d72 Switch locale to C
Remove the explicit installation of an English language pack and instead
just default LANG to C. This settls Perl's noisy warnings and also stops
any side effects of different language build hosts.

Note that this is set right at the start of the build process, so it
should be entirely possible to override if needs be.

Change-Id: Id3b31162d4198fa02dc5a4d11168e57dbcd14a5d
2013-01-28 18:08:50 +00:00
Clint Byrum
c8c33e3bb1 Add a simple implementation of element dependency
This adds a new optional file to the root of elements. The
file lists dependencies which will be added to the list
requested by the user during disk image creation.

Change-Id: Id71c3b333563604bbbaf90f9cf40e24fa9738fc8
2013-01-23 16:04:20 -08:00
Chris Jones
0f73578721 Allow manual installation of packages.
Rather than force creation of an element for a single package install,
allow people to do this from the command line.

Change-Id: I63e2e7e50c4a7dbb8a8e198581dfadce91773621
2012-12-21 17:46:44 +00:00
Chris Jones
219aa52958 Fix disk-image-create's getopt error handling:
We originally checked the return code of getopt, but subsequently
a debugging echo has been inserted and now we are checking the
return code of "echo", which is very rarely going to be useful.

Change-Id: I054fecaf6576d5e9d42b07159bba7d50335bf6cc
2012-12-17 17:00:13 +00:00
Robert Collins
e8a8e99a08 Make it possible to not recompress the qcow2 image at the end.
Change-Id: Icca922853ee583dcccb6d3970d080fbec11edac7
2012-12-14 13:25:18 +13:00
Monty Taylor
1eff4a436e Rename flavour to element.
Flavour is overloaded in openstack due to it being used by nova. Element
seems to have the same feeling of combinability without using a term already
in active use in the openstack community.

Change-Id: Ia4c028d4062a8f69c66665821c94dd4bcdf06031
2012-12-05 14:04:58 -08:00
Arata Notsu
0b81a3e0e8 Remove udev stuff from the source tree
Pick udev stuff from the running evironment instead.

Change-Id: I90a004d0a2b83d3451e73ae351cf474a7bfd62ee
2012-11-29 12:32:06 +09:00
Devananda van der Veen
fb426153f4 remove force-xtrace in disk-image-get-kernel 2012-11-18 08:17:54 -08:00
Devananda van der Veen
e5fcb27790 add some tooling that baremetal-devstack wants
- add bin/disk-image-get-kernel to extract vmlinuz/initrd from .qcow2
- add corresponding methods to lib/common-functions
  - map_nbd
  - (un)mount_qcow_image
2012-11-18 07:20:27 -08:00
Robert Collins
4b7cdb4fdc Fix copyrights for HP work. 2012-11-15 16:20:32 +13:00
Chris Jones
f7091cb2fc Add support for flavours to ship udev rules.d files and port over the mellanox variant to use this 2012-11-12 16:57:13 +00:00
Robert Collins
40e954735c Make it possible to set a size from within a flavour, and use that for devstack. 2012-11-12 11:47:26 +13:00
Chris Jones
eb584f14ed Port old baremetal-mkinitrd.sh to the new image creation standards 2012-11-10 10:50:15 +13:00
Robert Collins
5cc195e391 Bring across disk image code. 2012-11-10 00:04:13 +13:00