Commit Graph

214 Commits

Author SHA1 Message Date
SamYaple
0c32784663 Revert "Zerofree the image if possible"
This reverts commit 5184d02a7c.

The decision was made to go with fstrim because it is faster and more
universal that zerofree. The related-id has the patchset that implements
fstrim.

Related-Id: I269b4416be450369616f9b8e030f84c30e329804
Change-Id: If40cf2fc0ecd8686768cbfeac9ecee90907674e7
2016-03-10 20:38:01 +00:00
Clark Boylan
5184d02a7c Zerofree the image if possible
If the image has an ext filesystem and the zerofree utility is present
on the build system then run zerofree. This should make images as
compressable as possible which is a nice feature when building
compressed qcow2 images.

Change-Id: Ia6062c291f7a3f58b85a4f408ecb3d0574c65d53
2016-03-07 17:45:14 -08:00
Ian Wienand
53c57d75c7 Cleanup unmount_dir function
Cleanup this function to work with a symlinked directory.  Document
it's behaviour more exactly, and add a simple unit-test for it (not
run by default, due to doing things like mounting and unmounting
system dirs on a live system, which doesn't seem safe for CI.  But it
is useful for developers ensuring sanity).

Change-Id: I335316019ef948758392b03e91f9869102a472b9
2016-01-28 14:47:43 +11:00
Matthew Thode
176ae7bff6
Initial add of gentoo support for diskimage-builder
uses upstream's stage4 images, includes all the needed bells and
whistles for openstack on kvm.

Change-Id: Ibca43173c30c2a74a73a2e2d9dd6d6d832c62694
Closes-Bug: 1530911
2016-01-12 19:15:00 -06:00
Gregory Haynes
966f12dc4c Properly account for pipefail during cleanup
Due to a bug in how we were running some of our phases we were not
detecting some nonzero exit's. When this is fixed, dib fails early during
cleanup (leaving some resources attatched) due to not propertly
accounting for pipefail.

Change-Id: Icc0b35acbe035cac12a9291e2d07b6c690c3a6ad
2015-12-31 19:43:55 +00:00
James Arendt
12cb94a75d Fix unmount/remove race in cleanup_build_dir
With a slow file system, umount can return 0 and the immediately
following remove can fail with a "Device or resource busy" error.
This happened in DevStack in disk-image-create where unmount_image
is followed by an immediate cleanup_build_dir.

Solution is to apply same logic from bug 1332521 to allow the
remove to retry on failure (up to 5s) in case the umount has not
completed.

Change-Id: I3337e2b4ad0111e77f79dc179439cdfea8ebdeda
Closes-Bug: #1527721
2015-12-17 20:55:02 -08:00
Jenkins
af2226305d Merge "Prevent overwriting of user modified blacklist.conf" 2015-10-05 18:57:01 +00:00
Nisha Agarwal
d9ce041237 Prevent overwriting of user modified blacklist.conf
The ramdisk-functions overwrites user modified
/etc/modprobe.d/blacklist.conf with a single entry
"blacklist evbug". Due to this, ramdisk fails to recognise
the hardware/disks etc for which user has modified
/etc/modprobe.d/blacklist.conf.
The commit enables copying all the *.conf files under
/etc/modprobe.d to the ramdisk. It creates
a config file /etc/modprobe.d/blacklist-dib-ramdisk.conf
instead of overwriting /etc/modprobe.d/blacklist.conf.

Closes bug: #1492804

Change-Id: Ib88272e4b8f4641c58e7e68bb0c2e4b82efc4fc1
2015-09-30 02:04:15 -07:00
Ian Wienand
494a833987 Add #!/bin/bash to library functions
Currently when these files are opened your editor doesn't know what to
do with them.  Add #!/bin/bash to library functions so that editors,
diff-tools, etc can do syntax highlighting.

There are other ways to skin this cat, such as renaming to ".sh",
adding -* style editor flags, etc.  We had this discussion in DevStack
too, and came to the conclusion the simplest thing that works for
everyone is to just put the #! at the top.

Change-Id: I4cf64321e14844696139f5d40e4d719436390b35
2015-09-16 13:54:07 +10:00
Jenkins
1663c42dab Merge "ramdisk: enable ppc64 support for symlink" 2015-07-16 23:07:12 +00:00
Jenkins
cdf4467a8a Merge "Cleanup the build directories earlier" 2015-07-14 11:50:53 +00:00
Antoine Musso
9bd84d4044 Only chown tmp dirs when they are a tmpfs mount
Temp dirs are created with mktemp and thus belong to the user. There
is no need to chown them unless we used `mount -t tmpfs`.

Move chown under the tmpfs_check conditional.

Change-Id: I37efe18ced3a06d461364dc5cb20600f1527e995
2015-06-24 13:58:31 +00:00
Dan Prince
96b01a7d2a Revert "Remove unused map_nbd function"
This reverts commit ea4a823810.

This function was actually still in use in lib/common-functions
and removing it causes the disk-image-get-kernel /bin
to fail entirely.

Change-Id: Icddb3ca369922a6ea915af8b1b62c434cb1bdf28
Closes-bug: 1464031
2015-06-10 18:37:08 -04:00
yogananth subramanian
f0f945d136 ramdisk: enable ppc64 support for symlink
Patch enables symlink /usr/lib -> ../lib  support for ppc64 Fedora deployment ramdisk.

Change-Id: I01994685292d7c1cae0f9ec1a1052d68420b4b82
Closes-Bug: #1463357
2015-06-09 16:10:13 +05:30
Pino Toscano
987a832351 Cleanup the build directories earlier
Split the cleanup_dirs function in two, i.e. cleanup of the build dir
and the image dir, and use the former to cleanup the temporary build
subdirs after their unmount, before the conversion to other disk
formats; they are not needed anyway at that point, and allows to save
disk space during the conversion phase.

Change-Id: Ie30d7e6033613d6979148423326ae7e17a7342e7
2015-06-04 10:46:18 +02:00
Tristan Cacqueray
356404f229 Follow symlink for elements
This allow custom elements to be added with symlink. Without -follow
a symlinked element is valid but scripts in *.d directory aren't used.

Change-Id: If50b7d9c3b1f6fe278c28488146709efe5cf065f
Closes-Bug: 1461124
2015-06-02 15:10:32 +00:00
Ben Nemec
1b92a11268 Don't log tmpfs message during cleanup
By that point in the build it isn't generally useful, and it causes
confusion when builds fail because people think that's the error.

Change-Id: I26dee4ac0947b71a4a065ef6c5a18103e7df6667
2015-06-01 10:48:48 -05:00
Jenkins
8d5923be11 Merge "Make it clear that tmpfs is optional" 2015-05-28 23:53:32 +00:00
Jenkins
38a3d98099 Merge "Cleanup /tmp in the guest" 2015-05-28 23:38:04 +00:00
Dougal Matthews
a6751c53a0 Make it clear that tmpfs is optional
Given this is often the final output, it can look like an error occured.
Changing the wording makes this clearer.

Change-Id: I70f157054e3120cffee6fa5241b1ffe0b7bfa650
2015-05-27 12:49:38 +01:00
Pino Toscano
37572a548c Cleanup /tmp in the guest
Clean all the content in the /tmp directory of the guest, leaving the
directory itself (usually has special attributes).

Modern distributions usually either setup a tmpfs on /tmp, or clean it
at every boot, so the leftovers will be just few bytes in the generated
image.
Regarding other distributions, a clean /tmp at their first boot will
surely not be a bad idea anyway.

Change-Id: I2b0f8864bc4909542d924f5bd9296dca5d0189f2
2015-05-20 11:24:17 +02:00
Gregory Haynes
b9b6640fa7 Initial element tests
Adding a test function which allows us to use elements to perform
element-specific tests. In order for this to work sanely, also adding
some configuration to our break system so we can assert on negative
tests.

Also adding a test for apt-sources to verify this code actually works.

Change-Id: I378a74255010eca192f5766b653f8a42404be5ea
2015-05-17 02:07:40 +00:00
Pino Toscano
cb37ef5384 ramdisk: switch from lsb_release to $DISTRO_NAME
Instead of executing `lsb_release` to know the current distro being
built, use the $DISTRO_NAME environment variable, already provided by
distribution elements.

Change-Id: I22b67afb481983cc40c198fd408ad5c7f4d68bec
2015-05-12 17:03:46 +02:00
Jenkins
9ba196394c Merge "Remove unused map_nbd function" 2015-05-06 02:50:17 +00:00
Jenkins
b2c2c47dc1 Merge "Actually set a sane PATH for inside chroot" 2015-05-05 01:53:11 +00:00
Jenkins
f8302171f5 Merge "Add element ubuntu-signed to provide signed kernel" 2015-04-28 11:02:44 +00:00
Gregory Haynes
f2a893b74c Actually set a sane PATH for inside chroot
In I084aff7e449f5de811a6169ec90e352ada7da439 we attemped to address a
bug for systems which dont have a path that works well for inside a
chroot. Turns out there were multiple issues:
 * The PATH we were setting was after we attempted to call sh.
 * PATH was being set in a sibling process to the command being run.
 * PATH was not being exported so it was not effecting child processes.

Using env to set a sane path before we attempt to run our commands
addresses these issues.

Change-Id: I4285f8048465ee5c2490116447d32033007bd185
2015-04-28 00:02:15 +00:00
Jenkins
74b777dae8 Merge "Fix dhclient in Fedora ramdisks" 2015-04-23 03:58:33 +00:00
Jenkins
5440d4b166 Merge "Dont try to unmount if were not using tmpfs" 2015-04-23 03:55:10 +00:00
Jenkins
f0736e7d97 Merge "Append full path to img-functions:run_in_target" 2015-04-22 19:06:12 +00:00
Jenkins
44d6d5cd88 Merge "Support VHD output format" 2015-04-22 05:31:04 +00:00
Jenkins
212da2f149 Merge "Try 5 times for rmdir command call after umount" 2015-04-21 21:22:19 +00:00
Ben Kero
4a2939c207 Append full path to img-functions:run_in_target
The build host might not necessarily have a complete set of dirs
required in its PATH. It will likely be better to statically code a
complete(ish) set of PATH entries inside the function.

Some distros (ex: archlinux) are lacking some dirs commonly found in a
PATH. If this is used as a host OS, it will transfer this incomplete
PATH into the chroot, where other guests (ubuntu, centos) will fail to
find basic binaries due to the lack of a /bin entry.

Change-Id: I084aff7e449f5de811a6169ec90e352ada7da439
2015-04-16 11:19:44 -07:00
Gregory Haynes
ea4a823810 Remove unused map_nbd function
This is a leftover from when we used nbd. We no longer do and this
function is not used.

Change-Id: If037bdd0100ef27401ddd5931ec3db7017302675
2015-04-15 20:20:12 +00:00
Gregory Haynes
7937fb5392 Support VHD output format
This useful for building images for xen.

Change-Id: Ib716bd7fc1ed979968ea92f46f1644038e40df66
2015-04-14 13:39:18 -04:00
Shivanand Tendulker
06e3d7c767 Add element ubuntu-signed to provide signed kernel
ubuntu-signed element would install 'linux-signed-image-generic' that
provides signed kernel that can be used for deploy in UEFI secure boot mode.

Package 'linux-signed-image-generic' ships signed kernel with extension
'.efi.signed' (Ex. '/boot/vmlinuz-3.13.0-49-generic.efi.signed').

The kernel modules directory for signed kernel and unsigned kernel is same.
It is without 'efi.signed' extension to its name. This is different from normal
practice of directory naming in '/lib/modules' (Ex. For signed kernel
'vmlinuz-3.13.0-49-generic.efi.signed', modules directory is
'/lib/modules/3.13.0-49-generic').
This needed some changes in '/lib/ramdisk-functions' and 'ramdisk' element to
copy kernel modules.

The signed kernel package contains both signed and unsigned kernel. The
unsiged kernel is without extension '.efi.signed' (Ex.
'/boot/vmlinuz-3.13.0-49-generic'). This required change into
'/lib/img-functions' and 'baremetal' element to pick up signed kernel version
when this element is used.

Closes-Bug: 1443076
Change-Id: I60061cbea847b47fa752b9463cfd387e8e7f0635
2015-04-12 11:36:17 -07:00
Haomeng, Wang
cc9870ec11 Try 5 times for rmdir command call after umount
Sometime, we will get "Device or resource busy" during the mount point
deleting, umount return 0, but the resouce is busy for a while, so need
to add sleep interval to wait resource free, then we can delete it.
Change-Id: Idaa219d12e847824960eec8907739add5d619d1a
Closes-Bug: 1332521
2015-04-07 02:01:29 +00:00
Gregory Haynes
d9bcde8a4d Export image properties
Some elements (such as the manifests element) want to use these
variables. We currently do not consistently export them (IMAGE_NAME is
only exported if you actually specify a name).

Change-Id: I43d17ddcdd7d0ff3cbb4c530caeebb8da915f4ef
2015-04-01 20:33:17 +00:00
Derek Higgins
5671a38e8e Fix dhclient in Fedora ramdisks
Copy all of the necessary parts for a Fedora based dhclient to work. This
includes a number of network scripts. Also grab the ip command supplied by
the iproute package, the busybox "ip addr" command was missing the valid_lft
and preferred_lft options.

This will allow the dhcp to work in the ramdisk instead of getting passed the
PXE net config.

Related-Bug: #1417026
Change-Id: I8feee9a740855dab7b47162c5727bf91db77fcc6
2015-03-31 12:27:40 +01:00
Gregory Haynes
2d79e9d395 Short circuit qemu-img convert for raw images
We currently use qemu-img convert with a raw source and dest when
building raw images. We can just mv the file for increased speed.

Change-Id: I3da095cb9ecad7224a121a434a9fb204132bf6df
2015-03-25 22:50:42 +00:00
Gregory Haynes
f06e3fdb01 Dont try to unmount if were not using tmpfs
I regularly see users report that their build fails because this unmount
line reports an error. Even though we dont bail here because of the ||
true, as a user it is hard to distinguish this from an error.

Change-Id: Ic43f4fb24c53c58329fdf501bba6ba14024ec2aa
2015-03-24 18:30:55 +00:00
Jenkins
8e313b112f Merge "ramdisk-image-create: add support for vmlinux file" 2015-02-04 06:51:51 +00:00
Jenkins
df7ae94df2 Merge "Corrected element-info usage" 2015-02-03 15:50:06 +00:00
jodewey
8fc027227a Corrected element-info usage
Deprecated the `--expand-dependencies` flag from `element-info` usage.
The flag was required and not optional.  We can rely on argparse to exit non-0
when the required positional argument is not provided.

Change-Id: Iaf8eb962eb600760974bc33c30b809a07a23278e
Closes-Bug: 1265649
2015-01-29 13:43:48 -08:00
Michael Kerrin
6337e6fe17 Fix issue with leaking /tmp/image.* directories
When building the ramdisk we don't cleanup the temporary
directories after ourselves. This leaves /tmp/image.* directories
mounted and /tmp/image.* directories on the system.

Also the ramdisk-functions duplicate, from what I can see,
the cleanup function from common-functions. So when a job
is killed off it ends up leaving /tmp/image.* directories
on the system.

Change-Id: I2d73aabd0eb176027b4e7368580db08902e2b6ab
2015-01-28 12:07:07 +00:00
yogananth subramanian
57705694e7 ramdisk-image-create: add support for vmlinux file
ramdisk-image-create assumes the presence of vmlinuz file in OS image.
But ubuntu PPC LE images have only vmlinux file. This patch adds
support for using vmlinux file.

Change-Id: I5dd0b8ceb46b73be57d4c15b39b96f99b524fa3f
Closes-Bug: 1413362
2015-01-22 02:10:39 +05:30
AzherKhan
bb38d0e325 Optimize speed of deletion in find command
I would like to recommend to use + instead of \; in the find
command. As this will ensure the removal of all selected
directories in a single invocation.

Hence improve the speed of deletion.

Change-Id: I409fe11aae217afb6f790491591005c679264ed4
2014-10-27 21:04:09 -07:00
Jenkins
34c273d8df Merge "Make diskimage-builder work in Docker." 2014-10-24 20:13:45 +00:00
Jenkins
2ea4f6ec8a Merge "Avoid overwritting of hooks" 2014-10-20 11:04:07 +00:00
Jenkins
bb07de7fe9 Merge "Enable dracut deploy ramdisks" 2014-10-20 11:03:54 +00:00