Commit Graph

240 Commits

Author SHA1 Message Date
Andreas Florath
022d93ee82 Unify tidy up logs in lib/img-functions
Cleaning logs was split, some was done in the
img-functions.finalise_base, some was done in the base element.
The version unifies tidy up logs in the lib/img-functions.
Especially when building docker container images the base element
cannot be used. This patch removes about some hundreds KB of
useless logs in cases when the base element is not used.

Change-Id: I165bafb73daf9144c2f3a83930e85e8d8cf5fae3
Signed-off-by: Andreas Florath <andreas@florath.net>
2017-01-14 09:40:44 +00:00
Gregory Haynes
54b2733ad8 Dont run with VIRTUAL_ENV set
debian ships a modified site.py which has some interesting behavior when
VIRTUAL_ENV is set. In this case it will add
/usr/lib/pythonx.x/site-packages to the start of sys.path. This causes
pip to install packages to this location (rather than /usr/local). As a
result, later on when booting where VIRTUAL_ENV is not set this branch
is not hit and the path where python packages were installed is not part
of sys.path.

Change-Id: Icdb769541eee9793f261b4b8ec563be76ee13fe2
2016-12-21 22:40:46 +00:00
Paul Belanger
da41ee6012 Add output image format tgz support
We now support tgz (tar.gz) as an output format.

Change-Id: Iadec92f2f96c3f904f28bd49f87ffc7d48ef7bd7
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2016-12-17 16:41:42 -05:00
Jenkins
389b372976 Merge "Improve checksum performance for images" 2016-11-30 05:07:21 +00:00
Paul Belanger
22952b7ea0 Improve checksum performance for images
Do md5 and sha256 in parallel to speed things up for larger images.

Change-Id: Ib782fe54e4286ba2749a7ab7247f5d41a887a370
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2016-11-30 03:42:24 +00:00
Jenkins
0a453febc6 Merge "Fedora AArch64 (64-bit ARM) support in diskimage-builder" 2016-11-25 12:09:21 +00:00
Ian Wienand
e0c346d479 Turn off tracing around pid/chroot check
In the error case, we get a spew of output as this check goes though
every pid checking if its in the chroot.  Disable tracing around the
call.

Change-Id: Ie84f12974755c0c2c51d7e7697337ed9b32a4a1c
2016-11-18 14:51:20 +11:00
d.marlin
9b4d2a22e4 Fedora AArch64 (64-bit ARM) support in diskimage-builder
Add some checks for AArch64 to avoid the "Unknown architecture" or
"architecture not supported" messages, and allow builds to complete.

Change-Id: I89ba609abaeeb7019eb317cf13473929b2065230
2016-11-16 21:47:26 -05:00
Markos Chandras
a1f57b8cad lib: common-functions: Fix tmpfs umounting
It has been observed that some chroot operations spawn additional
processes which rely on chroot files. More specifically, zypper, uses
gpg-agent to import and validate gpg keys for its repositories. This
gpg-agent process may stay alive for longer which prevents unmounting of
the tmpfs directory since the gpg-agent process still uses libraries etc
which were present in the chroot. We try to solve this by using walking
all the pids in /proc to find out the running processes in the chroot and
kill them gracefully. If that fails for whatever reason, then we simply
keep trying to umount the tmpfs directory before we give up.

The gpg-agent process usually terminates soon after its home directory
disappears but on fast systems we can reach the 'umount tmpfs' point
before gpg-agent terminates by itself. The solution is generic enough so
other 'chroot processes' can also be handled appropriately.

Change-Id: Iccf332678c79266113e76f062884fc5ee79e515d
2016-11-16 15:44:01 +00:00
Paul Belanger
2ea5feca5c
Create (md5|sha256) checksum files for images
In shade, we use both md5 and sha256 checksums to help validate the
integrity of an image. Rather then having nodepool do this each time
for every time, have diskimage-builder create these files when we
build the image.

We've added a flag (disabled by default) to toggle this functionality.

Change-Id: I5815ba69b7d477f1e91dc8ec0c69c86168770964
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2016-10-05 19:34:03 -04:00
Jonas Sticha
224acc6456 Correct order of parameters in call to qemu-img convert
Under some systems this leads to an error if the oder of parameters
does not comply exactly with the way it is specified.

Change-Id: I9561b85985e3f0917f1b4c7801c9048b4e73ae3b
2016-07-21 10:57:40 +02:00
liyuanzhen
5e1fcca299 Fix the bug that "mktemp: failed to create directory"
In the function "run_in_target", it is failed that force an empty
TMPDIR inside the chroot. The TMPDIR is still the user defined
tmp dir. Due to the TMPDIR dir is not exist, using "mktemp" to
create tmp dir is failed.

Change-Id: I898f80099bc2a7c32e8676014d0f4263807f7039
Closes-Bug: #1597569
2016-06-30 06:04:56 +00:00
Jenkins
867bfaa44d Merge "Install docker for tests" 2016-06-29 21:23:41 +00:00
Jenkins
59bb62d1ee Merge "Refactor: remove unused functions" 2016-06-07 01:01:13 +00:00
Jenkins
fb6fa48f85 Merge "Export FS_TYPE and remove hardcoded ext4 values" 2016-06-07 01:01:04 +00:00
Jenkins
13323c9495 Merge "Export die() function" 2016-06-07 01:00:50 +00:00
Andreas Florath
a6a5c385eb Refactor: rename temporary directories
The temporary directories for image creation and building the OS both
started with 'image' as their names followed by some random
characters.  During debugging this is annoying, because on first sight
it is not clear, where which files are stored.

This patch renames them to dib_build.XXXXXXXX and dib_image.XXXXXXXX.

This patch introduces no user-visible change:
the temporary directories are only used during the
run of disk-image-builder.

Change-Id: I249cdb7750fe9a746b375b462789cd9b82681a2e
Signed-off-by: Andreas Florath <andreas@florath.net>
2016-06-03 07:21:03 +02:00
Andreas Florath
b365d3c833 Refactor: remove unused functions
There were a couple of functions which were unused:
ensure_nbd, map_nbd, unmount_qcow_image, mount_qcow_image, ensure_sudo
Because some of them use 'trap ... EXIT' this hinders introducing a
separate exit phase - therefore they are removed.

(It would also be impossible to use them in the current setup, because
they overwrite the 'trap ... EXIT' of the disk-image-creates 'main'.)

Change-Id: If932a557dca9aea4864154ad6c4f286373d6dd7c
Signed-off-by: Andreas Florath <andreas@florath.net>
2016-06-02 21:06:12 +02:00
Ian Wienand
7550d25db0 Export die() function
I realised I'd been using die() in a few places assuming it was
available, but it wasn't exported.  I guess it didn't matter because
whatever was wrong, we were failing anyway :)

This exports the function to make it available to sub-processes, which
should remove the need to source it as done in several places.

Change-Id: I7b9a5a6db406e160099b6ed9fde80455ae227327
2016-05-27 09:25:22 +10:00
Paul Belanger
be521bdec6 Export FS_TYPE and remove hardcoded ext4 values
Export FS_TYPE from img-defaults and use it to remove hard-coded
defaults in the debootstrap mounting.  Also, cleanup the suse element
as it should have access to the exported variable.

Change-Id: Ie9b671ca9336060a5ad294be48aa7eff442bf066
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2016-05-19 17:33:36 +10:00
Ben Kero
9a00bf144f Revert "Revert "Properly fail/trap in eval_run_d"" and fix PIPESTATUS
This reverts commit f07e33a2e9.

This change reverts the revert while fixing the underlying issue --
$PIPESTATUS needed to be encapsulated in ${}s

Change-Id: I1df06ffa7aecf4ea4b8e187dc756e9fc779786bc
2016-05-16 16:25:45 -07:00
Derek Higgins
f07e33a2e9 Revert "Properly fail/trap in eval_run_d"
This reverts commit 0d1d6bec7c.
This patch breaks tripleo-ci (the instack.qcow2 images is failing
to build) and was merged without passing CI.

Closes-Bug: #1582115
Change-Id: Ic4725ad0689c937fb4c8c792e1eaff5f4ea9ada9
2016-05-16 13:16:12 +01:00
Jenkins
e7625214f6 Merge "Properly fail/trap in eval_run_d" 2016-05-16 03:04:56 +00:00
Gregory Haynes
0d1d6bec7c Properly fail/trap in eval_run_d
In phases which are called from eval_run_d (block-device.d) we do not
listen to exit 1's nor do we allow break=after-error. This is because
the run_d function is called in a subshell in order to grab its output.

This also turns on pipefail in the main disk-image-create script.

Change-Id: I88ab2e7104148437eabfe6880e3a1e5ebbb2c15d
2016-05-16 01:37:02 +00:00
Gregory Haynes
edc06a20e5 Install docker for tests
We have some test cases which attempt to build docker images, therefore
we need docker.

Fix a few bugs that showed up when we run docker tests - we need to
docker rm with sudo and docker images don't always have a /tmp so check
before unmounting it.

Change-Id: I147d0ef3f2ea83f35bac568214573a6bde0b1967
2016-05-13 17:07:16 +00:00
SamYaple
5b6716cee8 Use fstrim to prep the block device
This cuts the image size down alot, esspecially if there were lots of
small file deletes.

The fstrim utility is in the util-linux package and should be on
most all systems. fstrim also works with XFS, ext4, btrfs, etc
prodiving the kernel is new enough.

A reduction of 25% or more in size is common.

Change-Id: I269b4416be450369616f9b8e030f84c30e329804
2016-03-13 16:24:59 +00:00
SamYaple
0c32784663 Revert "Zerofree the image if possible"
This reverts commit 5184d02a7c.

The decision was made to go with fstrim because it is faster and more
universal that zerofree. The related-id has the patchset that implements
fstrim.

Related-Id: I269b4416be450369616f9b8e030f84c30e329804
Change-Id: If40cf2fc0ecd8686768cbfeac9ecee90907674e7
2016-03-10 20:38:01 +00:00
Clark Boylan
5184d02a7c Zerofree the image if possible
If the image has an ext filesystem and the zerofree utility is present
on the build system then run zerofree. This should make images as
compressable as possible which is a nice feature when building
compressed qcow2 images.

Change-Id: Ia6062c291f7a3f58b85a4f408ecb3d0574c65d53
2016-03-07 17:45:14 -08:00
Ian Wienand
53c57d75c7 Cleanup unmount_dir function
Cleanup this function to work with a symlinked directory.  Document
it's behaviour more exactly, and add a simple unit-test for it (not
run by default, due to doing things like mounting and unmounting
system dirs on a live system, which doesn't seem safe for CI.  But it
is useful for developers ensuring sanity).

Change-Id: I335316019ef948758392b03e91f9869102a472b9
2016-01-28 14:47:43 +11:00
Matthew Thode
176ae7bff6
Initial add of gentoo support for diskimage-builder
uses upstream's stage4 images, includes all the needed bells and
whistles for openstack on kvm.

Change-Id: Ibca43173c30c2a74a73a2e2d9dd6d6d832c62694
Closes-Bug: 1530911
2016-01-12 19:15:00 -06:00
Gregory Haynes
966f12dc4c Properly account for pipefail during cleanup
Due to a bug in how we were running some of our phases we were not
detecting some nonzero exit's. When this is fixed, dib fails early during
cleanup (leaving some resources attatched) due to not propertly
accounting for pipefail.

Change-Id: Icc0b35acbe035cac12a9291e2d07b6c690c3a6ad
2015-12-31 19:43:55 +00:00
James Arendt
12cb94a75d Fix unmount/remove race in cleanup_build_dir
With a slow file system, umount can return 0 and the immediately
following remove can fail with a "Device or resource busy" error.
This happened in DevStack in disk-image-create where unmount_image
is followed by an immediate cleanup_build_dir.

Solution is to apply same logic from bug 1332521 to allow the
remove to retry on failure (up to 5s) in case the umount has not
completed.

Change-Id: I3337e2b4ad0111e77f79dc179439cdfea8ebdeda
Closes-Bug: #1527721
2015-12-17 20:55:02 -08:00
Jenkins
af2226305d Merge "Prevent overwriting of user modified blacklist.conf" 2015-10-05 18:57:01 +00:00
Nisha Agarwal
d9ce041237 Prevent overwriting of user modified blacklist.conf
The ramdisk-functions overwrites user modified
/etc/modprobe.d/blacklist.conf with a single entry
"blacklist evbug". Due to this, ramdisk fails to recognise
the hardware/disks etc for which user has modified
/etc/modprobe.d/blacklist.conf.
The commit enables copying all the *.conf files under
/etc/modprobe.d to the ramdisk. It creates
a config file /etc/modprobe.d/blacklist-dib-ramdisk.conf
instead of overwriting /etc/modprobe.d/blacklist.conf.

Closes bug: #1492804

Change-Id: Ib88272e4b8f4641c58e7e68bb0c2e4b82efc4fc1
2015-09-30 02:04:15 -07:00
Ian Wienand
494a833987 Add #!/bin/bash to library functions
Currently when these files are opened your editor doesn't know what to
do with them.  Add #!/bin/bash to library functions so that editors,
diff-tools, etc can do syntax highlighting.

There are other ways to skin this cat, such as renaming to ".sh",
adding -* style editor flags, etc.  We had this discussion in DevStack
too, and came to the conclusion the simplest thing that works for
everyone is to just put the #! at the top.

Change-Id: I4cf64321e14844696139f5d40e4d719436390b35
2015-09-16 13:54:07 +10:00
Jenkins
1663c42dab Merge "ramdisk: enable ppc64 support for symlink" 2015-07-16 23:07:12 +00:00
Jenkins
cdf4467a8a Merge "Cleanup the build directories earlier" 2015-07-14 11:50:53 +00:00
Antoine Musso
9bd84d4044 Only chown tmp dirs when they are a tmpfs mount
Temp dirs are created with mktemp and thus belong to the user. There
is no need to chown them unless we used `mount -t tmpfs`.

Move chown under the tmpfs_check conditional.

Change-Id: I37efe18ced3a06d461364dc5cb20600f1527e995
2015-06-24 13:58:31 +00:00
Dan Prince
96b01a7d2a Revert "Remove unused map_nbd function"
This reverts commit ea4a823810.

This function was actually still in use in lib/common-functions
and removing it causes the disk-image-get-kernel /bin
to fail entirely.

Change-Id: Icddb3ca369922a6ea915af8b1b62c434cb1bdf28
Closes-bug: 1464031
2015-06-10 18:37:08 -04:00
yogananth subramanian
f0f945d136 ramdisk: enable ppc64 support for symlink
Patch enables symlink /usr/lib -> ../lib  support for ppc64 Fedora deployment ramdisk.

Change-Id: I01994685292d7c1cae0f9ec1a1052d68420b4b82
Closes-Bug: #1463357
2015-06-09 16:10:13 +05:30
Pino Toscano
987a832351 Cleanup the build directories earlier
Split the cleanup_dirs function in two, i.e. cleanup of the build dir
and the image dir, and use the former to cleanup the temporary build
subdirs after their unmount, before the conversion to other disk
formats; they are not needed anyway at that point, and allows to save
disk space during the conversion phase.

Change-Id: Ie30d7e6033613d6979148423326ae7e17a7342e7
2015-06-04 10:46:18 +02:00
Tristan Cacqueray
356404f229 Follow symlink for elements
This allow custom elements to be added with symlink. Without -follow
a symlinked element is valid but scripts in *.d directory aren't used.

Change-Id: If50b7d9c3b1f6fe278c28488146709efe5cf065f
Closes-Bug: 1461124
2015-06-02 15:10:32 +00:00
Ben Nemec
1b92a11268 Don't log tmpfs message during cleanup
By that point in the build it isn't generally useful, and it causes
confusion when builds fail because people think that's the error.

Change-Id: I26dee4ac0947b71a4a065ef6c5a18103e7df6667
2015-06-01 10:48:48 -05:00
Jenkins
8d5923be11 Merge "Make it clear that tmpfs is optional" 2015-05-28 23:53:32 +00:00
Jenkins
38a3d98099 Merge "Cleanup /tmp in the guest" 2015-05-28 23:38:04 +00:00
Dougal Matthews
a6751c53a0 Make it clear that tmpfs is optional
Given this is often the final output, it can look like an error occured.
Changing the wording makes this clearer.

Change-Id: I70f157054e3120cffee6fa5241b1ffe0b7bfa650
2015-05-27 12:49:38 +01:00
Pino Toscano
37572a548c Cleanup /tmp in the guest
Clean all the content in the /tmp directory of the guest, leaving the
directory itself (usually has special attributes).

Modern distributions usually either setup a tmpfs on /tmp, or clean it
at every boot, so the leftovers will be just few bytes in the generated
image.
Regarding other distributions, a clean /tmp at their first boot will
surely not be a bad idea anyway.

Change-Id: I2b0f8864bc4909542d924f5bd9296dca5d0189f2
2015-05-20 11:24:17 +02:00
Gregory Haynes
b9b6640fa7 Initial element tests
Adding a test function which allows us to use elements to perform
element-specific tests. In order for this to work sanely, also adding
some configuration to our break system so we can assert on negative
tests.

Also adding a test for apt-sources to verify this code actually works.

Change-Id: I378a74255010eca192f5766b653f8a42404be5ea
2015-05-17 02:07:40 +00:00
Pino Toscano
cb37ef5384 ramdisk: switch from lsb_release to $DISTRO_NAME
Instead of executing `lsb_release` to know the current distro being
built, use the $DISTRO_NAME environment variable, already provided by
distribution elements.

Change-Id: I22b67afb481983cc40c198fd408ad5c7f4d68bec
2015-05-12 17:03:46 +02:00
Jenkins
9ba196394c Merge "Remove unused map_nbd function" 2015-05-06 02:50:17 +00:00