Commit Graph

99 Commits

Author SHA1 Message Date
Michael Johnson
8b08d212c3 Fix DIB scripts python version
Now that DIB is python3 only we can remove a hack that made sure
scripts outside the chroot ran with the correct version of python.
This is necessary as python3 does not resolve symbolic links to the
binary like python2.x did, which causes element scripts to fail finding
modules when DIB was run from inside a venv.

This patch does the following:
1. Reverts 9c7b8d1714 which was the
   workaround for mixed python2/3 environments.
2. Updates the scripts to use "python3" instead of "python".

Change-Id: If2402bb02fc8a4778fa9434fa167ea1fafd87c28
2020-07-07 12:53:51 -07:00
Marcin Juszkiewicz
18e35893f7 Do not try to use MBR on AArch64
When I tried to build CentOS8 image for AArch64 I got error saying that
MBR is not supported. So make sure that it will not be used by default.

Change-Id: Ib67ab7f808d727c3c61932c540d398dbe723972f
2020-04-14 10:12:57 +00:00
Zuul
74dff74907 Merge "Set correct python version for non-chroot scripts" 2019-12-12 16:48:17 +00:00
Michael Johnson
9c7b8d1714 Set correct python version for non-chroot scripts
Some phases of diskimage-builder run outside the chroot environment,
such as the extra-data.d scripts, and don't have access to dib-python.
This means these scripts may choose the wrong python version by using
"#!/usr/bin/env python" to execute. The svc-map element is an example.

This patch creates a temporary directory and symbolic link for the
correct version of python, then manipulates the environment PATH
to preference the symbolic link "python" command.
This will allow elements with these scripts to work correctly with
the version of python diskimage-builder is running under.

Change-Id: I289d621e1bfbba0eb174dff977d1a5c92c04e4fa
Co-Authored-By: Ian Wienand <iwienand@redhat.com>
2019-12-11 22:52:28 +00:00
Ian Wienand
801fa77a16 Work around Trusty ext4 metadata_csum errors on Bionic
As described inline, Bionic hosts will build invalid Trusty images.
Hack around this by disabling metadata_csum in the ext4 mkfs.

Change-Id: Ibd67d58ca830a9e60605d0700ee2b17906c804e6
2019-12-11 16:02:13 +11:00
Michele Baldessari
f9dcbd30cc Make sure DIB_DEBUG_TRACE has a default value
After the introduction of 'Add output for mis-configured element
scripts' we started seeing CI failures in tripleo where
instack-undercloud is being used (rocky/queens):

  /usr/lib/python2.7/site-packages/diskimage_builder/lib/dib-run-parts: line 108: DIB_DEBUG_TRACE: unbound variable
  INFO: 2019-12-02 16:24:33,423 -- ############### End stdout/stderr logging ###############
  ERROR: 2019-12-02 16:24:33,423 -- Hook FAILED.

Let's make sure that by default the env variable is set
to 0.

Change-Id: I38c76c0edee436f1e7dd0c9a868cea1e6ee3271d
Closes-Bug: #1854904
2019-12-03 08:26:12 +01:00
Zuul
fdb6225acc Merge "Add output for mis-configured element scripts" 2019-11-22 06:57:40 +00:00
Ian Wienand
bee30b43d2 Only wait for checksum processes
When running under nodepool in a foreground, non-daemonized situation
without a tty (i.e. within a container) we're seeing this "wait" hang
indefinitely.

It is probably related to "outfilter.py" and output file descriptors,
although TBH we haven't completely root-caused it.  I won't claim this
is a great solution, but it should hopefully let the dib process
finish and just die, where outfilter will disappear.

Change-Id: If78da54df3d4c240fee16aee4413ec554b37c1d6
2019-11-21 14:04:09 +11:00
Michael Johnson
824042d608 Add output for mis-configured element scripts
I commonly get asked for help when people are attempting to create
local image elements and they cannot get them to work.
diskimage-builder silently ignores element scripts that it doesn't
find to it's liking, such as non-executable or files with extensions
(.sh is a common mistake).
This patch extends the '-x' tracing flag down to dib-run-parts and
will cause it to print out helpful messages when these files would
otherwise be silently ignored.

Examples:
Ignoring non-executable files: 10-do-not-run-me
Ignoring non-conforming filenames: 10-I-can-run.sh

I am not enabling these by default as they can create extra noise
and require additional filesystem IO to produce.

Change-Id: Ic804efca3015c199440b4b10da951d71a815c64f
2019-11-20 15:05:30 -08:00
Ian Wienand
b52b560fb0 Revert "Drop vhdutil dependency, use qemu-img"
This reverts commit a3e9e7f89e.

We still have some issues with vhd creation on RAX

In short, it appears that images fail to resize unless they have a
specific "creator" field.  Revert this while we consider the options.

[1] https://bugs.launchpad.net/nova/+bug/862653

Change-Id: I2b6a3bfbfe28432fbb6a2ce4a0211939d224b8d5
2019-10-30 09:28:58 +11:00
Ian Wienand
a3e9e7f89e Drop vhdutil dependency, use qemu-img
The vhdutil utility is completely dead; the whole subsystem it relies
on was removed with [1] so it's not even vaguely possible to keep it
up-to-date.

I took the .raw images on a nb and used the qemu-img there (so Xenial)
and generated some VPC images; uploaded them to rackspace and the all
seemed to boot fine.  If there was a problem, maybe it's been fixed on
either the qemu or RAX side in the previous few years.

Thus swith to qemu-img to generate the vhd images too.

[1] https://xenbits.xen.org/gitweb/?p=xen.git;a=commit;h=5c883cf036cf5ab8b1b79390549e2475f7a568dd

Change-Id: I3099d2ebb958370fcec623087a093b2c8dbdc6c4
2019-10-16 16:23:25 +11:00
Zuul
48edd472e7 Merge "Allow configurable gzip binary name" 2019-09-06 09:47:06 +00:00
Logan V
d9e85efd7c Allow configurable gzip binary name
Add a new environment variable $DIB_GZIP_BIN allowing builders to
specify a different gzip (such as pigz) to be used when compressing
tgz images.

Change-Id: Ifb617568140a149e2fda241e07ff8a59429e6697
2019-08-30 17:46:20 +02:00
Ian Wienand
5492843aa8 block-device-efi : expand disk size calculation
As noted in the change, 7fd52ba841
increased the size of the EFI partition considerably.  This has meant
that our padding upwards of the disk size is insufficient and EFI
builds (arm64 in particular) is failing due to out-of-disk errors
during final image operations like installing kernels.

Similar to the discussion we had in
I65fa13a088eecdfe61636678578577ea2cfb3c0c, this feels a bit ugly
because we're mixing logic here with sizes specified in block-device
config files.  But it boils down to the same problem; we are
calculating the disk size here and passing it to the block-layer, so
unless we want to make large changes to the status quo about where
these sizes are calculated, small adjustments here are the most KISS
solution.

Thus we check if we have selected the EFI bootloader element, and thus
assume there will be a large system EFI partition and expand the disk
size accordingly.

Change-Id: Ifa05366c2f2b95259f3312e4dde8c85347075ba1
2019-08-14 15:49:38 +10:00
Andreas Jaeger
9a145cf0d0 Stop regex warning
python 3.6 warns about regexes like:
DeprecationWarning: invalid escape sequence \+

I noticed that debugging a trove job and it really led me in the wrong
way. Fix this with making it a raw string.

Change-Id: I58ee1a49d62316c6c3f0588832c97f659f7e460b
2019-08-08 15:31:52 +02:00
Michael Johnson
5f4b764a11 Remove the rhel 8 check for xfs
This patch removes the check and default for rhel 8 requiring
xfs filesystem as rhel 8 images can successfully be built with
ext4 filesystems.

Change-Id: I1a6bfa26324fd43ae0c77c2c977dda0dd56e26e5
2019-06-12 07:01:36 -07:00
Nir Magnezi
ee46e2f9b7 Add version-less RHEL element for RHEL7 and RHEL8
Make a version-less RHEL element to handle both '7' and '8' DIB_RELEASE.
The element usage should align with other elements which operate in the
same way such as the Fedora element.

Additionally, this patch adds support for RHEL8 that operates with
Python 3.
As of now, users of diskimage-builder will still be able to use the
'rhel7' element, or migrate to 'rhel' and specify their respective
DIB_RELEASE value.

* mount the xfs file-system for extraction as read-only.  vaguely
  based on explaination in [1] and the fact we only read the image
  data into a tar, so can ignore this.

    XFS (dm-1): Superblock has unknown read-only compatible features (0x4) enabled.

* Use the redhat system python as the dib-python version.  dib was
  ahead of it's time making an abstracted python interpreter for
  system work ;) the system python should work for running the various
  dib element scripts.

[1] https://unix.stackexchange.com/questions/247550/unmountable-xfs-filesystem

Redhat-Bug: https://bugzilla.redhat.com/show_bug.cgi?id=1700253
Co-Authored-By: Ian Wienand <iwienand@redhat.com>
Change-Id: I90540675c70bb475d9db2ae24f81c648a31f3f95
2019-05-29 11:28:53 +03:00
Logan V
87a18f51e3 Use megabyte granularity for image extra space
I want to use the new --image-extra-size flag[1] but my use-case
calls for megabyte granularity of this value. Rather than adding
60% to an 800MB image, maybe I only want to add 100 or 200MB, etc.

[1] https://review.opendev.org/#/c/655127/

Change-Id: I8fb9685d60ebb1260d5efcf03c5c23c561c24384
2019-05-15 13:38:25 +10:00
Zuul
fa34eb7fe4 Merge "Support defining the free space in the image" 2019-05-07 10:14:01 +00:00
Zuul
8bf37a064e Merge "Allow specification of filesystem journal size" 2019-05-07 10:14:00 +00:00
Tobias Henkel
778d007150 Support defining the free space in the image
Currently diskimage-builder supports two ways to specify the image
size. One is defining a fixed image size using DIB_IMAGE_SIZE, the
other one is auto-detection while adding a security margin of 60% as
free space. This means when building larger images (e.g. >100GB) with
unknown size upfront we end up with much wasted space, IO and network
traffic when uploading the images to several cloud providers. This can
be optimized by adding a third way by defining DIB_IMAGE_EXTRA_SIZE to
specify the free space in GB. This makes it possible to easily build
images of varying sizes while still minimizing the overhead by keeping
the free space constant to e.g. 1GB.

Change-Id: I114c739d11d0cfe3b8d8abc6df5ff989edfb67f2
2019-04-29 20:18:43 +10:00
Logan V
11142f75b4 Allow specification of filesystem journal size
In many cases, the statically sized 64MB journal is far below the
e2fstools default calculation[0] which calls for a 64MB journal only
on filesystems smaller than 16GB. On bare metal in particular, the
correct default journal size will often be in the 512MB-1GB range.

Since we cannot know what the target system is, this should be a
tunable parameter that the user can set depending on the intended
image usage.

Add a DIB_JOURNAL_SIZE envvar and --mkfs-journal-size parameter
to the image creation so users can override the default journal
size.

[0] https://git.kernel.org/pub/scm/fs/ext2/e2fsprogs.git/tree/lib/ext2fs/mkjournal.c#n333

Change-Id: I65fa13a088eecdfe61636678578577ea2cfb3c0c
2019-04-29 17:00:30 +10:00
Ian Wienand
5284564071 Unmount internal mounts on finalise errors
This is only one line, but it takes a lot to untangle ...  basically
the current "correct" path is:

---
 mk_build_dir()
  -> sets trap trap_cleanup EXIT

 ... stuff ..

 mount_proc_dev_sys
  -> mounts $TMP_MOUNT_PATH/<proc,dev.sysfs>

 pre-finalise.d
 finalise.d

 unmount_image $TMP_BUILD_DIR/mnt # nb == $TMP_MOUNT_PATH
  -> unmount_dir()
   -> recursive unmount everything inside TMP_MOUNT_PATH

 TMP_IMAGE_PATH=$(dib-block-device getval image-path)
 export TMP_IMAGE_PATH

 dib-block-device umount
 dib-block-device cleanup

 ... actually cleanup directories ...
---

Our current failure exit trap does:

---
 dib-block-device umount
 unmount_image
 ...
---

Note this is the *opposite* of what is done in the correct exit path.
In the failure case, if a script fails in the finalise stages it leads
to /proc, /sys, /dev etc. still being mounted inside the image; the
"dib-block-device umount" call doesn't know anything about these
mounts and tries to unmount the parent directory, and we get a hard
failure with a busy mount, and all the mounts are subsequently leaked.

Note that "unmount_dir", which is ultimately called by
"unmount_image", already knows to skip those mounts that
"dib-block-device umount" manages (this is the DIB_MOUNTPOINTS list).
This is further evidence it should be called *before* the
dib-block-device umount.

Change-Id: Ibef3ce9d1167b9c4ff3d5717b113cd3ed374f5e3
2019-03-13 16:38:49 +11:00
Michael Johnson
cfba9ea79d Make sure $TMP_BUILD_DIR/mnt is owned by root
The path $TMP_BUILD_DIR/mnt becomes the / inside the chroot during
the chroot phases of diskimage-builder. Previously this path was being
created using the account running diskimage-builder. This account may
not be valid inside the chroot. This causes path validation, when running
on a Ubuntu bionic host, to fail.
This patch chown's the $TMP_BUILD_DIR/mnt to root.root to make sure
that / is owned by a valid account inside the chroot.

Change-Id: Ifedc136baa67c7952942aed2c8cb1041902fef91
Closes-Bug: 1811113
2019-01-09 20:08:15 -08:00
Zuul
3eab481ab8 Merge "Fix a typo in the help message of disk-image-create" 2018-11-29 11:42:19 +00:00
Ian Wienand
0da1d3a419 Fix unit tests for elements
It looks like we dropped running these probably when we moved the
elements around.  For testtools to find the test scripts we need to
add the __init__.py files to make the directories look like modules.
Also prevent copying any .pyc or cache files in as hooks.

Change-Id: I66d5f6ee62cc4d9ee14c64e819b4db57d035d09f
2018-11-28 11:04:50 +11:00
Pierre Riteau
a64aa0cb47 Fix a typo in the help message of disk-image-create
Change-Id: I092e5ea88747b80c0e59c0aea4301d19009e0241
2018-11-23 09:09:31 +00:00
Zuul
16d5c4280b Merge "Turn on quiet mode when logfile specified" 2018-10-31 00:15:27 +00:00
Ian Wienand
86d5534352 Turn on quiet mode when logfile specified
I'm not really sure why I originally had --logfile also log to stdout
in I202e1cb200bde17f6d7770cf1e2710bbf4cca64c, but it seem
counter-intuitive (indeed, I just tripped myself up thinking that in a
devstack job "--logfile" would put the logs into a separate file and
avoid the stdout logging, and I wrote it!).

Make it so specifying a --logfile puts dib into quiet mode for stdout.
Explicitly overriding DIB_QUIET will allow both if someone wants that.

Change-Id: I3279c9253eee1c9db69c958b87a0ce73efc0be9b
2018-10-24 12:40:09 +11:00
Tobias Henkel
eff5b2312b
Add a post-root.d phase
While trying to get docker image pre-caching to work we couldn't get a
docker daeomon to run within the chrooted environment. However we got
docker running with the help of bwrap outside of the chrooted
environment. The only option so far for this is the block-device.d
phase. But this has the problem that it runs after the image size has
been calculated. This leads to broken builds if the docker images
being pulled are big.

This can be solved by adding a post-root.d phase that runs outside the
chroot but before the image size calculation.

Change-Id: I36c2a81e2d9f5069f18ce5b0d52c5f1c7212c3ae
2018-10-19 10:33:56 +02:00
Zuul
56634e0938 Merge "Turn down pkg-map and hook copy tracing output" 2018-10-18 03:56:12 +00:00
Ian Wienand
6c077d7c2a Turn down pkg-map and hook copy tracing output
This is a lot of very low value noise in the logs as these iterate
through all the elements (often doing nothing).  Turn it down and add
an echo so we just see what elements it is working on.

Change-Id: I0687de4722766189db9d4a7bd7d3cfb45d387b62
2018-10-18 11:03:17 +11:00
Ian Wienand
fadf99af05 Add a pre-finalise.d phase
In exploring Gentoo caching, it was realised that we have no way to
bind mount the cache into the finalised image for the finalise.d
phases.

By adding a pre-finalise.d phase that runs outside the chroot, we can
mount outside things into the hierarchy at $TMP_BUILD_DIR/mnt which
are then seen by the in-chroot finalise.d phase.

This is similar to the pre-install phase

Change-Id: I9d782994843383ddf90f62c40498af9925fd9558
2018-10-15 12:45:23 +11:00
Ian Wienand
ee156e8232 Minor documentation updates
Some minor things after looking at these parts.

The dib-run-parts element doesn't do any of the copying any more, so
these comments are wrong.

The reason for the multiple mounts in the bind mount was non-obvious
to modern eyes (as util-linux has handled this for some time).

Formatting fix for the rst

Change-Id: Idb4c9ff32c49aced2c68a5c905bf7a8b2832a5a2
2018-10-12 11:51:56 +11:00
melissaml
b85bc24761 fix a typo
Change-Id: I4867c39a2995e35a0a1e9cba8f786cf4e6188141
2018-10-05 17:26:04 +08:00
Ian Wienand
8fb2f5cb55 Save and close stdout on exit
Redirecting our output through outfilter.py is inherently a bit racy,
since the disk-image-create process will exit, and then you might get
outfilter.py flushing any remaining output as it closes.

On an interactive prompt this might lead to final output overwriting
the prompt, etc.  This can be a bit confusing when you start running
things in a loop.

If we save the original fd, then on the exit path close the redirected
fd's and wait a little bit for final output (as a result of the
close), we get a more consistent output.

Change-Id: I8efe57ab421c1941e99bdecab62c6e21a87e4584
2018-05-25 11:48:20 +10:00
Ian Wienand
7b98433c46 Reduce path length in PS4 for debug
Strip everything before "site-packages" in the output filename for the
PS4 prompt.  This makes the line in debug logs significantly shorter
as we don't have the full virtualenv path every single time.  The
important thing -- the file being called in the lib/ dir, is retained.

Change-Id: I00706b6f6c0425c7795f997c08ceda3374dc84b5
2018-05-25 11:48:20 +10:00
Ian Wienand
596062b0f9 Use surrogateescape with outfilter.py
When switching to using log-file capture, we're getting

 [gentoo/build-succeeds] outfile.write(ts_line.encode('utf-8'))
 [gentoo/build-succeeds] UnicodeEncodeError: 'utf-8' codec can't
   encode character 'udcc5' in position 59: surrogates not allowed

Use surrogateescape [1] on the output to avoid this

[1] https://www.python.org/dev/peps/pep-0383/

Change-Id: I2c2c537296edfa5a8fe661a41bd5bfb3bfcf57e3
2018-05-25 11:48:14 +10:00
Clark Boylan
f3d58d9042 Don't use -e to test for what might be broken symlink
The behavior of test -e and [[ -e against broken symlinks is to fail
even if the symlink exists. However we want to test if the link exists
or if there is a file in that location. Therefore switch from test -e to
test -L and test -f to check if the file or link exists regardless of
link target validity.

Change-Id: I84a9b6731eccf950707be50aef464a2de1e33e8e
2018-04-06 15:04:56 -07:00
Ian Wienand
e443700b5c Formalise saving of /etc/resolv.conf
systemd-resolved has a new behaviour in bionic, in that if there is no
/etc/resolv.conf file when it installs, it assumes it is a fresh
system and makes /etc/resolf.conf a symlink into its compatability
files.

dib ends up saving & restoring whatever /etc/resolv.conf we have after
the inital chroot creation, which may not be what we want -- in the
above case it restores the system-resolved symlink.  For
openstack-infra, we use unbound and want simply "127.0.0.1" in a
/etc/resolv.conf file [1].

Formalise the ability to save specific contents into the final image.
Add documentation, and a note in the code that it's an external
interface.

I would have preferred to namespace the .ORIG file with DIB_ or
similar, but this unofficial interface has already escaped into the
wild.  Leave it as is for simplicity.

[1] Note that systemd-resolved will obey /etc/resolv.conf as you would
expect, if file exists.

Change-Id: Ie0e97d8072e2b21a54b053fa6fb07b62960c686d
2018-04-04 15:17:45 +10:00
Ian Wienand
855ab0d850 Restore tracing on exit points of block_device_create_config_file
We exit in several places and don't restore tracing.  Previously in
nodepool we relied on the default fallback, which did restore the
tracing.  Since we now use the MBR config file, we take the different
exit path without it and the debugging output is incomplete.

Change-Id: I586fc95517926025705ce376ec5c4aaf4122773f
2018-04-03 14:33:09 +10:00
Ian Wienand
adb0341064 Add block-device defaults
This moves the block-device default out of the "vm" element and into a
selection of other elements.  There's "mbr" which retains the status
quo.  There's an EFI version that has the boot/grub partitions as
required.  In between there's the GPT only version, which is useful
for architectures like power without EFI, but still want possible
larger disks using GPT.

Change-Id: I4a566a97d073fc0dda0ab2494ac988fe015800a9
2018-02-23 10:04:40 +11:00
Andreas Florath
46a07de480 Fix /dev/pts mount options handling
The current implementation - as introduced in
Iee44703297a15b14c715f4bfb7bae67f613aceee - has some shortcomings / bugs,
like:

* the 'grep' check is too sloppy
* when /dev/pts is already mounted multiple times the current implementation
  fails:
  $ mount | grep devpts | sed 's/.*(\(.*\))/\1/'
  rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000
  rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000
  rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000
* code duplication
* Using the undocumented and non-robust output
  of 'mount'.

This patch fixed the above problems.

Change-Id: Ib0c7358772480c56d405659a6a32afd60c311686
Signed-off-by: Andreas Florath <andreas@florath.net>
2017-11-24 07:14:56 +00:00
Markos Chandras
da02f37de1 elements: Respect devpts mount options
This is a continuation for f2cc647dae ("diskimage_builder: lib:
common-functions: Fix options for devpts mount"). We also need to
respect the devpts mount options when the dib elements are mounting
this virtual filesystems themselves.

Change-Id: Iee44703297a15b14c715f4bfb7bae67f613aceee
2017-11-14 08:31:55 +00:00
Zuul
b5b9d98fa0 Merge "diskimage_builder: lib: common-functions: Fix options for devpts mount" 2017-11-09 15:18:44 +00:00
Zuul
bc6c928bb9 Merge "Move to a common lock-file directory" 2017-11-07 17:32:28 +00:00
Markos Chandras
f2cc647dae diskimage_builder: lib: common-functions: Fix options for devpts mount
Commit cebfcf85f9 ("Use -t devpts for
/dev/pts mounts") switched from using '--bind' to '-t devpts' for
mounting the /dev/pts virtual filesystem. However, mounting devpts to
another location also affects the host's /dev/pts mountpoint. Since we
are now mounting devpts without options we end up with the following one
on openSUSE

devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,mode=600,ptmxmode=000)

instead of the one we want

devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)

The missing gid=5 options results to boot problems for virtual machines

So in order to fix that, we need to use the existing devpts options for
/dev/pts so we don't lose them in the new mount.

Change-Id: I17f2c2bb96b807f8dbc07185ae0147bff3230f92
2017-11-06 10:23:03 +02:00
Ian Wienand
b25d0337b8 Move to a common lock-file directory
In a couple of places we use flock for critical sections, but we leave
lockfiles around in various locations which can be confusing.

Introduce DIB_LOCKFILES global (under ~/.cache/dib/lockfiles) and
write lockfiles in there.

Fix up removal of the lockfile in the yum path; we just want to make
sure we cleanup the .rpmmacros file, but we don't need to remove the
lockfile as well.

Co-Authored-By: Andreas Florath <andreas@florath.net>

Change-Id: Ie810b2836be521325afe923708d046112e1e1e20
2017-10-26 16:27:59 +11:00
Andreas Florath
cebfcf85f9 Use -t devpts for /dev/pts mounts
Currently a bind is used when mounting /dev/pts in chroot.
This leads to problems - especially when running DIB in parallel:
It was observed that the /dev/pts mount vanishes from the host
system.

This patch uses '-t devpts' - as it is done for /sys and /proc -
for handling /dev/pts.

Change-Id: Id7775ae6fca6502af800e7b73a00862ef320206b
Signed-off-by: Andreas Florath <andreas@florath.net>
2017-10-23 07:13:02 +00:00
Jenkins
7a76270227 Merge "Add timestamp output filter" 2017-10-09 15:56:31 +00:00