As motivation for this; we have had two breakouts of dib in recent
memory. One was a failure to unmount through symlinks in the core
code (I335316019ef948758392b03e91f9869102a472b9) and the other was
removing host keys on the build-system
(Ib01d71ff9415a0ae04d963f6e380aab9ac2260ce).
For the most part, dib runs unprivileged. Bits of the core code are
hopefully well tested (modulo bugs like the first one!). We give free
reign inside the chroot (although there is still some potential there
for adverse external affects via bind mounts). Where we could be a
bit safer (and could have prevented at least the second of these
breakouts) is with some better checking that the "sudo" calls
*outside* the chroot at least looked sane.
This adds a basic check that we're using chroot or image paths when
calling sudo in those parts of elements that run *outside* the chroot.
Various files are updated to accomodate this check; mostly by just
ignoring it for existing code (I have not audited these calls).
Nobody is pretending this type of checking makes dib magically safe,
or removes the issues with it needing to do things as root during the
build. But this can help find egregious errors like the key removal.
Change-Id: I161a5aea1d29dcdc7236f70d372c53246ec73749
In certain cases, with packages cached that need an upgrade,
when performing that action the system hangs waiting for
a prompt.
Add force-confdef option, that will let dpkg overwrite
configuration packages that were not modified. In combination
with already existing force-confold flag, it will allow
to avoid any conffile prompt.
Change-Id: Ifb177f9ac2c9ad29f8b92309c5b8cfe8e60a4e14
The Ubuntu Xenial cloud server images set the mode of
/var/lib/apt/lists/partial to 700, so when mounted it's inaccessible to
an unprivileged user, resulting in an error:
find: `/tmp/image.aDQKdkRi/mnt/var/lib/apt/lists/partial': Permission denied
There's no reason an image should come with anything already in
/var/lib/apt/lists/partial, so just avoid trying to descend into that
directory when fixing the apt translations packages.
Change-Id: Id27f0166bfb09d67200f337a5ffff2f2037b7c1c
The wrong APT config name is used to disable download of translations.
It's Acquire::Languages, not APT::Acquire::Languages.
Change-Id: Ie0c12d444bab19b4486845944ef51031e9133470
Closes-bug: #1436523
This allows them to install packages already specified in .yaml files
for them, which are not installed otherwise since the migration to .yaml
files.
Change-Id: Iac8c7eb55116938616e5299b84487cd52e0cbf6f
There is a wide variety of tracing options through the various shell
scripts. Some use "set -eux", others explicity set xtrace and others
do nothing. There is a "-x" option to bin/disk-image-create but it
doesn't flow down to the many scripts it calls.
This adds a global integer variable set by disk-image-create
DIB_DEBUG_TRACE. All scripts have a stanza added to detect this and
turn on tracing. Any other tracing methods are rolled into this. So
the standard header is
---
if [ "${DIB_DEBUG_TRACE:-0}" -gt 0 ]; then
set -x
fi
set -eu
set -o pipefail
---
Multiple -x options can be specified to dib-create-image, which
increases the value of DIB_DEBUG_TRACE. If script authors feel their
script should only trace at higher levels, they should modify the
"-gt" value. If they feel it should trace by default, they can modify
the default value also.
Changes to pachset 16 : scripts which currently trace themselves by
default have retained this behaviour with DIB_DEBUG_TRACE defaulting
to "1". This was done by running [1] on patch set 15. See the thread
beginning at [2]
dib-lint is also updated to look for the variable being matched.
[1] https://gist.github.com/ianw/71bbda9e6acc74ccd0fd
[2] http://lists.openstack.org/pipermail/openstack-dev/2014-November/051575.html
Change-Id: I6c5a962260741dcf6f89da9a33b96372a719b7b0
The dpkg element was hardcoding ~/.cache/image-create as the cache path,
but elements should use DIB_IMAGE_CACHE as the root for the cache to
allow users to override the path. Replace the hardcoded path with one
that is rooted at $DIB_IMAGE_CACHE.
Change-Id: I76030654e7dfc7b7d4dfa87e082c14e7cee479b9
The docker build folks obsess about image size and speed a bit. Grab a
few of their optimizations from their debootstrap build process and
apply them to ours.
Change-Id: Ic3d3565423b0ae090896d99fd3bf1145eca6303d
I would like to recommend to use + instead of \; in the find
command. As this will ensure the removal of all selected
directories in a single invocation.
Hence improve the speed of deletion.
Change-Id: I409fe11aae217afb6f790491591005c679264ed4
As some finalise steps can install packages we need to generate the
dpkg manifest after that has occurred.
Change-Id: I2177db2e64d4d9c21deeac7cf017919888a2d524
Grub installation happens in finalize.d so cleanup should be done after this.
This reapplies I6322b8b529e31fd1b7251dd9e07cc85f442f7e81 after the revert,
but working.
Change-Id: Ie1873e64d407552da37a8dbedc13c3adbf79c085
This calls chroot as a non-root user, which will always fail with
EPERM.
This reverts commit ab2d1a31f2.
Change-Id: Ie674fef694ad66e1ebc22083dec2a0bc34371e7b
After running through pkg-map we could have no packages to install,
if so, don't attempt to run a malformed command.
Change-Id: Ia8e0aed62bcf814bf85c86b54ff0837da49ae7dd
Updates the dpkg element's bin/install-packages script
so that if supports using pkg-map for package mapping.
To make use of the new change simply add the -m <element name>
to allow install-packages to know which element namespace to use
when installing packages.
Use of the new -m option is off by default.
As part of this change we also updated install-packages
to use getopt for in script argument parsing.
Change-Id: Idfc40f2d75828a0f09d227f0332ccef8f0183efc
While an end user would never use these elements at the same time we
should always use unique names for things per DIB conventions.
Change-Id: I53b10522183e8566c62922b91878e5cf767fe2c6
With this patch, /var/cache/apt/archives directory content is preserved.
The directory is actually a bind mount of the
~/.cache/image-create/apt/$DISTRO_NAME directory, much like what we do
for ccache.
You can use DIB_APT_LOCAL_CACHE=0 to disable this behavior.
This trivial change improve performance A LOT (>30%), even if a local
HTTP proxy because:
- there is no need to copy again and again the same files
- we avoid network latency
The patch has been tested with Debian and Ubuntu with every elements
from the tripleo-image-elements repository, the final size of the cache
directory is about 700MB per distribution subdirectory.
Change-Id: I4fab499493f734c7c546d4d23b1a98f0e7523a39
As with the previous similar changes, this is intended to catch
problems as they happen instead of ignoring them and continuing on
to potentially fail later. Setting this on all existing scripts
will allow us to enforce use via Jenkins.
Change-Id: Iad2d490c86dceab148ea9ab08f457c49a5d5352e
Using set -e in all of our scripts will prevent some subtle bugs
from slipping in, and will allow us to enforce use of set -e with
tooling.
This change also adds -u and set -o pipefail in the less complex
scripts where it is unlikely to cause problems. A follow-up change
will enable those options in the complex scripts so that if it
breaks something it can be reverted easily.
Change-Id: I0ad358ccb98da7277a0ee2e9ce8fda98438675eb
The phrase is no longer needed as of August 23, 2000 with Nicaragua's
joining of the Berne Convention.
Additionally, in at least one instance,
elements/cache-url/bin/cache-url, its existence in the file between
Copyright lines is just weird and feels misleading, even though it is
not.
Remove all of the lines, because sanity.
Change-Id: I24fd76c2b4f66b8036010b5079db39ead729abee
Fixes problems found by set -eu and pipefail, including:
-Many unset variables
-Commands that can fail under normal circumstances, which breaks
with set -e. This change swallows those expected errors to allow
our existing error code to handle them.
-The dkms element was not finding Fedora kernel versions correctly.
This may be an issue for other distros too, but since Fedora was
working fine without this functionality I only changed it to print
a warning message rather than failing the build when it happens.
-The ramdisk init script will not be set -eu because if it fails
the result is a kernel panic, which can be tricky to debug.
However, in testing with set -e a few failing commands were found
and have been fixed in this patch.
Change-Id: I44cf98dfc80cfcaec54b88cc83be80a3dbf2cec3
* Create a standard element to copy manifests out of the image and
into the build area.
* Ensure all current manifests are copied into a standard manifests
directory
Change-Id: I37eff5a13a14564e1adc33eb4f0144d571267617
Adds an environment variable DIB_ADD_APT_KEYS pointing to a folder of
apt repo GPG keys. Each key will be verified and installed with apt-key
to the apt keyring.
Change-Id: I4b15347a76aa3241e741931bb94f17598641ae42
To create an auditable log of packages installed in an image use
dpkg-query to create a json data file with the package names, version
and install status.
Change-Id: I59018221edcb944a876ad47b402538c63c426bc7
Currently the dpkg element fakes start-stop-daemon, initctl and
invoke-rc.d to stop dpkg from starting a bunch of daemons in the
chroot. This is problematic when packages use service, start, stop or
restart commands.
This patch uses a policy-rc.d instead of faking invoke-rc.d to achieve
the same thing. This approach now aligns exactly with debootstrap.
Without this patch DIB runs on some debian distributions fail
when trying to umount the chroot loop device as there are daemons
running.
The log will now show "invoke-rc.d: policy-rc.d denied execution of
start." instead of "Warning: Fake invoke-rc.s called, doing nothing."
Change-Id: I6db192127aca19b5b7915179b781f5192078bfc7
Related-Bug: #1211564
install-packages is an executable script, it should be +x. The install
command that was installing it is setting permissions to 755, so we were
fine there. However, I was trying to test the script outside of an
image build to test a change, and it's helpful to have it +x in the
repository for that context like all the other scripts under bin.
Change-Id: Ic300cc56d463640d119db4e310d375c66ed133ca
Since we are using bash syntax in some of the element fragments,
we should make sure we use bash for all of them, so that things don't
break on systems where /bin/sh != /bin/bash.
Change-Id: If2f043c57aa4e1492b7f9839213ef6123f683612
Check if Upstart exists before blocking the daemon. Not all dpkg based
OS's have Upstart installed so this should help increase compatibility.
Closes-Bug: #1251949
Change-Id: I2dcb1ff3641778b5653ca5762a728398adb57da1
Post install scripts are useful because they can perform tasks you want
to handle after the OS/application install but before the first boot
of the image. e.g: Clean the cache left by the package manager reducing
the size of the image.
Change-Id: I03e77f602192bbdce29c02999d1b57fac8051ddc
Fixes: Bug #1145786
ARM doesn't have a generic Linux image due to the soc-specific nature of Linux
kernels today, so we drop the manual installation of that package, replacing it
with a dist-upgrade instead. This involved tweaks to the dpkg and fedora
install-package scripts.
Change-Id: I97924b80ca87781307e1087b9fe4b18215770e84
This includes the install-packages implementation for dpkg, apt http proxy
config, daemon blocking and unblocking.
Change-Id: I8f159021d2b223d7003cec067de3aa605ad06974
