While we already clean a number of things off the ironic-agent
ramdisk, there are a few more significant ones that we should add
to the list.
First is the kernel source. If you're rebuilding your kernel on
the agent ramdisk after the initial image build, then you need to
re-examine your life choices. ;-)
Second is /var/cache. On yum-based distros, this contains a large
number of yum cache files that take up significant space. We don't
really want to be copying around caches when booting a ramdisk
anyway, so cleaning this is the right thing to do regardless.
Third is all *.pyc or *.pyo files. There are a lot of these, so
they eat up significant space and bloat the number of files in the
ramdisk, which makes it take longer to build. the only purpose for
the files is to slightly speed up Python app startup, and we
probably lose more time transferring the files over the network
than we would gain in quicker start times. Note that we were
already trying to remove these, but for some reason I was still
seeing them show up in my final images. It makes more sense to
put them in the same pruning command as all the others anyway.
Fourth is /usr/include. These are files only needed for
compilation. See above for my thoughts on compiling in a ramdisk.
These changes have reduced the agent ramdisk from 391 MB to 333 MB
in my local centos 7 builds, and have reduced the number of files
in the ramdisk by over 18000.
Change-Id: I550f9904b9afd12d48da9ba24559acb23133d076
Fedora 24 has split locales into separate packages. Testing revealed
what is possibly a bug in the choosing of default packages, so add a
small work-around to ensure the minimal locale pack is installed.
This appears to be the only change required for fedora-minimal with
Fedora 24; at least to build with the project-config infra elements.
Change-Id: I64438c34c572ed96211384ae1bfb45b2949e4318
If we're installing pip/virtualenv from source, we need to make sure
we pre-install the packaged versions before the upstream
versions. Otherwise, CI jobs later on that depend on packaged versions
of pip/virtualenv can bring them in and overwrite the upstream
versions we have installed, which leads to a heck of a mess and
usually very confusing failures.
I have also moved in a small hack from system-config:install_puppet.sh
that we found was necessary when using pip versions from upstream.
Note this is not as much of an issue on Debian/Ubuntu, as they keep
their pip packages in a separate place to the system packages, so you
don't have these overwite conflicts as much.
Change-Id: Ib40708c07b939b84661c44df88a5a308fd0c7216
The initramfs file created by the ironic-agent element is
owned by the user running disk-image-create; ensure that
the other files created by the element are also owned by
the user.
Change-Id: I829db5b8e8bf1fc68face9cd2bda52d2a5ccdd4f
Closes-Bug: 1593010
This adds some fixtures to ensure the log output is what we expect,
along with ensuring we test the operation of the debug environment
variables.
Change-Id: I01c36299539b15a633b9307da5a348a5ae4e2563
This adds a common usable logging configuration for DIB, that
can be used by different python tools or elements.
This change is a factored out patch from the block device
refactoring.
Change-Id: I34d44610f74030936e7c0f602340b183baec3f1b
Signed-off-by: Andreas Florath <andreas@florath.net>
The latest Fedora/Ubuntu images don't ship python2 by default, so we
need to use our dib-python wrapper for this so we work in python3 only
environments.
Closes-bug: 1577105
Change-Id: I1048ceef35f269960216066924986eec6117ca00
This element takes up a *huge* part of the logs of openstack-infra
builds as we go and cache every git tree.
This silences most of the noise which will reduce the mess
in the logs considerably.
Note that we've had logging on this turned down since
I91c5e55814ba9840769357261d203f4850e2eba6 but it has been
ineffective in stopping the log-spew output, see the
dependency change.
Change-Id: I60f06f84d57087c82b3907575bff125015d35171
Depends-On: I1e39822f218dc0322e2490a770f3dc867a55802c
The temporary directories for image creation and building the OS both
started with 'image' as their names followed by some random
characters. During debugging this is annoying, because on first sight
it is not clear, where which files are stored.
This patch renames them to dib_build.XXXXXXXX and dib_image.XXXXXXXX.
This patch introduces no user-visible change:
the temporary directories are only used during the
run of disk-image-builder.
Change-Id: I249cdb7750fe9a746b375b462789cd9b82681a2e
Signed-off-by: Andreas Florath <andreas@florath.net>
There were a couple of functions which were unused:
ensure_nbd, map_nbd, unmount_qcow_image, mount_qcow_image, ensure_sudo
Because some of them use 'trap ... EXIT' this hinders introducing a
separate exit phase - therefore they are removed.
(It would also be impossible to use them in the current setup, because
they overwrite the 'trap ... EXIT' of the disk-image-creates 'main'.)
Change-Id: If932a557dca9aea4864154ad6c4f286373d6dd7c
Signed-off-by: Andreas Florath <andreas@florath.net>
It turns out our manual locale cleanup is causing issues (see
I54490b17a7f8b2f977369044fcc6bb49cc13768e). Upon further
investigation, I think this is a better approach than manually
deleting repos.
glibc on Fedora obeys the %_install_langs macro for reducing the
installed locales (as mentioned in the comments, F24 has moved to
having different packages, but worry about that later).
So our existing clear-out is really only required for CentOS, whose
glibc does not have any way to indicate to build less locales.
However, %_install_langs is still correct there, as it restricts some
of the translation files and other things installed with the %lang
macro in spec files.
This is complicated by us having to set this at glibc-common install
time, which happens with the "yum" from outside the chroot (i.e. on
trusty). Since this is too old to have flags to pass this, we need to
fiddle with rpmmacros.
I've tested this with fedora-minimal builds and the locales file is
about 2MiB, which is what it was after the cleanups, and the listed
locales are only those we expect (i.e. it appears to be working).
Change-Id: I528a68beeb7b2ceec25ccbec1900670501608158
Because Fedora 21 is EOL, the appropriate cloud images were
removed from the mirrors during the last days.
Because of the removal, currently all CI tests are failing.
This patch removes the Fedora 21 CI tests.
(Adding and supporting Fedora 23 is done with another patch -
because some additional changes are needed here.)
Change-Id: Ib85bb6fafd4f56ecc55dd420048f4d9e6e6969f3
Signed-off-by: Andreas Florath <andreas@florath.net>
I realised I'd been using die() in a few places assuming it was
available, but it wasn't exported. I guess it didn't matter because
whatever was wrong, we were failing anyway :)
This exports the function to make it available to sub-processes, which
should remove the need to source it as done in several places.
Change-Id: I7b9a5a6db406e160099b6ed9fde80455ae227327
If REPOREF="*", HAS_REF will be used without initialization. As -u is set
the script will terminate with error.
Change-Id: Ic1d88415adfef66dfc6c1d92610a45a9eb6359f3
Signed-off-by: Liu Qing <liuqing@chinac.com>
Allow file test to return all possible mime-types and ensure anything
that matches the python file type is parsed by flake8 instead of
relying on the first match returned.
Closes-Bug: #1585688
Change-Id: Iba31f1853537fe9234ab6f83d66f13dc1c578abb
The "set to localhost by default" behavior for manage_etc_hosts has been
deprecated for more than a year now by change
Ia8582883f737548e2911d3f36a1943e5b236281b.
Setting that value to "localhost" is still possible, but it won't be the
default anymore. If the previous behavior of assigning the hostname and
FQDN to 127.0.0.1 is still desired in some environments, it can be
achieved by setting the DIB_CLOUD_INIT_ETC_HOSTS environment variable,
as the deprecation warning message suggested.
Change-Id: I5a19d46e2f305769a0c89c9d25d2e6be02910221
Add a pkg-map to install cloud-initramfs-growroot on Precise, as it is
required there because the kernel is too old to use growroot on a live
file-system. This was generically removed in
e23b087505.
Change-Id: I5e6cbc4b74dc72a6f23f73a018c028f3ff7c0157
Add ability to filter in pkg-map on release, which defaults to
DIB_RELEASE. As per the examples, release is a more specific target
than distro, but distro still has to match.
Added a debug flag and ability to use a pkg-map directly for
debugging/development.
Change-Id: Ie282f96966e46236b06bc276de0168fc7a66c5da
Export FS_TYPE from img-defaults and use it to remove hard-coded
defaults in the debootstrap mounting. Also, cleanup the suse element
as it should have access to the exported variable.
Change-Id: Ie9b671ca9336060a5ad294be48aa7eff442bf066
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
For something fairly simple, I went back-and-forward with this a bit.
Firstly, I realise calling readlink constantly sucks. Due to the way
we call dib and source various files, you end up with the source-file
from "caller" being usually a very ugly path including levels of "../"
indirection. Cleaning this up to something canonical is the only sane
way to present it.
Because we evaluate _ps4() from a sub-shell in the PS4 string, there's
no way for it to do something like build a global in-memory cache in
an associative array or similar. It could write out a temp file or
some other side-band method, but the overheads of managing this don't
seem any different to just calling readlink. If anyone can think of a
bash-hack around this that doesn't involve a fork() I'm interested.
We could potentially strip some of the leading paths in the assumption
you know what they are; but it gets complex when things are split
across /usr/bin & /usr/lib and external elements, etc. I thought
about arbitrarily shortening it (e.g. just take last 20 characters)
which gives you enough of an idea of the file, but looks a bit ugly.
Or we could just leave the file-name out all together and assume the
function name is unique enough; this also seemed a bit ugly.
Obviously it's a matter of taste in the output. It is certainly
wider, but it also adds a lot of information. It also makes it fairly
clear where there are things we can make less verbose,
e.g. I1e39822f218dc0322e2490a770f3dc867a55802c disables tracing in
run-parts which is just noise. There's a few other frequently used
loops that we could disable tracing for by default to benefit
signal:noise.
tl;dr : take a look at the logs. I think it is a step in the right
direction of making the logs more usable for debugging.
Change-Id: I8054a3050415fcb527baeb7012bf133e5c864bf3
This reverts commit f07e33a2e9.
This change reverts the revert while fixing the underlying issue --
$PIPESTATUS needed to be encapsulated in ${}s
Change-Id: I1df06ffa7aecf4ea4b8e187dc756e9fc779786bc
