To avoid any confusion, commands passed to exec_sudo() should be a
list of "str"s. Log a message if we see unicode issues.
This also adds a debug trace of all output. stderr is captured.
This is modified to raise CalledProcessError on failure, like
check_call(). Calls that are ok to fail will need to explicitly catch
and ignore this.
The two calls that we expect to fail are wrapped
We wish to try rolling back if one of these command raises an
exception. Modify the create handler to initiate rollback on all
exceptions.
Change-Id: Iee4fa41ffaf243e4728bf3a5eeec5c8fa8d2dadc
The await function is essentially a non-standard check_output call.
Let's use standard calls to increase maintainability.
Change-Id: I2c25e1cd7122791fcaa86b46bd801e661471bc9e
As this method can be introduced without any dependency,
provide it on an independent change to simplify reviews.
This is a partial refactor based on
I592c0b1329409307197460cfa8fd69798013f1f8
Change-Id: Idaf3d2b3b3e23d0b9d6bc071d67b961a829ae422
Co-Authored-By: Andreas Florath <andreas@florath.net>
Add the bits of documentation that talk about image creation
at the scope of level 1.
This is a partial refactor of change
I592c0b1329409307197460cfa8fd69798013f1f8
Change-Id: I2619c9ebf3ecfeea67fe9063a169d8324d7ffdf2
Co-Authored-By: Andreas Florath <andreas@florath.net>
This is a partial refactor from change
I592c0b1329409307197460cfa8fd69798013f1f8
Change-Id: I8822e68e41c4ebd47eea9ffed4557efc130a7bf7
Co-Authored-By: Andreas Florath <andreas@florath.net>
Add a new method in the block device library called
exec_sudo, so it can be reused.
This is a partial refactor of change
I592c0b1329409307197460cfa8fd69798013f1f8
Change-Id: Id621f6d029e1275a35c4fd3f19b57c8518076134
Co-Authored-By: Andreas Florath <andreas@florath.net>
As part of the final steps, refactor the bits belonging
to block device and functions. This is a partial refactor
from I3600c6a3d663c697b59d91bd3fbb5e408af345e4
Change-Id: I7aa4fe0466e44846d8fa3194575d446fe4b5b2e6
Co-Authored-By: Andreas Florath <andreas@florath.net>
I think this has gone missing on test nodes during unpuppeting. Add,
as the gentoo job grabs some bz2 tarballs for extraction.
Change-Id: Icb82930702124f7bf112b03a469d90d4163d398b
Introducing the refactors of the block device to allow a tree-like
configuration, and start using it for the partitions level.
Based on patch I3600c6a3d663c697b59d91bd3fbb5e408af345e4
Change-Id: I58bb3c256a1dfd100d29266571c333c2d43334f7
Co-Authored-By: Andreas Florath <andreas@florath.net>
It seems that the redhat nodepool job is quite reliably geting a
"floating point" error during centos image build. This happens after
03-yum-cleanup which is pruning the locales. This might be a
red-herring, since the logs are full of
/bin/bash: warning: setlocale: LC_ALL: cannot change locale (C.UTF-8)
I think in our recent de-puppetisation of hosts, something might have
changed that is setting LC_ALL=C.UTF-8 for the jenkins user, at least
on Ubuntu. This is a problem for centos, as it doesn't have C.UTF-8
locale. I then think using the invalid locale is what leads the the
floating-point error when doing some maths in dib-run-parts to
calculate runtimes.
We are currently overriding LANG, but we really want LC_ALL to ensure
this applies globally.
Change-Id: I8e7cae093c4b32e0d20b73ae0086f14c7cc6a9cb
Add the tree-like vs complete digraph configuration
for images. This is a partial refactor from
I3600c6a3d663c697b59d91bd3fbb5e408af345e4
Change-Id: Ia7a8321e63d59771fe47d8e262b9aacffd60d8d9
Co-Authored-By: Andreas Florath <andreas@florath.net>
Add a new getval call that allows to retrieve values
from the block device. Also isolating the block device
information into a 'blockdev' dictionary entry, to better
return it with the getval command.
This is a refactor from the original code at
I3600c6a3d663c697b59d91bd3fbb5e408af345e4.
Change-Id: I93d33669a3a0ae644eab9f9b955bb5a9470cadeb
Co-Authored-By: Andreas Florath <andreas@florath.net>
The original approach was to pass each and every command
line parameter to the block device. While the block device
functionality gets extended, this is not any longer practical.
Instead of passing in all the parameters separately this patch
collects these in a YAML file that is passed in to the block device
layer.
Change-Id: I9d07593a01441b62632234468ac25a982cf1a9f0
Signed-off-by: Andreas Florath <andreas@florath.net>
Some package updates are more complex and require things like --backtrack=99 to
be passed to emerge. We also try harder to ensure the system is in a consistent
state as a last step.
Change-Id: Ia5d3514e8b2a6cb2d656ade997cebb798d9c0a47
With 8e822768f9 we added the ability to
disable the EPEL repository, however we need yum-utils to use
yum-config-manager.
Change-Id: Iea445f84494fd9a89fd93e9b35f920eb5e55211d
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
Recent changes in the default configuration of cloud-init in Ubuntu
cause warnings when the Ec2 datasource is used on non-Amazon clouds,
see https://bugs.launchpad.net/cloud-init/+bug/1660385
We explicitly select the previous behavior when an Ec2 datasource is
desired.
Change-Id: Iebad8f6c0017fe08013dd5fe667c6132158b71cd
Closes-bug: 1683038
We only need dib-python when we build the image, no need to leak it to
the final product. Remove it in cleanup.d outside the chroot so
nothing can be using it.
Change-Id: I1e229caad7968fb3ab8e44ecdda427e174088d2d
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
If DIB_PYTHON_VERSION is < 3 on the !redhat path, that means we're on
an older platform that may not have python3-virtualenv packages. Skip
install.
Ensure the order of operations happens by forcing the installs
Also add a note about limited platform support (patches welcome :)
Change-Id: I18412767f0ebf946d557a0a126285369e96af159
Recent changes in project-config have shown that we leave the system
in an inconsistent state when installing from source. On fedora, we
will have installed the python2 packages, but then used $DIB_PYTHON to
install python3 pip from source!
This tries to clarify the situation. As described in the document,
with package installs, we just install the $DIB_PYTHON packaged
versions.
Source installs want to take over the global namespace. This is the
price you pay for running the latest versions outside package managers
:) The only sane thing seems to be for us to normalise python2 &
python3 versions of pip, setuptools and virtualenv and then hacking
things such that "/usr/bin/pip" and "/usr/bin/virtalenv" remain
defaulted to python2 versions.
Documentation is added
Change-Id: Ibc6572b89e256d1f48b7fe7c672b8b9524dc704f
Currently we install pip/virtualenv with "/usr/local/bin/dib-python".
This means that every time you create a virtualenv, the python
interpreter inside it is called "dib-python" which is confusing.
Add an env var DIB_PYTHON that points directly the to interpreter
available during build, for use when running scripts.
Change-Id: I88ad3c9eb958d58db4631d9b27bc2c592f970345
This change move "do_extra_package_install" from pre-install to install
phase.
Extra packages are added by user request using the flag "-p", This
package should not be something the elements depend on.
The reason behind this patch is to move the extra package install to
a proper phase, Also more reasonable if base element run package update
to be before we install extra packages.
Change-Id: I68cc773aba9aa01743f0dda9f4e635e4cac2a282
Apt gets confused if it talks to a mirror with an older index than the
index currently cached by apt. This can happen when image builds use a
newer index than the booted image. Avoid these problems entirely by
removing those index caches at the end of image building.
Change-Id: I245d516ee8a44831b2c29612b782bad555c48a3f
Co-Author: Clark Boylan <clark.boylan@gmail.com>
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
This patch removes three nearly-copies of debootstrap documentation
and fixes some documentation aspects.
Change-Id: Ief7794f5c1abad73788c063af6c862472cd34744
Signed-off-by: Andreas Florath <andreas@florath.net>
The current output for package-installs-v2 is inscrutable [1]
The problem starts with process_output() which is not capturing
stderr. This means that any stderr output is dislocated from any
stdout output around it. This is *really* confusing as you get a
bunch of seemingly meaningless stderr output from any calls before you
see any stdout (e.g. in [1] you can see random yum error output that
should have been with the yum call)). The simplest thing to do is to
redirect stderr to stdout which keeps everything in sync.
This causes a slight problem, however, because pkg-map outputs both
status information and errors on stderr. To work around this but
maintain compatibility, we add a "--prefix" argument that prepends
mapped packages from pkg-map with a value we can match on. The
existing status/debug output from pkg-map is low-value; modify the
call so that it will be traced only at higher debug levels (e.g. -x
-x).
The current loop is also calling pkg-map for every package in every
element (this is why in [1] the same message is repeated over and
over). This is unnecessary; it only needs to pkg-map once for each
element, giving the package list as the arguments. Create package
lists by element and pass those to pkg-map.
As a cleanup, there is no point in printing e.output if the
process_output fails for the install because we are already tracing
it; i.e. the output, even for failures, is already in the logs.
Printing it again just duplicates the output.
[2] is an extract showing what I feel is a much more understandable
log output for a fairly complex install.
[1] http://paste.openstack.org/show/595118/
[2] http://paste.openstack.org/show/595303/
Change-Id: Ia74602a5d2db032a476481caec0e45dab013d54f
DIB_PYTHON_EXEC was added in I5fab0e192c3a2dad8f60e821c184479e24e33bcd
to export the python that disk-image-create is running under. If dib
is installed with python3 then just calls "python" (python2) to run
sub-scripts, it fails to find itself.
This has been tested in devstack gates that use py3x
Change-Id: Ia1028972bfc0517b468b279aab9decdbcd7424ca
If the path is missing, unmount_dir currently exits with an error which
unintentionally aborts cleanup efforts early. This change makes
unmount_dir idempotent by exiting successfully if a directory doesn't
exist.
Change-Id: I1491b4344e8569ecb2833f44baee445a89a39d61
The dib-run-parts element was copying our internal version of
dib-run-parts into /usr/local/bin to be used running scripts inside
the target chroot. However, it never cleaned up after itself. This
means all images were left with an unmanaged local install of
dib-run-parts.
This copies dib-run-parts into the hooks directory of the chroot and
runs it from there. It is cleaned up automatically on the exit path.
The dib-run-parts element is no longer required and it has been
removed from all dependencies. It is left with a deprecation notice
in the README. For compatability we convert it to simply install
dib-utils.
Codesearch shows no users depending on this unintentional implicit
install. Note os-refresh-config depends on dib-utils and thus will
have an explicitly installed version.
Partial-Bug: #1673144
Change-Id: Ia2e96c00a4246c04beb96c17f83b8aefb69219ca
It was an oversight during v2 development for dib to start providing
dib-run-parts. The intention was for dib to use a vendored
dib-run-parts directly from $_LIB and have no dependencies on
dib-utils at all. By exporting dib-run-parts, we created an
unintentional conflict with the dib-utils package which provides the
same script.
Tools that depend on dib-utils are unaffected by this
(os-refresh-config).
The only tool that installs diskimage-builder and then assumes
dib-run-parts is available in the path is instack. I have proposed
Ibfe972208df40fa092b11b5419043524c903f1b4 to modify that to use our
internal version.
Change-Id: I149c345d38d761a49b3a6ccc4833482f09f1cd05
