I noticed in debugging that with python3 all the pkg-installs output
is preceeded by a b'foo', which suggests coding issues lurking.
The universal_newlines argument makes subprocess readline() returns a
decoded str (via locale) instead of a byte-string. This clears up a
couple of points where we explicitly decode and cleans up the print
output.
Minor formatting cleanup of command and exit display. Don't strip the
leading spaces so indents retain in the output
Change-Id: I2894f10a0c2fc618563641b9d106b716f4a544aa
DIB automatically includes it's base elements in the ELEMENT_PATH;
this double-set means the base elements are in the path twice, leading
to some tools that walk the element list (package-installs-squash, for
example) to do things twice.
Change-Id: Icf631427601cfd2abed436711cedb9e002fff8f9
There's a few things going on here
Firstly, we need to install coverage tool in test-requirements
Secondly, .testr.conf has to use PYTHON because the coverage report
works by resetting PYTHON='coverage run ...'
Thirdly, because we call ourselves diskimage-builder but the python
module is diskimage_builder that seems to confuse things. We need to
use "setup.py test" (note, that is different to "setup.py testr"!) to
use the PBR testr wrapper. That exposes a --coverage-package-name
argument that calls the coverage tool with the right argument.
With this I got a coverage report for our unit tests
Change-Id: I9012e18eb7d01bee035140e70afa76c47c27eb01
Per the bug report, %I results in -'s in the interface name being
replaced with /'s, which means when we try to look up the interface
by path in dhcp-all-interfaces.sh we end up at an invalid path.
Using %i instead should fix the problem.
See https://www.freedesktop.org/software/systemd/man/systemd.unit.html
Essentially what is happening is that we start with a name like
br-ex that wasn't escaped in the first place. However, because of
the - it looks like it could have been escaped to systemd. When
we use %I, which tries to unescape escaped values, it unescapes a
value that was already not escaped.
Change-Id: I434ed2e084d4477dc7a2b7827164586f8ea0c1e3
Closes-Bug: 1649409
We can use ${DIB_DEBIAN_COMPONENTS} to get this information. We also
already set the mirror.
Change-Id: Idd11c2b7df1d247c6d32a5f936b8601b4741b519
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
Currently we run grub-mkconfig then go and edit the generated output
in /boot/grub/grub.cfg to override the "root=" arguments to our label.
If another element like project-config's
finalise.d/99-fix-grub-timeout then goes and re-runs grub2-install, it
overwrites these changes.
Grub has GRUB_DEVICE which should override the root device. Let's try
using this. Maybe it wasn't around at the time, this code is
more-or-less unchanged from I7d83bb2b359e7a8c3858eca04c96e35cf4e1fe9e
Change-Id: Ibaaa81124098f3c6febe48e455d3e1cd0a5f1761
Avoid dangerous file parsing and object serialization libraries.
yaml.load is the obvious function to use but it is dangerous[1]
Because yaml.load return Python object may be dangerous if you
receive a YAML document from an untrusted source such as the
Internet. The function yaml.safe_load limits this ability to
simple Python objects like integers or lists.
In addition, Bandit flags yaml.load() as security risk so replace
all occurrences with yaml.safe_load(). Thus I replace yaml.load()
with yaml.safe_load()
[1]https://security.openstack.org/guidelines/dg_avoid-dangerous-input-parsing-libraries.html
Change-Id: I84640973fd9f45a69d2b21f6d594cd5bf10660a6
Closes-Bug: #1634265
Cleaning logs was split, some was done in the
img-functions.finalise_base, some was done in the base element.
The version unifies tidy up logs in the lib/img-functions.
Especially when building docker container images the base element
cannot be used. This patch removes about some hundreds KB of
useless logs in cases when the base element is not used.
Change-Id: I165bafb73daf9144c2f3a83930e85e8d8cf5fae3
Signed-off-by: Andreas Florath <andreas@florath.net>
As described in the bug, there are conditions with certain switches
in which the interface is 'admin down'ed during initialization.
Doing a 'cat' on /sys/class/net/<interface>/carrier when it is
'admin down'ed produces an 'Invalid Argument' error and the script
terminates. What this fix does is ignore failures of the 'cat'
operation (by '|| echo 0') and place the link up inside the retry
loop.
Change-Id: I4f098aa5078b8482681394a3e9a6b17ed4bd4451
Closes-Bug: 1654046
Xenial's bind of /dev into the chroot includes /dev/shm which is in
use by the host. An alternitive fix for this would be to use rbind
to recursivly bind mount /dev instead of just the base bind of /dev
Change-Id: I2c0f70afd1e82dd52a522f0dd2b3ea618b30b6c6
As noted in the bug, there may be circumstances where a longer
timeout than the current default is needed. This patch allows users
to tune this timeout for their environment if need be.
Change-Id: I173f3dad684894fbc3c27dece5ae15b5f63bae5a
Closes-Bug: 1654027
When we configure dhcp interfaces before network.target has run,
network.target will try to bring up those interfaces a second time
after our service does so. This causes two issues - first, the
network target will always fail because it can't bring up an
interface that is already up, and second, when configuring interfaces
that don't actually have an available DHCP server it will result in
a five minute delay waiting for DHCP on those interfaces. This will
also cause the network target to fail and is an unnecessary delay.
By moving the dhcp-interface service to run after the network
target we avoid both of these problems. network.target will still
bring up the interfaces on subsequent boots. This could result in
the five minute delay happening on reboots, but the expected use
case for interfaces without DHCP is that they would be configured
statically on initial deployment so this should be a minor issue.
The dhcp-interface service is also configured to run before the
network-online target so that services which depend on the network
actually being available will not race the DHCP process.
A snippet from /var/log/messages on a node with this patch applied
is included in the bug to demonstrate the behavior described above.
Change-Id: I5cfabf20f920beea52abf4c42362b6f6ac0b37c4
Closes-Bug: 1653812
The components documentation was previously referring to the
ramdisk image for deployment, which was previously deprecated.
Corrected to point to the ironic-agent element.
Change-Id: I770460041eb13523896aaadb7705bdc3db1a54ca
We landed the fix for this in
Icdb769541eee9793f261b4b8ec563be76ee13fe2.
This reverts commit 2978ff885b.
Change-Id: Iecfc41ab2aad57bc4f6f86a13810b534d19a8fd5
debian ships a modified site.py which has some interesting behavior when
VIRTUAL_ENV is set. In this case it will add
/usr/lib/pythonx.x/site-packages to the start of sys.path. This causes
pip to install packages to this location (rather than /usr/local). As a
result, later on when booting where VIRTUAL_ENV is not set this branch
is not hit and the path where python packages were installed is not part
of sys.path.
Change-Id: Icdb769541eee9793f261b4b8ec563be76ee13fe2
When using up to date distributions for dib development, pep8
installs using python3. This patch fixes the problem, that
not the complete dib-lint (which is called) is compatible
with python3.
Change-Id: I417d03746edb4d34011b997edf8b5b9662ea6f09
Signed-off-by: Andreas Florath <andreas@florath.net>
There are issues with pip packages and a python3 only Xenial systems.
This is occuring after Ie609de51cc5fcde701296c9474e315981d9778a2.
We believe the issue is with VIRTUAL_ENV being set within the chroot
and messing up pip installs
(Icdb769541eee9793f261b4b8ec563be76ee13fe2) but a full solution is not
yet clear.
For now, set Xenial to ensure we use python2. Install the package for
the ubuntu element (75-debian-minimal-baseinstall will install python2
for the minimal elements).
Change-Id: Id403919b0af93b375a900186c01a0d3a3bdfafea
On Debian network configuration can be done via /etc/network/interfaces.
It can accept a statement to load additional files, which varied in
history:
Wheezy only supports 'source' (see b822581)
Jessie supports 'source-directory' and comes with the statement by
default.
However since 754dd05 we inconditionally inject 'source', thus on Jessie
the configuration ends up with:
source-directory /etc/network/interfaces.d/*
source /etc/network/interfaces.d/*
When networking is started, 'ifup -a' parses the list of interfaces
twice. When configured with dhcp, that causes two dhclient to spawn
which might conflict with each other.
Inject the source statement only if there is neither a source or
source-directory with the same path.
Change-Id: Iefa9c9584f676e50481c621b4111eded3125a50b
Because we run this image in openstack-infra, we want to increase our
test coverage to help avoid potential breaks to our CI systems.
Change-Id: I26405e3f7465654075278ec35b5e0da1338bb45e
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
Since we still run these 3 version of ubuntu-minimal elements in
openstack-infra, also run functional testing for them.
Trusty and xenial will be in voting gate, precise added as skipped for
non-voting.
Add the default skip/run status to the "-l" output just to confirm
this too.
Change-Id: Icfbfd0cb7d9acae824972474b77e2fe0486c4f69
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
Every run we are doing a full tar.gz of the chroot environment that
never gets used.
It's not suitable for CI since we use fresh images each time there.
The cache in general isn't really isn't a very safe thing to have
around, because there's no invalidation procedure and no real way to
make one -- we've no guarantee that a new chroot build even moments
after a previous one wouldn't bring in or different packages, etc (of
course this is *unlikely*, but the longer you go between builds the
worse the problem becomes. Also, tons of packages get installed after
this not from any cache, so potential speed-up is rather marginal.
Debian turned this off with I58fc485aacacaa17243bf9ce760ed91256d1f182.
However, given the reasons above and it's complete lack of testing, I
don't see this as useful.
If we really want this type of thing, I think we should come up with a
way to use a persistent external yum/dnf cache that yum/dnf keeps in
sync with it's usual invalidation rules.
Change-Id: I66789c35db75c41bc45ea1ad2e26f87456de4e4d
Set the grub timeout to 5 seconds by default, and add notes on how to
update this. This will stop infra having to carry an element that
goes and rewrites the grub configuration.
Change-Id: I556b3f48eff1b67ee8c4b9b64f749af95100fb99
