I noticed in debugging that with python3 all the pkg-installs output
is preceeded by a b'foo', which suggests coding issues lurking.
The universal_newlines argument makes subprocess readline() returns a
decoded str (via locale) instead of a byte-string. This clears up a
couple of points where we explicitly decode and cleans up the print
output.
Minor formatting cleanup of command and exit display. Don't strip the
leading spaces so indents retain in the output
Change-Id: I2894f10a0c2fc618563641b9d106b716f4a544aa
Avoid dangerous file parsing and object serialization libraries.
yaml.load is the obvious function to use but it is dangerous[1]
Because yaml.load return Python object may be dangerous if you
receive a YAML document from an untrusted source such as the
Internet. The function yaml.safe_load limits this ability to
simple Python objects like integers or lists.
In addition, Bandit flags yaml.load() as security risk so replace
all occurrences with yaml.safe_load(). Thus I replace yaml.load()
with yaml.safe_load()
[1]https://security.openstack.org/guidelines/dg_avoid-dangerous-input-parsing-libraries.html
Change-Id: I84640973fd9f45a69d2b21f6d594cd5bf10660a6
Closes-Bug: #1634265
Now that we are explicit about what python version we intend to use
for dib we can have package installs optionally install packages
depending on this. Add a new dib_python_version that matches on the
DIB_PYTHON_VERSION string set by dib-python.
Co-Authored-By: Adam Harwell <flux.adam@gmail.com>
Change-Id: I70659aab7d12924bdb9bc0489a7f02d5fd0dbb39
It seems that on Xenial, it does not take much to confuse "file" and
it's mime guessing such that it thinks some files are not python.
"package-installs-v2" is a good example, since it has an interpreter
"dib-python" that "file" doesn't know about, and no extension. While
looking at this, I've added emacs vars here so it opens in python
mode.
Change-Id: I01994b08c5ad8987925f1eec4062f5b6ee72eb8f
Icf8a075224833fcfbbe2128e8802ff41c39f3c09 looked rather ugly, and it's
easy for us to expand the processing done in the arch list.
Change "arch" to a comma-separated list of architectures that should
match for install.
Add a "not-arch" list which will exclude the package from installation
on those architectures. (An aside -- I considered making it just he
one list with foo,!bar,moo but ! has special meaning in YAML, so it's
easier to have two lists).
$ ARCH=ppc64 package-installs-squash --elements ironic-agent --path=./elements/ /dev/stdout | grep dmidecode
$ ARCH=ppc64 package-installs-squash --elements ironic-agent --path=./elements/ /dev/stdout | grep lshw
"lshw",
$ ARCH=amd64 package-installs-squash --elements ironic-agent --path=./elements/ /dev/stdout | grep lshw
$ ARCH=amd64 package-installs-squash --elements ironic-agent --path=./elements/ /dev/stdout | grep dmidecode
"dmidecode",
Change-Id: Ic69dd02a09e6f3ba9078a2377d8df29871a20db2
subprocess.check_call() returns a byte-string which needs to be turned
into a unicode string for python3 compatability.
Also some minor refactoring while we're here.
Closes-Bug: 1536462
Change-Id: Icd957bc4d93ccad94b1246ad62e6e02ee14d9ca5
Use dib-python to run package-installs using the provided python
version. Automatically detect the python version for our
package-installs-squash since that runs outside the chroot.
Change-Id: I926022bcf8cbcd81b051026ffd5d6477650045ad
subprocess.CalledProcessError in Python 2.6 does not have the 'out'
parameter for __init__, so pass only two of them and manually set
'output' in that case.
Fixes/improves commit 7f410aaff2.
Change-Id: I279bdf433b1272a9c3af4d66a2a52c78a7ac5de2
In some cases, like linux-image-* on debian, we need to only install
packages for a specific target architecture.
Change-Id: Ic0009d0c1e121d6f3f1f21345c544e2d98f080f9
Previously, this code was not checking for the proper environment
variable for an element's installtype. There was a line replacing '-'
with '_' as is required, but that value was not actually used when
searching for the environment variable.
Change-Id: I0bbd56969188389db81844d9276269464870f776
subprocess.check_output() has been introduced in Python 2.7, so the
script will fail when trying to install stuff in guests with Python 2.6
and older (like RHEL 6 / CentOS 6, for example).
Thus gracefully fallback to subprocess.Popen() when
subprocess.check_output() is not available.
Change-Id: I335148397932177810f095a942b993b249991107
Closes-Bug: #1415240
There is a wide variety of tracing options through the various shell
scripts. Some use "set -eux", others explicity set xtrace and others
do nothing. There is a "-x" option to bin/disk-image-create but it
doesn't flow down to the many scripts it calls.
This adds a global integer variable set by disk-image-create
DIB_DEBUG_TRACE. All scripts have a stanza added to detect this and
turn on tracing. Any other tracing methods are rolled into this. So
the standard header is
---
if [ "${DIB_DEBUG_TRACE:-0}" -gt 0 ]; then
set -x
fi
set -eu
set -o pipefail
---
Multiple -x options can be specified to dib-create-image, which
increases the value of DIB_DEBUG_TRACE. If script authors feel their
script should only trace at higher levels, they should modify the
"-gt" value. If they feel it should trace by default, they can modify
the default value also.
Changes to pachset 16 : scripts which currently trace themselves by
default have retained this behaviour with DIB_DEBUG_TRACE defaulting
to "1". This was done by running [1] on patch set 15. See the thread
beginning at [2]
dib-lint is also updated to look for the variable being matched.
[1] https://gist.github.com/ianw/71bbda9e6acc74ccd0fd
[2] http://lists.openstack.org/pipermail/openstack-dev/2014-November/051575.html
Change-Id: I6c5a962260741dcf6f89da9a33b96372a719b7b0
If there are no old-style package-installs and the package-installs
element is included we fail because we call install-packages with no
arguments.
Change-Id: I3c78b51d0a2673ecc581f9faded078974dbc27b7
The latest update to package-install captures both stderr and stdout
from pkg-map, unfortunately, pkg-map has a 'missing-ok' option
which causes it to print an error message on stderr.
The result is that package-install tries to look for packages named
"Missing", "package", "name", etc.
Change-Id: I86b3b71a64b29d533b42fd0cae020e8ecf22cac2
Closes-bug: 1402085
Instead of doing the work in the image of parsing through the element's
package-install declarations, we can squash it on hostside, where we
have both YAML and JSON available to us, and then emit a single
pre-processed file into the target to be used later.
Change-Id: I3f182aa3aae0a79b2f3ea4e66c1878ad12878b0a
We currently support package-installs definitions which has some
limitations and oddities. This new format requires only one definition
which does not reside in our run-parts directories and follows a
consistent naming scheme (package-installs.yaml).
Change-Id: Ie51a7c4fdc15634ae8e069728e5e07cc1dc36095
Our package-installs script fails when installing a package which does
not have a pkg-map but a pkg-map file exist for the element.
Change-Id: I3dab802e23bccfc916efcc1c70c6ce6c4a9ccf67
Packages are often also installed in both the pre-install.d and
post-install.d phases. This patch expands the package-installs element
to support declarative package support for these phases in addition to
the existing support for install.d. The actual install/uninstall logic
is moved to common scripts under bin/ so that it can be reused across
the different phases.
Change-Id: Id51d0bbad232737fc8b5ffaf016dec50cd5b66c9
