Avoid dangerous file parsing and object serialization libraries.
yaml.load is the obvious function to use but it is dangerous[1]
Because yaml.load return Python object may be dangerous if you
receive a YAML document from an untrusted source such as the
Internet. The function yaml.safe_load limits this ability to
simple Python objects like integers or lists.
In addition, Bandit flags yaml.load() as security risk so replace
all occurrences with yaml.safe_load(). Thus I replace yaml.load()
with yaml.safe_load()
[1]https://security.openstack.org/guidelines/dg_avoid-dangerous-input-parsing-libraries.html
Change-Id: I84640973fd9f45a69d2b21f6d594cd5bf10660a6
Closes-Bug: #1634265
Now that we are explicit about what python version we intend to use
for dib we can have package installs optionally install packages
depending on this. Add a new dib_python_version that matches on the
DIB_PYTHON_VERSION string set by dib-python.
Co-Authored-By: Adam Harwell <flux.adam@gmail.com>
Change-Id: I70659aab7d12924bdb9bc0489a7f02d5fd0dbb39
It seems that on Xenial, it does not take much to confuse "file" and
it's mime guessing such that it thinks some files are not python.
"package-installs-v2" is a good example, since it has an interpreter
"dib-python" that "file" doesn't know about, and no extension. While
looking at this, I've added emacs vars here so it opens in python
mode.
Change-Id: I01994b08c5ad8987925f1eec4062f5b6ee72eb8f
Icf8a075224833fcfbbe2128e8802ff41c39f3c09 looked rather ugly, and it's
easy for us to expand the processing done in the arch list.
Change "arch" to a comma-separated list of architectures that should
match for install.
Add a "not-arch" list which will exclude the package from installation
on those architectures. (An aside -- I considered making it just he
one list with foo,!bar,moo but ! has special meaning in YAML, so it's
easier to have two lists).
$ ARCH=ppc64 package-installs-squash --elements ironic-agent --path=./elements/ /dev/stdout | grep dmidecode
$ ARCH=ppc64 package-installs-squash --elements ironic-agent --path=./elements/ /dev/stdout | grep lshw
"lshw",
$ ARCH=amd64 package-installs-squash --elements ironic-agent --path=./elements/ /dev/stdout | grep lshw
$ ARCH=amd64 package-installs-squash --elements ironic-agent --path=./elements/ /dev/stdout | grep dmidecode
"dmidecode",
Change-Id: Ic69dd02a09e6f3ba9078a2377d8df29871a20db2
subprocess.check_call() returns a byte-string which needs to be turned
into a unicode string for python3 compatability.
Also some minor refactoring while we're here.
Closes-Bug: 1536462
Change-Id: Icd957bc4d93ccad94b1246ad62e6e02ee14d9ca5
Use dib-python to run package-installs using the provided python
version. Automatically detect the python version for our
package-installs-squash since that runs outside the chroot.
Change-Id: I926022bcf8cbcd81b051026ffd5d6477650045ad
subprocess.CalledProcessError in Python 2.6 does not have the 'out'
parameter for __init__, so pass only two of them and manually set
'output' in that case.
Fixes/improves commit 7f410aaff2.
Change-Id: I279bdf433b1272a9c3af4d66a2a52c78a7ac5de2
In some cases, like linux-image-* on debian, we need to only install
packages for a specific target architecture.
Change-Id: Ic0009d0c1e121d6f3f1f21345c544e2d98f080f9
Previously, this code was not checking for the proper environment
variable for an element's installtype. There was a line replacing '-'
with '_' as is required, but that value was not actually used when
searching for the environment variable.
Change-Id: I0bbd56969188389db81844d9276269464870f776
subprocess.check_output() has been introduced in Python 2.7, so the
script will fail when trying to install stuff in guests with Python 2.6
and older (like RHEL 6 / CentOS 6, for example).
Thus gracefully fallback to subprocess.Popen() when
subprocess.check_output() is not available.
Change-Id: I335148397932177810f095a942b993b249991107
Closes-Bug: #1415240
There is a wide variety of tracing options through the various shell
scripts. Some use "set -eux", others explicity set xtrace and others
do nothing. There is a "-x" option to bin/disk-image-create but it
doesn't flow down to the many scripts it calls.
This adds a global integer variable set by disk-image-create
DIB_DEBUG_TRACE. All scripts have a stanza added to detect this and
turn on tracing. Any other tracing methods are rolled into this. So
the standard header is
---
if [ "${DIB_DEBUG_TRACE:-0}" -gt 0 ]; then
set -x
fi
set -eu
set -o pipefail
---
Multiple -x options can be specified to dib-create-image, which
increases the value of DIB_DEBUG_TRACE. If script authors feel their
script should only trace at higher levels, they should modify the
"-gt" value. If they feel it should trace by default, they can modify
the default value also.
Changes to pachset 16 : scripts which currently trace themselves by
default have retained this behaviour with DIB_DEBUG_TRACE defaulting
to "1". This was done by running [1] on patch set 15. See the thread
beginning at [2]
dib-lint is also updated to look for the variable being matched.
[1] https://gist.github.com/ianw/71bbda9e6acc74ccd0fd
[2] http://lists.openstack.org/pipermail/openstack-dev/2014-November/051575.html
Change-Id: I6c5a962260741dcf6f89da9a33b96372a719b7b0
If there are no old-style package-installs and the package-installs
element is included we fail because we call install-packages with no
arguments.
Change-Id: I3c78b51d0a2673ecc581f9faded078974dbc27b7
The latest update to package-install captures both stderr and stdout
from pkg-map, unfortunately, pkg-map has a 'missing-ok' option
which causes it to print an error message on stderr.
The result is that package-install tries to look for packages named
"Missing", "package", "name", etc.
Change-Id: I86b3b71a64b29d533b42fd0cae020e8ecf22cac2
Closes-bug: 1402085
Instead of doing the work in the image of parsing through the element's
package-install declarations, we can squash it on hostside, where we
have both YAML and JSON available to us, and then emit a single
pre-processed file into the target to be used later.
Change-Id: I3f182aa3aae0a79b2f3ea4e66c1878ad12878b0a
We currently support package-installs definitions which has some
limitations and oddities. This new format requires only one definition
which does not reside in our run-parts directories and follows a
consistent naming scheme (package-installs.yaml).
Change-Id: Ie51a7c4fdc15634ae8e069728e5e07cc1dc36095
Our package-installs script fails when installing a package which does
not have a pkg-map but a pkg-map file exist for the element.
Change-Id: I3dab802e23bccfc916efcc1c70c6ce6c4a9ccf67
Packages are often also installed in both the pre-install.d and
post-install.d phases. This patch expands the package-installs element
to support declarative package support for these phases in addition to
the existing support for install.d. The actual install/uninstall logic
is moved to common scripts under bin/ so that it can be reused across
the different phases.
Change-Id: Id51d0bbad232737fc8b5ffaf016dec50cd5b66c9
