It seems libmagic changed it's Python output to
"text/x-script.python", which I see on Fedora. Handle this too so we
detect Python files correctly.
[1] eb373e431c
Change-Id: I35992c70523a8f2bc5efff2e5167ed1ac1514d34
This reverts commit 6ee2995214 and
e85c2a6f03.
I missed that if you pip install and then run dib-lint, it's not going
to pick up the .yamllint file shipped here. Thus it gives spurious
errors.
The reason for this was simply better duplicate key detection in yaml
files, which caused us problems with the kernel installs. However, at
this point it seems just the old "does it load" test from pyyaml will
be enough.
Change-Id: I87a9fc9bb119cfeffad48fc0fa0df31f0181825d
This gives us better linting of YAML files that just opening them.
This would have detected the duplicate keys in
I34e27d821fbefe274e7b007f37b0bd34db2e1d26.
The .yamllint is taken from zuul-jobs where it is also used as a
fairly sane set of default rules.
A few minor newline fixes are added.
Change-Id: I96d6644ae24f7deb84fa50fefbda0f0d33e0e009
Add a basic test to ensure that all elements have a README.rst file.
This way they will be exhaustively listed in the Sphinx documentation.
Add dummy README.rst for 'disable-selinux' and 'rpm-distro' elements.
Change-Id: Ia5252ddd89b5ae5c6e9a12a66ef10f912fd54da5
Because we read this in via a bash loop, without a trailing newline we
can hit one of the oldest bash gotcha tricks and end up skipping the
final line (or only line, if there's only one) when the description
files don't have trailing newlines. Add a check.
Change-Id: I23e293b957cd4f008611656cf9166391b1b537a2
When running dib-lint vi temp files .*.sw? files are included which leads to
false positives. ALso all editor files are checked when looking at indents,
again this results in false positives.
Exclude those files by checking if they're in the user/project's
.gitignore setup.
Change-Id: I0a48174f22a8dad9e8f15bf3f70835d021a2d46f
Instead, either use the bash built-in of type to ensure it exists. Since
which is an external dep, things can fail oddly in a constrained
environment.
Also add a dib-lint test for this.
Change-Id: I645029f5b5bfe1198c89ce10fd3246be8636e8af
Signed-off-by: Jesse Keating <omgjlk@us.ibm.com>
Prior to version 2.0.0 of diskimage-builder, the elements dir was hard
coded as 'elements' in dib-lint which allowed dib-lint to be consumed by
other projects (instack-undercloud). The 2.0.0 release has changed this
path and it is not overridable so that the other projects can still use
dib-lint. This change makes ELEMENTS_DIR and LIB_DIR check for an
existing environment variable so that dib-lint can be consumed outside
of the diskimage-builder project.
Change-Id: Iae1dade1e20a950e63ffe8dacfea0f23cf9923a0
Related-Bug: #1672730
Scripts have moved from bin/ to diskimage_builder/lib (and are not
executable, since they are called from dib). Add a match so we run
checkers over those files.
Clarify the situation with .py files while we're here.
Change-Id: I8a1ef4b00a185b83c8b1f29c563b85e78bc233ca
Avoid dangerous file parsing and object serialization libraries.
yaml.load is the obvious function to use but it is dangerous[1]
Because yaml.load return Python object may be dangerous if you
receive a YAML document from an untrusted source such as the
Internet. The function yaml.safe_load limits this ability to
simple Python objects like integers or lists.
In addition, Bandit flags yaml.load() as security risk so replace
all occurrences with yaml.safe_load(). Thus I replace yaml.load()
with yaml.safe_load()
[1]https://security.openstack.org/guidelines/dg_avoid-dangerous-input-parsing-libraries.html
Change-Id: I84640973fd9f45a69d2b21f6d594cd5bf10660a6
Closes-Bug: #1634265
When using up to date distributions for dib development, pep8
installs using python3. This patch fixes the problem, that
not the complete dib-lint (which is called) is compatible
with python3.
Change-Id: I417d03746edb4d34011b997edf8b5b9662ea6f09
Signed-off-by: Andreas Florath <andreas@florath.net>
Files in $element/environment.d are meant to be sourced, so drop
the executable bit. Moreover, drop the executable bit from a couple
of other scripts that are either meant to be sourced or simply because
they are configuration files.
Change-Id: I7f724dd9d409f4a835a136f12f48a84aa9acc41e
When you source a file that just does
export FOO=$(bar)
you miss any invalid return codes from "bar" (even under -e) because
bash returns the value of the "export", which is 0
On centos-minimal, we stopped bringing in systemd early and this was
causing dib-init-system to not know what init was available. Since it
did not fail correctly, it lead to confusing errors much later in the
build when service files were not copied correctly. See also
I24ce648485c3d6f3c27ab8f87a638516b3727017
A dib-lint check is added. One minor fixup is in 00-set-apt-sources
(this one is less likely to cause problems). I have run dib-lint over
project-config elements and none use this pattern.
Change-Id: I076c08190d40c315ad6a6d96a3823e9fc52630be
It seems that on Xenial, it does not take much to confuse "file" and
it's mime guessing such that it thinks some files are not python.
"package-installs-v2" is a good example, since it has an interpreter
"dib-python" that "file" doesn't know about, and no extension. While
looking at this, I've added emacs vars here so it opens in python
mode.
Change-Id: I01994b08c5ad8987925f1eec4062f5b6ee72eb8f
Currently we have all our elements and library files in a top-level
directory and install them into
<root>/share/diskimage-builder/[elements|lib] (where root is either /
or the root of a virtualenv).
The problem with this is that editable/development installs (pip -e)
do *not* install data_files. Thus we have no canonical location to
look for elements -- leading to the various odd things we do such as a
whole bunch of guessing at the top of disk-image-create and having a
special test-loader in tests/test_elements.py so we can run python
unit tests on those elements that have it.
data_files is really the wrong thing to use for what are essentially
assets of the program. data_files install works well for things like
config-files, init.d files or dropping documentation files.
By moving the elements under the diskimage_builder package, we always
know where they are relative to where we import from. In fact,
pkg_resources has an api for this which we wrap in the new
diskimage_builder/paths.py helper [1].
We use this helper to find the correct path in the couple of places we
need to find the base-elements dir, and for the paths to import the
library shell functions.
Elements such as svc-map and pkg-map include python unit-tests, which
we do not need tests/test_elements.py to special-case load any more.
They just get found automatically by the normal subunit loader.
I have a follow-on change (I69ca3d26fede0506a6353c077c69f735c8d84d28)
to move disk-image-create to a regular python entry-point.
Unfortunately, this has to move to work with setuptools. You'd think
a symlink under diskimage_builder/[elements|lib] would work, but it
doesn't.
[1] this API handles stuff like getting files out of .zip archive
modules, which we don't do. Essentially for us it's returning
__file__.
Change-Id: I5e3e3c97f385b1a4ff2031a161a55b231895df5b
Because environment files are sourced into the current environment,
they shouldn't be setting global settings like tracing else they
affect every preceeding import. This is quite confusing when only
half your imports are traced in the logs, because it was either turned
on, or off, by a preceeding environment import.
There is a corresponding dib-run-parts change in
I29f7df1514aeb988222d1094e8269eddb485c2a0 that will greatly increase
debugability for environment files by deliberately logging what files
are sourced and consistently turning on tracing around their import.
This isn't strictly necessary (since dib-run-parts with the prior
change will just turn tracing off after import anyway) but it's a
decent cleanup for consistency. A bare-minimum dib-lint check is
added. Documentation is updated.
Change-Id: I10f68be0642835a04af7e5a2bc101502f61e5357
Allow file test to return all possible mime-types and ensure anything
that matches the python file type is parsed by flake8 instead of
relying on the first match returned.
Closes-Bug: #1585688
Change-Id: Iba31f1853537fe9234ab6f83d66f13dc1c578abb
As motivation for this; we have had two breakouts of dib in recent
memory. One was a failure to unmount through symlinks in the core
code (I335316019ef948758392b03e91f9869102a472b9) and the other was
removing host keys on the build-system
(Ib01d71ff9415a0ae04d963f6e380aab9ac2260ce).
For the most part, dib runs unprivileged. Bits of the core code are
hopefully well tested (modulo bugs like the first one!). We give free
reign inside the chroot (although there is still some potential there
for adverse external affects via bind mounts). Where we could be a
bit safer (and could have prevented at least the second of these
breakouts) is with some better checking that the "sudo" calls
*outside* the chroot at least looked sane.
This adds a basic check that we're using chroot or image paths when
calling sudo in those parts of elements that run *outside* the chroot.
Various files are updated to accomodate this check; mostly by just
ignoring it for existing code (I have not audited these calls).
Nobody is pretending this type of checking makes dib magically safe,
or removes the issues with it needing to do things as root during the
build. But this can help find egregious errors like the key removal.
Change-Id: I161a5aea1d29dcdc7236f70d372c53246ec73749
It turns out that invalid JSON can be valid YAML ... thus if you mess
up a pkg-map file that still works as a YAML file dib-lint will let it
pass, but when pkg-map later tries to open it as a JSON file, it
fails.
Parse each type separately to catch these problems.
Change-Id: Ib3985e7d1599ed6bf3b7a73b786a53177b71fae0
It's hard to tell if dib-lint is working as it outputs nothing. Add
some minimal output strings at some key points.
Change-Id: Id11cc9ecb8d5215d6fc8d8ef3584bfeeba53ff13
It's better to report all of the failures in one shot, so we should
make sure a flake8 failure doesn't immediately end the dib-lint
run, and instead just sets the error flag like the other checks.
Change-Id: Ib13fc71bb12a6565888bdd89f33fc6ada89f8d8c
Our dib-lint checking is only considering scripts with #!/bin/bash.
While there's nothing really wrong with some other shebang line like
"#!/usr/bin/env bash" let's keep things consistent.
We can use the same regex match to reduce a few forks in the main
checking.
Also a minor cleanup to the file matching
Change-Id: I609721b2671e704ea26075dad7e5b39a8b858f6b
Some of the elements-deps in the project-config repo have a blank line
at the end, which throws out the ordering. Strip blank lines, like
comment lines, before processing.
As an additional help, show a side-by-side diff of what is provided
versus what is expected when showing an error about sorting.
Change-Id: I007851ee01d6853ad992ce4437331e8bd79bbfce
When dib-lint complains about wrong indents, it doesn't give you any
indication where the problem is. This repeats the grep on failure,
outputting the line and line-number.
As a bonus, skip *.orig files from merges
Change-Id: Ifbbdf854ea19191f66e9823468dbc0afc2f93e1f
Look for files .yaml and pkg-map configurations, and try to load them
either as json or yaml. This way, invalid ones can be detected before
they are committed unnoticed.
Also, exclude .yaml files from being searched while checking bash and
python scripts.
Change-Id: I2478837cfe66929ae1b0d7dd96e049773a35e11c
Pass an explicit template to mktemp since that is required on Mac OS X.
Hardcoded to /tmp since I did not want to bother validating TMPDIR which
is user supplied.
Explicit define a repeating group which causes a grep error:
repetition-operator operand invalid
Change-Id: Ib23cc15db7f7dfdd611280884e46342562d545df
I noticed dib-lint was running on my editor backup files and checking
python files (covered by flake8) and RST files
Change-Id: Ide2e8a4c1aa8d619bc4b0326a2cfc5e63c74475f
There is a wide variety of tracing options through the various shell
scripts. Some use "set -eux", others explicity set xtrace and others
do nothing. There is a "-x" option to bin/disk-image-create but it
doesn't flow down to the many scripts it calls.
This adds a global integer variable set by disk-image-create
DIB_DEBUG_TRACE. All scripts have a stanza added to detect this and
turn on tracing. Any other tracing methods are rolled into this. So
the standard header is
---
if [ "${DIB_DEBUG_TRACE:-0}" -gt 0 ]; then
set -x
fi
set -eu
set -o pipefail
---
Multiple -x options can be specified to dib-create-image, which
increases the value of DIB_DEBUG_TRACE. If script authors feel their
script should only trace at higher levels, they should modify the
"-gt" value. If they feel it should trace by default, they can modify
the default value also.
Changes to pachset 16 : scripts which currently trace themselves by
default have retained this behaviour with DIB_DEBUG_TRACE defaulting
to "1". This was done by running [1] on patch set 15. See the thread
beginning at [2]
dib-lint is also updated to look for the variable being matched.
[1] https://gist.github.com/ianw/71bbda9e6acc74ccd0fd
[2] http://lists.openstack.org/pipermail/openstack-dev/2014-November/051575.html
Change-Id: I6c5a962260741dcf6f89da9a33b96372a719b7b0
We now guard against diff returning non-zero (which it does when it
discovers changes). This allows dib-lint runs to continue past the point
of displaying a dependency ordering nit.
Change-Id: Ib7801597400d6d87146181935922c8832baf5aaa
We check python files with dib-lint rather than flake8 which have
conflicting opinions. This means weve been (forcibly) writing non pep8
python.
Also fixing pep8 issues so tests pass.
Change-Id: Idc9db40334f6e15738a7802c06697270df68741c
The current dib-lint script has an exclusion for indent but it is
erroneously used as a label around the tab indention check. This
patch renames the tab indention exclusion and adds the indent
exclusion where it belongs.
Change-Id: I701595e7a08f4764a45c6701d67cbb931c3d47f3
This patch adds exclusions for the indentation and newline
rules in dib-lint. The indentation exclusion is needed for
files using heredoc and wanting to preserve indentation.
The newline exclusion is added here for consisently treating
all dib-lint rules the same.
Change-Id: Ic8a2164eba3cf99e55473846e3a5fe663a3c1095
This will be useful for adopting dib-lint in other projects, in
particular tripleo-image-elements. It allows some dib-lint rules
to be used even if the project can't pass all of the checks. The
failing checks can then be fixed one at a time and removed from the
exclusions list.
For consistency, this change reads the new exclusions from tox.ini
in the same way as flake8. To use the exclusion mechanism, the
the following section can be added to tox.ini:
[dib-lint]
ignore = sete setu setpipefail
Change-Id: I6d8024e7613890e935ceb5e50d0d507bd554d8dd
Currently when dib-lint finds a problem it does something like:
echo "ERROR: Problem found"
rc=1
This is repetitive and error-prone since it's easy to forget to set
rc to actually fail the check. This change makes those two steps
a single function call.
Change-Id: I40b5bf39348a69add1f955c49f310e3bda21be0e
With this change, dib-lint raises an error if when finding an element
using space indentation that is not multiple of 4.
Co-Authored-By: Jon-Paul Sullivan <jonpaul.sullivan@hp.com>
Change-Id: I470e1fdfc38a3f3c7ba5644c5103f2a9ef073005
