The `diskimage-builder` command provides a yaml file based interface
to `disk-image-create` and `ramdisk-image-create`. Every argument to
these scripts has a YAML equivalent. The command has the following
features:
- Environment values can be provided from the calling environment as
well as YAML
- All arguments are validated with jsonschema in the most appropriate
YAML type
- Schema is self-documenting and printed when running with --help
- Multiple YAML files can be specified and each file can have multiple
images defined
- Entries with duplicate image names will be merged into a single
image build, with attributes overwritten, elements appended, and
environment values updated/overwritten. A missing image name implies
the same image name as the previous entry.
- --dry-run and --stop-on-failure flags
A simple YAML defintion would resemble:
- imagename: centos-minimal
checksum: true
install-type: package
elements: [centos, vm]
- imagename: ironic-python-agent
elements:
- ironic-python-agent-ramdisk
- extra-hardware
The TripleO project has managed image build options with YAML files
and it has proved useful having git history and a diff friendly
format, specifically for the following situations:
- Managing differences between distros (centos, rhel)
- Managing changes in major distro releases (centos-8, centos-9-stream)
- Managing the python2 to python3 transition, within and across major
distro releases
Now that the TripleO toolchain is being retired this tool is being
proposed to be used for the image builds of TripleO's successor, as
well as the rest of the community.
Subsequent commits will add documentation and switch some tests to
using `diskimage-builder`.
Change-Id: I95cba3530d1b1c6c52cf547338762e33738f7225
decorator-3.4.0 now errors on setup due to using use_2to3, and
networkx-1.1.0 attempts to import fractions.gcd, which is math.gcd
from python-3.5 onwards.
Change-Id: I4f4ddebb079a95708be3dd76086b039db18d5948
This repo is now testing only with Python 3, so let's make
a few cleanups:
- Remove python 2.7 stanza from setup.py
- Remove obsolete sections from setup.cfg
- Update classifiers
- Update requirements, no need for python_version anymore
- Cleanup doc/source/conf.py to remove now obsolete content,
use sphinxcontrib.apidoc to build module index again (this
seems to have been broken when switching to sphinx-build).
- Remove install_command from tox.ini, the default is fine
- Use TOX_CONSTRAINTS_FILE instead of obsolete UPPER_CONSTRAINTS_FILE.
- Update tox.ini for python3 only support.
Change-Id: Id8738ecfb0f578d2a7953c63ffe10779f835bcaf
This reverts commit 6ee2995214 and
e85c2a6f03.
I missed that if you pip install and then run dib-lint, it's not going
to pick up the .yamllint file shipped here. Thus it gives spurious
errors.
The reason for this was simply better duplicate key detection in yaml
files, which caused us problems with the kernel installs. However, at
this point it seems just the old "does it load" test from pyyaml will
be enough.
Change-Id: I87a9fc9bb119cfeffad48fc0fa0df31f0181825d
DIB does not currently have any translations or translatable strings,
so removing local configuration from setup.cfg.
Additionally, Babel is only required if you import it directly,
removing it as well.
Change-Id: I036de97641df38eef408290b50cecfe216c69f6d
putting it in requirements means it's installed for anything that
installs diskimage-builder. Not just for tests, but in actual prod
installations. Because of the way flake8 plugins work, this means
then that hacking is injected into all flake8 invocations in that
environment.
This is currently causing issues in nodepool where we do not
actually use hacking but it's still spewing spurious messages to
the screen because it is incorrect finding our apache header to be
incorrect.
Just depend on flake8, since that's what dib-lint depends on.
Change-Id: I8d2b2076b604d596906a3b12a5518e2a56c0fe88
This causes problems for other projects incorporating dib; we don't
have a specific need for a cap.
Fix a few issues, mostly spacing or regex matches. No functional
changes.
W503 and W504 relate to leaving artithmetic operators at the start or
end of lines, and are mutually exclusive and, due to "ignore"
overriding the defaults both get enabled. It seems everyone gets this
wrong (https://gitlab.com/pycqa/flake8/issues/466). Don't take a
position on this and ignore both.
Use double # around comments including YAML snippets using "# type: "
which now gets detected as PEP484/mypy type hints.
Change-Id: I8b7ce6dee02dcce31c82427a2441c931d136ef57
This reverts commit 74317a3445.
What I missed was that "dib-lint" is distributed and people use this
to lint their out-of-tree elements (e.g. project-config). Because
dib-lint runs flake8 that really makes it a valid runtime dependency.
However, I don't think we need to pin to these particular versions of
flake8 in requirements (which I think was causing the original issues,
as it's fairly specific and easily conflicts). It seems to make more
sense to use the current uncapped "hacking" versions; so move them
from test-requirements.txt. Add a note so we don't forget again.
Change-Id: Ic2eee8f5b64c7020e98e0b1ef43a40f24411891a
Flake8 was wrongly listed as a runtime dependency. That mistake messed
system flake8 version. Flake8 is a test requirement not a runtime one.
Also flake8 version is already determined by hacking version, so we
update it to recent version.
Change-Id: Iccd9dc6f2508127f17712d16dfce6538ff83807e
This review squashes:
Iac9afc7766d3640815dc20cfd6de1245d36a09cc
Ie894b5801bd7b3815432882cd626941e89d9f9a1
We need to do this as we can't fix pylint without networkx as that
failes requirements-chak due to us having a cap on networkx and we can't
uncap networkx as part of tripleo-buildimage installs without
constratints which gets us 2.1 and DIB desn't support 2.x
This is the commit message Iac9afc7766d3640815dc20cfd6de1245d36a09cc
---
One of the pylint dependencies has updated to be python3 only; this
version of pylint correctly caps things so it still works with
python2.
This also exposes that we need to uncap networkx due to
I34045f87ca19c2f184b040f4d89347374cce518b. We should remain on
version 1 for now thanks to upper-constraints, but we need to maintain
the lower-constraint.
---
This is the commit message Ie894b5801bd7b3815432882cd626941e89d9f9a1
---
Support different versions of networkx
Since the entry of networkx 2.0 nodes has a different
behaviour. Checking if dg.nodes is iterable is enough to add
compatibility for new/older versions.
---
Change-Id: I82dc61fac6c156a4f0d574290c7632077aa53195
This reverts commit a47ff0dd4a.
Since this merged, a global-requirements pin to keep networkx <2.0 has
also merged. The plan is:
1. revert our 2.0 support and
1a. take the <2.0 pin from global requirements
2. figure out how to use constraints properly in our testing
3. restore this, with a depends-on for a 2.0 bump in requirements
(which will self-test, see 3.)
4. when other projects are ready for a global 2.0 bump, merge
in a controlled fashion
This reverts the 2.0 support, and adds the pin for networkx <2.0
Change-Id: I18f6a1115da779581245e3dd423fd90516974a33
This switches the code to use networkx for the digraph implementation.
Note that the old implementation specifically isn't removed in this
change -- for review clarity. It will be replaced by a base class
that defines things properly to the API described below.
Plugins return a node object with three functions
get_name() : return the unique name of this node
get_nodes() : return a list of nodes for insertion into the graph.
Usually this is just "self". Some special things like partitioning
add extra nodes at this point, however.
get_edges() : return a tuple of two lists; edges_from and edges_to
As you would expect the first is a list of node names that points to
us, and the second is a list of node names we point to. Usually
this is only populated as ([self.base],[]) -- i.e. our "base" node
points to us. Some plugins, such as mounting, create links both to
and from themselves, however.
Plugins have been updated, some test cases added (error cases
specifically)
Change-Id: Ic5a61365ef0132476b11bdbf1dd96885e91c3cb6
This patch introduces stevedore plugin mechanism for use
with the block device layer. This makes it possible that
other projects pass in their own block device plugins.
Change-Id: Id3ea56aaf75f5a20a4e1b6ac2a68adb12c56b574
Signed-off-by: Andreas Florath <andreas@florath.net>
Sphinx 1.5 fails with current pbr (needs new pbr release, see change
I52d45fa0a0d42de690d3a492068f7bb03483a224.
For now, set warnerrors to False to be able to use this.
Also, remove duplicate requirements line.
Also, update to new sphinx version to show that this works.
This is in reaction to the failure in
Ia184446fe34c49a48ca079c828157ea34e662d4b.
Change-Id: I9e7261c4124b71eeb6bddd9e21747b61bbdc16fa
Move dib-run-parts from dib-utils into diskimage-builder directly.
For calling outside the chroot, we provide a standard entry-point
script. However, as noted in the warning comment, the underlying
script is still copied directly into the chroot by the dib-run-parts
element. I believe this to be the KISS approach.
This removes the dependency on dib-utils. We have discussed this
previously and nobody seemed to think retiring dib-utils was going to
be an issue.
This also updates the documentation to not mention dib-utils, or using
disk-image-create via $PATH setup, but rather gives instructions on
installing from pip with a virtualenv.
Change-Id: Ic1e22ba498d2c368da7d72e2e2b70ff34324feb8
Currently, running sphinx_build fails for us because we depend on
diskimage-builder in our sphinx conf.py. This causes doc generation
on sites like rtfd to fail unless they install the diskimage-builder
module beforehand. We can, alternatively, import pbr directly and not
require the module as part of doc generation.
Change-Id: I41f222ff9c67950fc30841935a6a603f5718395e
Being able to discover DIB's version from the command itself is
convenient. This patch adds a --version option to the disk-image-create
command, failing gracefully if diskimage-builder is not installed.
This adds an explicit dependency on pbr to the requirements since this
is required to run diskimage_builder/version.py outside of a test
environment.
This patch consciously chooses to only provide the long-form option
and no '-v' to allow for the future possibility that a '-v' might
indicate '--verbose' in the future.
Change-Id: I9fc084774d6c7a39a944b07680b3eb8be8e34f9c
flake8 is called directly by dib-lint, so this is a hard requirement
to be installed for dib-lint to work outside of tox. This came up
when proposing a dib-lint job for the nodepool elements in
project-config (I28459a9bbaaa6175d03dbe753498849a75942538)
Change-Id: I3052f860cebc66844cd21f34be64aea8474d2248
The svc-map element expects to be able to "import yaml", we should ensure
it's installed alongside DIB.
Change-Id: I1e4c19bbb2a7143e61974cffb3d7232d90414388
Now that dib-run-parts has been moved to the dib-utils project, we
need to update diskimage-builder to use it instead of the version
directly in diskimage-builder.
This change removes the old copy of the dib-run-parts script in
the element, adds dib-utils as a dependency of diskimage-builder,
and updates the uses of dib-run-parts to correctly handle the fact
that it is now external to the project.
Requires I0be1f876d0e4a7d38e0d5c6010a552a8ebb158a4
Change-Id: Ia0a0df7784a14c49b5c47ac0b03e6c2602c84b3b
