Commit Graph

48 Commits

Author SHA1 Message Date
Arx Cruz
1d928a1b8a Fix jsonschema version to match lower-constraint
The jsonschema lower constraint for train is version 3.0.2. This is
making some jobs fail because it can't install the diskimage-builder.

Closes-Bug: #2012489
Change-Id: I620265f6bb09966ff5eaafe9b7b5ee960d0a4977
2023-03-22 10:03:55 +01:00
Steve Baker
7e38f85724 A new diskimage-builder command for yaml image builds
The `diskimage-builder` command provides a yaml file based interface
to `disk-image-create` and `ramdisk-image-create`. Every argument to
these scripts has a YAML equivalent. The command has the following
features:
- Environment values can be provided from the calling environment as
  well as YAML
- All arguments are validated with jsonschema in the most appropriate
  YAML type
- Schema is self-documenting and printed when running with --help
- Multiple YAML files can be specified and each file can have multiple
  images defined
- Entries with duplicate image names will be merged into a single
  image build, with attributes overwritten, elements appended, and
  environment values updated/overwritten. A missing image name implies
  the same image name as the previous entry.
- --dry-run and --stop-on-failure flags

A simple YAML defintion would resemble:

- imagename: centos-minimal
  checksum: true
  install-type: package
  elements: [centos, vm]
- imagename: ironic-python-agent
  elements:
  - ironic-python-agent-ramdisk
  - extra-hardware

The TripleO project has managed image build options with YAML files
and it has proved useful having git history and a diff friendly
format, specifically for the following situations:
- Managing differences between distros (centos, rhel)
- Managing changes in major distro releases (centos-8, centos-9-stream)
- Managing the python2 to python3 transition, within and across major
  distro releases

Now that the TripleO toolchain is being retired this tool is being
proposed to be used for the image builds of TripleO's successor, as
well as the rest of the community.

Subsequent commits will add documentation and switch some tests to
using `diskimage-builder`.

Change-Id: I95cba3530d1b1c6c52cf547338762e33738f7225
2023-03-13 09:01:49 +13:00
Tristan Cacqueray
2c5eb04ba9 Allow flake8 version 5
This change enables using the latest flake8.

Change-Id: Iaa19c36f8cab8482a01f764c588375db8e7d8be3
2022-08-25 18:31:41 +00:00
yatinkarel
8b4ad6dae6 Drop lower version requirement for networkx
With [1] to fix lower constraints job, networkx
requirement was bumped to 2.4.0, but since diskimage-builder
is branchless and to get it work with stable/train upper-constraints
for networkx i.e 2.3.0 can't bump to 2.4.0.
Also networkx-2.3.0 has dep on decorator>=4.3.0
so actual decorator issue with 3.4.0 shouldn't be there.

[1] https://review.opendev.org/c/openstack/diskimage-builder/+/811048
[2] https://github.com/networkx/networkx/blob/networkx-2.3/requirements/default.txt

Closes-Bug: #1945926
Change-Id: I93a2826019d9bb0d004f7daa661fdd1ca4269331
2021-10-05 12:55:43 +05:30
Steve Baker
517e7eca95 Fix lower constraints
decorator-3.4.0 now errors on setup due to using use_2to3, and
networkx-1.1.0 attempts to import fractions.gcd, which is math.gcd
from python-3.5 onwards.

Change-Id: I4f4ddebb079a95708be3dd76086b039db18d5948
2021-09-27 16:03:22 +13:00
Andreas Jaeger
4493208048 Drop six usage
With python3, six is not needed anymore, drop it.

Change-Id: I70bb679270605ac32ca0cceb9414ea3a210e5842
2020-06-05 12:04:37 +02:00
Andreas Jaeger
e7885499dd Cleanup py27 and docs support
This repo is now testing only with Python 3, so let's make
a few cleanups:
- Remove python 2.7 stanza from setup.py
- Remove obsolete sections from setup.cfg
- Update classifiers
- Update requirements, no need for python_version anymore
- Cleanup doc/source/conf.py to remove now obsolete content,
  use sphinxcontrib.apidoc to build module index again (this
  seems to have been broken when switching to sphinx-build).
- Remove install_command from tox.ini, the default is fine
- Use TOX_CONSTRAINTS_FILE instead of obsolete UPPER_CONSTRAINTS_FILE.
- Update tox.ini for python3 only support.

Change-Id: Id8738ecfb0f578d2a7953c63ffe10779f835bcaf
2020-06-05 12:04:30 +02:00
Ian Wienand
0c94eef7be Revert "dib-lint: use yamllint to parse YAML files"
This reverts commit 6ee2995214 and
e85c2a6f03.

I missed that if you pip install and then run dib-lint, it's not going
to pick up the .yamllint file shipped here.  Thus it gives spurious
errors.

The reason for this was simply better duplicate key detection in yaml
files, which caused us problems with the kernel installs.  However, at
this point it seems just the old "does it load" test from pyyaml will
be enough.

Change-Id: I87a9fc9bb119cfeffad48fc0fa0df31f0181825d
2020-05-28 16:44:49 +10:00
Monty Taylor
e85c2a6f03 Add dependency on yamllint
dib-lint requires yamllint now, so we need to depend on it.

Change-Id: Iccd2cdbb04bd9b429d8d19b31ff3b10cdd568e15
2020-05-27 10:28:19 -05:00
Dmitry Tantsur
97cadc4249 Remove Babel and any signs of translations
DIB does not currently have any translations or translatable strings,
so removing local configuration from setup.cfg.

Additionally, Babel is only required if you import it directly,
removing it as well.

Change-Id: I036de97641df38eef408290b50cecfe216c69f6d
2020-04-17 17:05:15 +02:00
Monty Taylor
1038f9c8ba Remove hacking from requirements
putting it in requirements means it's installed for anything that
installs diskimage-builder. Not just for tests, but in actual prod
installations. Because of the way flake8 plugins work, this means
then that hacking is injected into all flake8 invocations in that
environment.

This is currently causing issues in nodepool where we do not
actually use hacking but it's still spewing spurious messages to
the screen because it is incorrect finding our apache header to be
incorrect.

Just depend on flake8, since that's what dib-lint depends on.

Change-Id: I8d2b2076b604d596906a3b12a5518e2a56c0fe88
2020-03-12 16:20:37 -05:00
Ian Wienand
28ebd24844 Uncap hacking
This causes problems for other projects incorporating dib; we don't
have a specific need for a cap.

Fix a few issues, mostly spacing or regex matches.  No functional
changes.

W503 and W504 relate to leaving artithmetic operators at the start or
end of lines, and are mutually exclusive and, due to "ignore"
overriding the defaults both get enabled.  It seems everyone gets this
wrong (https://gitlab.com/pycqa/flake8/issues/466).  Don't take a
position on this and ignore both.

Use double # around comments including YAML snippets using "# type: "
which now gets detected as PEP484/mypy type hints.

Change-Id: I8b7ce6dee02dcce31c82427a2441c931d136ef57
2020-02-24 10:34:46 +11:00
Ian Wienand
8c85a7ed5d Revert "Fixed use of flake8"
This reverts commit 74317a3445.

What I missed was that "dib-lint" is distributed and people use this
to lint their out-of-tree elements (e.g. project-config).  Because
dib-lint runs flake8 that really makes it a valid runtime dependency.

However, I don't think we need to pin to these particular versions of
flake8 in requirements (which I think was causing the original issues,
as it's fairly specific and easily conflicts).  It seems to make more
sense to use the current uncapped "hacking" versions; so move them
from test-requirements.txt.  Add a note so we don't forget again.

Change-Id: Ic2eee8f5b64c7020e98e0b1ef43a40f24411891a
2019-09-16 10:17:49 +10:00
Sorin Sbarnea
74317a3445 Fixed use of flake8
Flake8 was wrongly listed as a runtime dependency. That mistake messed
system flake8 version. Flake8 is a test requirement not a runtime one.

Also flake8 version is already determined by hacking version, so we
update it to recent version.

Change-Id: Iccd9dc6f2508127f17712d16dfce6538ff83807e
2019-09-04 13:40:35 +00:00
Fatih Degirmenci
8d7e9b3805 Constraint networkx to <2.3 for Python 2
NetworkX released 2.3, dropping support for Python 2.

https://github.com/networkx/networkx/blob/master/doc/news.rst

Depends-On: https://review.openstack.org/#/c/652094

Change-Id: I59afe81d7868605434d2a42658c5b09a6a546732
2019-04-13 22:11:58 +02:00
Ian Wienand
1107326723 Update pylint to 1.7.6, uncap networkx
This review squashes:
    Iac9afc7766d3640815dc20cfd6de1245d36a09cc
    Ie894b5801bd7b3815432882cd626941e89d9f9a1

We need to do this as we can't fix pylint without networkx as that
failes requirements-chak due to us having a cap on networkx and we can't
uncap networkx as part of tripleo-buildimage installs without
constratints which gets us 2.1 and DIB desn't support 2.x

This is the commit message Iac9afc7766d3640815dc20cfd6de1245d36a09cc
---
One of the pylint dependencies has updated to be python3 only; this
version of pylint correctly caps things so it still works with
python2.

This also exposes that we need to uncap networkx due to
I34045f87ca19c2f184b040f4d89347374cce518b.  We should remain on
version 1 for now thanks to upper-constraints, but we need to maintain
the lower-constraint.
---

This is the commit message Ie894b5801bd7b3815432882cd626941e89d9f9a1
---
Support different versions of networkx

Since the entry of networkx 2.0 nodes has a different
behaviour. Checking if dg.nodes is iterable is enough to add
compatibility for new/older versions.
---

Change-Id: I82dc61fac6c156a4f0d574290c7632077aa53195
2018-07-18 09:27:01 +10:00
OpenStack Proposal Bot
a565f5ddc1 Updated from global requirements
Change-Id: Idf7fa1936a9ee0fc6c3569f8c399f2039cd873a9
2018-03-04 10:01:45 +00:00
OpenStack Proposal Bot
f0d6e04c52 Updated from global requirements
Change-Id: I03dcd2a53ac16425dfa02ce67de4dd959d1bd53d
2017-11-16 11:01:41 +00:00
Ian Wienand
06b5ce4573 Revert "Support networkx 2.0"
This reverts commit a47ff0dd4a.

Since this merged, a global-requirements pin to keep networkx <2.0 has
also merged.  The plan is:

1. revert our 2.0 support and
1a. take the <2.0 pin from global requirements
2. figure out how to use constraints properly in our testing
3. restore this, with a depends-on for a 2.0 bump in requirements
   (which will self-test, see 3.)
4. when other projects are ready for a global 2.0 bump, merge
   in a controlled fashion

This reverts the 2.0 support, and adds the pin for networkx <2.0

Change-Id: I18f6a1115da779581245e3dd423fd90516974a33
2017-09-22 08:02:44 +10:00
OpenStack Proposal Bot
058debc7dd Updated from global requirements
Change-Id: Ic1d5fe9de580f445e88758e7adc279548ce87e5a
2017-08-24 11:37:04 +00:00
Ian Wienand
75817ef205 Use networkx for digraph
This switches the code to use networkx for the digraph implementation.

Note that the old implementation specifically isn't removed in this
change -- for review clarity.  It will be replaced by a base class
that defines things properly to the API described below.

Plugins return a node object with three functions

 get_name() : return the unique name of this node

 get_nodes() : return a list of nodes for insertion into the graph.
  Usually this is just "self".  Some special things like partitioning
  add extra nodes at this point, however.

 get_edges() : return a tuple of two lists; edges_from and edges_to
  As you would expect the first is a list of node names that points to
  us, and the second is a list of node names we point to.  Usually
  this is only populated as ([self.base],[]) -- i.e. our "base" node
  points to us.  Some plugins, such as mounting, create links both to
  and from themselves, however.

Plugins have been updated, some test cases added (error cases
specifically)

Change-Id: Ic5a61365ef0132476b11bdbf1dd96885e91c3cb6
2017-05-26 11:42:10 +10:00
OpenStack Proposal Bot
f3cfeab5ec Updated from global requirements
Change-Id: I09b47aa75008bfd832ec8eec9108501f18223875
2017-04-12 04:10:50 +00:00
OpenStack Proposal Bot
93af629970 Updated from global requirements
Change-Id: I1a8a956293dd7f1973938c60f827705d8b739c3e
2017-04-07 14:22:14 +00:00
Andreas Florath
57c9e0bb41 Use stevedore for plugin config of block device
This patch introduces stevedore plugin mechanism for use
with the block device layer.  This makes it possible that
other projects pass in their own block device plugins.

Change-Id: Id3ea56aaf75f5a20a4e1b6ac2a68adb12c56b574
Signed-off-by: Andreas Florath <andreas@florath.net>
2017-03-24 21:42:47 +00:00
OpenStack Proposal Bot
acbe59ce10 Updated from global requirements
Change-Id: Ia7ab7528e2d4b4626724f454823801c7917017c8
2017-03-13 19:30:19 +00:00
Ian Wienand
f1776d72aa Merge branch 'master' into merge-branch
Change-Id: Ib3c4b0030c94459a8c2eca836a16e42edf1accd4
2017-03-02 13:49:29 +11:00
Andreas Jaeger
b833960c69 Fix requirements update
Sphinx 1.5 fails with current pbr (needs new pbr release, see change
I52d45fa0a0d42de690d3a492068f7bb03483a224.

For now, set warnerrors to False to be able to use this.

Also, remove duplicate requirements line.

Also, update to new sphinx version to show that this works.

This is in reaction to the failure in
Ia184446fe34c49a48ca079c828157ea34e662d4b.

Change-Id: I9e7261c4124b71eeb6bddd9e21747b61bbdc16fa
2017-02-12 16:59:06 +01:00
Ian Wienand
7d5afecfd9 Merge remote-tracking branch 'origin/master' into merge-branch
Change-Id: Ibab1bb95521292ae818bd91f7073c3749a2cc0cb
2016-11-18 13:53:56 +11:00
OpenStack Proposal Bot
0f742ce24a Updated from global requirements
Change-Id: I4b8e6f6fdf78706dbb735e25195a1f83413ab3fd
2016-11-09 04:14:55 +00:00
Ian Wienand
d65678678e Move dib-run-parts into diskimage-builder
Move dib-run-parts from dib-utils into diskimage-builder directly.

For calling outside the chroot, we provide a standard entry-point
script.  However, as noted in the warning comment, the underlying
script is still copied directly into the chroot by the dib-run-parts
element.  I believe this to be the KISS approach.

This removes the dependency on dib-utils.  We have discussed this
previously and nobody seemed to think retiring dib-utils was going to
be an issue.

This also updates the documentation to not mention dib-utils, or using
disk-image-create via $PATH setup, but rather gives instructions on
installing from pip with a virtualenv.

Change-Id: Ic1e22ba498d2c368da7d72e2e2b70ff34324feb8
2016-11-04 17:07:37 +11:00
OpenStack Proposal Bot
1f10f4fa35 Updated from global requirements
Change-Id: I0f152cb4160915bbcad273bca7d28f947b82eae4
2016-09-30 19:58:01 +00:00
OpenStack Proposal Bot
c48cef7f1d Updated from global requirements
Change-Id: Iab736f98c26e50caa1705f5f772afe6fb2e2fd99
2016-06-29 04:21:33 +00:00
Gregory Haynes
8b3419b0a9 Fix sphinx-build to not depend on diskimage-builder
Currently, running sphinx_build fails for us because we depend on
diskimage-builder in our sphinx conf.py. This causes doc generation
on sites like rtfd to fail unless they install the diskimage-builder
module beforehand. We can, alternatively, import pbr directly and not
require the module as part of doc generation.

Change-Id: I41f222ff9c67950fc30841935a6a603f5718395e
2016-06-20 18:36:42 -07:00
OpenStack Proposal Bot
4bb5a5f4f3 Updated from global requirements
Change-Id: I109295e1d88e672235a7d5fc470f9f1fa12d3170
2016-05-13 00:39:26 +00:00
OpenStack Proposal Bot
e3f92e4a52 Updated from global requirements
Change-Id: Ia75dfa7c0ef9cbbbe5b7dba51d13c50c91c52922
2016-04-15 01:52:46 +00:00
OpenStack Proposal Bot
3bfc44cb99 Updated from global requirements
Change-Id: Id519f2c3129418821eae1692d4da8305729202e8
2016-03-16 13:26:57 +00:00
Colleen Murphy
76a8d3704b Add --version option to disk-image-create
Being able to discover DIB's version from the command itself is
convenient. This patch adds a --version option to the disk-image-create
command, failing gracefully if diskimage-builder is not installed.

This adds an explicit dependency on pbr to the requirements since this
is required to run diskimage_builder/version.py outside of a test
environment.

This patch consciously chooses to only provide the long-form option
and no '-v' to allow for the future possibility that a '-v' might
indicate '--verbose' in the future.

Change-Id: I9fc084774d6c7a39a944b07680b3eb8be8e34f9c
2016-02-29 16:13:01 -08:00
OpenStack Proposal Bot
be68534ec0 Updated from global requirements
Change-Id: If7e380cf61c284ea65e390e3cee130999746f2f7
2016-01-28 13:19:08 +00:00
Andreas Jaeger
dee406dd7c Remove argparse from requirements
argparse was external in python 2.6 but not anymore, remove it from
requirements.

Change-Id: Ib7e74912b36c1b5ccb514e31fac35efeff57378d
2016-01-22 19:49:50 +01:00
OpenStack Proposal Bot
64e9c6eac3 Updated from global requirements
Change-Id: I5229ba6b3d21f7902d1af847e0c567a0debedf7a
2016-01-22 04:03:13 +00:00
Martin André
8309df5f25 Add missing six requirement for svc-map element
Add constraint for versions >= 1.9.0 to be in line with global
requirements.

Change-Id: I91bcbfa00b1b03774d507dd5c9c5fcd1c4d809b4
2015-11-05 01:58:00 +00:00
OpenStack Proposal Bot
943ab4b88e Updated from global requirements
Change-Id: I93e693c255787272399d53d841627459e46e8974
2015-11-03 04:29:38 +00:00
Ian Wienand
4a936fe741 Add flake8 to requirements
flake8 is called directly by dib-lint, so this is a hard requirement
to be installed for dib-lint to work outside of tox.  This came up
when proposing a dib-lint job for the nodepool elements in
project-config (I28459a9bbaaa6175d03dbe753498849a75942538)

Change-Id: I3052f860cebc66844cd21f34be64aea8474d2248
2015-10-01 19:08:29 +10:00
OpenStack Proposal Bot
c4ab253653 Updated from global requirements
Change-Id: Iff075e25eeb091650ac85ca1e1af7a1fd3394d3e
2015-06-22 08:20:17 +00:00
Kiall Mac Innes
ac349eda4f Install PyYAML for the svc-map element
The svc-map element expects to be able to "import yaml", we should ensure
it's installed alongside DIB.

Change-Id: I1e4c19bbb2a7143e61974cffb3d7232d90414388
2014-11-10 14:45:36 +00:00
OpenStack Proposal Bot
6a71077c28 Updated from global requirements
Change-Id: Ie2bb534b456ed3aac79b706cf51ecee9789527e4
2014-09-23 01:55:13 +00:00
Ben Nemec
644598941c Use dib-run-parts from dib-utils
Now that dib-run-parts has been moved to the dib-utils project, we
need to update diskimage-builder to use it instead of the version
directly in diskimage-builder.

This change removes the old copy of the dib-run-parts script in
the element, adds dib-utils as a dependency of diskimage-builder,
and updates the uses of dib-run-parts to correctly handle the fact
that it is now external to the project.

Requires I0be1f876d0e4a7d38e0d5c6010a552a8ebb158a4

Change-Id: Ia0a0df7784a14c49b5c47ac0b03e6c2602c84b3b
2014-08-26 16:44:18 -05:00
JUN JIE NAN
7fd212c019 Move Babel and argparse from test-requirements.txt
to requirements.txt, to avoid disk-image-create failed without
dependencies installed.

Change-Id: I110809d298593890b2b050d6da2dcbd036db1ff3
2014-01-16 17:01:50 +08:00