Commit Graph

2786 Commits

Author SHA1 Message Date
Andreas Florath
20389d755f Fix dib-init-system for Debian Jessie and Debian Stretch
On Debian Jessie and Debian Stretch systemctl is in /bin.
If the package systemd-sysv is not installed the script
dib-init-system did not find the init system.
This patch fixes the problem: it also looks in /bin
for systemctl and if found decides for systemd.

Change-Id: I5a18052a070bad5e16b14672237a1e2b38513949
Signed-off-by: Andreas Florath <andreas@florath.net>
2017-03-11 14:04:13 +00:00
Jenkins
958ea8a337 Merge "Run indent checks in diskimage_builder/lib" into feature/v2
Change-Id: I471413a7a8ff601483752ab0b9a35aa0b6ecda27
2017-03-08 19:51:49 +11:00
Ian Wienand
c8206d1ab6 Run indent checks in diskimage_builder/lib
Scripts have moved from bin/ to diskimage_builder/lib (and are not
executable, since they are called from dib).  Add a match so we run
checkers over those files.

Clarify the situation with .py files while we're here.

Change-Id: I8a1ef4b00a185b83c8b1f29c563b85e78bc233ca
2017-03-08 11:57:49 +11:00
Yolanda Robla
96504a4de0 Use OrderedDict for partitions instead of simple dictionary
The order of the partitions is important, it needs to be preserved.
If using a simple dict, this is not happening. As a consequence,
checks like 'primary partition being first' are failing because the
dictionary sorts the partitions randomly.
Switched to OrderedDict solved the problem, as it preserves the
ordering it gets from the yaml blob.

Change-Id: Icfa9bd95ffd0203d7c3f6af95de3a6f848c2a954
2017-03-07 15:55:20 +01:00
Andreas Florath
866a06f92d Refactor: block-device partitioning cleanup
Now that the main partitioning refactor patch is merged, there is
a small relict of handling partitions still in the disk-image-create
main.

This patch moves the functionality from disk-image-create to the
block-device/partitioning module: it is mostly a rewrite of the
original bash code in python.

Change-Id: Ia73baeca74180a7bc9ea487da03ff56d6a3070ce
Signed-off-by: Andreas Florath <andreas@florath.net>
2017-03-07 18:43:09 +11:00
Ian Wienand
f1776d72aa Merge branch 'master' into merge-branch
Change-Id: Ib3c4b0030c94459a8c2eca836a16e42edf1accd4
2017-03-02 13:49:29 +11:00
Dirk Mueller
65c6da58c3 [suse] Handle pip-and-virtualenv install for opensuse
Similarly to Fedora we need to register the virtualenv package
with the rpm database to avoid things becoming messy later on
when package dependencies are conflicting with the from-source
install.

Change-Id: I67654fe5533e6086a17b38e2ae79a630a609ff92
2017-02-28 22:51:25 +01:00
Jenkins
2d919e7ca0 Merge "[suse] remove --no-confirm from zypper invocation" 2017-02-28 01:31:50 +00:00
Jenkins
f7669f60da Merge "gentoo: do not manually clean /tmp" 2017-02-28 01:31:44 +00:00
Bernard Cafarelli
fedea46c9e
pip-and-virtualenv: also handle rhel distros
The current pip install script only checked for centos/fedora
This causes setuptools errrors like (with pbr install):
"SyntaxError: '<' operator not allowed in environment markers"

Explictly list distro names in that situation

Change-Id: I5c894ab6152acf5441231acc1215fe00967f4f31
2017-02-21 16:09:13 +01:00
Dirk Mueller
661c4c32d2 [suse] remove --no-confirm from zypper invocation
this is basically the deprecated version of --non-interactive, which is
already being used, so this is duplicate and only causes warnings but
doesn't help with anything

Change-Id: Ic98790461636c4136b7005bdc31ce08f8ca81e0f
2017-02-20 20:48:50 +01:00
Pino Toscano
e2192fc7b7 gentoo: do not manually clean /tmp
disk-image-create already does that globally after the guest is
finalized, see lib/img-functions:finalise_base.  Thus, there is no need
to remove part of /tmp manually, and no other distro element does that.

Change-Id: Id3b72d41fce258af98b31976ef726d57a0a9fae4
2017-02-20 17:27:25 +01:00
Andreas Jaeger
b833960c69 Fix requirements update
Sphinx 1.5 fails with current pbr (needs new pbr release, see change
I52d45fa0a0d42de690d3a492068f7bb03483a224.

For now, set warnerrors to False to be able to use this.

Also, remove duplicate requirements line.

Also, update to new sphinx version to show that this works.

This is in reaction to the failure in
Ia184446fe34c49a48ca079c828157ea34e662d4b.

Change-Id: I9e7261c4124b71eeb6bddd9e21747b61bbdc16fa
2017-02-12 16:59:06 +01:00
Andreas Florath
003fc17208 Fix #1627402: fix grub2 element for ubuntu xenial
Looks that the special handling for Ubuntu is not needed any longer
(its a pity that there are no detailed comments...).

The grub2 element is a second implementation of the bootstrap element
- but because there are some features that come only here, e.g. efi
boot, it should be working as long as this is not implemented in the
bootloader element.

Change-Id: I74269116ea30b84f3259805720d5cd1616f960c5
Signed-off-by: Andreas Florath <andreas@florath.net>
Closes-Bug: #1627402
2017-02-11 18:44:18 +00:00
Ian Wienand
7a155e08bf Merge branch 'master' into merge-branch
Change-Id: I28e4c7837d84e8b66eff3d182666c5a87a9e3c9b
2017-02-09 13:35:53 +11:00
Andreas Florath
fdd2c4b236 Semi-automatic doc generation of element dependency
Currently there is no description of dependencies in the generated
documentation of the elements: therefore a user of an element does not
know which other elements are automatically included and e.g. which
configuration options are available. In addition there are some
copy&pastes of parts of the README.rst scattered thought different
Ubuntu and Debian specific elements.

This patch adds a semi-automatic generation of dependency information
of all elements.  Nevertheless these are not automatically included.
The author of the element's README.rst can decide if and where the
dependency information should appear and can use the descriptor

.. element_deps::

for this.

This patch adds the dependency information for some Debian and
Ubuntu patches - and creates the base for later removing the
duplicated parts.

A call is added to element_dependencies._find_all_elements() to
populate reverse dependencies for Element objects.

(This is a reworking of I31d2b6050b6c46fefe37378698e9a330025db430 for
the feature/v2 branch)

Change-Id: Iebb83916fed71565071246baa550849eef40560b
2017-02-09 09:50:30 +11:00
Ian Wienand
f3121f0fb7 Fix up doc errors
It seems that pbr's "warnerrors" isn't actually doing anything at the
moment (I680b448471e687919d202e8f2abe57f8ba3b22ee) meaning we're able
to commit docs with RST parser errors.  Fix all these up in some
minimal way (I have not audited content, just made it pass build).

Link the specs in so they're referenced from the top level.

Change-Id: Id67b9ea7ba8f49b43969c58ca3fb7fa1243538a4
2017-02-08 16:07:01 +11:00
Ian Wienand
3654aa4205 Fix up element doc generation
Turns out that you really have to have the elements dir symlinked
under the source directory for the globbing to match anything.  AFAICT
there's no way to add external directories to the sphinx build.

Change-Id: Iaa3576ef250822b0d57bebce1ebbe03e5bafa042
2017-02-08 14:32:43 +11:00
Jenkins
234f44d703 Merge "python-brickclient: accommodate python2/3 changes" 2017-02-08 00:32:29 +00:00
Jenkins
24628b135b Merge "Move Ubuntu specific use_tempaddr setting to ubuntu-common element" 2017-02-07 23:25:38 +00:00
Anshul Jain
dfc8fbf907 python-brickclient: accommodate python2/3 changes
Following a similar fix to pip-and-virtualenv element in
Ia730850a48e2478fd5461710a9d2619408725cd8, python-brickclient is also
required to be fixed to handle python3 only systems.

Change-Id: I442469beb5a20074425e7059eec7f78604d6cc5a
2017-02-08 09:33:02 +11:00
Andreas Florath
23ac49bf83 Move Ubuntu specific use_tempaddr setting to ubuntu-common element
By default (during boot) the use_tempaddr is set to <=0 for all
up-to date kernels.  Only Ubuntu installes a sysctl setting which
sets the use_tempaddr to 2 (/etc/sysctl.d/10-ipv6-privacy.conf) [1].
The 80-disable-rfc3041 overwrites this setting and sets
use_tempaddr back to 0.

Because this only affects Ubuntu it makes sense to move the script
to the ubuntu-common element. The other motivation for the move is,
to clear the base element that it can be removed.

[1] https://bugs.launchpad.net/ubuntu/+source/procps/+bug/1068756

Change-Id: Ibf261818ca8243874fde9eb3650bb65188fa228d
Signed-off-by: Andreas Florath <andreas@florath.net>
2017-02-07 20:41:13 +00:00
Jenkins
740c0a85df Merge "Move generation of dib_[environment|args] to manifest element" 2017-02-07 20:23:06 +00:00
Jenkins
15969b9c21 Merge "Preinstall pyOpenSSL" 2017-02-07 08:53:23 +00:00
Jenkins
0713387588 Merge "Turn down some low-value tracing output" 2017-02-07 08:20:02 +00:00
Jenkins
cac1471cc8 Merge "yum/install-packages output cleanup" 2017-02-07 06:34:26 +00:00
Jenkins
ab478545b1 Merge "Target map-packages deprecation message" 2017-02-07 06:24:41 +00:00
Jenkins
6372de09fe Merge "Use strings in package-installs follow output" 2017-02-07 05:50:40 +00:00
Nam Nguyen Hoai
e3b67a3d45 Fix typo in README.rst
There is a wrong word, it should be updated.

Change-Id: Iff70cc9154ef3b3121207df627e83271fd3159a9
2017-02-07 13:58:24 +07:00
Ian Wienand
cc177cc215 yum/install-packages output cleanup
Rather than echo the package list separately, turn on tracing around
the yum/dnf call so that we get more complete output.  No loss of
info, as this effecitvely shows the packgae list; plus it has the
bonus that it makes more sense in the logs, because otherwise you get
all the yum/dnf output without knowing what the call was.

Change-Id: I44df6e944af602e6b03f669c15a521056de1ad79
2017-02-07 15:49:47 +11:00
Ian Wienand
124cec0bba Preinstall pyOpenSSL
With the warnings added in Ibfe69dc84246662ed8caa0d4c3e2edf68314c87e
we can see that this element is relying on map-packages to translate
the debian package name python-pyopenssl -> pyOpenSSL.

This makes no sense because this is not a generic element; it should
just use the fedora/redhat names directly.

Change-Id: Id6ecb6f978b60d6a527692692a408d1d35828de2
2017-02-07 15:47:37 +11:00
Ian Wienand
5bd8158862 Target map-packages deprecation message
Currently every run of install-packages puts out a warning that
map-packages is obsolete.  This happens even if map-package does no
mapping.  The caller can't prevent the call (it's part of
install-packages) and it gives no actionable help if there is
something wrong.

Keep track of any mappings we are doing in the obsolete map-packages
run and only output a warning if we actually translate anything.  If
we do output, tell the caller what packages were translated so they
can make appropriate pkg-map entries.

Change-Id: Ibfe69dc84246662ed8caa0d4c3e2edf68314c87e
2017-02-07 15:44:51 +11:00
Ian Wienand
3c6972a5c0 Turn down some low-value tracing output
There are a couple of loops identified here that output a lot of
tracing each iteration for little value.  Make them only trace at
higher levels (-x -x and above).

Points of actual interest within the loops are replaced with an
explicit "echo" statement.

Additionally, export DIB_DEBUG_TRACE explicitly and in all cases, not
just when tracing is turned on.

Change-Id: Id710c0b111fc1f5e1ae87fc35f6db28b24867bad
2017-02-07 15:26:11 +11:00
Andreas Florath
870374c8da Move generation of dib_[environment|args] to manifest element
dib_[environment|args] manifest files are currently generated by the
base element and then moved by the manifest element.

This creates too many corner cases -- if you don't include the base
element (we are trying to empty it ATM) you don't get the env/args
saved at all; if you include base but don't include the manifest
element they're saved to /etc, but if you do have the manifest element
they're moved to the manifest dir.

Move generation of these into the manifest element directly and update
the documentation to reflect this.  In practice this doesn't change
things, because the "manifests" element gets pulled in via deps for
most builds.

Change-Id: I3f23037058137d166b29f0b70fd1a02c22c07fc8
Signed-off-by: Andreas Florath <andreas@florath.net>
2017-02-07 11:33:16 +11:00
Jenkins
0f224d3ae6 Merge "Check return of _load_state" into feature/v2 2017-02-06 23:41:50 +00:00
Andreas Florath
df1f583903 Check return of _load_state
In error scenarios there might be no loadable state available.
This patch adds some robustness handling this scenarios
by checking the result and possible bailing out.

Change-Id: I02e2731b14bec22c22db08d60d8d15db1911a84e
Signed-off-by: Andreas Florath <andreas@florath.net>
2017-02-04 19:21:48 +00:00
Andreas Florath
a8953dd277 block-device: change top level config from dict to list
With the old configuration structure it was only possible
to use one image and one partition layout.  The new
block-device configuration uses a list at top level;
therefore it is possible to use multiple instances
of each element type.

Change-Id: I9db4327486b676887d6ce09609994116dbebfc89
Signed-off-by: Andreas Florath <andreas@florath.net>
2017-02-03 20:15:39 +00:00
Ian Wienand
3f8800832a Release 1.27.0
-----BEGIN PGP SIGNATURE-----
 
 iQEcBAABAgAGBQJYk8+9AAoJEBty/58O8cX8LdIH+wU/VrEVs0XYohiL6DUgabzs
 112U3UUihH5xMc/ca9Tarx+XwEvfMZkwYN2Qr0JoRJjmSt2AL6AezUhGSV+98vaY
 iQEccaFDFYlyDHm4V2r7N1xwS0B3mx87FPqVQQSUKlc3JsQxCy4o9RtD9aM8Gvqy
 +gAxMxL3p3O131K0Rvb0U5lC1FLgft9SuljCV8i5nU4/HdoryD6hedz2/ss8a9KG
 KKEdBKvPBKn73+nb8peQD/VXpej9C31r87q5VEjUsZkJ7gduY/qYLlGGgoBQqAXN
 WQ/ef1RkQKW5ba2jsjnk7fdOrA0+wYENxorR2WecuZbe2ieXw6fP3lYiD6VeWsM=
 =IUuh
 -----END PGP SIGNATURE-----

Merge tag '1.27.0' into merge-branch

Release 1.27.0

Change-Id: I9f6948636cae6d375d1d8315976504021f5a3bbb
2017-02-03 11:49:45 +11:00
Jenkins
111cb51055 Merge "Unify tidy up logs in lib/img-functions" 2017-02-03 00:29:48 +00:00
Jenkins
03d1af7380 Merge "Don't run unit tests from run_functests.sh" into feature/v2 2017-02-02 22:51:04 +00:00
Ian Wienand
adf39c52cf Release 1.21.1
-----BEGIN PGP SIGNATURE-----
 
 iQEcBAABAgAGBQJYW2GoAAoJEBty/58O8cX8uSAH/15dJsglP6Zie7jSSJcR6k+e
 PJembHn9qrqrCjmJ5EwakojySaaLhwEJKvlP54OU9v7pmUXL9gJtK2OzW54LQ41g
 xBHIu0Pg4z7juyHm9+1P2Sr7Mzs1pVSEbsIYpDYUU19eghI1EAeIj3I1woKgajN7
 JlI61j3r67G6EAVtPOnmD1jvXS8CrtjiJ9wtWTH20pWfmksovg/GuXUCZrLkAAhO
 NcK35CdMii1Hkr7XOH424La/Ar+3qfUX18ZkbJY6yHzkq/ityTzzKOFjAaDl2Jg9
 WNc+SLCVYpPhPwgt7miTywamUNj3ZviA5/Hd8fuLXmtHSLQ23WOtBiaQMLtwXHs=
 =8dIl
 -----END PGP SIGNATURE-----

Merge tag '1.26.1' into merge-branch

Release 1.21.1

Change-Id: Ib9eb3dd1d384fc5b6a9846608216e056c57a173a
2017-02-02 20:36:23 +11:00
Ian Wienand
19f13958f7 Handle blank lines in element-deps
When merging feature/v2 we have re-enabled the centos-minimal test
which has a blank line in the element-deps.  The modified get_elements
functions added in If97bcd21e45de1b5ed91023fdc441a4617051a6b were not
handling this.

Fixing this, we can extract this into some common code for
element-deps and element-provides.

Also, I noticed while debugging that the element list is an iterator
from reverse(), so printing it doesn't help.  Fix up the debug output.

Change-Id: I8e9b079f586e8a36dba3797a0fc03ca1b2041a04
2017-02-02 18:18:46 +11:00
Ian Wienand
f1af76f228 Don't run unit tests from run_functests.sh
Change ec7f56c1b2 add added unit tests
in diskimage_builder/tests/functional (this is probably misnamed).

These are found and run by testr just as part of the normal "setup.py
test" run.  Don't run them as part of the functional tests (this
breaks "-h"/"-l" because it installs a virtualenv and runs tests, and
also is incorrect in the gate where it's creating a nested virtualenv
underneath the testing virtualenv).

Change-Id: I9908e080042d3026a198ba89eb653c6eff376d22
2017-02-02 15:20:24 +11:00
Ian Wienand
bfca36c772 Release 1.25.2
-----BEGIN PGP SIGNATURE-----
 
 iQEcBAABAgAGBQJYV1yqAAoJEBty/58O8cX8hLwIAKP66w6MdPN8PDgUOteui/Sx
 N0UFKJ9yR4GQOAP0NffPLjch5/g0iJLs3eFKOhtGC1LjbDjpVgjX8vW18ib8wBZK
 GemOZPF3uxg8FROrZF1vpoDy/cHgL1YV10hCnwdjN/r9rb8zOuSabqjW+Dennj2n
 fZ0SJfa8Owfudn3YxGuOymVb/wMtEloDmVGBEI1Y+h7osELCCDi3OXmwsA8qMsdl
 cTwbeugBs4PlOVbZUK/JKGuwIHKgPnDYzYu5KpXw77/MdjGT0fo5Tlq5AOBDI2sC
 9JOFEBDli4Ro05VwvI58ADMpvvOax+9EvOhLbB1dRPdZl21Iyb6gOdy2PUbFO0c=
 =aKxq
 -----END PGP SIGNATURE-----

Merge tag '1.25.2' into merge-branch

Release 1.25.2

Change-Id: I698bcf2e82117bd81649cd065a7af5cac85990c7
2017-02-02 11:20:00 +11:00
Jenkins
b4266001d0 Merge "Spec for changing the block device handling: partitioning" 2017-01-31 17:45:19 +00:00
Jenkins
47e1ab5c9e Merge "Create ubuntu/fedora test for pip-and-virtualenv" 2017-01-31 17:45:14 +00:00
Jenkins
7f08bd6234 Merge "Speed up chroot checking loop" 2017-01-31 17:42:04 +00:00
Jenkins
f054e5f24a Merge "debootstrap: avoid duplicate network configuration" 2017-01-31 17:39:28 +00:00
Ian Wienand
83bdfe179a Git ignore coverage stuff
One less piece of clutter when you're trying to merge ...

Change-Id: I794027280359f5d1e02d4f3b99222d35783075d5
2017-01-31 14:21:21 +11:00
Ian Wienand
b6e631360f Release 1.24.0
-----BEGIN PGP SIGNATURE-----
 
 iQEcBAABAgAGBQJYP+r7AAoJEBty/58O8cX8ZcMIAI7Yy2BfGqY/jfyLYNryATSu
 g78McMxAeOiPYII+KKyVkeWsChKJUafLZr/ppO95pm7ZLXeHQeC4MHDI68RmghIX
 HVXn4hPYzwlZ+Yr4hqaEf11H+vw0TDxo2a5bxlQZw+FCx9+3lkR60QxWf8a6QCxv
 nLqKIhtpwGi9mIyvAya+MaOmHMHqbNYgIBHKP0yXhTdxoHZoJWZeXN1rkelwAF9N
 TS1+KwIk+uemn1ZlqzYDWj8euyFyAv8L0WlF0ztCKe7TAFvNeMo26fLykeKHVDEo
 GC5R8MshiOtihPC972rNIponIzd62AetTlDxfK+gcUZO21pNTrlgdLX5Jn04fo0=
 =BpQc
 -----END PGP SIGNATURE-----

Merge tag '1.24.0' into merge-branch

Release 1.24.0

Change-Id: I23cfa6f8dc66bb6a1d957a49a5b468cfd47c005b
2017-01-31 14:14:19 +11:00