Commit Graph

3647 Commits

Author SHA1 Message Date
Jenkins
a2f910d702 Merge "Make our virtualenv source py3 safe" 2017-03-14 01:35:36 +00:00
Jenkins
429dad946f Merge "blockdevice.py: python3 fixes" 2017-03-14 01:22:27 +00:00
OpenStack Proposal Bot
acbe59ce10 Updated from global requirements
Change-Id: Ia7ab7528e2d4b4626724f454823801c7917017c8
2017-03-13 19:30:19 +00:00
Michael Johnson
fae8004e4d squash-package-install to use the correct python
99-squash-package-install in the package-installs element does not
know which python environments the requirements were installed into.
This can cause it to select the wrong python to run the
package-installs-squash script.

Co-Authored-By: Adam Harwell <flux.adam@gmail.com>
Change-Id: I5fab0e192c3a2dad8f60e821c184479e24e33bcd
2017-03-13 20:24:39 +11:00
Ian Wienand
85985cdadc Make our virtualenv source py3 safe
execfile() has gone in python3, and if you google various
stackoverflow results, python-dev mailing list threads and other
projects it seems runpy is considered one of the better solutions.
This should be py2.7 safe too.

Change-Id: I18077ba9d603752492cc81f260e12710981f4dff
2017-03-13 20:22:27 +11:00
Ian Wienand
b04c58eca4 blockdevice.py: python3 fixes
keys() is an iterator in python3, so we need to make it list before
finding the first element

Change-Id: Ic158c7b2901c1a34ff417d3432fcefed4760ce24
2017-03-13 19:33:07 +11:00
Jenkins
8b19f7f99d Merge "Fix dib-init-system for Debian Jessie and Debian Stretch" 2017-03-13 03:29:41 +00:00
Jenkins
ace33c0097 Merge "Send custom parameters in bootloader to GRUB_CMDLINE" 2017-03-13 03:14:46 +00:00
Andreas Florath
20389d755f Fix dib-init-system for Debian Jessie and Debian Stretch
On Debian Jessie and Debian Stretch systemctl is in /bin.
If the package systemd-sysv is not installed the script
dib-init-system did not find the init system.
This patch fixes the problem: it also looks in /bin
for systemctl and if found decides for systemd.

Change-Id: I5a18052a070bad5e16b14672237a1e2b38513949
Signed-off-by: Andreas Florath <andreas@florath.net>
2017-03-11 14:04:13 +00:00
Andreas Florath
c99ea56726 Replace architecture-emulation-binaries with qemu-debootstrap
The architecture-emulation-binaries element does for a limited number
of architectures on Ubuntu exactly the same as `qemu-debootstrap`.
`qemu-debootstrap` comes in the qemu-user-static package that is
anyhow used when cross-building an image.

This patch replaces the complete architecture-emulation-binaries
element with a call to qemu-debootstrap.

Change-Id: Ib9667307bfd3ff7592444a2ec5b04aa5365a1872
Signed-off-by: Andreas Florath <andreas@florath.net>
2017-03-11 13:58:37 +00:00
Ian Wienand
6887e796e1 Add 2.0.0 release notes
This is a distillation of a longer email I sent to openstack-dev [1]
as an omnibus update for 2.0.0 release notes.

[1] http://lists.openstack.org/pipermail/openstack-dev/2017-February/111833.html

Change-Id: Ic0b012626e0850c41d532e5dfdf82538cc6397d8
2017-03-10 14:29:30 +11:00
Yolanda Robla
11275c212b Send custom parameters in bootloader to GRUB_CMDLINE
Depending on the types of deployment (security, nfv...) some extra
kernel flags are needed on the images. This change exposes the
DIB_BOOTLOADER_DEFAULT_CMDLINE parameter, defaulting to the
existing 'nofb nomodeset vga=normal', that will allow to modify
these boot params.

Change-Id: I67d191fa5ca44a57f776cb9739a02dd71212969c
Closes-Bug: #1668890
2017-03-08 15:43:46 +01:00
Jenkins
958ea8a337 Merge "Run indent checks in diskimage_builder/lib" into feature/v2
Change-Id: I471413a7a8ff601483752ab0b9a35aa0b6ecda27
2017-03-08 19:51:49 +11:00
Ian Wienand
c8206d1ab6 Run indent checks in diskimage_builder/lib
Scripts have moved from bin/ to diskimage_builder/lib (and are not
executable, since they are called from dib).  Add a match so we run
checkers over those files.

Clarify the situation with .py files while we're here.

Change-Id: I8a1ef4b00a185b83c8b1f29c563b85e78bc233ca
2017-03-08 11:57:49 +11:00
Yolanda Robla
96504a4de0 Use OrderedDict for partitions instead of simple dictionary
The order of the partitions is important, it needs to be preserved.
If using a simple dict, this is not happening. As a consequence,
checks like 'primary partition being first' are failing because the
dictionary sorts the partitions randomly.
Switched to OrderedDict solved the problem, as it preserves the
ordering it gets from the yaml blob.

Change-Id: Icfa9bd95ffd0203d7c3f6af95de3a6f848c2a954
2017-03-07 15:55:20 +01:00
Andreas Florath
866a06f92d Refactor: block-device partitioning cleanup
Now that the main partitioning refactor patch is merged, there is
a small relict of handling partitions still in the disk-image-create
main.

This patch moves the functionality from disk-image-create to the
block-device/partitioning module: it is mostly a rewrite of the
original bash code in python.

Change-Id: Ia73baeca74180a7bc9ea487da03ff56d6a3070ce
Signed-off-by: Andreas Florath <andreas@florath.net>
2017-03-07 18:43:09 +11:00
Ian Wienand
f1776d72aa Merge branch 'master' into merge-branch
Change-Id: Ib3c4b0030c94459a8c2eca836a16e42edf1accd4
2017-03-02 13:49:29 +11:00
Dirk Mueller
65c6da58c3 [suse] Handle pip-and-virtualenv install for opensuse
Similarly to Fedora we need to register the virtualenv package
with the rpm database to avoid things becoming messy later on
when package dependencies are conflicting with the from-source
install.

Change-Id: I67654fe5533e6086a17b38e2ae79a630a609ff92
2017-02-28 22:51:25 +01:00
Jenkins
2d919e7ca0 Merge "[suse] remove --no-confirm from zypper invocation" 2017-02-28 01:31:50 +00:00
Jenkins
f7669f60da Merge "gentoo: do not manually clean /tmp" 2017-02-28 01:31:44 +00:00
Bernard Cafarelli
fedea46c9e
pip-and-virtualenv: also handle rhel distros
The current pip install script only checked for centos/fedora
This causes setuptools errrors like (with pbr install):
"SyntaxError: '<' operator not allowed in environment markers"

Explictly list distro names in that situation

Change-Id: I5c894ab6152acf5441231acc1215fe00967f4f31
2017-02-21 16:09:13 +01:00
Dirk Mueller
661c4c32d2 [suse] remove --no-confirm from zypper invocation
this is basically the deprecated version of --non-interactive, which is
already being used, so this is duplicate and only causes warnings but
doesn't help with anything

Change-Id: Ic98790461636c4136b7005bdc31ce08f8ca81e0f
2017-02-20 20:48:50 +01:00
Pino Toscano
e2192fc7b7 gentoo: do not manually clean /tmp
disk-image-create already does that globally after the guest is
finalized, see lib/img-functions:finalise_base.  Thus, there is no need
to remove part of /tmp manually, and no other distro element does that.

Change-Id: Id3b72d41fce258af98b31976ef726d57a0a9fae4
2017-02-20 17:27:25 +01:00
Andreas Jaeger
b833960c69 Fix requirements update
Sphinx 1.5 fails with current pbr (needs new pbr release, see change
I52d45fa0a0d42de690d3a492068f7bb03483a224.

For now, set warnerrors to False to be able to use this.

Also, remove duplicate requirements line.

Also, update to new sphinx version to show that this works.

This is in reaction to the failure in
Ia184446fe34c49a48ca079c828157ea34e662d4b.

Change-Id: I9e7261c4124b71eeb6bddd9e21747b61bbdc16fa
2017-02-12 16:59:06 +01:00
Andreas Florath
003fc17208 Fix #1627402: fix grub2 element for ubuntu xenial
Looks that the special handling for Ubuntu is not needed any longer
(its a pity that there are no detailed comments...).

The grub2 element is a second implementation of the bootstrap element
- but because there are some features that come only here, e.g. efi
boot, it should be working as long as this is not implemented in the
bootloader element.

Change-Id: I74269116ea30b84f3259805720d5cd1616f960c5
Signed-off-by: Andreas Florath <andreas@florath.net>
Closes-Bug: #1627402
2017-02-11 18:44:18 +00:00
Ian Wienand
7a155e08bf Merge branch 'master' into merge-branch
Change-Id: I28e4c7837d84e8b66eff3d182666c5a87a9e3c9b
2017-02-09 13:35:53 +11:00
Andreas Florath
fdd2c4b236 Semi-automatic doc generation of element dependency
Currently there is no description of dependencies in the generated
documentation of the elements: therefore a user of an element does not
know which other elements are automatically included and e.g. which
configuration options are available. In addition there are some
copy&pastes of parts of the README.rst scattered thought different
Ubuntu and Debian specific elements.

This patch adds a semi-automatic generation of dependency information
of all elements.  Nevertheless these are not automatically included.
The author of the element's README.rst can decide if and where the
dependency information should appear and can use the descriptor

.. element_deps::

for this.

This patch adds the dependency information for some Debian and
Ubuntu patches - and creates the base for later removing the
duplicated parts.

A call is added to element_dependencies._find_all_elements() to
populate reverse dependencies for Element objects.

(This is a reworking of I31d2b6050b6c46fefe37378698e9a330025db430 for
the feature/v2 branch)

Change-Id: Iebb83916fed71565071246baa550849eef40560b
2017-02-09 09:50:30 +11:00
Ian Wienand
f3121f0fb7 Fix up doc errors
It seems that pbr's "warnerrors" isn't actually doing anything at the
moment (I680b448471e687919d202e8f2abe57f8ba3b22ee) meaning we're able
to commit docs with RST parser errors.  Fix all these up in some
minimal way (I have not audited content, just made it pass build).

Link the specs in so they're referenced from the top level.

Change-Id: Id67b9ea7ba8f49b43969c58ca3fb7fa1243538a4
2017-02-08 16:07:01 +11:00
Ian Wienand
3654aa4205 Fix up element doc generation
Turns out that you really have to have the elements dir symlinked
under the source directory for the globbing to match anything.  AFAICT
there's no way to add external directories to the sphinx build.

Change-Id: Iaa3576ef250822b0d57bebce1ebbe03e5bafa042
2017-02-08 14:32:43 +11:00
Jenkins
234f44d703 Merge "python-brickclient: accommodate python2/3 changes" 2017-02-08 00:32:29 +00:00
Jenkins
24628b135b Merge "Move Ubuntu specific use_tempaddr setting to ubuntu-common element" 2017-02-07 23:25:38 +00:00
Anshul Jain
dfc8fbf907 python-brickclient: accommodate python2/3 changes
Following a similar fix to pip-and-virtualenv element in
Ia730850a48e2478fd5461710a9d2619408725cd8, python-brickclient is also
required to be fixed to handle python3 only systems.

Change-Id: I442469beb5a20074425e7059eec7f78604d6cc5a
2017-02-08 09:33:02 +11:00
Andreas Florath
23ac49bf83 Move Ubuntu specific use_tempaddr setting to ubuntu-common element
By default (during boot) the use_tempaddr is set to <=0 for all
up-to date kernels.  Only Ubuntu installes a sysctl setting which
sets the use_tempaddr to 2 (/etc/sysctl.d/10-ipv6-privacy.conf) [1].
The 80-disable-rfc3041 overwrites this setting and sets
use_tempaddr back to 0.

Because this only affects Ubuntu it makes sense to move the script
to the ubuntu-common element. The other motivation for the move is,
to clear the base element that it can be removed.

[1] https://bugs.launchpad.net/ubuntu/+source/procps/+bug/1068756

Change-Id: Ibf261818ca8243874fde9eb3650bb65188fa228d
Signed-off-by: Andreas Florath <andreas@florath.net>
2017-02-07 20:41:13 +00:00
Jenkins
740c0a85df Merge "Move generation of dib_[environment|args] to manifest element" 2017-02-07 20:23:06 +00:00
Jenkins
15969b9c21 Merge "Preinstall pyOpenSSL" 2017-02-07 08:53:23 +00:00
Jenkins
0713387588 Merge "Turn down some low-value tracing output" 2017-02-07 08:20:02 +00:00
Jenkins
cac1471cc8 Merge "yum/install-packages output cleanup" 2017-02-07 06:34:26 +00:00
Jenkins
ab478545b1 Merge "Target map-packages deprecation message" 2017-02-07 06:24:41 +00:00
Jenkins
6372de09fe Merge "Use strings in package-installs follow output" 2017-02-07 05:50:40 +00:00
Nam Nguyen Hoai
e3b67a3d45 Fix typo in README.rst
There is a wrong word, it should be updated.

Change-Id: Iff70cc9154ef3b3121207df627e83271fd3159a9
2017-02-07 13:58:24 +07:00
Ian Wienand
cc177cc215 yum/install-packages output cleanup
Rather than echo the package list separately, turn on tracing around
the yum/dnf call so that we get more complete output.  No loss of
info, as this effecitvely shows the packgae list; plus it has the
bonus that it makes more sense in the logs, because otherwise you get
all the yum/dnf output without knowing what the call was.

Change-Id: I44df6e944af602e6b03f669c15a521056de1ad79
2017-02-07 15:49:47 +11:00
Ian Wienand
124cec0bba Preinstall pyOpenSSL
With the warnings added in Ibfe69dc84246662ed8caa0d4c3e2edf68314c87e
we can see that this element is relying on map-packages to translate
the debian package name python-pyopenssl -> pyOpenSSL.

This makes no sense because this is not a generic element; it should
just use the fedora/redhat names directly.

Change-Id: Id6ecb6f978b60d6a527692692a408d1d35828de2
2017-02-07 15:47:37 +11:00
Ian Wienand
5bd8158862 Target map-packages deprecation message
Currently every run of install-packages puts out a warning that
map-packages is obsolete.  This happens even if map-package does no
mapping.  The caller can't prevent the call (it's part of
install-packages) and it gives no actionable help if there is
something wrong.

Keep track of any mappings we are doing in the obsolete map-packages
run and only output a warning if we actually translate anything.  If
we do output, tell the caller what packages were translated so they
can make appropriate pkg-map entries.

Change-Id: Ibfe69dc84246662ed8caa0d4c3e2edf68314c87e
2017-02-07 15:44:51 +11:00
Ian Wienand
3c6972a5c0 Turn down some low-value tracing output
There are a couple of loops identified here that output a lot of
tracing each iteration for little value.  Make them only trace at
higher levels (-x -x and above).

Points of actual interest within the loops are replaced with an
explicit "echo" statement.

Additionally, export DIB_DEBUG_TRACE explicitly and in all cases, not
just when tracing is turned on.

Change-Id: Id710c0b111fc1f5e1ae87fc35f6db28b24867bad
2017-02-07 15:26:11 +11:00
Andreas Florath
870374c8da Move generation of dib_[environment|args] to manifest element
dib_[environment|args] manifest files are currently generated by the
base element and then moved by the manifest element.

This creates too many corner cases -- if you don't include the base
element (we are trying to empty it ATM) you don't get the env/args
saved at all; if you include base but don't include the manifest
element they're saved to /etc, but if you do have the manifest element
they're moved to the manifest dir.

Move generation of these into the manifest element directly and update
the documentation to reflect this.  In practice this doesn't change
things, because the "manifests" element gets pulled in via deps for
most builds.

Change-Id: I3f23037058137d166b29f0b70fd1a02c22c07fc8
Signed-off-by: Andreas Florath <andreas@florath.net>
2017-02-07 11:33:16 +11:00
Jenkins
0f224d3ae6 Merge "Check return of _load_state" into feature/v2 2017-02-06 23:41:50 +00:00
Andreas Florath
df1f583903 Check return of _load_state
In error scenarios there might be no loadable state available.
This patch adds some robustness handling this scenarios
by checking the result and possible bailing out.

Change-Id: I02e2731b14bec22c22db08d60d8d15db1911a84e
Signed-off-by: Andreas Florath <andreas@florath.net>
2017-02-04 19:21:48 +00:00
Andreas Florath
a8953dd277 block-device: change top level config from dict to list
With the old configuration structure it was only possible
to use one image and one partition layout.  The new
block-device configuration uses a list at top level;
therefore it is possible to use multiple instances
of each element type.

Change-Id: I9db4327486b676887d6ce09609994116dbebfc89
Signed-off-by: Andreas Florath <andreas@florath.net>
2017-02-03 20:15:39 +00:00
Ian Wienand
3f8800832a Release 1.27.0
-----BEGIN PGP SIGNATURE-----
 
 iQEcBAABAgAGBQJYk8+9AAoJEBty/58O8cX8LdIH+wU/VrEVs0XYohiL6DUgabzs
 112U3UUihH5xMc/ca9Tarx+XwEvfMZkwYN2Qr0JoRJjmSt2AL6AezUhGSV+98vaY
 iQEccaFDFYlyDHm4V2r7N1xwS0B3mx87FPqVQQSUKlc3JsQxCy4o9RtD9aM8Gvqy
 +gAxMxL3p3O131K0Rvb0U5lC1FLgft9SuljCV8i5nU4/HdoryD6hedz2/ss8a9KG
 KKEdBKvPBKn73+nb8peQD/VXpej9C31r87q5VEjUsZkJ7gduY/qYLlGGgoBQqAXN
 WQ/ef1RkQKW5ba2jsjnk7fdOrA0+wYENxorR2WecuZbe2ieXw6fP3lYiD6VeWsM=
 =IUuh
 -----END PGP SIGNATURE-----

Merge tag '1.27.0' into merge-branch

Release 1.27.0

Change-Id: I9f6948636cae6d375d1d8315976504021f5a3bbb
2017-02-03 11:49:45 +11:00
Jenkins
111cb51055 Merge "Unify tidy up logs in lib/img-functions" 2017-02-03 00:29:48 +00:00