Commit Graph

3462 Commits

Author SHA1 Message Date
Jenkins
f128d08cf0 Merge "Use correct Ubuntu distro url on non-x86 arches" 2017-03-24 04:42:59 +00:00
Jenkins
94e2973785 Merge "functests: skip qcow2 generically but add specific test" 2017-03-24 01:07:10 +00:00
Mikhail S Medvedev
4d4417c9a6 Use correct Ubuntu distro url on non-x86 arches
diskimage-builder usually provides defaults that work out of the box.
One default that does not work outside of x86 land is Ubuntu distro
mirror url. Considering there are only two valid default options, we can
automatically choose a better default.

This patch changes behavior only for architectures known to be using
http://ports.ubuntu.com/ubuntu-ports. All others still would use
http://archive.ubuntu.com/ubuntu as default. It provides some guarantee
that we do not introduce a regression.

Change-Id: If95a64bac0c88f30736da4bae7f1fdce126c0bf6
2017-03-23 09:27:25 -05:00
gecong1973
8ff6f51b81 Typo fix: curent => current
Change-Id: I5ce5a98c355310d6f5f6c09e19993b01ed025f0e
2017-03-23 15:02:20 +08:00
Ian Wienand
76ae374413 functests: skip qcow2 generically but add specific test
We somewhat discussed skipping qcow2 generation previously in
I9372e195913798a851c96e62eee89029e067baa1.  As recent issues with PPC
testing have shown, we are not actually testing the "vm" element and
hence the bootloader path in the functional tests.

I don't think we need to test this on every element; it overlaps
somewhat with the testing done by the nodepool jobs which build full
images and boot them.  I also didn't want to introduce a separate run
for this.  Thus it seems valuable to at least have one element
enhanced to do this installation and conversion in our default tests
for basic sanity.

This disables qcow generation by default, as per the other change, but
allows an element to drop a file that will override the output
formats.  The Xenial element is modified to produce a qcow2 using
this, and also introduces a dependency on the "vm" element so it tries
to install the bootloader.

We now exit if the .qcow2 fails to build as well.

Change-Id: I1a6acefe52f8c696c39b2d592fdc7ae32a87e6fe
2017-03-23 13:49:24 +11:00
Jenkins
2ea3a434f7 Merge "Add default PPC block-device layout" 2017-03-23 01:03:13 +00:00
Jenkins
f7999f86e4 Merge "Capture output in _exec_sudo" 2017-03-23 01:01:13 +00:00
Ian Wienand
19fcd263d0 Add default PPC block-device layout
Add a default PPC block-device layout.  I've extracted this into
separate yaml files for ease of editing and to facilitate things like
longer comments.

This is not sufficient to get PPC images working, but it is required.

Change-Id: I09e5d1ed92260bdb632333f5203dd7e70d512dc8
2017-03-23 09:44:01 +11:00
Ian Wienand
a8f2eaded8 Capture output in _exec_sudo
The stdout of the script is captured, so anything coming out from
these commands needs to be captured.  Move to check_process and show
the output as part of an error log in failure case.

Change-Id: I1150375cdc479d4f19b8ddeb49a824ab16fdf831
2017-03-23 09:31:37 +11:00
Dan Radez
06576a02f0 Adding aarch64 support for CentOS7
Using delorian and delorian-deps for aarch64
this will produce an overcloud-full image

Change-Id: Id0dc83d9c558b0c0bbfaed7727883541d140472c
2017-03-22 10:46:54 -04:00
Yolanda Robla
8a3a3ffe6e Fix typo in CMDLINE env var for bootloader
The correct spelling of the env var needs to be
DIB_BOOTLOADER_DEFAULT_CMDLINE

Change-Id: I4ae2aed928723588e736b63283a54eca5f4b4c3b
2017-03-22 12:02:37 +01:00
Jenkins
f1d53f2e31 Merge "Use sphinx warning-is-error" 2017-03-15 23:56:59 +00:00
Alex Schultz
02e5353a9f Allow ELEMENTS_DIR to be configurable
Prior to version 2.0.0 of diskimage-builder, the elements dir was hard
coded as 'elements' in dib-lint which allowed dib-lint to be consumed by
other projects (instack-undercloud). The 2.0.0 release has changed this
path and it is not overridable so that the other projects can still use
dib-lint. This change makes ELEMENTS_DIR and LIB_DIR check for an
existing environment variable so that dib-lint can be consumed outside
of the diskimage-builder project.

Change-Id: Iae1dade1e20a950e63ffe8dacfea0f23cf9923a0
Related-Bug: #1672730
2017-03-14 09:57:10 -06:00
Ian Wienand
fea6ab1624 Use sphinx warning-is-error
Sphix 1.5 (I9e7261c4124b71eeb6bddd9e21747b61bbdc16fa) includes
"warning-is-error" which supersedes pbr's warnerrors.  Enable this and
fix up the resulting failures

 - trailing lines for lists in element_deps directive
 - missing README's that are linked
 - syntax error and highlighting in building instructions

Change-Id: I6549551b4a9bf47076c9811a7a38a666cbea2a50
2017-03-14 14:49:49 +11:00
Jenkins
756923c85b Merge "squash-package-install to use the correct python" 2017-03-14 01:35:42 +00:00
Jenkins
a2f910d702 Merge "Make our virtualenv source py3 safe" 2017-03-14 01:35:36 +00:00
Jenkins
429dad946f Merge "blockdevice.py: python3 fixes" 2017-03-14 01:22:27 +00:00
OpenStack Proposal Bot
acbe59ce10 Updated from global requirements
Change-Id: Ia7ab7528e2d4b4626724f454823801c7917017c8
2017-03-13 19:30:19 +00:00
Michael Johnson
fae8004e4d squash-package-install to use the correct python
99-squash-package-install in the package-installs element does not
know which python environments the requirements were installed into.
This can cause it to select the wrong python to run the
package-installs-squash script.

Co-Authored-By: Adam Harwell <flux.adam@gmail.com>
Change-Id: I5fab0e192c3a2dad8f60e821c184479e24e33bcd
2017-03-13 20:24:39 +11:00
Ian Wienand
85985cdadc Make our virtualenv source py3 safe
execfile() has gone in python3, and if you google various
stackoverflow results, python-dev mailing list threads and other
projects it seems runpy is considered one of the better solutions.
This should be py2.7 safe too.

Change-Id: I18077ba9d603752492cc81f260e12710981f4dff
2017-03-13 20:22:27 +11:00
Ian Wienand
b04c58eca4 blockdevice.py: python3 fixes
keys() is an iterator in python3, so we need to make it list before
finding the first element

Change-Id: Ic158c7b2901c1a34ff417d3432fcefed4760ce24
2017-03-13 19:33:07 +11:00
Jenkins
8b19f7f99d Merge "Fix dib-init-system for Debian Jessie and Debian Stretch" 2017-03-13 03:29:41 +00:00
Jenkins
ace33c0097 Merge "Send custom parameters in bootloader to GRUB_CMDLINE" 2017-03-13 03:14:46 +00:00
Andreas Florath
20389d755f Fix dib-init-system for Debian Jessie and Debian Stretch
On Debian Jessie and Debian Stretch systemctl is in /bin.
If the package systemd-sysv is not installed the script
dib-init-system did not find the init system.
This patch fixes the problem: it also looks in /bin
for systemctl and if found decides for systemd.

Change-Id: I5a18052a070bad5e16b14672237a1e2b38513949
Signed-off-by: Andreas Florath <andreas@florath.net>
2017-03-11 14:04:13 +00:00
Andreas Florath
c99ea56726 Replace architecture-emulation-binaries with qemu-debootstrap
The architecture-emulation-binaries element does for a limited number
of architectures on Ubuntu exactly the same as `qemu-debootstrap`.
`qemu-debootstrap` comes in the qemu-user-static package that is
anyhow used when cross-building an image.

This patch replaces the complete architecture-emulation-binaries
element with a call to qemu-debootstrap.

Change-Id: Ib9667307bfd3ff7592444a2ec5b04aa5365a1872
Signed-off-by: Andreas Florath <andreas@florath.net>
2017-03-11 13:58:37 +00:00
Ian Wienand
6887e796e1 Add 2.0.0 release notes
This is a distillation of a longer email I sent to openstack-dev [1]
as an omnibus update for 2.0.0 release notes.

[1] http://lists.openstack.org/pipermail/openstack-dev/2017-February/111833.html

Change-Id: Ic0b012626e0850c41d532e5dfdf82538cc6397d8
2017-03-10 14:29:30 +11:00
Yolanda Robla
11275c212b Send custom parameters in bootloader to GRUB_CMDLINE
Depending on the types of deployment (security, nfv...) some extra
kernel flags are needed on the images. This change exposes the
DIB_BOOTLOADER_DEFAULT_CMDLINE parameter, defaulting to the
existing 'nofb nomodeset vga=normal', that will allow to modify
these boot params.

Change-Id: I67d191fa5ca44a57f776cb9739a02dd71212969c
Closes-Bug: #1668890
2017-03-08 15:43:46 +01:00
Jenkins
958ea8a337 Merge "Run indent checks in diskimage_builder/lib" into feature/v2
Change-Id: I471413a7a8ff601483752ab0b9a35aa0b6ecda27
2017-03-08 19:51:49 +11:00
Ian Wienand
c8206d1ab6 Run indent checks in diskimage_builder/lib
Scripts have moved from bin/ to diskimage_builder/lib (and are not
executable, since they are called from dib).  Add a match so we run
checkers over those files.

Clarify the situation with .py files while we're here.

Change-Id: I8a1ef4b00a185b83c8b1f29c563b85e78bc233ca
2017-03-08 11:57:49 +11:00
Yolanda Robla
96504a4de0 Use OrderedDict for partitions instead of simple dictionary
The order of the partitions is important, it needs to be preserved.
If using a simple dict, this is not happening. As a consequence,
checks like 'primary partition being first' are failing because the
dictionary sorts the partitions randomly.
Switched to OrderedDict solved the problem, as it preserves the
ordering it gets from the yaml blob.

Change-Id: Icfa9bd95ffd0203d7c3f6af95de3a6f848c2a954
2017-03-07 15:55:20 +01:00
Andreas Florath
866a06f92d Refactor: block-device partitioning cleanup
Now that the main partitioning refactor patch is merged, there is
a small relict of handling partitions still in the disk-image-create
main.

This patch moves the functionality from disk-image-create to the
block-device/partitioning module: it is mostly a rewrite of the
original bash code in python.

Change-Id: Ia73baeca74180a7bc9ea487da03ff56d6a3070ce
Signed-off-by: Andreas Florath <andreas@florath.net>
2017-03-07 18:43:09 +11:00
Ian Wienand
f1776d72aa Merge branch 'master' into merge-branch
Change-Id: Ib3c4b0030c94459a8c2eca836a16e42edf1accd4
2017-03-02 13:49:29 +11:00
Dirk Mueller
65c6da58c3 [suse] Handle pip-and-virtualenv install for opensuse
Similarly to Fedora we need to register the virtualenv package
with the rpm database to avoid things becoming messy later on
when package dependencies are conflicting with the from-source
install.

Change-Id: I67654fe5533e6086a17b38e2ae79a630a609ff92
2017-02-28 22:51:25 +01:00
Jenkins
2d919e7ca0 Merge "[suse] remove --no-confirm from zypper invocation" 2017-02-28 01:31:50 +00:00
Jenkins
f7669f60da Merge "gentoo: do not manually clean /tmp" 2017-02-28 01:31:44 +00:00
Bernard Cafarelli
fedea46c9e
pip-and-virtualenv: also handle rhel distros
The current pip install script only checked for centos/fedora
This causes setuptools errrors like (with pbr install):
"SyntaxError: '<' operator not allowed in environment markers"

Explictly list distro names in that situation

Change-Id: I5c894ab6152acf5441231acc1215fe00967f4f31
2017-02-21 16:09:13 +01:00
Dirk Mueller
661c4c32d2 [suse] remove --no-confirm from zypper invocation
this is basically the deprecated version of --non-interactive, which is
already being used, so this is duplicate and only causes warnings but
doesn't help with anything

Change-Id: Ic98790461636c4136b7005bdc31ce08f8ca81e0f
2017-02-20 20:48:50 +01:00
Pino Toscano
e2192fc7b7 gentoo: do not manually clean /tmp
disk-image-create already does that globally after the guest is
finalized, see lib/img-functions:finalise_base.  Thus, there is no need
to remove part of /tmp manually, and no other distro element does that.

Change-Id: Id3b72d41fce258af98b31976ef726d57a0a9fae4
2017-02-20 17:27:25 +01:00
Andreas Jaeger
b833960c69 Fix requirements update
Sphinx 1.5 fails with current pbr (needs new pbr release, see change
I52d45fa0a0d42de690d3a492068f7bb03483a224.

For now, set warnerrors to False to be able to use this.

Also, remove duplicate requirements line.

Also, update to new sphinx version to show that this works.

This is in reaction to the failure in
Ia184446fe34c49a48ca079c828157ea34e662d4b.

Change-Id: I9e7261c4124b71eeb6bddd9e21747b61bbdc16fa
2017-02-12 16:59:06 +01:00
Andreas Florath
003fc17208 Fix #1627402: fix grub2 element for ubuntu xenial
Looks that the special handling for Ubuntu is not needed any longer
(its a pity that there are no detailed comments...).

The grub2 element is a second implementation of the bootstrap element
- but because there are some features that come only here, e.g. efi
boot, it should be working as long as this is not implemented in the
bootloader element.

Change-Id: I74269116ea30b84f3259805720d5cd1616f960c5
Signed-off-by: Andreas Florath <andreas@florath.net>
Closes-Bug: #1627402
2017-02-11 18:44:18 +00:00
Ian Wienand
7a155e08bf Merge branch 'master' into merge-branch
Change-Id: I28e4c7837d84e8b66eff3d182666c5a87a9e3c9b
2017-02-09 13:35:53 +11:00
Andreas Florath
fdd2c4b236 Semi-automatic doc generation of element dependency
Currently there is no description of dependencies in the generated
documentation of the elements: therefore a user of an element does not
know which other elements are automatically included and e.g. which
configuration options are available. In addition there are some
copy&pastes of parts of the README.rst scattered thought different
Ubuntu and Debian specific elements.

This patch adds a semi-automatic generation of dependency information
of all elements.  Nevertheless these are not automatically included.
The author of the element's README.rst can decide if and where the
dependency information should appear and can use the descriptor

.. element_deps::

for this.

This patch adds the dependency information for some Debian and
Ubuntu patches - and creates the base for later removing the
duplicated parts.

A call is added to element_dependencies._find_all_elements() to
populate reverse dependencies for Element objects.

(This is a reworking of I31d2b6050b6c46fefe37378698e9a330025db430 for
the feature/v2 branch)

Change-Id: Iebb83916fed71565071246baa550849eef40560b
2017-02-09 09:50:30 +11:00
Ian Wienand
f3121f0fb7 Fix up doc errors
It seems that pbr's "warnerrors" isn't actually doing anything at the
moment (I680b448471e687919d202e8f2abe57f8ba3b22ee) meaning we're able
to commit docs with RST parser errors.  Fix all these up in some
minimal way (I have not audited content, just made it pass build).

Link the specs in so they're referenced from the top level.

Change-Id: Id67b9ea7ba8f49b43969c58ca3fb7fa1243538a4
2017-02-08 16:07:01 +11:00
Ian Wienand
3654aa4205 Fix up element doc generation
Turns out that you really have to have the elements dir symlinked
under the source directory for the globbing to match anything.  AFAICT
there's no way to add external directories to the sphinx build.

Change-Id: Iaa3576ef250822b0d57bebce1ebbe03e5bafa042
2017-02-08 14:32:43 +11:00
Jenkins
234f44d703 Merge "python-brickclient: accommodate python2/3 changes" 2017-02-08 00:32:29 +00:00
Jenkins
24628b135b Merge "Move Ubuntu specific use_tempaddr setting to ubuntu-common element" 2017-02-07 23:25:38 +00:00
Anshul Jain
dfc8fbf907 python-brickclient: accommodate python2/3 changes
Following a similar fix to pip-and-virtualenv element in
Ia730850a48e2478fd5461710a9d2619408725cd8, python-brickclient is also
required to be fixed to handle python3 only systems.

Change-Id: I442469beb5a20074425e7059eec7f78604d6cc5a
2017-02-08 09:33:02 +11:00
Andreas Florath
23ac49bf83 Move Ubuntu specific use_tempaddr setting to ubuntu-common element
By default (during boot) the use_tempaddr is set to <=0 for all
up-to date kernels.  Only Ubuntu installes a sysctl setting which
sets the use_tempaddr to 2 (/etc/sysctl.d/10-ipv6-privacy.conf) [1].
The 80-disable-rfc3041 overwrites this setting and sets
use_tempaddr back to 0.

Because this only affects Ubuntu it makes sense to move the script
to the ubuntu-common element. The other motivation for the move is,
to clear the base element that it can be removed.

[1] https://bugs.launchpad.net/ubuntu/+source/procps/+bug/1068756

Change-Id: Ibf261818ca8243874fde9eb3650bb65188fa228d
Signed-off-by: Andreas Florath <andreas@florath.net>
2017-02-07 20:41:13 +00:00
Jenkins
740c0a85df Merge "Move generation of dib_[environment|args] to manifest element" 2017-02-07 20:23:06 +00:00
Jenkins
15969b9c21 Merge "Preinstall pyOpenSSL" 2017-02-07 08:53:23 +00:00