Merge remote-tracking branch 'origin/master' into feature/v2

Change-Id: I796546f3873a0c9072215e63703aaa64753c5afd

doc/source/specs/README.rst

@@ -0,0 +1,82 @@
+=======
+README
+=======
+
+diskimage-builder Specifications
+================================
+
+
+This directory is used to hold approved design specifications for changes to
+the diskimage-builder project. Reviews of the specs are done in gerrit, using a
+similar workflow to how we review and merge changes to the code itself. For
+specific policies around specification review, refer to the end of this
+document.
+
+The layout of this directory is::
+
+  specs/v<major_version>/
+
+Where there are two sub-directories:
+
+- specs/v<major_version>/approved: specifications approved but not yet
+  implemented
+- specs/v<major_version>/implemented: implemented specifications
+- specs/v<major_version>/backlog: unassigned specifications
+
+The lifecycle of a specification
+--------------------------------
+
+Developers proposing a specification should propose a new file in the
+``approved`` directory. diskimage-builder-core will review the change in the
+usual manner for the project, and eventually it will get merged if a consensus
+is reached.
+
+When a specification has been implemented either the developer or someone
+from diskimage-builder-core will move the implemented specification from the
+``approved`` directory to the ``implemented`` directory. It is important to
+create redirects when this is done so that existing links to the approved
+specification are not broken. Redirects aren't symbolic links, they are
+defined in a file which sphinx consumes. An example is at
+``specs/v1/redirects``.
+
+This directory structure allows you to see what we thought about doing,
+decided to do, and actually got done. Users interested in functionality in a
+given release should only refer to the ``implemented`` directory.
+
+Example specifications
+----------------------
+
+You can find an example spec in ``specs/template.rst``.
+
+Backlog specifications
+----------------------
+
+Additionally, we allow the proposal of specifications that do not have a
+developer assigned to them. These are proposed for review in the same manner as
+above, but are added to::
+
+  specs/backlog/approved
+
+Specifications in this directory indicate the original author has either
+become unavailable, or has indicated that they are not going to implement the
+specification. The specifications found here are available as projects for
+people looking to get involved with diskimage-builder. If you are interested in
+claiming a spec, start by posting a review for the specification that moves it
+from this directory to the next active release. Please set yourself as the new
+`primary assignee` and maintain the original author in the `other contributors`
+list.
+
+Specification review policies
+=============================
+
+There are some special review policies which diskimage-builder-core will apply
+when reviewing proposed specifications. They are:
+
+Trivial specifications
+----------------------
+
+Proposed changes which are trivial (very small amounts of code) and don't
+change any of our public APIs are sometimes not required to provide a
+specification. The decision of whether something is trivial or not is a
+judgement made by the author or by consensus of the project cores, generally
+trying to err on the side of spec creation.

doc/source/specs/v1/approved/block-device-overview.rst

@@ -0,0 +1,173 @@
+..
+ This work is licensed under a Creative Commons Attribution 3.0 Unported
+ License.
+
+ http://creativecommons.org/licenses/by/3.0/legalcode
+
+==================
+Block Device Setup
+==================
+
+During the creation of a disk image (e.g. for a VM), there is the need
+to create, setup, configure and afterwards detach some kind of storage
+where the newly installed OS can be copied to or directly installed
+in.
+
+Problem description
+===================
+
+Currently dib is somewhat limited when it comes to setting up the
+block device: only one partition that can be used for data. LVM,
+encryption, multi-device or installation in an already existing block
+device is not supported.
+
+In addition there are several places (main, lib, elements) where the
+current way of handling the block device is used (spread knowledge and
+implementation).
+
+Also it is not possible, to implement the handling as different
+elements: it is not possible to pass results of one element in the
+same phase to another element.  Passing results from one phase to dib
+main is limited.
+
+Use Cases
+---------
+
+Possible use cases are (Actor: End User)
+
+#. User wants to use an existing block device to install an system
+   image in (like hd, iSCSI, SAN lun, ...).
+#. User wants that the system will be installed in multiple
+   partitions.
+#. User wants that the partitioning is done in a specific way
+   (optimize for speed, optimize for size).
+#. User wants to use LVM to install the system in (multiple PV, VG and
+   LV).
+#. User wants to encrypt a partition or a LV where (parts) of the
+   system are installed in.
+#. User wants specific file systems on specific partitions or LVs.
+
+Please note that these are only examples and details are described and
+implemented by different sub-specs.
+
+Proposed change
+===============
+
+Because of the current way to execute elements, it is not possible to
+have different elements for each feature.  Instead the changes will be
+implemented in a python module 'block_device' placed in the
+'diskimage_builder' directory.
+
+The entry-point mechanism is used to create callable python programs.
+These python programs are directly called from within the dib-main.
+
+There is the need to implement some functions or classes that take
+care about common used new functionality: e.g. storing state between
+phases, calling python sub-modules and passing arguments around.
+These functionality is implemented as needed - therefore it is most
+likely that the first patch implements also big parts of these
+infrastructure tasks.
+
+Alternatives
+------------
+#. Rewrite DIB in the way that elements can interchange data, even if
+   they are called during one phase.
+   This would influence the way all existing elements are called - and
+   might lead to unpredictable results.
+#. In addition there is the need to introduce at least two additional
+   phases: because major parts of the block device handling are
+   currently done in main and these must be passed over to elements.
+#. Another way would be to implement everything in one element:
+   this has the disadvantage, that other elements are not allowed to
+   use the 'block_device' phase any longer and also passing around
+   configuration and results is still not possible (see [3]).
+
+API impact
+----------
+
+Is described in the sub-elements.
+
+Security impact
+---------------
+
+Is described in the sub-elements.
+
+Other end user impact
+---------------------
+
+Paradigm changes from execute script to configuration for block_device
+phase.
+
+Performance Impact
+------------------
+
+Is described in the sub-elements.
+
+Implementation
+==============
+
+Assignee(s)
+-----------
+
+Primary assignee:
+  ansreas (andreas@florath.net)
+
+Would be good, if other people would support this - and specify and
+implement modules.
+
+Work Items
+----------
+
+This is an overview over changes in the block device layer.  Each
+level or module needs it's own spec.
+
+A first step is to reimplement the existing functionality, this
+contains:
+#. Level 0: Local Loop module
+   Use loop device on local image file
+   (This is already implemented: [1])
+#. Level 1: partitioning module
+   (This is already implemented: [4])
+#. Level 2: Create File System
+   An initial module uses ext4 only
+#. Level 3: Mounting
+
+As a second step the following functionality can be added:
+* Level 1: LVM module
+* Level 2: Create File System
+  (swap)
+* Level 2: Create File System
+  (vfat, needed for UEFI)
+* Level 2: Create File System
+  (xfs)
+
+Of course any other functionality can also be added when needed and wanted.
+
+Dependencies
+============
+
+Is described in the sub-elements.
+
+Testing
+=======
+
+Is described in the sub-elements.
+
+Documentation Impact
+====================
+
+Is described in the sub-elements.
+
+References
+==========
+
+[1] Implementation of Level 0: Local Loop module
+    https://review.openstack.org/319591
+[2] 'Block Device Setup for Disk-Image-Builder'
+    https://etherpad.openstack.org/p/C80jjsAs4x
+[3] partitioning-parted
+    This was a first try to implement everything
+    as an element - it shows the limitation.
+    https://review.openstack.org/313938
+[4] Implementation of Level 1: partitioning module
+    https://review.openstack.org/322671

doc/source/specs/v1/approved/v1-template.rst

@@ -0,0 +1,185 @@
+..
+ This work is licensed under a Creative Commons Attribution 3.0 Unported
+ License.
+
+ http://creativecommons.org/licenses/by/3.0/legalcode
+
+==============================================
+Example Spec - The title of your specification
+==============================================
+
+Introduction paragraph -- why are we doing anything? A single paragraph of
+prose that operators can understand. The title and this first paragraph
+should be used as the subject line and body of the commit message
+respectively.
+
+Some notes about the diskimage-bulider spec process:
+
+* Not all changes need a spec. For more information see
+  <add_url_here>
+
+* The aim of this document is first to define the problem we need to solve,
+  and second agree the overall approach to solve that problem.
+
+* This is not intended to be extensive documentation for a new feature.
+
+* You should aim to get your spec approved before writing your code.
+  While you are free to write prototypes and code before getting your spec
+  approved, its possible that the outcome of the spec review process leads
+  you towards a fundamentally different solution than you first envisaged.
+
+* But, API changes are held to a much higher level of scrutiny.
+  As soon as an API change merges, we must assume it could be in production
+  somewhere, and as such, we then need to support that API change forever.
+  To avoid getting that wrong, we do want lots of details about API changes
+  upfront.
+
+Some notes about using this template:
+
+* Your spec should be in ReSTructured text, like this template.
+
+* Please wrap text at 79 columns.
+
+* Please do not delete any of the sections in this template.  If you have
+  nothing to say for a whole section, just write: None
+
+* For help with syntax, see http://sphinx-doc.org/rest.html
+
+* If you would like to provide a diagram with your spec, ascii diagrams are
+  required.  http://asciiflow.com/ is a very nice tool to assist with making
+  ascii diagrams.  The reason for this is that the tool used to review specs is
+  based purely on plain text.  Plain text will allow review to proceed without
+  having to look at additional files which can not be viewed in gerrit.  It
+  will also allow inline feedback on the diagram itself.
+
+
+Problem description
+===================
+
+A detailed description of the problem. What problem is this blueprint
+addressing?
+
+Use Cases
+---------
+
+What use cases does this address? What impact on actors does this change have?
+Ensure you are clear about the actors in each use case: Developer, End User,
+etc.
+
+Proposed change
+===============
+
+Here is where you cover the change you propose to make in detail. How do you
+propose to solve this problem?
+
+If this is one part of a larger effort make it clear where this piece ends. In
+other words, what's the scope of this effort?
+
+At this point, if you would like to just get feedback on if the problem and
+proposed change fit in diskimage-builder, you can stop here and post this for
+review to get preliminary feedback. If so please say:
+Posting to get preliminary feedback on the scope of this spec.
+
+Alternatives
+------------
+
+What other ways could we do this thing? Why aren't we using those? This doesn't
+have to be a full literature review, but it should demonstrate that thought has
+been put into why the proposed solution is an appropriate one.
+
+API impact
+----------
+
+Describe how this will effect our public interfaces. Will this be adding new
+environment variables? Deprecating existing ones? Adding a new command line
+argument?
+
+Security impact
+---------------
+
+Describe any potential security impact on the system.
+
+Other end user impact
+---------------------
+
+Aside from the API, are there other ways a user will interact with this
+feature?
+
+Performance Impact
+------------------
+
+Describe any potential performance impact on the system, for example
+how often will new code be called, does it perform any intense processing
+or data manipulation.
+
+Implementation
+==============
+
+Assignee(s)
+-----------
+
+Who is leading the writing of the code? Or is this a blueprint where you're
+throwing it out there to see who picks it up?
+
+If more than one person is working on the implementation, please designate the
+primary author and contact.
+
+Primary assignee:
+  <launchpad-id or None>
+
+Other contributors:
+  <launchpad-id or None>
+
+Work Items
+----------
+
+Work items or tasks -- break the feature up into the things that need to be
+done to implement it. Those parts might end up being done by different people,
+but we're mostly trying to understand the timeline for implementation.
+
+
+Dependencies
+============
+
+* Include specific references to specs in diskimage-builder or in other
+  projects, that this one either depends on or is related to.
+
+* If this requires functionality of another project that is not currently used
+  by diskimage-builder document that fact.
+
+
+Testing
+=======
+
+Please discuss the important scenarios needed to test here, as well as
+specific edge cases we should be ensuring work correctly. For each
+scenario please specify if this requires specialized hardware, or software.
+
+Is this untestable in gate given current limitations (specific hardware /
+software configurations available)? If so, are there mitigation plans (gate
+enhancements, etc).
+
+
+Documentation Impact
+====================
+
+Which audiences are affected most by this change, and which documentation
+files need to be changed. Do we need to add information about this change to
+the developer guide, the user guide, certain elements, etc.
+
+References
+==========
+
+Please add any useful references here. You are not required to have any
+reference. Moreover, this specification should still make sense when your
+references are unavailable. Examples of what you could include are:
+
+* Links to mailing list or IRC discussions
+
+* Links to notes from a summit session
+
+* Links to relevant research, if appropriate
+
+* Related specifications as appropriate
+
+* Anything else you feel it is worthwhile to refer to

elements/base/install.d/99-dkms

@@ -1,9 +0,0 @@
-#!/bin/bash
-
-if [ ${DIB_DEBUG_TRACE:-0} -gt 0 ]; then
-    set -x
-fi
-set -eu
-set -o pipefail
-
-install-packages -m base dkms_package

elements/base/pkg-map

@@ -3,14 +3,10 @@
     "redhat": {
       "iscsi_package": "iscsi-initiator-utils"
     },
-    "suse": {
-      "dkms_package": ""
-    },
     "gentoo": {
       "ccache_package": "dev-util/ccache",
       "curl": "net-misc/curl",
       "dhcp_client": "net-misc/dhcp",
-      "dkms_package": "",
       "extlinux": "sys-boot/syslinux",
       "git": "dev-vcs/git",
       "grub_bios": "sys-boot/grub",
@@ -31,7 +27,6 @@
   },
   "default": {
     "ccache_package": "ccache",
-    "dkms_package": "dkms",
     "iscsi_package": "open-iscsi"
   }
 }

elements/centos-minimal/element-deps

@@ -1,3 +1,2 @@
-epel
 yum-minimal
 

elements/centos/element-deps

@@ -1,6 +1,5 @@
 cache-url
 dib-run-parts
-epel
 redhat-common
 rpm-distro
 yum

elements/centos7/element-deps

@@ -1,6 +1,5 @@
 cache-url
 dib-run-parts
-epel
 redhat-common
 rpm-distro
 source-repositories

elements/mellanox/element-deps

@@ -0,0 +1,2 @@
+package-installs
+pkg-map

elements/mellanox/init.d/01-mellanox

@@ -1,3 +1,7 @@
 # extra load for mellanox
 modprobe mlx4_en
-
+modprobe mlx4_ib
+modprobe ib_ipoib
+modprobe mlx5_ib
+modprobe ib_umad
+modprobe ib_uverbs

elements/mellanox/install.d/65-mellanox

@@ -10,4 +10,12 @@ set -o pipefail
 home=$(dirname $0)
 
 install -m 0644 -o root -g root $home/mellanox-rules.udev /etc/udev/rules.d/81-mellanox.rules
+
+# needed kernel modules; mlx4_en mlx4_ib ib_ipoib mlx5_ib ib_umad ib_uverbs
+# mlx5_core loaded by mlx5_ib
 echo "mlx4_en" >>/etc/modules
+echo "mlx4_ib" >>/etc/modules
+echo "ib_ipoib" >>/etc/modules
+echo "mlx5_ib" >>/etc/modules
+echo "ib_umad" >>/etc/modules
+echo "ib_uverbs" >>/etc/modules

elements/mellanox/install.d/mellanox-rules.udev

@@ -1,6 +1,11 @@
 ACTION!="add", GOTO="drivers_end"
 
 SUBSYSTEM=="net", RUN+="/sbin/modprobe mlx4_en"
+SUBSYSTEM=="net", RUN+="/sbin/modprobe mlx4_ib"
+SUBSYSTEM=="net", RUN+="/sbin/modprobe ib_ipoib"
+SUBSYSTEM=="net", RUN+="/sbin/modprobe mlx5_ib"
+SUBSYSTEM=="net", RUN+="/sbin/modprobe ib_umad"
+SUBSYSTEM=="net", RUN+="/sbin/modprobe ib_uverbs"
 
 LABEL="drivers_end"
 

elements/mellanox/package-installs.yaml

@@ -0,0 +1,26 @@
+dkms:
+ibacm:
+ibutils:
+ibverbs-utils:
+infiniband-diags:
+libibcm:
+libibcommon:
+libibmad:
+libibumad:
+libibverbs:
+libibverbs-runtime:
+libmlx4:
+libmlx4-dev:
+libmlx5:
+librdmacm:
+librdmacm-dev:
+librdmacm-runtime:
+mstflint:
+opensm:
+pciutils:
+perftest:
+qperf:
+rdma:
+rpm-build:
+srptools:
+vlan:

elements/mellanox/pkg-map

@@ -0,0 +1,67 @@
+{
+    "family": {
+        "debian":{
+            "dkms": "dkms",
+            "libibverbs": "libibverbs*",
+            "ibacm": "ibacm",
+            "librdmacm": "librdmacm*",
+            "libmlx4": "libmlx4*",
+            "libmlx5": "libmlx5*",
+            "libibcm": "libibcm.*",
+            "libibmad": "libibmad.*",
+            "libibumad": "libibumad*",
+            "libmlx4-dev": "libmlx4-dev",
+            "librdmacm-dev": "librdmacm-dev",
+            "rdma": "rdmacm-utils",
+            "vlan": "vlan",
+            "ibverbs-utils": "ibverbs-utils"
+        },
+        "redhat": {
+            "libibverbs": "libibverbs.*",
+            "librdmacm": "librdmacm.*",
+            "libmlx4": "libmlx4.*",
+            "libmlx5": "libmlx5.*",
+            "libibcm": "libibcm.*",
+            "libibmad": "libibmad.*",
+            "libibumad": "libibumad.*",
+            "rdma": "rdma",
+            "qperf": "qperf",
+            "pciutils": "pciutils"
+        },
+        "suse":{
+            "libibverbs": "libibverbs",
+            "librdmacm": "librdmacm",
+            "libmlx4": "libmlx4",
+            "libmlx5": "libmlx5",
+            "libibcm": "libibcm",
+            "libibmad": "libibmad",
+            "libibumad": "libibumad",
+            "rdma": "rdma",
+            "qperf": "qperf",
+            "rpm-build": "rpm-build",
+            "libibverbs-runtime": "libibverbs-runtime",
+            "librdmacm-runtime": "librdmacm-runtime",
+            "libibcommon": "libibcommon"
+        }
+    },
+    "default": {
+        "infiniband-diags": "infiniband-diags",
+        "mstflint": "mstflint",
+        "opensm": "opensm",
+        "srptools": "srptools",
+        "libmlx4-dev": "",
+        "librdmacm-dev": "",
+        "ibutils": "ibutils",
+        "perftest": "perftest",
+        "vlan": "",
+        "pciutils": "",
+        "ibverbs-utils": "",
+        "rpm-build": "",
+        "libibverbs-runtime": "",
+        "librdmacm-runtime": "",
+        "ibacm": "",
+        "qperf": "",
+        "dkms": "",
+        "libibcommon": ""
+    }
+}

elements/mellanox/udev.d/81-mellanox-drivers.rules

@@ -1,6 +1,11 @@
 ACTION!="add", GOTO="drivers_end"
 
 SUBSYSTEM=="net", RUN+="/sbin/modprobe mlx4_en"
+SUBSYSTEM=="net", RUN+="/sbin/modprobe mlx4_ib"
+SUBSYSTEM=="net", RUN+="/sbin/modprobe ib_ipoib"
+SUBSYSTEM=="net", RUN+="/sbin/modprobe mlx5_ib"
+SUBSYSTEM=="net", RUN+="/sbin/modprobe ib_umad"
+SUBSYSTEM=="net", RUN+="/sbin/modprobe ib_uverbs"
 
 LABEL="drivers_end"
 

elements/yum-minimal/package-installs.yaml

@@ -7,3 +7,4 @@ man-pages:
 redhat-lsb-core:
 selinux-policy:
 selinux-policy-targeted:
+libselinux-python:

elements/yum-minimal/root.d/08-yum-chroot

@@ -156,6 +156,7 @@ function _install_pkg_manager {
         fi
 
         sudo -E yum -y \
+            --disableexcludes=all \
             --setopt=cachedir=$YUM_CACHE/$ARCH/$DIB_RELEASE \
             --setopt=reposdir=$TARGET_ROOT/etc/yum.repos.d \
             --releasever=$DIB_RELEASE \