Merge branch 'master' into merge-branch

Change-Id: I28e4c7837d84e8b66eff3d182666c5a87a9e3c9b

diskimage_builder/elements/base/extra-data.d/50-store-build-settings

@@ -1,13 +0,0 @@
-#!/bin/bash
-# Store the build-time environment and command line arguments
-
-if [ ${DIB_DEBUG_TRACE:-0} -gt 0 ]; then
-    set -x
-fi
-set -eu
-set -o pipefail
-
-[ -n "$TMP_HOOKS_PATH" ] || die "Temp hook path not set"
-
-echo "$DIB_ENV" > $TMP_HOOKS_PATH/dib_environment
-echo "$DIB_ARGS" > $TMP_HOOKS_PATH/dib_arguments

diskimage_builder/elements/base/install.d/50-store-build-settings

@@ -1,16 +0,0 @@
-#!/bin/bash
-# Store build-time environment and command line arguments
-
-if [ ${DIB_DEBUG_TRACE:-0} -gt 0 ]; then
-    set -x
-fi
-set -eu
-set -o pipefail
-
-if [ -e "/tmp/in_target.d/dib_environment" ]; then
-    cp /tmp/in_target.d/dib_environment /etc/
-fi
-
-if [ -e "/tmp/in_target.d/dib_arguments" ]; then
-    cp /tmp/in_target.d/dib_arguments /etc/
-fi

diskimage_builder/elements/cloud-init/README.rst

@@ -1,6 +1,6 @@
-========
+==========
 cloud-init
-========
+==========
 
 Install's and enables cloud-init for systems that don't come with it
 pre-installed

diskimage_builder/elements/debian-minimal/README.rst

@@ -2,48 +2,51 @@
 debian-minimal
 ==============
 
-Create a minimal image based on Debian. We default to unstable but `DIB_RELEASE`
+Create a minimal image based on Debian. We default to unstable but
-can be set to any series of Debian.
+``DIB_RELEASE`` can be set to any series of Debian.
 
 There are two ways to configure apt-sources:
 
 1. Using the standard way of defining the default, backports, updates
    and security repositories is the default. In this case you can
    overwrite the two environment variables to adapt the behavior:
-   `DIB_DISTRIBUTION_MIRROR`: the mirror to use
-      default: http://ftp.us.debian.org/debian
-   `DIB_DEBIAN_COMPONENTS`: (default) `main`
-      a comma separated list of components. For Debian this can be
-      e.g. `main,contrib,non-free`.
 
-   Note it is not recommended to use http://httpredir.debian.org/ for
+   * ``DIB_DISTRIBUTION_MIRROR``: the mirror to use (default:
-   `DIB_DISTRIBUTION_MIRROR` due to how unreliable it is.  Be sure to
+     `<http://ftp.us.debian.org/debian>`__)
-   select a mirror from the official mirror list:
 
-       https://www.debian.org/mirror/list
+   * ``DIB_DEBIAN_COMPONENTS``: (default: ``main``) a comma
+     separated list of components. For Debian this can be
+     e.g. ``main,contrib,non-free``.
 
-   By default only `main` component is used. If
+   Note it is not recommended to use
-   `DIB_DEBIAN_COMPONENTS` (comma separated) from the `debootstrap`
+   `<http://httpredir.debian.org/>`__ for ``DIB_DISTRIBUTION_MIRROR``
-   element has been set, that list of components will be used instead.
+   due to how unreliable it is.  Be sure to select a mirror from the
+   official mirror list at `<https://www.debian.org/mirror/list>`__
 
-   Backports, updates and security are included unless `DIB_RELEASE`
+   By default only the ``main`` component is used. If
-   is `unstable`.
+   ``DIB_DEBIAN_COMPONENTS`` (comma separated) from the
+   ``debootstrap`` element has been set, that list of components will
+   be used instead.
+
+   Backports, updates and security are included unless ``DIB_RELEASE``
+   is ``unstable``.
+
+2. Complete configuration given in the variable ``DIB_APT_SOURCES_CONF``.
 
-2. Complete configuration given in the variable
-   `DIB_APT_SOURCES_CONF`.
    Each line contains exactly one entry for the sources.list.d
-   directory.
+   directory.  The first word must be the logical name (which is used
-   The first word must be the logical name (which is used as file name
+   as file name with ``.list`` automatically appended), followed by a
-   with `.list` automatically appended), followed by a colon `:`,
+   colon ``:``, followed by the complete repository specification.
-   followed by the complete repository specification.
+
-   Example:
+   .. code-block:: bash
-   DIB_APT_SOURCES_CONF=\
+
-       "default:deb http://10.0.0.10/ stretch main contrib
+      DIB_APT_SOURCES_CONF=\
-        mysecurity:deb http://10.0.0.10/ stretch-security main contrib"
+        "default:deb http://10.0.0.10/ stretch main contrib
+         mysecurity:deb http://10.0.0.10/ stretch-security main contrib"
 
 If necessary, a custom apt keyring and debootstrap script can be
-supplied to the `debootstrap` command via `DIB_APT_KEYRING` and
+supplied to the ``debootstrap`` command via ``DIB_APT_KEYRING`` and
-`DIB_DEBIAN_DEBOOTSTRAP_SCRIPT` respectively. Both options require the
+``DIB_DEBIAN_DEBOOTSTRAP_SCRIPT`` respectively. Both options require the
 use of absolute rather than relative paths.
 
 Use of this element will also require the tool 'debootstrap' to be
@@ -51,26 +54,26 @@ available on your system. It should be available on Ubuntu, Debian,
 and Fedora. It is also recommended that the 'debian-keyring' package
 be installed.
 
-The `DIB_OFFLINE` or more specific `DIB_DEBIAN_USE_DEBOOTSTRAP_CACHE`
+The ``DIB_OFFLINE`` or more specific ``DIB_DEBIAN_USE_DEBOOTSTRAP_CACHE``
 variables can be set to prefer the use of a pre-cached root filesystem
 tarball.
 
-The `DIB_DEBOOTSTRAP_EXTRA_ARGS` environment variable may be used to
+The ``DIB_DEBOOTSTRAP_EXTRA_ARGS`` environment variable may be used to
 pass extra arguments to the debootstrap command used to create the
-base filesystem image. If --keyring is is used in `DIB_DEBOOTSTRAP_EXTRA_ARGS`,
+base filesystem image. If --keyring is is used in ``DIB_DEBOOTSTRAP_EXTRA_ARGS``,
-it will override `DIB_APT_KEYRING` if that is used as well.
+it will override ``DIB_APT_KEYRING`` if that is used as well.
 
-For further information about `DIB_DEBIAN_DEBOOTSTRAP_SCRIPT` ,
+For further information about ``DIB_DEBIAN_DEBOOTSTRAP_SCRIPT`` ,
-`DIB_DEBIAN_USE_DEBOOTSTRAP_CACHE` and `DIB_DEBOOTSTRAP_EXTRA_ARGS`
+``DIB_DEBIAN_USE_DEBOOTSTRAP_CACHE`` and ``DIB_DEBOOTSTRAP_EXTRA_ARGS``
 please consult "README.rst" of the debootstrap element.
 
 -------------------
 Note on ARM systems
 -------------------
 
-Because there is not a one-to-one mapping of `ARCH` to a kernel package, if
+Because there is not a one-to-one mapping of ``ARCH`` to a kernel package, if
 you are building an image for ARM on debian, you need to specify which kernel
-you want in the environment variable `DIB_ARM_KERNEL`. For instance, if you want
+you want in the environment variable ``DIB_ARM_KERNEL``. For instance, if you want
-the `linux-image-mx5` package installed, set `DIB_ARM_KERNEL` to `mx5`.
+the ``linux-image-mx5`` package installed, set ``DIB_ARM_KERNEL`` to ``mx5``.
 
 .. element_deps::

diskimage_builder/elements/fedora/package-installs.yaml

@@ -17,7 +17,7 @@ openssl:
 # dependency of the python-paste package needed for the heat element,
 # this seems to be conflicting and causing the image building process to
 # fail. The problem is hapenning on a Fedora 18 system.
-python-pyopenssl:
+pyOpenSSL:
 
 # Workaround for:
 # https://bugzilla.redhat.com/show_bug.cgi?id=1066983

diskimage_builder/elements/manifests/README.rst

@@ -1,13 +1,25 @@
 =========
 manifests
 =========
-Copy any manifests generated into the build area post-image creation
 
-This element should be a dependency of any element that writes a manifest
+An framework for saving manifest information generated during the
-into the `DIB_MANIFEST_IMAGE_DIR`, which defaults to `/etc/dib-manifests`.
+build for later inspection.  Manifests are kept in the final image and
-This is created in extra-data.d rather than pre-install.d to allow the
+also copied to the build area post-image creation.
-source-repositories element to make use of it
 
-The manifests are copied to `DIB_MANIFEST_SAVE_DIR`, which defaults to
+Elements that wish to save any form of manifest should depend on this
-`${IMAGE_NAME}.d/`, resulting in the manifests being available as
+element and can save their data to into the ``DIB_MANIFEST_IMAGE_DIR`` (
-`${IMAGE_NAME}.d/dib-manifests` by default
+which defaults to ``/etc/dib-manifests``).  Note this is created in
+``extra-data.d`` rather than ``pre-install.d`` to allow the
+``source-repositories`` element to make use of it
+
+The manifests are copied to ``DIB_MANIFEST_SAVE_DIR``, which defaults
+to ``${IMAGE_NAME}.d/``, resulting in the manifests being available as
+``${IMAGE_NAME}.d/dib-manifests`` by default after the build.
+
+Extra status
+------------
+
+This element will also add the files ``dib_environment`` and
+``dib_arguments`` to the manifest recording the ``diskimage-builder``
+specific environment (``DIB_*`` variables) and command-line arguments
+respectively.

diskimage_builder/elements/manifests/cleanup.d/01-copy-manifests-dir

@@ -1,6 +1,7 @@
 #!/bin/bash
 #
 # Copyright 2014 Hewlett-Packard Development Company, L.P.
+# Copyright 2017 Andreas Florath (andreas@florath.net)
 #
 # Licensed under the Apache License, Version 2.0 (the "License"); you may
 # not use this file except in compliance with the License. You may obtain
@@ -21,15 +22,15 @@ fi
 set -eu
 set -o pipefail
 
-if [ -d $TMP_MOUNT_PATH/${DIB_MANIFEST_IMAGE_DIR} ]; then
+MANIFEST_IMAGE_PATH=${TMP_MOUNT_PATH}/${DIB_MANIFEST_IMAGE_DIR}
-    # Move the dib_environment and dib_arguments files into the manifests dir
+
-    if [ -e $TMP_MOUNT_PATH/etc/dib_arguments ]; then
+# Double check: directory must be created in extra-data.d/20-manifest-dir
-        sudo mv $TMP_MOUNT_PATH/etc/dib_arguments $TMP_MOUNT_PATH/${DIB_MANIFEST_IMAGE_DIR}
+[ -d ${MANIFEST_IMAGE_PATH} ] || {
-    fi
+    echo "Error: MANIFEST_IMAGE_PATH [${MANIFEST_IMAGE_PATH}] does not exist";
-    if [ -e $TMP_MOUNT_PATH/etc/dib_environment ]; then
+    exit 1; }
-        sudo mv $TMP_MOUNT_PATH/etc/dib_environment $TMP_MOUNT_PATH/${DIB_MANIFEST_IMAGE_DIR}
+
-    fi
+echo "$DIB_ENV" | sudo dd of=${MANIFEST_IMAGE_PATH}/dib_environment # dib-lint: safe_sudo
-    mkdir -p ${DIB_MANIFEST_SAVE_DIR}
+echo "$DIB_ARGS" | sudo dd of=${MANIFEST_IMAGE_PATH}/dib_arguments  # dib-lint: safe_sudo
-    cp --no-preserve=ownership -rv $TMP_MOUNT_PATH/${DIB_MANIFEST_IMAGE_DIR} \
+
-        ${DIB_MANIFEST_SAVE_DIR}
+mkdir -p ${DIB_MANIFEST_SAVE_DIR}
-fi
+cp --no-preserve=ownership -rv ${MANIFEST_IMAGE_PATH} ${DIB_MANIFEST_SAVE_DIR}

diskimage_builder/elements/package-installs/bin/package-installs-v2

@@ -14,6 +14,8 @@
 # License for the specific language governing permissions and limitations
 # under the License.
 
+from __future__ import print_function
+
 import argparse
 import json
 import subprocess
@@ -23,18 +25,19 @@ import sys
 # run a command, return output
 #  if follow is set, output will be echoed to stdout
 def process_output(cmdline, follow=False):
-    proc = subprocess.Popen(cmdline, stdout=subprocess.PIPE)
+    proc = subprocess.Popen(cmdline, stdout=subprocess.PIPE,
+                            universal_newlines=True)
     if follow:
         print("Running command: %s" % cmdline)
         out = ""
         with proc.stdout:
-            for line in iter(proc.stdout.readline, b''):
+            for line in iter(proc.stdout.readline, ''):
-                out += line.decode('utf-8')
+                out += line
-                print("> %s" % line.strip())
+                print("> %s" % line, end="")
         proc.wait()
-        print("> -- done")
+        print("returncode: %d" % proc.returncode)
     else:
-        out = proc.communicate()[0].decode('utf-8')
+        out = proc.communicate()[0]
 
     if proc.returncode:
         e = subprocess.CalledProcessError(proc.returncode, cmdline)

diskimage_builder/elements/pip-and-virtualenv/install.d/pip-and-virtualenv-source-install/04-install-pip

@@ -19,7 +19,7 @@ if [[ $DISTRO_NAME =~ (centos|fedora) ]]; then
     ${YUM:-yum} install -y python-virtualenv python-pip python-setuptools
 
     # install pip; this overwrites packaged pip
-    python /tmp/get-pip.py
+    /usr/local/bin/dib-python /tmp/get-pip.py
 
     # pip and setuptools are closely related; we want to ensure the
     # latest for sanity.  Because distro packages don't include enough
@@ -50,6 +50,6 @@ if [[ $DISTRO_NAME =~ (centos|fedora) ]]; then
     fi
     echo "exclude=python-virtualenv,python-pip,python-setuptools" >> ${conf}
 else
-    python /tmp/get-pip.py
+    /usr/local/bin/dib-python /tmp/get-pip.py
     pip install virtualenv
 fi

diskimage_builder/elements/python-brickclient/README.rst

@@ -1,34 +1,48 @@
 python-brickclient
 ==================
 
-* This element is aimed for providing cinder local attach/detach functionality.
+* This element is aimed for providing cinder local attach/detach
+  functionality.
+
 * Currently the feature has a dependency on a known bug
-  "https://launchpad.net/bugs/1623549", which has been resolved and will be part
+  `<https://launchpad.net/bugs/1623549>`__, which has been resolved
-  of the upstream with the next release of python-brick-cinderclient-ext.
+  and will be part of the upstream with the next release of
-  Note: Current version of python-brick-cinderclient-ext i.e. 0.2.0 requires and update
+  ``python-brick-cinderclient-ext``.  Note: Current version of
-  to be made in Line32 fo below script.
+  ``python-brick-cinderclient-ext`` i.e. 0.2.0 requires and update to
-  /usr/share/python-brickclient/venv/lib/python2.7/site-packages/brick_cinderclient_ext/__init__.py
+  be made in Line32 for
-  update "brick-python-cinderclient-ext" to "python-brick-cinderclient-ext".
+  ``/usr/share/python-brickclient/venv/lib/python2.7/site-packages/brick_cinderclient_ext/__init__.py``:
+  update ``brick-python-cinderclient-ext`` to
+  ``python-brick-cinderclient-ext``.
 
-* Usage:
+Usage
-  Pass the below shell script to parameter 'user-data' and set 'config-drive=true'
+-----
-  at the time of provisioning the node via nova-boot to make cinder local
-  attach/detach commands talk to your cloud controller.
-  [Example of Config Drive Script]
-    #!/bin/bash
-    FILE="/etc/bash.bashrc"
-    [ ! -f "$FILE" ] && touch "$FILE"
-    echo 'export OS_AUTH_URL="http://<controller_ip>:5000/v2.0"' >> "$FILE"
-    echo 'export OS_PASSWORD="password"'  >> "$FILE"
-    echo 'export OS_USERNAME="demo"' >> "$FILE"
-    echo 'export OS_TENANT_NAME="demo"'  >> "$FILE"
-    echo 'export OS_PROJECT_NAME="demo"' >> "$FILE"
-    exec bash
-  To attach: /usr/share/python-brickclient/venv/bin/cinder local-attach <volume_id>
-  To detach: /usr/share/python-brickclient/venv/bin/cinder local-detach <volume_id>
 
-* Alternatively, the same action can be completed manually at the node which does
+Pass the below shell script to parameter ``user-data`` and set
-  not require setting up of config drive such as:
+``config-drive=true`` at the time of provisioning the node via
-  /usr/share/python-brickclient/venv/bin/cinder --os-username demo --os-password \
+nova-boot to make cinder local attach/detach commands talk to your
-  password  --os-tenant-name demo --os-project-name demo \
+cloud controller.
-  --os-auth-url=http://<controller_ip>:5000/v2.0 local-attach <volume_id>
+
+.. code-block:: bash
+
+   #!/bin/bash
+   FILE="/etc/bash.bashrc"
+   [ ! -f "$FILE" ] && touch "$FILE"
+   echo 'export OS_AUTH_URL="http://<controller_ip>:5000/v2.0"' >> "$FILE"
+   echo 'export OS_PASSWORD="password"'  >> "$FILE"
+   echo 'export OS_USERNAME="demo"' >> "$FILE"
+   echo 'export OS_TENANT_NAME="demo"'  >> "$FILE"
+   echo 'export OS_PROJECT_NAME="demo"' >> "$FILE"
+   exec bash
+
+ To attach: ``/usr/share/python-brickclient/venv/bin/cinder local-attach <volume_id>``
+ To detach: ``/usr/share/python-brickclient/venv/bin/cinder local-detach <volume_id>``
+
+Alternatively, the same action can be completed manually at the node
+which does not require setting up of config drive such as:
+
+.. code-block:: bash
+
+   /usr/share/python-brickclient/venv/bin/cinder \
+     --os-username demo --os-password password \
+     --os-tenant-name demo --os-project-name demo \
+     --os-auth-url=http://<controller_ip>:5000/v2.0 local-attach <volume_id>

diskimage_builder/elements/python-brickclient/package-installs.yaml

@@ -1,3 +1,8 @@
 libssl-dev:
 libffi-dev:
 python-dev:
+  installtype: source
+  dib_python_version: 2
+python3-dev:
+  installtype: source
+  dib_python_version: 3

diskimage_builder/elements/python-brickclient/pkg-map

@@ -1,13 +1,15 @@
 {
   "family": {
     "redhat": {
-      "python-dev": "python2-devel",
+      "python2-dev": "python2-devel",
+      "python3-dev": "python3-devel",
       "libssl-dev": "openssl-devel",
       "libffi-dev": "libffi-devel"
     }
   },
   "default": {
-    "python-dev": "python-dev",
+    "python2-dev": "python2-dev",
+    "python3-dev": "python3-dev",
     "libssl-dev": "libssl-dev",
     "libffi-dev": "libffi-dev"
   }

diskimage_builder/elements/redhat-common/bin/map-packages

@@ -22,6 +22,13 @@ import sys
 # Manually maintained for brevity; consider making this compiled from
 # distromatch or other rich data sources.
 # Debian name on the left, Fedora/RHEL on the right.
+
+#
+#           !!! DO NOT ADD ANY ENTRIES TO THIS FILE !!!
+#
+# This global list has been deprecated by the pkg-map element.  New
+# package mappings should go in pkg-map files inside each element.
+#
 package_map = {
     'apache2': 'httpd',
     'arping': 'iputils',
@@ -81,13 +88,30 @@ package_map = {
     'openstack-neutron-dhcp-agent': 'openstack-neutron',
 }
 
-print("WARNING: map-packages is deprecated.  Please use the pkg-map element.",
+deprecated = []
-      file=sys.stderr)
-
 for arg in sys.argv[1:]:
     if arg not in package_map and arg.endswith('-dev'):
         # convert -dev into devel
-        print('%s%s' % (arg, 'el'))
+        converted = '%s%s' % (arg, 'el')
+        deprecated.append((arg, converted))
+        print(converted)
     else:
-        print(package_map.get(arg, arg))
+        converted = package_map.get(arg, arg)
+        if converted != arg:
+            deprecated.append((arg, converted))
+        print(converted)
+
+if deprecated:
+    print("WARNING: The following packages were re-mapped by "
+          "redhat-common map-packages\n"
+          "They should be converted to pkg-map:", file=sys.stderr)
+    for arg, converted in deprecated:
+        print(" %s -> %s" % (arg, converted), file=sys.stderr)
+
 sys.exit(0)
+
+
+# Tell emacs to use python-mode
+# Local variables:
+# mode: python
+# End:

diskimage_builder/elements/rhel/bin/map-packages

@@ -18,6 +18,14 @@ import sys
 # Manually maintained for brevity; consider making this compiled from
 # distromatch or other rich data sources.
 # Debian name on the left, RHEL on the right.
+
+#
+#           !!! DO NOT ADD ANY ENTRIES TO THIS FILE !!!
+#
+#  This global list has been deprecated by the pkg-map element.  New
+#  package mappings should go in pkg-map files inside each element.
+#
+
 package_map = {
     'augeas-tools': 'augeas',
     'build-essential': 'make automake gcc gcc-c++ kernel-devel',
@@ -40,9 +48,18 @@ package_map = {
     'vlan': 'vconfig',
 }
 
-print("WARNING: map-packages is deprecated.  Please use the pkg-map element.",
+deprecated = []
-      file=sys.stderr)
-
 for arg in sys.argv[1:]:
-    print(package_map.get(arg, arg))
+    mapped = package_map.get(arg, arg)
+    if mapped != arg:
+        deprecated.append((arg, mapped))
+    print(mapped)
+
+if deprecated:
+    print("WARNING: The following packages were re-mapped by "
+          "rhel map-packages.\n"
+          "They should be converted to pkg-map:\n", file=sys.stderr)
+    for arg, converted in deprecated:
+        print(" %s -> %s" % (arg, converted), file=sys.stderr)
+
 sys.exit(0)

diskimage_builder/elements/rhel7/README.rst

@@ -13,7 +13,10 @@ https://access.redhat.com/downloads/content/69/ver=/rhel---7/7.1/x86_64/product-
 
 Then before running the image build, define DIB_LOCAL_IMAGE (replace the file
 name with the one downloaded, if it differs from the example)::
-  export DIB_LOCAL_IMAGE=rhel-guest-image-7.1-20150224.0.x86_64.qcow2
+
+.. code-block:: bash
+
+   export DIB_LOCAL_IMAGE=rhel-guest-image-7.1-20150224.0.x86_64.qcow2
 
 The downloaded file will then be used as the basis for any subsequent image
 builds.

diskimage_builder/elements/ubuntu-common/README.rst

@@ -0,0 +1,5 @@
+ubuntu-common
+=============
+
+This element holds configuration and scripts that are common for all
+Ubuntu images.

diskimage_builder/elements/ubuntu-common/install.d/80-disable-rfc3041

@@ -21,7 +21,17 @@ if [ ${DIB_DEBUG_TRACE:-0} -gt 0 ]; then
 fi
 set -e
 
-# This will disable the disable Privacy extensions for IPv6 (RFC3041)
+# This will disable the privacy extensions for IPv6 (RFC4941)
+# (RFC4941 obsoletes RFC3041)
+#
+# Note that depending on the implementation of how the sysctl
+# settings are applied there is a race condition: there might be
+# a small time-frame where the original Ubuntu settings from
+# /etc/sysctl.d/10-ipv6-privacy.conf are already applied but the
+# new settings from /etc/sysctl.d/99-cloudimg-ipv6.conf
+# not yet.
+# If during this time all started network interfaces will
+# use the privacy extension.
 cat > /etc/sysctl.d/99-cloudimg-ipv6.conf <<EOF
 # See https://bugs.launchpad.net/ubuntu/+source/procps/+bug/1068756
 net.ipv6.conf.all.use_tempaddr=0

diskimage_builder/elements/ubuntu-core/element-deps

@@ -1,3 +1,4 @@
 cache-url
 dib-run-parts
 dpkg
+ubuntu-common

diskimage_builder/elements/ubuntu-minimal/element-deps

@@ -1,2 +1,3 @@
 debootstrap
 package-installs
+ubuntu-common

diskimage_builder/elements/ubuntu/element-deps

@@ -3,3 +3,4 @@ cloud-init-datasources
 dib-run-parts
 dkms
 dpkg
+ubuntu-common

diskimage_builder/elements/yum/bin/install-packages

@@ -27,6 +27,9 @@ MAP_ELEMENT=
 # allow override for dnf, as shipped by default with >=F22
 YUM=${YUM:-yum}
 
+# save global xtrace state
+_xtrace=$(set +o | grep xtrace)
+
 SCRIPTNAME=$(basename $0)
 function show_options () {
     echo "Usage: $SCRIPTNAME [package ...]"
@@ -103,7 +106,7 @@ if [ -n "$WHITELIST" ]; then
         exit 0
     fi
 
-    echo "Running install-packages ${ACTION}. Package list: $PKGS"
+    echo "Running install-packages ${ACTION}."
 
     if [ "$ACTION" == "download" ]; then
         mkdir -p $DOWNLOAD_PATH
@@ -130,6 +133,7 @@ if [ -n "$WHITELIST" ]; then
     # point.
     #
     # [1] https://bugzilla.redhat.com/show_bug.cgi?id=965567
+    set -o xtrace
     ${YUM} -v -y $ACTION $EXTRA_ARGS $PKGS
 
     if [ "$ACTION" == "install" ]; then
@@ -138,6 +142,7 @@ if [ -n "$WHITELIST" ]; then
             dnf mark install $PKGS
         fi
     fi
+    $_xtrace
 
     # probably not the right place for this; but python-pip package on
     # fedora/rh calls pip "pip-python" while the rest of the work

diskimage_builder/elements/zypper-minimal/README.rst

@@ -3,7 +3,7 @@ zypper-minimal
 ==============
 Base element for creating minimal SUSE-based images
 
-This element is incomplete by itself so you probaby want to use it along
+This element is incomplete by itself so you probably want to use it along
 with the opensuse-minimal one. It requires 'zypper' to be installed on the
 host.
 

diskimage_builder/lib/common-functions

@@ -77,6 +77,7 @@ function copy_hooks_not_overwrite () {
     test -d $TMP_HOOKS_PATH/$_DIR || mkdir $TMP_HOOKS_PATH/$_DIR
     for _HOOK in $(ls $1); do
       if [ ! -f $TMP_HOOKS_PATH/$_DIR/$_HOOK ]; then
+        echo "Copying hooks $1/$_HOOK"
         cp -t $TMP_HOOKS_PATH/$_DIR -a $1/$_HOOK
       else
         echo "There is a duplicated hook in your elements: $_ELEMENT/$_DIR/$_HOOK"
@@ -220,6 +221,7 @@ function run_d() {
     check_element
     check_break before-$1 ${break_cmd:-bash}
     if [ -d ${TMP_HOOKS_PATH}/$1.d ] ; then
+        echo "Running hooks from ${TMP_HOOKS_PATH}/$1.d"
         if [ -n "$2" ]; then
             dib-run-parts ${TMP_HOOKS_PATH}/$1.d | tee $2
             if [[ ${PIPESTATUS[0]} != 0 ]]; then

diskimage_builder/lib/disk-image-create

@@ -52,7 +52,7 @@ function show_options () {
     echo "       File types should be comma separated. VHD outputting requires the vhd-util"
     echo "       executable be in your PATH. ACI outputting requires the ACI_MANIFEST "
     echo "       environment variable be a path to a manifest file."
-    echo "    -x -- turn on tracing (use -x -x for very detailed tracing)"
+    echo "    -x -- turn on tracing (use -x -x for very detailed tracing)."
     echo "    -u -- uncompressed; do not compress the image - larger but faster"
     echo "    -c -- clear environment before starting work"
     echo "    --checksum -- generate MD5 and SHA256 checksum files for the created image"
@@ -107,6 +107,7 @@ function show_version() {
     python -c "from diskimage_builder import version; print(version.version_info.version_string())"
 }
 
+DIB_DEBUG_TRACE=${DIB_DEBUG_TRACE:-0}
 INSTALL_PACKAGES=""
 IMAGE_TYPES=("qcow2")
 COMPRESS_IMAGE="true"
@@ -128,7 +129,7 @@ while true ; do
         -t) IFS="," read -a IMAGE_TYPES <<< "$2"; export IMAGE_TYPES ; shift 2 ;;
         -h|--help) show_options; exit 0;;
         --version) show_version; exit 0;;
-        -x) shift; export DIB_DEBUG_TRACE=$(( $DIB_DEBUG_TRACE + 1 )); set -x;;
+        -x) shift; DIB_DEBUG_TRACE=$(( $DIB_DEBUG_TRACE + 1 ));;
         -u) shift; export COMPRESS_IMAGE="";;
         -c) shift ; export CLEAR_ENV=1;;
         -n) shift; export SKIP_BASE="1";;
@@ -151,6 +152,8 @@ while true ; do
     esac
 done
 
+export DIB_DEBUG_TRACE
+
 export DIB_IMAGE_CACHE=${DIB_IMAGE_CACHE:-~/.cache/image-create}
 mkdir -p $DIB_IMAGE_CACHE
 
@@ -170,7 +173,6 @@ function _ps4 {
 export -f _ps4
 export PS4='+ $(_ps4):   '
 
-
 source $_LIB/img-defaults
 source $_LIB/common-functions
 source $_LIB/img-functions
@@ -189,6 +191,11 @@ if [ -z "$*" ]; then
 fi
 arg_to_elements "$@"
 
+# start tracing after most boilerplate
+if [ ${DIB_DEBUG_TRACE} -gt 0 ]; then
+    set -x
+fi
+
 if [ "${#IMAGE_TYPES[@]}" = "1" ]; then
   export IMAGE_NAME=${IMAGE_NAME%%\.${IMAGE_TYPES[0]}}
 fi
@@ -290,8 +297,8 @@ fi
 unmount_image
 mv $TMP_BUILD_DIR/mnt $TMP_BUILD_DIR/built
 
-# save xtrace state, as we want to turn it off to avoid spamming the
+# save xtrace state, as we always want to turn it off to avoid
-# logs with du output below.
+# spamming the logs with du output below.
 xtrace=$(set +o | grep xtrace)
 
 # temp file for holding du output

doc/source/developer/developing_elements.rst

@@ -350,10 +350,6 @@ exported in bash).
 Debugging elements
 ------------------
 
-The build-time environment and command line arguments are captured by the
-:doc:`../elements/base/README` element and written to ``/etc/dib_environment``
-and ``/etc/dib_arguments`` inside the image.
-
 Export ``break`` to drop to a shell during the image build. Break points can be
 set either before or after any of the hook points by exporting
 "break=[before|after]-hook-name". Multiple break points can be specified as a
@@ -366,6 +362,12 @@ comma-delimited string. Some examples:
 
 * ``break=after-error`` will break after an error during an in target hookpoint.
 
+The :doc:`../elements/manifests/README` element will make a range of
+manifest information generated by other elements available for
+inspection inside and outside the built image.  Environment and
+command line arguments are captured as described in the documentation
+and can be useful for debugging.
+
 Images are built such that the Linux kernel is instructed not to switch into
 graphical consoles (i.e. it will not activate KMS). This maximises
 compatibility with remote console interception hardware, such as HP's iLO.

doc/source/index.rst

@@ -8,7 +8,7 @@ It includes support for building images based on many major
 distributions and can produce cloud-images in all common formats
 (``qcow2``, ``vhd``, ``raw``, etc), bare metal file-system images and
 ram-disk images.  These images are composed from the many included
-`elements`; ``diskimage-builder`` acts as a framework to easily add
+``elements``; ``diskimage-builder`` acts as a framework to easily add
 your own elements for even further customization.
 
 ``diskimage-builder`` is used extensively by the `TripleO project
@@ -35,15 +35,15 @@ Issues
 
 Issues are tracked on launchpad at:
 
- * `https://bugs.launchpad.net/diskimage-builder/+bugs`
+ * `<https://bugs.launchpad.net/diskimage-builder/+bugs>`__
-   <https://bugs.launchpad.net/diskimage-builder/+bugs>
 
 
 Communication
 -------------
 
 Communication among the diskimage-builder developers happens on IRC in
-#openstack-dib on freenode and on the openstack-dev mailing list (openstack-dev@lists.openstack.org).
+``#openstack-dib`` on freenode and on the ``openstack-dev`` mailing list
+(``openstack-dev@lists.openstack.org``).
 
 
 Table of Contents
@@ -55,3 +55,4 @@ Table of Contents
    user_guide/index
    developer/index
    elements
+   specs/README

doc/source/specs/README.rst

@@ -1,10 +1,9 @@
-=======
+================================
-README
-=======
-
 diskimage-builder Specifications
 ================================
 
+Overview
+========
 
 This directory is used to hold approved design specifications for changes to
 the diskimage-builder project. Reviews of the specs are done in gerrit, using a
@@ -46,7 +45,7 @@ given release should only refer to the ``implemented`` directory.
 Example specifications
 ----------------------
 
-You can find an example spec in ``specs/template.rst``.
+You can find an example spec in :doc:`v1/approved/v1-template`
 
 Backlog specifications
 ----------------------
@@ -80,3 +79,11 @@ change any of our public APIs are sometimes not required to provide a
 specification. The decision of whether something is trivial or not is a
 judgement made by the author or by consensus of the project cores, generally
 trying to err on the side of spec creation.
+
+Approved Specifications
+=======================
+
+.. toctree::
+   :glob:
+
+   v1/approved/*

doc/source/specs/v1/approved/block-device-lvl1-partitioning.rst

@@ -20,7 +20,9 @@ The implementation for this proposed changed already exists, was
 discussed and is currently waiting for reviews [1].  To have a
 complete overview over the block device setup, this document is
 provided.
+
 The dependencies are not implemented as they should be, because
+
 * the spec process is currently in the phase of discussion and not
   finalized [2],
 * the implementation was finished and reviewed before the spec process
@@ -137,14 +139,15 @@ size
 
 Example:
 
-::
+.. code-block:: yaml
-        ["partitioning",
+
-          {"rootdisk": {
+    ["partitioning",
-              "label": "mbr",
+       {"rootdisk": {
-              "partitions":
+         "label": "mbr",
-              [{"name": "part-01",
+         "partitions":
-                "flags": ["boot"],
+            [{"name": "part-01",
-                "size": "100%"}]}}]
+              "flags": ["boot"],
+              "size": "100%"}]}}]
 
 Security impact
 ---------------

doc/source/specs/v1/approved/block-device-overview.rst

@@ -123,6 +123,7 @@ level or module needs it's own spec.
 
 A first step is to reimplement the existing functionality, this
 contains:
+
 #. Level 0: Local Loop module
    Use loop device on local image file
    (This is already implemented: [1])
@@ -133,6 +134,7 @@ contains:
 #. Level 3: Mounting
 
 As a second step the following functionality can be added:
+
 * Level 1: LVM module
 * Level 2: Create File System
   (swap)