Adding element to get source for elements.

Fixes bug 1184943

Adding an element that allows other elements to register
repositories it would like to be retrieved for it.

Doing this outside of the chroot allows locally cached
repositories to be used. It also gives the ability to d-i-b
to specify what revision to use if an alternative to the
most recent is required.

Effectively allowing a CI system to test d-i-b and elements
without being effected by unavailable git repositories or
breakages in actively developed source code being used by
the elements.

Change-Id: I1527facebaad256a357af680e017b34b1788575d

elements/source-repositories/README.md

@@ -0,0 +1,33 @@
+With this element other elements can register source code repositories by
+placing their details in the file source-repository-\*. An example
+of an element "custom-element" that wants to retrieve the ironic source
+from git and pbr from a tarball would be
+
+*File : elements/custom-element/source-repository-ironic*
+
+    #<name> <type> <destination> <location> [<ref>]
+    # <ref> defaults to master if not specified
+    ironic git /usr/local/ironic git://github.com/openstack/ironic.git
+
+*File : elements/custom-element/source-repository-pbr*
+
+    pbr tar /usr/local/pbr http://tarballs.openstack.org/pbr/pbr-master.tar.gz
+
+diskimage-builder will then retrieve the sources specified and place them
+at the directory \<destination\>
+
+A number of environment variables can be set by the process calling
+diskimage-builder which can change the details registered by the element, these are
+
+    DIB_REPOTYPE_<name>     : change the registered type
+    DIB_REPOLOCATION_<name> : change the registered location
+    DIB_REPOREF_<name>      : change the registered reference
+
+for example if you would like diskimage-builder to get ironic from a local
+mirror you could set DIB_REPOLOCATION_ironic=git://localgitserver/ironic.git
+
+Git sources will be cloned to \<destination\>
+
+Tarballs will be extracted to \<destination\>. Tarballs should contain a
+single topleval directory, regardless of the name of this top level directory
+it will be renamed to \<destination\>

elements/source-repositories/extra-data.d/99-getsources

@@ -0,0 +1,75 @@
+#!/bin/bash
+
+set -e
+
+# Gets Repositories listed in the a repository file and places them in
+# the repository directory.
+# The format of the repository file is one or more lines matching
+# <name> <type> <destination> <location> [<ref>]
+function get_repos_for_element(){
+    local REPO_SOURCES=$1
+
+    local REGEX="^([^ ]+) (git|tar) (/[^ ]+) ([^ ]+) ?([^ ]*)$"
+    while read line ; do
+
+        # ignore blank lines and lines begining in '#'
+        [[ "$line" == \#* ]] || [[ -z "$line" ]] && continue
+
+        if [[ "$line" =~ $REGEX ]]  ; then
+            local REPONAME=${BASH_REMATCH[1]}
+            local REPOTYPE=${BASH_REMATCH[2]}
+            local REPOPATH=${BASH_REMATCH[3]}
+            local REPOLOCATION=${BASH_REMATCH[4]}
+            local REPOREF=${BASH_REMATCH[5]:-master}
+
+            local REPO_DIRECTORY=$TMP_MOUNT_PATH$REPOPATH
+            local REPO_SUB_DIRECTORY=$(dirname $REPO_DIRECTORY)
+
+            # REPOTYPE can be overridden with DIB_REPOTYPE_{name}
+            local REPOTYPE_OVERRIDE=DIB_REPOTYPE_${REPONAME/-/_}
+            REPOTYPE=${!REPOTYPE_OVERRIDE:-$REPOTYPE}
+
+            # REPOLOCATION can be overridden with DIB_REPOLOCATION_{name}
+            local REPOLOCATION_OVERRIDE=DIB_REPOLOCATION_${REPONAME/-/_}
+            REPOLOCATION=${!REPOLOCATION_OVERRIDE:-$REPOLOCATION}
+
+            # REPOREF can be overridden with DIB_REPOREF_{name}
+            local REPOREF_OVERRIDE=DIB_REPOREF_${REPONAME/-/_}
+            REPOREF=${!REPOREF_OVERRIDE:-$REPOREF}
+
+            case $REPOTYPE in
+            git)
+                sudo mkdir -p $REPO_SUB_DIRECTORY
+                sudo git clone $REPOLOCATION $REPO_DIRECTORY
+                pushd $REPO_DIRECTORY
+                sudo git reset --hard $REPOREF
+                popd
+                ;;
+            tar)
+                # The top level directory of the tarball mightn't have a fixed name i.e.
+                # it could contain version numbers etc... so we write it to a tmpdir
+                # the then move the contents into the directory we want it in, this does
+                # assume the tarball only contains a single top level directory
+                local tmpdir=$(mktemp --tmpdir=$TMP_MOUNT_PATH/tmp -d)
+                curl $REPOLOCATION | tar -C $tmpdir -xzf -
+                sudo mkdir -p $REPO_DIRECTORY
+                sudo mv $tmpdir/*/* $REPO_DIRECTORY
+                rm -rf $tmpdir
+                ;;
+            *)
+                echo "Unsupported repository type"
+                return 1
+                ;;
+            esac
+
+        else
+            echo "Couldn't parse '$line' as a source repository"
+            return 1
+        fi
+    done < $REPO_SOURCES
+}
+
+# Get source repositories for the target
+for _SOURCEREPO in $(find $TMP_HOOKS_PATH -maxdepth 1 -name "source-repository-*") ; do
+    get_repos_for_element $_SOURCEREPO
+done

sudoers.d/img-build-sudoers

@@ -57,3 +57,5 @@ ALL ALL=(root) NOPASSWD: /usr/bin/du --block-size=* -x -s /tmp/*/built
 ALL ALL=(root) NOPASSWD: /bin/mount -t tmpfs tmpfs /tmp/image.*
 ALL ALL=(root) NOPASSWD: /bin/umount -f /tmp/image.*
 ALL ALL=(root) NOPASSWD: /bin/chown *\:* /tmp/image.*
+ALL ALL=(root) NOPASSWD: /bin/git clone * /tmp/image.*
+ALL ALL=(root) NOPASSWD: /bin/git reset --hard *