ubuntu: Add options to ignore mirror components and use insecure repos

When using the upstream cloud images with the "ubuntu" element, they
have universe and multiverse enabled which we don't mirror.

To use the infra mirrors as a DIB_DISTRIBUTION_MIRROR with this
element, we need to be able to skip redirecting to universe and
multiverse, and additionally enable insecure repos (as we don't gpg
sign our mirrors).

Add and document two new variables with the ubuntu element to do this.
This is then setup by the openstack-ci-mirrors element so that we use
local mirrors duing dib functional testing for the "ubuntu" element.

Change-Id: I6ffbde07fa0e103641ee5c5f9d9e854e5b2168dc

diskimage_builder/elements/openstack-ci-mirrors/environment.d/11-dib-distribution-mirror.bash

@@ -27,8 +27,12 @@ if [ -f /etc/ci/mirror_info.sh ]; then
 
 fi
 
-# This is repo files pre-created for the fedora/centos-minimal jobs in
-# the gate
+# Infra doesn't mirror non-free repos, so instruct to ignore these
+export DIB_DISTRIBUTION_MIRROR_UBUNTU_IGNORE="(universe|multiverse)"
+export DIB_DISTRIBUTION_MIRROR_UBUNTU_INSECURE=1
+
+# These repo files are pre-created for the fedora/centos-minimal jobs
+# in the gate.  Not relevant inside the chroot.
 if [[ -d ${WORKSPACE:-/not/a/path/}/dib-mirror ]]; then
 
     if [[ "${DISTRO_NAME}" == "fedora" ]]; then

diskimage_builder/elements/ubuntu/README.rst

@@ -11,5 +11,15 @@ Overrides:
  * To download a non-default release of Ubuntu cloud images, use the
    environment variable ``DIB_RELEASE``. This element will export the
    ``DIB_RELEASE`` variable.
+ * Use ``DIB_DISTRIBUTION_MIRROR`` to override the ``sources.list``
+   with an alternative mirror
+ * Setting ``DIB_DISTRIBUTION_MIRROR_UBUNTU_IGNORE`` to an
+   extended-regexp (i.e. the argument to the ``=~`` bash comparitor)
+   which, when matched, will *not* set that line to the
+   ``DIB_DISTRIBUTION_MIRROR``.  For example, if your local mirror
+   does not mirror the universe and multiverse components, set this to
+   ``(universe|multiverse)``
+ * Setting ``DIB_DISTRIBUTION_MIRROR_UBUNTU_INSECURE`` updates apt
+   settings to allow insecure/unuthenticated repositories.
 
 .. element_deps::

diskimage_builder/elements/ubuntu/pre-install.d/01-set-ubuntu-mirror

@@ -1,6 +1,6 @@
 #!/bin/bash
 
-if [ ${DIB_DEBUG_TRACE:-0} -gt 0 ]; then
+if [ ${DIB_DEBUG_TRACE:-0} -gt 1 ]; then
     set -x
 fi
 set -eu
@@ -10,5 +10,21 @@ DIB_DISTRIBUTION_MIRROR=${DIB_DISTRIBUTION_MIRROR:-}
 
 [ -n "$DIB_DISTRIBUTION_MIRROR" ] || exit 0
 
-sudo sed -ie "s&http://\(archive\|security\).ubuntu.com/ubuntu&$DIB_DISTRIBUTION_MIRROR&" \
-    /etc/apt/sources.list
+while IFS= read line
+do
+    if [[ "$line" =~ "${DIB_DISTRIBUTION_MIRROR_UBUNTU_IGNORE:-}" ]]; then
+        # append line unmodified
+        echo "$line" | tee --append /etc/apt/sources.list.new
+    else
+        echo "$line" | \
+            sed -e "s&http://\(archive\|security\).ubuntu.com/ubuntu&$DIB_DISTRIBUTION_MIRROR&" | \
+            tee --append /etc/apt/sources.list.new
+    fi
+done < /etc/apt/sources.list
+
+if [[ -n "${DIB_DISTRIBUTION_MIRROR_UBUNTU_INSECURE:-}" ]]; then
+    echo "APT::Get::AllowUnauthenticated \"true\";" | tee /etc/apt/apt.conf.d/95allow-unauthenticated
+    echo "Acquire::AllowInsecureRepositories \"true\";" | tee -a /etc/apt/apt.conf.d/95allow-unauthenticated
+fi
+
+mv /etc/apt/sources.list.new /etc/apt/sources.list