Commit Graph

477 Commits

Author SHA1 Message Date
Jenkins
308783d382 Merge "Switch openSUSE to 42.3 by default" 2017-07-28 05:14:23 +00:00
Alfredo Moralejo
b1961e14ea Use SELinuxfs to check selinux status
Currently, the cleanup script is using existence of
semanage binary to check if selinux is enabled. However
this is misleading and can lead to problems when selinux
is disabled in a system where the binary exist.

This patch changes the detection logic to use /sys/fs/selinux
directory which is a in-memory filesystem created only when
selinux is really enabled.

Change-Id: I008f8bbc9c8414ce948c601e3907e27764e15a52
Related-Bug: 1706386
2017-07-26 18:57:25 +02:00
Dirk Mueller
1c4c4fd734 Switch openSUSE to 42.3 by default
This is the latest stable release, so we should default to it.

Change-Id: I05643787002d339ccbf7a718847fe4ed6f39eacc
2017-07-26 08:56:02 +02:00
Jenkins
609bcee27b Merge "zypper: Clean caches and don't cache packages locally" 2017-07-26 02:25:40 +00:00
Markos Chandras
81e72d4045 elements: zypper-minimal: Install tar package
tar is an essential package but nothing pulls it explicitly. This causes
some issues in the openSUSE CI jobs like the following one

"Failed to execute tar: No such file or directory", "Failed to write
file: Broken pipe", "Failed to retrieve image file. (Wrong URL?)",
"Exiting."], "stdout": "", "stdout_lines": []}

Just like 'sed', add 'tar' to the list of packages for the openSUSE
minimal builds.

Change-Id: Ia36e3d9fd6b78862a6831ba80b43d4614a349ca0
2017-07-25 16:27:25 +01:00
Jenkins
a6da39acb8 Merge "Move setfiles to outside chroot with runcon" 2017-07-24 02:04:21 +00:00
Ian Wienand
5089e4e541 Move setfiles to outside chroot with runcon
As described in the comments inline, on a selinux enabled kernel (such
as a centos build host) you need to have permissions to change the
contexts to those the kernel doesn't understand -- such as when you're
building a fedora image.

For some reason, setfiles has an arbitrary limit of 10 errors before
it stops.  I believe we previously had 9 errors (this mean 9
mis-labeled files, which were just waiting to cause problems).
Something changed with F26 setfiles and it started erroring
immediately, which lead to investigation.  Infra builds, on
non-selinux Ubuntu kernel's, would not have hit this issue.

This means we need to move this to run with a manual chroot into the
image under restorecon.

I'm really not sure why ironic-agent removes all the selinux tools
from the image, it seems like an over-optimisation (it's been like
that since Id6333ca5d99716ccad75ea1964896acf371fa72a).  Keep them so
we can run the relabel.

Change-Id: I4f5b591817ffcd776cbee0a0f9ca9f48de72aa6b
2017-07-24 10:14:07 +10:00
Dirk Mueller
bfeb9d9e99 zypper: Clean caches and don't cache packages locally
For builds inside the infra, we don't want to pack the cache
inside the image (as it might be different at the time the image
runs). In an opensuse-minimal image this saves about 10MB of image
size.

Change-Id: I5ecabd46f0a662798bda3e4468395ad8308d0055
2017-07-23 17:24:24 +02:00
Jenkins
55971717b6 Merge "elements: openstack-ci-mirrors: Use openSUSE mirrors for gating jobs" 2017-07-22 05:22:34 +00:00
Jenkins
e029af993b Merge "Remove DIB_[DISTRO]_DISTRIBUTION_MIRROR" 2017-07-22 05:22:04 +00:00
Jenkins
7a70299668 Merge "Enable console during kernel boot on Power" 2017-07-20 03:55:19 +00:00
Jenkins
d66dbc679c Merge "The correct option for label name in fat and vfat is '-n'" 2017-07-20 03:54:23 +00:00
Ian Wienand
7ffe6856d6
Add -m flag to setfiles for Fedora 26
As described in the comment and associated bugzilla, the behaviour of
setfiles has changed in Fedora 26 to require "-m" situations where
labeled file-systems are mounted below non-labeled file-systems.  Our
loopback/chroot system appears to trigger this nicely, leading to a
setfiles call that does nothing without this.

Change-Id: I276c6f6a4fb44f4bea5004f6b4214f94757728ae
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2017-07-19 08:01:19 -04:00
Ian Wienand
6ffde2e596 yum-minimal: pre-install coreutils
As described in the referenced bug, the dependency solver in yum
doesn't handle weak dependencies well and in some cases, such as
Fedora 26, can end up choosing coreutils-single (the busybox-esque
single binary) instead of actual coreutils, which then causes problems
with conflicting packages later.

Change-Id: I2907bf3b74c146986b483d52cc6ac437036330b4
2017-07-18 14:51:18 +10:00
Ian Wienand
b8ad9c2e37 Force install during pip-and-virtualenv
On a system where the packaged pip/virtualenv is up-to-date with
upstream (such as Fedora 26 ... for now), we don't reinstall, which
then violates a bunch of assumptions later on.  Force install.

Change-Id: I6ebcda0351997fa7e32f0e6e77a98b2c33764e3f
2017-07-18 14:50:09 +10:00
Ian Wienand
da90ef4743 Fix latest-limit command line
It turns out dnf argparse can't handle negative numbers without "=".
It's actually documented in the man page

    --latest-limit <number> ...  If <number> is negative skip <number>
      of latest packages. If a negative number is used use syntax
      --latest-limit=<number>

But who reads that :)  This started failing with Fedora 26

Change-Id: I884af94c07fa11b010f69863047a04711b14f21e
2017-07-18 13:17:10 +10:00
Jenkins
016606c81d Merge "opensuse-minimal: install glibc-locale" 2017-07-18 00:40:48 +00:00
Dirk Mueller
59721d3c74 opensuse-minimal: install glibc-locale
We expect LC_ALL for non-C locales to be working inside
images, so always install glibc-locale for openSUSE.

Change-Id: I8fe92773e377539070d9d9fe2960a6202bb80a18
2017-07-17 22:50:25 +02:00
Markos Chandras
6be09152c2 elements: openstack-ci-mirrors: Use openSUSE mirrors for gating jobs
In preparation for promoting the openSUSE jobs to voting ones we should
use the OpenStack mirrors. As such, the opensuse elements are modified
to make use of the DIB_DISTRIBUTION_MIRROR variable which is normally
exported by the openstack-ci-mirrors element.

Change-Id: Ie588c1c1eec13190cfb2ec718ba51f8c9878283f
2017-07-17 10:54:03 +01:00
Jenkins
c18a3ff029 Merge "Replace architecture-emulation-binaries with qemu-debootstrap" 2017-07-17 05:36:09 +00:00
Ian Wienand
3457d2f8e8 Remove DIB_[DISTRO]_DISTRIBUTION_MIRROR
We added the DIB_distro_DISTRIBUTION_MIRROR arguments with
I92964b17ec3e47cf97e3a3091f054b2a205ac768 as a way that we could
source a list of mirrors and then have the distro elements choose
which one applied to them.

However, this hasn't worked out to be so useful.  The
openstack-ci-mirrors element is working as a mirror setup script -- it
translates the openstack CI mirror list variables into the generic
"DIB_DISTRIBUTION_MIRROR" as appropriate for each distro's build.
Also, it turns out there's other things that need to be done, such as
turning off gpg checking, which mean the idea of "just export
variables" hasn't turned out as valid ... you need actual code
involved to get it right.

AFAICT we never actually documented these, and they do not seem to be
in use.  They have caused considerable confusion when dealing with new
platforms as we try to keep consistency.  Remove them.

[1] http://codesearch.openstack.org/?q=DIB_.*_DISTRIBUTION_MIRROR&i=nope&files=&repos=

Change-Id: Ifc4ab700631ffdfbe790068558f670f9a11dde5e
2017-07-17 14:47:31 +10:00
Jenkins
787e76b916 Merge "Remove additional Bumblebee repository for opensuse element" 2017-07-17 00:50:46 +00:00
Jenkins
64a8c6e1dc Merge "zypper-minimal: No point in preserving the environment here" 2017-07-17 00:26:11 +00:00
Amrith Kumar
59f416ae20 The correct option for label name in fat and vfat is '-n'
The code in mkfs correctly extends the command line with a '-n' for
vfat but does not currently do it for fat. This means that mkfs for
fat ends up with a '-L' which is what you'd do for everything like
ext[234].

The change just treats fat like vfat in the one place where this check
is required.

Change-Id: If65dfd949acdadff33a564640fb42ea73026a786
Closes-Bug: #1703063
2017-07-15 22:48:52 -04:00
Dirk Mueller
02d33f2ca7 zypper-minimal: No point in preserving the environment here
Change-Id: I46442e841d1f718b683bca4d2a348f0013306907
2017-07-13 22:50:47 +02:00
Dirk Mueller
05ba445ade Remove additional Bumblebee repository for opensuse element
The purpose of the openSUSE element is to build openSUSE distribution
based images, so an additional community repo shouldn't be pulled into
the image. In addition the dkms dependency is blacklisted for SUSE
in the dkms element anyway, so this should be a noop.

Change-Id: I0aa06d9f4f110546032f910e3361840693d02de7
2017-07-11 23:24:05 +02:00
Jenkins
0327d775f1 Merge "pip-and-virtualenv: Install python3 on openSUSE" 2017-07-11 08:11:16 +00:00
Jenkins
997a6ea6cb Merge "Add symlink test for resolv.conf restore" 2017-07-11 08:07:23 +00:00
Rafael Folco
bfdf7dc0f6 Enable console during kernel boot on Power
On Power systems console should be added the kernel command line
in the following order: 'console=tty0 console=hvc0'.
The first one is the graphical console. The last one is the serial
console. The kernel enables all the consoles pointed through the
kernel command line. However, only the last one will receive
input/output during kernel boot. All the other consoles will be
enabled after the boot.

Change-Id: I0069f608e0ab104d3778954e033fb82ed5ea7693
2017-07-07 17:55:56 +00:00
Amrith Kumar
43e32116bd fix readme.rst to reflect correct environment variable
The readme.rst incorrectly refers to the environment variable
DIB_APT_KEYS which should be DIB_ADD_APT_KEYS. See [1] for usage in
code.

This is a minor correction to the readme only, no runnable code is
modified.

[1] http://git.openstack.org/cgit/openstack/diskimage-builder/tree/diskimage_builder/elements/dpkg/extra-data.d/01-copy-apt-keys#n23

Change-Id: I04129cef9f40ec75a206c126bfd40ee61e4e6a2b
2017-07-06 22:54:08 -04:00
Ian Wienand
5fa6e3e13c Add symlink test for resolv.conf restore
We replace the base resolv.conf with an "outside" copy so that
resolving works when we're in the chroot.

Installing resolvconf package modifies the in-chroot resolv.conf to a
symlink (to /var/run) which it wants maintained in the final image.
We have the existing "immutable" check for a created resolv.conf file,
but no eqivalent for a symlink.

This adds a check to see if the resolv.conf is a symlink and leave it
alone if it is, assuming it has been re-created in the chroot.

I have tested this with ubuntu-minimal+resolvconf with
dhcp-all-interfaces and the system seems to work with resolvconf
working correctly.

Change-Id: Idd5a26e9d55979bd951577d5b098ed4bfba91ad3
2017-07-06 13:48:27 +10:00
Jenkins
e8ad2a3799 Merge "elements: pip-and-virtualenv: Use common packages for openSUSE" 2017-07-04 11:20:35 +00:00
Markos Chandras
5fe35b0d7a pip-and-virtualenv: Install python3 on openSUSE
The python3 package actually contains some core modules (like the xml
one) which are not present in the python3-base on which is pulled by
the python3-devel package. As such, it's best to have it installed
similar to python-xml for python2.

Change-Id: I5cd5d1127ae62d6753c2ace44965179c5400bb9a
2017-07-04 08:40:34 +01:00
Jenkins
fad72745d2 Merge "Support for Cloud Images on ppc64le for rhel7 and centos7" 2017-07-04 01:13:24 +00:00
Jenkins
2ed643a734 Merge "Use the dib python to do cleanup" 2017-06-29 21:22:36 +00:00
Jenkins
6b45497ff6 Merge "Remove centos and rhel elements" 2017-06-29 21:16:57 +00:00
Jenkins
f0fb835db9 Merge "Avoid hanging endlessly on unreachable cache urls" 2017-06-29 08:03:25 +00:00
Chhavi Agarwal
6d69d7909d Support for Cloud Images on ppc64le for rhel7 and centos7
In order to support {CentOS,RHEL}7 for building cloud images we need to
handle the differences in grub packaging from Ubuntu.  We also need to
populate the defualt location for cloud images for CentOS builds.

Change-Id: Ie0d82ff21a42b08c4cb94b7a5635f80bfabf684e
2017-06-29 15:44:26 +10:00
Dirk Mueller
959226c55e Avoid hanging endlessly on unreachable cache urls
When a download redirector redirects to a broken mirror, timeout
quickly rather than waiting until the overall job is being timed out.

Change-Id: If7eb63d406aaf61f71aa9203cf708c474aa63fd0
2017-06-28 22:14:55 +02:00
Markos Chandras
c46b6da65f elements: pip-and-virtualenv: Use common packages for openSUSE
The 'packages' variable already contains the packages we need so
use it instead of duplicating the packages.

Change-Id: Id22e1862f9654e66252d03a0fed9839cf004d750
2017-06-28 17:59:25 +01:00
Ian Wienand
a00d02f6a1 Remove centos and rhel elements
Several people have popped up in IRC recently with failures in these
elements.  Without Python 2.7 available in the image they are
unsupported (OpenStack hasn't supported it for a long time).  Remove
these to avoid further confusion.

The centos/centos7 DISTRO split that has happened with centos-minimal
is unfortunate but I don't think it helps to rename centos7/rhel7 ATM.
To summarise; DISTRO=centos7 means image based build,
DISTRO=centos && DIB_RELEASE=7 means the minimal build.

In the future, I think it is important that the minimal builds and
image builds set the same DISTRO.  This reflects that "upper" layers
shouldn't care about the exact building of the lower layers.  I see
CentOS 8 going one of two ways

1) the changes are so significant, we start separate centos8 /
centos8-minimal elements.  They both set DISTRO=centos8 (and
DIB_RELEASE to point-release maybe?).  This means we have to update
all "if DISTRO == centos || DISTRO == centos7" branches to also check
for "centos8".  Evenually (!)  "centos" goes away for versioned DISTRO
only

2) we restore centos element with DISTRO=centos and DIB_RELEASE=8, and
centos-minimal remains the same.  This means we have to audit all "if
DISTRO == centos" calls to make sure they're appropriate for version 8
(stick a "&& DIB_RELEASE=7" on them all basically).

I'm not sure we can fully decide until we start to see excatly how the
distro switching/matching bits look, but (2) is consistent with Ubuntu
and probably the preferred solution.

Some "rhel" parts have been cleaned up.  More could be done in
rhel-common, but given our lack of coverage of that I'd prefer to
leave it for now.

Change-Id: I6ea784116ef59ca22878c8512c963f29c815a00a
2017-06-28 12:26:24 +10:00
Tony Breeds
c383428727 As far as block-device layout is concerned ppc64le == ppc64el
Change-Id: I06a89f256f66eba2f73dd110f5d8a61e0f0a924c
2017-06-27 00:34:33 +00:00
Jenkins
c1facd7980 Merge "Fix mkfs use wrong label option for vfat" 2017-06-26 02:41:30 +00:00
Ian Wienand
b0e0dd991c Move image download tests to default skip
The image download tests have long been too unreliable for the gate.
We need to cache the base images similar to how devstack caches it's
testing images.  Let's move them to non-voting jobs for the time
being.

This means that the gate jobs are now all based on "-minimal" and are
using infra mirrors.  Unfortunately, there is still some unreliability
because we currently have issues with infra mirrors being very slow
after AFS updates, leading to job timeouts.  But we're on the right
path...

Also, I noticed we don't have tests of the "ubuntu" image-download
based tests, which were tacitly being tested by apt-sources before we
moved that to -minimal.  Add simple tests for these.

Change-Id: Ie33ee49656872467ef68d753210032156bb6b2cb
2017-06-23 10:58:47 +10:00
Xinliang Liu
178db0c97b Fix mkfs use wrong label option for vfat
For vfat type, mkfs should use '-n' option for label.
e.g.:
mkfs -t vfat -n LABEL-STRING

Change-Id: I1414c5b8e0aeb240c3e6884e35ba75dde677db0c
2017-06-22 14:50:53 +08:00
Jamie Lennox
aa03e24c30 Use the dib python to do cleanup
In a system where python2 is not installed and /usr/bin/python is not
linked then the cleanup process will fail trying to invoke the python
script. Use the previously determined DIB_PYTHON_EXEC if it's available.

Change-Id: I128292808ccef92cc1803988b35caae5aa6fa541
2017-06-22 11:20:51 +10:00
Clark Boylan
559de43694 On suse the python2 dev package is python-devel
This was previously defined as python2-devel (which is what rhel uses),
but the actual package name is python-devel. See:

  https://software.opensuse.org/package/python-devel

Change-Id: Id61e5b05772d10c32b33d3e70cb64d5ebdcba6e4
2017-06-21 15:52:05 +10:00
Ian Wienand
18a0d970fa Move ironic-agent test to fedora-minimal
I'm uncertain as to why this is using the "fedora" element for testing
... but it requires downloading the fedora .qcow on every test which
has shown to be unreliable.  An easy thing to do is to switch it to
fedora-minimal; that will only involve downloads from local mirrors in
the gate.

Add redhat-rpm-config for minimal.  I admit I have not fully gone
through why this is not pulled in.  It's been an issue since
I459f2203fa145049dda185da952813118193d573 and there's all sorts of
bugs.

Change-Id: I37458e3926dae32a259bd5aa9efc645561b029a0
2017-06-21 15:05:36 +10:00
Ian Wienand
649f0b66d9 Start at using CI mirrors for fedora/centos
fedora/centos-minimal don't obey DIB_DISTRIBUTION_MIRROR currently.  I
don't really want them too -- we want to be able to separate the
mirrors used during the build process from those embedded into the
final image.  Add DIB_YUM_MINIMAL_BOOTSTRAP_REPOS which is a directory
with repo files to use during the install.

This introduces setup-gate-mirrors.sh which is intended to setup
repo/sources/whatever files in the openstack gate that point to the
local region mirror.  It pulls the info from the mirror_info.sh script
on each CI node.

The openstack-ci-mirrors element is updated to export these variables.
elements are updated to depend on it.  Tests are restored

Change-Id: I7604fc4d41cb1483be16b8d628a24e8fc764f515
2017-06-21 12:02:27 +10:00
Ian Wienand
f0b70211c6 Use local mirror for ubuntu-minimal jobs
This adds "openstack-ci-mirrors" element which performs various
settings to get builds using local mirrors.  As a first step, we
convert ubuntu-minimal jobs

The main trick is that since infra mirrors are created with rerepo
they are not signed (they are recreated, not cloned, and not signing
is seen as a feature in that it deters external use).  So we need to
instruct debootstrap to ignore signing and also turn it off for
in-chroot apt.  Other than that, the existing DIB_DISTRIBUTION_MIRROR
works to redirect installs.

Remove "restricted" as it's not mirrored, and I don't think we want it
in here by default.

(I think DIB_DISTRIBUTION_MIRROR is a bit of an anti-pattern, because
it leaves the mirrors in the final image -- just because you use them
to build, doesn't mean you want them at runtime).  But we don't need
to fix that now, and we don't use any created images.)

This pauses fedora testing until the next change, which moves to using
local mirrors for testing on fedora/centos

Change-Id: I778bd05a1e615c27edf1c9f0a1409119a6b3a850
2017-06-21 12:01:31 +10:00
Ian Wienand
0d37351031 Move apt-sources to ubuntu-minimal / move debian to skip list
The gate is currently extremley unstable, and these two issues are
causing most of the problems.  We need to commit them atomically so we
can get anything moving again

---

The gate is very unstable downloading the ubuntu tarballs from
upstream at the moment.  Move this to ubuntu-minimal which, in a later
change will source files from our local mirror.

We need a caching mechanism for these large files to avoid this
instability.  This is future work for the various image-based jobs.

---

Move debian to default skip lists

I don't know if it's mirrors being worked hard for the Stretch
release, but this is constantly failing the gate.  I will move this to
the -nv extras job

I am working on having the voting job use local mirrors for
everything.  Unfortunately debian infra mirrors don't have stretch yet
and we need to do some fiddling to get "stable" available.  Once we
have all this, we can consider making it voting again.

Change-Id: Iaf7b3888ef06c7aef63cbf76a94b33f96bc9c5c2
2017-06-21 10:34:53 +10:00
Ian Wienand
5d5fa06e5c Sync after writing partition table
We introduced the "settle" in
I90103b59357edebbac7a641e8980cb282d37561b thinking that maybe kpartx
had not finished writing the partition.  This probably wasn't a bad
first assumption, since we used to have this -- but is seems
insufficient.

The other failiure here seems to be if kpartx hasn't actually seen the
updated partition table in the image, so it has correctly (in it's
mind) not mounted the partition.

Looking at strace of fdisk run manually on a loopback, it will do a
fsync on the raw device after writing and then a global sync as it
exits.

This replicates this; we flush and fsync in mbr.py in the exit handler
after writing the partition, before closing the file (i've updated one
of the unit tests to double-check the call).  In the partitioning.py
caller we execute a sync call too.

Since it does seem unlikely the "-s" option of kpartx is not working,
I've removed the udev settle work-around too.

Change-Id: Ia77a0ffe4c76854b326ed76490479d9c691b49aa
Partial-Bug: #1698337
2017-06-19 17:13:36 +10:00
Ian Wienand
a0f747932d Install systemd earlier for Stretch
Debian Stretch released as stable recently, and the init system is
less tightly specified in the base dependencies (for some info, see
[1]).  It seems, probably unintentionally, that in the previous
release systemd-sysv was brought in by debootstrap, but that is no
longer happening.

Add systemd as an early dependency of debian-minimal.

Remove the package-installs.yaml as that happens too late (other
things need to know the init system to write out service files, etc
and probe for systemd utils before package-installs).  As mentioned, I
do not believe the "only install systemd on testing" idea was actually
working here, because it was being brought in during the initial
debootstrap.

Update some documentation to explain what's going on

[1] https://lists.debian.org/debian-boot/2015/05/msg00156.html

Change-Id: Id67c0cf08728407d234976f9807d3bd71d12f758
2017-06-19 13:27:33 +10:00
Michael Johnson
250aeb5d21 Fix mkfs failure when loop device is not ready
There was a race in diskimage-builder where the mkfs call after a
kpartx -avs for the loop device would fail because the device was
not yet ready.  This adds a udevadm settle call after the kpartx
to make sure the udev event queue has cleared.

Change-Id: I90103b59357edebbac7a641e8980cb282d37561b
Closes-Bug: #1698337
2017-06-17 09:00:13 +10:00
Ian Wienand
f60dd38482 Add timestamp output filter
This adds a devstack-inspired output filter to standardise
timestamping.

Currently, python tools timestamp always (timestamp setup in
logging_config.py) but all the surrounding bash does not.

We have extra timestamps added in run_functests.sh for our own
purposes to get the bash timestamps; but this ends up giving us
double-timestamps for the python bits.  Additionally, callers such as
nodepool capture our output and put their own timestamps on it, and
again have the double-timestamps.

This uses a lightly modified outfilter.py from devstack to standardise
this.

All output is run through this filter, which will timestamp it.  I
have removed the places where we double-timestamp -- logging_config.py
and the prefix in dib-run-parts.

An env option is added to turn timestamps off completely (does not
seem worth taking up a command-line option for).  For callers like
nodepool, they can set this and will just have their own timestamps as
they collect the lines.

Since all logging is going through outfilter, it's easy to add a
--logfile option.  I think this will be quite handy; personally I'm
always redirecting dib runs to files for debugging.

I've also added a "quiet" option.  I think this could be useful in
run_tests.sh if we were to start logging the output of each test to
individual files.  This would be much easier to deal with than the
very large log files we get (especially if we wanted to turn on
parallel running...)

Change-Id: I202e1cb200bde17f6d7770cf1e2710bbf4cca64c
2017-06-16 15:58:50 +10:00
Jenkins
1324f5b7db Merge "Remove use of 'which'." 2017-06-11 09:30:34 +00:00
Ian Wienand
5ac8a98e9a PPC bootloader; install to boot partition
Using the newly exposed variables from the prior change, install the
ppc bootloader to the boot partition, not the underlying loopback
device.

Change-Id: I0918e8df8797d6dbabf7af618989ab7f79ee9580
2017-06-08 17:14:22 +10:00
Ian Wienand
6c394f5746 Pass all blockdevices to bootloader
Currently we only export "image-block-device" which is the loopback
device (/dev/loopX) for the underlying image.  This is the device we
install grub to (from inside the chroot ...)

This is ok for x86, but is insufficient for some platforms like PPC
which have a separate boot partition.  They do not want to install to
the loop device, but do things like dd special ELF files into special
boot partitions.

The first problem seems to be that in level1/partitioning.py we have a
whole bunch of different paths that either call partprobe on the loop
device, or kpartx.  We have _all_part_devices_exist() that gates the
kpartx for unknown reasons.  We have detach_loopback() that does not
seem to remove losetup created devices.  I don't think this does
cleanup if it uses kpartx correctly.  It is extremley unclear what's
going to be mapped where.

This moves to us *only* using kpartx to map the partitions of the loop
device.  We will *not* call partprobe and create the /dev/loopXpN
devices and will only have the devicemapper nodes kpartx creates.
This seems to be best.  Cleanup happens inside partitioning.py.
practice.  Deeper thinking about this, and more cleanup of the
variables will be welcome.

This adds "image-block-devices" (note the extra "s") which exports all
the block devices with name and path.  This is in a string format that
can be eval'd to an array (you can't export arrays).

This is then used in a follow-on
(I0918e8df8797d6dbabf7af618989ab7f79ee9580) to pick the right
partition on PPC.

Change-Id: If8e33106b4104da2d56d7941ce96ffcb014907bc
2017-06-08 17:14:22 +10:00
Ian Wienand
1d1e4ccb3e Move rollback into NodeBase object
Currently we pass a reference to a global "rollback" list to create()
to keep rollback functions.  Other nodes don't need to know about
global rollback state, and by passing by reference we're giving them
the chance to mess it up for everyone else.

Add a "add_rollback()" function in NodeBase for create() calls to
register rollback calls within themselves.  As they hit rollback
points they can add a new entry.  lambda v arguments is much of a
muchness -- but this is similar to the standard atexit() call so with
go with that pattern.  A new "rollback()" call is added that the
driver will invoke on each node as it works its way backwards in case
of failure.

On error, nodes will have rollback() called in reverse order (which
then calls registered rollbacks in reverse order).

A unit test is added to test rollback behaviour

Change-Id: I65214e72c7ef607dd08f750a6d32a0b10fe97ac3
2017-06-08 17:14:20 +10:00
Ian Wienand
09dee46579 Move global mount tracking into state
Keep track of the mount-point ordering in a state variable, rather
than a global.  This path is tested by existing unit tests.

Note a prior change inserted the MountNode objects directly into a
list in self.state, which makes sorting quite easy as it can just
implement __lt__.  Unfortunately we still json dump the state, and
thus we can't have aribtrary objects in it (future work may be to
check keys inserted into the status object...).  So we have to do a
bit of wrangling with tuple lists and comparision functions here, but
it's not too bad.

Change-Id: I0c51e0c53c4efdb7a65ab0efe09a6780cb1affa8
2017-06-08 17:13:28 +10:00
Ian Wienand
886f925b13 Use global state to check for duplicate fs labels
As we add file-systems, add them to global state and check the labels
are uniqiue.  Add a unit test and remove the old global value.

Bonus fixup to the length check, and a test for that too.

Change-Id: I0f5a96f687c92e000afc9c98a26c49c4b1d3f28d
2017-06-08 17:13:28 +10:00
Ian Wienand
b708918b85 Remove 'state' argument from later cmd_* calls
With I468dbf5134947629f125504513703d6f2cdace59 each node has a
reference to the global state object.  This means it gets pickled into
the node-list, which is loaded for later calls.  There is no need to
reload the state.json it and pass it for later cmd_* calls, as the
nodes can see it via the unpickled self.state

Change-Id: I9e2f8910f17599d92ee33e7df8e36d8ed4d44575
2017-06-08 17:13:28 +10:00
Ian Wienand
824a9e91c4 Add state to NodeBase class
Making the global state reference a defined part of the node makes
some parts of the block device processing easier and removes the need
for other global values.

The state is passed to PluginNodeBase.__init__() and expected to be
passed into all nodes as they are created.  NodeBase.__init__() is
updated with the new paramater 'state'.

The parameter is removed from the create() call as nodes can simply
reference it at any point as "self.state".

This is similar to 1cdc8b20373c5d582ea928cfd7334469ff36dbce, except it
is based on I68840594a34af28d41d9522addcfd830bd203b97 which loads the
node-list from pickled state for later cmd_* calls.  Thus we only
build the state *once*, at cmd_create() time as we build the node
list.

Change-Id: I468dbf5134947629f125504513703d6f2cdace59
2017-06-08 17:13:26 +10:00
Ian Wienand
e82e0097a9 Use picked nodes for later cmd_* calls
Currently the later cmd_* calls -- umount, cleanup, delete -- all
recreate the node graph by parsing the config file using
create_graph()

There is some need, however, to have a sense of global state when
building the node list.  The problem is, this is a one time operation
-- we do not want to rebuild that state for these later calls (see the
"loaded" checks in proposed
Ic3b805f9258128d5233b21ff25579c03487c7fcc).

An insight here seems to be that these cmd_* calls do not actually
want to re-parse the configuration file and rebuild the node list;
they just want to walk the node list in reverse with the state as
provided after cmd_create().

So, rather than re-creating the node list, we might as well just
pickle it, save it to disk along side the state dictionary dump and
reload it for cmd_*.

After this, I think we can safely have PluginBase.__init__() be passed
the state.  We will now know that this will only be called once,
during initial creation.

Change-Id: I68840594a34af28d41d9522addcfd830bd203b97
2017-06-08 17:10:10 +10:00
Ian Wienand
9a8b135267 Don't make image & loopdev functions static
You can't pickle a static method reference which complicates being
able to save the node graph when the "rollback" call-back wants to
hold references to these functions.  The outer module (localoop.py) is
small anyway, so from an organisation point of view the difference is
minimal.  Since these are really only called with parameters from the
containing class, they could be class methods with no parameters, at
the small expense of having to fiddle the mbr test-case a bit.

Change-Id: I6f9592a4295abe1b41294b79828bc2f3c2da01c6
2017-06-08 17:10:10 +10:00
Jenkins
60a5484ae8 Merge "Add env var to dump config graph" 2017-06-08 06:59:51 +00:00
Ian Wienand
d5c3863b87 Add env var to dump config graph
Make this a bit easier during debugging.  Add env var and some
developer instructions.

Change-Id: I34978ddb47d6642dfa22cae0f4c0543c0ba5475f
2017-06-08 05:04:58 +00:00
Ian Wienand
6fe1ef94f1 Use class as super() argument
Fix a few typos using the inherited class for super()

Change-Id: If9f2f423f136fb78ee93018d5c299d0dae603aad
2017-06-08 09:43:47 +10:00
Ian Wienand
90b56b3aab Move ppc block-device default to right $ARCH
The supported ppc ${ARCH} is "ppc64el" (at least in the gate testing
...) so move the file to that, so gets picked up by
block_device_create_config_file

Change-Id: I9273f35cdbfb0a62404461cbc1df9b2a92155fb0
2017-06-07 13:30:38 +10:00
Ian Wienand
89a85f6fbb Update tracing in block_device_create_config_file
Something seems to be going on with the ppc matching in the gate test.
Small updates to see what's going on...

Change-Id: Ie48cd4ce1f983a58932a577a43746240f6866936
2017-06-07 13:30:38 +10:00
Ian Wienand
7661da1341 Pad state dump
Because we append the function/line info after debug lines in the gate
logs, the pretty-print ends up not looking all that pretty.  Pad it.

Change-Id: Ice013428342614300cd51e8b7be56e79b75b31fc
2017-06-06 12:34:00 +10:00
Jenkins
ec70cb61f0 Merge "Trivial fix typos" 2017-06-05 05:54:50 +00:00
Ian Wienand
cdb1a95be1 Move "functional" unit tests under block-device
This is code motion with some small changes to make follow-on's
easier.

test_blockdevice_mbr.py is moved alongside the other tests.  It is
modified slightly to use the standard base class and remove a lot of
repeated test setup; a fixture is used for the tempdir (so it doesn't
have to be torn-down, and is removed properly on error) and the partx
args are moved into the setUp() so each test doesn't have to create
it.  No functional change.  renamed test_mbr.py for shortness.

test_blockdevice_utils.py is merged with existing test_utils.py.  No
change to the tests.

test_blockdevice.py is removed.  It isn't doing anything currently; to
work it will need to take an approach based more on mocking of calls
that require elevated permissions.  It's in history if we need it.

Change-Id: I87b1ea94afaaa0b44e6a57b9d073f95a63a04cf0
2017-06-05 12:22:52 +10:00
Vu Cong Tuan
cae44c7eea Replace assertRaisesRegexp with assertRaisesRegex
assertRaisesRegexp was renamed to assertRaisesRegex in Py3.2
For more details, please check:
https://docs.python.org/3/library/
unittest.html#unittest.TestCase.assertRaisesRegex

Change-Id: I705c958c0dbf1daa409ed29ccbc038426298c306
Closes-Bug: #1436957
2017-06-03 13:27:37 +07:00
Jenkins
5a045e036d Merge "dhcp-all-interfaces.sh - Add support for InfiniBand interface DHCP" 2017-06-02 06:11:19 +00:00
Jenkins
80cc1d0ea4 Merge "Adjust package mapping for SUSE family" 2017-06-02 02:56:16 +00:00
Dirk Mueller
d0a398c167 Adjust package mapping for SUSE family
package-installs.yaml is installing python-dev, not python2-dev,
so we need to adjust the mapping accordingly.

In addition, zypper-minimal used an dpkg specific package name,
while there is a SUSE equivalent (and zypper-minimal is anyway
SUSE family specific)

Change-Id: Ia9dd061fa46a514781808d62e5e93b03f75c6745
2017-05-31 21:09:53 +02:00
Dirk Mueller
f58bf252de Drop support for Ubuntu precise
Ubuntu 12.04 LTS reached its regular End of Life on April 28, 2017.

Depends-On: I5e145095a10db112bb27516bfe652d2cdc052a61
Change-Id: I64af4c5183d77a75dcd062895d19b0a1330c8da8
2017-05-31 14:36:30 +02:00
Jenkins
016b1f1522 Merge "Make BlockDeviceState implement a dict" 2017-05-31 10:55:32 +00:00
Jenkins
d1c49c1ae4 Merge "Refactor mount-point sorting" 2017-05-31 10:50:26 +00:00
Jenkins
b312c06dbb Merge "Decode string to bytes in dracut-regenerate" 2017-05-31 10:49:51 +00:00
Jenkins
09543cf52b Merge "Add state object, rename "results", add unit tests" 2017-05-31 05:52:24 +00:00
Jenkins
d0e0714f71 Merge "Test openSUSE 42.2/42.3 image builds" 2017-05-31 04:37:57 +00:00
Vu Cong Tuan
6a72052108 Trivial fix typos
Change-Id: Ib86aa9938fd852610ec0a6d8d868181f87bd2f24
2017-05-31 11:17:05 +07:00
Jenkins
2bdc154df5 Merge "drop deprecated map-services/packages from zypper element" 2017-05-31 02:11:21 +00:00
Jenkins
05d64b99ce Merge "Remove ccache" 2017-05-31 01:48:01 +00:00
Ian Wienand
4253cab773 Make BlockDeviceState implement a dict
While plugins treat the state as just a dictionary, it's nice for the
driver functions to keep state related functions encapsulated in the
state object singleton.  Wrap the internal state dictionary so we can
pass the BlockDeviceState directly without dereferencing.

Change-Id: Ic0193c64d645ed1312f898cbfef87841f460799c
2017-05-31 11:24:55 +10:00
Ian Wienand
35a1e7bee9 Refactor mount-point sorting
Currently we keep a global list of mount-points defined in the
configuration and automatically setup dependencies between mount nodes
based on their global "mount order" (i.e. higher directories mount
first).

The current method for achieving this is roughly to add the mount
points to a dictionary indexed my mount-point, then at "get_edge()"
call build the sorted list ... unless it has already been built
because this gets called for every node.

It seems much simpler to simply keep a sorted list of the
MountPointNode objects as we add them.  We don't need to implement a
sorting algorithm then, we can just use sort() and implement __lt__
for the nodes.

I believe the existing mount-order unit testing is sufficient; I'm
struggling to find a valid configuration where the mount-order is
*not* correctly specified in the configuration graph.

Change-Id: Idc05cdf42d95e230b9906773aa2b4a3b0f075598
2017-05-31 11:05:50 +10:00
Jenkins
edaf577bad Merge "Remove dracut-network element" 2017-05-31 00:14:01 +00:00
Mark Goddard
54765fd2f4 Remove dracut-network element
This element has not been functioning correctly for some time due to
an incorrect path to select-boot-kernel-initrd (should be /usr/local/bin).

The dracut-regenerate element can be used to regenerate dracut ramdisks
and is more flexible than this element.

Change-Id: I33d555ffd4a92b2948b2ea4a66b151f0422ccb8c
Closes-Bug: #1688546
2017-05-31 08:36:56 +10:00
Andreas Florath
b107606a75 Remove ccache
This patch removes the ccache handling from the base element.  For
mostly all systems this was never used at all.

This is working towards the removal of the base element from DIB

Change-Id: Ieb16ef612ebd98470993dcd6f55b3a22d37084ba
Signed-off-by: Andreas Florath <andreas@florath.net>
2017-05-31 08:28:09 +10:00
Mark Goddard
aa6c1d01a9 Decode string to bytes in dracut-regenerate
In python3, the standard out data returned by
subprocess.Popen.communicate() will in most cases be bytes rather than a
string and must therefore be decoded.

Without this fix we hit the following error:

TypeError: a bytes-like object is required, not 'str'

Change-Id: I6d75f867ebfdb925970c3397175214b9050d7632
Closes-Bug: #1694463
2017-05-30 16:15:06 +01:00
Dirk Mueller
5d39f83f74 Test openSUSE 42.2/42.3 image builds
Currently openSUSE 42.3 has entered feature freeze mode
so it is a good point in time to verify that 42.3 builds
are working successfully. Also test opensuse-minimal for
platforms that support it (need working zypper package)

Change-Id: I4c613e1e68cb7375c29d544bbf70b5da9bf21414
2017-05-30 13:07:04 +02:00
Ian Wienand
b85de3cd9e Add state object, rename "results", add unit tests
A couple of things going on, but I think it makes sense to do them
atomically.

The NodeBase.create() argument "results" is the global state
dictionary that will be saved to "state.json", and re-loaded in later
phases and passed to them as the argument "state".  So for
consistency, call this argument "state" (this fits with the change out
to start building the state dictionary earlier in the
PluginBase.__init__() calls).

Since the "state" is a pretty important part of how everything works,
move it into a separate object.  This is treated as essentially a
singleton.  It bundles it nicely together for some added
documentation [1].

We move instantiation of this object out of the generic
BlockDevice.__init__() call and into the actual cmd_* drivers.  This
is because there's two distinct instantiation operations -- creating a
new state (during cmd_create) and loading an existing state (other
cmd_*).  This is also safer -- since we know the cmd_* arguments are
looking for an existing state.json, we will fail if it somehow goes
missing.

To more fully unit test this, some testing plugins and new
entry-points are added.  These add known state values which we check
for.  These should be a good basis for further tests.

[1] as noted, we could probably do some fun things in the future like
make this implement a dictionary and have some saftey features like
r/o keys.

Change-Id: I90eb711b3e9b1ce139eb34bdf3cde641fd06828f
2017-05-30 20:39:00 +10:00
Jenkins
634e9ac043 Merge "Refactor: use lazy logging" 2017-05-30 09:30:28 +00:00
Jenkins
2275ccb97b Merge "allow uninstalls to fail on gentoo" 2017-05-30 07:38:56 +00:00
Matthew Thode
ce7ea9d34c
allow uninstalls to fail on gentoo
The cleanup of packages should be more opertunistic, if it's not there
then fail quietly.

Change-Id: I207a1162abc9ca5e9636b8de192f21424db0f569
2017-05-29 23:46:42 -05:00
Andreas Florath
f314df12c3 Refactor: use lazy logging
As described in pep282 [1], the variable part of a log message
should be passed in via parameter.  In this case the parameters
are evaluated only when they need to be.

This patch fixes (unifies) this for DIB.

A check using pylint was added that this kind of passing parameters to
the logging subsystem is enforced in future.  As a blueprint a similar
(stripped-down) approach from cinder [2] was used.

[1] https://www.python.org/dev/peps/pep-0282/
[2] https://github.com/openstack/cinder/blob/master/tox.ini

Change-Id: I2d7bcc863e4e9583d82d204438b3c781ac99824e
Signed-off-by: Andreas Florath <andreas@florath.net>
2017-05-30 14:39:58 +10:00
Ian Wienand
543dc1baa6 Add pylint with indent check
This is an initial creation of pylint with a basic indent checker.
Small issues corrected.  Job added to gate with
Ib554a284e92583cc1d6a5c2219b3922852ca4c73

Change-Id: I7e24d8348db3aef79e1395d12692199a1f80161a
Co-Authored-By: Andreas Florath <andreas@florath.net>
2017-05-29 16:12:35 +10:00
Jenkins
3ac57740c6 Merge "Move create_graph into config.py" 2017-05-27 10:22:08 +00:00
Jenkins
795ec082ed Merge "Create and use plugin/node abstract classes" 2017-05-27 10:22:01 +00:00
Jenkins
3b85cad420 Merge "Use networkx for digraph" 2017-05-27 10:10:31 +00:00
Jenkins
68393b6f9f Merge "Disable recommended package installations for zypper-minimal" 2017-05-26 17:34:02 +00:00
Jenkins
da7bfd2336 Merge "Add a more generic tree->graph parser" 2017-05-26 15:27:41 +00:00
Dirk Mueller
b4edb7d0eb Disable recommended package installations for zypper-minimal
This is consistent with how dpkg based images are configured
and minimizes the nodepool images drastically (avoid installing
texlive for example)

Change-Id: I98fb31bc0e06869e9770fae3dbd62f0d86acb879
2017-05-26 09:47:07 +02:00
Jenkins
da754520f6 Merge "Adding unit testing for configuration" 2017-05-26 06:45:54 +00:00
Ian Wienand
3fdd9df983 Move create_graph into config.py
This was suggested in a review comment in
I8a5d62a076a5a50597f2f1df3a8615afba6dadb2.  It works out quite nicely
because the BlockDevice() driver now doesn't need to know anything
about stevedore or plugins, and just works on the node list.  It also
simplifies the unit testing by not having to call create_graph through
a BlockDevice object.

Change-Id: I98512f6cf42e256d2ea8225a0b496d303bf357b8
2017-05-26 11:48:39 +10:00
Ian Wienand
deb832d685 Create and use plugin/node abstract classes
This completes the transitions started in
Ic5a61365ef0132476b11bdbf1dd96885e91c3cb6

The new file plugin.py is the place to start with this change.  The
abstract base classes PluginBase and NodeBase are heavily documented.
NodeBase essentially replaces Digraph.Node

The changes in level?/*.py make no functional changes, but are just
refactoring to implement the plugin and node classes consistently.
Additionally we have added asserts during parsing & generation to
ensure plugins are implemented PluginBase, and get_nodes() is always
returning NodeBase objects for the graph.

Change-Id: Ie648e9224749491260dea65d7e8b8151a6824b9c
2017-05-26 11:48:11 +10:00
Ian Wienand
75817ef205 Use networkx for digraph
This switches the code to use networkx for the digraph implementation.

Note that the old implementation specifically isn't removed in this
change -- for review clarity.  It will be replaced by a base class
that defines things properly to the API described below.

Plugins return a node object with three functions

 get_name() : return the unique name of this node

 get_nodes() : return a list of nodes for insertion into the graph.
  Usually this is just "self".  Some special things like partitioning
  add extra nodes at this point, however.

 get_edges() : return a tuple of two lists; edges_from and edges_to
  As you would expect the first is a list of node names that points to
  us, and the second is a list of node names we point to.  Usually
  this is only populated as ([self.base],[]) -- i.e. our "base" node
  points to us.  Some plugins, such as mounting, create links both to
  and from themselves, however.

Plugins have been updated, some test cases added (error cases
specifically)

Change-Id: Ic5a61365ef0132476b11bdbf1dd96885e91c3cb6
2017-05-26 11:42:10 +10:00
Ian Wienand
00da1982ce Add a more generic tree->graph parser
This moves to a more generic config parser that doesn't have plugins
parsing part of the tree.

I understand why it ended up that way; we have "partitions" key which
has special semantics compared to others keys and there was a desire
to keep it isolated from core tree->graph code.  But this isn't really
isolated; you have to reverse-engineer several module-crossing
boundaries, extras classes and repetitive recursive functions.

Ultimately, plugins should have access to the node graph, but not
participate in configuration parsing.  This way we ensure that plugins
can't invent new methods of configuration parsing.

Note: unit tests produce the same tree -> graph conversion as the old
method.  i.e. this is not intended to have a functional change.

Change-Id: I8a5d62a076a5a50597f2f1df3a8615afba6dadb2
2017-05-26 10:13:14 +10:00
Ian Wienand
7341542f2c Adding unit testing for configuration
Add a range of unit-testing for configuration parsing, graph
generation and mount-point generation.  Unfortunately there's some
global variable hacks, and some stubs, but it's a start.

Change-Id: I9e4f950c2c2ea656fc0c1a14594059fb4c62fa35
2017-05-26 09:44:19 +10:00
Dirk Mueller
f039a9b796 drop deprecated map-services/packages from zypper element
Change-Id: Ie3065dcc6aefccba93c02085e9977681d1b0535c
2017-05-25 23:43:21 +02:00
Jenkins
b7924c0c83 Merge "Produce API documentation" 2017-05-25 08:03:31 +00:00
Ian Wienand
78c0766bec Produce API documentation
There's an increasing amount of pydoc based documentation.  Output the
module reference and link it into the developers main page.

One fixup to the rst needed

Change-Id: I1d43a1fe1c735eb4559e3d2b40834d1c8115c586
2017-05-25 14:26:31 +10:00
Jenkins
0208f83a97 Merge "Set manifest permissions in the image" 2017-05-24 06:58:10 +00:00
Noam Angel
f1369a1add Set manifest permissions in the image
This is a follow-on to 57ef187632.

There's two things going on here; DIB_MANIFEST_IMAGE_DIR is *outside*
the chroot on the build host.  We copy the files here for posterity, I
guess.  MANIFEST_IMAGE_PATH is *inside* the chroot and are the files
we want to ensure are locked to root.

The prior change modified the permissions on DIB_MANIFEST_IMAGE_DIR.
So the first time you build, it works -- then the second time,
assuming you're using the same output filename, it hits the root-owned
manifest directories and causes a build failure.

I have built with this and checked that the manifest files in the
image are locked to root:

 $ virt-ls -a ./test.qcow2 -l /etc/dib-manifests
 total 32
 drwxr-xr-x  2 0 0  4096 May 24 03:39 .
 drwxr-xr-x 53 0 0  4096 May 24 03:39 ..
 -rw-------  1 0 0 15236 May 24 03:39 dib-manifest-dpkg-test
 -rw-------  1 0 0    35 May 24 03:39 dib_arguments
 -rw-------  1 0 0   137 May 24 03:39 dib_environment

Related-Bug: #1671842
Change-Id: I08319d0b5fcc461d40fe0be8427dcf0e37ad21e6
2017-05-24 15:20:55 +10:00
Jenkins
57c40a2ac4 Merge "Add dracut-regenerate elements" 2017-05-23 07:35:51 +00:00
Angel Noam
ba4f72f4f0 dhcp-all-interfaces.sh - Add support for InfiniBand interface DHCP
Change-Id: Ic2a9e2909a8086903257d43fbda97694baa339b4
2017-05-22 07:48:32 +00:00
Ian Wienand
bc58b5c515 Move parts of Partition creation into object
Move Partition() object creation into the actual Partition object,
rather than having the logic within the Partitioning() object

Change-Id: I833ed419a0fca38181a9e2db28e5af87500d8ba4
2017-05-20 06:44:39 +00:00
Ian Wienand
d013496ba0 Split partition into it's own file
Split Partition() into it's own file for clarity.  This will be
followed-on by less dependence between Partitions and Partition

Change-Id: I860f6a1787c0e4fe99f93919ac37cf7d80bfaae9
2017-05-20 06:44:39 +00:00
Ian Wienand
4e08765f87 Move exception to it's own file (again)
Moving the exception didn't cause problems in
I925ed62bdc808f0e07862f6e0905e80b50fbe942, but in later changes where
we split blockdevice.py up a bit more, we can get a bit tangled with
circular imports.

Change-Id: I8297483f64c4e1deecd5ec88ee40e9198bb83589
2017-05-20 06:44:39 +00:00
Andreas Florath
45272343c5 Add weights to digraph
Because in some cases (e.g. partitioning) the order is needed,
add weights to the digraph to get an (somewhat) stable
topological sort.

Change-Id: I5ef1acc6338ac93c593faa0eafe26cbed42ed887
Signed-off-by: Andreas Florath <andreas@florath.net>
2017-05-20 06:42:48 +00:00
Ian Wienand
9eb71a1fe0 Switch debian to deb.debian.org
Per [1] this is the "official" CDN mirror, which I think is the most
appropriate for the default.  I think this addresses the concerns
httpredir service, which I don't think ever quite got out of beta.

[1] https://wiki.debian.org/DebianGeoMirror

Change-Id: I55f2a00b8bbb0f0a20d3be229e4c2c32a7b69057
2017-05-19 20:10:40 +00:00
Jesse Keating
84d10dce57 Remove use of 'which'.
Instead, either use the bash built-in of type to ensure it exists. Since
which is an external dep, things can fail oddly in a constrained
environment.

Also add a dib-lint test for this.

Change-Id: I645029f5b5bfe1198c89ce10fd3246be8636e8af
Signed-off-by: Jesse Keating <omgjlk@us.ibm.com>
2017-05-19 12:43:36 -07:00
Yolanda Robla
f23ea63341 Add dracut-regenerate elements
This new element will allow to regenerate dracut
on the produced images, to enable different modules. It
relies on a yaml blob to specify modules and packages
needed. It defaults to installing lvm and crypt.

Change-Id: I292fb70cde41ee6053b7b81a67931bcdaaa6d664
2017-05-19 09:43:11 +02:00
Jenkins
94ab9e2e7e Merge "Set manifests to mode 600 and owner root" 2017-05-18 08:37:05 +00:00
Jenkins
ca04348393 Merge "Remove _config_error thrower" 2017-05-18 02:37:53 +00:00
Ian Wienand
b91207ae47 Remove _config_error thrower
"log and throw" is arguably an anti-pattern; the error message either
bubles-up into the exception, or the handler figures it out.  We have
an example where this logs, and then the handler in blockdevice.py
catches it and logs it again.

Less layers is better; just raise the exception, and use log.exception
to get tracebacks where handled.

Change-Id: I8efd94fbe52a3911253753f447afdb7565849185
2017-05-18 10:37:56 +10:00
Jenkins
812563f90c Merge "Remove PluginBase/NodePluginBase class" 2017-05-18 00:24:33 +00:00
Gregory Haynes
57ef187632 Set manifests to mode 600 and owner root
Manifests files can release sensitive information and therefore should
have restrictive permissions.

Change-Id: I64d6c830217a7d8b0172df2dc774079dcd1e2a68
Related-Bug: #1671842
2017-05-18 10:10:15 +10:00
Jenkins
c54e9fddfa Merge "Use fakelogger in test_blockdevice_mbr" 2017-05-17 23:59:18 +00:00
Ian Wienand
08e8471360 Use fakelogger in test_blockdevice_mbr
Capture logging instead of putting it on stdout

Change-Id: Ic0c9f25549a1984fecf8e912e5b14845e2363730
2017-05-17 18:49:12 +10:00
Ian Wienand
2d2b2725bd Remove PluginBase/NodePluginBase class
A majority of the "plugins" aren't implementing the plugin class.
Clearly we need some refactoring of the ideas here.  Remove for
simplicity.

Change-Id: If399a371b171f4fd17cfa5856fe55daca4c86e60
2017-05-17 09:03:42 +02:00
Yolanda Robla
5b305ffa7b Only unmount directories that are mounted
To avoid failures with double unmount, skip unmounting
the mountpoints that are managed by block device.

Change-Id: I228779eb9bf544a27a53e5017c87573023fd375a
2017-05-17 08:37:25 +02:00
Yolanda Robla
6d0b9abc0f Apply setfiles on all mountpoints
With new block device definition, where content of the image
can be mounted on different partitions, is not enough with
executing setfiles on root directory. Instead of that, expose
all the mountpoints on the image, and apply setfiles on them.

Change-Id: I153f979722eaec49eab93d7cd398c5589b9bfc44
2017-05-16 07:51:48 +02:00
Andreas Florath
e4e23897a1 Refactor: block-device filesystem creation, mount and fstab
This patch finalizes the block device refactoring.  It moves the three
remaining levels (filesystem creation, mount and fstab handling) into
the new python module.

Now it is possible to use any number of disk images, any number of
partitions and used them mounted to different directories.

Notes:

 * unmount_dir : modified to only unmount the subdirs mounted by
   mount_proc_sys_dev().  dib-block-device unmounts
   $TMP_MOUNT_PATH/mnt (see I85e01f3898d3c043071de5fad82307cb091a64a9)

Change-Id: I592c0b1329409307197460cfa8fd69798013f1f8
Signed-off-by: Andreas Florath <andreas@florath.net>
Closes-Bug: #1664924
2017-05-12 13:52:02 +02:00
Ian Wienand
b9c065de28 Remove args from BlockDevice() init
The args agument was only used to find the symbol for the getval
command.  Have the command pass the symbol to find in directly.  We
can therefore remove the args paramater to the BlockDevice() creation.

Change-Id: I8e357131b70a00e4a2c4792c009f6058d1d5ae9e
2017-05-12 09:36:23 +10:00
Ian Wienand
c74ba2fe74 Move to subparsers
Move argument parsing to subparsers, rather than positional arguments.
This better reflects the tool's role as a driver and allows
sub-commands to deal with arguments in a natural way.

Change-Id: Iae8c368e0f3fe47abfddb9e0a1558bd5b3423aee
2017-05-11 21:03:33 +10:00
Jenkins
b39b5cb5a5 Merge "Clear __init__.py from cmd move" 2017-05-11 10:35:22 +00:00
Ian Wienand
f76d68020b Clear __init__.py from cmd move
I accidentally dropped the clearing of this file when it moved to
cmd.py during rebase of I1919f6e865acae14ee95cd025c9c7b75ca266a9c

Change-Id: Ibe9fcde594770cb51c732cc253987308dc038083
2017-05-11 18:52:14 +10:00
Ian Wienand
0368052a2f Take --params from environment
DIB_BLOCK_DEVICE_PARAMS_YAML should be exported, and the
dib-block-device will take this as the value of --params.  Remove this
to simplify the command-line

Change-Id: I6764ed223ecd36f9d24e19f164b6a927380b410f
2017-05-11 17:36:51 +10:00
Ian Wienand
ff5b30db8a Create BlockDeviceCmd object
This creates a BlockDeviceCmd object to hold the main() function.
This doesn't really do anything different right now, but sets a base
for using argparse subparsers to handle the command-line

Change-Id: I4acf95ff4d554a3b4e7e2244ab1706631b98458f
2017-05-11 15:48:47 +10:00
Ian Wienand
a06c610a8c Move YAML parsing into cmd.py; default to env
This moves the YAML parameter parsing into the command-line driver.
It makes the argument optional so it can be taken from the environment
variable directly.  The parsed YAML is passed to the BlockDevice
object.

Change-Id: I6fa5e5b7d1fccfc7cf47d6e4a1fa6e560734680d
2017-05-11 15:22:41 +10:00
Ian Wienand
a9d8e51ec2 Move dib-block-device implementation into cmd.py
Move the command-line driver components into cmd.py.  No functional
change.

Change-Id: I1919f6e865acae14ee95cd025c9c7b75ca266a9c
2017-05-11 15:22:41 +10:00
Ian Wienand
47140293b6 Move blockdevicesetupexception.py into blockdevice.py
Less is more when it comes to code :)

Change-Id: I925ed62bdc808f0e07862f6e0905e80b50fbe942
2017-05-11 15:22:41 +10:00
Jenkins
fae8484e6b Merge "block_device: reorder imports" 2017-05-11 04:53:40 +00:00
Jenkins
bd7d7b052a Merge "Remove unused val_else_none" 2017-05-11 03:24:35 +00:00
Ian Wienand
9a8184ab4c Remove unused val_else_none
This function is unused

Change-Id: I8fe3e5452b95a639618a37a02f34b5b9f63ca43a
2017-05-11 11:53:08 +10:00
Ian Wienand
2a185ec6b6 block_device: reorder imports
Reorder imports to hacking standard (stdlib, third-party, project) [1]

[1] https://docs.openstack.org/developer/hacking/#import-order-template

Change-Id: I4aa73321e1e796ef6b8b079e42f90bf5c75388fe
2017-05-11 10:38:55 +10:00
Ian Wienand
0d8c4270c0 exec_sudo: check cmd for str, log output and raise exception
To avoid any confusion, commands passed to exec_sudo() should be a
list of "str"s.  Log a message if we see unicode issues.

This also adds a debug trace of all output.  stderr is captured.

This is modified to raise CalledProcessError on failure, like
check_call().  Calls that are ok to fail will need to explicitly catch
and ignore this.

The two calls that we expect to fail are wrapped

We wish to try rolling back if one of these command raises an
exception.  Modify the create handler to initiate rollback on all
exceptions.

Change-Id: Iee4fa41ffaf243e4728bf3a5eeec5c8fa8d2dadc
2017-05-11 09:45:25 +10:00
Ian Wienand
6f90daca7f Use check_output
The await function is essentially a non-standard check_output call.
Let's use standard calls to increase maintainability.

Change-Id: I2c25e1cd7122791fcaa86b46bd801e661471bc9e
2017-05-09 15:49:39 +10:00
Yolanda Robla
59a1fc6546 Add sort_mount_point method
As this method can be introduced without any dependency,
provide it on an independent change to simplify reviews.
This is a partial refactor based on
I592c0b1329409307197460cfa8fd69798013f1f8

Change-Id: Idaf3d2b3b3e23d0b9d6bc071d67b961a829ae422
Co-Authored-By: Andreas Florath <andreas@florath.net>
2017-05-05 17:29:53 +00:00
Yolanda Robla
fb70a49ba2 Add a test to validate we can build debian vms
This is a partial refactor from change
I592c0b1329409307197460cfa8fd69798013f1f8

Change-Id: I8822e68e41c4ebd47eea9ffed4557efc130a7bf7
Co-Authored-By: Andreas Florath <andreas@florath.net>
2017-05-05 19:17:39 +02:00
Yolanda Robla
9b75eda51a Introduce exec_sudo command
Add a new method in the block device library called
exec_sudo, so it can be reused.

This is a partial refactor of change
I592c0b1329409307197460cfa8fd69798013f1f8

Change-Id: Id621f6d029e1275a35c4fd3f19b57c8518076134
Co-Authored-By: Andreas Florath <andreas@florath.net>
2017-05-05 16:11:41 +02:00
Adam Harwell
196b44f38a Fix py3 error in block-device
This is breaking the octavia py3 gates.
Introduced by: I7aa4fe0466e44846d8fa3194575d446fe4b5b2e6

Change-Id: I3d86482a2999197a60a81d42afc5ef7a6e71e313
2017-05-04 15:55:22 -07:00
Yolanda Robla
e21935626b Refactor block-device base functions.
As part of the final steps, refactor the bits belonging
to block device and functions. This is a partial refactor
from I3600c6a3d663c697b59d91bd3fbb5e408af345e4

Change-Id: I7aa4fe0466e44846d8fa3194575d446fe4b5b2e6
Co-Authored-By: Andreas Florath <andreas@florath.net>
2017-05-04 19:54:18 +00:00
Yolanda Robla
08c36e4bf8 Add refactor of tree-like vs graph
Introducing the refactors of the block device to allow a tree-like
configuration, and start using it for the partitions level.
Based on patch I3600c6a3d663c697b59d91bd3fbb5e408af345e4

Change-Id: I58bb3c256a1dfd100d29266571c333c2d43334f7
Co-Authored-By: Andreas Florath <andreas@florath.net>
2017-05-03 05:27:43 +00:00
Ian Wienand
91ba21dd6c Set LC_ALL in disk-image-create
It seems that the redhat nodepool job is quite reliably geting a
"floating point" error during centos image build.  This happens after
03-yum-cleanup which is pruning the locales.  This might be a
red-herring, since the logs are full of

  /bin/bash: warning: setlocale: LC_ALL: cannot change locale (C.UTF-8)

I think in our recent de-puppetisation of hosts, something might have
changed that is setting LC_ALL=C.UTF-8 for the jenkins user, at least
on Ubuntu.  This is a problem for centos, as it doesn't have C.UTF-8
locale.  I then think using the invalid locale is what leads the the
floating-point error when doing some maths in dib-run-parts to
calculate runtimes.

We are currently overriding LANG, but we really want LC_ALL to ensure
this applies globally.

Change-Id: I8e7cae093c4b32e0d20b73ae0086f14c7cc6a9cb
2017-05-03 11:31:11 +10:00
Jenkins
629b81bc9b Merge "Refactor block_device: isolate the getval call" 2017-05-01 14:45:13 +00:00
Yolanda Robla
943f1ccf04 Refactor block_device: isolate the getval call
Add a new getval call that allows to retrieve values
from the block device. Also isolating the block device
information into a 'blockdev' dictionary entry, to better
return it with the getval command.

This is a refactor from the original code at
I3600c6a3d663c697b59d91bd3fbb5e408af345e4.

Change-Id: I93d33669a3a0ae644eab9f9b955bb5a9470cadeb
Co-Authored-By: Andreas Florath <andreas@florath.net>
2017-05-01 12:22:52 +02:00
Jenkins
83ea4a17b8 Merge "Clean out apt index caches at end of image build" 2017-05-01 08:09:07 +00:00
Jenkins
ead4a50c2a Merge "Make Gentoo package updates work more often" 2017-05-01 08:05:04 +00:00
Jenkins
c36ac99458 Merge "Turn off strict_id mode for Ec2 datasource" 2017-05-01 08:04:52 +00:00
Andreas Florath
803d40b0c6 Refactor block_device: passing command line parameters
The original approach was to pass each and every command
line parameter to the block device.  While the block device
functionality gets extended, this is not any longer practical.

Instead of passing in all the parameters separately this patch
collects these in a YAML file that is passed in to the block device
layer.

Change-Id: I9d07593a01441b62632234468ac25a982cf1a9f0
Signed-off-by: Andreas Florath <andreas@florath.net>
2017-04-23 09:51:18 +00:00
Matthew Thode
6f51fbe355
Make Gentoo package updates work more often
Some package updates are more complex and require things like --backtrack=99 to
be passed to emerge.  We also try harder to ensure the system is in a consistent
state as a last step.

Change-Id: Ia5d3514e8b2a6cb2d656ade997cebb798d9c0a47
2017-04-22 19:34:11 -05:00
Paul Belanger
1ce16a987b
Add yum-utils as EPEL dependency
With 8e822768f9 we added the ability to
disable the EPEL repository, however we need yum-utils to use
yum-config-manager.

Change-Id: Iea445f84494fd9a89fd93e9b35f920eb5e55211d
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2017-04-20 17:01:00 -04:00
Simon Leinen
0ff40cf63c Turn off strict_id mode for Ec2 datasource
Recent changes in the default configuration of cloud-init in Ubuntu
cause warnings when the Ec2 datasource is used on non-Amazon clouds,
see https://bugs.launchpad.net/cloud-init/+bug/1660385

We explicitly select the previous behavior when an Ec2 datasource is
desired.

Change-Id: Iebad8f6c0017fe08013dd5fe667c6132158b71cd
Closes-bug: 1683038
2017-04-19 09:47:56 +02:00
Paul Belanger
3e17ee3f48 Clean up dib-python symlink
We only need dib-python when we build the image, no need to leak it to
the final product.  Remove it in cleanup.d outside the chroot so
nothing can be using it.

Change-Id: I1e229caad7968fb3ab8e44ecdda427e174088d2d
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2017-04-12 14:33:57 +10:00
Ian Wienand
7a962e9d1c Skip python3-virtualenv on <= trusty
If DIB_PYTHON_VERSION is < 3 on the !redhat path, that means we're on
an older platform that may not have python3-virtualenv packages.  Skip
install.

Ensure the order of operations happens by forcing the installs

Also add a note about limited platform support (patches welcome :)

Change-Id: I18412767f0ebf946d557a0a126285369e96af159
2017-04-12 06:36:20 +10:00
Ian Wienand
79d4113cbe pip-and-virtualenv : install python2 & 3, and default to 2
Recent changes in project-config have shown that we leave the system
in an inconsistent state when installing from source.  On fedora, we
will have installed the python2 packages, but then used $DIB_PYTHON to
install python3 pip from source!

This tries to clarify the situation.  As described in the document,
with package installs, we just install the $DIB_PYTHON packaged
versions.

Source installs want to take over the global namespace.  This is the
price you pay for running the latest versions outside package managers
:) The only sane thing seems to be for us to normalise python2 &
python3 versions of pip, setuptools and virtualenv and then hacking
things such that "/usr/bin/pip" and "/usr/bin/virtalenv" remain
defaulted to python2 versions.

Documentation is added

Change-Id: Ibc6572b89e256d1f48b7fe7c672b8b9524dc704f
2017-04-11 18:59:11 +10:00
Ian Wienand
ffd4820d59 Install pip with python interpreter
Currently we install pip/virtualenv with "/usr/local/bin/dib-python".
This means that every time you create a virtualenv, the python
interpreter inside it is called "dib-python" which is confusing.

Add an env var DIB_PYTHON that points directly the to interpreter
available during build, for use when running scripts.

Change-Id: I88ad3c9eb958d58db4631d9b27bc2c592f970345
2017-04-11 18:59:09 +10:00
Jenkins
5c1579f0d6 Merge "Unify and fix doc of several Debian and Ubuntu elements" 2017-04-10 01:25:01 +00:00
Jenkins
a076bf66d6 Merge "Move do_extra_package_install to run in install phase" 2017-04-10 01:24:55 +00:00
Noam Angel
95503b4201 Move do_extra_package_install to run in install phase
This change move "do_extra_package_install" from pre-install to install
phase.

Extra packages are added by user request using the flag "-p", This
package should not be something the elements depend on.
The reason behind this patch is to move the extra package install to
a proper phase, Also more reasonable if base element run package update
to be before we install extra packages.

Change-Id: I68cc773aba9aa01743f0dda9f4e635e4cac2a282
2017-04-10 08:32:47 +10:00
Paul Belanger
1778fb57db Clean out apt index caches at end of image build
Apt gets confused if it talks to a mirror with an older index than the
index currently cached by apt. This can happen when image builds use a
newer index than the booted image. Avoid these problems entirely by
removing those index caches at the end of image building.

Change-Id: I245d516ee8a44831b2c29612b782bad555c48a3f
Co-Author: Clark Boylan <clark.boylan@gmail.com>
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2017-04-09 11:57:23 -04:00
Jenkins
2d178e43e1 Merge "Use DIB_PYTHON_EXEC to run commands" 2017-04-07 20:29:30 +00:00
Jenkins
347833856e Merge "Ignore missing path in unmount_dir" 2017-04-07 15:59:47 +00:00
Jenkins
3f5c558753 Merge "Fix package-installs-v2 output" 2017-04-07 13:24:17 +00:00
Jenkins
1b5045c563 Merge "Basic logging for package-installs-squash" 2017-04-07 13:24:10 +00:00
Jenkins
08ad06f86d Merge "Fix opensuse-minimal element on non-SUSE hosts" 2017-04-07 12:06:31 +00:00
Andreas Florath
ee8fa3c4bb Unify and fix doc of several Debian and Ubuntu elements
This patch removes three nearly-copies of debootstrap documentation
and fixes some documentation aspects.

Change-Id: Ief7794f5c1abad73788c063af6c862472cd34744
Signed-off-by: Andreas Florath <andreas@florath.net>
2017-04-07 05:56:50 +00:00
Ian Wienand
f068e6aa6e Fix package-installs-v2 output
The current output for package-installs-v2 is inscrutable [1]

The problem starts with process_output() which is not capturing
stderr.  This means that any stderr output is dislocated from any
stdout output around it.  This is *really* confusing as you get a
bunch of seemingly meaningless stderr output from any calls before you
see any stdout (e.g. in [1] you can see random yum error output that
should have been with the yum call)).  The simplest thing to do is to
redirect stderr to stdout which keeps everything in sync.

This causes a slight problem, however, because pkg-map outputs both
status information and errors on stderr.  To work around this but
maintain compatibility, we add a "--prefix" argument that prepends
mapped packages from pkg-map with a value we can match on.  The
existing status/debug output from pkg-map is low-value; modify the
call so that it will be traced only at higher debug levels (e.g. -x
-x).

The current loop is also calling pkg-map for every package in every
element (this is why in [1] the same message is repeated over and
over).  This is unnecessary; it only needs to pkg-map once for each
element, giving the package list as the arguments.  Create package
lists by element and pass those to pkg-map.

As a cleanup, there is no point in printing e.output if the
process_output fails for the install because we are already tracing
it; i.e. the output, even for failures, is already in the logs.
Printing it again just duplicates the output.

[2] is an extract showing what I feel is a much more understandable
log output for a fairly complex install.

[1] http://paste.openstack.org/show/595118/
[2] http://paste.openstack.org/show/595303/

Change-Id: Ia74602a5d2db032a476481caec0e45dab013d54f
2017-04-07 13:48:53 +10:00
Ian Wienand
facabe585c Basic logging for package-installs-squash
Some basic info logging and debugging for package-installs-squash.

Change-Id: If7c227cf0a53b71d334b5f5a6075120f2a7bf3a7
2017-04-07 13:48:52 +10:00
Adam Harwell
f06231ee08 Use DIB_PYTHON_EXEC to run commands
DIB_PYTHON_EXEC was added in I5fab0e192c3a2dad8f60e821c184479e24e33bcd
to export the python that disk-image-create is running under.  If dib
is installed with python3 then just calls "python" (python2) to run
sub-scripts, it fails to find itself.

This has been tested in devstack gates that use py3x

Change-Id: Ia1028972bfc0517b468b279aab9decdbcd7424ca
2017-04-07 07:40:58 +10:00
Corey O'Brien
0ea7e927de Fix typo in pip-and-virtualenv
Change-Id: I3058b45fff037106eba0267fd6629707a5ebb8b1
2017-04-06 10:19:40 -04:00
Corey O'Brien
aa90f7991a Ignore missing path in unmount_dir
If the path is missing, unmount_dir currently exits with an error which
unintentionally aborts cleanup efforts early. This change makes
unmount_dir idempotent by exiting successfully if a directory doesn't
exist.

Change-Id: I1491b4344e8569ecb2833f44baee445a89a39d61
2017-04-06 10:08:13 -04:00
Jenkins
10baebe1e4 Merge "Run dib-run-parts out of /tmp" 2017-04-05 10:18:55 +00:00
Jenkins
0279258ac6 Merge "Don't provide dib-run-parts" 2017-04-05 07:01:29 +00:00
Jenkins
221c44f24c Merge "Adding aarch64 support for CentOS7" 2017-04-05 05:10:05 +00:00
Ian Wienand
6802cf7100 Run dib-run-parts out of /tmp
The dib-run-parts element was copying our internal version of
dib-run-parts into /usr/local/bin to be used running scripts inside
the target chroot.  However, it never cleaned up after itself.  This
means all images were left with an unmanaged local install of
dib-run-parts.

This copies dib-run-parts into the hooks directory of the chroot and
runs it from there.  It is cleaned up automatically on the exit path.

The dib-run-parts element is no longer required and it has been
removed from all dependencies.  It is left with a deprecation notice
in the README.  For compatability we convert it to simply install
dib-utils.

Codesearch shows no users depending on this unintentional implicit
install.  Note os-refresh-config depends on dib-utils and thus will
have an explicitly installed version.

Partial-Bug: #1673144
Change-Id: Ia2e96c00a4246c04beb96c17f83b8aefb69219ca
2017-04-05 13:11:22 +10:00
Ian Wienand
fd424757a6 Don't provide dib-run-parts
It was an oversight during v2 development for dib to start providing
dib-run-parts.  The intention was for dib to use a vendored
dib-run-parts directly from $_LIB and have no dependencies on
dib-utils at all.  By exporting dib-run-parts, we created an
unintentional conflict with the dib-utils package which provides the
same script.

Tools that depend on dib-utils are unaffected by this
(os-refresh-config).

The only tool that installs diskimage-builder and then assumes
dib-run-parts is available in the path is instack.  I have proposed
Ibfe972208df40fa092b11b5419043524c903f1b4 to modify that to use our
internal version.

Change-Id: I149c345d38d761a49b3a6ccc4833482f09f1cd05
2017-04-05 13:11:20 +10:00
Ian Wienand
8e822768f9 Add flag to disable EPEL
Add DIB_EPEL_DISABLED flag that allows installation of the EPEL repo,
but to have it disabled by default.  This will help when you have
unavoidable EPEL dependencies, but want to make sure you only pull
specific things in with "--enablerepo" calls when installing those
packages.

Change-Id: Iedf6167a7cd69418255ebbee095aea04c50d73fd
2017-04-05 05:59:40 +10:00
Jenkins
6e7d7e77ad Merge "Use stevedore for plugin config of block device" 2017-04-04 08:13:45 +00:00
Hironori Shiina
eaa0569ac5 Fix code-block in README of rhel7 element
A code-block in README of rhel7 element is not rendered as expected.
This patch fixes it to be rendered correctly.

Change-Id: Ie8f4c05edd1dd93314290682e4b2734622894e15
2017-04-03 03:56:22 +00:00
Dirk Mueller
1bdeaadc95 Fix opensuse-minimal element on non-SUSE hosts
zypper on non-suse hosts is not parsing the pattern repodata because
those are marked as an inofficial extension to the repomd specification.
This is not a big issue as there is meanwhile in newer openSUSE
distributions a pattern *package* that depends on the same packages like
the pattern would do, so we can just replace it with that.

Change-Id: I0c8f713075bd7e5bf1d425f81933b4666654add7
Depends-On: I34e98f0f7693859ed05011b008334628adff612f
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2017-04-02 19:37:01 -04:00
Paul Belanger
87236af75c Have simple-init enable network.service
When a glean is running on centos with multiple NICs, it will try to
systemctl enable network.service multiple times for each interface.
Because of systemd magic, it is possible for the systemctl command to
fail in a race condition.

glean shouldn't be enabling network.service during boot in
pre-networking phases (Ib2b618dd975ca44e9c6b0a2c9027642ffc46b9b0).  I
have proposed I8319f1ed6498a9d447950c2b4b34bca59e7b97e4 to remove this
and document the behaviour.

This also bring across suse's version
(I20bffabd333ea290d8712ec2a467f2b2d5678f3a)

Change-Id: I89d9443cb61e287bd0d9da3f48315272218ee335
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2017-03-28 19:28:51 +11:00
Andreas Florath
57c9e0bb41 Use stevedore for plugin config of block device
This patch introduces stevedore plugin mechanism for use
with the block device layer.  This makes it possible that
other projects pass in their own block device plugins.

Change-Id: Id3ea56aaf75f5a20a4e1b6ac2a68adb12c56b574
Signed-off-by: Andreas Florath <andreas@florath.net>
2017-03-24 21:42:47 +00:00
Jenkins
f128d08cf0 Merge "Use correct Ubuntu distro url on non-x86 arches" 2017-03-24 04:42:59 +00:00
Jenkins
94e2973785 Merge "functests: skip qcow2 generically but add specific test" 2017-03-24 01:07:10 +00:00
Mikhail S Medvedev
4d4417c9a6 Use correct Ubuntu distro url on non-x86 arches
diskimage-builder usually provides defaults that work out of the box.
One default that does not work outside of x86 land is Ubuntu distro
mirror url. Considering there are only two valid default options, we can
automatically choose a better default.

This patch changes behavior only for architectures known to be using
http://ports.ubuntu.com/ubuntu-ports. All others still would use
http://archive.ubuntu.com/ubuntu as default. It provides some guarantee
that we do not introduce a regression.

Change-Id: If95a64bac0c88f30736da4bae7f1fdce126c0bf6
2017-03-23 09:27:25 -05:00
gecong1973
8ff6f51b81 Typo fix: curent => current
Change-Id: I5ce5a98c355310d6f5f6c09e19993b01ed025f0e
2017-03-23 15:02:20 +08:00
Ian Wienand
76ae374413 functests: skip qcow2 generically but add specific test
We somewhat discussed skipping qcow2 generation previously in
I9372e195913798a851c96e62eee89029e067baa1.  As recent issues with PPC
testing have shown, we are not actually testing the "vm" element and
hence the bootloader path in the functional tests.

I don't think we need to test this on every element; it overlaps
somewhat with the testing done by the nodepool jobs which build full
images and boot them.  I also didn't want to introduce a separate run
for this.  Thus it seems valuable to at least have one element
enhanced to do this installation and conversion in our default tests
for basic sanity.

This disables qcow generation by default, as per the other change, but
allows an element to drop a file that will override the output
formats.  The Xenial element is modified to produce a qcow2 using
this, and also introduces a dependency on the "vm" element so it tries
to install the bootloader.

We now exit if the .qcow2 fails to build as well.

Change-Id: I1a6acefe52f8c696c39b2d592fdc7ae32a87e6fe
2017-03-23 13:49:24 +11:00
Jenkins
2ea3a434f7 Merge "Add default PPC block-device layout" 2017-03-23 01:03:13 +00:00
Jenkins
f7999f86e4 Merge "Capture output in _exec_sudo" 2017-03-23 01:01:13 +00:00
Ian Wienand
19fcd263d0 Add default PPC block-device layout
Add a default PPC block-device layout.  I've extracted this into
separate yaml files for ease of editing and to facilitate things like
longer comments.

This is not sufficient to get PPC images working, but it is required.

Change-Id: I09e5d1ed92260bdb632333f5203dd7e70d512dc8
2017-03-23 09:44:01 +11:00
Ian Wienand
a8f2eaded8 Capture output in _exec_sudo
The stdout of the script is captured, so anything coming out from
these commands needs to be captured.  Move to check_process and show
the output as part of an error log in failure case.

Change-Id: I1150375cdc479d4f19b8ddeb49a824ab16fdf831
2017-03-23 09:31:37 +11:00
Dan Radez
06576a02f0 Adding aarch64 support for CentOS7
Using delorian and delorian-deps for aarch64
this will produce an overcloud-full image

Change-Id: Id0dc83d9c558b0c0bbfaed7727883541d140472c
2017-03-22 10:46:54 -04:00
Yolanda Robla
8a3a3ffe6e Fix typo in CMDLINE env var for bootloader
The correct spelling of the env var needs to be
DIB_BOOTLOADER_DEFAULT_CMDLINE

Change-Id: I4ae2aed928723588e736b63283a54eca5f4b4c3b
2017-03-22 12:02:37 +01:00
Jenkins
f1d53f2e31 Merge "Use sphinx warning-is-error" 2017-03-15 23:56:59 +00:00
Ian Wienand
fea6ab1624 Use sphinx warning-is-error
Sphix 1.5 (I9e7261c4124b71eeb6bddd9e21747b61bbdc16fa) includes
"warning-is-error" which supersedes pbr's warnerrors.  Enable this and
fix up the resulting failures

 - trailing lines for lists in element_deps directive
 - missing README's that are linked
 - syntax error and highlighting in building instructions

Change-Id: I6549551b4a9bf47076c9811a7a38a666cbea2a50
2017-03-14 14:49:49 +11:00
Michael Johnson
fae8004e4d squash-package-install to use the correct python
99-squash-package-install in the package-installs element does not
know which python environments the requirements were installed into.
This can cause it to select the wrong python to run the
package-installs-squash script.

Co-Authored-By: Adam Harwell <flux.adam@gmail.com>
Change-Id: I5fab0e192c3a2dad8f60e821c184479e24e33bcd
2017-03-13 20:24:39 +11:00
Ian Wienand
85985cdadc Make our virtualenv source py3 safe
execfile() has gone in python3, and if you google various
stackoverflow results, python-dev mailing list threads and other
projects it seems runpy is considered one of the better solutions.
This should be py2.7 safe too.

Change-Id: I18077ba9d603752492cc81f260e12710981f4dff
2017-03-13 20:22:27 +11:00
Ian Wienand
b04c58eca4 blockdevice.py: python3 fixes
keys() is an iterator in python3, so we need to make it list before
finding the first element

Change-Id: Ic158c7b2901c1a34ff417d3432fcefed4760ce24
2017-03-13 19:33:07 +11:00
Jenkins
8b19f7f99d Merge "Fix dib-init-system for Debian Jessie and Debian Stretch" 2017-03-13 03:29:41 +00:00
Andreas Florath
20389d755f Fix dib-init-system for Debian Jessie and Debian Stretch
On Debian Jessie and Debian Stretch systemctl is in /bin.
If the package systemd-sysv is not installed the script
dib-init-system did not find the init system.
This patch fixes the problem: it also looks in /bin
for systemctl and if found decides for systemd.

Change-Id: I5a18052a070bad5e16b14672237a1e2b38513949
Signed-off-by: Andreas Florath <andreas@florath.net>
2017-03-11 14:04:13 +00:00
Andreas Florath
c99ea56726 Replace architecture-emulation-binaries with qemu-debootstrap
The architecture-emulation-binaries element does for a limited number
of architectures on Ubuntu exactly the same as `qemu-debootstrap`.
`qemu-debootstrap` comes in the qemu-user-static package that is
anyhow used when cross-building an image.

This patch replaces the complete architecture-emulation-binaries
element with a call to qemu-debootstrap.

Change-Id: Ib9667307bfd3ff7592444a2ec5b04aa5365a1872
Signed-off-by: Andreas Florath <andreas@florath.net>
2017-03-11 13:58:37 +00:00
Yolanda Robla
11275c212b Send custom parameters in bootloader to GRUB_CMDLINE
Depending on the types of deployment (security, nfv...) some extra
kernel flags are needed on the images. This change exposes the
DIB_BOOTLOADER_DEFAULT_CMDLINE parameter, defaulting to the
existing 'nofb nomodeset vga=normal', that will allow to modify
these boot params.

Change-Id: I67d191fa5ca44a57f776cb9739a02dd71212969c
Closes-Bug: #1668890
2017-03-08 15:43:46 +01:00
Yolanda Robla
96504a4de0 Use OrderedDict for partitions instead of simple dictionary
The order of the partitions is important, it needs to be preserved.
If using a simple dict, this is not happening. As a consequence,
checks like 'primary partition being first' are failing because the
dictionary sorts the partitions randomly.
Switched to OrderedDict solved the problem, as it preserves the
ordering it gets from the yaml blob.

Change-Id: Icfa9bd95ffd0203d7c3f6af95de3a6f848c2a954
2017-03-07 15:55:20 +01:00
Andreas Florath
866a06f92d Refactor: block-device partitioning cleanup
Now that the main partitioning refactor patch is merged, there is
a small relict of handling partitions still in the disk-image-create
main.

This patch moves the functionality from disk-image-create to the
block-device/partitioning module: it is mostly a rewrite of the
original bash code in python.

Change-Id: Ia73baeca74180a7bc9ea487da03ff56d6a3070ce
Signed-off-by: Andreas Florath <andreas@florath.net>
2017-03-07 18:43:09 +11:00
Ian Wienand
f1776d72aa Merge branch 'master' into merge-branch
Change-Id: Ib3c4b0030c94459a8c2eca836a16e42edf1accd4
2017-03-02 13:49:29 +11:00
Andreas Florath
003fc17208 Fix #1627402: fix grub2 element for ubuntu xenial
Looks that the special handling for Ubuntu is not needed any longer
(its a pity that there are no detailed comments...).

The grub2 element is a second implementation of the bootstrap element
- but because there are some features that come only here, e.g. efi
boot, it should be working as long as this is not implemented in the
bootloader element.

Change-Id: I74269116ea30b84f3259805720d5cd1616f960c5
Signed-off-by: Andreas Florath <andreas@florath.net>
Closes-Bug: #1627402
2017-02-11 18:44:18 +00:00
Ian Wienand
7a155e08bf Merge branch 'master' into merge-branch
Change-Id: I28e4c7837d84e8b66eff3d182666c5a87a9e3c9b
2017-02-09 13:35:53 +11:00
Andreas Florath
fdd2c4b236 Semi-automatic doc generation of element dependency
Currently there is no description of dependencies in the generated
documentation of the elements: therefore a user of an element does not
know which other elements are automatically included and e.g. which
configuration options are available. In addition there are some
copy&pastes of parts of the README.rst scattered thought different
Ubuntu and Debian specific elements.

This patch adds a semi-automatic generation of dependency information
of all elements.  Nevertheless these are not automatically included.
The author of the element's README.rst can decide if and where the
dependency information should appear and can use the descriptor

.. element_deps::

for this.

This patch adds the dependency information for some Debian and
Ubuntu patches - and creates the base for later removing the
duplicated parts.

A call is added to element_dependencies._find_all_elements() to
populate reverse dependencies for Element objects.

(This is a reworking of I31d2b6050b6c46fefe37378698e9a330025db430 for
the feature/v2 branch)

Change-Id: Iebb83916fed71565071246baa550849eef40560b
2017-02-09 09:50:30 +11:00
Jenkins
0f224d3ae6 Merge "Check return of _load_state" into feature/v2 2017-02-06 23:41:50 +00:00
Andreas Florath
df1f583903 Check return of _load_state
In error scenarios there might be no loadable state available.
This patch adds some robustness handling this scenarios
by checking the result and possible bailing out.

Change-Id: I02e2731b14bec22c22db08d60d8d15db1911a84e
Signed-off-by: Andreas Florath <andreas@florath.net>
2017-02-04 19:21:48 +00:00
Andreas Florath
a8953dd277 block-device: change top level config from dict to list
With the old configuration structure it was only possible
to use one image and one partition layout.  The new
block-device configuration uses a list at top level;
therefore it is possible to use multiple instances
of each element type.

Change-Id: I9db4327486b676887d6ce09609994116dbebfc89
Signed-off-by: Andreas Florath <andreas@florath.net>
2017-02-03 20:15:39 +00:00
Ian Wienand
3f8800832a Release 1.27.0
-----BEGIN PGP SIGNATURE-----
 
 iQEcBAABAgAGBQJYk8+9AAoJEBty/58O8cX8LdIH+wU/VrEVs0XYohiL6DUgabzs
 112U3UUihH5xMc/ca9Tarx+XwEvfMZkwYN2Qr0JoRJjmSt2AL6AezUhGSV+98vaY
 iQEccaFDFYlyDHm4V2r7N1xwS0B3mx87FPqVQQSUKlc3JsQxCy4o9RtD9aM8Gvqy
 +gAxMxL3p3O131K0Rvb0U5lC1FLgft9SuljCV8i5nU4/HdoryD6hedz2/ss8a9KG
 KKEdBKvPBKn73+nb8peQD/VXpej9C31r87q5VEjUsZkJ7gduY/qYLlGGgoBQqAXN
 WQ/ef1RkQKW5ba2jsjnk7fdOrA0+wYENxorR2WecuZbe2ieXw6fP3lYiD6VeWsM=
 =IUuh
 -----END PGP SIGNATURE-----

Merge tag '1.27.0' into merge-branch

Release 1.27.0

Change-Id: I9f6948636cae6d375d1d8315976504021f5a3bbb
2017-02-03 11:49:45 +11:00
Ian Wienand
adf39c52cf Release 1.21.1
-----BEGIN PGP SIGNATURE-----
 
 iQEcBAABAgAGBQJYW2GoAAoJEBty/58O8cX8uSAH/15dJsglP6Zie7jSSJcR6k+e
 PJembHn9qrqrCjmJ5EwakojySaaLhwEJKvlP54OU9v7pmUXL9gJtK2OzW54LQ41g
 xBHIu0Pg4z7juyHm9+1P2Sr7Mzs1pVSEbsIYpDYUU19eghI1EAeIj3I1woKgajN7
 JlI61j3r67G6EAVtPOnmD1jvXS8CrtjiJ9wtWTH20pWfmksovg/GuXUCZrLkAAhO
 NcK35CdMii1Hkr7XOH424La/Ar+3qfUX18ZkbJY6yHzkq/ityTzzKOFjAaDl2Jg9
 WNc+SLCVYpPhPwgt7miTywamUNj3ZviA5/Hd8fuLXmtHSLQ23WOtBiaQMLtwXHs=
 =8dIl
 -----END PGP SIGNATURE-----

Merge tag '1.26.1' into merge-branch

Release 1.21.1

Change-Id: Ib9eb3dd1d384fc5b6a9846608216e056c57a173a
2017-02-02 20:36:23 +11:00
Ian Wienand
19f13958f7 Handle blank lines in element-deps
When merging feature/v2 we have re-enabled the centos-minimal test
which has a blank line in the element-deps.  The modified get_elements
functions added in If97bcd21e45de1b5ed91023fdc441a4617051a6b were not
handling this.

Fixing this, we can extract this into some common code for
element-deps and element-provides.

Also, I noticed while debugging that the element list is an iterator
from reverse(), so printing it doesn't help.  Fix up the debug output.

Change-Id: I8e9b079f586e8a36dba3797a0fc03ca1b2041a04
2017-02-02 18:18:46 +11:00
Ian Wienand
bfca36c772 Release 1.25.2
-----BEGIN PGP SIGNATURE-----
 
 iQEcBAABAgAGBQJYV1yqAAoJEBty/58O8cX8hLwIAKP66w6MdPN8PDgUOteui/Sx
 N0UFKJ9yR4GQOAP0NffPLjch5/g0iJLs3eFKOhtGC1LjbDjpVgjX8vW18ib8wBZK
 GemOZPF3uxg8FROrZF1vpoDy/cHgL1YV10hCnwdjN/r9rb8zOuSabqjW+Dennj2n
 fZ0SJfa8Owfudn3YxGuOymVb/wMtEloDmVGBEI1Y+h7osELCCDi3OXmwsA8qMsdl
 cTwbeugBs4PlOVbZUK/JKGuwIHKgPnDYzYu5KpXw77/MdjGT0fo5Tlq5AOBDI2sC
 9JOFEBDli4Ro05VwvI58ADMpvvOax+9EvOhLbB1dRPdZl21Iyb6gOdy2PUbFO0c=
 =aKxq
 -----END PGP SIGNATURE-----

Merge tag '1.25.2' into merge-branch

Release 1.25.2

Change-Id: I698bcf2e82117bd81649cd065a7af5cac85990c7
2017-02-02 11:20:00 +11:00
Ian Wienand
b6e631360f Release 1.24.0
-----BEGIN PGP SIGNATURE-----
 
 iQEcBAABAgAGBQJYP+r7AAoJEBty/58O8cX8ZcMIAI7Yy2BfGqY/jfyLYNryATSu
 g78McMxAeOiPYII+KKyVkeWsChKJUafLZr/ppO95pm7ZLXeHQeC4MHDI68RmghIX
 HVXn4hPYzwlZ+Yr4hqaEf11H+vw0TDxo2a5bxlQZw+FCx9+3lkR60QxWf8a6QCxv
 nLqKIhtpwGi9mIyvAya+MaOmHMHqbNYgIBHKP0yXhTdxoHZoJWZeXN1rkelwAF9N
 TS1+KwIk+uemn1ZlqzYDWj8euyFyAv8L0WlF0ztCKe7TAFvNeMo26fLykeKHVDEo
 GC5R8MshiOtihPC972rNIponIzd62AetTlDxfK+gcUZO21pNTrlgdLX5Jn04fo0=
 =BpQc
 -----END PGP SIGNATURE-----

Merge tag '1.24.0' into merge-branch

Release 1.24.0

Change-Id: I23cfa6f8dc66bb6a1d957a49a5b468cfd47c005b
2017-01-31 14:14:19 +11:00
Andreas Florath
ec7f56c1b2 Refactor: block-device handling (partitioning)
During the creation of a disk image (e.g. for a VM), there is the need
to create, setup, configure and afterwards detach some kind of storage
where the newly installed OS can be copied to or directly installed
in.

This patch implements partitioning handling.

Change-Id: I0ca6a4ae3a2684d473b44e5f332ee4225ee30f8c
Signed-off-by: Andreas Florath <andreas@florath.net>
2017-01-24 19:59:10 +00:00
Jenkins
6c0eb3d905 Merge "Add get_elements; preserve backwards compat for expand_dependencies" into feature/v2 2017-01-20 10:25:54 +00:00
Jenkins
1b7c415b4a Merge "Activate virtualenv in disk-image-create" into feature/v2 2016-12-20 03:41:19 +00:00
Gregory Haynes
5da7574aee Add get_elements; preserve backwards compat for expand_dependencies
expand_dependencies() was a public interface so we should try and
preserve backwards compat. However, since the interface is really
broken, add a new exported function "get_elements" that instack can
switch to.  This returns the canonical list of elements without
duplicates, and gives the path to each element too.

This highlighted that the unit tests were really a bit wrong.  They're
testing inner functions when we have an "API" in the get_elements()
function.  Convert all unit-tests to use this function instead.  Since
this is a library call, convert the sys.exit() calls to raised
exceptions.

Refactor the variable output into a separate function so we can do a
sanity check on it.

The added flake8 ignores are for the "over-indented for ... indent"
which happens a lot with these new longer lines.  Most other projects
ignore them.

This is an alternative proposal to
I15609389c18adf3017220fc94552514d195b323a

Change-Id: If97bcd21e45de1b5ed91023fdc441a4617051a6b
2016-12-20 08:18:59 +11:00
Ian Wienand
5adfc9365a Fix --version display
We do not have the concept of "not installed" in v2, so remove the
obsolete code looking for a now non-existent variable.

Also, log the version at startup.  This can help when
debugging from logs

Change-Id: I964c4cf207c10666afc5bc7ab9f2bfb9b1897c1e
2016-12-01 04:37:04 +00:00
Ian Wienand
04208e7c79 Activate virtualenv in disk-image-create
Because we're still fundamentally a python program calling a
shell-script, there's some oddities like not having the virtualenv
bin/ in the $PATH if we call disk-image-create directly.

We can detect this, however, and activate the virtualenv before we
fork the disk-image-create shell script so everything "just works".

See also nodepool change I0537cbf167bb18edf26f84ac269cbd9c8a1ea6a2

Change-Id: Ibfea6cf6a6fd0c7f1e468d501c61ae0b58992042
2016-11-30 06:11:09 +11:00
Ian Wienand
448a2602fe Merge remote-tracking branch 'origin/master' into merge-branch
Change-Id: I05cc51c699008018d557ed3874d71af26fd60240
2016-11-29 07:43:46 +11:00
Ian Wienand
7d5afecfd9 Merge remote-tracking branch 'origin/master' into merge-branch
Change-Id: Ibab1bb95521292ae818bd91f7073c3749a2cc0cb
2016-11-18 13:53:56 +11:00
Jenkins
cc3a28755d Merge "Move dib-run-parts into diskimage-builder" into feature/v2 2016-11-15 00:43:49 +00:00
Ian Wienand
3c6e589226 Remove execute perm from disk-image-create
Remove the x bit from lib/disk-image-create; because it's called
directly by the entry-point, it doesn't need to be exectuable.
This should also be clearer that you're not supposed to run it
by hand.

Remove some boilerplate from old file

Change-Id: Ibb6cdae613e6c9cf21dd6aecc8e1f739bc3a2643
2016-11-04 17:11:27 +11:00
Ian Wienand
d65678678e Move dib-run-parts into diskimage-builder
Move dib-run-parts from dib-utils into diskimage-builder directly.

For calling outside the chroot, we provide a standard entry-point
script.  However, as noted in the warning comment, the underlying
script is still copied directly into the chroot by the dib-run-parts
element.  I believe this to be the KISS approach.

This removes the dependency on dib-utils.  We have discussed this
previously and nobody seemed to think retiring dib-utils was going to
be an issue.

This also updates the documentation to not mention dib-utils, or using
disk-image-create via $PATH setup, but rather gives instructions on
installing from pip with a virtualenv.

Change-Id: Ic1e22ba498d2c368da7d72e2e2b70ff34324feb8
2016-11-04 17:07:37 +11:00
Ian Wienand
7446c32197 Move diskimage-image-create to an entry point
It has always been a weird thing that dib is a python package, but
is totally driven by the disk-image-create script.  It creates this
strange division that is hard to explain.

This moves disk-image-create to a regular python entry-point

Currently, this simply exec()s the original disk-image-create script.

However, we now have a (private) interface between disk-image-create
written in python and the driver shell script.  Here's some things we
could do, for example:

* Argument parsing is generally nicer in Python, and then end result
  is mostly just setting environment variables to flag different things
  in the shell script.  I could see us moving the argument-parsing into
  diskimage_builder.disk_image_create:main() and just setting things in
  os.environ before the exec()).

* I7092e1845942f249175933d67ab121188f3511fd sets IMAGE_ELEMENT_YAML in
  disk-image-create by calling-back to element-info.  We can just call
  element_dependencies.find_all_elements() in here an export is to
  os.environ before disk-image-create starts.

* remove need for ramdisk-image-create symlink by just exporting
  IS_RAMDISK based on sys.argv[1] value

* you could even unit test some of this :)

Change-Id: I69ca3d26fede0506a6353c077c69f735c8d84d28
2016-11-02 05:12:11 +00:00
Ian Wienand
97c01e48ed Move elements & lib relative to diskimage_builder package
Currently we have all our elements and library files in a top-level
directory and install them into
<root>/share/diskimage-builder/[elements|lib] (where root is either /
or the root of a virtualenv).

The problem with this is that editable/development installs (pip -e)
do *not* install data_files.  Thus we have no canonical location to
look for elements -- leading to the various odd things we do such as a
whole bunch of guessing at the top of disk-image-create and having a
special test-loader in tests/test_elements.py so we can run python
unit tests on those elements that have it.

data_files is really the wrong thing to use for what are essentially
assets of the program.  data_files install works well for things like
config-files, init.d files or dropping documentation files.

By moving the elements under the diskimage_builder package, we always
know where they are relative to where we import from.  In fact,
pkg_resources has an api for this which we wrap in the new
diskimage_builder/paths.py helper [1].

We use this helper to find the correct path in the couple of places we
need to find the base-elements dir, and for the paths to import the
library shell functions.

Elements such as svc-map and pkg-map include python unit-tests, which
we do not need tests/test_elements.py to special-case load any more.
They just get found automatically by the normal subunit loader.

I have a follow-on change (I69ca3d26fede0506a6353c077c69f735c8d84d28)
to move disk-image-create to a regular python entry-point.

Unfortunately, this has to move to work with setuptools.  You'd think
a symlink under diskimage_builder/[elements|lib] would work, but it
doesn't.

[1] this API handles stuff like getting files out of .zip archive
modules, which we don't do.  Essentially for us it's returning
__file__.

Change-Id: I5e3e3c97f385b1a4ff2031a161a55b231895df5b
2016-11-01 17:27:41 -07:00
Gregory Haynes
eff793fc2f Remove deprecated expand-dependencies arg
This argument has been deprecated in element-dependencies for some time.
Removing in v2.

Change-Id: I9e40ec7bef7afbebba3958043db46975ea7b24ff
2016-10-06 16:14:32 +00:00
Ian Wienand
91b431ce78 Move element-info to a standard entry-point
Move element-info from a wrapper script to a standard entry-point
console_script.

Update the documentation to explain how to run it for development.  I
don't think we should support the idea that you can check-out the code
and run ./bin/disk-image-create -- it has dependencies (dib-utils,
etc) and needs to be run from a virtualenv (this is what CI in the
gate does).  A follow-up can clean-up some of the path munging stuff
we have for this in disk-image-create.

Change-Id: Ic0c03995667f320a27ac30441279f3e6abb6bca8
2016-09-08 15:29:56 +10:00
Andreas Florath
3d48a528c1 Refactor: block-device handling (local loop)
Block device handling can be somewhat complex - especially
when taking things like md, lvm or encryption into account.

This patch factors out the creation and deletion of the local
loop image device handling into a python library.

The main propose of this patch is to implement the needed
infrastructure.  Based on this, more advanced functions can be added.
Example: (advanced) partitioning, LVM, handling different boot
scenarios (BIOS, UEFI, ...), possibility of handling multiple images
(local loop image, iSCSI, physical hard disk, ...), handling of
different filesystems for different partitions / LVs.

Change-Id: Ib626b36a00f8a5dc3dbde8df3e2619a2438eaaf1
Signed-off-by: Andreas Florath <andreas@florath.net>
2016-09-08 04:31:01 +00:00
Ian Wienand
37a53354ec Add IMAGE_ELEMENT_YAML and get_image_element_array
These new variables are a list of elements chosen for the build along
with their full paths.  For Python elements, IMAGE_ELEMENT_YAML is a
YAML formatted list that can be easily parsed.  For bash elements,
"get_image_element_array" will produce an associative-array of the
same (working around lack of array export in Bash).

This list is intended for consumption of elements who need to copy
files from other elements, such as pkg-map and svc-map.  As discussed
in I2a29861c67de2d25c595cb35d850e92807d26ac6, this list has already
been pruned and had overrides processed, so it is safe to simply walk
over this list with no further processing.

Since we're presenting the element list in a couple of different ways,
we combine it all into the element-info script.  It will output an
eval-able string that declares the appropriate variables.

I've added some inline documentation so they still appear in grep.
The documentation is updated with examples, and moved to a more
appropriate location as a sub-section of the element sytle guide.

To test this out, use the associative-array in generate_hooks, where
we can now find the element's directory without searching.

Change-Id: Ibbd07d082ec827441def2d3f6240df3efdc6eae3
2016-09-08 11:08:07 +10:00
Ian Wienand
274be6de55 Making element overriding explicit
This is a re-factor of element_dependencies to achieve two things --
centralising override policy and storing path names.

Firstly we want to make the override policy for elements completely
explicit.  Currently, elements that wish to copy parts of other
elements walk ELEMENTS_PATH themselves and look for elements in
IMAGE_ELEMENT.  How they handle duplicate elements can differ, leading
to inconsistent behaviour.

We introduce logic in element-info to find elements in each of the
directories in ELEMENT_PATHS in *reverse* order -- that is to say,
earlier entries in the paths will overwrite later ones.

For example

 ELEMENT_PATHS=foo:bar:baz

will mean that "foo/element" will override "baz/element", since "foo"
is first.  This should be sane to anyone familiar with $PATH.
Documentation is clarified around this point and a test-case is added.

The second thing is that we want to keep the complete path of the
elements we have chosen.  We want the aforementioned elements that
walk the element list to use these canonical paths to pickup files;
this way they don't need to make local decisions about element
overrides, but can simply iterate a list and copy/merge files if they
exist.

A follow-on change (I7092e1845942f249175933d67ab121188f3511fd) will
expose this data in a separate variable that can be parsed by elements
(a further follow-on I0a64b45e9f2cfa28e84b2859d76b065a6c4590f0
modifies the elements to use this information).  Thus this does not
change the status-quo -- elements that are walking ELEMENTS_PATH
themselves and can/will continue doing that.

Change-Id: I2a29861c67de2d25c595cb35d850e92807d26ac6
2016-09-08 10:58:19 +10:00
Ian Wienand
452f7b8d5a Clear up "already provided" message
I got quite confused what this was trying to tell me at first.  It's
saying that you requested an element but another element already
provides that element, so we don't know which one to choose.

To help clarify the situation to the user, keep track of what is
providing elements so we can describe where the conflict came from.

Change-Id: Ie7471ac900a8cbee5684c928badd1b8ce6d3e3cf
2016-07-01 11:53:07 +10:00