Commit Graph

373 Commits

Author SHA1 Message Date
Zuul
906a3f4a57 Merge "Use -t devpts for /dev/pts mounts" 2017-11-02 14:20:05 +00:00
Zuul
1b0631da84 Merge "Update proliant-tools to support Gen10 Proliant servers" 2017-11-01 07:22:03 +00:00
Ian Wienand
b25d0337b8 Move to a common lock-file directory
In a couple of places we use flock for critical sections, but we leave
lockfiles around in various locations which can be confusing.

Introduce DIB_LOCKFILES global (under ~/.cache/dib/lockfiles) and
write lockfiles in there.

Fix up removal of the lockfile in the yum path; we just want to make
sure we cleanup the .rpmmacros file, but we don't need to remove the
lockfile as well.

Co-Authored-By: Andreas Florath <andreas@florath.net>

Change-Id: Ie810b2836be521325afe923708d046112e1e1e20
2017-10-26 16:27:59 +11:00
Zuul
8f025691ba Merge "Dont install python-virtualenv for py3k in deb" 2017-10-24 06:33:45 +00:00
Zuul
c5f713b1ec Merge "Change to install a package in 'proliant-tools'" 2017-10-24 05:48:25 +00:00
Yolanda Robla
ba11376328 Create rescue user on ironic agent
Create a new service, that will be launched after ironic
agent has been exited. This will launch an script that will
take the rescue password, and create the rescue user with
that credentials.

Depends-On: I7898ff22800dedba73d7fbfb3801378867abe183
Change-Id: Ic3a241e2789a122d3d966e7e2148306fd0cf6aed
Partial-Bug: 1526449
2017-10-23 12:50:32 +00:00
Andreas Florath
cebfcf85f9 Use -t devpts for /dev/pts mounts
Currently a bind is used when mounting /dev/pts in chroot.
This leads to problems - especially when running DIB in parallel:
It was observed that the /dev/pts mount vanishes from the host
system.

This patch uses '-t devpts' - as it is done for /sys and /proc -
for handling /dev/pts.

Change-Id: Id7775ae6fca6502af800e7b73a00862ef320206b
Signed-off-by: Andreas Florath <andreas@florath.net>
2017-10-23 07:13:02 +00:00
Gregory Haynes
00d7c619e9 Dont install python-virtualenv for py3k in deb
On ubuntu we detect that in python3 we need to install
python3-virtualenv, but append this to the packages to install rather
than replace python-virtualenv which results in both being installed
(and therefore grabbing python2).

Change-Id: I422490ebe9a9c655552685bc2ff342d288335a9c
Closes-Bug: #1724656
2017-10-18 23:11:55 +00:00
Zuul
a0045985f2 Merge "Remove dd from LVM element" 2017-10-17 08:15:03 +00:00
Zhangfei Gao
6cc155fd66 Add debian minimal requirement for arm64
Debian system building fails, fixed by adding arm64 package.

DIB_RELEASE=jessie disk-image-create debian ironic-agent \
    grub2 devuser -a arm64 -o deploy-jessie
dib-run-parts Running /tmp/dib_build.v5FEtaKx/hooks/cleanup.d/99-extract-kernel-and-ramdisk
ls: cannot access /tmp/dib_build.v5FEtaKx/mnt/boot/vmlinu*: No such file or directory

Change-Id: I610d767785df49fed954f12854be5ae78ff9baa6
Signed-off-by: Zhangfei Gao <zhangfei.gao@linaro.org>
2017-10-16 13:39:50 +08:00
Zhangfei Gao
a8635d3a3b Fix grub2 dependency on arm64
Unable to locate package reported when
DIB_RELEASE=jessie disk-image-create debian ironic-agent \
grub2 devuser -a arm64 -o deploy-jessie

E: Unable to locate package grub-pc-bin
E: Unable to locate package shim-signed
E: Unable to locate package grub-efi-amd64-signed

Fix the issue via adding arch dependency and arm64 packages

Change-Id: I40650a887b575a9c2b00a8c5036c35354d548673
Signed-off-by: Zhangfei Gao <zhangfei.gao@linaro.org>
2017-10-16 13:39:34 +08:00
Jenkins
7a76270227 Merge "Add timestamp output filter" 2017-10-09 15:56:31 +00:00
Jenkins
fd8f92e542 Merge "Update Gentoo element for element changes" 2017-10-08 23:58:20 +00:00
Andreas Florath
bb6cf52d85 Remove dd from LVM element
This patch removes the unneeded dd calls in the lvm block device
plugin.

After removing the underlying block device, there is the need to call
'pvscan --cache'.  This is done by a dedicated LVM cleanup node which
is cleaned up after the the underlying block device.

Change-Id: Id8eaede77fbdc107d2ba1035cd6b8eb5c10160c3
Signed-off-by: Andreas Florath <andreas@florath.net>
2017-10-08 17:21:21 +00:00
Matthew Thode
e29f031bec
Update Gentoo element for element changes
There have been a few changes over the past few months, here we make the
following changes.

* change from backtrack=99 to complete-graph as a more correct flag
* make python version selection more in line with what gentoo supports
* set up python before stuff gets pip installed
* ensure we have the proper pip so we can install pip packages as root
* ensure we have the proper use flags for the disk formatting changes
* set DIB_RELEASE like other distros
* fix openssh-server element for gentoo

Change-Id: I17202de3016616ce34c8cbead7d0fb047a64e96b
2017-10-08 12:02:46 -05:00
Anshul Jain
afb7084a4d Update proliant-tools to support Gen10 Proliant servers
This commits make update to ssacli version to point to latest
ssacli release that has support for HPE P/E-Class SR Gen10 controllers.

Change-Id: Ia9a0eaec78d601f56b4036e57601554b87f21acc
Closes-Bug: 1721185
2017-10-04 07:02:07 +00:00
Andreas Florath
fa6c731132 Move fstrim to block device layer
The call to fstrim in disk-image-create is currently useless, because
at the time this is called, the file systems were already umounted by
the block device layer.

The current implementation of the block-device mount plugin does not
call fstrim at all: resulting in larger image sizes.

This patch removes the useless fstrim call from the disk-image-create
script and moves this into the block-device mount.py.

The resulting image might be much smaller.  Example: Ubuntu Xenial
with some elements; once with and once without this patch:

-rw-r--r-- 1 dib dib 475661824 Sep 16 06:43 ubuntu-xenial-without-fstrim.qcow2
-rw-r--r-- 1 dib dib 364249088 Sep 16 09:30 ubuntu-xenial-with-fstrim.qcow2

Change-Id: I4e21ae50c5e6e26dc9f50f004ed6413132c81047
Signed-off-by: Andreas Florath <andreas@florath.net>
2017-09-28 17:48:59 +10:00
Jenkins
dc215674f8 Merge "Actually sort mount-point list" 2017-09-27 07:20:29 +00:00
Ian Wienand
06b5ce4573 Revert "Support networkx 2.0"
This reverts commit a47ff0dd4a.

Since this merged, a global-requirements pin to keep networkx <2.0 has
also merged.  The plan is:

1. revert our 2.0 support and
1a. take the <2.0 pin from global requirements
2. figure out how to use constraints properly in our testing
3. restore this, with a depends-on for a 2.0 bump in requirements
   (which will self-test, see 3.)
4. when other projects are ready for a global 2.0 bump, merge
   in a controlled fashion

This reverts the 2.0 support, and adds the pin for networkx <2.0

Change-Id: I18f6a1115da779581245e3dd423fd90516974a33
2017-09-22 08:02:44 +10:00
Ian Wienand
a47ff0dd4a Support networkx 2.0
Networkx 2.0 released recently.  The main difference for us is that
"node" is no longer a dictionary and should be accessed via "nodes",
and the topological_sort returns an interator

Closes-Bug: 1712693
Change-Id: I78e89f2261b8b8d28c68b517c1e61691ab40016c
2017-09-21 09:43:01 +00:00
Ian Wienand
df00e9adcb Add initramfs-tools for ubuntu-minimal
A small update was made to 4.4.0-96.119 that dropped the
initramfs-tools dependency from the kernel [1].  This had the
unfortunate affect of removing the initramfs from ubuntu-minimal and
making it unbootable, since we specify the root device via LABEL=.
Add the package explicitly alongside the kernel.

Also, small fix to pass unit tests

[1] https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/1700972

Change-Id: I57a0f08cd5e082ecdf8dba0ab34fb3062c50836d
2017-09-21 10:42:11 +02:00
Ian Wienand
ed3c5d9711 Actually sort mount-point list
We intended to do an in-place sort of the mount-point list, but
sorted() returns a new list that wasn't captured.  Move to the .sort()
function.

It seems the existing unit-test missed this.  Add a new test taken
from the bug which does exhibit a sorting issue.  Also added a
unit-test of just the comparitor for sanity.

Closes-Bug: 1699437
Change-Id: I8101e4a1804a4af7dbda20d48bf362c3f4ad2742
2017-09-19 11:30:36 +10:00
Aparna
7bc2b23290 Change to install a package in 'proliant-tools'
This commit adds change in 'proliant-tools' element to
install a package 'unzip' which is required to perform
SUM based firmware update for HPE Proliant servers.

Change-Id: Ib8f6d18402439edd93d100cc7a4fb2094c863715
2017-09-18 08:04:34 +00:00
Jenkins
22e03f9820 Merge "Add missing package dependency for yaml" 2017-09-15 13:52:35 +00:00
Jenkins
254875dbde Merge "Create /etc/machine-id for fedora" 2017-09-15 04:05:51 +00:00
Jenkins
6e266399fb Merge "Use latest Fedora .qcow URL" 2017-09-15 02:36:24 +00:00
Ian Wienand
768c5e188c Create /etc/machine-id for fedora
As described in the comment, we need to create the /etc/machine-id for
the image-based build when systemd isn't updated (as is usually the
case for a new distro)

Work on clearing this out continues, but this brings it to parity with
fedora-minimal.

Change-Id: Icbbbabb4114d4d95909648d8e39a6bae6d2a7b7b
Depends-On: I761e425f8a658669d9b8a70ce4260cec263ea51a
2017-09-15 11:54:01 +10:00
Ian Wienand
7774260b76 Use latest Fedora .qcow URL
The URL we are using seems to have disappeared.  Update this to
download.fedoraproject.org.  The new URL requires a "subrelease" now,
add it, along with a note on where it comes from.

Change-Id: I761e425f8a658669d9b8a70ce4260cec263ea51a
2017-09-15 11:06:22 +10:00
Yolanda Robla
da5c926fe9 Add missing package dependency for yaml
This element was assuming that yaml was included as package,
but there are systems not including it. So properly add yaml
as a dependency.

Change-Id: I72da2776674a3963657052b9a9715abcb4fab1e2
Partially-Fixes-Bug: #1715686
2017-09-13 14:16:41 +02:00
Yolanda Robla
3ff8d1e10b Move the ordering of the dracut regenerate command
When using combined with rhel7 image, the unregister of repos
has already happened, because it is executed under 60- ordering.
As dracut-regenerate may need to install extra packages for it,
it causes this step to fail, because it cannot find repos where
to pull the packages from.

Change-Id: I35e37df7990ad76a5004cb90fdd863ec743a5483
2017-09-06 12:52:05 +02:00
Ben Nemec
72d0d22cdf Remove nested quotes from TAROPTS
Per the bug report, these seem to be causing issues with maintaining
file capabilities.  They aren't necessary so let's just remove them.

Change-Id: I06c90fdc85655986142b936cadbe04d75dd27427
Closes-Bug: 1714604
2017-09-01 17:29:15 -05:00
Jenkins
39d84d2059 Merge "Use [[ for =~ matches" 2017-08-29 05:59:42 +00:00
Ian Wienand
c448864901 Use [[ for =~ matches
Avoid incorrect use of [ with =~ matching

I guess this doesn't trip "-e" because it's in an if-conditional.  I'm
looking at making bashate detect this; maybe we can run bashate over
things we know are scripts

Change-Id: Ia3fe2b978fae5bdaadbb1789058180d3ad950d00
2017-08-28 17:01:03 +10:00
Julia Kreger
6d64a2aee6 Fix cylical systemd config for dhcp-all-interfaces
In Ubuntu/Debian, the default dependencies cannot be relied
upon as we enter into a cyclical dependency relationship which
prevents the unit from starting.

Added the required configuration to the systemd unit file.

This issue has also been observed in glean[0], which has a nearly
identical unit file for interface start-up.

[0]: https://review.openstack.org/#/c/485748
Closes-Bug: #1708685

Change-Id: I23ac9510d1a21c7073bd33f76ba66fa04a8be035
2017-08-25 15:51:23 +10:00
Jenkins
fa3797848b Merge "LVM support for dib-block-device" 2017-08-24 09:18:26 +00:00
Yolanda Robla
c2dc3dc78e LVM support for dib-block-device
This provides a basic LVM support to dib-block-device.

Co-Authored-By: Ian Wienand <iwienand@redhat.com>

Change-Id: Ibd624d9f95ee68b20a15891f639ddd5b3188cdf9
2017-08-24 16:22:56 +10:00
Jenkins
02601a1295 Merge "Increase timeout for removal" 2017-08-16 09:45:01 +00:00
Jenkins
ea23aa13a5 Merge "Add netbase to ensure /etc/protocols is placed for debian" 2017-08-15 00:37:27 +00:00
Yolanda Robla
81f495ad00 Increase timeout for removal
Under certain environments, this timeout was causing failures
because it was too short. Increasing to 10, to give time to
perform the specified tasks.

Change-Id: I01dd3553f38e1137b2fcb04b4ee12202be3ad1a8
2017-08-11 16:29:26 +02:00
Julia Kreger
f19c45eb29 Add netbase to ensure /etc/protocols is placed for debian
Many programs rely upon /etc/protocols to be present
however the default debian image that is generated lacks
/etc/protocols. This is observable when building an image
for use with ironic via the ironic-agent element, since
the IPA agent fails to start as python needs /etc/protocols
to open a socket connection.

Added to debian-minimal as it is inherited into the debian
element.

Change-Id: Icc81635870961943707cf6b3f61a9ddbd51cb8fd
Closes-Bug: #1708531
2017-08-11 14:17:30 +00:00
Ian Wienand
a88a768e98 Clear up debian element documentation
There is some confusion in the readme's over what is happening.  The
original change (Iaf46c8e61bf1cac9a096cbfd75d6d6a9111b701e) split out
debian-minimal and made debian "... simply be a collection of the
extra things we do to make it look like a cloud-init based cloud
image"

Make this clearer in the documentation

Change-Id: Ibe6fad9c67b70a5e31e43e06419968135174fef3
2017-08-09 13:15:38 +10:00
Jenkins
e04cf78fa5 Merge "Bump fedora/fedora-minimal DIB_RELEASE 26" 2017-08-08 00:26:45 +00:00
Dave Hill
6c2b1465cc Clear /etc/machine-id to avoid duplicate machine-ids
Deploying many nodes with the generated image shouldn't have the same
/etc/machine-id so clearing it and letting systemd generate a new
id upon first boot seems to be the best way to achieve this.

Change-Id: I73d0577d31464521b3989312fd9d982a1312a268
Closes-bug: 1707526
Closes-bug: 1672461
2017-08-06 13:56:58 -04:00
Jenkins
ced9b51f6e Merge "Allow users to specify partition type in the MBR PTE" 2017-08-04 05:19:02 +00:00
Paul Belanger
7cbbee7ea3 Bump fedora/fedora-minimal DIB_RELEASE 26
Fedora 26 is now the latest release:

  https://fedoraproject.org/wiki/Releases/26/Schedule

We are building and using these in infra now

Change-Id: I012c2d28255be274e88abc2751d968bafaf76fbb
Depends-On: Ieba5f69020a13681074f72cfca2955071801b63a
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2017-08-04 14:18:05 +10:00
Ian Wienand
818b75af41 Move selinux packages to redhat-common
Change I008f8bbc9c8414ce948c601e3907e27764e15a52 has shown that we
build redhat images without the "semange" tool available, which comes
from the policycoreutils-python package (see also
I3f9e2c322d042a5dddba33451c0fc21a4d32a88a).

I403e7806ae10d5dd96d0727832f4da20e34b94c7 added some of the selinux
libraries to yum-minimal for ansible support, but not to others.

Given both these changes, it seems that selinux[-targeted],
libselinux[-python] and policycoreutils[-python] can reasonably
considered part of all base images.  Move the selinux related packages
into redhat-common.

This also adds it explicitly to install_test_deps.sh.  It was actually
being dragged in by the docker install, but is a required component
for building (should be in bindep, but not there with that yet).

Change-Id: Idd4ae71ee6deee84604823b6b5dc4a845f316e01
Related-Bug: #1707788
2017-08-01 11:08:54 +10:00
Amrith Kumar
52faa0e1d9 Allow users to specify partition type in the MBR PTE
The MBR Partition Table Entry (PTE) allows one to specify many
possible partition types and one of the benefits of this is being able
to specify the CHS variant or the LBA variant.

By default, LBA only creates partitions of type 0x83 (of course,
that's only because the documentation doesn't tell you how to make it
do anything else).

I will take up Ian's suggestion in patch set 2 for a more rigorous
test in an independent patch set.

Change-Id: If3068535980eac2e58d4025444c65147a8c7fedc
Closes-Bug:#1703352
2017-07-29 06:34:25 -04:00
Jenkins
308783d382 Merge "Switch openSUSE to 42.3 by default" 2017-07-28 05:14:23 +00:00
Alfredo Moralejo
b1961e14ea Use SELinuxfs to check selinux status
Currently, the cleanup script is using existence of
semanage binary to check if selinux is enabled. However
this is misleading and can lead to problems when selinux
is disabled in a system where the binary exist.

This patch changes the detection logic to use /sys/fs/selinux
directory which is a in-memory filesystem created only when
selinux is really enabled.

Change-Id: I008f8bbc9c8414ce948c601e3907e27764e15a52
Related-Bug: 1706386
2017-07-26 18:57:25 +02:00
Dirk Mueller
1c4c4fd734 Switch openSUSE to 42.3 by default
This is the latest stable release, so we should default to it.

Change-Id: I05643787002d339ccbf7a718847fe4ed6f39eacc
2017-07-26 08:56:02 +02:00
Jenkins
609bcee27b Merge "zypper: Clean caches and don't cache packages locally" 2017-07-26 02:25:40 +00:00
Markos Chandras
81e72d4045 elements: zypper-minimal: Install tar package
tar is an essential package but nothing pulls it explicitly. This causes
some issues in the openSUSE CI jobs like the following one

"Failed to execute tar: No such file or directory", "Failed to write
file: Broken pipe", "Failed to retrieve image file. (Wrong URL?)",
"Exiting."], "stdout": "", "stdout_lines": []}

Just like 'sed', add 'tar' to the list of packages for the openSUSE
minimal builds.

Change-Id: Ia36e3d9fd6b78862a6831ba80b43d4614a349ca0
2017-07-25 16:27:25 +01:00
Jenkins
a6da39acb8 Merge "Move setfiles to outside chroot with runcon" 2017-07-24 02:04:21 +00:00
Ian Wienand
5089e4e541 Move setfiles to outside chroot with runcon
As described in the comments inline, on a selinux enabled kernel (such
as a centos build host) you need to have permissions to change the
contexts to those the kernel doesn't understand -- such as when you're
building a fedora image.

For some reason, setfiles has an arbitrary limit of 10 errors before
it stops.  I believe we previously had 9 errors (this mean 9
mis-labeled files, which were just waiting to cause problems).
Something changed with F26 setfiles and it started erroring
immediately, which lead to investigation.  Infra builds, on
non-selinux Ubuntu kernel's, would not have hit this issue.

This means we need to move this to run with a manual chroot into the
image under restorecon.

I'm really not sure why ironic-agent removes all the selinux tools
from the image, it seems like an over-optimisation (it's been like
that since Id6333ca5d99716ccad75ea1964896acf371fa72a).  Keep them so
we can run the relabel.

Change-Id: I4f5b591817ffcd776cbee0a0f9ca9f48de72aa6b
2017-07-24 10:14:07 +10:00
Dirk Mueller
bfeb9d9e99 zypper: Clean caches and don't cache packages locally
For builds inside the infra, we don't want to pack the cache
inside the image (as it might be different at the time the image
runs). In an opensuse-minimal image this saves about 10MB of image
size.

Change-Id: I5ecabd46f0a662798bda3e4468395ad8308d0055
2017-07-23 17:24:24 +02:00
Jenkins
55971717b6 Merge "elements: openstack-ci-mirrors: Use openSUSE mirrors for gating jobs" 2017-07-22 05:22:34 +00:00
Jenkins
e029af993b Merge "Remove DIB_[DISTRO]_DISTRIBUTION_MIRROR" 2017-07-22 05:22:04 +00:00
Jenkins
7a70299668 Merge "Enable console during kernel boot on Power" 2017-07-20 03:55:19 +00:00
Jenkins
d66dbc679c Merge "The correct option for label name in fat and vfat is '-n'" 2017-07-20 03:54:23 +00:00
Ian Wienand
7ffe6856d6
Add -m flag to setfiles for Fedora 26
As described in the comment and associated bugzilla, the behaviour of
setfiles has changed in Fedora 26 to require "-m" situations where
labeled file-systems are mounted below non-labeled file-systems.  Our
loopback/chroot system appears to trigger this nicely, leading to a
setfiles call that does nothing without this.

Change-Id: I276c6f6a4fb44f4bea5004f6b4214f94757728ae
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2017-07-19 08:01:19 -04:00
Ian Wienand
6ffde2e596 yum-minimal: pre-install coreutils
As described in the referenced bug, the dependency solver in yum
doesn't handle weak dependencies well and in some cases, such as
Fedora 26, can end up choosing coreutils-single (the busybox-esque
single binary) instead of actual coreutils, which then causes problems
with conflicting packages later.

Change-Id: I2907bf3b74c146986b483d52cc6ac437036330b4
2017-07-18 14:51:18 +10:00
Ian Wienand
b8ad9c2e37 Force install during pip-and-virtualenv
On a system where the packaged pip/virtualenv is up-to-date with
upstream (such as Fedora 26 ... for now), we don't reinstall, which
then violates a bunch of assumptions later on.  Force install.

Change-Id: I6ebcda0351997fa7e32f0e6e77a98b2c33764e3f
2017-07-18 14:50:09 +10:00
Ian Wienand
da90ef4743 Fix latest-limit command line
It turns out dnf argparse can't handle negative numbers without "=".
It's actually documented in the man page

    --latest-limit <number> ...  If <number> is negative skip <number>
      of latest packages. If a negative number is used use syntax
      --latest-limit=<number>

But who reads that :)  This started failing with Fedora 26

Change-Id: I884af94c07fa11b010f69863047a04711b14f21e
2017-07-18 13:17:10 +10:00
Jenkins
016606c81d Merge "opensuse-minimal: install glibc-locale" 2017-07-18 00:40:48 +00:00
Dirk Mueller
59721d3c74 opensuse-minimal: install glibc-locale
We expect LC_ALL for non-C locales to be working inside
images, so always install glibc-locale for openSUSE.

Change-Id: I8fe92773e377539070d9d9fe2960a6202bb80a18
2017-07-17 22:50:25 +02:00
Markos Chandras
6be09152c2 elements: openstack-ci-mirrors: Use openSUSE mirrors for gating jobs
In preparation for promoting the openSUSE jobs to voting ones we should
use the OpenStack mirrors. As such, the opensuse elements are modified
to make use of the DIB_DISTRIBUTION_MIRROR variable which is normally
exported by the openstack-ci-mirrors element.

Change-Id: Ie588c1c1eec13190cfb2ec718ba51f8c9878283f
2017-07-17 10:54:03 +01:00
Jenkins
c18a3ff029 Merge "Replace architecture-emulation-binaries with qemu-debootstrap" 2017-07-17 05:36:09 +00:00
Ian Wienand
3457d2f8e8 Remove DIB_[DISTRO]_DISTRIBUTION_MIRROR
We added the DIB_distro_DISTRIBUTION_MIRROR arguments with
I92964b17ec3e47cf97e3a3091f054b2a205ac768 as a way that we could
source a list of mirrors and then have the distro elements choose
which one applied to them.

However, this hasn't worked out to be so useful.  The
openstack-ci-mirrors element is working as a mirror setup script -- it
translates the openstack CI mirror list variables into the generic
"DIB_DISTRIBUTION_MIRROR" as appropriate for each distro's build.
Also, it turns out there's other things that need to be done, such as
turning off gpg checking, which mean the idea of "just export
variables" hasn't turned out as valid ... you need actual code
involved to get it right.

AFAICT we never actually documented these, and they do not seem to be
in use.  They have caused considerable confusion when dealing with new
platforms as we try to keep consistency.  Remove them.

[1] http://codesearch.openstack.org/?q=DIB_.*_DISTRIBUTION_MIRROR&i=nope&files=&repos=

Change-Id: Ifc4ab700631ffdfbe790068558f670f9a11dde5e
2017-07-17 14:47:31 +10:00
Jenkins
787e76b916 Merge "Remove additional Bumblebee repository for opensuse element" 2017-07-17 00:50:46 +00:00
Jenkins
64a8c6e1dc Merge "zypper-minimal: No point in preserving the environment here" 2017-07-17 00:26:11 +00:00
Amrith Kumar
59f416ae20 The correct option for label name in fat and vfat is '-n'
The code in mkfs correctly extends the command line with a '-n' for
vfat but does not currently do it for fat. This means that mkfs for
fat ends up with a '-L' which is what you'd do for everything like
ext[234].

The change just treats fat like vfat in the one place where this check
is required.

Change-Id: If65dfd949acdadff33a564640fb42ea73026a786
Closes-Bug: #1703063
2017-07-15 22:48:52 -04:00
Dirk Mueller
02d33f2ca7 zypper-minimal: No point in preserving the environment here
Change-Id: I46442e841d1f718b683bca4d2a348f0013306907
2017-07-13 22:50:47 +02:00
Dirk Mueller
05ba445ade Remove additional Bumblebee repository for opensuse element
The purpose of the openSUSE element is to build openSUSE distribution
based images, so an additional community repo shouldn't be pulled into
the image. In addition the dkms dependency is blacklisted for SUSE
in the dkms element anyway, so this should be a noop.

Change-Id: I0aa06d9f4f110546032f910e3361840693d02de7
2017-07-11 23:24:05 +02:00
Jenkins
0327d775f1 Merge "pip-and-virtualenv: Install python3 on openSUSE" 2017-07-11 08:11:16 +00:00
Jenkins
997a6ea6cb Merge "Add symlink test for resolv.conf restore" 2017-07-11 08:07:23 +00:00
Rafael Folco
bfdf7dc0f6 Enable console during kernel boot on Power
On Power systems console should be added the kernel command line
in the following order: 'console=tty0 console=hvc0'.
The first one is the graphical console. The last one is the serial
console. The kernel enables all the consoles pointed through the
kernel command line. However, only the last one will receive
input/output during kernel boot. All the other consoles will be
enabled after the boot.

Change-Id: I0069f608e0ab104d3778954e033fb82ed5ea7693
2017-07-07 17:55:56 +00:00
Amrith Kumar
43e32116bd fix readme.rst to reflect correct environment variable
The readme.rst incorrectly refers to the environment variable
DIB_APT_KEYS which should be DIB_ADD_APT_KEYS. See [1] for usage in
code.

This is a minor correction to the readme only, no runnable code is
modified.

[1] http://git.openstack.org/cgit/openstack/diskimage-builder/tree/diskimage_builder/elements/dpkg/extra-data.d/01-copy-apt-keys#n23

Change-Id: I04129cef9f40ec75a206c126bfd40ee61e4e6a2b
2017-07-06 22:54:08 -04:00
Ian Wienand
5fa6e3e13c Add symlink test for resolv.conf restore
We replace the base resolv.conf with an "outside" copy so that
resolving works when we're in the chroot.

Installing resolvconf package modifies the in-chroot resolv.conf to a
symlink (to /var/run) which it wants maintained in the final image.
We have the existing "immutable" check for a created resolv.conf file,
but no eqivalent for a symlink.

This adds a check to see if the resolv.conf is a symlink and leave it
alone if it is, assuming it has been re-created in the chroot.

I have tested this with ubuntu-minimal+resolvconf with
dhcp-all-interfaces and the system seems to work with resolvconf
working correctly.

Change-Id: Idd5a26e9d55979bd951577d5b098ed4bfba91ad3
2017-07-06 13:48:27 +10:00
Jenkins
e8ad2a3799 Merge "elements: pip-and-virtualenv: Use common packages for openSUSE" 2017-07-04 11:20:35 +00:00
Markos Chandras
5fe35b0d7a pip-and-virtualenv: Install python3 on openSUSE
The python3 package actually contains some core modules (like the xml
one) which are not present in the python3-base on which is pulled by
the python3-devel package. As such, it's best to have it installed
similar to python-xml for python2.

Change-Id: I5cd5d1127ae62d6753c2ace44965179c5400bb9a
2017-07-04 08:40:34 +01:00
Jenkins
fad72745d2 Merge "Support for Cloud Images on ppc64le for rhel7 and centos7" 2017-07-04 01:13:24 +00:00
Jenkins
2ed643a734 Merge "Use the dib python to do cleanup" 2017-06-29 21:22:36 +00:00
Jenkins
6b45497ff6 Merge "Remove centos and rhel elements" 2017-06-29 21:16:57 +00:00
Jenkins
f0fb835db9 Merge "Avoid hanging endlessly on unreachable cache urls" 2017-06-29 08:03:25 +00:00
Chhavi Agarwal
6d69d7909d Support for Cloud Images on ppc64le for rhel7 and centos7
In order to support {CentOS,RHEL}7 for building cloud images we need to
handle the differences in grub packaging from Ubuntu.  We also need to
populate the defualt location for cloud images for CentOS builds.

Change-Id: Ie0d82ff21a42b08c4cb94b7a5635f80bfabf684e
2017-06-29 15:44:26 +10:00
Dirk Mueller
959226c55e Avoid hanging endlessly on unreachable cache urls
When a download redirector redirects to a broken mirror, timeout
quickly rather than waiting until the overall job is being timed out.

Change-Id: If7eb63d406aaf61f71aa9203cf708c474aa63fd0
2017-06-28 22:14:55 +02:00
Markos Chandras
c46b6da65f elements: pip-and-virtualenv: Use common packages for openSUSE
The 'packages' variable already contains the packages we need so
use it instead of duplicating the packages.

Change-Id: Id22e1862f9654e66252d03a0fed9839cf004d750
2017-06-28 17:59:25 +01:00
Ian Wienand
a00d02f6a1 Remove centos and rhel elements
Several people have popped up in IRC recently with failures in these
elements.  Without Python 2.7 available in the image they are
unsupported (OpenStack hasn't supported it for a long time).  Remove
these to avoid further confusion.

The centos/centos7 DISTRO split that has happened with centos-minimal
is unfortunate but I don't think it helps to rename centos7/rhel7 ATM.
To summarise; DISTRO=centos7 means image based build,
DISTRO=centos && DIB_RELEASE=7 means the minimal build.

In the future, I think it is important that the minimal builds and
image builds set the same DISTRO.  This reflects that "upper" layers
shouldn't care about the exact building of the lower layers.  I see
CentOS 8 going one of two ways

1) the changes are so significant, we start separate centos8 /
centos8-minimal elements.  They both set DISTRO=centos8 (and
DIB_RELEASE to point-release maybe?).  This means we have to update
all "if DISTRO == centos || DISTRO == centos7" branches to also check
for "centos8".  Evenually (!)  "centos" goes away for versioned DISTRO
only

2) we restore centos element with DISTRO=centos and DIB_RELEASE=8, and
centos-minimal remains the same.  This means we have to audit all "if
DISTRO == centos" calls to make sure they're appropriate for version 8
(stick a "&& DIB_RELEASE=7" on them all basically).

I'm not sure we can fully decide until we start to see excatly how the
distro switching/matching bits look, but (2) is consistent with Ubuntu
and probably the preferred solution.

Some "rhel" parts have been cleaned up.  More could be done in
rhel-common, but given our lack of coverage of that I'd prefer to
leave it for now.

Change-Id: I6ea784116ef59ca22878c8512c963f29c815a00a
2017-06-28 12:26:24 +10:00
Tony Breeds
c383428727 As far as block-device layout is concerned ppc64le == ppc64el
Change-Id: I06a89f256f66eba2f73dd110f5d8a61e0f0a924c
2017-06-27 00:34:33 +00:00
Jenkins
c1facd7980 Merge "Fix mkfs use wrong label option for vfat" 2017-06-26 02:41:30 +00:00
Ian Wienand
b0e0dd991c Move image download tests to default skip
The image download tests have long been too unreliable for the gate.
We need to cache the base images similar to how devstack caches it's
testing images.  Let's move them to non-voting jobs for the time
being.

This means that the gate jobs are now all based on "-minimal" and are
using infra mirrors.  Unfortunately, there is still some unreliability
because we currently have issues with infra mirrors being very slow
after AFS updates, leading to job timeouts.  But we're on the right
path...

Also, I noticed we don't have tests of the "ubuntu" image-download
based tests, which were tacitly being tested by apt-sources before we
moved that to -minimal.  Add simple tests for these.

Change-Id: Ie33ee49656872467ef68d753210032156bb6b2cb
2017-06-23 10:58:47 +10:00
Xinliang Liu
178db0c97b Fix mkfs use wrong label option for vfat
For vfat type, mkfs should use '-n' option for label.
e.g.:
mkfs -t vfat -n LABEL-STRING

Change-Id: I1414c5b8e0aeb240c3e6884e35ba75dde677db0c
2017-06-22 14:50:53 +08:00
Jamie Lennox
aa03e24c30 Use the dib python to do cleanup
In a system where python2 is not installed and /usr/bin/python is not
linked then the cleanup process will fail trying to invoke the python
script. Use the previously determined DIB_PYTHON_EXEC if it's available.

Change-Id: I128292808ccef92cc1803988b35caae5aa6fa541
2017-06-22 11:20:51 +10:00
Clark Boylan
559de43694 On suse the python2 dev package is python-devel
This was previously defined as python2-devel (which is what rhel uses),
but the actual package name is python-devel. See:

  https://software.opensuse.org/package/python-devel

Change-Id: Id61e5b05772d10c32b33d3e70cb64d5ebdcba6e4
2017-06-21 15:52:05 +10:00
Ian Wienand
18a0d970fa Move ironic-agent test to fedora-minimal
I'm uncertain as to why this is using the "fedora" element for testing
... but it requires downloading the fedora .qcow on every test which
has shown to be unreliable.  An easy thing to do is to switch it to
fedora-minimal; that will only involve downloads from local mirrors in
the gate.

Add redhat-rpm-config for minimal.  I admit I have not fully gone
through why this is not pulled in.  It's been an issue since
I459f2203fa145049dda185da952813118193d573 and there's all sorts of
bugs.

Change-Id: I37458e3926dae32a259bd5aa9efc645561b029a0
2017-06-21 15:05:36 +10:00
Ian Wienand
649f0b66d9 Start at using CI mirrors for fedora/centos
fedora/centos-minimal don't obey DIB_DISTRIBUTION_MIRROR currently.  I
don't really want them too -- we want to be able to separate the
mirrors used during the build process from those embedded into the
final image.  Add DIB_YUM_MINIMAL_BOOTSTRAP_REPOS which is a directory
with repo files to use during the install.

This introduces setup-gate-mirrors.sh which is intended to setup
repo/sources/whatever files in the openstack gate that point to the
local region mirror.  It pulls the info from the mirror_info.sh script
on each CI node.

The openstack-ci-mirrors element is updated to export these variables.
elements are updated to depend on it.  Tests are restored

Change-Id: I7604fc4d41cb1483be16b8d628a24e8fc764f515
2017-06-21 12:02:27 +10:00
Ian Wienand
f0b70211c6 Use local mirror for ubuntu-minimal jobs
This adds "openstack-ci-mirrors" element which performs various
settings to get builds using local mirrors.  As a first step, we
convert ubuntu-minimal jobs

The main trick is that since infra mirrors are created with rerepo
they are not signed (they are recreated, not cloned, and not signing
is seen as a feature in that it deters external use).  So we need to
instruct debootstrap to ignore signing and also turn it off for
in-chroot apt.  Other than that, the existing DIB_DISTRIBUTION_MIRROR
works to redirect installs.

Remove "restricted" as it's not mirrored, and I don't think we want it
in here by default.

(I think DIB_DISTRIBUTION_MIRROR is a bit of an anti-pattern, because
it leaves the mirrors in the final image -- just because you use them
to build, doesn't mean you want them at runtime).  But we don't need
to fix that now, and we don't use any created images.)

This pauses fedora testing until the next change, which moves to using
local mirrors for testing on fedora/centos

Change-Id: I778bd05a1e615c27edf1c9f0a1409119a6b3a850
2017-06-21 12:01:31 +10:00
Ian Wienand
0d37351031 Move apt-sources to ubuntu-minimal / move debian to skip list
The gate is currently extremley unstable, and these two issues are
causing most of the problems.  We need to commit them atomically so we
can get anything moving again

---

The gate is very unstable downloading the ubuntu tarballs from
upstream at the moment.  Move this to ubuntu-minimal which, in a later
change will source files from our local mirror.

We need a caching mechanism for these large files to avoid this
instability.  This is future work for the various image-based jobs.

---

Move debian to default skip lists

I don't know if it's mirrors being worked hard for the Stretch
release, but this is constantly failing the gate.  I will move this to
the -nv extras job

I am working on having the voting job use local mirrors for
everything.  Unfortunately debian infra mirrors don't have stretch yet
and we need to do some fiddling to get "stable" available.  Once we
have all this, we can consider making it voting again.

Change-Id: Iaf7b3888ef06c7aef63cbf76a94b33f96bc9c5c2
2017-06-21 10:34:53 +10:00
Ian Wienand
5d5fa06e5c Sync after writing partition table
We introduced the "settle" in
I90103b59357edebbac7a641e8980cb282d37561b thinking that maybe kpartx
had not finished writing the partition.  This probably wasn't a bad
first assumption, since we used to have this -- but is seems
insufficient.

The other failiure here seems to be if kpartx hasn't actually seen the
updated partition table in the image, so it has correctly (in it's
mind) not mounted the partition.

Looking at strace of fdisk run manually on a loopback, it will do a
fsync on the raw device after writing and then a global sync as it
exits.

This replicates this; we flush and fsync in mbr.py in the exit handler
after writing the partition, before closing the file (i've updated one
of the unit tests to double-check the call).  In the partitioning.py
caller we execute a sync call too.

Since it does seem unlikely the "-s" option of kpartx is not working,
I've removed the udev settle work-around too.

Change-Id: Ia77a0ffe4c76854b326ed76490479d9c691b49aa
Partial-Bug: #1698337
2017-06-19 17:13:36 +10:00
Ian Wienand
a0f747932d Install systemd earlier for Stretch
Debian Stretch released as stable recently, and the init system is
less tightly specified in the base dependencies (for some info, see
[1]).  It seems, probably unintentionally, that in the previous
release systemd-sysv was brought in by debootstrap, but that is no
longer happening.

Add systemd as an early dependency of debian-minimal.

Remove the package-installs.yaml as that happens too late (other
things need to know the init system to write out service files, etc
and probe for systemd utils before package-installs).  As mentioned, I
do not believe the "only install systemd on testing" idea was actually
working here, because it was being brought in during the initial
debootstrap.

Update some documentation to explain what's going on

[1] https://lists.debian.org/debian-boot/2015/05/msg00156.html

Change-Id: Id67c0cf08728407d234976f9807d3bd71d12f758
2017-06-19 13:27:33 +10:00