Commit Graph

3178 Commits

Author SHA1 Message Date
Matthew Thode
c4f83f2311 Enable support for Gentoo overlays
This is intended to eventually support building musl-libc based images,
which need the musl overlay.

Change-Id: I8f5429ffa64e74c860772d9a00ff0b7eebb7721a
2017-11-30 05:14:00 +00:00
Zuul
bbbe762dff Merge "elements: zypper-minimal: Refresh repositories where necessary" 2017-11-30 04:36:06 +00:00
Ian Wienand
1b203f8a38 Pre-install curl
As described, Fedora 27 has a curl-minimal package that comes in to
satisfy the rpm package dependency.  It conflicts with the "real" curl
package -- which is so commonly installed (by infra elements, etc)
that this becomes an annoying problem.  Just pre-install the full curl
package.

Fedora 24 is old enough to not worry about, so remove some old
workarounds to make the flow a little simpler.

Change-Id: I67baf96377109ac4521ba00243a0d91b35fafba0
2017-11-30 15:15:42 +11:00
Ian Wienand
bf8de79940 Install fedora-gpg-keys for F27
The repo GPG keys moved into a separate package [1] which now needs to
be installed.

Since the fedora-release/fedora-repos split is *long* since over,
remove that work-around and add this one.

[1] https://pagure.io/fedora-repos/c/f69f3729511c3eba5f470b1d90ea2bfee372eb29?branch=f27

Change-Id: I9ad28d5bdb78375ae21dbb16e2d8c4effb32cb35
2017-11-30 15:15:37 +11:00
Matthew Thode
c886c4cbe1 Make preinstall.d more deterministic
Reorders the preinstall, making the order more explicit.  Also dedupes
some folder setup.

Change-Id: I423dcba169558ff6037a3382b997675722e77405
2017-11-30 13:48:18 +11:00
Zuul
f74e48799d Merge "Enable gentoo in pip-and-virtualenv element" 2017-11-30 02:10:09 +00:00
Zuul
247c68b5a3 Merge "Clear /etc/machine-id to avoid duplicate machine-ids" 2017-11-30 01:56:26 +00:00
Zuul
d01d3d8832 Merge "Make python changes more reliable" 2017-11-30 01:49:07 +00:00
Zuul
71ca627d30 Merge "Add debian minimal requirement for arm64" 2017-11-29 23:47:12 +00:00
Zuul
c146d2f3b7 Merge "Fix grub2 dependency on arm64" 2017-11-29 23:47:10 +00:00
K Jonathan Harker
bce29e5c22 Use EPEL for debootstrap on centos
I747c2b8754effbc6ec82af3bf7543fd9599a6c14 removed the
centos-openstack-octata repos from the base image.  We were previously
getting debootstrap from there, so this broke the centos-7 based jobs.
Enable EPEL for the install so we use that version of debootstrap.

Additionally we noticed that dpkg wasn't installed on the build host
either, but is actually required (calls were silently failing ... a
problem for another day).  Add this.

Change-Id: I1e9db5130a05a004c3c877d1444d7097023c06e2
2017-11-28 15:21:38 +11:00
Andreas Florath
46a07de480 Fix /dev/pts mount options handling
The current implementation - as introduced in
Iee44703297a15b14c715f4bfb7bae67f613aceee - has some shortcomings / bugs,
like:

* the 'grep' check is too sloppy
* when /dev/pts is already mounted multiple times the current implementation
  fails:
  $ mount | grep devpts | sed 's/.*(\(.*\))/\1/'
  rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000
  rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000
  rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000
* code duplication
* Using the undocumented and non-robust output
  of 'mount'.

This patch fixed the above problems.

Change-Id: Ib0c7358772480c56d405659a6a32afd60c311686
Signed-off-by: Andreas Florath <andreas@florath.net>
2017-11-24 07:14:56 +00:00
Matthew Thode
ade82904a1
Make python changes more reliable
We oneshot emerge without calculating dependencies a few things to solve
for possible dependency loops.

Python 3.5 also became stable, so don't need to do special things for
it.

Matched the uninstall with the install lines (no need for a full if
statement).

Change-Id: I7c5e546612ac47d659e73a46a52e34d39ca81949
2017-11-20 23:19:46 -06:00
Zuul
8133166a63 Merge "zuul: add tripleo jobs" 2017-11-21 01:41:44 +00:00
Zuul
fb9a8b20a7 Merge "Remove setting of version/release from releasenotes" 2017-11-20 09:52:08 +00:00
Zuul
b879a03c75 Merge "Dont install python-pip for py3k" 2017-11-20 05:38:24 +00:00
Emilien Macchi
cc11be49b8 zuul: add tripleo jobs
Import the new jobs from tripleo-ci, with parity from what we had before
with legacy jobs:
- buildimage jobs
- ovb-ha

Change-Id: If93f9f3b6df1dec4c6edef43f6c5f126eeac8a32
2017-11-18 12:02:31 -08:00
Andreas Jaeger
ec2c955019 Remove setting of version/release from releasenotes
Release notes are version independent, so remove version/release
values. We've found that projects now require the service package
to be installed in order to build release notes, and this is entirely
due to the current convention of pulling in the version information.

Release notes should not need installation in order to build, so this
unnecessary version setting needs to be removed.

This is needed for new release notes publishing, see
I56909152975f731a9d2c21b2825b972195e48ee8 and the discussion starting
at
http://lists.openstack.org/pipermail/openstack-dev/2017-November/124480.html
.

Change-Id: I2e0ad6dd4088ee6d72373c5ffed0c66dee443f32
2017-11-16 22:06:48 +01:00
Markos Chandras
7f051add61 elements: zypper-minimal: Refresh repositories where necessary
We should always refresh the Tumbleweed repositories and the 'update'
one for Leap in order to always have the latest information from the
repositories.

Change-Id: I85db9d8bb7fa153f01222129e9b36fecc2632f57
2017-11-16 16:54:47 +00:00
OpenStack Proposal Bot
f0d6e04c52 Updated from global requirements
Change-Id: I03dcd2a53ac16425dfa02ce67de4dd959d1bd53d
2017-11-16 11:01:41 +00:00
Markos Chandras
da02f37de1 elements: Respect devpts mount options
This is a continuation for f2cc647dae ("diskimage_builder: lib:
common-functions: Fix options for devpts mount"). We also need to
respect the devpts mount options when the dib elements are mounting
this virtual filesystems themselves.

Change-Id: Iee44703297a15b14c715f4bfb7bae67f613aceee
2017-11-14 08:31:55 +00:00
Gregory Haynes
ab89c7d69c Dont install python-pip for py3k
We want to install python3-pip, not python-pip when we are building a
py3k image less we pull in python2. Once we stop installing python2 we
have to stop calling python2 during pip install.

Change-Id: I7d8ba9300039cce90965410a4e16ca9e711904c3
2017-11-13 23:00:52 +00:00
Adam Harwell
d4fd7f1217 Enable gentoo in pip-and-virtualenv element
Currently it will hit the `else` and try to apt-get, which fails.

Change-Id: I951882cf3897ced165e167f12877c05ee62a5054
2017-11-11 09:00:50 +09:00
Zuul
b5b9d98fa0 Merge "diskimage_builder: lib: common-functions: Fix options for devpts mount" 2017-11-09 15:18:44 +00:00
Zuul
bc6c928bb9 Merge "Move to a common lock-file directory" 2017-11-07 17:32:28 +00:00
Zhiguo Deng
271dc36f33 Add zipl element as s390x architecture bootloader
s390x architecture uses zipl as bootloader. When used in combination
with the vm element it replaces the existing bootloader element.
It's mandatory for s390x vm images.

Use cases
---------

* Allow users to create s390x images that run on nova with s390x
  libvirt/kvm backend
* Building nodepool images for s390x third party CI

Supported Distros
-----------------
The following listing shows all Distros that officially support
s390x and how those Distros are supported in DIB with this patch.

* SLES - not supported (SLES is not supported in DIB)
* RHEL - not suppoprted (RHEL is not supported as KVM guest on s390x,
                         therefore there's no rhel7 qcow image for s390x available
                         like it is for other archictectures)
* Ubuntu - supported

Ubuntu images can for example be built using the following commands:

  $ disk-image-create ubuntu-minimal zipl vm
  $ disk-image-create ubuntu-minimal zipl
  $ disk-image-create ubuntu zipl vm

Testing
-------

Cross architecture building of s390x images is not supported so far.

The plan is to set up a ThirdParty CI that builds the image for s390x and
provides the logs.

Co-Authored-By: Andreas Scheuring <andreas.scheuring@de.ibm.com>
Co-Authored-By: Holger Smolinsky <holger@smolinski.name>
Co-Authored-By: Zhiguo Deng <bjzgdeng@linux.vnet.ibm.com>
Co-Authored-By: Arne Recknagel <arne.recknagel@hotmail.com>

Closes-Bug: #1730641

Change-Id: I576e7edda68da12e97c60af38f457915efe7b934
2017-11-07 17:19:27 +01:00
Markos Chandras
f2cc647dae diskimage_builder: lib: common-functions: Fix options for devpts mount
Commit cebfcf85f9 ("Use -t devpts for
/dev/pts mounts") switched from using '--bind' to '-t devpts' for
mounting the /dev/pts virtual filesystem. However, mounting devpts to
another location also affects the host's /dev/pts mountpoint. Since we
are now mounting devpts without options we end up with the following one
on openSUSE

devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,mode=600,ptmxmode=000)

instead of the one we want

devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)

The missing gid=5 options results to boot problems for virtual machines

So in order to fix that, we need to use the existing devpts options for
/dev/pts so we don't lose them in the new mount.

Change-Id: I17f2c2bb96b807f8dbc07185ae0147bff3230f92
2017-11-06 10:23:03 +02:00
Zuul
906a3f4a57 Merge "Use -t devpts for /dev/pts mounts" 2017-11-02 14:20:05 +00:00
Zuul
1b0631da84 Merge "Update proliant-tools to support Gen10 Proliant servers" 2017-11-01 07:22:03 +00:00
James E. Blair
5008358d52 Zuul: add file extension to playbook path
Zuul now supports including the file extension on the playbook path
and omitting the extension is now deprecrated.  Update references
to include the extension.

Change-Id: I358a4ee4d7faed68e449592480cb5d8fafa70cd1
2017-10-29 12:43:09 -07:00
Ian Wienand
b25d0337b8 Move to a common lock-file directory
In a couple of places we use flock for critical sections, but we leave
lockfiles around in various locations which can be confusing.

Introduce DIB_LOCKFILES global (under ~/.cache/dib/lockfiles) and
write lockfiles in there.

Fix up removal of the lockfile in the yum path; we just want to make
sure we cleanup the .rpmmacros file, but we don't need to remove the
lockfile as well.

Co-Authored-By: Andreas Florath <andreas@florath.net>

Change-Id: Ie810b2836be521325afe923708d046112e1e1e20
2017-10-26 16:27:59 +11:00
Zuul
8f025691ba Merge "Dont install python-virtualenv for py3k in deb" 2017-10-24 06:33:45 +00:00
Zuul
c5f713b1ec Merge "Change to install a package in 'proliant-tools'" 2017-10-24 05:48:25 +00:00
Yolanda Robla
ba11376328 Create rescue user on ironic agent
Create a new service, that will be launched after ironic
agent has been exited. This will launch an script that will
take the rescue password, and create the rescue user with
that credentials.

Depends-On: I7898ff22800dedba73d7fbfb3801378867abe183
Change-Id: Ic3a241e2789a122d3d966e7e2148306fd0cf6aed
Partial-Bug: 1526449
2017-10-23 12:50:32 +00:00
Andreas Florath
cebfcf85f9 Use -t devpts for /dev/pts mounts
Currently a bind is used when mounting /dev/pts in chroot.
This leads to problems - especially when running DIB in parallel:
It was observed that the /dev/pts mount vanishes from the host
system.

This patch uses '-t devpts' - as it is done for /sys and /proc -
for handling /dev/pts.

Change-Id: Id7775ae6fca6502af800e7b73a00862ef320206b
Signed-off-by: Andreas Florath <andreas@florath.net>
2017-10-23 07:13:02 +00:00
Gregory Haynes
00d7c619e9 Dont install python-virtualenv for py3k in deb
On ubuntu we detect that in python3 we need to install
python3-virtualenv, but append this to the packages to install rather
than replace python-virtualenv which results in both being installed
(and therefore grabbing python2).

Change-Id: I422490ebe9a9c655552685bc2ff342d288335a9c
Closes-Bug: #1724656
2017-10-18 23:11:55 +00:00
Zuul
a0045985f2 Merge "Remove dd from LVM element" 2017-10-17 08:15:03 +00:00
Ian Wienand
cc61e6ac12 Import legacy playbooks
Import legacy playbooks for zuulv3 transition

Change-Id: I30763af7a4062d3c254a21eda532e3a4e90192da
2017-10-17 10:41:30 +11:00
Zhangfei Gao
6cc155fd66 Add debian minimal requirement for arm64
Debian system building fails, fixed by adding arm64 package.

DIB_RELEASE=jessie disk-image-create debian ironic-agent \
    grub2 devuser -a arm64 -o deploy-jessie
dib-run-parts Running /tmp/dib_build.v5FEtaKx/hooks/cleanup.d/99-extract-kernel-and-ramdisk
ls: cannot access /tmp/dib_build.v5FEtaKx/mnt/boot/vmlinu*: No such file or directory

Change-Id: I610d767785df49fed954f12854be5ae78ff9baa6
Signed-off-by: Zhangfei Gao <zhangfei.gao@linaro.org>
2017-10-16 13:39:50 +08:00
Zhangfei Gao
a8635d3a3b Fix grub2 dependency on arm64
Unable to locate package reported when
DIB_RELEASE=jessie disk-image-create debian ironic-agent \
grub2 devuser -a arm64 -o deploy-jessie

E: Unable to locate package grub-pc-bin
E: Unable to locate package shim-signed
E: Unable to locate package grub-efi-amd64-signed

Fix the issue via adding arch dependency and arm64 packages

Change-Id: I40650a887b575a9c2b00a8c5036c35354d548673
Signed-off-by: Zhangfei Gao <zhangfei.gao@linaro.org>
2017-10-16 13:39:34 +08:00
Jenkins
7a76270227 Merge "Add timestamp output filter" 2017-10-09 15:56:31 +00:00
Jenkins
fd8f92e542 Merge "Update Gentoo element for element changes" 2017-10-08 23:58:20 +00:00
Andreas Florath
bb6cf52d85 Remove dd from LVM element
This patch removes the unneeded dd calls in the lvm block device
plugin.

After removing the underlying block device, there is the need to call
'pvscan --cache'.  This is done by a dedicated LVM cleanup node which
is cleaned up after the the underlying block device.

Change-Id: Id8eaede77fbdc107d2ba1035cd6b8eb5c10160c3
Signed-off-by: Andreas Florath <andreas@florath.net>
2017-10-08 17:21:21 +00:00
Matthew Thode
e29f031bec
Update Gentoo element for element changes
There have been a few changes over the past few months, here we make the
following changes.

* change from backtrack=99 to complete-graph as a more correct flag
* make python version selection more in line with what gentoo supports
* set up python before stuff gets pip installed
* ensure we have the proper pip so we can install pip packages as root
* ensure we have the proper use flags for the disk formatting changes
* set DIB_RELEASE like other distros
* fix openssh-server element for gentoo

Change-Id: I17202de3016616ce34c8cbead7d0fb047a64e96b
2017-10-08 12:02:46 -05:00
Anshul Jain
afb7084a4d Update proliant-tools to support Gen10 Proliant servers
This commits make update to ssacli version to point to latest
ssacli release that has support for HPE P/E-Class SR Gen10 controllers.

Change-Id: Ia9a0eaec78d601f56b4036e57601554b87f21acc
Closes-Bug: 1721185
2017-10-04 07:02:07 +00:00
Andreas Florath
fa6c731132 Move fstrim to block device layer
The call to fstrim in disk-image-create is currently useless, because
at the time this is called, the file systems were already umounted by
the block device layer.

The current implementation of the block-device mount plugin does not
call fstrim at all: resulting in larger image sizes.

This patch removes the useless fstrim call from the disk-image-create
script and moves this into the block-device mount.py.

The resulting image might be much smaller.  Example: Ubuntu Xenial
with some elements; once with and once without this patch:

-rw-r--r-- 1 dib dib 475661824 Sep 16 06:43 ubuntu-xenial-without-fstrim.qcow2
-rw-r--r-- 1 dib dib 364249088 Sep 16 09:30 ubuntu-xenial-with-fstrim.qcow2

Change-Id: I4e21ae50c5e6e26dc9f50f004ed6413132c81047
Signed-off-by: Andreas Florath <andreas@florath.net>
2017-09-28 17:48:59 +10:00
Jenkins
dc215674f8 Merge "Actually sort mount-point list" 2017-09-27 07:20:29 +00:00
Tony Breeds
ab1ed1e59a Add Constraints support
Adding constraints support to libraries is slightly more complex than
services as the libraries themselves are listed in upper-constraints.txt
which leads to errors that you can't install a specific version and a
constrained version.

This change adds constraints support by also adding a helper script to
edit the constraints to remove diskimage-builder.

Change-Id: I795800589ea1491efd0e489a38b60e6adf33e7f3
2017-09-22 14:54:42 +10:00
Ian Wienand
06b5ce4573 Revert "Support networkx 2.0"
This reverts commit a47ff0dd4a.

Since this merged, a global-requirements pin to keep networkx <2.0 has
also merged.  The plan is:

1. revert our 2.0 support and
1a. take the <2.0 pin from global requirements
2. figure out how to use constraints properly in our testing
3. restore this, with a depends-on for a 2.0 bump in requirements
   (which will self-test, see 3.)
4. when other projects are ready for a global 2.0 bump, merge
   in a controlled fashion

This reverts the 2.0 support, and adds the pin for networkx <2.0

Change-Id: I18f6a1115da779581245e3dd423fd90516974a33
2017-09-22 08:02:44 +10:00
Ian Wienand
a47ff0dd4a Support networkx 2.0
Networkx 2.0 released recently.  The main difference for us is that
"node" is no longer a dictionary and should be accessed via "nodes",
and the topological_sort returns an interator

Closes-Bug: 1712693
Change-Id: I78e89f2261b8b8d28c68b517c1e61691ab40016c
2017-09-21 09:43:01 +00:00