Commit Graph

3693 Commits

Author SHA1 Message Date
Zuul
0668cd37fe Merge "Add ironic jobs to the CI" 2020-02-04 06:23:52 +00:00
Dmitry Tantsur
3b27ae1578 Add ironic jobs to the CI
Ironic is now using diskimage-builder for producing and publishing
official production-ready IPA ramdisks. These jobs test ironic
against changes in diskimage-builder to ensure they still work:
* Two check-image jobs merely build an IPA image with DIB:
  * The CentOS 8 job is our primary now and should eventually
    become voting.
  * The CentOS 7 one is used on stable/train and will be removed
    once DIB stops supporting this version.
* The two other jobs build and use the IPA image to deploy a node
  in two different ways (pushed via iSCSI and pulled via HTTP).
  Since they're likely to be affected by factors outside of DIB
  control, they're added non-voting.

Depends-On: https://review.opendev.org/703585
Change-Id: Ibce594657bb90db1bac63fa9f965ee88586f8246
2020-01-21 12:48:32 +01:00
Antoine Musso
168127b60a dib-lint: test elements have README.rst file
Add a basic test to ensure that all elements have a README.rst file.
This way they will be exhaustively listed in the Sphinx documentation.

Add dummy README.rst for 'disable-selinux' and 'rpm-distro' elements.

Change-Id: Ia5252ddd89b5ae5c6e9a12a66ef10f912fd54da5
2020-01-20 11:43:43 +01:00
Andreas Jaeger
71ef0a8b3c Remove trusty jobs
OpenDev is removing trusty from the infrastructure, remove the jobs
using ubuntu-trusty.

Depends-On: https://review.opendev.org/702817
Change-Id: I0d328f4d1125be7e3801a9dd2c9374086024b903
2020-01-17 09:23:05 +01:00
Carlos Goncalves
ae2be0b464 Fix Yum repositories and GPG keys for CentOS 8.1
CentOS 8.1 split repositories and GPG keys out into subpackages. This
broke DIB support for CentOS 8.

7e41cef418
26a0d73ced

Change-Id: If3de6efa6074e059dc9fdd47c7bdc19d26d4d7f2
2020-01-15 19:39:00 +01:00
Matthew Thode
87e7f7b869
allow building of Gentoo images for non-systemd profiles
Change-Id: I61e3ba391eaaf9b300f88b082c03116388401ba3
2019-12-20 17:51:55 -06:00
Mohammed Naser
6f1b51627f modprobe.d: use $TMP_MOUNT_PATH
The hook inside extra-data.d runs outside the chroot when
building the image which means that we need to prefix paths
inside the hook to avoid running things on the host.

We also run it with sudo because if we're running DIB not
as root, /etc is uid 0 and we'll get a permission denied.

Change-Id: I1838890fe124c84c879285a471bcc78fe47d6c23
2019-12-18 11:42:02 -05:00
Zuul
71e1bcbdb8 Merge "Install rng-tools in Red Hat family distro images" 2019-12-17 06:25:32 +00:00
Zuul
f6d32a684f Merge "Add arm64 based functional test" 2019-12-16 21:31:51 +00:00
Ian Wienand
082397a86a Add arm64 based functional test
An initial functional test for bionic/arm64 builds, put it in the new
arm64 check queue.

Change-Id: I5f8a047f41c6555da7211b758c55f7a87b3aa5d1
2019-12-16 10:51:01 +11:00
Zuul
db9501cab2 Merge "Fix wrong URL in ironic-agent element" 2019-12-15 23:43:33 +00:00
Carlos Goncalves
3ff7365ee8 Install rng-tools in Red Hat family distro images
Make sure rngd, a hardware RNG entropy gatherer daemon, is installed on
all DIB-built Red Hat family distro images. rngd comes installed by
default in a typical base installation as it's proven to help speed
things up.

Nova attaches the virtio-rng-pci device to VMs. virtio-rng-pci is a
device that provides feed random data. However, it is of little to no
use if the virtual machine is not configured to make use of given
device. That is where rngd can help by facilitating entropy to the pool
from virtio-rng-pci.

$ openstack image set --property hw_rng_model=virtio [...]
$ openstack flavor set --property hw_rng:allowed=True [...]

DIB-built minimal images do not come with rngd installed. This patch
makes sure the daemon is installed. Its systemd service comes already
enabled.

Change-Id: I34a989dbfc57d4c98113ac25c81dfb500945ff0a
2019-12-12 20:26:06 +00:00
Carlos Goncalves
45d9917c00 Update bindep for RHEL/CentOS 8
Install policycoreutils-python in RHEL/CentOS 7.
Install policycoreutils-python-utils in Fedora and RHEL/CentOS 8.

This patch also drops python3-setuptools from bindep as Fedora 28 is
EOL and RPM dependency is fixed in >= Fedora 29.

Change-Id: I0ecd29e0f113005d7e993add84d2c4fb90fd16e3
2019-12-12 18:28:56 +01:00
Zuul
74dff74907 Merge "Set correct python version for non-chroot scripts" 2019-12-12 16:48:17 +00:00
Madhuri Kumari
46491eb1b5 Fix wrong URL in ironic-agent element
Change-Id: Iaa107094b1409f93b101b23cea2e874c60ea26f5
2019-12-12 16:39:48 +05:30
Michael Johnson
9c7b8d1714 Set correct python version for non-chroot scripts
Some phases of diskimage-builder run outside the chroot environment,
such as the extra-data.d scripts, and don't have access to dib-python.
This means these scripts may choose the wrong python version by using
"#!/usr/bin/env python" to execute. The svc-map element is an example.

This patch creates a temporary directory and symbolic link for the
correct version of python, then manipulates the environment PATH
to preference the symbolic link "python" command.
This will allow elements with these scripts to work correctly with
the version of python diskimage-builder is running under.

Change-Id: I289d621e1bfbba0eb174dff977d1a5c92c04e4fa
Co-Authored-By: Ian Wienand <iwienand@redhat.com>
2019-12-11 22:52:28 +00:00
Ian Wienand
801fa77a16 Work around Trusty ext4 metadata_csum errors on Bionic
As described inline, Bionic hosts will build invalid Trusty images.
Hack around this by disabling metadata_csum in the ext4 mkfs.

Change-Id: Ibd67d58ca830a9e60605d0700ee2b17906c804e6
2019-12-11 16:02:13 +11:00
Ian Wienand
4ca306cc0e Drop Xenial from functional test
Devstack dropped Xenial support with
Iefcca99904dde76b34efbbfc0e04515dfa5a09e5.

I have ported the required debootstrap in the openstackci PPA to
Bionic, so these builds should work there now.

Unfortunatley, there's no current solution for Suse builds as there is
no zypper for the minimal builds on bionic.  These will either have to
fix that, or figure out how to pin devstack to Train branch for the
jobs.  Since it causes retries in the gate as non-voting, put it in
experimental for now until we figure something out.

Trusty also fails due to issues with ext4 versions using bionic hosts.
This is fixed in a follow-on
Ibd67d58ca830a9e60605d0700ee2b17906c804e6; the job is disabled for now.

Change-Id: I71823236731583e28fddcceb71f44d09b58664a4
2019-12-11 16:01:52 +11:00
Zuul
361a751d36 Merge "Allow zypper repos to be overrideable" 2019-12-04 06:14:55 +00:00
Zuul
68b41854b2 Merge "Install ndisc6 package in element script" 2019-12-03 22:41:09 +00:00
Zuul
9b00377f31 Merge "Fix regex for mirror URL substitution" 2019-12-03 21:56:19 +00:00
Zuul
0b1eba25d5 Merge "Break retry loop on success in dhcp-all-interfaces" 2019-12-03 21:56:18 +00:00
Zuul
e3a904d4a6 Merge "Fix login.defs config for tumbleweed" 2019-12-03 21:56:16 +00:00
Carlos Goncalves
670df3326d Fix regex for mirror URL substitution
The base URL of EPEL repository installed by the epel-release package in
CentOS 8 at least now defaults to https.

The error seen when building an CentOS 8 image was:
"Error: Cannot find a valid baseurl for repo: epel"

This patch fixes it so that it will always match regardless of being
http or https.

Change-Id: I9ec5536ee72047c929a1ef6950ff4e9092842a4c
2019-12-03 19:13:18 +01:00
Harald Jensås
7948fee7e2 Install ndisc6 package in element script
The ndisc6 package is not yet available in EPEL 8.
See: https://bugzilla.redhat.com/show_bug.cgi?id=1779134

Until the package is available set the pkg-map to "" for
the ndisc6 package when distro is redhat and install the
package using || true in the element script instead so
that CentOS 8 build's do not fail because of the missing
package.

Once the package is in EPEL 8 this change can be reverted.

Related-Bug: #1754219
Change-Id: Icd4bad8852ce5ba40fb0e7b0d335191efbe88c67
2019-12-03 14:57:33 +01:00
Michele Baldessari
f9dcbd30cc Make sure DIB_DEBUG_TRACE has a default value
After the introduction of 'Add output for mis-configured element
scripts' we started seeing CI failures in tripleo where
instack-undercloud is being used (rocky/queens):

  /usr/lib/python2.7/site-packages/diskimage_builder/lib/dib-run-parts: line 108: DIB_DEBUG_TRACE: unbound variable
  INFO: 2019-12-02 16:24:33,423 -- ############### End stdout/stderr logging ###############
  ERROR: 2019-12-02 16:24:33,423 -- Hook FAILED.

Let's make sure that by default the env variable is set
to 0.

Change-Id: I38c76c0edee436f1e7dd0c9a868cea1e6ee3271d
Closes-Bug: #1854904
2019-12-03 08:26:12 +01:00
Colleen Murphy
ad67aa213b Allow zypper repos to be overrideable
Without this change, operating system elements that use the
zypper-minimal element always must use download.opensuse.org as their
repository source. This change makes ZYPPER_REPOS overrideable, which
allows the user to create custom operating system elements that can use
private repositories as their source for base packages. For example,
with only this change, it is possible to create a sles-minimal element
that generates a SLE 15 SP1 image just by overriding DIB_ZYPPER_REPOS
and DIB_OPENSUSE_PATTERNS.

Change-Id: I46e40fbe4408d4204056a27b182b21213f1176ff
2019-12-02 16:25:53 -08:00
Colleen Murphy
4fb5a57b8a Fix login.defs config for tumbleweed
On openSUSE Tumbleweed, the login.defs config file was moved under
/usr[1]. This change allows the login.defs config change to work for
both old and new locations.

[1] https://build.opensuse.org/request/show/736424

Change-Id: Ia5eff5e7b0709836278361b1b8daa788619eff75
2019-12-02 16:21:45 -08:00
Hervé Rousseau
b91e212434 Break retry loop on success in dhcp-all-interfaces
If rdisc6 is available, a node using this element will loops until
DIB_DHCP_TIMEOUT is reached because of a missing 'break' when rdisc6
return code is 0.
This will mark the dhcp-interface@.service unit as failed (because it
has the same timeout) and not bring any network interface online.

Change-Id: I034dcda94d765f236950ebcbee36789f5bdc515f
Closes-Bug: #1854717
Signed-off-by: Hervé Rousseau <hroussea@cern.ch>
2019-12-02 15:50:20 +01:00
Zuul
a231bc6b9f Merge "Add IPv6 support in dhcp-all-interfaces" 2019-11-26 23:54:54 +00:00
Zuul
20e468c604 Merge "Stop installing pydistutils.cfg" 2019-11-26 02:12:41 +00:00
Monty Taylor
75341292f9 Stop installing pydistutils.cfg
Support for easy_install codepaths is increasingly broken, and now
putting allow-hosts in this file breaks most recent pip. Just stop
installing the file - people should be using pip anyway.

Change-Id: I0a6b2432f81d80fbcbb336403fe555003880fa9f
2019-11-25 11:46:42 +11:00
Zuul
4248858925 Merge "Update bindep.txt for some missing dependencies" 2019-11-22 07:43:05 +00:00
Zuul
fdb6225acc Merge "Add output for mis-configured element scripts" 2019-11-22 06:57:40 +00:00
Zuul
9dd161471a Merge "Ensure nouveau is blacklisted in initramfs too" 2019-11-22 06:57:39 +00:00
Zuul
f268c72ed9 Merge "Adds support for GPG keyring" 2019-11-22 06:57:38 +00:00
Zuul
05a6f898fd Merge "Only wait for checksum processes" 2019-11-22 06:57:37 +00:00
Zuul
6143d40def Merge "Introduce manual setting of DIB_INIT_SYSTEM" 2019-11-22 06:43:25 +00:00
Ian Wienand
0500f67fda Update bindep.txt for some missing dependencies
We rely on curl for a few things that like to download remote files.

The failure cleanup processes rely on "ps" to kill left-over chroot
processes (_get_chroot_processes)

git is required for source installs

Change-Id: Icddcc13d568b8d5cd3e8de05907ff327136088df
2019-11-22 05:44:01 +00:00
Ian Wienand
bee30b43d2 Only wait for checksum processes
When running under nodepool in a foreground, non-daemonized situation
without a tty (i.e. within a container) we're seeing this "wait" hang
indefinitely.

It is probably related to "outfilter.py" and output file descriptors,
although TBH we haven't completely root-caused it.  I won't claim this
is a great solution, but it should hopefully let the dib process
finish and just die, where outfilter will disappear.

Change-Id: If78da54df3d4c240fee16aee4413ec554b37c1d6
2019-11-21 14:04:09 +11:00
Andreas Florath
3636b40f74 Introduce manual setting of DIB_INIT_SYSTEM
The current implementation evauates the dib-init-system
script too early.  Also it looks that there is no simple
way of getting the info about the init system automatically:
another element can install (later on) a different
init system.  Therefore the only reliable way of setting
this is manual.

Change-Id: I6e9ffa1bdb3154f488f4fd335b197699b86aacd4
Signed-off-by: Andreas Florath <andreas@florath.net>
2019-11-21 12:38:15 +11:00
Michael Johnson
824042d608 Add output for mis-configured element scripts
I commonly get asked for help when people are attempting to create
local image elements and they cannot get them to work.
diskimage-builder silently ignores element scripts that it doesn't
find to it's liking, such as non-executable or files with extensions
(.sh is a common mistake).
This patch extends the '-x' tracing flag down to dib-run-parts and
will cause it to print out helpful messages when these files would
otherwise be silently ignored.

Examples:
Ignoring non-executable files: 10-do-not-run-me
Ignoring non-conforming filenames: 10-I-can-run.sh

I am not enabling these by default as they can create extra noise
and require additional filesystem IO to produce.

Change-Id: Ic804efca3015c199440b4b10da951d71a815c64f
2019-11-20 15:05:30 -08:00
David Hill
f7ee1cd733 Add grub-efi-x86_64 pkg-map
Add grub-efi-x86_64 pkg-map

Change-Id: I64fab237c3fcd5e4fe1e74eb17ddd2a9aa8110f5
Closes-bug: #1852762
2019-11-16 21:15:28 -05:00
Harald Jensås
f94508d537 Add IPv6 support in dhcp-all-interfaces
When the rdisc6 utility is available probe for router
advertisement. configure eni and rhel-netscripts interfaces
to do IPv6 address configuration according to the flags
in the RA recived from the router.

The systemd service file timeout is DIB_DHCP_TIMEOUT * 2,
so that DHCPv4 can timout, and dhcpv6 run before the service
times out.

Retries are commented in dhclient.conf, without it we end up
trying DIB_DHCP_TIMEOUT * 60 before the client move on to
IPv6.

WHEN:
  Stateful address conf.    :          No
  Stateful other conf.      :          No
THEN:
  Do not run dhclient at all, autoconfiguration via
  SLAAC only.

WHEN:
  Stateful address conf.    :          No
  Stateful other conf.      :          Yes
THEN:
  Run "dhclient -6 -S", The ``-S`` option makes the
  dhcp client not request an address, only other
  options such as DNS servers and NTP servers from
  DHCPv6 server.

WHEN:
  Stateful address conf.    :          Yes
  Stateful other conf.      :          Yes
THEN:
  The dhcp client should request an address _and_ other
  options such as DNS servers and NTP servers from
  DHCPv6 server.

NOTE: No IPv6 support added for suse-netscripts

Closes-Bug: 1754219
Change-Id: Icdc79875c33f894ab7eaec8afdfb33a731efff99
2019-11-13 09:31:01 +01:00
Andreas Jaeger
76389d0afe Remove unused job
The job dib-nodepool-functional-openstack-opensuse-150-src has been
replaced by  dib-nodepool-functional-openstack-opensuse-15-src, remove
the now obsolete job.

openSUSE 15.0 is EOL in infra.

Change-Id: I54d9a0f43240c4e76b645e632ba24a793ba79aee
2019-11-13 08:00:30 +01:00
Felipe Alencastro
48ff601098 Adds support for GPG keyring
Currently DIB_ADD_APT_KEYS only supports GPG armor keys, while
default Debuntu apt gpg keys are in keyring format.

Change-Id: I361c375e25b03a08b19052b10c6733939c8df921
2019-11-07 17:32:05 -03:00
Ian Wienand
6758849b66 vhd-util : note on Xen/RAX images
Add a note covering what we've discovered about creating RAX/Xenserver
compatible images over the last few days.

Change-Id: Iffd9dab1ca54f27390ec7850093e828ca7576e98
2019-10-31 08:36:07 +11:00
Ian Wienand
b52b560fb0 Revert "Drop vhdutil dependency, use qemu-img"
This reverts commit a3e9e7f89e.

We still have some issues with vhd creation on RAX

In short, it appears that images fail to resize unless they have a
specific "creator" field.  Revert this while we consider the options.

[1] https://bugs.launchpad.net/nova/+bug/862653

Change-Id: I2b6a3bfbfe28432fbb6a2ce4a0211939d224b8d5
2019-10-30 09:28:58 +11:00
Ian Wienand
371669c981 Revert "Pause Fedora jobs"
This reverts commit dad99e54f6 and
c26cb9c2cfe20ba60ba7e6dbcd428bc14c990034; fixed by [1]

[1] https://bodhi.fedoraproject.org/updates/FEDORA-2019-2368fc415f

Change-Id: I397637c1de4414397305158d6a4f51c63c76141a
2019-10-29 10:02:18 +11:00
nishagbkar
9e149ce8bb Deprecates the existing "ironic-agent" element in DIB
The "ironic-agent" is copied to ironic-python-agent-builder and
hence it is deprecated from DIB.

Remove from functional testing

Change-Id: Ibc4f75b9d7e2a31994fc86d05bd57975f00fb74f
Task: 36198
Story: 2005114
2019-10-29 10:00:47 +11:00