Commit Graph

522 Commits

Author SHA1 Message Date
Zuul
fe15b197fb Merge "ubuntu: Add options to ignore mirror components and use insecure repos" 2018-10-18 04:34:00 +00:00
Zuul
56634e0938 Merge "Turn down pkg-map and hook copy tracing output" 2018-10-18 03:56:12 +00:00
Zuul
d3d17b2e43 Merge "Fix DIB ubuntu-minimal running on bionic (18.04)" 2018-10-18 03:08:37 +00:00
Ian Wienand
8b0111477e Set EPEL mirror during openstack-ci-mirrors
We want to set DIB_EPEL_MIRROR for the epel element, which means we
need access to the CI mirror info script in the chroot.  Copy it into
the temp directory with extra-data.d and update the environment file
to find it.

Change-Id: Ia12f0cbdeb194eef3155497ceb5ffc4a452aad76
2018-10-18 13:53:53 +11:00
Ian Wienand
c47ee6e121 ubuntu: Add options to ignore mirror components and use insecure repos
When using the upstream cloud images with the "ubuntu" element, they
have universe and multiverse enabled which we don't mirror.

To use the infra mirrors as a DIB_DISTRIBUTION_MIRROR with this
element, we need to be able to skip redirecting to universe and
multiverse, and additionally enable insecure repos (as we don't gpg
sign our mirrors).

Add and document two new variables with the ubuntu element to do this.
This is then setup by the openstack-ci-mirrors element so that we use
local mirrors duing dib functional testing for the "ubuntu" element.

Change-Id: I6ffbde07fa0e103641ee5c5f9d9e854e5b2168dc
2018-10-18 13:53:48 +11:00
Matthew Thode
3ad8235180 simplify overlay logic for Gentoo
openssl/cryptography versions are updated/stable
musl profiles need newer versions of open-iscsi masked as upstream
doesn't want to work with multiple libcs

Change-Id: If5baf339516390ae332015928557c6bb734486c2
2018-10-18 00:21:31 +00:00
Matthew Thode
1e257f59cc simplify python3.6 selection on gentoo
python 3.6 is now stable in gentoo

Change-Id: Ide96c039b03817c216ab58930ff5c897556e6c8a
2018-10-18 00:21:16 +00:00
Ian Wienand
6c077d7c2a Turn down pkg-map and hook copy tracing output
This is a lot of very low value noise in the logs as these iterate
through all the elements (often doing nothing).  Turn it down and add
an echo so we just see what elements it is working on.

Change-Id: I0687de4722766189db9d4a7bd7d3cfb45d387b62
2018-10-18 11:03:17 +11:00
Matthew Thode
89f95a5351
enable caching for gentoo builds
To facilitate this I've created two new environment variables to set
the environment and default options for package actions.

eval is needed for the export as it preserves quotes.

Change-Id: Ib03651ee8dacd48cd1c135afd57cd31101356056
Signed-off-by: Matthew Thode <mthode@mthode.org>
2018-10-15 23:17:37 -05:00
Ian Wienand
39fb794ebc Allow debootstrap to cleanup without a kernel
Currently there's more-or-less an assumption that a kernel is
installed, so module blacklists are simply echoed into the modprobe
blacklist.  This may not be the case with some ongoing container work.

Although we don't need to blacklist modules for containers, it also
doesn't hurt.  Move the debootstrap element to the new modprobe
element, and allow it to create the blacklist directory.

Change-Id: I0f057caf473951df56a2af9633e3a5b53e0809b1
2018-10-08 16:02:25 +11:00
Pierre Riteau
bacceba41d Fail build due to missing kauditd only when SELinux is enabled
With the check added in commit 7566819139,
diskimage-builder fails to build RPM-based images if kauditd is not
running. However, this is only valid for environments where SELinux is
enabled. If SELinux is disabled (which is identified by an empty _runcon
variable), proceed with running setfiles.

Change-Id: I1b056f20a3a55f7333391207d9e1049d25ece041
Closes-Bug: #1779273
2018-10-05 10:02:14 +01:00
Zuul
3e6d658687 Merge "allow building non-gentoo images on gentoo hosts" 2018-10-03 11:08:11 +00:00
Michael Johnson
330626482a Fix DIB ubuntu-minimal running on bionic (18.04)
Diskimage-builder fails to build ubuntu-minimal images when run on
a Ubuntu bionic-beaver (18.04) instance.
The user gets "Couldn't create tempfiles for splitting up" when
apt-get update is run in the ubuntu-minimal element root.d.

The issue is that the /tmp inside the chroot is not getting the
proper permissions applied from the base-files package. This is likely
because the pip-cache element has already created the directory before
the base-files package is installed.

This patch changes the order of pip-cache to root.d/11-pip-cache so that
it runs after teh base OS root.d elements run.

Change-Id: I6fd1cb2a23422206884165eb502b260f0c1e52f7
2018-10-01 19:59:30 -07:00
Jesse Pretorius
f4c5326b8e ubuntu-common: Update default DIB_RELEASE to bionic
The ubuntu-minimal README states that the latest Ubuntu LTS
is the default, but currently that is not true. This patch
changes the default to the current LTS.

Change-Id: I10f28314d1a5969c20094194637cfe31219d228c
2018-09-20 02:06:04 +01:00
Jesse Pretorius
8c69da65ac Remove redundant sources change/update
The apt sources are set out in root.d/75-ubuntu-minimal-baseinstall
and the cache is updated, cleaned and a dist-upgrade is done there.

As such, this file is unnecessary.

Change-Id: Idab5ede3f235bc204c4bdebf40fbcf4a12e5bc2f
2018-09-20 00:05:07 +00:00
Jesse Pretorius
30493f7655 Move common ubuntu environment setting to ubuntu-common element
The ubuntu, and ubuntu-minimal elements both make use of a common set
of environment settings to determine the distribution name.

The ubuntu-minimal element also does a few extra things which would
appear to apply to both sets and bring in extra architecture support.
As such, these are included in the common element.

This intends to be part of a series of patches which will eventually
create a new element to build a minimal ubuntu-systemd-container
element which can be used for lxc/nspawn containers.

Change-Id: Ia4e620f7d3fa6215484a8d218cea2f28bd1ffaee
2018-09-20 00:04:55 +00:00
Zuul
b86af3dc6a Merge " IPA requires iptables" 2018-09-13 02:05:42 +00:00
Ian Wienand
f6a2452d4c Only append DIB_BOOTLOADER_DEFAULT_CMDLINE to default grub entry
The grub.cfg has two variables [1]

 GRUB_CMDLINE_LINUX : used on all boots
 GRUB_CMDLINE_LINUX_DEFAULT : additionally used on all "normal" boots

The problem with I2298675dda1f699c572b3423e7274bc8bd7c1c9d is that it
appened the values in DIB_BOOTLOADER_DEFAULT_CMDLINE to both of these,
resulting in duplicated arguments.  I don't think we considered that
GRUB_CMDLINE_LINUX_DEFAULT actually already appends to the
GRUB_CMDLINE_LINUX values.

Make DIB_BOOTLOADER_DEFAULT_CMDLINE only append itself to
GRUB_CMDLINE_LINUX_DEFAULT.  That seems to line up sensibly with the
name of the variable.

Documentation is enhanced around this, and a releasenote added.

[1] https://help.ubuntu.com/community/Grub2/Setup

Change-Id: I76b5442a9090c19a6540ed2d4ab324546f241ebf
Closes: #1791736
2018-09-13 09:51:50 +10:00
Zuul
b29aee3383 Merge "Install sudo on Gentoo images by deault" 2018-09-11 00:09:33 +00:00
Zuul
9f93b2ce3d Merge "Fix CentOS image build failure when dib runs on system where audit disabled" 2018-09-06 11:24:59 +00:00
Olivier Bourdon
7566819139 Fix CentOS image build failure when dib runs on system where audit disabled
Without this fix, building a CentOS image on Ubuntu where audit=0 is passed
as a kernel boot parameter will lead to the following error:
disk-image-create centos7 dhcp-all-interfaces cloud-init-nocloud \
    devuser yum epel baremetal
... dib-run-parts Running tmpdir/hooks/cleanup.d/99-selinux-fixfiles-restore
... Error connecting to audit system.

Change-Id: I229d9b72f88bffddca42da57f01c27e902427071
2018-09-04 08:23:02 +02:00
Charalampos Kominos
c85141291e Fix bootloader packages for aarch64
Due to the arm naming convention, building centos images for arm64 and
aarch64 does not yield the same result. In order to locate grub2 on
aarch64 the correct mapping is added.

Change-Id: I1bb227b2523e420e394fec8c52c6c79fcdd31c53
Closes-Bug:#1789414
Signed-off-by: Charalampos Kominos <Charalampos.Kominos@enea.com>
2018-08-31 17:01:47 +02:00
Zuul
b602c05551 Merge "Add netcat to redhat-common map-packages" 2018-08-23 11:26:12 +00:00
Zuul
f33526d20c Merge "better handle existing keywords files/directories" 2018-08-10 06:07:06 +00:00
Carlos Goncalves
138b14b010 Install ca-certificate with redhat-common
Change-Id: I36d44ca8b4f966476657ec2bb1ecc1458bb524fd
2018-08-04 00:30:49 +02:00
Carlos Goncalves
cf553ce0ac Add netcat to redhat-common map-packages
Change-Id: I273038611febe5e30f30dd8d3ba8990dbdc94647
2018-08-03 17:20:19 +02:00
Zuul
53186f23a0 Merge "cache-url requires curl" 2018-07-31 00:20:56 +00:00
Oliver Walsh
a514feba99 Add DIB element to blacklist nouveau
This ensures nouveau is not loaded at boot, which is required when installing
NVIDIA GPU drivers and to avoid issues with PCI passthrough of NVIDIA GPUs.

The option to disable kernel modesets ensures that it can be unloaded again if
it happens to be loaded after boot (e.g manually or implicitly by X).

bp tripleo-vgpu

Change-Id: I60815de86e7b22dfb39555af9d2d53564841e2ab
Related-bug: 1774674
2018-07-25 16:35:58 +01:00
Oliver Walsh
73e27a8504 modprobe DIB_MODPROBE_BLACKLIST should be optional
modprobe element currently fails when DIB_MODPROBE_BLACKLIST is not set.
As there are now two methods to control blacklisting this should be optional.

Change-Id: Ibf3c31a95177ba88c1b93228490c7f36f5b70b57
2018-07-25 16:33:47 +01:00
Zuul
9adf12fe4a Merge "Fix for proper LVM support" 2018-07-24 07:32:11 +00:00
Clint Byrum
d8907e78b5 cache-url requires curl
In some cases cache-url can get pulled in without curl, causing it to
fail.

Co-Authored-By: Adam Harwell <flux.adam@gmail.com>

Change-Id: Ibd66c2ca4f8cc423783555d8a99b1184f43adff2
2018-07-23 09:56:58 +00:00
Zuul
b79952af2a Merge "Add new modprobe element" 2018-07-11 07:21:40 +00:00
Zuul
bbf69a90f3 Merge "Fix /etc/network/interfaces file contents" 2018-07-10 07:46:56 +00:00
Zuul
2343d4b577 Merge "Add keyring if supplied" 2018-07-10 07:46:55 +00:00
Olivier Bourdon
caf565673b Fix for proper LVM support
Without this fix, a LVM based ubuntu-minimal image will fail
booting due to the fact that the boot process will not be able
to retrieve the root filesystem using LABEL=(cloud)img-rootfs

Change-Id: If4ecf65868563f7b799160a58af6312bedf956bf
2018-07-09 14:15:57 +00:00
Hironori Shiina
7e4e6cfff4 Add expected semicolons for dhclient.conf
This patch adds an expected semicolon to an end of statement in
dhclient.conf for dhcp-all-interfaces element. Without this fix, an
error occurs when an image is booted with a message,
'semicolon expected.'.

Change-Id: I8311dbc67cc2815223111da01e7a7517c7d6f059
2018-07-06 13:42:25 +09:00
Sam Yaple
c144246cc9
Add keyring if supplied
When building with debootstrap, debootstrap will use the key to check
that everything is properly signed. It will not `apt-key add` the key
into the final environment, however.

Early adding the key after debootstrap before we need to read from the
private repo again prevents unsigned issues. This also maintains the
integrity of the packages in the environment throughout the build.

Change-Id: I5ca75ae4620c9fb26b512cb30f8cd79fa7a0373a
2018-07-02 14:33:35 -04:00
Zuul
927e8115f6 Merge "Fix bootloader for efi on rhel systems" 2018-06-28 15:02:30 +00:00
Yolanda Robla
31383970c7 Add new modprobe element
This element will replace modprobe-blacklist element. It wil
still have the blacklist functionality, but it also adds
the feature of passing a complete file with settings to the
modprobe.d directory. Adding this functionality, that will
allow elements that depends on this module, to just copy the
specified files to the final directory.

Change-Id: I9a44f7d11520b8b1e604956d3c1db2fc7e2bf457
2018-06-28 13:55:53 +02:00
Matthew Thode
b9f1c7a22f
better handle existing keywords files/directories
The existing directories are needed for stage building (a part of the
Gentoo build process).  Normally these directories are empty, but there
are times where overrides need to be defined.  This commit handles
existing overrides for keywords.  For historical reasons the overrides
were able to be put in different files and directories, this
centralizes them.

This also updates the version of openssl/cryptography that works with
or without bindist.

Change-Id: I62c934ed305a711a4a9a3ef01fa55ad142aebb78
2018-06-25 09:20:52 -05:00
Hoang Trung Hieu
a4648872ba Add iscsi-boot element for CentOS images
This patch adds an element that handles the configuration for
creating a disk capable of being a remote root filesystem through
iSCSI on CentOS images.

Tested on Fujitsu Server and boot with BIOS and UEFI mode successfully.
- Tested Boot-From-Volume + EFI for centos7 with following elements:
  "centos7 vm devuser cloud-init-datasources dhcp-all-interfaces
   iscsi-boot dracut-regenerate block-device-efi"

Co-authored-By: Nguyen Van Trung <trungnv@vn.fujitsu.com>

Change-Id: Ia1f23d722dced6f254fd7aee86abe8066a72fa42
2018-06-25 12:01:52 +00:00
Olivier Bourdon
11d91501d0 Fix /etc/network/interfaces file contents
According to http://bit.ly/2HA4oDO and
the official Ubuntu manual
http://manpages.ubuntu.com/manpages/xenial/man5/interfaces.5.html
source-dir support has been removed from Ubuntu >= 16.04/Xenial

Once an image is generated and booted, moving the dhcp interface(s)
declaration(s) from /etc/network/interfaces into specific subentries
of /etc/network/interfaces.d and calling 'service networking restart'
just make your instance unreachable and all interfaces are left
unconfigured.

This patchset fixes this issue

Change-Id: I6b6b99c81490c874c5db5405c2fbf3c180c87464
2018-06-19 11:26:21 +02:00
Yolanda Robla
61e6566c48 Fix bootloader for efi on rhel systems
When building the image on a non-efi environment, it generates
linux16/initrd16 entries. But to boot from UEFI they need to have
linuxefi/initrdefi entries.
Use sed to replace those entries, in case we have an EFI image.

Change-Id: I47c96450e10f34b91bcc32888532bd7ab87cf316
2018-06-19 08:37:30 +02:00
Zuul
e210f79500 Merge "Don't run setfiles on /boot/efi" 2018-06-15 08:42:13 +00:00
Nguyen Van Trung
8d86ff1bec Don't run setfiles on /boot/efi
setfiles isn't supported on the vfat /boot/efi partition.  Add it to
the skip list.

Tested on Fujitsu Server successfully.

Change-Id: Iab262c4bdb0ecc25ca6b77ee4aff1ce442c0c578
2018-06-15 14:53:38 +10:00
Michael Turek
91e3b72a23 Add iscsi-boot element
This patch adds an element that handles the configuration for
creating a disk capable of being a remote root filesystem through
iSCSI on Ubuntu and Debian images.

Change-Id: Ibf9e39d2bdab530106015f156d23d28029d12b0d
Closes-bug: #1716794
2018-06-14 08:56:03 +07:00
Yolanda Robla
bfc60958bf Fix bootloader packages for rhel
When using uefi in rhel, the package mapping is incorrect.
We need to add specific grub-efi* mappings to use grub2-efi

Change-Id: I2db96ae85fd5e4638c794015b2f8164c018420e3
2018-06-08 17:14:19 +02:00
Zuul
fc3748f49b Merge "Fix encoding issue during processing output" 2018-06-06 07:58:14 +00:00
Zuul
225c09b245 Merge "elements: pip-and-virtualenv: Handle openSUSE Leap 15" 2018-05-31 21:54:59 +00:00
Zuul
c4c2fb746c Merge "Remove duplicate GRUB command line entry" 2018-05-31 16:00:29 +00:00
Markos Chandras
f37e85d547 elements: pip-and-virtualenv: Handle openSUSE Leap 15
We need to handle openSUSE Leap 15 when installing pip and virtualenv
packages. This fixes the following problem when the pip-and-virtualenv
elements is used:

2018-05-31 09:42:12.014 | + [[ opensuse = opensuse ]]
2018-05-31 09:42:12.014 | /tmp/in_target.d/install.d/04-install-pip: line 57: packages: unbound variable

Change-Id: Id7911b0a0836fa8dcc003e23fa515b78fba67126
2018-05-31 10:55:28 +01:00
Zuul
4c04b46db6 Merge "Allow to rebuild arbitrary images" 2018-05-29 17:06:48 +00:00
Zuul
df9402c81c Merge "elements: zypper-minimal: Add support for openSUSE Leap 15.X" 2018-05-25 08:55:53 +00:00
Roman Gorshunov
84eea81efe Allow to rebuild arbitrary images
Patch allows to rebuild arbitrary images, which location, filename and
sha256sum are specified in variables, not only hardcoded $DIB_RELEASE/current.

Change-Id: I05418932a0c40d885fe00a49f1f49d7e86c67518
2018-05-24 10:19:59 +00:00
Zuul
668c93b118 Merge "Replace the ubuntu-minimal trusty test with a bionic one" 2018-05-21 06:01:53 +00:00
Ian Wienand
82eb1ca837 Replace the ubuntu-minimal trusty test with a bionic one
Add a bionic test in replacement of trusty.  We are already building
bionic images in the gate, so this seems like a good time to switch.

Change-Id: I20d4c25e9b79e7326c86767c36be8615ba0888a3
2018-05-21 12:51:31 +10:00
Roman Gorshunov
09af52a08c Remove non-maintained ubuntu-core element
Removing no longer working and no longer maintained ubuntu-core element, which
intent is unclear, and not documented.

Change-Id: Id847591d04fd7cd32c8903967da01ee0d303b267
Closes-Bug: 1771614
2018-05-18 09:28:02 -07:00
Markos Chandras
bb9ad09010 elements: zypper-minimal: Add support for openSUSE Leap 15.X
openSUSE Leap shares the same repo structure with 42.X

Change-Id: I23de11d81020c8aae641dfc01c1cbddf769f5c75
2018-05-18 11:55:56 +01:00
Stanislav Makar
2e6a19a018 Add Ubuntu 18.04 support
Use squashfs for more recent Ubuntu releases

Change-Id: I80df28be6e2a5e03ae1450e84fc05715f21a7750
Closes-bug: 1766850
2018-05-18 14:47:20 +10:00
Matthew Thode
e1f01d513a
IPA requires iptables
iptables -L is used in log collection

Change-Id: Id7e69a9e9f87db6621b928a2104df4b94f10044e
2018-05-17 11:14:44 -05:00
Doug Szumski
c1b1534c87 Remove duplicate GRUB command line entry
Without this change DIB appends a second command line entry to the GRUB
config. This causes the original command line entry to be ignored
when Linux is booted.

The expected behaviour is that DIB appends to the existing entry as
it does for Ubuntu and SUSE.

Following discussion on the review, this also removes the distro specific
switch statement, as update-grub just calls grub-mkconfig, meaning that
there was nothing distro specific in the first place.

Change-Id: I2298675dda1f699c572b3423e7274bc8bd7c1c9d
Closes-Bug: #1771366
2018-05-16 09:25:59 +01:00
Zuul
a8df61edbb Merge "rpm-distro: set the contentdir yum var" 2018-05-14 12:36:11 +00:00
Zuul
d79a7aaf01 Merge "Add pip cache cleanup to pip-and-virtualenv" 2018-05-14 09:03:49 +00:00
Zuul
1412e957ba Merge "Fixes add-apt-keys in dpkg element" 2018-05-14 08:52:44 +00:00
Tristan Cacqueray
25964c5c5b rpm-distro: set the contentdir yum var
Since CentOS-7.5, a new yum variable is needed for SIG repositories.
This change replicates the %post task of the centos-release package
to setup the contentdir yum var. This should fix issues when
repository url uses $contentdir and yum fail with:
http://mirror.centos.org/%24contentdir ... [Errno 14] HTTP Error 404 - Not Found

For more details see:

https://lists.centos.org/pipermail/centos-devel/2018-March/016542.html

This change also drops support for fedora without dnf.

Change-Id: I1819a48b94670577b0c5e29b24cebfb20ea07d28
2018-05-14 05:38:37 +00:00
XiaojueGuan
93b42f0f01 Trivial: update url to new url
Change-Id: I8d26fd0598626c8666a1852724f0620af9dc028d
2018-05-13 23:06:00 +08:00
Oded Le'Sage
10b2a5a4ee Fixes add-apt-keys in dpkg element
This commit addresses the issue described in bug 1768354 when using the
apt-sources element and adding a key to a custom repo, subsequent deb
package installs fail due no update of the repo before package install

Change-Id: I968b3422fab2fb2305426d49215391d8ba7499df
Closes-Bug: 1768354
2018-05-09 12:06:20 -05:00
Michael Johnson
7e79a933db Add pip cache cleanup to pip-and-virtualenv
The pip-and-virtualenv element provides pip inside the created images.
When this pip is used inside the chroot of the image it, by default, creates
a cache directory with the http packages and the resulting wheels.
In the case of the octavia ubuntu-minimal image this cache is using ~50MB of
cache that is not needed after the image finished building.

This package creates a finalise.d task that removes the cache from the
default location.

Change-Id: I4715437b068d04993ef755bd1e27963db1d22417
2018-05-09 09:31:17 -07:00
Matthew Thode
f7818e4a57
Install sudo on Gentoo images by deault
Change-Id: I30640017e3c652a467be421169029045a8b675d2
Close-Bug: #1766484
2018-04-24 11:44:39 -05:00
Tristan Cacqueray
abd63b01aa pip-and-virtualenv: fix install-pip when centos-release-openstack is enabled
When RDO projects repository is installed, the python-setuptools
package is obsoleted by python2-setuptools, this makes the install-pip
script failed:

  Package python-setuptools-0.9.8-7.el7.noarch is obsoleted by
  python2-setuptools-22.0.5-1.el7.noarch which is already installed

Then the "rpm -ql python-setuptools | xargs rm -rf" exit 1.  Check if
we have a record of the updated package obsoleting then old one; if
so, use it.

Change-Id: I2b0051bd9e81908c187098a7b82e120b999b111d
2018-04-23 08:18:04 +10:00
Paul Belanger
99eb9a2d7d
Fix epel element for centos-minimal
We no longer install wget / yum-utils for centos-minimal, this fixes
that.

Change-Id: I8d89026bd48cf7398cc1cbe41e3b7f00f682dbb8
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2018-04-18 20:15:07 -04:00
Paul Belanger
12a760f7b7 Revert "debootstrap: Call update-initramfs explicitly"
This currently breaks glean on rackspace, revert until we can figure
out why that is.

This reverts commit 43bc352c59.

Change-Id: Iae88a3b0457bab0b8f0fd1febf58732ca95e5dc9
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2018-04-17 21:47:46 +00:00
Ian Wienand
b423292cd0 Remove installed packages before pip install
The release of pip10 has shown up a few issues here

Firstly, pip10 now refuses to overwrite distutils installed packages,
which includes "python-virtualenv" on centos.  History has shown us
that we want the packages installed and overwritten, to avoid the
packages coming back and messing things up.

Pre-install all the packages, then list the files in the packages with
"rpm" directly and remove them.  This way pip is happy to install.

We need to take better account of the package names for this; on
Fedora things have switch to "python2-virtualenv" instead of
"python-virtualenv" and we can't use an alias to list the package
contents.

This also highlighted that python2-pip is in EPEL for centos, so
enable that when we install it.  Make the epel element a no-op for non
centos/rhe distros.

There is a related change in recent fedora that python3 now installs
binaries into /usr/local/bin.  There are commented swizzles in here to
ensure we retain the status quo of "pip" and "virtualenv" both being
python2 based, with the python3 versions being called explicitly
"pip3" and "virtualenv3" respectively.

Change-Id: I2ffdd9f615ae6b00428c17249e4f216774991b99
2018-04-17 16:09:04 +10:00
Matthew Thode
2dd6dd357a
allow building non-gentoo images on gentoo hosts
Can't use a variable that's not set.

Change-Id: I4a7cf2ac47c2254da2fec778437f67c1fe3707f7
2018-04-12 13:24:59 -05:00
Zuul
0769bfd4aa Merge "Set the dhclient timeout to match DIB_DHCP_TIMEOUT" 2018-04-11 10:12:45 +00:00
Ian Wienand
f52b385818 Don't only install python3-virtualenv
We added this sed in I422490ebe9a9c655552685bc2ff342d288335a9c to
avoid installing python2 packages on python3-only systems and thus
dragging in all of python2.

We made a similar change to python-pip in
I7d8ba9300039cce90965410a4e16ca9e711904c3; however we realised that
the gate (and other consumers) were relying on this element having
installed the python2 & 3 packages for consistency -- otherwise jobs
would install the python-pip packages and overwrite the
pip-from-source and mess everything up.  We reverted that in
I419dbdf4682394db68974944af1e5c432f3e0565 and added some clearer notes
that this element brings in python2 & 3, and if you want something
that doesn't do that then this element isn't for you.

However, we never fixed up the virtualenv package install -- currently
our Xenial images have a global virtualenv installed from source, but
the python-virtualenv packages aren't installed.  Thus if a job does
"apt-get install python-virtualenv" it overwrites the from-source
virtualenv with older parts and again messes everything up.

Probably most jobs just call "virtualenv" and assume it is there;
however in bringing up some rspec test for puppet I have hit this
issue as some modules specify dependencies on the virtualenv packages.

Thus install the python-virtualenv AND python3-virtualenv packages in
this element.

Change-Id: Ia84c38dc3c40a6080e144b563e10abca7dac2881
2018-04-10 12:34:03 +10:00
Derek Higgins
97399e9bb1 Set the dhclient timeout to match DIB_DHCP_TIMEOUT
On initial boot when networking is brought up by cloud-init this
is the timeout that dhclient adheres to. Centos configures
"timeout 300" (for an EC2 bug) in their cloud image, which results
in a 5 minutes delay to boot in cases where no dhcp available (e.g. IPv6
SLAAC). To reduce this boot delay and to provide consistency with
places where we have set other dhcp timeouts set this to DIB_DHCP_TIMEOUT.

Change-Id: I119a002070501c3dfe7c6730b07ee25f422b85b0
Related-Bug: #1758324
2018-04-05 14:29:22 +01:00
Zuul
09e5b2d357 Merge "debootstrap: Call update-initramfs explicitly" 2018-03-29 06:02:58 +00:00
Zuul
c07e7349d5 Merge "Change the GENTOO_PORTAGE_CLEANUP variable default" 2018-03-29 05:33:55 +00:00
Zuul
e8744cf32b Merge "zypper-minimal: Set default locale env to C.UTF-8" 2018-03-29 05:33:52 +00:00
Zuul
e444de83e6 Merge "Clean up dib-python symlink" 2018-03-29 05:33:51 +00:00
Andreas Florath
43bc352c59 debootstrap: Call update-initramfs explicitly
Many elements install additional distribution packages.
In addition the user can provide a set of packages to be installed
via the '-p' switch.
Some of them influence the boot process and therefore the initramfs
needs to be updated. Because the package manager during the image
creation process is configured not to run package scripts, this needs
to be done explicitly.

This issue was found during development and debugging of the
block-device LVM plugin: Even when the e.g. the lvm2 package
was installed in the image, it was missing in the initramfs
because of the missing update.

Change-Id: I7c92033b3ca80cdd23d081002059d83ca3f53bdb
Signed-off-by: Andreas Florath <andreas@florath.net>
2018-03-29 04:14:52 +00:00
Matthew Thode
6a6d78e63c
Change the GENTOO_PORTAGE_CLEANUP variable default
Default the GENTOO_PORTAGE_CLEANUP to True.  By default we should not
ship package info, this bloats the image and is usually outdated by the
time it'd be consumed.

Change-Id: I14c2530d91807cbc6a3806e01c7e4f6f472b190d
2018-03-26 23:26:49 -05:00
Clark Boylan
301eac8e8b Fix element-provides in debian element
The debian element depends on debian-minimal now which provides
operating-system. This means that the debian element can no longer
provide operating-system and doing so results in an error when using the
debian element.

The fix is simple just rely on the fact that debian-minimal provides
operating-system and remove this element-provides from debian.

Fixes-Bug: 1758000
Change-Id: I524feeb82c19046ec987eb1186c7f4568309e559
2018-03-26 10:58:04 -07:00
Zuul
ffc06874ef Merge "install sudo in the devuser element" 2018-03-26 00:58:57 +00:00
Zuul
8ab38dc5cf Merge "Update Fedora defaults to 27" 2018-03-25 23:41:45 +00:00
Zuul
9d751463f1 Merge "enable systemd profile for Gentoo" 2018-03-23 19:01:35 +00:00
Zuul
45ff8175b6 Merge "proliant-tools: add net-tools package to support hpsum utility" 2018-03-23 05:36:12 +00:00
Matthew Thode
cfa7935e43
enable systemd profile for Gentoo
Change-Id: Id3ac1d97b280f10f9938a60c4871d08f59b85002
2018-03-22 15:12:59 -05:00
Matthew Thode
cfa5b237c0
install sudo in the devuser element
The devuser element can set up passwordless sudo, which requiers the
/etc/sudoers.d directory, which requires the sudo package, so we ensure
the sudo package is installed.

Change-Id: I80d6c669d4ac0d97b49d01cb621bf05b8e7f8ef1
2018-03-22 00:16:09 -05:00
Matthew Thode
5e2f3646ad
remove portage git directory
this shrinks thinks by ~50%, from 722M to 352M

Change-Id: I1267cc05700ee28c45a331de7f571b9ee075c6b5
2018-03-16 18:29:40 -05:00
Zuul
651d913fcc Merge "arm64: use HWE kernel and fix console" 2018-03-16 08:31:14 +00:00
Zuul
5e2168aefd Merge "Choose appropriate bootloader for block-device" 2018-03-16 08:31:12 +00:00
Zuul
96af247400 Merge "Add block-device defaults" 2018-03-16 08:31:10 +00:00
Zuul
c60a20b59d Merge "GPT partitioning support" 2018-03-16 08:14:27 +00:00
Anshul Jain
a7135a0d8f proliant-tools: add net-tools package to support hpsum utility
Hpsum utiltity of proliant-tools requires net-tools to be installed
as part of base image. This commit adds support for installation of
net-tools for all distros.

Change-Id: I2a1e81059ed1aee975db78cfa5e61bbf1b98e06f
Closes-bug: 1751777
2018-03-09 02:29:23 -06:00
Zuul
30d5ee2ec7 Merge "Fix for passing user defined value for satellite cert for rhel-common." 2018-03-08 07:05:40 +00:00
Zuul
dda4e2e0f1 Merge "secondary architectures use different url" 2018-03-08 07:05:39 +00:00
Zuul
d2133ec5e7 Merge "Fix for rhel7 iso image creation." 2018-03-08 07:05:38 +00:00
Mark Hamzy
cdb423eeb9 secondary architectures use different url
The Fedora-Cloud-Base*qcow2 images are stored on a different
server for secondary architectures.

Change-Id: I90d48ce4175fd251e8f5ab7a70190ad952256a94
2018-03-01 19:38:14 -06:00
Tobias Henkel
b62ed1823c
Fix encoding issue during processing output
When using the package-installs element there can be some encoding
problems if the package installation emits unparsable output
[1]. However in this case we just want to forward the output to the
console which normally can handle this correctly. In order to fix this
switch off universal_newlines processing such that we just operate on
bytes.

Further we have to decode the lines without setting the locale and
ignoring errors. This is required because print encodes without
setting the locale and thus we need to filter/modify the stream such
that it doesn't crash.

[1] Traceback:
2018-03-01 09:58:00.515 | Traceback (most recent call last):
2018-03-01 09:58:00.515 |   File "/usr/local/bin/package-installs-v2", line 137, in <module>
2018-03-01 09:58:00.515 |     main()
2018-03-01 09:58:00.515 |   File "/usr/local/bin/package-installs-v2", line 130, in main
2018-03-01 09:58:00.515 |     process_output(install_args, follow=True)
2018-03-01 09:58:00.515 |     for line in iter(proc.stdout.readline, ''):
2018-03-01 09:58:00.515 |   File "/usr/lib/python3.5/encodings/ascii.py", line 26, in decode
2018-03-01 09:58:00.515 |     return codecs.ascii_decode(input, self.errors)[0]
2018-03-01 09:58:00.515 | UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 34: ordinal not in range(128)

Change-Id: Ie4af9b4523459a630cfb98d09093bfe9ef7aa61e
2018-03-01 16:09:25 +00:00
Anshul Jain
73a9ba0f1f Fix for rhel7 iso image creation.
Currently rhel7 image creation fails because it tries to copy
default bootloaders which is ubuntu way. This commit updates `iso`
element to correct the path of bootloaders required for rhel image.

Change-Id: I526d75b2db609fc77be0fc778b4d00f2d3df38ec
Closes-bug: 1750725
2018-02-28 06:31:34 -06:00
Zuul
3fb0950808 Merge "update Gentoo Hardened profiles (now stable)" 2018-02-28 11:42:51 +00:00
Zuul
05a44e2181 Merge "Checking link status according to DIB_DHCP_TIMEOUT" 2018-02-28 11:42:50 +00:00
Anshul Jain
b37a1e27cb Fix for passing user defined value for satellite cert for rhel-common.
For 'satellite' mode of registration, rpm for rhel SSL certificate is
hard coded to 'katello-ca-consumer-latest.noarch.rpm'. This commit adds
functionality that provides an option to set this as defined in their
satellite server.

Change-Id: Ib176cfa209f5ac8a4b5da71419327b4237330904
Closes-Bug: 1749947
2018-02-28 02:43:39 -06:00
Ian Wienand
e9ed983324 arm64: use HWE kernel and fix console
Install hwe kernel for ubuntu-minimal.  As noted this is currently
Xenial specific; we need this for initial bring-up so let's tackle
future releases as things progress.

Ensure we use ttyAMA0 for arm64 console too.

Change-Id: Ic607cf8369666dc24929aff6f2ef8a72e7980599
2018-02-23 10:04:48 +11:00
Ian Wienand
7b4c8abce3 Choose appropriate bootloader for block-device
In the prior change we added block-device-[mbr|gpt|efi] elements to
create appropriate disk-layouts.

This adds an environment flag to each so the bootloader can install
the right thing.  The EFI install path is updated to work with this
(this part a copy of I572937945adbb5adaa5cb09200752e323c2c9531)

We do some basic sanity checking in the block-device elements;
e.g. mbr is not suitable for aarch64, and efi is not suitable for
power.

This updates the bootloader to install EFI where appropriate

Co-Authored-By: Marcin Juszkiewicz <marcin.juszkiewicz@linaro.org>
Change-Id: Ib80acbfd9a12efd976c3fa15a5d1081eb0799305
2018-02-23 10:04:44 +11:00
Ian Wienand
adb0341064 Add block-device defaults
This moves the block-device default out of the "vm" element and into a
selection of other elements.  There's "mbr" which retains the status
quo.  There's an EFI version that has the boot/grub partitions as
required.  In between there's the GPT only version, which is useful
for architectures like power without EFI, but still want possible
larger disks using GPT.

Change-Id: I4a566a97d073fc0dda0ab2494ac988fe015800a9
2018-02-23 10:04:40 +11:00
Ian Wienand
55b479b54f GPT partitioning support
This adds support for a GPT label type to the partitioning code.  This
is relatively straight-forward translation of the partition config
into a sgparted command-line and subsequent call.

A unit test is added based on a working GPT/EFI configuration and the
fedora-minimal functional test is updated to build a single-partition
GPT based using the new block-device-gpt override element.  See notes
in the sample configuration files about partition requirements and
types.

Documentation has been updated.

Co-Authored-By: Marcin Juszkiewicz <marcin.juszkiewicz@linaro.org>
Change-Id: I6b819a8071389e7e4eb4874ff7750bd192695ff2
2018-02-23 10:04:26 +11:00
Matthew Thode
6b3c22dd75
update Gentoo Hardened profiles (now stable)
Change-Id: I6d89f27bfd62fc0e86fec1a0bc6cb37f9ec6c82f
Signed-off-by: Matthew Thode <mthode@mthode.org>
2018-02-21 11:06:16 -06:00
Lenny Verkhovsky
f249cec9f3 Checking link status according to DIB_DHCP_TIMEOUT
In slow networks like Infiniband it takes much time for the
interface to get the carrier. This patch enables this service
to run more then 20 seconds and limited by DIB_DHCP_TIMEOUT.

Change-Id: I8a6015567ac25e37b5a5aba4b1fda71170cc144a
2018-02-21 12:52:57 +00:00
Zuul
e8eb291f57 Merge "update gentoo vars for new profile and python" 2018-02-20 05:26:20 +00:00
Paul Belanger
e9e7ac2ee1
Install systemd earlier for Ubuntu Bionic
Like we did in https://review.openstack.org/475206 we need to install
systemd sooner because of the new world order of containers.

Change-Id: Ia60d751fee3af6f8d72ad664107acb337360feca
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2018-02-17 14:32:35 -05:00
Matthew Thode
bfee8bf094
update gentoo vars for new profile and python
Change-Id: I810b09f98c479e6ebdbf8de6fae31166a3e87667
2018-02-13 17:02:11 -06:00
Zuul
2080055155 Merge "Create rescue user on ironic agent" 2018-02-05 22:43:00 +00:00
Zuul
cb599b13de Merge "upgrade pip before using -c option" 2018-02-05 22:34:00 +00:00
Zuul
66444e8b93 Merge "Add support for Fedora 27, remove EOL Fedora 25" 2018-01-29 05:23:28 +00:00
Tim Flink
f8bcc51b55 Don't install dmidecode on Fedora ppc64le
While Debian-based distros use the label of ppc64el for ppc64 little
endian, Fedora uses ppc64le.

The ironic-agent was doing arch specific package install of lshw over
dmidecode for ppc64 and ppc64el but was attempting to install dmidecode
on Fedora ppc64le which caused the test to fail due to a missing
package.

This change just adds ppc64le to the arch-specific package installation
description for the ironic-agent element.

Change-Id: I38c3c1480bbbb2df817856614e6b740a0c02723a
Closes-Bug: 1744944
2018-01-29 10:53:43 +11:00
Zuul
9bfa45f0c8 Merge "Remove architecture rules on lshw dependency in ironic-agent" 2018-01-28 22:07:53 +00:00
Tim Flink
490cf3aa49 Add support for Fedora 27, remove EOL Fedora 25
This updates diskimage-builder to support current Fedora releases (26
and 27) and removes support for Fedora 25 which is EOL as of December
12, 2017.

Change-Id: I227a607c6c468cc8b7bb154a189e9c8ce2021192
2018-01-23 11:31:21 +00:00
Mark Hamzy
34ff72f253 upgrade pip before using -c option
The installed pip can be an older version which does not support
the -c argument. Therefore, upgrade pip before using -c.

Change-Id: If18d8ea822a62c8551c9c4d47354d58b0299fed2
Closes-Bug: 1744403
2018-01-19 16:46:59 -06:00
Dirk Mueller
4858340b42 Add SUSE Mapping
Change-Id: I8bc12435c62ef7c9c3fa8e21e00738698a532f56
2018-01-11 19:00:40 +01:00
Ian Wienand
e9df83b2b3 Revert "Dont install python-pip for py3k"
This reverts commit ab89c7d69c.

This commit checked for DIB_PYTHON_VERSION and only installed the v3
packages.  This is unfortunately backwards-incompatible, as consumers
such as the openstack gate are relying on this package installing pip
& virtualenv packages for python2 AND python3.

This was sort-of expressed in the docs, where it discusses what the
resulting setup of the system will be, but I've added a note to make
it clearer.

If we want to change this, I think we'll need either a new element, or
a non-defaulting flag.

Change-Id: I419dbdf4682394db68974944af1e5c432f3e0565
2018-01-10 15:48:03 +11:00
sayalilunkad
acc44c0651 Adding mapping for SUSE package
Mapping PyYAML to python-PyYAML as in opensuse.

Change-Id: I54864c96afdb20975bf1e3ac9a117d5263f21f3b
2017-12-20 17:51:33 +01:00
Ian Wienand
af50200863 Update Fedora defaults to 27
Update the Fedora defaults to now released F27

Change-Id: I7655ad010dfa668bd68b7a619d85e5f694f96806
2017-12-07 13:59:51 +11:00
Ian Wienand
dcbb7e7ebb ironic-agent: don't remove make
It turns out make has always been a tacit dependency of openssl as it
ships a Makefile for certificates [1].  This just recently changed to
be a hard dependency in F27, so this now fails as openssl is a
dependency of protected packages such as dnf.  Since it's always been
wrong to remove it, we take it out of the purge list.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=783446

Change-Id: I69efb3a56878ab97c4587bbbf5356bea752f2846
2017-12-07 13:59:51 +11:00
Michael Turek
7054a71f7d Remove architecture rules on lshw dependency in ironic-agent
There's a patch in flight in ironic-python-agent to switch the
default hardware manager to use lshw instead of dmidecode. [0]
This would require lshw to be installed regardless of
architecture. This patch removes the architecture rules from
lshw in the package-installs list.

[0] Ie370331df6bb5ef131c5cb60f458877e2a7ad71a

Change-Id: Idaf05b8efce28cd0cbf339cf693db4f55a693d9b
Partial-Bug: #1715790
2017-12-06 17:02:46 -05:00
K Jonathan Harker
7470ee26e0 zypper: fix package removal
zypper only supports the --no-recommends option during installs, giving
the option during removals results in an error.

When setting ACTION=remove, remove --no-recommends from EXTRA_ARGS, and
set --clean-deps to also remove no-longer-needed dependencies.

Rename EXTRA_ARGS to ACTION_ARGS for increased readability.

Change-Id: Ifbd168992b1a20658b6b4a99ba175234f6c78f6d
2017-12-05 22:59:20 +00:00
Zuul
7beb27ca30 Merge "Add zipl element as s390x architecture bootloader" 2017-12-01 02:35:58 +00:00
Zuul
6ab9655ca3 Merge "Fix /dev/pts mount options handling" 2017-12-01 02:01:46 +00:00
goldyfruit
c247cb41bb Fix wrong epel-release-7* package URL
When "epel" element is used during a build process
with "rhel7" distribution, the build failed
because the "epel-release-7*" package cannot be
installed.

The reason is because the URL is not correct, it
should be:
  URL=$BASE_URL/$RELEASE/x86_64/Packages/e/

Change-Id: I90c26892361f7611645b85f2eddc949b2f0d76fc
Closes-Bug: #1735547
2017-11-30 15:31:10 -05:00
Zuul
bfd61258ba Merge "Add the groundwork for musl profile support" 2017-11-30 09:31:40 +00:00
Zuul
5677a30a5a Merge "Pre-install curl" 2017-11-30 08:16:14 +00:00
Zuul
8182acb0fc Merge "Install fedora-gpg-keys for F27" 2017-11-30 07:54:38 +00:00
Matthew Thode
7223f2ce20 Add the groundwork for musl profile support
At the moment all musl needs in addition to an official stage4 file is a
few keywords and use flag changes.

Change-Id: Ibf4a6d616aca1aef876967e2aa34170c96ac9ef8
2017-11-30 18:17:21 +11:00
Zuul
6a28810ad8 Merge "Enable support for Gentoo overlays" 2017-11-30 06:40:43 +00:00
Zuul
87392cf1c8 Merge "Make preinstall.d more deterministic" 2017-11-30 05:50:03 +00:00
Matthew Thode
c4f83f2311 Enable support for Gentoo overlays
This is intended to eventually support building musl-libc based images,
which need the musl overlay.

Change-Id: I8f5429ffa64e74c860772d9a00ff0b7eebb7721a
2017-11-30 05:14:00 +00:00
Zuul
bbbe762dff Merge "elements: zypper-minimal: Refresh repositories where necessary" 2017-11-30 04:36:06 +00:00
Ian Wienand
1b203f8a38 Pre-install curl
As described, Fedora 27 has a curl-minimal package that comes in to
satisfy the rpm package dependency.  It conflicts with the "real" curl
package -- which is so commonly installed (by infra elements, etc)
that this becomes an annoying problem.  Just pre-install the full curl
package.

Fedora 24 is old enough to not worry about, so remove some old
workarounds to make the flow a little simpler.

Change-Id: I67baf96377109ac4521ba00243a0d91b35fafba0
2017-11-30 15:15:42 +11:00
Ian Wienand
bf8de79940 Install fedora-gpg-keys for F27
The repo GPG keys moved into a separate package [1] which now needs to
be installed.

Since the fedora-release/fedora-repos split is *long* since over,
remove that work-around and add this one.

[1] https://pagure.io/fedora-repos/c/f69f3729511c3eba5f470b1d90ea2bfee372eb29?branch=f27

Change-Id: I9ad28d5bdb78375ae21dbb16e2d8c4effb32cb35
2017-11-30 15:15:37 +11:00
Matthew Thode
c886c4cbe1 Make preinstall.d more deterministic
Reorders the preinstall, making the order more explicit.  Also dedupes
some folder setup.

Change-Id: I423dcba169558ff6037a3382b997675722e77405
2017-11-30 13:48:18 +11:00
Zuul
f74e48799d Merge "Enable gentoo in pip-and-virtualenv element" 2017-11-30 02:10:09 +00:00
Zuul
247c68b5a3 Merge "Clear /etc/machine-id to avoid duplicate machine-ids" 2017-11-30 01:56:26 +00:00
Zuul
d01d3d8832 Merge "Make python changes more reliable" 2017-11-30 01:49:07 +00:00
Zuul
71ca627d30 Merge "Add debian minimal requirement for arm64" 2017-11-29 23:47:12 +00:00
Zuul
c146d2f3b7 Merge "Fix grub2 dependency on arm64" 2017-11-29 23:47:10 +00:00
Andreas Florath
46a07de480 Fix /dev/pts mount options handling
The current implementation - as introduced in
Iee44703297a15b14c715f4bfb7bae67f613aceee - has some shortcomings / bugs,
like:

* the 'grep' check is too sloppy
* when /dev/pts is already mounted multiple times the current implementation
  fails:
  $ mount | grep devpts | sed 's/.*(\(.*\))/\1/'
  rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000
  rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000
  rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000
* code duplication
* Using the undocumented and non-robust output
  of 'mount'.

This patch fixed the above problems.

Change-Id: Ib0c7358772480c56d405659a6a32afd60c311686
Signed-off-by: Andreas Florath <andreas@florath.net>
2017-11-24 07:14:56 +00:00
Matthew Thode
ade82904a1
Make python changes more reliable
We oneshot emerge without calculating dependencies a few things to solve
for possible dependency loops.

Python 3.5 also became stable, so don't need to do special things for
it.

Matched the uninstall with the install lines (no need for a full if
statement).

Change-Id: I7c5e546612ac47d659e73a46a52e34d39ca81949
2017-11-20 23:19:46 -06:00
Zuul
b879a03c75 Merge "Dont install python-pip for py3k" 2017-11-20 05:38:24 +00:00
Markos Chandras
7f051add61 elements: zypper-minimal: Refresh repositories where necessary
We should always refresh the Tumbleweed repositories and the 'update'
one for Leap in order to always have the latest information from the
repositories.

Change-Id: I85db9d8bb7fa153f01222129e9b36fecc2632f57
2017-11-16 16:54:47 +00:00
Markos Chandras
da02f37de1 elements: Respect devpts mount options
This is a continuation for f2cc647dae ("diskimage_builder: lib:
common-functions: Fix options for devpts mount"). We also need to
respect the devpts mount options when the dib elements are mounting
this virtual filesystems themselves.

Change-Id: Iee44703297a15b14c715f4bfb7bae67f613aceee
2017-11-14 08:31:55 +00:00
Gregory Haynes
ab89c7d69c Dont install python-pip for py3k
We want to install python3-pip, not python-pip when we are building a
py3k image less we pull in python2. Once we stop installing python2 we
have to stop calling python2 during pip install.

Change-Id: I7d8ba9300039cce90965410a4e16ca9e711904c3
2017-11-13 23:00:52 +00:00
Adam Harwell
d4fd7f1217 Enable gentoo in pip-and-virtualenv element
Currently it will hit the `else` and try to apt-get, which fails.

Change-Id: I951882cf3897ced165e167f12877c05ee62a5054
2017-11-11 09:00:50 +09:00
Dirk Mueller
4e258bdad6 zypper-minimal: Set default locale env to C.UTF-8
Currently in Leap 42.x the bootup scripts don't actually make use of
locale.conf yet, so we need to set it in /etc/sysconfig/language. For
future distro compatibility the setting in locale.conf is kept in sync.

Also fix default timezone link.

Change-Id: I59e5dccad8a5ae132d3039851e7aa1db86a609d7
2017-11-10 16:09:05 +01:00
Zuul
bc6c928bb9 Merge "Move to a common lock-file directory" 2017-11-07 17:32:28 +00:00
Zhiguo Deng
271dc36f33 Add zipl element as s390x architecture bootloader
s390x architecture uses zipl as bootloader. When used in combination
with the vm element it replaces the existing bootloader element.
It's mandatory for s390x vm images.

Use cases
---------

* Allow users to create s390x images that run on nova with s390x
  libvirt/kvm backend
* Building nodepool images for s390x third party CI

Supported Distros
-----------------
The following listing shows all Distros that officially support
s390x and how those Distros are supported in DIB with this patch.

* SLES - not supported (SLES is not supported in DIB)
* RHEL - not suppoprted (RHEL is not supported as KVM guest on s390x,
                         therefore there's no rhel7 qcow image for s390x available
                         like it is for other archictectures)
* Ubuntu - supported

Ubuntu images can for example be built using the following commands:

  $ disk-image-create ubuntu-minimal zipl vm
  $ disk-image-create ubuntu-minimal zipl
  $ disk-image-create ubuntu zipl vm

Testing
-------

Cross architecture building of s390x images is not supported so far.

The plan is to set up a ThirdParty CI that builds the image for s390x and
provides the logs.

Co-Authored-By: Andreas Scheuring <andreas.scheuring@de.ibm.com>
Co-Authored-By: Holger Smolinsky <holger@smolinski.name>
Co-Authored-By: Zhiguo Deng <bjzgdeng@linux.vnet.ibm.com>
Co-Authored-By: Arne Recknagel <arne.recknagel@hotmail.com>

Closes-Bug: #1730641

Change-Id: I576e7edda68da12e97c60af38f457915efe7b934
2017-11-07 17:19:27 +01:00
Zuul
906a3f4a57 Merge "Use -t devpts for /dev/pts mounts" 2017-11-02 14:20:05 +00:00
Zuul
1b0631da84 Merge "Update proliant-tools to support Gen10 Proliant servers" 2017-11-01 07:22:03 +00:00
Ian Wienand
b25d0337b8 Move to a common lock-file directory
In a couple of places we use flock for critical sections, but we leave
lockfiles around in various locations which can be confusing.

Introduce DIB_LOCKFILES global (under ~/.cache/dib/lockfiles) and
write lockfiles in there.

Fix up removal of the lockfile in the yum path; we just want to make
sure we cleanup the .rpmmacros file, but we don't need to remove the
lockfile as well.

Co-Authored-By: Andreas Florath <andreas@florath.net>

Change-Id: Ie810b2836be521325afe923708d046112e1e1e20
2017-10-26 16:27:59 +11:00
Zuul
8f025691ba Merge "Dont install python-virtualenv for py3k in deb" 2017-10-24 06:33:45 +00:00
Zuul
c5f713b1ec Merge "Change to install a package in 'proliant-tools'" 2017-10-24 05:48:25 +00:00
Yolanda Robla
ba11376328 Create rescue user on ironic agent
Create a new service, that will be launched after ironic
agent has been exited. This will launch an script that will
take the rescue password, and create the rescue user with
that credentials.

Depends-On: I7898ff22800dedba73d7fbfb3801378867abe183
Change-Id: Ic3a241e2789a122d3d966e7e2148306fd0cf6aed
Partial-Bug: 1526449
2017-10-23 12:50:32 +00:00
Andreas Florath
cebfcf85f9 Use -t devpts for /dev/pts mounts
Currently a bind is used when mounting /dev/pts in chroot.
This leads to problems - especially when running DIB in parallel:
It was observed that the /dev/pts mount vanishes from the host
system.

This patch uses '-t devpts' - as it is done for /sys and /proc -
for handling /dev/pts.

Change-Id: Id7775ae6fca6502af800e7b73a00862ef320206b
Signed-off-by: Andreas Florath <andreas@florath.net>
2017-10-23 07:13:02 +00:00
Gregory Haynes
00d7c619e9 Dont install python-virtualenv for py3k in deb
On ubuntu we detect that in python3 we need to install
python3-virtualenv, but append this to the packages to install rather
than replace python-virtualenv which results in both being installed
(and therefore grabbing python2).

Change-Id: I422490ebe9a9c655552685bc2ff342d288335a9c
Closes-Bug: #1724656
2017-10-18 23:11:55 +00:00
Zhangfei Gao
6cc155fd66 Add debian minimal requirement for arm64
Debian system building fails, fixed by adding arm64 package.

DIB_RELEASE=jessie disk-image-create debian ironic-agent \
    grub2 devuser -a arm64 -o deploy-jessie
dib-run-parts Running /tmp/dib_build.v5FEtaKx/hooks/cleanup.d/99-extract-kernel-and-ramdisk
ls: cannot access /tmp/dib_build.v5FEtaKx/mnt/boot/vmlinu*: No such file or directory

Change-Id: I610d767785df49fed954f12854be5ae78ff9baa6
Signed-off-by: Zhangfei Gao <zhangfei.gao@linaro.org>
2017-10-16 13:39:50 +08:00
Zhangfei Gao
a8635d3a3b Fix grub2 dependency on arm64
Unable to locate package reported when
DIB_RELEASE=jessie disk-image-create debian ironic-agent \
grub2 devuser -a arm64 -o deploy-jessie

E: Unable to locate package grub-pc-bin
E: Unable to locate package shim-signed
E: Unable to locate package grub-efi-amd64-signed

Fix the issue via adding arch dependency and arm64 packages

Change-Id: I40650a887b575a9c2b00a8c5036c35354d548673
Signed-off-by: Zhangfei Gao <zhangfei.gao@linaro.org>
2017-10-16 13:39:34 +08:00
Matthew Thode
e29f031bec
Update Gentoo element for element changes
There have been a few changes over the past few months, here we make the
following changes.

* change from backtrack=99 to complete-graph as a more correct flag
* make python version selection more in line with what gentoo supports
* set up python before stuff gets pip installed
* ensure we have the proper pip so we can install pip packages as root
* ensure we have the proper use flags for the disk formatting changes
* set DIB_RELEASE like other distros
* fix openssh-server element for gentoo

Change-Id: I17202de3016616ce34c8cbead7d0fb047a64e96b
2017-10-08 12:02:46 -05:00
Anshul Jain
afb7084a4d Update proliant-tools to support Gen10 Proliant servers
This commits make update to ssacli version to point to latest
ssacli release that has support for HPE P/E-Class SR Gen10 controllers.

Change-Id: Ia9a0eaec78d601f56b4036e57601554b87f21acc
Closes-Bug: 1721185
2017-10-04 07:02:07 +00:00
Ian Wienand
df00e9adcb Add initramfs-tools for ubuntu-minimal
A small update was made to 4.4.0-96.119 that dropped the
initramfs-tools dependency from the kernel [1].  This had the
unfortunate affect of removing the initramfs from ubuntu-minimal and
making it unbootable, since we specify the root device via LABEL=.
Add the package explicitly alongside the kernel.

Also, small fix to pass unit tests

[1] https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/1700972

Change-Id: I57a0f08cd5e082ecdf8dba0ab34fb3062c50836d
2017-09-21 10:42:11 +02:00
Aparna
7bc2b23290 Change to install a package in 'proliant-tools'
This commit adds change in 'proliant-tools' element to
install a package 'unzip' which is required to perform
SUM based firmware update for HPE Proliant servers.

Change-Id: Ib8f6d18402439edd93d100cc7a4fb2094c863715
2017-09-18 08:04:34 +00:00
Jenkins
22e03f9820 Merge "Add missing package dependency for yaml" 2017-09-15 13:52:35 +00:00
Jenkins
254875dbde Merge "Create /etc/machine-id for fedora" 2017-09-15 04:05:51 +00:00
Jenkins
6e266399fb Merge "Use latest Fedora .qcow URL" 2017-09-15 02:36:24 +00:00
Ian Wienand
768c5e188c Create /etc/machine-id for fedora
As described in the comment, we need to create the /etc/machine-id for
the image-based build when systemd isn't updated (as is usually the
case for a new distro)

Work on clearing this out continues, but this brings it to parity with
fedora-minimal.

Change-Id: Icbbbabb4114d4d95909648d8e39a6bae6d2a7b7b
Depends-On: I761e425f8a658669d9b8a70ce4260cec263ea51a
2017-09-15 11:54:01 +10:00
Ian Wienand
7774260b76 Use latest Fedora .qcow URL
The URL we are using seems to have disappeared.  Update this to
download.fedoraproject.org.  The new URL requires a "subrelease" now,
add it, along with a note on where it comes from.

Change-Id: I761e425f8a658669d9b8a70ce4260cec263ea51a
2017-09-15 11:06:22 +10:00
Yolanda Robla
da5c926fe9 Add missing package dependency for yaml
This element was assuming that yaml was included as package,
but there are systems not including it. So properly add yaml
as a dependency.

Change-Id: I72da2776674a3963657052b9a9715abcb4fab1e2
Partially-Fixes-Bug: #1715686
2017-09-13 14:16:41 +02:00
Yolanda Robla
3ff8d1e10b Move the ordering of the dracut regenerate command
When using combined with rhel7 image, the unregister of repos
has already happened, because it is executed under 60- ordering.
As dracut-regenerate may need to install extra packages for it,
it causes this step to fail, because it cannot find repos where
to pull the packages from.

Change-Id: I35e37df7990ad76a5004cb90fdd863ec743a5483
2017-09-06 12:52:05 +02:00
Ben Nemec
72d0d22cdf Remove nested quotes from TAROPTS
Per the bug report, these seem to be causing issues with maintaining
file capabilities.  They aren't necessary so let's just remove them.

Change-Id: I06c90fdc85655986142b936cadbe04d75dd27427
Closes-Bug: 1714604
2017-09-01 17:29:15 -05:00
Jenkins
39d84d2059 Merge "Use [[ for =~ matches" 2017-08-29 05:59:42 +00:00
Ian Wienand
c448864901 Use [[ for =~ matches
Avoid incorrect use of [ with =~ matching

I guess this doesn't trip "-e" because it's in an if-conditional.  I'm
looking at making bashate detect this; maybe we can run bashate over
things we know are scripts

Change-Id: Ia3fe2b978fae5bdaadbb1789058180d3ad950d00
2017-08-28 17:01:03 +10:00
Julia Kreger
6d64a2aee6 Fix cylical systemd config for dhcp-all-interfaces
In Ubuntu/Debian, the default dependencies cannot be relied
upon as we enter into a cyclical dependency relationship which
prevents the unit from starting.

Added the required configuration to the systemd unit file.

This issue has also been observed in glean[0], which has a nearly
identical unit file for interface start-up.

[0]: https://review.openstack.org/#/c/485748
Closes-Bug: #1708685

Change-Id: I23ac9510d1a21c7073bd33f76ba66fa04a8be035
2017-08-25 15:51:23 +10:00
Jenkins
ea23aa13a5 Merge "Add netbase to ensure /etc/protocols is placed for debian" 2017-08-15 00:37:27 +00:00
Julia Kreger
f19c45eb29 Add netbase to ensure /etc/protocols is placed for debian
Many programs rely upon /etc/protocols to be present
however the default debian image that is generated lacks
/etc/protocols. This is observable when building an image
for use with ironic via the ironic-agent element, since
the IPA agent fails to start as python needs /etc/protocols
to open a socket connection.

Added to debian-minimal as it is inherited into the debian
element.

Change-Id: Icc81635870961943707cf6b3f61a9ddbd51cb8fd
Closes-Bug: #1708531
2017-08-11 14:17:30 +00:00
Ian Wienand
a88a768e98 Clear up debian element documentation
There is some confusion in the readme's over what is happening.  The
original change (Iaf46c8e61bf1cac9a096cbfd75d6d6a9111b701e) split out
debian-minimal and made debian "... simply be a collection of the
extra things we do to make it look like a cloud-init based cloud
image"

Make this clearer in the documentation

Change-Id: Ibe6fad9c67b70a5e31e43e06419968135174fef3
2017-08-09 13:15:38 +10:00
Dave Hill
6c2b1465cc Clear /etc/machine-id to avoid duplicate machine-ids
Deploying many nodes with the generated image shouldn't have the same
/etc/machine-id so clearing it and letting systemd generate a new
id upon first boot seems to be the best way to achieve this.

Change-Id: I73d0577d31464521b3989312fd9d982a1312a268
Closes-bug: 1707526
Closes-bug: 1672461
2017-08-06 13:56:58 -04:00
Paul Belanger
7cbbee7ea3 Bump fedora/fedora-minimal DIB_RELEASE 26
Fedora 26 is now the latest release:

  https://fedoraproject.org/wiki/Releases/26/Schedule

We are building and using these in infra now

Change-Id: I012c2d28255be274e88abc2751d968bafaf76fbb
Depends-On: Ieba5f69020a13681074f72cfca2955071801b63a
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2017-08-04 14:18:05 +10:00
Ian Wienand
818b75af41 Move selinux packages to redhat-common
Change I008f8bbc9c8414ce948c601e3907e27764e15a52 has shown that we
build redhat images without the "semange" tool available, which comes
from the policycoreutils-python package (see also
I3f9e2c322d042a5dddba33451c0fc21a4d32a88a).

I403e7806ae10d5dd96d0727832f4da20e34b94c7 added some of the selinux
libraries to yum-minimal for ansible support, but not to others.

Given both these changes, it seems that selinux[-targeted],
libselinux[-python] and policycoreutils[-python] can reasonably
considered part of all base images.  Move the selinux related packages
into redhat-common.

This also adds it explicitly to install_test_deps.sh.  It was actually
being dragged in by the docker install, but is a required component
for building (should be in bindep, but not there with that yet).

Change-Id: Idd4ae71ee6deee84604823b6b5dc4a845f316e01
Related-Bug: #1707788
2017-08-01 11:08:54 +10:00
Jenkins
308783d382 Merge "Switch openSUSE to 42.3 by default" 2017-07-28 05:14:23 +00:00
Alfredo Moralejo
b1961e14ea Use SELinuxfs to check selinux status
Currently, the cleanup script is using existence of
semanage binary to check if selinux is enabled. However
this is misleading and can lead to problems when selinux
is disabled in a system where the binary exist.

This patch changes the detection logic to use /sys/fs/selinux
directory which is a in-memory filesystem created only when
selinux is really enabled.

Change-Id: I008f8bbc9c8414ce948c601e3907e27764e15a52
Related-Bug: 1706386
2017-07-26 18:57:25 +02:00
Dirk Mueller
1c4c4fd734 Switch openSUSE to 42.3 by default
This is the latest stable release, so we should default to it.

Change-Id: I05643787002d339ccbf7a718847fe4ed6f39eacc
2017-07-26 08:56:02 +02:00
Jenkins
609bcee27b Merge "zypper: Clean caches and don't cache packages locally" 2017-07-26 02:25:40 +00:00
Markos Chandras
81e72d4045 elements: zypper-minimal: Install tar package
tar is an essential package but nothing pulls it explicitly. This causes
some issues in the openSUSE CI jobs like the following one

"Failed to execute tar: No such file or directory", "Failed to write
file: Broken pipe", "Failed to retrieve image file. (Wrong URL?)",
"Exiting."], "stdout": "", "stdout_lines": []}

Just like 'sed', add 'tar' to the list of packages for the openSUSE
minimal builds.

Change-Id: Ia36e3d9fd6b78862a6831ba80b43d4614a349ca0
2017-07-25 16:27:25 +01:00
Jenkins
a6da39acb8 Merge "Move setfiles to outside chroot with runcon" 2017-07-24 02:04:21 +00:00
Ian Wienand
5089e4e541 Move setfiles to outside chroot with runcon
As described in the comments inline, on a selinux enabled kernel (such
as a centos build host) you need to have permissions to change the
contexts to those the kernel doesn't understand -- such as when you're
building a fedora image.

For some reason, setfiles has an arbitrary limit of 10 errors before
it stops.  I believe we previously had 9 errors (this mean 9
mis-labeled files, which were just waiting to cause problems).
Something changed with F26 setfiles and it started erroring
immediately, which lead to investigation.  Infra builds, on
non-selinux Ubuntu kernel's, would not have hit this issue.

This means we need to move this to run with a manual chroot into the
image under restorecon.

I'm really not sure why ironic-agent removes all the selinux tools
from the image, it seems like an over-optimisation (it's been like
that since Id6333ca5d99716ccad75ea1964896acf371fa72a).  Keep them so
we can run the relabel.

Change-Id: I4f5b591817ffcd776cbee0a0f9ca9f48de72aa6b
2017-07-24 10:14:07 +10:00
Dirk Mueller
bfeb9d9e99 zypper: Clean caches and don't cache packages locally
For builds inside the infra, we don't want to pack the cache
inside the image (as it might be different at the time the image
runs). In an opensuse-minimal image this saves about 10MB of image
size.

Change-Id: I5ecabd46f0a662798bda3e4468395ad8308d0055
2017-07-23 17:24:24 +02:00
Jenkins
55971717b6 Merge "elements: openstack-ci-mirrors: Use openSUSE mirrors for gating jobs" 2017-07-22 05:22:34 +00:00
Jenkins
e029af993b Merge "Remove DIB_[DISTRO]_DISTRIBUTION_MIRROR" 2017-07-22 05:22:04 +00:00
Jenkins
7a70299668 Merge "Enable console during kernel boot on Power" 2017-07-20 03:55:19 +00:00
Ian Wienand
7ffe6856d6
Add -m flag to setfiles for Fedora 26
As described in the comment and associated bugzilla, the behaviour of
setfiles has changed in Fedora 26 to require "-m" situations where
labeled file-systems are mounted below non-labeled file-systems.  Our
loopback/chroot system appears to trigger this nicely, leading to a
setfiles call that does nothing without this.

Change-Id: I276c6f6a4fb44f4bea5004f6b4214f94757728ae
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2017-07-19 08:01:19 -04:00
Ian Wienand
6ffde2e596 yum-minimal: pre-install coreutils
As described in the referenced bug, the dependency solver in yum
doesn't handle weak dependencies well and in some cases, such as
Fedora 26, can end up choosing coreutils-single (the busybox-esque
single binary) instead of actual coreutils, which then causes problems
with conflicting packages later.

Change-Id: I2907bf3b74c146986b483d52cc6ac437036330b4
2017-07-18 14:51:18 +10:00
Ian Wienand
b8ad9c2e37 Force install during pip-and-virtualenv
On a system where the packaged pip/virtualenv is up-to-date with
upstream (such as Fedora 26 ... for now), we don't reinstall, which
then violates a bunch of assumptions later on.  Force install.

Change-Id: I6ebcda0351997fa7e32f0e6e77a98b2c33764e3f
2017-07-18 14:50:09 +10:00
Ian Wienand
da90ef4743 Fix latest-limit command line
It turns out dnf argparse can't handle negative numbers without "=".
It's actually documented in the man page

    --latest-limit <number> ...  If <number> is negative skip <number>
      of latest packages. If a negative number is used use syntax
      --latest-limit=<number>

But who reads that :)  This started failing with Fedora 26

Change-Id: I884af94c07fa11b010f69863047a04711b14f21e
2017-07-18 13:17:10 +10:00
Jenkins
016606c81d Merge "opensuse-minimal: install glibc-locale" 2017-07-18 00:40:48 +00:00
Dirk Mueller
59721d3c74 opensuse-minimal: install glibc-locale
We expect LC_ALL for non-C locales to be working inside
images, so always install glibc-locale for openSUSE.

Change-Id: I8fe92773e377539070d9d9fe2960a6202bb80a18
2017-07-17 22:50:25 +02:00
Markos Chandras
6be09152c2 elements: openstack-ci-mirrors: Use openSUSE mirrors for gating jobs
In preparation for promoting the openSUSE jobs to voting ones we should
use the OpenStack mirrors. As such, the opensuse elements are modified
to make use of the DIB_DISTRIBUTION_MIRROR variable which is normally
exported by the openstack-ci-mirrors element.

Change-Id: Ie588c1c1eec13190cfb2ec718ba51f8c9878283f
2017-07-17 10:54:03 +01:00
Jenkins
c18a3ff029 Merge "Replace architecture-emulation-binaries with qemu-debootstrap" 2017-07-17 05:36:09 +00:00
Ian Wienand
3457d2f8e8 Remove DIB_[DISTRO]_DISTRIBUTION_MIRROR
We added the DIB_distro_DISTRIBUTION_MIRROR arguments with
I92964b17ec3e47cf97e3a3091f054b2a205ac768 as a way that we could
source a list of mirrors and then have the distro elements choose
which one applied to them.

However, this hasn't worked out to be so useful.  The
openstack-ci-mirrors element is working as a mirror setup script -- it
translates the openstack CI mirror list variables into the generic
"DIB_DISTRIBUTION_MIRROR" as appropriate for each distro's build.
Also, it turns out there's other things that need to be done, such as
turning off gpg checking, which mean the idea of "just export
variables" hasn't turned out as valid ... you need actual code
involved to get it right.

AFAICT we never actually documented these, and they do not seem to be
in use.  They have caused considerable confusion when dealing with new
platforms as we try to keep consistency.  Remove them.

[1] http://codesearch.openstack.org/?q=DIB_.*_DISTRIBUTION_MIRROR&i=nope&files=&repos=

Change-Id: Ifc4ab700631ffdfbe790068558f670f9a11dde5e
2017-07-17 14:47:31 +10:00
Jenkins
787e76b916 Merge "Remove additional Bumblebee repository for opensuse element" 2017-07-17 00:50:46 +00:00
Jenkins
64a8c6e1dc Merge "zypper-minimal: No point in preserving the environment here" 2017-07-17 00:26:11 +00:00
Dirk Mueller
02d33f2ca7 zypper-minimal: No point in preserving the environment here
Change-Id: I46442e841d1f718b683bca4d2a348f0013306907
2017-07-13 22:50:47 +02:00
Dirk Mueller
05ba445ade Remove additional Bumblebee repository for opensuse element
The purpose of the openSUSE element is to build openSUSE distribution
based images, so an additional community repo shouldn't be pulled into
the image. In addition the dkms dependency is blacklisted for SUSE
in the dkms element anyway, so this should be a noop.

Change-Id: I0aa06d9f4f110546032f910e3361840693d02de7
2017-07-11 23:24:05 +02:00
Jenkins
0327d775f1 Merge "pip-and-virtualenv: Install python3 on openSUSE" 2017-07-11 08:11:16 +00:00
Rafael Folco
bfdf7dc0f6 Enable console during kernel boot on Power
On Power systems console should be added the kernel command line
in the following order: 'console=tty0 console=hvc0'.
The first one is the graphical console. The last one is the serial
console. The kernel enables all the consoles pointed through the
kernel command line. However, only the last one will receive
input/output during kernel boot. All the other consoles will be
enabled after the boot.

Change-Id: I0069f608e0ab104d3778954e033fb82ed5ea7693
2017-07-07 17:55:56 +00:00
Amrith Kumar
43e32116bd fix readme.rst to reflect correct environment variable
The readme.rst incorrectly refers to the environment variable
DIB_APT_KEYS which should be DIB_ADD_APT_KEYS. See [1] for usage in
code.

This is a minor correction to the readme only, no runnable code is
modified.

[1] http://git.openstack.org/cgit/openstack/diskimage-builder/tree/diskimage_builder/elements/dpkg/extra-data.d/01-copy-apt-keys#n23

Change-Id: I04129cef9f40ec75a206c126bfd40ee61e4e6a2b
2017-07-06 22:54:08 -04:00
Jenkins
e8ad2a3799 Merge "elements: pip-and-virtualenv: Use common packages for openSUSE" 2017-07-04 11:20:35 +00:00
Markos Chandras
5fe35b0d7a pip-and-virtualenv: Install python3 on openSUSE
The python3 package actually contains some core modules (like the xml
one) which are not present in the python3-base on which is pulled by
the python3-devel package. As such, it's best to have it installed
similar to python-xml for python2.

Change-Id: I5cd5d1127ae62d6753c2ace44965179c5400bb9a
2017-07-04 08:40:34 +01:00
Jenkins
fad72745d2 Merge "Support for Cloud Images on ppc64le for rhel7 and centos7" 2017-07-04 01:13:24 +00:00
Jenkins
6b45497ff6 Merge "Remove centos and rhel elements" 2017-06-29 21:16:57 +00:00
Jenkins
f0fb835db9 Merge "Avoid hanging endlessly on unreachable cache urls" 2017-06-29 08:03:25 +00:00
Chhavi Agarwal
6d69d7909d Support for Cloud Images on ppc64le for rhel7 and centos7
In order to support {CentOS,RHEL}7 for building cloud images we need to
handle the differences in grub packaging from Ubuntu.  We also need to
populate the defualt location for cloud images for CentOS builds.

Change-Id: Ie0d82ff21a42b08c4cb94b7a5635f80bfabf684e
2017-06-29 15:44:26 +10:00
Dirk Mueller
959226c55e Avoid hanging endlessly on unreachable cache urls
When a download redirector redirects to a broken mirror, timeout
quickly rather than waiting until the overall job is being timed out.

Change-Id: If7eb63d406aaf61f71aa9203cf708c474aa63fd0
2017-06-28 22:14:55 +02:00
Markos Chandras
c46b6da65f elements: pip-and-virtualenv: Use common packages for openSUSE
The 'packages' variable already contains the packages we need so
use it instead of duplicating the packages.

Change-Id: Id22e1862f9654e66252d03a0fed9839cf004d750
2017-06-28 17:59:25 +01:00
Ian Wienand
a00d02f6a1 Remove centos and rhel elements
Several people have popped up in IRC recently with failures in these
elements.  Without Python 2.7 available in the image they are
unsupported (OpenStack hasn't supported it for a long time).  Remove
these to avoid further confusion.

The centos/centos7 DISTRO split that has happened with centos-minimal
is unfortunate but I don't think it helps to rename centos7/rhel7 ATM.
To summarise; DISTRO=centos7 means image based build,
DISTRO=centos && DIB_RELEASE=7 means the minimal build.

In the future, I think it is important that the minimal builds and
image builds set the same DISTRO.  This reflects that "upper" layers
shouldn't care about the exact building of the lower layers.  I see
CentOS 8 going one of two ways

1) the changes are so significant, we start separate centos8 /
centos8-minimal elements.  They both set DISTRO=centos8 (and
DIB_RELEASE to point-release maybe?).  This means we have to update
all "if DISTRO == centos || DISTRO == centos7" branches to also check
for "centos8".  Evenually (!)  "centos" goes away for versioned DISTRO
only

2) we restore centos element with DISTRO=centos and DIB_RELEASE=8, and
centos-minimal remains the same.  This means we have to audit all "if
DISTRO == centos" calls to make sure they're appropriate for version 8
(stick a "&& DIB_RELEASE=7" on them all basically).

I'm not sure we can fully decide until we start to see excatly how the
distro switching/matching bits look, but (2) is consistent with Ubuntu
and probably the preferred solution.

Some "rhel" parts have been cleaned up.  More could be done in
rhel-common, but given our lack of coverage of that I'd prefer to
leave it for now.

Change-Id: I6ea784116ef59ca22878c8512c963f29c815a00a
2017-06-28 12:26:24 +10:00
Ian Wienand
b0e0dd991c Move image download tests to default skip
The image download tests have long been too unreliable for the gate.
We need to cache the base images similar to how devstack caches it's
testing images.  Let's move them to non-voting jobs for the time
being.

This means that the gate jobs are now all based on "-minimal" and are
using infra mirrors.  Unfortunately, there is still some unreliability
because we currently have issues with infra mirrors being very slow
after AFS updates, leading to job timeouts.  But we're on the right
path...

Also, I noticed we don't have tests of the "ubuntu" image-download
based tests, which were tacitly being tested by apt-sources before we
moved that to -minimal.  Add simple tests for these.

Change-Id: Ie33ee49656872467ef68d753210032156bb6b2cb
2017-06-23 10:58:47 +10:00
Clark Boylan
559de43694 On suse the python2 dev package is python-devel
This was previously defined as python2-devel (which is what rhel uses),
but the actual package name is python-devel. See:

  https://software.opensuse.org/package/python-devel

Change-Id: Id61e5b05772d10c32b33d3e70cb64d5ebdcba6e4
2017-06-21 15:52:05 +10:00
Ian Wienand
18a0d970fa Move ironic-agent test to fedora-minimal
I'm uncertain as to why this is using the "fedora" element for testing
... but it requires downloading the fedora .qcow on every test which
has shown to be unreliable.  An easy thing to do is to switch it to
fedora-minimal; that will only involve downloads from local mirrors in
the gate.

Add redhat-rpm-config for minimal.  I admit I have not fully gone
through why this is not pulled in.  It's been an issue since
I459f2203fa145049dda185da952813118193d573 and there's all sorts of
bugs.

Change-Id: I37458e3926dae32a259bd5aa9efc645561b029a0
2017-06-21 15:05:36 +10:00
Ian Wienand
649f0b66d9 Start at using CI mirrors for fedora/centos
fedora/centos-minimal don't obey DIB_DISTRIBUTION_MIRROR currently.  I
don't really want them too -- we want to be able to separate the
mirrors used during the build process from those embedded into the
final image.  Add DIB_YUM_MINIMAL_BOOTSTRAP_REPOS which is a directory
with repo files to use during the install.

This introduces setup-gate-mirrors.sh which is intended to setup
repo/sources/whatever files in the openstack gate that point to the
local region mirror.  It pulls the info from the mirror_info.sh script
on each CI node.

The openstack-ci-mirrors element is updated to export these variables.
elements are updated to depend on it.  Tests are restored

Change-Id: I7604fc4d41cb1483be16b8d628a24e8fc764f515
2017-06-21 12:02:27 +10:00
Ian Wienand
f0b70211c6 Use local mirror for ubuntu-minimal jobs
This adds "openstack-ci-mirrors" element which performs various
settings to get builds using local mirrors.  As a first step, we
convert ubuntu-minimal jobs

The main trick is that since infra mirrors are created with rerepo
they are not signed (they are recreated, not cloned, and not signing
is seen as a feature in that it deters external use).  So we need to
instruct debootstrap to ignore signing and also turn it off for
in-chroot apt.  Other than that, the existing DIB_DISTRIBUTION_MIRROR
works to redirect installs.

Remove "restricted" as it's not mirrored, and I don't think we want it
in here by default.

(I think DIB_DISTRIBUTION_MIRROR is a bit of an anti-pattern, because
it leaves the mirrors in the final image -- just because you use them
to build, doesn't mean you want them at runtime).  But we don't need
to fix that now, and we don't use any created images.)

This pauses fedora testing until the next change, which moves to using
local mirrors for testing on fedora/centos

Change-Id: I778bd05a1e615c27edf1c9f0a1409119a6b3a850
2017-06-21 12:01:31 +10:00
Ian Wienand
0d37351031 Move apt-sources to ubuntu-minimal / move debian to skip list
The gate is currently extremley unstable, and these two issues are
causing most of the problems.  We need to commit them atomically so we
can get anything moving again

---

The gate is very unstable downloading the ubuntu tarballs from
upstream at the moment.  Move this to ubuntu-minimal which, in a later
change will source files from our local mirror.

We need a caching mechanism for these large files to avoid this
instability.  This is future work for the various image-based jobs.

---

Move debian to default skip lists

I don't know if it's mirrors being worked hard for the Stretch
release, but this is constantly failing the gate.  I will move this to
the -nv extras job

I am working on having the voting job use local mirrors for
everything.  Unfortunately debian infra mirrors don't have stretch yet
and we need to do some fiddling to get "stable" available.  Once we
have all this, we can consider making it voting again.

Change-Id: Iaf7b3888ef06c7aef63cbf76a94b33f96bc9c5c2
2017-06-21 10:34:53 +10:00
Ian Wienand
a0f747932d Install systemd earlier for Stretch
Debian Stretch released as stable recently, and the init system is
less tightly specified in the base dependencies (for some info, see
[1]).  It seems, probably unintentionally, that in the previous
release systemd-sysv was brought in by debootstrap, but that is no
longer happening.

Add systemd as an early dependency of debian-minimal.

Remove the package-installs.yaml as that happens too late (other
things need to know the init system to write out service files, etc
and probe for systemd utils before package-installs).  As mentioned, I
do not believe the "only install systemd on testing" idea was actually
working here, because it was being brought in during the initial
debootstrap.

Update some documentation to explain what's going on

[1] https://lists.debian.org/debian-boot/2015/05/msg00156.html

Change-Id: Id67c0cf08728407d234976f9807d3bd71d12f758
2017-06-19 13:27:33 +10:00
Jenkins
1324f5b7db Merge "Remove use of 'which'." 2017-06-11 09:30:34 +00:00
Ian Wienand
5ac8a98e9a PPC bootloader; install to boot partition
Using the newly exposed variables from the prior change, install the
ppc bootloader to the boot partition, not the underlying loopback
device.

Change-Id: I0918e8df8797d6dbabf7af618989ab7f79ee9580
2017-06-08 17:14:22 +10:00
Ian Wienand
6c394f5746 Pass all blockdevices to bootloader
Currently we only export "image-block-device" which is the loopback
device (/dev/loopX) for the underlying image.  This is the device we
install grub to (from inside the chroot ...)

This is ok for x86, but is insufficient for some platforms like PPC
which have a separate boot partition.  They do not want to install to
the loop device, but do things like dd special ELF files into special
boot partitions.

The first problem seems to be that in level1/partitioning.py we have a
whole bunch of different paths that either call partprobe on the loop
device, or kpartx.  We have _all_part_devices_exist() that gates the
kpartx for unknown reasons.  We have detach_loopback() that does not
seem to remove losetup created devices.  I don't think this does
cleanup if it uses kpartx correctly.  It is extremley unclear what's
going to be mapped where.

This moves to us *only* using kpartx to map the partitions of the loop
device.  We will *not* call partprobe and create the /dev/loopXpN
devices and will only have the devicemapper nodes kpartx creates.
This seems to be best.  Cleanup happens inside partitioning.py.
practice.  Deeper thinking about this, and more cleanup of the
variables will be welcome.

This adds "image-block-devices" (note the extra "s") which exports all
the block devices with name and path.  This is in a string format that
can be eval'd to an array (you can't export arrays).

This is then used in a follow-on
(I0918e8df8797d6dbabf7af618989ab7f79ee9580) to pick the right
partition on PPC.

Change-Id: If8e33106b4104da2d56d7941ce96ffcb014907bc
2017-06-08 17:14:22 +10:00
Ian Wienand
90b56b3aab Move ppc block-device default to right $ARCH
The supported ppc ${ARCH} is "ppc64el" (at least in the gate testing
...) so move the file to that, so gets picked up by
block_device_create_config_file

Change-Id: I9273f35cdbfb0a62404461cbc1df9b2a92155fb0
2017-06-07 13:30:38 +10:00
Jenkins
ec70cb61f0 Merge "Trivial fix typos" 2017-06-05 05:54:50 +00:00
Jenkins
5a045e036d Merge "dhcp-all-interfaces.sh - Add support for InfiniBand interface DHCP" 2017-06-02 06:11:19 +00:00
Jenkins
80cc1d0ea4 Merge "Adjust package mapping for SUSE family" 2017-06-02 02:56:16 +00:00
Dirk Mueller
d0a398c167 Adjust package mapping for SUSE family
package-installs.yaml is installing python-dev, not python2-dev,
so we need to adjust the mapping accordingly.

In addition, zypper-minimal used an dpkg specific package name,
while there is a SUSE equivalent (and zypper-minimal is anyway
SUSE family specific)

Change-Id: Ia9dd061fa46a514781808d62e5e93b03f75c6745
2017-05-31 21:09:53 +02:00
Dirk Mueller
f58bf252de Drop support for Ubuntu precise
Ubuntu 12.04 LTS reached its regular End of Life on April 28, 2017.

Depends-On: I5e145095a10db112bb27516bfe652d2cdc052a61
Change-Id: I64af4c5183d77a75dcd062895d19b0a1330c8da8
2017-05-31 14:36:30 +02:00
Jenkins
b312c06dbb Merge "Decode string to bytes in dracut-regenerate" 2017-05-31 10:49:51 +00:00
Jenkins
d0e0714f71 Merge "Test openSUSE 42.2/42.3 image builds" 2017-05-31 04:37:57 +00:00
Vu Cong Tuan
6a72052108 Trivial fix typos
Change-Id: Ib86aa9938fd852610ec0a6d8d868181f87bd2f24
2017-05-31 11:17:05 +07:00
Jenkins
2bdc154df5 Merge "drop deprecated map-services/packages from zypper element" 2017-05-31 02:11:21 +00:00
Jenkins
05d64b99ce Merge "Remove ccache" 2017-05-31 01:48:01 +00:00
Jenkins
edaf577bad Merge "Remove dracut-network element" 2017-05-31 00:14:01 +00:00
Mark Goddard
54765fd2f4 Remove dracut-network element
This element has not been functioning correctly for some time due to
an incorrect path to select-boot-kernel-initrd (should be /usr/local/bin).

The dracut-regenerate element can be used to regenerate dracut ramdisks
and is more flexible than this element.

Change-Id: I33d555ffd4a92b2948b2ea4a66b151f0422ccb8c
Closes-Bug: #1688546
2017-05-31 08:36:56 +10:00
Andreas Florath
b107606a75 Remove ccache
This patch removes the ccache handling from the base element.  For
mostly all systems this was never used at all.

This is working towards the removal of the base element from DIB

Change-Id: Ieb16ef612ebd98470993dcd6f55b3a22d37084ba
Signed-off-by: Andreas Florath <andreas@florath.net>
2017-05-31 08:28:09 +10:00
Mark Goddard
aa6c1d01a9 Decode string to bytes in dracut-regenerate
In python3, the standard out data returned by
subprocess.Popen.communicate() will in most cases be bytes rather than a
string and must therefore be decoded.

Without this fix we hit the following error:

TypeError: a bytes-like object is required, not 'str'

Change-Id: I6d75f867ebfdb925970c3397175214b9050d7632
Closes-Bug: #1694463
2017-05-30 16:15:06 +01:00
Dirk Mueller
5d39f83f74 Test openSUSE 42.2/42.3 image builds
Currently openSUSE 42.3 has entered feature freeze mode
so it is a good point in time to verify that 42.3 builds
are working successfully. Also test opensuse-minimal for
platforms that support it (need working zypper package)

Change-Id: I4c613e1e68cb7375c29d544bbf70b5da9bf21414
2017-05-30 13:07:04 +02:00