Commit Graph

1224 Commits

Author SHA1 Message Date
liyingjun
bf0d7ede55 Fixes packages for arm64 bootloader
Should install "grub2-efi-aa64 grub2-efi-aa64-modules" instead of
"grub2-efi grub2-efi-modules" for arm64

Change-Id: Iee3191b0944b3b862890d166a9d36bd592fe8f7e
Closes-bug: #1839816
2019-08-12 17:18:36 +08:00
Manuel Torrinha
a38ac762f1 Fixes DIB_DISTRIBUTION_MIRROR_UBUNTU_IGNORE matching when empty
- DIB_DISTRIBUTION_MIRROR_UBUNTU_IGNORE matches when it is empty or not
 set and DIB_DISTRIBUTION_MIRROR is being used. Checking for it being 
 set and not empty solves this.
 - Normalizing bash conditionals for readability

Closes-Bug: #1808359
Change-Id: I87853fcda4c8b29a3f1720a2778debeb3acc3a53
Signed-off-by: Manuel Torrinha <manuel.torrinha@tecnico.ulisboa.pt>
2019-08-09 10:26:48 +00:00
Michael Johnson
bac0fa3eb2 Fix the pypi element for multiple mirror URLs
The 'pypi' mirror element is generating invalid pip.conf files
when more than one "DIB_PYPI_MIRROR_URL" is specified.
This patch fixes the pip.conf file rendering to use a proper form
when there are multiple "extra_index_url" provided.

Closes-Bug: #1839558
Change-Id: Ibda0e7390955683560e09b486f636775643ff57c
2019-08-08 22:52:43 +00:00
Andreas Jaeger
9a145cf0d0 Stop regex warning
python 3.6 warns about regexes like:
DeprecationWarning: invalid escape sequence \+

I noticed that debugging a trove job and it really led me in the wrong
way. Fix this with making it a raw string.

Change-Id: I58ee1a49d62316c6c3f0588832c97f659f7e460b
2019-08-08 15:31:52 +02:00
Ian Wienand
1f2a874e8e Create /etc/machine-id for RHEL images
Per the inline comment, a machine-id is required for kernels to
install correctly (this may well be a bug, but the linked issue
remained inconclusive).

Add a call to make the machine-id before install packages.

Change-Id: If75d04376e62bfdfe14ee3ca4d0bd5c8b383c1b0
Redhat-Bug: https://bugzilla.redhat.com/show_bug.cgi?id=1737355
2019-08-07 18:18:34 +10:00
Zuul
37909a0e81 Merge "Cleanup: remove useless statement" 2019-08-05 02:46:12 +00:00
Zuul
a0c6d16e1c Merge "journal-to-console: element to send systemd journal to console" 2019-08-05 02:00:15 +00:00
Zuul
f59ffcd9f3 Merge "fix comments / spelling errors in gentoo element" 2019-07-30 07:22:26 +00:00
Zuul
3d995645e6 Merge "update version of open-iscsi that is installed on musl" 2019-07-30 02:37:18 +00:00
Matthew Thode
559d2bcf3b
fix comments / spelling errors in gentoo element
Change-Id: I41d906a99470599da3cfac7aaa0350232ce79316
2019-07-29 08:54:16 -05:00
Matthew Thode
f1d7e902e3
support alternate portage directories
The 17.1 profile changed the defaults used in portage for where we store
our repo, distfiles and binpkgs.  Some portage related variables need to
be set deterministically.  17.1 is no enabled for Systemd's profile.

Change-Id: Ib55f6875c5cb461c3c530b51d7420ce3dc8da360
2019-07-26 19:30:01 -05:00
Ian Wienand
a5bd03ec6b journal-to-console: element to send systemd journal to console
This element configures systemd to send its journal to the console,
which can then be retreived by server commands.  In the case of
nodepool, if the image failed to boot the console will be dumped into
the logs when nodepool decides the node is not responding.  Having
this can be very helpful diagnosing early boot errors.

Needed-By: https://review.opendev.org/#/c/669787/
Change-Id: I6b6df7023acb6b2f967b84840bc4b542ebc03727
2019-07-25 11:24:49 +10:00
Matthew Thode
ed95b4eba1
update version of open-iscsi that is installed on musl
Newer versions of open-iscsi seem to compile on Gentoo / musl.  Use them
if we can.  This also removes the cap on open-iscsi.

Change-Id: I596cb61494e459a419bce6a63deff89f9e78fe23
2019-07-22 14:30:32 -05:00
Zuul
f35d29c356 Merge "ironic-agent: Use targetcli & python3-devel on rhel8" 2019-07-15 00:31:01 +00:00
Zuul
fc1709b0fb Merge "set default sources conf for buster as it now has a release" 2019-07-15 00:31:00 +00:00
Zuul
c7eb556098 Merge "disable autounmask for emerge" 2019-07-15 00:25:57 +00:00
Clark Boylan
abb6aed459 Only enable dbus-daemon on fedora-29
Previously we were trying to enable dbus-daemon service on all prior to
fedora 30. Unfortunately 28 and older don't have this service so this
broke those releases and only worked for 29. Fix this by only enabling
this service on fedora 29.

Change-Id: I1bd15dcf0bbe270afccb0c0c3ea6ad08862a53f1
2019-07-12 10:21:49 -07:00
Zuul
709c9e70c9 Merge "Set router solicitation delay with using NM" 2019-07-10 19:14:00 +00:00
Clark Boylan
5b5b78bf59 Set router solicitation delay with using NM
The linux kernel and NetworkManager fight each other over control for
interface management when router advertisements are in use. Long story
short if the linux kernel configures a network interface for ipv6
before NetworkManager attempts to manage that interface then NM will
ignore the interface and not configure ipv4 on it.

This can happen because the kernel is configured to send router
advertisements solicitations which result in router advertisements which
the kernel uses to configure the interface(s). There is a default of a 1
second delay before sending the solicitation which in many cases is long
enough that NM has started before then. However, in slower environments
like those used for testing with qemu this isn't long enough.

Some testing by hand indicates that 15 seconds is about right so
increase the delay to 15 seconds via sysctl.conf.

Note this may increase boot times in ipv6 only environments (though it
is hard to be sure due to how systemd starts everything at once and does
socket activation and the like).

Change-Id: I475a253091cbaf63687b91c748c31a6753bb0f57
2019-07-10 08:33:17 -07:00
Chandan Kumar (raukadah)
08caa8034d [RHEL-8] Set _clear_old_files=0 in install-pip element
When virtualenv and setuptools gots installed from source and rpm
then their installation path lives at different places but when
the python script got called then that time it choses either of
rpm or source based path on system wide installation and leads to
different failure as their methods are not implemeted.

So by setting _clear_old_files to 0 will install
python3-virtualenv python3-pip python3-setuptools from rpms only
and avoid these failures.

Change-Id: I0c162f1fe8168513e352546ab8dd2b68fa65b88c
Signed-off-by: Chandan Kumar (raukadah) <chkumar@redhat.com>
2019-07-08 19:50:25 +05:30
Matthew Thode
de94e07a86
disable autounmask for emerge
autounmask=y (default) changes portage depsolving, causing errors
(mostly often seen in perl and binpkg related issues).

Disabling this functionality for DIB builds is OK as the enviroment is
not passed on post build and the build process is not interactive
anyway.

Change-Id: Ife9ace246bec16864ee4982bc456763af5dff2e8
Signed-off-by: Matthew Thode <mthode@mthode.org>
2019-07-05 12:41:51 -05:00
Lon Hohberger
92f87d0d0a ironic-agent: Use targetcli & python3-devel on rhel8
Change-Id: I2e596f5c3dc23b21441a4eb61c773725ebd25d34
Signed-off-by: Lon Hohberger <lhh@redhat.com>
2019-07-02 21:39:13 -04:00
Matthew Thode
d8f796e153
install gnupg2 by default in debian-minimal
debian-minimal depends on debootstrap which depends on dpkg

This needs to be installed early as dpkg installs the apt keys early via
02-add-apt-keys in pre-install.d

Change-Id: I8580849ceaa7a5152c94f29afa890ac6d6983fb1
2019-07-01 14:28:01 -05:00
Matthew Thode
03933b3ab7
set default sources conf for buster as it now has a release
Change-Id: Ica014a1a267b55e2f29a0c333213e0a4c897108b
2019-07-01 13:59:14 -05:00
Maksim Malchuk
32dd3305d2 Cleanup: remove useless statement
This change removes useless statement accidentally added in the
06576a02f0

Change-Id: I7ea4a24d8c72c9e72f5f87247403af0f9bf69b40
Signed-off-by: Maksim Malchuk <maksim.malchuk@gmail.com>
2019-07-01 10:28:12 +03:00
Zuul
091a4e2c6e Merge "debootstrap: make default network interface names configurable" 2019-06-28 06:30:46 +00:00
Zuul
2c5fc5f8f9 Merge "Enable nodepool testing for opensuse 15.1" 2019-06-28 06:26:58 +00:00
Dirk Mueller
97444ad92e Enable nodepool testing for opensuse 15.1
There are several jobs depending on working opensuse 15.1
images in nodepool, so it makes sense to ensure its working.
Also upgrade the previously marked experimental opensuse-15.0
job as it tests the xenial->opensuse combination, which is
particularly difficult to keep working and we'll need it in
the CI.

Change-Id: Icb6d998756ce5221e017959dcb59b21f0f023454
2019-06-27 19:59:45 +02:00
Zuul
f27a0dfc26 Merge "Add DIB_UBUNTU_KERNEL to ubuntu-minimal" 2019-06-25 22:48:50 +00:00
Michael Johnson
e433aebf7d Add DIB_UBUNTU_KERNEL to ubuntu-minimal
This patch adds a new environment variable to the ubuntu-minimal
element called DIB_UBUNTU_KERNEL that allows you to specify the kernel
meta package that will be using to install the kernel inside the image.
It supports "linux-image-generic" (The default), "linux-image-kvm", and
"linux-image-virtual".
This allows building images that are smaller in size (~200MB smaller
qcow2) that have only the kernel modules necessary for virtual
machines.

Change-Id: I8ce65e1d357d15e8ed8995ad1dcaea02bbd1986f
2019-06-20 10:18:23 -07:00
jacky06
8dd7ca720f Sync Sphinx requirement
1. Sync sphinx dependency with global requirements. It caps python 2 since
sphinx 2.0 no longer supports Python 2.7.
2. Update some URLs to latest
3. Remove the unnecessary space

Change-Id: I5464be9e055feecd80918f691448acf5f100e701
2019-06-18 23:29:52 +08:00
Zuul
4afcba1fea Merge "Move pypi to dib-python" 2019-06-18 06:49:37 +00:00
Zuul
e281c4dae9 Merge "Use architecture-specific grub2 RPMs on RHEL8" 2019-06-18 06:49:35 +00:00
Zuul
c46f8714e3 Merge "Remove the rhel 8 check for xfs" 2019-06-18 06:49:34 +00:00
Zuul
fe618002ff Merge "Update test coverage for openSUSE/-minimal to 15.1" 2019-06-18 06:21:21 +00:00
Dirk Mueller
a81cf9e231 Update test coverage for openSUSE/-minimal to 15.1
Use openSUSE 15.1 as default, which is the latest released stable
openSUSE release.

Remove leftovers for unmaintained openSUSE 42.2 images.

Depends-On: https://review.opendev.org/#/c/660126/
Change-Id: I0b204b7b3d7ae74b6749320b3bfe1ca89d154ebb
2019-06-13 09:20:40 +02:00
Michael Johnson
5f4b764a11 Remove the rhel 8 check for xfs
This patch removes the check and default for rhel 8 requiring
xfs filesystem as rhel 8 images can successfully be built with
ext4 filesystems.

Change-Id: I1a6bfa26324fd43ae0c77c2c977dda0dd56e26e5
2019-06-12 07:01:36 -07:00
Andreas Florath
05af0fc863 debootstrap: make default network interface names configurable
Nowadays, in the time of Predictable Network Interface Names, the
network interface names 'ethX' are not used that often any more.
Depending on the virtualization layer and the guest OS names like
'ens3', 'enp1s0' or 'enp0s31f6' are used.
This patch enables the user to set DIB_NETWORK_INTERFACE_NAMES to a
list of network interfaces which are brought up using DHCP during
(first) boot.

Change-Id: I04cc2ee710f0389a8085b1c91d9329784cb28048
Signed-off-by: Andreas Florath <Andreas.Florath@telekom.de>
2019-06-12 13:53:38 +00:00
Maksim Malchuk
ea9ab89829 Move pypi to dib-python
The latest Fedora/Ubuntu images don't ship python2 by default, so we
need to use our dib-python wrapper for this so we work in python3 only
environments.

This change also correctly creates the pip.conf and .pydistutils.cfg
files with trusted host extracted from the index-url.

Related-bug: 1577105

Change-Id: Ibb5348af3e3bbe46b19affe90a8930a4b4ad4cad
Signed-off-by: Maksim Malchuk <maksim.malchuk@gmail.com>
2019-06-10 20:43:09 +03:00
Dmitry Tantsur
928c6e61f0 ironic-agent: install mdadm on the ramdisk
The newly introduced software RAID support requires it.

Change-Id: Ic438865006f1472abc0c9f4d40cc40c91b4ada71
2019-06-07 14:05:41 +02:00
Lon Hohberger
0cf0942068 Use architecture-specific grub2 RPMs on RHEL8
RHEL8 ships a bunch of grub2-efi-X-modules in its main
repository, each of which provides grub2-efi-modules,
potentially causing nondeterminism when building images.

This changes the DIB elements to always use architecture-
specific RPMs when RHEL8 is selected.

Change-Id: If94f3721195d5ecd80036e4234a3ca223a19c349
Related: https://bugzilla.redhat.com/show_bug.cgi?id=1716672
2019-06-06 10:50:51 -04:00
Zuul
7f469e3e83 Merge "Increase size of EFI system partition (again)" 2019-05-31 09:05:17 +00:00
Zuul
d323928af0 Merge "Makes image caching more resilient" 2019-05-31 08:54:31 +00:00
Zuul
d6f43865ed Merge "fail early when lates build information can not be fetched" 2019-05-31 08:45:50 +00:00
Zuul
21676bd350 Merge "Add option to skip update packages" 2019-05-31 08:13:38 +00:00
Zuul
49930da885 Merge "Deprecate rhel7 in favor of rhel" 2019-05-31 08:01:13 +00:00
Zuul
4367dd2dd3 Merge "Add version-less RHEL element for RHEL7 and RHEL8" 2019-05-31 07:56:50 +00:00
Pierre Riteau
7fd52ba841 Increase size of EFI system partition (again)
When I said in I8594d1fe05242f246a5809740a115ab2f84ac5a3 that 12 MiB
ought to be enough, I should have expected that I would be proven wrong.
While 12 MiB is enough to fit shim-x64 and grub2-efi-x64, yum fails to
update these packages to newer versions:

Transaction check error:
  installing package shim-x64-15-2.el7.centos.x86_64 needs 7MB on the /boot/efi filesystem
  installing package grub2-efi-x64-1:2.02-0.76.el7.centos.1.x86_64 needs 3MB on the /boot/efi filesystem

Error Summary
-------------
Disk Requirements:
  At least 7MB more space needed on the /boot/efi filesystem.

It is recommended that the ESP partition be much bigger. This commit
bumps its size to 550MiB, following guidelines from Rod Smith to avoid
incompatibilities with some EFIs [1].

[1] https://www.rodsbooks.com/efi-bootloaders/principles.html

Change-Id: If9515234f1a803cda32b2482f8abe10ddf0e6d26
2019-05-31 17:10:08 +10:00
Sorin Sbarnea
fb656718fb Makes image caching more resilient
Avoids failing on the first attempt to download the image to cache as
mirrors hosting them can randomly go down, usually with a connection
refused.

Change-Id: I9de9f33c2cc16596d04b35c4eb92621e6a2c7511
2019-05-31 16:31:43 +10:00
Dirk Mueller
421a0fa541 fail early when lates build information can not be fetched
When the mirror returns a error, it was trying to interpret the error
message (e.g. <html><title>Internal server error..) as a download link.
By using -f on curl we get an empty reply and an exit code, which, as
we run in set -e mode, aborts.

Change-Id: Ibaa39aedb7db286f859c4b090114c6a233b150c7
2019-05-31 16:09:25 +10:00
Zuul
ed6dfd87e5 Merge "allow the use of non-bzip compressed stages for building gentoo" 2019-05-31 04:42:07 +00:00
Nir Magnezi
433a374748 Deprecate rhel7 in favor of rhel
The rhel7 element is deprecated and is left only for backward
compatibility.
The rhel element should be used instead. Users should set DIB_RELEASE to
'7' to indicate which release you are using.

The new element is a version-less RHEL element to handle both '7'
and '8' DIB_RELEASE, which aligns with other elements which operate in
the same way such as the Fedora element.

Change-Id: Ic39ed85cacae9942448eb18ad685763f9369c2ed
2019-05-29 12:07:44 +00:00
Nir Magnezi
ee46e2f9b7 Add version-less RHEL element for RHEL7 and RHEL8
Make a version-less RHEL element to handle both '7' and '8' DIB_RELEASE.
The element usage should align with other elements which operate in the
same way such as the Fedora element.

Additionally, this patch adds support for RHEL8 that operates with
Python 3.
As of now, users of diskimage-builder will still be able to use the
'rhel7' element, or migrate to 'rhel' and specify their respective
DIB_RELEASE value.

* mount the xfs file-system for extraction as read-only.  vaguely
  based on explaination in [1] and the fact we only read the image
  data into a tar, so can ignore this.

    XFS (dm-1): Superblock has unknown read-only compatible features (0x4) enabled.

* Use the redhat system python as the dib-python version.  dib was
  ahead of it's time making an abstracted python interpreter for
  system work ;) the system python should work for running the various
  dib element scripts.

[1] https://unix.stackexchange.com/questions/247550/unmountable-xfs-filesystem

Redhat-Bug: https://bugzilla.redhat.com/show_bug.cgi?id=1700253
Co-Authored-By: Ian Wienand <iwienand@redhat.com>
Change-Id: I90540675c70bb475d9db2ae24f81c648a31f3f95
2019-05-29 11:28:53 +03:00
Matthew Thode
afcac1922c
allow the use of non-bzip compressed stages for building gentoo
Upstream is switching to xz so we need to be able to support it.

Change-Id: I382cc3e8038e2e552c553c526a990a01e51aeb12
2019-05-24 09:32:57 -05:00
Zuul
c47a9d5001 Merge "Replace git.openstack.org URLs with opendev.org URLs" 2019-05-24 08:03:47 +00:00
melissaml
a6322c6ed0 Replace git.openstack.org URLs with opendev.org URLs
Change-Id: I03e9162d5a59a2aa1631a9ecf6f6833bb7ac6050
2019-05-16 14:45:52 +08:00
Zuul
3d3ba26edd Merge "Use megabyte granularity for image extra space" 2019-05-15 06:51:10 +00:00
Logan V
87a18f51e3 Use megabyte granularity for image extra space
I want to use the new --image-extra-size flag[1] but my use-case
calls for megabyte granularity of this value. Rather than adding
60% to an 800MB image, maybe I only want to add 100 or 200MB, etc.

[1] https://review.opendev.org/#/c/655127/

Change-Id: I8fb9685d60ebb1260d5efcf03c5c23c561c24384
2019-05-15 13:38:25 +10:00
Dirk Mueller
c7ac6ee0cb Update test coverage for openSUSE/-minimal to 15.0
Use openSUSE 15.0 as default, which is the latest released stable
openSUSE release. Switch to https for accessing download.o.org
as encrypted transfers should be used by default.

Remove leftovers for definitely unmaintained openSUSE 13.x images
and split into old/new leap style versioning scheme for clarity.

Change-Id: Iab129eeee2b1a2563f0f0d2cb17bbad57c068e38
2019-05-08 14:59:51 +00:00
Zuul
4665e79245 Merge "openssh-server: harden sshd config" 2019-05-07 19:35:42 +00:00
Paul Belanger
5d60979e93 Use fedora-release-common for fedora 30+
It looks like fedora-release on fedora 30+ has been split into sub
packages. Use fedora-release-common to avoid package conflicts.

Change-Id: I8f8711044fc4074b91939e0a6dfdac4d7a14a35b
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2019-05-07 12:08:50 +00:00
Zuul
fa34eb7fe4 Merge "Support defining the free space in the image" 2019-05-07 10:14:01 +00:00
Zuul
8bf37a064e Merge "Allow specification of filesystem journal size" 2019-05-07 10:14:00 +00:00
Zuul
8c8b856c27 Merge "Only enable dbus-daemon for fedora-29 and below" 2019-05-07 05:47:57 +00:00
Paul Belanger
38d7574127 Only enable dbus-daemon for fedora-29 and below
In fedora-30 is when we migrate to dbus-broker, fedora-29 is still using
dbus-daemon.

Change-Id: I1e1d3a3826157b8b22386c211eaa58b6439b5f3c
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2019-05-06 17:49:42 +10:00
Paul Belanger
daf5a4e4bd Switch simple-init to support python3
Depending on the version of $DIB_PYTHON_VERSION, we can either use pip /
pip3 to install glean.  This is helpful for newer OSes that might not
want to ship python2 (pip).

Change-Id: I25c5927a1eb55ee16b919dd64403184f335839b6
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2019-05-02 19:38:16 -04:00
Tristan Cacqueray
11ec95b779 openssh-server: harden sshd config
Harden sshd configuration by adding KexAlgorithms, Ciphers and MACs for sshd,
following good pratices on https://infosec.mozilla.org/guidelines/openssh

Change-Id: I3051320d867a5033e82deef10c5e723ca9829884
Co-Authored-By: Nicolas Hicher <nhicher@redhat.com>
2019-05-01 11:42:21 -04:00
Tobias Henkel
778d007150 Support defining the free space in the image
Currently diskimage-builder supports two ways to specify the image
size. One is defining a fixed image size using DIB_IMAGE_SIZE, the
other one is auto-detection while adding a security margin of 60% as
free space. This means when building larger images (e.g. >100GB) with
unknown size upfront we end up with much wasted space, IO and network
traffic when uploading the images to several cloud providers. This can
be optimized by adding a third way by defining DIB_IMAGE_EXTRA_SIZE to
specify the free space in GB. This makes it possible to easily build
images of varying sizes while still minimizing the overhead by keeping
the free space constant to e.g. 1GB.

Change-Id: I114c739d11d0cfe3b8d8abc6df5ff989edfb67f2
2019-04-29 20:18:43 +10:00
caoyuan
0329a6de5e Replace git.openstack.org URLs with opendev.org URLs
Change-Id: Iac5a9da62db84365a769ea07146281866215a9c5
2019-04-29 20:15:25 +10:00
Logan V
11142f75b4 Allow specification of filesystem journal size
In many cases, the statically sized 64MB journal is far below the
e2fstools default calculation[0] which calls for a 64MB journal only
on filesystems smaller than 16GB. On bare metal in particular, the
correct default journal size will often be in the 512MB-1GB range.

Since we cannot know what the target system is, this should be a
tunable parameter that the user can set depending on the intended
image usage.

Add a DIB_JOURNAL_SIZE envvar and --mkfs-journal-size parameter
to the image creation so users can override the default journal
size.

[0] https://git.kernel.org/pub/scm/fs/ext2/e2fsprogs.git/tree/lib/ext2fs/mkjournal.c#n333

Change-Id: I65fa13a088eecdfe61636678578577ea2cfb3c0c
2019-04-29 17:00:30 +10:00
Zuul
6a4bf78e0c Merge "Fix Fedora aarch64 image location" 2019-04-18 12:22:06 +00:00
Zuul
5b47dc3a5b Merge "debian-minimal buster support" 2019-04-18 08:32:47 +00:00
Zuul
2205741de6 Merge "Also use selinuxenabled to check selinux status" 2019-04-18 08:32:15 +00:00
Pedro Alvarez
f034dd00d9 Fix Fedora aarch64 image location
It used to be considered a 'secondary architecture' but that
is not the case anymore.

Change-Id: I8e5e9cfa915c8a3c979ff9db26477c0542d271db
2019-04-09 15:50:55 +00:00
Ian Wienand
105d201e1f debian-minimal buster support
Due to the referenced bug, many versions of debootstrap can't bring up
a buster environment.  Unfortunately, these include versions we use to
do this on Xenial/Bionic nodes.

Also, there isn't backports or security updates, so elide these for
now.

I did get a working build (I haven't gone so far as a full boot+glean)
with this, at least.

Change-Id: If2420e92cb728ab6e91b0d70547da4483679b391
Paritial-Bug: #1822927
2019-04-04 16:10:08 +11:00
Serena Ziviani
19cc00041a Also use selinuxenabled to check selinux status
Currently, the cleanup script is using the existence of the folder
/sys/fs/selinux to check if SELinux is enabled. This, however, is
misleading in case disk-image-builder is used inside a Docker
container on a selinux-enabled host. In this case, the folder exists
in the container but SELinux is disabled.

This patch addresses the problem by checking, in addition to the
check already in place, the output of the command selinuxenabled.

Change-Id: I83e58f2467e60df9f0f00f7b7a58d0e2ce357a9a
Closes-Bug: #1820077
2019-03-28 14:20:24 +01:00
Zuul
36b4bc87f9 Merge "Minor clarifications in centos7 element docs" 2019-03-28 03:50:52 +00:00
Ian Wienand
20c5c98426 Replace openstack.org git:// URLs with https://
This is a mechanically generated change to replace openstack.org
git:// URLs with https:// equivalents.

This is in aid of a planned future move of the git hosting
infrastructure to a self-hosted instance of gitea (https://gitea.io),
which does not support the git wire protocol at this stage.

This update should result in no functional change.

For more information see the thread at

 http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003825.html

Change-Id: Id26bec14c3d94e2f81b2148fc85d17f07866398c
2019-03-22 01:35:42 +00:00
Daniel Abad
965d6f97aa Minor clarifications in centos7 element docs
Change-Id: I6aef77513efa37262269ca24b296acbdc823a039
2019-03-20 17:23:34 +01:00
Ian Wienand
5284564071 Unmount internal mounts on finalise errors
This is only one line, but it takes a lot to untangle ...  basically
the current "correct" path is:

---
 mk_build_dir()
  -> sets trap trap_cleanup EXIT

 ... stuff ..

 mount_proc_dev_sys
  -> mounts $TMP_MOUNT_PATH/<proc,dev.sysfs>

 pre-finalise.d
 finalise.d

 unmount_image $TMP_BUILD_DIR/mnt # nb == $TMP_MOUNT_PATH
  -> unmount_dir()
   -> recursive unmount everything inside TMP_MOUNT_PATH

 TMP_IMAGE_PATH=$(dib-block-device getval image-path)
 export TMP_IMAGE_PATH

 dib-block-device umount
 dib-block-device cleanup

 ... actually cleanup directories ...
---

Our current failure exit trap does:

---
 dib-block-device umount
 unmount_image
 ...
---

Note this is the *opposite* of what is done in the correct exit path.
In the failure case, if a script fails in the finalise stages it leads
to /proc, /sys, /dev etc. still being mounted inside the image; the
"dib-block-device umount" call doesn't know anything about these
mounts and tries to unmount the parent directory, and we get a hard
failure with a busy mount, and all the mounts are subsequently leaked.

Note that "unmount_dir", which is ultimately called by
"unmount_image", already knows to skip those mounts that
"dib-block-device umount" manages (this is the DIB_MOUNTPOINTS list).
This is further evidence it should be called *before* the
dib-block-device umount.

Change-Id: Ibef3ce9d1167b9c4ff3d5717b113cd3ed374f5e3
2019-03-13 16:38:49 +11:00
Zuul
bdfc13a5c0 Merge "[lvm] Add Ubuntu bionic as supported distro" 2019-03-11 09:19:49 +00:00
Zuul
186db05ffb Merge "Add DIB_APT_MINIMAL_CREATE_INTERFACES toggle" 2019-03-05 07:24:24 +00:00
Logan V
8756cbea1b Add DIB_APT_MINIMAL_CREATE_INTERFACES toggle
Add a DIB_APT_MINIMAL_CREATE_INTERFACES boolean to the debootstrap
element which functions identically to
DIB_YUM_MINIMAL_CREATE_INTERFACES in the yum-minimal element.

This can be used to disable the creation of the
/etc/network/interfaces.d/eth[01] dhcp configuration files, which
are not needed on systems where cloud-init or other means are used
to configure networking.

The flag is enabled by default to keep creating the dhcp interface
files, maintaining backwards compatibility.

Change-Id: I1fdaca8350a5ceefd9e437af4fd000ce6a3ee7f3
2019-03-05 16:27:57 +11:00
Gaëtan Trellu
bbde9bb320 [lvm] Add Ubuntu bionic as supported distro
The way how LVM is created on Ubuntu Xenial and Ubuntu Bionic
is the same.

Change-Id: I16d548f6393dd3cdfd5a9befa5c0ef0f6db92df1
2019-03-04 15:05:37 -05:00
Zuul
0f8d340c6c Merge "Update gentoo-releng gpg key" 2019-02-28 00:33:06 +00:00
Matthew Thode
b2cc91d276
Update gentoo-releng gpg key
A new signing subkey is used.

Change-Id: Idc9aceba7ee144fd0307737c24991acfacf68985
2019-02-27 15:18:25 -06:00
Zuul
703549412d Merge "update spelling errors" 2019-02-25 08:49:56 +00:00
Zuul
b2e2d121f0 Merge "set rhel minor release" 2019-02-25 08:11:56 +00:00
Noam Angel
8b83196024 Add option to skip update packages
in same cases it is required to avoid update all existing packages,
doing so can result in release update which is currently not possible
unless you not include "base" element.

"base" element used for most distribution (rhel, debain), and is
necessary for most cloud operations, this patch add 
"DIB_AVOID_PACKAGES_UPDATE" parameter to skip updating all packages.

usecases for this patch can be:
 * Avoid release update when building old release ex. RHEL7.5.
 * build on network-less environment.

usage:
DIB_AVOID_PACKAGES_UPDATE=1

or
DIB_AVOID_PACKAGES_UPDATE=0

Change-Id: I71192b23c8f0bc48b348fe7377bf8a2399b53792
2019-02-25 06:31:30 +00:00
Ian Wienand
37dff9738a Fix opensuse 42.3 pip-and-virtualenv
Related to I041a141366099093805e6052b1bbf64efd277e1e, we also need to
remove this on opensuse.  The files for gate testing are added, but
the test is not added to any jobs at this point in the interests of
gate time.

Change-Id: I1af9e84d76bedcb2607717edc6d2abe2920b0584
2019-02-25 15:37:17 +11:00
Quique Llorente
5b1844acf9 Keep git after ironic-agent post
New versions of pbr depends on git [1] and IPA depends on pbr [2] so
removing git will remove pbr, IPA and friends.

[1] https://src.fedoraproject.org/rpms/python-pbr/blob/master/f/python-pbr.spec#_64
[2] https://github.com/rdo-packages/ironic-python-agent-distgit/blob/rpm-master/openstack-ironic-python-agent.spec#L85

Closes-Bug: #1816017
Depends-On: https://review.openstack.org/637668
Change-Id: I97f6b593e88e1cb81cd4bb2d77787bc012fb8271
2019-02-19 16:32:22 +11:00
Noam Angel
802dc35a61 set rhel minor release
Change-Id: I52a38c16dbbbe9fa1d4d6b6daffde01f63f664e6
2019-02-10 14:59:41 +00:00
Zuul
25ba034a0e Merge "pip-and-virtualenv: handle centos image-based builds" 2019-02-07 07:54:23 +00:00
Ian Wienand
d0906ad473 pip-and-virtualenv: handle centos image-based builds
This fixes a regression in I041a141366099093805e6052b1bbf64efd277e1e
where we starting skipping the removal of old files for image-based
builds (confusingly named centos7 rather than centos for historical
reasons).  Fix the check

Change-Id: I74688a9e91d833b5d654056431729bed0585616c
2019-02-07 10:56:29 +11:00
Zuul
dd8bbd5c4f Merge "fix systemd import-tar for gentoo" 2019-02-04 02:15:09 +00:00
Zuul
46ac931513 Merge "pip-and-virtualenv : only remove system files on centos" 2019-02-02 01:38:48 +00:00
Ian Wienand
ea1735b6a2 pip-and-virtualenv : only remove system files on centos
As described inline, we only want to remove the system package files
on centos; it causes problems on Fedora where some system tools expect
these to be there.

But there is an additional bug -- pip actually removes the system
package files anyway.  To work around this, reinstall the system
package.

Closes-Bug: #1813232
Change-Id: I041a141366099093805e6052b1bbf64efd277e1e
2019-02-01 11:01:45 +11:00
Ian Wienand
7cb5916a76 Enable dbus-broker for Fedora 29
As described in the comments, it seems the transition between
dbus-daemon -> dbus-broker in Fedora 29 has made it so the packages
can get into a state where neither service is enabled.

Explicitly install and enable dbus-broker for F29

Change-Id: I06753043a75be2f635653899c6c251b9fbdd7c67
2019-01-31 18:08:37 +11:00
Matthew Thode
f4a1c7f89f
fix systemd import-tar for gentoo
use a newer version of systemd
Fixes https://review.openstack.org/#/c/608102

Change-Id: I23fd671adb893f3abd9fbc65382f2aec5a317c24
2019-01-27 18:16:51 -06:00
Zuul
379f1bdfc0 Merge "support cracklib in pam for Gentoo's musl profile" 2019-01-21 05:03:48 +00:00
Zuul
ad23eef69c Merge "Change phase to check for dracut-regenerate in iscsi-boot" 2019-01-18 09:57:14 +00:00
Matthew Thode
1bde2591ae
support cracklib in pam for Gentoo's musl profile
Needed for any musl build, is default in other profiles

Change-Id: Ib7cae9124f5846d33c05f26befd8f13646a08610
2019-01-15 09:47:46 -06:00
Michael Johnson
cfba9ea79d Make sure $TMP_BUILD_DIR/mnt is owned by root
The path $TMP_BUILD_DIR/mnt becomes the / inside the chroot during
the chroot phases of diskimage-builder. Previously this path was being
created using the account running diskimage-builder. This account may
not be valid inside the chroot. This causes path validation, when running
on a Ubuntu bionic host, to fail.
This patch chown's the $TMP_BUILD_DIR/mnt to root.root to make sure
that / is owned by a valid account inside the chroot.

Change-Id: Ifedc136baa67c7952942aed2c8cb1041902fef91
Closes-Bug: 1811113
2019-01-09 20:08:15 -08:00
Zuul
b0d41230e4 Merge "Delete the duplicate words in 50-zipl" 2019-01-10 00:17:53 +00:00
Zuul
8d3fa3a85c Merge "simple-init: allow for NetworkManager support" 2019-01-09 03:38:25 +00:00
Zuul
18c0c42c8d Merge "package-installs: provide for skip from env var" 2019-01-09 03:34:08 +00:00
Zuul
234f000e6b Merge "Update to Fedora 29" 2019-01-08 23:56:43 +00:00
Matthew Thode
cf786bb175
change to python36 for gentoo
fixes build issues since dev-util/glib-utils was updated

Change-Id: Ie8b5c425f846619ab4fc07f5bd1902dc83172a59
2019-01-08 04:59:08 -06:00
weiyj
64a8fc7c58 update spelling errors
Change-Id: Ic206b8247acce1975409329faa29deccd4f886de
2019-01-08 14:31:06 +08:00
Zuul
31f3e9dbc3 Merge "source-repositories: Replace documentation http with https links" 2019-01-08 05:16:18 +00:00
chengebj5238
079a104c1a source-repositories: Replace documentation http with https links
Change-Id: I7db3f2a1ed4e0460db60635ab00367050b0300a5
2019-01-08 15:49:41 +11:00
zhouxinyong
e3d6b8b0ec Delete the duplicate words in 50-zipl
Change-Id: Icaa3678f8b46f2a02eb9254753ed30ab8215aa7f
2019-01-07 10:02:35 +08:00
Yolanda Robla
853e13c6c3 Change phase to check for dracut-regenerate in iscsi-boot
There is an use of get_image_element_array on the environment.d
phase, for the iscsi-boot element.
This function is not available on that step. So moving the check
at next step, extra-data-d, where it is available.

Change-Id: I89cfe565492142c2f7962109360fcbcebadfd469
2019-01-02 11:05:00 +01:00
Zuul
56c72a0139 Merge "Add an element to configure iBFT network interfaces" 2018-11-30 13:46:05 +00:00
Zuul
58790e15f2 Merge "Add missing ws separator between words" 2018-11-30 03:53:49 +00:00
Ian Wienand
8ec3750dda simple-init: allow for NetworkManager support
This plumbs through an "--use-nm" flag to glean which instructs it to
setup interface bringup with NetworkManager rather than legacy network
enablement scripts.

In this case, install the NetworkManager package.  In the non-nm case,
also install the network-scripts for Fedora 29 -- this has stopped
being installed by default (it's been deprecated since forever).

As noted in the docs, this is currently really only relevant on the
supported rpm distros which are using the ifcfg-rh NetworkManager
plugin to effectively re-use old config files.  However,
NetworkManager has similar plugins for other platforms, so support can
be expanded if changes are proposed.

Depends-On: https://review.openstack.org/618964
Change-Id: I4d76e88ce25e5675fd5ef48924acd09915a62a4b
2018-11-30 10:02:47 +11:00
Ian Wienand
c52c383f1b package-installs: provide for skip from env var
Provide a "when" option that provides for not installing packages
based on a = or != match on an environment variable.

Unit tests are added.

Change-Id: Ifa824dccaff69fd447f45d54cb4a3083bcabdd86
2018-11-30 10:02:47 +11:00
Zuul
3eab481ab8 Merge "Fix a typo in the help message of disk-image-create" 2018-11-29 11:42:19 +00:00
Ian Wienand
0da1d3a419 Fix unit tests for elements
It looks like we dropped running these probably when we moved the
elements around.  For testtools to find the test scripts we need to
add the __init__.py files to make the directories look like modules.
Also prevent copying any .pyc or cache files in as hooks.

Change-Id: I66d5f6ee62cc4d9ee14c64e819b4db57d035d09f
2018-11-28 11:04:50 +11:00
Pierre Riteau
a64aa0cb47 Fix a typo in the help message of disk-image-create
Change-Id: I092e5ea88747b80c0e59c0aea4301d19009e0241
2018-11-23 09:09:31 +00:00
zhufl
b5f247b04f Add missing ws separator between words
This is to add missing ws separator between words.

Change-Id: Ie192d296128fb785c344ac5d8a77cad59764080e
2018-11-21 16:07:59 +08:00
Dmitry Tantsur
f0f3e3bac4 Add an element to configure iBFT network interfaces
This allows nodes with remote devices configured via iBFT to be
correctly used during Ironic introspection and deployment,
at least for non-multipath configurations.

The new element is added as a dependency for ironic-agent.

Change-Id: If3dac6504d26535593f12e851092065b688ef696
2018-11-20 14:11:11 +01:00
Zuul
d9d59b70da Merge "move selinux-permissive configure to pre-install phase" 2018-11-20 10:34:42 +00:00
Noam Angel
6f1f60983f move selinux-permissive configure to pre-install phase
install-packages is running before install.d phase, there is a chance
that installing a package like "container-selinux" will failed the
build, moving "selinux-permissive" to run at pre-install stage make
more sense.

Change-Id: I32f988be725d4b385c3765c47a00cd57c53d7d71
2018-11-19 13:13:57 +11:00
Zuul
d591c53196 Merge "fix some errors for ill-syntax in README.rst" 2018-11-19 01:59:49 +00:00
Ian Wienand
5085f23fdf Update to Fedora 29
Update builds to Fedora 29.  Remove the openstack gate CI mirror
workaround for pre-28 versions as they're not building in the gate any
more.

Change-Id: Ia6a8ae8d66d69f6add39e571043328e7274ba26c
2018-11-16 09:05:08 +00:00
Zuul
04806ea971 Merge "Increase size of EFI system partition" 2018-11-15 10:14:26 +00:00
zhouxinyong
621cbfca26 fix some errors for ill-syntax in README.rst
Change-Id: I0daf36848e8ddb09fcae6cbd55f460a8e49d209c
2018-11-13 15:05:51 +08:00
Pierre Riteau
257f9f4d1d Increase size of EFI system partition
8 MiB is not enough when using the grub2 element with centos7 images,
which installs binaries from the shim-x64 and grub2-efi-x64 packages
under /boot/efi. 12 MiB ought to be enough for anybody.

Change-Id: I8594d1fe05242f246a5809740a115ab2f84ac5a3
2018-10-31 14:38:28 +00:00
Jesse Pretorius
d59a0c8786 Add ubuntu-systemd-container operating-system element
In order to allow the simple preparation of base images which
can be used for LXC/nspawn machine containers, we add this
element.

Containers inherit a kernel from the host, so there is no need
to build a kernel into the image. All the element needs is a
base init system which, in this case, is systemd.

Change-Id: I45651de2aa1b19bdeee301094f0bdffdd0a3b45c
2018-10-31 14:22:28 +11:00
Zuul
16d5c4280b Merge "Turn on quiet mode when logfile specified" 2018-10-31 00:15:27 +00:00
Zuul
f8c12712cc Merge "Native zuulv3 tests" 2018-10-29 05:45:18 +00:00
Ian Wienand
36d642a6a0 Native zuulv3 tests
This finalises the ports of the legacy jobs to zuul native jobs.

The dib-setup-gate-mirrors role preconfigures the repo templates,
etc. for the openstack-ci-mirrors element.

The dib-functests role runs the tests as specified by dib_functests,
and can run under python2 or 3.

Change-Id: Ied67a31f0d31503d13eccad8662c29740c93f33e
2018-10-29 12:46:15 +11:00
Zuul
1785bd3975 Merge "Fix epel repo rewrite, add to testing" 2018-10-28 23:41:21 +00:00
Ian Wienand
86d5534352 Turn on quiet mode when logfile specified
I'm not really sure why I originally had --logfile also log to stdout
in I202e1cb200bde17f6d7770cf1e2710bbf4cca64c, but it seem
counter-intuitive (indeed, I just tripped myself up thinking that in a
devstack job "--logfile" would put the logs into a separate file and
avoid the stdout logging, and I wrote it!).

Make it so specifying a --logfile puts dib into quiet mode for stdout.
Explicitly overriding DIB_QUIET will allow both if someone wants that.

Change-Id: I3279c9253eee1c9db69c958b87a0ce73efc0be9b
2018-10-24 12:40:09 +11:00
Ian Wienand
dfd1e4fdaf Fix epel repo rewrite, add to testing
It looks like epel-release switch from "mirrorlist" to "metalink"
(around release 7-10 Jun 2017 according to [1]).  Update our rewrite
matching to handle this "metalink" as well.

Add epel element to the centos7 (image-based) build for testing too

[1] https://koji.fedoraproject.org/koji/buildinfo?buildID=978473

Add epel element so it's tested during the centos7 functional test.

Change-Id: I2d6d4c2ec47bc69d2f16c96b5045b05c435a1af9
2018-10-24 12:39:59 +11:00
Zuul
6d3d830cab Merge "Add a post-root.d phase" 2018-10-23 09:16:35 +00:00
Zuul
668978b0c2 Merge "Remove redundant sources change/update" 2018-10-23 09:16:35 +00:00
Tobias Henkel
eff5b2312b
Add a post-root.d phase
While trying to get docker image pre-caching to work we couldn't get a
docker daeomon to run within the chrooted environment. However we got
docker running with the help of bwrap outside of the chrooted
environment. The only option so far for this is the block-device.d
phase. But this has the problem that it runs after the image size has
been calculated. This leads to broken builds if the docker images
being pulled are big.

This can be solved by adding a post-root.d phase that runs outside the
chroot but before the image size calculation.

Change-Id: I36c2a81e2d9f5069f18ce5b0d52c5f1c7212c3ae
2018-10-19 10:33:56 +02:00
Zuul
ec025ff03d Merge "Fix DIB_DISTRIBUTION_MIRROR_UBUNTU_IGNORE regex typo" 2018-10-19 04:01:21 +00:00
Zuul
07d263de0b Merge "Allow debootstrap to cleanup without a kernel" 2018-10-19 03:23:09 +00:00
Zuul
22cd19fea7 Merge "ubuntu-common: Update default DIB_RELEASE to bionic" 2018-10-19 03:23:08 +00:00
Zuul
39c6d7c747 Merge "Move common ubuntu environment setting to ubuntu-common element" 2018-10-19 02:50:36 +00:00
Zuul
46ec4992ff Merge "Add support for Fedora 28, remove EOL Fedora 26" 2018-10-18 23:44:51 +00:00
Ian Wienand
86080a7db0 Fix DIB_DISTRIBUTION_MIRROR_UBUNTU_IGNORE regex typo
This should not be quoted.  Introduced in
I6ffbde07fa0e103641ee5c5f9d9e854e5b2168dc

Change-Id: Iac54c86ebdb90351a2dd663b876ab4675ee0435a
2018-10-18 21:49:44 +11:00
Paul Belanger
7173658df2 Add support for Fedora 28, remove EOL Fedora 26
This updates diskimage-builder to support current Fedora releases (27
and 28) and removes support for Fedora 26 which is EOL as of June
2018.

Change-Id: I602b22ed4d5397b39dc1eef67964f6fbdcd93060
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2018-10-18 19:43:12 +11:00
Zuul
e09faf9a8d Merge "simplify overlay logic for Gentoo" 2018-10-18 05:51:05 +00:00
Zuul
b3d405f624 Merge "simplify python3.6 selection on gentoo" 2018-10-18 05:51:05 +00:00
Zuul
763493e1f1 Merge "enable caching for gentoo builds" 2018-10-18 05:51:03 +00:00
Zuul
2d327c2506 Merge "Set EPEL mirror during openstack-ci-mirrors" 2018-10-18 04:40:33 +00:00
Zuul
fe15b197fb Merge "ubuntu: Add options to ignore mirror components and use insecure repos" 2018-10-18 04:34:00 +00:00
Zuul
56634e0938 Merge "Turn down pkg-map and hook copy tracing output" 2018-10-18 03:56:12 +00:00
Zuul
d3d17b2e43 Merge "Fix DIB ubuntu-minimal running on bionic (18.04)" 2018-10-18 03:08:37 +00:00
Ian Wienand
8b0111477e Set EPEL mirror during openstack-ci-mirrors
We want to set DIB_EPEL_MIRROR for the epel element, which means we
need access to the CI mirror info script in the chroot.  Copy it into
the temp directory with extra-data.d and update the environment file
to find it.

Change-Id: Ia12f0cbdeb194eef3155497ceb5ffc4a452aad76
2018-10-18 13:53:53 +11:00
Ian Wienand
c47ee6e121 ubuntu: Add options to ignore mirror components and use insecure repos
When using the upstream cloud images with the "ubuntu" element, they
have universe and multiverse enabled which we don't mirror.

To use the infra mirrors as a DIB_DISTRIBUTION_MIRROR with this
element, we need to be able to skip redirecting to universe and
multiverse, and additionally enable insecure repos (as we don't gpg
sign our mirrors).

Add and document two new variables with the ubuntu element to do this.
This is then setup by the openstack-ci-mirrors element so that we use
local mirrors duing dib functional testing for the "ubuntu" element.

Change-Id: I6ffbde07fa0e103641ee5c5f9d9e854e5b2168dc
2018-10-18 13:53:48 +11:00
Matthew Thode
3ad8235180 simplify overlay logic for Gentoo
openssl/cryptography versions are updated/stable
musl profiles need newer versions of open-iscsi masked as upstream
doesn't want to work with multiple libcs

Change-Id: If5baf339516390ae332015928557c6bb734486c2
2018-10-18 00:21:31 +00:00
Matthew Thode
1e257f59cc simplify python3.6 selection on gentoo
python 3.6 is now stable in gentoo

Change-Id: Ide96c039b03817c216ab58930ff5c897556e6c8a
2018-10-18 00:21:16 +00:00
Ian Wienand
6c077d7c2a Turn down pkg-map and hook copy tracing output
This is a lot of very low value noise in the logs as these iterate
through all the elements (often doing nothing).  Turn it down and add
an echo so we just see what elements it is working on.

Change-Id: I0687de4722766189db9d4a7bd7d3cfb45d387b62
2018-10-18 11:03:17 +11:00
Matthew Thode
89f95a5351
enable caching for gentoo builds
To facilitate this I've created two new environment variables to set
the environment and default options for package actions.

eval is needed for the export as it preserves quotes.

Change-Id: Ib03651ee8dacd48cd1c135afd57cd31101356056
Signed-off-by: Matthew Thode <mthode@mthode.org>
2018-10-15 23:17:37 -05:00
Zuul
8add0cb398 Merge "Add a pre-finalise.d phase" 2018-10-15 06:01:48 +00:00
Ian Wienand
fadf99af05 Add a pre-finalise.d phase
In exploring Gentoo caching, it was realised that we have no way to
bind mount the cache into the finalised image for the finalise.d
phases.

By adding a pre-finalise.d phase that runs outside the chroot, we can
mount outside things into the hierarchy at $TMP_BUILD_DIR/mnt which
are then seen by the in-chroot finalise.d phase.

This is similar to the pre-install phase

Change-Id: I9d782994843383ddf90f62c40498af9925fd9558
2018-10-15 12:45:23 +11:00
Zuul
91efed30ef Merge "Minor documentation updates" 2018-10-15 00:37:18 +00:00
Ian Wienand
ee156e8232 Minor documentation updates
Some minor things after looking at these parts.

The dib-run-parts element doesn't do any of the copying any more, so
these comments are wrong.

The reason for the multiple mounts in the bind mount was non-obvious
to modern eyes (as util-linux has handled this for some time).

Formatting fix for the rst

Change-Id: Idb4c9ff32c49aced2c68a5c905bf7a8b2832a5a2
2018-10-12 11:51:56 +11:00
Zuul
bbbade0e4c Merge "Fail build due to missing kauditd only when SELinux is enabled" 2018-10-11 09:46:48 +00:00
Ian Wienand
39fb794ebc Allow debootstrap to cleanup without a kernel
Currently there's more-or-less an assumption that a kernel is
installed, so module blacklists are simply echoed into the modprobe
blacklist.  This may not be the case with some ongoing container work.

Although we don't need to blacklist modules for containers, it also
doesn't hurt.  Move the debootstrap element to the new modprobe
element, and allow it to create the blacklist directory.

Change-Id: I0f057caf473951df56a2af9633e3a5b53e0809b1
2018-10-08 16:02:25 +11:00
melissaml
b85bc24761 fix a typo
Change-Id: I4867c39a2995e35a0a1e9cba8f786cf4e6188141
2018-10-05 17:26:04 +08:00
Pierre Riteau
bacceba41d Fail build due to missing kauditd only when SELinux is enabled
With the check added in commit 7566819139,
diskimage-builder fails to build RPM-based images if kauditd is not
running. However, this is only valid for environments where SELinux is
enabled. If SELinux is disabled (which is identified by an empty _runcon
variable), proceed with running setfiles.

Change-Id: I1b056f20a3a55f7333391207d9e1049d25ece041
Closes-Bug: #1779273
2018-10-05 10:02:14 +01:00
Zuul
3e6d658687 Merge "allow building non-gentoo images on gentoo hosts" 2018-10-03 11:08:11 +00:00
Michael Johnson
330626482a Fix DIB ubuntu-minimal running on bionic (18.04)
Diskimage-builder fails to build ubuntu-minimal images when run on
a Ubuntu bionic-beaver (18.04) instance.
The user gets "Couldn't create tempfiles for splitting up" when
apt-get update is run in the ubuntu-minimal element root.d.

The issue is that the /tmp inside the chroot is not getting the
proper permissions applied from the base-files package. This is likely
because the pip-cache element has already created the directory before
the base-files package is installed.

This patch changes the order of pip-cache to root.d/11-pip-cache so that
it runs after teh base OS root.d elements run.

Change-Id: I6fd1cb2a23422206884165eb502b260f0c1e52f7
2018-10-01 19:59:30 -07:00
Jesse Pretorius
f4c5326b8e ubuntu-common: Update default DIB_RELEASE to bionic
The ubuntu-minimal README states that the latest Ubuntu LTS
is the default, but currently that is not true. This patch
changes the default to the current LTS.

Change-Id: I10f28314d1a5969c20094194637cfe31219d228c
2018-09-20 02:06:04 +01:00
Jesse Pretorius
8c69da65ac Remove redundant sources change/update
The apt sources are set out in root.d/75-ubuntu-minimal-baseinstall
and the cache is updated, cleaned and a dist-upgrade is done there.

As such, this file is unnecessary.

Change-Id: Idab5ede3f235bc204c4bdebf40fbcf4a12e5bc2f
2018-09-20 00:05:07 +00:00
Jesse Pretorius
30493f7655 Move common ubuntu environment setting to ubuntu-common element
The ubuntu, and ubuntu-minimal elements both make use of a common set
of environment settings to determine the distribution name.

The ubuntu-minimal element also does a few extra things which would
appear to apply to both sets and bring in extra architecture support.
As such, these are included in the common element.

This intends to be part of a series of patches which will eventually
create a new element to build a minimal ubuntu-systemd-container
element which can be used for lxc/nspawn containers.

Change-Id: Ia4e620f7d3fa6215484a8d218cea2f28bd1ffaee
2018-09-20 00:04:55 +00:00
Zuul
b86af3dc6a Merge " IPA requires iptables" 2018-09-13 02:05:42 +00:00
Ian Wienand
f6a2452d4c Only append DIB_BOOTLOADER_DEFAULT_CMDLINE to default grub entry
The grub.cfg has two variables [1]

 GRUB_CMDLINE_LINUX : used on all boots
 GRUB_CMDLINE_LINUX_DEFAULT : additionally used on all "normal" boots

The problem with I2298675dda1f699c572b3423e7274bc8bd7c1c9d is that it
appened the values in DIB_BOOTLOADER_DEFAULT_CMDLINE to both of these,
resulting in duplicated arguments.  I don't think we considered that
GRUB_CMDLINE_LINUX_DEFAULT actually already appends to the
GRUB_CMDLINE_LINUX values.

Make DIB_BOOTLOADER_DEFAULT_CMDLINE only append itself to
GRUB_CMDLINE_LINUX_DEFAULT.  That seems to line up sensibly with the
name of the variable.

Documentation is enhanced around this, and a releasenote added.

[1] https://help.ubuntu.com/community/Grub2/Setup

Change-Id: I76b5442a9090c19a6540ed2d4ab324546f241ebf
Closes: #1791736
2018-09-13 09:51:50 +10:00
Zuul
b29aee3383 Merge "Install sudo on Gentoo images by deault" 2018-09-11 00:09:33 +00:00
Zuul
9f93b2ce3d Merge "Fix CentOS image build failure when dib runs on system where audit disabled" 2018-09-06 11:24:59 +00:00
Zuul
8473e2e373 Merge "Replace assertRaisesRegexp with assertRaisesRegex" 2018-09-06 11:22:20 +00:00
Olivier Bourdon
7566819139 Fix CentOS image build failure when dib runs on system where audit disabled
Without this fix, building a CentOS image on Ubuntu where audit=0 is passed
as a kernel boot parameter will lead to the following error:
disk-image-create centos7 dhcp-all-interfaces cloud-init-nocloud \
    devuser yum epel baremetal
... dib-run-parts Running tmpdir/hooks/cleanup.d/99-selinux-fixfiles-restore
... Error connecting to audit system.

Change-Id: I229d9b72f88bffddca42da57f01c27e902427071
2018-09-04 08:23:02 +02:00
Charalampos Kominos
c85141291e Fix bootloader packages for aarch64
Due to the arm naming convention, building centos images for arm64 and
aarch64 does not yield the same result. In order to locate grub2 on
aarch64 the correct mapping is added.

Change-Id: I1bb227b2523e420e394fec8c52c6c79fcdd31c53
Closes-Bug:#1789414
Signed-off-by: Charalampos Kominos <Charalampos.Kominos@enea.com>
2018-08-31 17:01:47 +02:00
Chuck Short
b67cfc6950 Replace assertRaisesRegexp with assertRaisesRegex
This replaces the deprecated (in python 3.2) unittest.TestCase method
assertRaisesRegexp() with assertRaisesRegex().

Change-Id: I0ac482741ad4adc1127dd9e9f64c1c8101f370e5
Signed-off-by: Chuck Short <chucks@redhat.com>
2018-08-27 10:40:36 -04:00
Zuul
b602c05551 Merge "Add netcat to redhat-common map-packages" 2018-08-23 11:26:12 +00:00
Zuul
f33526d20c Merge "better handle existing keywords files/directories" 2018-08-10 06:07:06 +00:00
Carlos Goncalves
138b14b010 Install ca-certificate with redhat-common
Change-Id: I36d44ca8b4f966476657ec2bb1ecc1458bb524fd
2018-08-04 00:30:49 +02:00
Carlos Goncalves
cf553ce0ac Add netcat to redhat-common map-packages
Change-Id: I273038611febe5e30f30dd8d3ba8990dbdc94647
2018-08-03 17:20:19 +02:00
Zuul
3be4b0c1fd Merge "Only detach device if all partitions have been cleaned" 2018-07-31 08:21:27 +00:00
Zuul
3197a7ef1b Merge "Move LVM cleanup phase into cleanup" 2018-07-31 00:30:47 +00:00
Zuul
53186f23a0 Merge "cache-url requires curl" 2018-07-31 00:20:56 +00:00
Zuul
d50bd1deb3 Merge "Don't quote names with sgdisk" 2018-07-30 06:26:25 +00:00
Yolanda Robla
64bb87f7b5 Only detach device if all partitions have been cleaned
Currently there is a bug, that tries to detach the device from a
partition at the first try, without considering that there may be
other partitions and volumes on it. Ensure that the detach is done
properly, and add a test to ensure that this happens correctly.

Change-Id: I35c5a473509f17a70270a2cbf5bf579faaeb123a
Fixes-Bug: #1777861
2018-07-30 16:24:57 +10:00
Ian Wienand
7302f38f97 Move LVM cleanup phase into cleanup
A recap -- we run umount phase then cleanup phase.

Currently we register a object to do the final LVM cleanup based on
the parent PV.  In light of I697bfbf042816c5ddf170bde9534cc4f0c7279ff,
I believe this should just be done in the cleanup phase.  Note there
was probably additional confusion because the partition removal was
done in the cleanup phase until
I7af3c5cf66afd81a481f454b5207af552ad52a32, where is was moved into the
umount phase.

Thus it is moved into the cleanup() function and this should now run,
per the comment, after everything is unmounted in umount phase.

This also exposes that we didn't have the cleanup phase in the unit
tests (because it wasn't doing anything I guess).  Add it.

Change-Id: I1c5f4ffc9619c774f78d21b918a81647b3dc28f5
2018-07-30 14:35:16 +10:00
Oliver Walsh
a514feba99 Add DIB element to blacklist nouveau
This ensures nouveau is not loaded at boot, which is required when installing
NVIDIA GPU drivers and to avoid issues with PCI passthrough of NVIDIA GPUs.

The option to disable kernel modesets ensures that it can be unloaded again if
it happens to be loaded after boot (e.g manually or implicitly by X).

bp tripleo-vgpu

Change-Id: I60815de86e7b22dfb39555af9d2d53564841e2ab
Related-bug: 1774674
2018-07-25 16:35:58 +01:00
Oliver Walsh
73e27a8504 modprobe DIB_MODPROBE_BLACKLIST should be optional
modprobe element currently fails when DIB_MODPROBE_BLACKLIST is not set.
As there are now two methods to control blacklisting this should be optional.

Change-Id: Ibf3c31a95177ba88c1b93228490c7f36f5b70b57
2018-07-25 16:33:47 +01:00
Zuul
48645abff6 Merge "Call kpartx remove in umount, not cleanup" 2018-07-24 23:05:16 +00:00
Zuul
0a40f45094 Merge "Move localloop to exec_sudo" 2018-07-24 23:05:15 +00:00
Zuul
961235854b Merge "block-device lvm: fix umount phase" 2018-07-24 11:26:11 +00:00
Zuul
9adf12fe4a Merge "Fix for proper LVM support" 2018-07-24 07:32:11 +00:00
Clint Byrum
d8907e78b5 cache-url requires curl
In some cases cache-url can get pulled in without curl, causing it to
fail.

Co-Authored-By: Adam Harwell <flux.adam@gmail.com>

Change-Id: Ibd66c2ca4f8cc423783555d8a99b1184f43adff2
2018-07-23 09:56:58 +00:00
Ian Wienand
1107326723 Update pylint to 1.7.6, uncap networkx
This review squashes:
    Iac9afc7766d3640815dc20cfd6de1245d36a09cc
    Ie894b5801bd7b3815432882cd626941e89d9f9a1

We need to do this as we can't fix pylint without networkx as that
failes requirements-chak due to us having a cap on networkx and we can't
uncap networkx as part of tripleo-buildimage installs without
constratints which gets us 2.1 and DIB desn't support 2.x

This is the commit message Iac9afc7766d3640815dc20cfd6de1245d36a09cc
---
One of the pylint dependencies has updated to be python3 only; this
version of pylint correctly caps things so it still works with
python2.

This also exposes that we need to uncap networkx due to
I34045f87ca19c2f184b040f4d89347374cce518b.  We should remain on
version 1 for now thanks to upper-constraints, but we need to maintain
the lower-constraint.
---

This is the commit message Ie894b5801bd7b3815432882cd626941e89d9f9a1
---
Support different versions of networkx

Since the entry of networkx 2.0 nodes has a different
behaviour. Checking if dg.nodes is iterable is enough to add
compatibility for new/older versions.
---

Change-Id: I82dc61fac6c156a4f0d574290c7632077aa53195
2018-07-18 09:27:01 +10:00
Zuul
b79952af2a Merge "Add new modprobe element" 2018-07-11 07:21:40 +00:00
Zuul
bbf69a90f3 Merge "Fix /etc/network/interfaces file contents" 2018-07-10 07:46:56 +00:00
Zuul
2343d4b577 Merge "Add keyring if supplied" 2018-07-10 07:46:55 +00:00
Olivier Bourdon
caf565673b Fix for proper LVM support
Without this fix, a LVM based ubuntu-minimal image will fail
booting due to the fact that the boot process will not be able
to retrieve the root filesystem using LABEL=(cloud)img-rootfs

Change-Id: If4ecf65868563f7b799160a58af6312bedf956bf
2018-07-09 14:15:57 +00:00
Hironori Shiina
7e4e6cfff4 Add expected semicolons for dhclient.conf
This patch adds an expected semicolon to an end of statement in
dhclient.conf for dhcp-all-interfaces element. Without this fix, an
error occurs when an image is booted with a message,
'semicolon expected.'.

Change-Id: I8311dbc67cc2815223111da01e7a7517c7d6f059
2018-07-06 13:42:25 +09:00
Sam Yaple
c144246cc9
Add keyring if supplied
When building with debootstrap, debootstrap will use the key to check
that everything is properly signed. It will not `apt-key add` the key
into the final environment, however.

Early adding the key after debootstrap before we need to read from the
private repo again prevents unsigned issues. This also maintains the
integrity of the packages in the environment throughout the build.

Change-Id: I5ca75ae4620c9fb26b512cb30f8cd79fa7a0373a
2018-07-02 14:33:35 -04:00
Ian Wienand
f94943344f Call kpartx remove in umount, not cleanup
Similar to I697bfbf042816c5ddf170bde9534cc4f0c7279ff, the order of
things called is "dib-block-device umount" *then* "dib-block-device
cleanup".

Because we're doing the "kpartx -d" here in cleanup, it means that the
loop-device is removed in umount phase from level0/localloop.py, then
afterwards we try and remove the partitions.

Change-Id: I7af3c5cf66afd81a481f454b5207af552ad52a32
TODO: a test case to ensure the ordering
2018-06-29 11:22:33 +10:00
Ian Wienand
a1a549548a Move localloop to exec_sudo
One call in localloop requires the output of the command, so modify
exec_sudo to buffer up output and return it.  This is modelled on the
same thing in package-installs-v2 which seems to work.  Rather than
return a subprocess exception, return a dib exception which everything
should have imported anyway.

The overall reason for this is to make our external calls more
consistent for mocking in unit testing.

Change-Id: I10d23b873dee9f775daef2a4c8be5671d02c386e
2018-06-29 11:22:24 +10:00
Zuul
927e8115f6 Merge "Fix bootloader for efi on rhel systems" 2018-06-28 15:02:30 +00:00
Yolanda Robla
31383970c7 Add new modprobe element
This element will replace modprobe-blacklist element. It wil
still have the blacklist functionality, but it also adds
the feature of passing a complete file with settings to the
modprobe.d directory. Adding this functionality, that will
allow elements that depends on this module, to just copy the
specified files to the final directory.

Change-Id: I9a44f7d11520b8b1e604956d3c1db2fc7e2bf457
2018-06-28 13:55:53 +02:00
Andreas Florath
f5736f3178 block-device lvm: fix umount phase
As described in blockdevice.py detachment and (most) resources
release must be done in the umount phase of a block device module.

Until now these jobs were done in the lvm cleanup() phase - which
is too late - especially when using nested LVMs.

This patch moves the functionality of the cleanup() phase to the
umount() phase for the lvm module.
It includes a test case that fails without applying the provided
source code changes.

Change-Id: I697bfbf042816c5ddf170bde9534cc4f0c7279ff
Signed-off-by: Andreas Florath <andreas@florath.net>
2018-06-28 15:21:59 +10:00
Zuul
e796b3bc18 Merge "Add iscsi-boot element for CentOS images" 2018-06-27 09:54:10 +00:00
Ian Wienand
b0da703f46 Don't quote names with sgdisk
Our sgdisk calls are putting extra double-quotes around the names of
partitions.  This confuses sfdisk, which confuses growpart, which
confuses growroot ... and you don't get your partition grown for EFI
boot.

Ensure we just bunch arguments into the list directly (for Popen)
rather than string split and have to worry about quoting.  Add a check
for this to our GPT unit test, extending it to include a space in the
name of the root partition.

Change-Id: I0a8cb69bb4c9c0865fbaa63ba0d7210028da552e
2018-06-27 18:10:08 +10:00
Matthew Thode
b9f1c7a22f
better handle existing keywords files/directories
The existing directories are needed for stage building (a part of the
Gentoo build process).  Normally these directories are empty, but there
are times where overrides need to be defined.  This commit handles
existing overrides for keywords.  For historical reasons the overrides
were able to be put in different files and directories, this
centralizes them.

This also updates the version of openssl/cryptography that works with
or without bindist.

Change-Id: I62c934ed305a711a4a9a3ef01fa55ad142aebb78
2018-06-25 09:20:52 -05:00
Hoang Trung Hieu
a4648872ba Add iscsi-boot element for CentOS images
This patch adds an element that handles the configuration for
creating a disk capable of being a remote root filesystem through
iSCSI on CentOS images.

Tested on Fujitsu Server and boot with BIOS and UEFI mode successfully.
- Tested Boot-From-Volume + EFI for centos7 with following elements:
  "centos7 vm devuser cloud-init-datasources dhcp-all-interfaces
   iscsi-boot dracut-regenerate block-device-efi"

Co-authored-By: Nguyen Van Trung <trungnv@vn.fujitsu.com>

Change-Id: Ia1f23d722dced6f254fd7aee86abe8066a72fa42
2018-06-25 12:01:52 +00:00
Zuul
e39adcd65f Merge "Remove redundant word" 2018-06-22 14:23:47 +00:00
chenxiangui
c809160906 Remove redundant word
Remove the redundant word 'the' in config.py

Change-Id: I3e9cb6390ce196f0a9022aef10f6c7b1ace36c48
2018-06-19 18:00:11 +08:00
Olivier Bourdon
11d91501d0 Fix /etc/network/interfaces file contents
According to http://bit.ly/2HA4oDO and
the official Ubuntu manual
http://manpages.ubuntu.com/manpages/xenial/man5/interfaces.5.html
source-dir support has been removed from Ubuntu >= 16.04/Xenial

Once an image is generated and booted, moving the dhcp interface(s)
declaration(s) from /etc/network/interfaces into specific subentries
of /etc/network/interfaces.d and calling 'service networking restart'
just make your instance unreachable and all interfaces are left
unconfigured.

This patchset fixes this issue

Change-Id: I6b6b99c81490c874c5db5405c2fbf3c180c87464
2018-06-19 11:26:21 +02:00
Yolanda Robla
9687a1efe1 Convert labels to upper case
When booting on UEFI, there was an issue mounting the vfat
filesystem. It was caused because the mount was defined in
/etc/fstab in lowercase, but the disk had it labeled in upper
case, and system could not find it. Conver the label to upper
case in case of fat/vfat.

Change-Id: Id3dee735e6f8fb221d199c4aba648f3e9a6e4206
2018-06-19 11:12:54 +02:00
Yolanda Robla
61e6566c48 Fix bootloader for efi on rhel systems
When building the image on a non-efi environment, it generates
linux16/initrd16 entries. But to boot from UEFI they need to have
linuxefi/initrdefi entries.
Use sed to replace those entries, in case we have an EFI image.

Change-Id: I47c96450e10f34b91bcc32888532bd7ab87cf316
2018-06-19 08:37:30 +02:00
Zuul
e210f79500 Merge "Don't run setfiles on /boot/efi" 2018-06-15 08:42:13 +00:00
Nguyen Van Trung
8d86ff1bec Don't run setfiles on /boot/efi
setfiles isn't supported on the vfat /boot/efi partition.  Add it to
the skip list.

Tested on Fujitsu Server successfully.

Change-Id: Iab262c4bdb0ecc25ca6b77ee4aff1ce442c0c578
2018-06-15 14:53:38 +10:00
Michael Turek
91e3b72a23 Add iscsi-boot element
This patch adds an element that handles the configuration for
creating a disk capable of being a remote root filesystem through
iSCSI on Ubuntu and Debian images.

Change-Id: Ibf9e39d2bdab530106015f156d23d28029d12b0d
Closes-bug: #1716794
2018-06-14 08:56:03 +07:00
Yolanda Robla
bfc60958bf Fix bootloader packages for rhel
When using uefi in rhel, the package mapping is incorrect.
We need to add specific grub-efi* mappings to use grub2-efi

Change-Id: I2db96ae85fd5e4638c794015b2f8164c018420e3
2018-06-08 17:14:19 +02:00
Zuul
fc3748f49b Merge "Fix encoding issue during processing output" 2018-06-06 07:58:14 +00:00
Zuul
53c2e35f1b Merge "Save and close stdout on exit" 2018-06-06 06:30:05 +00:00
Zuul
8bdba50459 Merge "Reduce path length in PS4 for debug" 2018-06-06 06:20:28 +00:00
Zuul
29230d74c5 Merge "Use surrogateescape with outfilter.py" 2018-06-06 06:20:27 +00:00
Zuul
225c09b245 Merge "elements: pip-and-virtualenv: Handle openSUSE Leap 15" 2018-05-31 21:54:59 +00:00
Zuul
c4c2fb746c Merge "Remove duplicate GRUB command line entry" 2018-05-31 16:00:29 +00:00
Markos Chandras
f37e85d547 elements: pip-and-virtualenv: Handle openSUSE Leap 15
We need to handle openSUSE Leap 15 when installing pip and virtualenv
packages. This fixes the following problem when the pip-and-virtualenv
elements is used:

2018-05-31 09:42:12.014 | + [[ opensuse = opensuse ]]
2018-05-31 09:42:12.014 | /tmp/in_target.d/install.d/04-install-pip: line 57: packages: unbound variable

Change-Id: Id7911b0a0836fa8dcc003e23fa515b78fba67126
2018-05-31 10:55:28 +01:00
Zuul
4c04b46db6 Merge "Allow to rebuild arbitrary images" 2018-05-29 17:06:48 +00:00
Zuul
df9402c81c Merge "elements: zypper-minimal: Add support for openSUSE Leap 15.X" 2018-05-25 08:55:53 +00:00
Ian Wienand
8fb2f5cb55 Save and close stdout on exit
Redirecting our output through outfilter.py is inherently a bit racy,
since the disk-image-create process will exit, and then you might get
outfilter.py flushing any remaining output as it closes.

On an interactive prompt this might lead to final output overwriting
the prompt, etc.  This can be a bit confusing when you start running
things in a loop.

If we save the original fd, then on the exit path close the redirected
fd's and wait a little bit for final output (as a result of the
close), we get a more consistent output.

Change-Id: I8efe57ab421c1941e99bdecab62c6e21a87e4584
2018-05-25 11:48:20 +10:00
Ian Wienand
7b98433c46 Reduce path length in PS4 for debug
Strip everything before "site-packages" in the output filename for the
PS4 prompt.  This makes the line in debug logs significantly shorter
as we don't have the full virtualenv path every single time.  The
important thing -- the file being called in the lib/ dir, is retained.

Change-Id: I00706b6f6c0425c7795f997c08ceda3374dc84b5
2018-05-25 11:48:20 +10:00
Ian Wienand
596062b0f9 Use surrogateescape with outfilter.py
When switching to using log-file capture, we're getting

 [gentoo/build-succeeds] outfile.write(ts_line.encode('utf-8'))
 [gentoo/build-succeeds] UnicodeEncodeError: 'utf-8' codec can't
   encode character 'udcc5' in position 59: surrogates not allowed

Use surrogateescape [1] on the output to avoid this

[1] https://www.python.org/dev/peps/pep-0383/

Change-Id: I2c2c537296edfa5a8fe661a41bd5bfb3bfcf57e3
2018-05-25 11:48:14 +10:00
Roman Gorshunov
84eea81efe Allow to rebuild arbitrary images
Patch allows to rebuild arbitrary images, which location, filename and
sha256sum are specified in variables, not only hardcoded $DIB_RELEASE/current.

Change-Id: I05418932a0c40d885fe00a49f1f49d7e86c67518
2018-05-24 10:19:59 +00:00
Zuul
668c93b118 Merge "Replace the ubuntu-minimal trusty test with a bionic one" 2018-05-21 06:01:53 +00:00
Ian Wienand
82eb1ca837 Replace the ubuntu-minimal trusty test with a bionic one
Add a bionic test in replacement of trusty.  We are already building
bionic images in the gate, so this seems like a good time to switch.

Change-Id: I20d4c25e9b79e7326c86767c36be8615ba0888a3
2018-05-21 12:51:31 +10:00
Roman Gorshunov
09af52a08c Remove non-maintained ubuntu-core element
Removing no longer working and no longer maintained ubuntu-core element, which
intent is unclear, and not documented.

Change-Id: Id847591d04fd7cd32c8903967da01ee0d303b267
Closes-Bug: 1771614
2018-05-18 09:28:02 -07:00
Markos Chandras
bb9ad09010 elements: zypper-minimal: Add support for openSUSE Leap 15.X
openSUSE Leap shares the same repo structure with 42.X

Change-Id: I23de11d81020c8aae641dfc01c1cbddf769f5c75
2018-05-18 11:55:56 +01:00
Stanislav Makar
2e6a19a018 Add Ubuntu 18.04 support
Use squashfs for more recent Ubuntu releases

Change-Id: I80df28be6e2a5e03ae1450e84fc05715f21a7750
Closes-bug: 1766850
2018-05-18 14:47:20 +10:00
Matthew Thode
e1f01d513a
IPA requires iptables
iptables -L is used in log collection

Change-Id: Id7e69a9e9f87db6621b928a2104df4b94f10044e
2018-05-17 11:14:44 -05:00
Doug Szumski
c1b1534c87 Remove duplicate GRUB command line entry
Without this change DIB appends a second command line entry to the GRUB
config. This causes the original command line entry to be ignored
when Linux is booted.

The expected behaviour is that DIB appends to the existing entry as
it does for Ubuntu and SUSE.

Following discussion on the review, this also removes the distro specific
switch statement, as update-grub just calls grub-mkconfig, meaning that
there was nothing distro specific in the first place.

Change-Id: I2298675dda1f699c572b3423e7274bc8bd7c1c9d
Closes-Bug: #1771366
2018-05-16 09:25:59 +01:00
Zuul
a8df61edbb Merge "rpm-distro: set the contentdir yum var" 2018-05-14 12:36:11 +00:00
Zuul
d79a7aaf01 Merge "Add pip cache cleanup to pip-and-virtualenv" 2018-05-14 09:03:49 +00:00
Zuul
1412e957ba Merge "Fixes add-apt-keys in dpkg element" 2018-05-14 08:52:44 +00:00
Tristan Cacqueray
25964c5c5b rpm-distro: set the contentdir yum var
Since CentOS-7.5, a new yum variable is needed for SIG repositories.
This change replicates the %post task of the centos-release package
to setup the contentdir yum var. This should fix issues when
repository url uses $contentdir and yum fail with:
http://mirror.centos.org/%24contentdir ... [Errno 14] HTTP Error 404 - Not Found

For more details see:

https://lists.centos.org/pipermail/centos-devel/2018-March/016542.html

This change also drops support for fedora without dnf.

Change-Id: I1819a48b94670577b0c5e29b24cebfb20ea07d28
2018-05-14 05:38:37 +00:00
XiaojueGuan
93b42f0f01 Trivial: update url to new url
Change-Id: I8d26fd0598626c8666a1852724f0620af9dc028d
2018-05-13 23:06:00 +08:00
Oded Le'Sage
10b2a5a4ee Fixes add-apt-keys in dpkg element
This commit addresses the issue described in bug 1768354 when using the
apt-sources element and adding a key to a custom repo, subsequent deb
package installs fail due no update of the repo before package install

Change-Id: I968b3422fab2fb2305426d49215391d8ba7499df
Closes-Bug: 1768354
2018-05-09 12:06:20 -05:00
Michael Johnson
7e79a933db Add pip cache cleanup to pip-and-virtualenv
The pip-and-virtualenv element provides pip inside the created images.
When this pip is used inside the chroot of the image it, by default, creates
a cache directory with the http packages and the resulting wheels.
In the case of the octavia ubuntu-minimal image this cache is using ~50MB of
cache that is not needed after the image finished building.

This package creates a finalise.d task that removes the cache from the
default location.

Change-Id: I4715437b068d04993ef755bd1e27963db1d22417
2018-05-09 09:31:17 -07:00
Matthew Thode
f7818e4a57
Install sudo on Gentoo images by deault
Change-Id: I30640017e3c652a467be421169029045a8b675d2
Close-Bug: #1766484
2018-04-24 11:44:39 -05:00
Tristan Cacqueray
abd63b01aa pip-and-virtualenv: fix install-pip when centos-release-openstack is enabled
When RDO projects repository is installed, the python-setuptools
package is obsoleted by python2-setuptools, this makes the install-pip
script failed:

  Package python-setuptools-0.9.8-7.el7.noarch is obsoleted by
  python2-setuptools-22.0.5-1.el7.noarch which is already installed

Then the "rpm -ql python-setuptools | xargs rm -rf" exit 1.  Check if
we have a record of the updated package obsoleting then old one; if
so, use it.

Change-Id: I2b0051bd9e81908c187098a7b82e120b999b111d
2018-04-23 08:18:04 +10:00
Paul Belanger
99eb9a2d7d
Fix epel element for centos-minimal
We no longer install wget / yum-utils for centos-minimal, this fixes
that.

Change-Id: I8d89026bd48cf7398cc1cbe41e3b7f00f682dbb8
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2018-04-18 20:15:07 -04:00
Paul Belanger
12a760f7b7 Revert "debootstrap: Call update-initramfs explicitly"
This currently breaks glean on rackspace, revert until we can figure
out why that is.

This reverts commit 43bc352c59.

Change-Id: Iae88a3b0457bab0b8f0fd1febf58732ca95e5dc9
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2018-04-17 21:47:46 +00:00
Ian Wienand
b423292cd0 Remove installed packages before pip install
The release of pip10 has shown up a few issues here

Firstly, pip10 now refuses to overwrite distutils installed packages,
which includes "python-virtualenv" on centos.  History has shown us
that we want the packages installed and overwritten, to avoid the
packages coming back and messing things up.

Pre-install all the packages, then list the files in the packages with
"rpm" directly and remove them.  This way pip is happy to install.

We need to take better account of the package names for this; on
Fedora things have switch to "python2-virtualenv" instead of
"python-virtualenv" and we can't use an alias to list the package
contents.

This also highlighted that python2-pip is in EPEL for centos, so
enable that when we install it.  Make the epel element a no-op for non
centos/rhe distros.

There is a related change in recent fedora that python3 now installs
binaries into /usr/local/bin.  There are commented swizzles in here to
ensure we retain the status quo of "pip" and "virtualenv" both being
python2 based, with the python3 versions being called explicitly
"pip3" and "virtualenv3" respectively.

Change-Id: I2ffdd9f615ae6b00428c17249e4f216774991b99
2018-04-17 16:09:04 +10:00
Matthew Thode
2dd6dd357a
allow building non-gentoo images on gentoo hosts
Can't use a variable that's not set.

Change-Id: I4a7cf2ac47c2254da2fec778437f67c1fe3707f7
2018-04-12 13:24:59 -05:00
Zuul
0769bfd4aa Merge "Set the dhclient timeout to match DIB_DHCP_TIMEOUT" 2018-04-11 10:12:45 +00:00
Ian Wienand
f52b385818 Don't only install python3-virtualenv
We added this sed in I422490ebe9a9c655552685bc2ff342d288335a9c to
avoid installing python2 packages on python3-only systems and thus
dragging in all of python2.

We made a similar change to python-pip in
I7d8ba9300039cce90965410a4e16ca9e711904c3; however we realised that
the gate (and other consumers) were relying on this element having
installed the python2 & 3 packages for consistency -- otherwise jobs
would install the python-pip packages and overwrite the
pip-from-source and mess everything up.  We reverted that in
I419dbdf4682394db68974944af1e5c432f3e0565 and added some clearer notes
that this element brings in python2 & 3, and if you want something
that doesn't do that then this element isn't for you.

However, we never fixed up the virtualenv package install -- currently
our Xenial images have a global virtualenv installed from source, but
the python-virtualenv packages aren't installed.  Thus if a job does
"apt-get install python-virtualenv" it overwrites the from-source
virtualenv with older parts and again messes everything up.

Probably most jobs just call "virtualenv" and assume it is there;
however in bringing up some rspec test for puppet I have hit this
issue as some modules specify dependencies on the virtualenv packages.

Thus install the python-virtualenv AND python3-virtualenv packages in
this element.

Change-Id: Ia84c38dc3c40a6080e144b563e10abca7dac2881
2018-04-10 12:34:03 +10:00
Clark Boylan
f3d58d9042 Don't use -e to test for what might be broken symlink
The behavior of test -e and [[ -e against broken symlinks is to fail
even if the symlink exists. However we want to test if the link exists
or if there is a file in that location. Therefore switch from test -e to
test -L and test -f to check if the file or link exists regardless of
link target validity.

Change-Id: I84a9b6731eccf950707be50aef464a2de1e33e8e
2018-04-06 15:04:56 -07:00
Derek Higgins
97399e9bb1 Set the dhclient timeout to match DIB_DHCP_TIMEOUT
On initial boot when networking is brought up by cloud-init this
is the timeout that dhclient adheres to. Centos configures
"timeout 300" (for an EC2 bug) in their cloud image, which results
in a 5 minutes delay to boot in cases where no dhcp available (e.g. IPv6
SLAAC). To reduce this boot delay and to provide consistency with
places where we have set other dhcp timeouts set this to DIB_DHCP_TIMEOUT.

Change-Id: I119a002070501c3dfe7c6730b07ee25f422b85b0
Related-Bug: #1758324
2018-04-05 14:29:22 +01:00
Ian Wienand
e443700b5c Formalise saving of /etc/resolv.conf
systemd-resolved has a new behaviour in bionic, in that if there is no
/etc/resolv.conf file when it installs, it assumes it is a fresh
system and makes /etc/resolf.conf a symlink into its compatability
files.

dib ends up saving & restoring whatever /etc/resolv.conf we have after
the inital chroot creation, which may not be what we want -- in the
above case it restores the system-resolved symlink.  For
openstack-infra, we use unbound and want simply "127.0.0.1" in a
/etc/resolv.conf file [1].

Formalise the ability to save specific contents into the final image.
Add documentation, and a note in the code that it's an external
interface.

I would have preferred to namespace the .ORIG file with DIB_ or
similar, but this unofficial interface has already escaped into the
wild.  Leave it as is for simplicity.

[1] Note that systemd-resolved will obey /etc/resolv.conf as you would
expect, if file exists.

Change-Id: Ie0e97d8072e2b21a54b053fa6fb07b62960c686d
2018-04-04 15:17:45 +10:00
Ian Wienand
855ab0d850 Restore tracing on exit points of block_device_create_config_file
We exit in several places and don't restore tracing.  Previously in
nodepool we relied on the default fallback, which did restore the
tracing.  Since we now use the MBR config file, we take the different
exit path without it and the debugging output is incomplete.

Change-Id: I586fc95517926025705ce376ec5c4aaf4122773f
2018-04-03 14:33:09 +10:00
Zuul
09e5b2d357 Merge "debootstrap: Call update-initramfs explicitly" 2018-03-29 06:02:58 +00:00
Zuul
c07e7349d5 Merge "Change the GENTOO_PORTAGE_CLEANUP variable default" 2018-03-29 05:33:55 +00:00
Zuul
e8744cf32b Merge "zypper-minimal: Set default locale env to C.UTF-8" 2018-03-29 05:33:52 +00:00
Zuul
e444de83e6 Merge "Clean up dib-python symlink" 2018-03-29 05:33:51 +00:00
Andreas Florath
43bc352c59 debootstrap: Call update-initramfs explicitly
Many elements install additional distribution packages.
In addition the user can provide a set of packages to be installed
via the '-p' switch.
Some of them influence the boot process and therefore the initramfs
needs to be updated. Because the package manager during the image
creation process is configured not to run package scripts, this needs
to be done explicitly.

This issue was found during development and debugging of the
block-device LVM plugin: Even when the e.g. the lvm2 package
was installed in the image, it was missing in the initramfs
because of the missing update.

Change-Id: I7c92033b3ca80cdd23d081002059d83ca3f53bdb
Signed-off-by: Andreas Florath <andreas@florath.net>
2018-03-29 04:14:52 +00:00
Matthew Thode
6a6d78e63c
Change the GENTOO_PORTAGE_CLEANUP variable default
Default the GENTOO_PORTAGE_CLEANUP to True.  By default we should not
ship package info, this bloats the image and is usually outdated by the
time it'd be consumed.

Change-Id: I14c2530d91807cbc6a3806e01c7e4f6f472b190d
2018-03-26 23:26:49 -05:00
Clark Boylan
301eac8e8b Fix element-provides in debian element
The debian element depends on debian-minimal now which provides
operating-system. This means that the debian element can no longer
provide operating-system and doing so results in an error when using the
debian element.

The fix is simple just rely on the fact that debian-minimal provides
operating-system and remove this element-provides from debian.

Fixes-Bug: 1758000
Change-Id: I524feeb82c19046ec987eb1186c7f4568309e559
2018-03-26 10:58:04 -07:00
Zuul
ffc06874ef Merge "install sudo in the devuser element" 2018-03-26 00:58:57 +00:00
Zuul
8ab38dc5cf Merge "Update Fedora defaults to 27" 2018-03-25 23:41:45 +00:00
Zuul
9d751463f1 Merge "enable systemd profile for Gentoo" 2018-03-23 19:01:35 +00:00
Zuul
45ff8175b6 Merge "proliant-tools: add net-tools package to support hpsum utility" 2018-03-23 05:36:12 +00:00
Matthew Thode
cfa7935e43
enable systemd profile for Gentoo
Change-Id: Id3ac1d97b280f10f9938a60c4871d08f59b85002
2018-03-22 15:12:59 -05:00
Matthew Thode
cfa5b237c0
install sudo in the devuser element
The devuser element can set up passwordless sudo, which requiers the
/etc/sudoers.d directory, which requires the sudo package, so we ensure
the sudo package is installed.

Change-Id: I80d6c669d4ac0d97b49d01cb621bf05b8e7f8ef1
2018-03-22 00:16:09 -05:00
Zuul
a49f4c3a37 Merge "remove portage git directory" 2018-03-22 03:32:44 +00:00
Ian Wienand
f3f671cf10 Fix default partition type
There was a typo in I6b819a8071389e7e4eb4874ff7750bd192695ff2 that
modified this default partition type from "0x83" to just 83.  We are
now seeing failures relating to this as sfdisk checks for a "disk
manager" when it see Id 0x53 (== 83)

     Device Boot      Start         End      Blocks   Id  System
  /dev/vda1   *        2048    26664575    13331264   53  OnTrack DM6 Aux3

Restore to 0x83

Change-Id: Ib43038d2d740fbe01a21a13dd56367f7bc97f869
2018-03-22 10:10:47 +11:00
Matthew Thode
5e2f3646ad
remove portage git directory
this shrinks thinks by ~50%, from 722M to 352M

Change-Id: I1267cc05700ee28c45a331de7f571b9ee075c6b5
2018-03-16 18:29:40 -05:00
Zuul
651d913fcc Merge "arm64: use HWE kernel and fix console" 2018-03-16 08:31:14 +00:00
Zuul
5e2168aefd Merge "Choose appropriate bootloader for block-device" 2018-03-16 08:31:12 +00:00
Zuul
96af247400 Merge "Add block-device defaults" 2018-03-16 08:31:10 +00:00
Zuul
45eaace4e3 Merge "Fail if two elements provide the same thing" 2018-03-16 08:31:08 +00:00
Zuul
c60a20b59d Merge "GPT partitioning support" 2018-03-16 08:14:27 +00:00
Anshul Jain
a7135a0d8f proliant-tools: add net-tools package to support hpsum utility
Hpsum utiltity of proliant-tools requires net-tools to be installed
as part of base image. This commit adds support for installation of
net-tools for all distros.

Change-Id: I2a1e81059ed1aee975db78cfa5e61bbf1b98e06f
Closes-bug: 1751777
2018-03-09 02:29:23 -06:00
Zuul
30d5ee2ec7 Merge "Fix for passing user defined value for satellite cert for rhel-common." 2018-03-08 07:05:40 +00:00
Zuul
dda4e2e0f1 Merge "secondary architectures use different url" 2018-03-08 07:05:39 +00:00
Zuul
d2133ec5e7 Merge "Fix for rhel7 iso image creation." 2018-03-08 07:05:38 +00:00
Mark Hamzy
cdb423eeb9 secondary architectures use different url
The Fedora-Cloud-Base*qcow2 images are stored on a different
server for secondary architectures.

Change-Id: I90d48ce4175fd251e8f5ab7a70190ad952256a94
2018-03-01 19:38:14 -06:00
Tobias Henkel
b62ed1823c
Fix encoding issue during processing output
When using the package-installs element there can be some encoding
problems if the package installation emits unparsable output
[1]. However in this case we just want to forward the output to the
console which normally can handle this correctly. In order to fix this
switch off universal_newlines processing such that we just operate on
bytes.

Further we have to decode the lines without setting the locale and
ignoring errors. This is required because print encodes without
setting the locale and thus we need to filter/modify the stream such
that it doesn't crash.

[1] Traceback:
2018-03-01 09:58:00.515 | Traceback (most recent call last):
2018-03-01 09:58:00.515 |   File "/usr/local/bin/package-installs-v2", line 137, in <module>
2018-03-01 09:58:00.515 |     main()
2018-03-01 09:58:00.515 |   File "/usr/local/bin/package-installs-v2", line 130, in main
2018-03-01 09:58:00.515 |     process_output(install_args, follow=True)
2018-03-01 09:58:00.515 |     for line in iter(proc.stdout.readline, ''):
2018-03-01 09:58:00.515 |   File "/usr/lib/python3.5/encodings/ascii.py", line 26, in decode
2018-03-01 09:58:00.515 |     return codecs.ascii_decode(input, self.errors)[0]
2018-03-01 09:58:00.515 | UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 34: ordinal not in range(128)

Change-Id: Ie4af9b4523459a630cfb98d09093bfe9ef7aa61e
2018-03-01 16:09:25 +00:00
Anshul Jain
73a9ba0f1f Fix for rhel7 iso image creation.
Currently rhel7 image creation fails because it tries to copy
default bootloaders which is ubuntu way. This commit updates `iso`
element to correct the path of bootloaders required for rhel image.

Change-Id: I526d75b2db609fc77be0fc778b4d00f2d3df38ec
Closes-bug: 1750725
2018-02-28 06:31:34 -06:00
Zuul
3fb0950808 Merge "update Gentoo Hardened profiles (now stable)" 2018-02-28 11:42:51 +00:00
Zuul
05a44e2181 Merge "Checking link status according to DIB_DHCP_TIMEOUT" 2018-02-28 11:42:50 +00:00
Anshul Jain
b37a1e27cb Fix for passing user defined value for satellite cert for rhel-common.
For 'satellite' mode of registration, rpm for rhel SSL certificate is
hard coded to 'katello-ca-consumer-latest.noarch.rpm'. This commit adds
functionality that provides an option to set this as defined in their
satellite server.

Change-Id: Ib176cfa209f5ac8a4b5da71419327b4237330904
Closes-Bug: 1749947
2018-02-28 02:43:39 -06:00
Ian Wienand
e9ed983324 arm64: use HWE kernel and fix console
Install hwe kernel for ubuntu-minimal.  As noted this is currently
Xenial specific; we need this for initial bring-up so let's tackle
future releases as things progress.

Ensure we use ttyAMA0 for arm64 console too.

Change-Id: Ic607cf8369666dc24929aff6f2ef8a72e7980599
2018-02-23 10:04:48 +11:00
Ian Wienand
7b4c8abce3 Choose appropriate bootloader for block-device
In the prior change we added block-device-[mbr|gpt|efi] elements to
create appropriate disk-layouts.

This adds an environment flag to each so the bootloader can install
the right thing.  The EFI install path is updated to work with this
(this part a copy of I572937945adbb5adaa5cb09200752e323c2c9531)

We do some basic sanity checking in the block-device elements;
e.g. mbr is not suitable for aarch64, and efi is not suitable for
power.

This updates the bootloader to install EFI where appropriate

Co-Authored-By: Marcin Juszkiewicz <marcin.juszkiewicz@linaro.org>
Change-Id: Ib80acbfd9a12efd976c3fa15a5d1081eb0799305
2018-02-23 10:04:44 +11:00
Ian Wienand
adb0341064 Add block-device defaults
This moves the block-device default out of the "vm" element and into a
selection of other elements.  There's "mbr" which retains the status
quo.  There's an EFI version that has the boot/grub partitions as
required.  In between there's the GPT only version, which is useful
for architectures like power without EFI, but still want possible
larger disks using GPT.

Change-Id: I4a566a97d073fc0dda0ab2494ac988fe015800a9
2018-02-23 10:04:40 +11:00
Ian Wienand
a32a8f1ce1 Fail if two elements provide the same thing
The current check only validates that an element that specifies
"element-provides" doesn't conflict with a "real" element.  We also
want to check this against the provides of other elements.

A real example is with a "block-device" element.  There is no actual
"block-device" element; we can have multiple elements provide it
(block-device-[gpt,mbr,efi], say) but we only want one of them at a
time.

Update the unit test for this.

Change-Id: I59d4aa5f6f09e2892b213e154befa10d85e95ca3
2018-02-23 10:04:26 +11:00
Ian Wienand
55b479b54f GPT partitioning support
This adds support for a GPT label type to the partitioning code.  This
is relatively straight-forward translation of the partition config
into a sgparted command-line and subsequent call.

A unit test is added based on a working GPT/EFI configuration and the
fedora-minimal functional test is updated to build a single-partition
GPT based using the new block-device-gpt override element.  See notes
in the sample configuration files about partition requirements and
types.

Documentation has been updated.

Co-Authored-By: Marcin Juszkiewicz <marcin.juszkiewicz@linaro.org>
Change-Id: I6b819a8071389e7e4eb4874ff7750bd192695ff2
2018-02-23 10:04:26 +11:00
Matthew Thode
6b3c22dd75
update Gentoo Hardened profiles (now stable)
Change-Id: I6d89f27bfd62fc0e86fec1a0bc6cb37f9ec6c82f
Signed-off-by: Matthew Thode <mthode@mthode.org>
2018-02-21 11:06:16 -06:00
Lenny Verkhovsky
f249cec9f3 Checking link status according to DIB_DHCP_TIMEOUT
In slow networks like Infiniband it takes much time for the
interface to get the carrier. This patch enables this service
to run more then 20 seconds and limited by DIB_DHCP_TIMEOUT.

Change-Id: I8a6015567ac25e37b5a5aba4b1fda71170cc144a
2018-02-21 12:52:57 +00:00
Zuul
e8eb291f57 Merge "update gentoo vars for new profile and python" 2018-02-20 05:26:20 +00:00
Paul Belanger
e9e7ac2ee1
Install systemd earlier for Ubuntu Bionic
Like we did in https://review.openstack.org/475206 we need to install
systemd sooner because of the new world order of containers.

Change-Id: Ia60d751fee3af6f8d72ad664107acb337360feca
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2018-02-17 14:32:35 -05:00
Matthew Thode
bfee8bf094
update gentoo vars for new profile and python
Change-Id: I810b09f98c479e6ebdbf8de6fae31166a3e87667
2018-02-13 17:02:11 -06:00
Zuul
2080055155 Merge "Create rescue user on ironic agent" 2018-02-05 22:43:00 +00:00
Zuul
cb599b13de Merge "upgrade pip before using -c option" 2018-02-05 22:34:00 +00:00
Zuul
5cbc72ae30 Merge "Set default label for XFS disks" 2018-02-01 03:59:53 +00:00
Zuul
66444e8b93 Merge "Add support for Fedora 27, remove EOL Fedora 25" 2018-01-29 05:23:28 +00:00
Mark Hamzy
c7da8bc90a Set default label for XFS disks
As described, we want to set the default label for XFS disks to the
shorter value.

For example, you hit this when setting the old FS_TYPE environment
variable to 'xfs' (which sets the "root-fs-type" parameter, which gets
passed through to 'type'; but does not set a default label).

Change-Id: I41dce6e25766562db4366021309b8c2b74a8ab80
Closes-Bug: 1742170
2018-01-29 15:10:08 +11:00
Tim Flink
f8bcc51b55 Don't install dmidecode on Fedora ppc64le
While Debian-based distros use the label of ppc64el for ppc64 little
endian, Fedora uses ppc64le.

The ironic-agent was doing arch specific package install of lshw over
dmidecode for ppc64 and ppc64el but was attempting to install dmidecode
on Fedora ppc64le which caused the test to fail due to a missing
package.

This change just adds ppc64le to the arch-specific package installation
description for the ironic-agent element.

Change-Id: I38c3c1480bbbb2df817856614e6b740a0c02723a
Closes-Bug: 1744944
2018-01-29 10:53:43 +11:00
Zuul
9bfa45f0c8 Merge "Remove architecture rules on lshw dependency in ironic-agent" 2018-01-28 22:07:53 +00:00
Tim Flink
490cf3aa49 Add support for Fedora 27, remove EOL Fedora 25
This updates diskimage-builder to support current Fedora releases (26
and 27) and removes support for Fedora 25 which is EOL as of December
12, 2017.

Change-Id: I227a607c6c468cc8b7bb154a189e9c8ce2021192
2018-01-23 11:31:21 +00:00
Ian Wienand
fd9a8acecd Don't fstrim vfat partitions
This small change avoids running fstrim on vfat partitions.

The mount order test-case has been updated to also test the mkfs
creation components, and the input config modified to have a vfat
partition to cover this path.

Change-Id: I8952e748d4bdc12a5769706de9057c1e97d95e37
2018-01-23 13:24:09 +11:00
Mark Hamzy
34ff72f253 upgrade pip before using -c option
The installed pip can be an older version which does not support
the -c argument. Therefore, upgrade pip before using -c.

Change-Id: If18d8ea822a62c8551c9c4d47354d58b0299fed2
Closes-Bug: 1744403
2018-01-19 16:46:59 -06:00
Dirk Mueller
4858340b42 Add SUSE Mapping
Change-Id: I8bc12435c62ef7c9c3fa8e21e00738698a532f56
2018-01-11 19:00:40 +01:00
Ian Wienand
e9df83b2b3 Revert "Dont install python-pip for py3k"
This reverts commit ab89c7d69c.

This commit checked for DIB_PYTHON_VERSION and only installed the v3
packages.  This is unfortunately backwards-incompatible, as consumers
such as the openstack gate are relying on this package installing pip
& virtualenv packages for python2 AND python3.

This was sort-of expressed in the docs, where it discusses what the
resulting setup of the system will be, but I've added a note to make
it clearer.

If we want to change this, I think we'll need either a new element, or
a non-defaulting flag.

Change-Id: I419dbdf4682394db68974944af1e5c432f3e0565
2018-01-10 15:48:03 +11:00
sayalilunkad
acc44c0651 Adding mapping for SUSE package
Mapping PyYAML to python-PyYAML as in opensuse.

Change-Id: I54864c96afdb20975bf1e3ac9a117d5263f21f3b
2017-12-20 17:51:33 +01:00
Ian Wienand
af50200863 Update Fedora defaults to 27
Update the Fedora defaults to now released F27

Change-Id: I7655ad010dfa668bd68b7a619d85e5f694f96806
2017-12-07 13:59:51 +11:00
Ian Wienand
dcbb7e7ebb ironic-agent: don't remove make
It turns out make has always been a tacit dependency of openssl as it
ships a Makefile for certificates [1].  This just recently changed to
be a hard dependency in F27, so this now fails as openssl is a
dependency of protected packages such as dnf.  Since it's always been
wrong to remove it, we take it out of the purge list.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=783446

Change-Id: I69efb3a56878ab97c4587bbbf5356bea752f2846
2017-12-07 13:59:51 +11:00
Michael Turek
7054a71f7d Remove architecture rules on lshw dependency in ironic-agent
There's a patch in flight in ironic-python-agent to switch the
default hardware manager to use lshw instead of dmidecode. [0]
This would require lshw to be installed regardless of
architecture. This patch removes the architecture rules from
lshw in the package-installs list.

[0] Ie370331df6bb5ef131c5cb60f458877e2a7ad71a

Change-Id: Idaf05b8efce28cd0cbf339cf693db4f55a693d9b
Partial-Bug: #1715790
2017-12-06 17:02:46 -05:00
K Jonathan Harker
7470ee26e0 zypper: fix package removal
zypper only supports the --no-recommends option during installs, giving
the option during removals results in an error.

When setting ACTION=remove, remove --no-recommends from EXTRA_ARGS, and
set --clean-deps to also remove no-longer-needed dependencies.

Rename EXTRA_ARGS to ACTION_ARGS for increased readability.

Change-Id: Ifbd168992b1a20658b6b4a99ba175234f6c78f6d
2017-12-05 22:59:20 +00:00
Zuul
7beb27ca30 Merge "Add zipl element as s390x architecture bootloader" 2017-12-01 02:35:58 +00:00
Zuul
6ab9655ca3 Merge "Fix /dev/pts mount options handling" 2017-12-01 02:01:46 +00:00
goldyfruit
c247cb41bb Fix wrong epel-release-7* package URL
When "epel" element is used during a build process
with "rhel7" distribution, the build failed
because the "epel-release-7*" package cannot be
installed.

The reason is because the URL is not correct, it
should be:
  URL=$BASE_URL/$RELEASE/x86_64/Packages/e/

Change-Id: I90c26892361f7611645b85f2eddc949b2f0d76fc
Closes-Bug: #1735547
2017-11-30 15:31:10 -05:00
Zuul
bfd61258ba Merge "Add the groundwork for musl profile support" 2017-11-30 09:31:40 +00:00
Zuul
5677a30a5a Merge "Pre-install curl" 2017-11-30 08:16:14 +00:00
Zuul
8182acb0fc Merge "Install fedora-gpg-keys for F27" 2017-11-30 07:54:38 +00:00
Matthew Thode
7223f2ce20 Add the groundwork for musl profile support
At the moment all musl needs in addition to an official stage4 file is a
few keywords and use flag changes.

Change-Id: Ibf4a6d616aca1aef876967e2aa34170c96ac9ef8
2017-11-30 18:17:21 +11:00
Zuul
6a28810ad8 Merge "Enable support for Gentoo overlays" 2017-11-30 06:40:43 +00:00
Zuul
87392cf1c8 Merge "Make preinstall.d more deterministic" 2017-11-30 05:50:03 +00:00
Matthew Thode
c4f83f2311 Enable support for Gentoo overlays
This is intended to eventually support building musl-libc based images,
which need the musl overlay.

Change-Id: I8f5429ffa64e74c860772d9a00ff0b7eebb7721a
2017-11-30 05:14:00 +00:00
Zuul
bbbe762dff Merge "elements: zypper-minimal: Refresh repositories where necessary" 2017-11-30 04:36:06 +00:00
Ian Wienand
1b203f8a38 Pre-install curl
As described, Fedora 27 has a curl-minimal package that comes in to
satisfy the rpm package dependency.  It conflicts with the "real" curl
package -- which is so commonly installed (by infra elements, etc)
that this becomes an annoying problem.  Just pre-install the full curl
package.

Fedora 24 is old enough to not worry about, so remove some old
workarounds to make the flow a little simpler.

Change-Id: I67baf96377109ac4521ba00243a0d91b35fafba0
2017-11-30 15:15:42 +11:00
Ian Wienand
bf8de79940 Install fedora-gpg-keys for F27
The repo GPG keys moved into a separate package [1] which now needs to
be installed.

Since the fedora-release/fedora-repos split is *long* since over,
remove that work-around and add this one.

[1] https://pagure.io/fedora-repos/c/f69f3729511c3eba5f470b1d90ea2bfee372eb29?branch=f27

Change-Id: I9ad28d5bdb78375ae21dbb16e2d8c4effb32cb35
2017-11-30 15:15:37 +11:00
Matthew Thode
c886c4cbe1 Make preinstall.d more deterministic
Reorders the preinstall, making the order more explicit.  Also dedupes
some folder setup.

Change-Id: I423dcba169558ff6037a3382b997675722e77405
2017-11-30 13:48:18 +11:00
Zuul
f74e48799d Merge "Enable gentoo in pip-and-virtualenv element" 2017-11-30 02:10:09 +00:00
Zuul
247c68b5a3 Merge "Clear /etc/machine-id to avoid duplicate machine-ids" 2017-11-30 01:56:26 +00:00
Zuul
d01d3d8832 Merge "Make python changes more reliable" 2017-11-30 01:49:07 +00:00
Zuul
71ca627d30 Merge "Add debian minimal requirement for arm64" 2017-11-29 23:47:12 +00:00
Zuul
c146d2f3b7 Merge "Fix grub2 dependency on arm64" 2017-11-29 23:47:10 +00:00
Andreas Florath
46a07de480 Fix /dev/pts mount options handling
The current implementation - as introduced in
Iee44703297a15b14c715f4bfb7bae67f613aceee - has some shortcomings / bugs,
like:

* the 'grep' check is too sloppy
* when /dev/pts is already mounted multiple times the current implementation
  fails:
  $ mount | grep devpts | sed 's/.*(\(.*\))/\1/'
  rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000
  rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000
  rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000
* code duplication
* Using the undocumented and non-robust output
  of 'mount'.

This patch fixed the above problems.

Change-Id: Ib0c7358772480c56d405659a6a32afd60c311686
Signed-off-by: Andreas Florath <andreas@florath.net>
2017-11-24 07:14:56 +00:00
Matthew Thode
ade82904a1
Make python changes more reliable
We oneshot emerge without calculating dependencies a few things to solve
for possible dependency loops.

Python 3.5 also became stable, so don't need to do special things for
it.

Matched the uninstall with the install lines (no need for a full if
statement).

Change-Id: I7c5e546612ac47d659e73a46a52e34d39ca81949
2017-11-20 23:19:46 -06:00
Zuul
b879a03c75 Merge "Dont install python-pip for py3k" 2017-11-20 05:38:24 +00:00
Markos Chandras
7f051add61 elements: zypper-minimal: Refresh repositories where necessary
We should always refresh the Tumbleweed repositories and the 'update'
one for Leap in order to always have the latest information from the
repositories.

Change-Id: I85db9d8bb7fa153f01222129e9b36fecc2632f57
2017-11-16 16:54:47 +00:00
Markos Chandras
da02f37de1 elements: Respect devpts mount options
This is a continuation for f2cc647dae ("diskimage_builder: lib:
common-functions: Fix options for devpts mount"). We also need to
respect the devpts mount options when the dib elements are mounting
this virtual filesystems themselves.

Change-Id: Iee44703297a15b14c715f4bfb7bae67f613aceee
2017-11-14 08:31:55 +00:00
Gregory Haynes
ab89c7d69c Dont install python-pip for py3k
We want to install python3-pip, not python-pip when we are building a
py3k image less we pull in python2. Once we stop installing python2 we
have to stop calling python2 during pip install.

Change-Id: I7d8ba9300039cce90965410a4e16ca9e711904c3
2017-11-13 23:00:52 +00:00
Adam Harwell
d4fd7f1217 Enable gentoo in pip-and-virtualenv element
Currently it will hit the `else` and try to apt-get, which fails.

Change-Id: I951882cf3897ced165e167f12877c05ee62a5054
2017-11-11 09:00:50 +09:00
Dirk Mueller
4e258bdad6 zypper-minimal: Set default locale env to C.UTF-8
Currently in Leap 42.x the bootup scripts don't actually make use of
locale.conf yet, so we need to set it in /etc/sysconfig/language. For
future distro compatibility the setting in locale.conf is kept in sync.

Also fix default timezone link.

Change-Id: I59e5dccad8a5ae132d3039851e7aa1db86a609d7
2017-11-10 16:09:05 +01:00
Zuul
b5b9d98fa0 Merge "diskimage_builder: lib: common-functions: Fix options for devpts mount" 2017-11-09 15:18:44 +00:00
Zuul
bc6c928bb9 Merge "Move to a common lock-file directory" 2017-11-07 17:32:28 +00:00
Zhiguo Deng
271dc36f33 Add zipl element as s390x architecture bootloader
s390x architecture uses zipl as bootloader. When used in combination
with the vm element it replaces the existing bootloader element.
It's mandatory for s390x vm images.

Use cases
---------

* Allow users to create s390x images that run on nova with s390x
  libvirt/kvm backend
* Building nodepool images for s390x third party CI

Supported Distros
-----------------
The following listing shows all Distros that officially support
s390x and how those Distros are supported in DIB with this patch.

* SLES - not supported (SLES is not supported in DIB)
* RHEL - not suppoprted (RHEL is not supported as KVM guest on s390x,
                         therefore there's no rhel7 qcow image for s390x available
                         like it is for other archictectures)
* Ubuntu - supported

Ubuntu images can for example be built using the following commands:

  $ disk-image-create ubuntu-minimal zipl vm
  $ disk-image-create ubuntu-minimal zipl
  $ disk-image-create ubuntu zipl vm

Testing
-------

Cross architecture building of s390x images is not supported so far.

The plan is to set up a ThirdParty CI that builds the image for s390x and
provides the logs.

Co-Authored-By: Andreas Scheuring <andreas.scheuring@de.ibm.com>
Co-Authored-By: Holger Smolinsky <holger@smolinski.name>
Co-Authored-By: Zhiguo Deng <bjzgdeng@linux.vnet.ibm.com>
Co-Authored-By: Arne Recknagel <arne.recknagel@hotmail.com>

Closes-Bug: #1730641

Change-Id: I576e7edda68da12e97c60af38f457915efe7b934
2017-11-07 17:19:27 +01:00
Markos Chandras
f2cc647dae diskimage_builder: lib: common-functions: Fix options for devpts mount
Commit cebfcf85f9 ("Use -t devpts for
/dev/pts mounts") switched from using '--bind' to '-t devpts' for
mounting the /dev/pts virtual filesystem. However, mounting devpts to
another location also affects the host's /dev/pts mountpoint. Since we
are now mounting devpts without options we end up with the following one
on openSUSE

devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,mode=600,ptmxmode=000)

instead of the one we want

devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)

The missing gid=5 options results to boot problems for virtual machines

So in order to fix that, we need to use the existing devpts options for
/dev/pts so we don't lose them in the new mount.

Change-Id: I17f2c2bb96b807f8dbc07185ae0147bff3230f92
2017-11-06 10:23:03 +02:00
Zuul
906a3f4a57 Merge "Use -t devpts for /dev/pts mounts" 2017-11-02 14:20:05 +00:00
Zuul
1b0631da84 Merge "Update proliant-tools to support Gen10 Proliant servers" 2017-11-01 07:22:03 +00:00
Ian Wienand
b25d0337b8 Move to a common lock-file directory
In a couple of places we use flock for critical sections, but we leave
lockfiles around in various locations which can be confusing.

Introduce DIB_LOCKFILES global (under ~/.cache/dib/lockfiles) and
write lockfiles in there.

Fix up removal of the lockfile in the yum path; we just want to make
sure we cleanup the .rpmmacros file, but we don't need to remove the
lockfile as well.

Co-Authored-By: Andreas Florath <andreas@florath.net>

Change-Id: Ie810b2836be521325afe923708d046112e1e1e20
2017-10-26 16:27:59 +11:00
Zuul
8f025691ba Merge "Dont install python-virtualenv for py3k in deb" 2017-10-24 06:33:45 +00:00
Zuul
c5f713b1ec Merge "Change to install a package in 'proliant-tools'" 2017-10-24 05:48:25 +00:00
Yolanda Robla
ba11376328 Create rescue user on ironic agent
Create a new service, that will be launched after ironic
agent has been exited. This will launch an script that will
take the rescue password, and create the rescue user with
that credentials.

Depends-On: I7898ff22800dedba73d7fbfb3801378867abe183
Change-Id: Ic3a241e2789a122d3d966e7e2148306fd0cf6aed
Partial-Bug: 1526449
2017-10-23 12:50:32 +00:00
Andreas Florath
cebfcf85f9 Use -t devpts for /dev/pts mounts
Currently a bind is used when mounting /dev/pts in chroot.
This leads to problems - especially when running DIB in parallel:
It was observed that the /dev/pts mount vanishes from the host
system.

This patch uses '-t devpts' - as it is done for /sys and /proc -
for handling /dev/pts.

Change-Id: Id7775ae6fca6502af800e7b73a00862ef320206b
Signed-off-by: Andreas Florath <andreas@florath.net>
2017-10-23 07:13:02 +00:00
Gregory Haynes
00d7c619e9 Dont install python-virtualenv for py3k in deb
On ubuntu we detect that in python3 we need to install
python3-virtualenv, but append this to the packages to install rather
than replace python-virtualenv which results in both being installed
(and therefore grabbing python2).

Change-Id: I422490ebe9a9c655552685bc2ff342d288335a9c
Closes-Bug: #1724656
2017-10-18 23:11:55 +00:00
Zuul
a0045985f2 Merge "Remove dd from LVM element" 2017-10-17 08:15:03 +00:00
Zhangfei Gao
6cc155fd66 Add debian minimal requirement for arm64
Debian system building fails, fixed by adding arm64 package.

DIB_RELEASE=jessie disk-image-create debian ironic-agent \
    grub2 devuser -a arm64 -o deploy-jessie
dib-run-parts Running /tmp/dib_build.v5FEtaKx/hooks/cleanup.d/99-extract-kernel-and-ramdisk
ls: cannot access /tmp/dib_build.v5FEtaKx/mnt/boot/vmlinu*: No such file or directory

Change-Id: I610d767785df49fed954f12854be5ae78ff9baa6
Signed-off-by: Zhangfei Gao <zhangfei.gao@linaro.org>
2017-10-16 13:39:50 +08:00
Zhangfei Gao
a8635d3a3b Fix grub2 dependency on arm64
Unable to locate package reported when
DIB_RELEASE=jessie disk-image-create debian ironic-agent \
grub2 devuser -a arm64 -o deploy-jessie

E: Unable to locate package grub-pc-bin
E: Unable to locate package shim-signed
E: Unable to locate package grub-efi-amd64-signed

Fix the issue via adding arch dependency and arm64 packages

Change-Id: I40650a887b575a9c2b00a8c5036c35354d548673
Signed-off-by: Zhangfei Gao <zhangfei.gao@linaro.org>
2017-10-16 13:39:34 +08:00
Jenkins
7a76270227 Merge "Add timestamp output filter" 2017-10-09 15:56:31 +00:00
Jenkins
fd8f92e542 Merge "Update Gentoo element for element changes" 2017-10-08 23:58:20 +00:00
Andreas Florath
bb6cf52d85 Remove dd from LVM element
This patch removes the unneeded dd calls in the lvm block device
plugin.

After removing the underlying block device, there is the need to call
'pvscan --cache'.  This is done by a dedicated LVM cleanup node which
is cleaned up after the the underlying block device.

Change-Id: Id8eaede77fbdc107d2ba1035cd6b8eb5c10160c3
Signed-off-by: Andreas Florath <andreas@florath.net>
2017-10-08 17:21:21 +00:00
Matthew Thode
e29f031bec
Update Gentoo element for element changes
There have been a few changes over the past few months, here we make the
following changes.

* change from backtrack=99 to complete-graph as a more correct flag
* make python version selection more in line with what gentoo supports
* set up python before stuff gets pip installed
* ensure we have the proper pip so we can install pip packages as root
* ensure we have the proper use flags for the disk formatting changes
* set DIB_RELEASE like other distros
* fix openssh-server element for gentoo

Change-Id: I17202de3016616ce34c8cbead7d0fb047a64e96b
2017-10-08 12:02:46 -05:00
Anshul Jain
afb7084a4d Update proliant-tools to support Gen10 Proliant servers
This commits make update to ssacli version to point to latest
ssacli release that has support for HPE P/E-Class SR Gen10 controllers.

Change-Id: Ia9a0eaec78d601f56b4036e57601554b87f21acc
Closes-Bug: 1721185
2017-10-04 07:02:07 +00:00
Andreas Florath
fa6c731132 Move fstrim to block device layer
The call to fstrim in disk-image-create is currently useless, because
at the time this is called, the file systems were already umounted by
the block device layer.

The current implementation of the block-device mount plugin does not
call fstrim at all: resulting in larger image sizes.

This patch removes the useless fstrim call from the disk-image-create
script and moves this into the block-device mount.py.

The resulting image might be much smaller.  Example: Ubuntu Xenial
with some elements; once with and once without this patch:

-rw-r--r-- 1 dib dib 475661824 Sep 16 06:43 ubuntu-xenial-without-fstrim.qcow2
-rw-r--r-- 1 dib dib 364249088 Sep 16 09:30 ubuntu-xenial-with-fstrim.qcow2

Change-Id: I4e21ae50c5e6e26dc9f50f004ed6413132c81047
Signed-off-by: Andreas Florath <andreas@florath.net>
2017-09-28 17:48:59 +10:00
Jenkins
dc215674f8 Merge "Actually sort mount-point list" 2017-09-27 07:20:29 +00:00
Ian Wienand
06b5ce4573 Revert "Support networkx 2.0"
This reverts commit a47ff0dd4a.

Since this merged, a global-requirements pin to keep networkx <2.0 has
also merged.  The plan is:

1. revert our 2.0 support and
1a. take the <2.0 pin from global requirements
2. figure out how to use constraints properly in our testing
3. restore this, with a depends-on for a 2.0 bump in requirements
   (which will self-test, see 3.)
4. when other projects are ready for a global 2.0 bump, merge
   in a controlled fashion

This reverts the 2.0 support, and adds the pin for networkx <2.0

Change-Id: I18f6a1115da779581245e3dd423fd90516974a33
2017-09-22 08:02:44 +10:00
Ian Wienand
a47ff0dd4a Support networkx 2.0
Networkx 2.0 released recently.  The main difference for us is that
"node" is no longer a dictionary and should be accessed via "nodes",
and the topological_sort returns an interator

Closes-Bug: 1712693
Change-Id: I78e89f2261b8b8d28c68b517c1e61691ab40016c
2017-09-21 09:43:01 +00:00
Ian Wienand
df00e9adcb Add initramfs-tools for ubuntu-minimal
A small update was made to 4.4.0-96.119 that dropped the
initramfs-tools dependency from the kernel [1].  This had the
unfortunate affect of removing the initramfs from ubuntu-minimal and
making it unbootable, since we specify the root device via LABEL=.
Add the package explicitly alongside the kernel.

Also, small fix to pass unit tests

[1] https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/1700972

Change-Id: I57a0f08cd5e082ecdf8dba0ab34fb3062c50836d
2017-09-21 10:42:11 +02:00
Ian Wienand
ed3c5d9711 Actually sort mount-point list
We intended to do an in-place sort of the mount-point list, but
sorted() returns a new list that wasn't captured.  Move to the .sort()
function.

It seems the existing unit-test missed this.  Add a new test taken
from the bug which does exhibit a sorting issue.  Also added a
unit-test of just the comparitor for sanity.

Closes-Bug: 1699437
Change-Id: I8101e4a1804a4af7dbda20d48bf362c3f4ad2742
2017-09-19 11:30:36 +10:00
Aparna
7bc2b23290 Change to install a package in 'proliant-tools'
This commit adds change in 'proliant-tools' element to
install a package 'unzip' which is required to perform
SUM based firmware update for HPE Proliant servers.

Change-Id: Ib8f6d18402439edd93d100cc7a4fb2094c863715
2017-09-18 08:04:34 +00:00
Jenkins
22e03f9820 Merge "Add missing package dependency for yaml" 2017-09-15 13:52:35 +00:00
Jenkins
254875dbde Merge "Create /etc/machine-id for fedora" 2017-09-15 04:05:51 +00:00
Jenkins
6e266399fb Merge "Use latest Fedora .qcow URL" 2017-09-15 02:36:24 +00:00
Ian Wienand
768c5e188c Create /etc/machine-id for fedora
As described in the comment, we need to create the /etc/machine-id for
the image-based build when systemd isn't updated (as is usually the
case for a new distro)

Work on clearing this out continues, but this brings it to parity with
fedora-minimal.

Change-Id: Icbbbabb4114d4d95909648d8e39a6bae6d2a7b7b
Depends-On: I761e425f8a658669d9b8a70ce4260cec263ea51a
2017-09-15 11:54:01 +10:00
Ian Wienand
7774260b76 Use latest Fedora .qcow URL
The URL we are using seems to have disappeared.  Update this to
download.fedoraproject.org.  The new URL requires a "subrelease" now,
add it, along with a note on where it comes from.

Change-Id: I761e425f8a658669d9b8a70ce4260cec263ea51a
2017-09-15 11:06:22 +10:00
Yolanda Robla
da5c926fe9 Add missing package dependency for yaml
This element was assuming that yaml was included as package,
but there are systems not including it. So properly add yaml
as a dependency.

Change-Id: I72da2776674a3963657052b9a9715abcb4fab1e2
Partially-Fixes-Bug: #1715686
2017-09-13 14:16:41 +02:00
Yolanda Robla
3ff8d1e10b Move the ordering of the dracut regenerate command
When using combined with rhel7 image, the unregister of repos
has already happened, because it is executed under 60- ordering.
As dracut-regenerate may need to install extra packages for it,
it causes this step to fail, because it cannot find repos where
to pull the packages from.

Change-Id: I35e37df7990ad76a5004cb90fdd863ec743a5483
2017-09-06 12:52:05 +02:00
Ben Nemec
72d0d22cdf Remove nested quotes from TAROPTS
Per the bug report, these seem to be causing issues with maintaining
file capabilities.  They aren't necessary so let's just remove them.

Change-Id: I06c90fdc85655986142b936cadbe04d75dd27427
Closes-Bug: 1714604
2017-09-01 17:29:15 -05:00
Jenkins
39d84d2059 Merge "Use [[ for =~ matches" 2017-08-29 05:59:42 +00:00
Ian Wienand
c448864901 Use [[ for =~ matches
Avoid incorrect use of [ with =~ matching

I guess this doesn't trip "-e" because it's in an if-conditional.  I'm
looking at making bashate detect this; maybe we can run bashate over
things we know are scripts

Change-Id: Ia3fe2b978fae5bdaadbb1789058180d3ad950d00
2017-08-28 17:01:03 +10:00
Julia Kreger
6d64a2aee6 Fix cylical systemd config for dhcp-all-interfaces
In Ubuntu/Debian, the default dependencies cannot be relied
upon as we enter into a cyclical dependency relationship which
prevents the unit from starting.

Added the required configuration to the systemd unit file.

This issue has also been observed in glean[0], which has a nearly
identical unit file for interface start-up.

[0]: https://review.openstack.org/#/c/485748
Closes-Bug: #1708685

Change-Id: I23ac9510d1a21c7073bd33f76ba66fa04a8be035
2017-08-25 15:51:23 +10:00
Jenkins
fa3797848b Merge "LVM support for dib-block-device" 2017-08-24 09:18:26 +00:00
Yolanda Robla
c2dc3dc78e LVM support for dib-block-device
This provides a basic LVM support to dib-block-device.

Co-Authored-By: Ian Wienand <iwienand@redhat.com>

Change-Id: Ibd624d9f95ee68b20a15891f639ddd5b3188cdf9
2017-08-24 16:22:56 +10:00
Jenkins
02601a1295 Merge "Increase timeout for removal" 2017-08-16 09:45:01 +00:00
Jenkins
ea23aa13a5 Merge "Add netbase to ensure /etc/protocols is placed for debian" 2017-08-15 00:37:27 +00:00
Yolanda Robla
81f495ad00 Increase timeout for removal
Under certain environments, this timeout was causing failures
because it was too short. Increasing to 10, to give time to
perform the specified tasks.

Change-Id: I01dd3553f38e1137b2fcb04b4ee12202be3ad1a8
2017-08-11 16:29:26 +02:00
Julia Kreger
f19c45eb29 Add netbase to ensure /etc/protocols is placed for debian
Many programs rely upon /etc/protocols to be present
however the default debian image that is generated lacks
/etc/protocols. This is observable when building an image
for use with ironic via the ironic-agent element, since
the IPA agent fails to start as python needs /etc/protocols
to open a socket connection.

Added to debian-minimal as it is inherited into the debian
element.

Change-Id: Icc81635870961943707cf6b3f61a9ddbd51cb8fd
Closes-Bug: #1708531
2017-08-11 14:17:30 +00:00
Ian Wienand
a88a768e98 Clear up debian element documentation
There is some confusion in the readme's over what is happening.  The
original change (Iaf46c8e61bf1cac9a096cbfd75d6d6a9111b701e) split out
debian-minimal and made debian "... simply be a collection of the
extra things we do to make it look like a cloud-init based cloud
image"

Make this clearer in the documentation

Change-Id: Ibe6fad9c67b70a5e31e43e06419968135174fef3
2017-08-09 13:15:38 +10:00
Jenkins
e04cf78fa5 Merge "Bump fedora/fedora-minimal DIB_RELEASE 26" 2017-08-08 00:26:45 +00:00
Dave Hill
6c2b1465cc Clear /etc/machine-id to avoid duplicate machine-ids
Deploying many nodes with the generated image shouldn't have the same
/etc/machine-id so clearing it and letting systemd generate a new
id upon first boot seems to be the best way to achieve this.

Change-Id: I73d0577d31464521b3989312fd9d982a1312a268
Closes-bug: 1707526
Closes-bug: 1672461
2017-08-06 13:56:58 -04:00
Jenkins
ced9b51f6e Merge "Allow users to specify partition type in the MBR PTE" 2017-08-04 05:19:02 +00:00
Paul Belanger
7cbbee7ea3 Bump fedora/fedora-minimal DIB_RELEASE 26
Fedora 26 is now the latest release:

  https://fedoraproject.org/wiki/Releases/26/Schedule

We are building and using these in infra now

Change-Id: I012c2d28255be274e88abc2751d968bafaf76fbb
Depends-On: Ieba5f69020a13681074f72cfca2955071801b63a
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2017-08-04 14:18:05 +10:00
Ian Wienand
818b75af41 Move selinux packages to redhat-common
Change I008f8bbc9c8414ce948c601e3907e27764e15a52 has shown that we
build redhat images without the "semange" tool available, which comes
from the policycoreutils-python package (see also
I3f9e2c322d042a5dddba33451c0fc21a4d32a88a).

I403e7806ae10d5dd96d0727832f4da20e34b94c7 added some of the selinux
libraries to yum-minimal for ansible support, but not to others.

Given both these changes, it seems that selinux[-targeted],
libselinux[-python] and policycoreutils[-python] can reasonably
considered part of all base images.  Move the selinux related packages
into redhat-common.

This also adds it explicitly to install_test_deps.sh.  It was actually
being dragged in by the docker install, but is a required component
for building (should be in bindep, but not there with that yet).

Change-Id: Idd4ae71ee6deee84604823b6b5dc4a845f316e01
Related-Bug: #1707788
2017-08-01 11:08:54 +10:00
Amrith Kumar
52faa0e1d9 Allow users to specify partition type in the MBR PTE
The MBR Partition Table Entry (PTE) allows one to specify many
possible partition types and one of the benefits of this is being able
to specify the CHS variant or the LBA variant.

By default, LBA only creates partitions of type 0x83 (of course,
that's only because the documentation doesn't tell you how to make it
do anything else).

I will take up Ian's suggestion in patch set 2 for a more rigorous
test in an independent patch set.

Change-Id: If3068535980eac2e58d4025444c65147a8c7fedc
Closes-Bug:#1703352
2017-07-29 06:34:25 -04:00
Jenkins
308783d382 Merge "Switch openSUSE to 42.3 by default" 2017-07-28 05:14:23 +00:00
Alfredo Moralejo
b1961e14ea Use SELinuxfs to check selinux status
Currently, the cleanup script is using existence of
semanage binary to check if selinux is enabled. However
this is misleading and can lead to problems when selinux
is disabled in a system where the binary exist.

This patch changes the detection logic to use /sys/fs/selinux
directory which is a in-memory filesystem created only when
selinux is really enabled.

Change-Id: I008f8bbc9c8414ce948c601e3907e27764e15a52
Related-Bug: 1706386
2017-07-26 18:57:25 +02:00
Dirk Mueller
1c4c4fd734 Switch openSUSE to 42.3 by default
This is the latest stable release, so we should default to it.

Change-Id: I05643787002d339ccbf7a718847fe4ed6f39eacc
2017-07-26 08:56:02 +02:00
Jenkins
609bcee27b Merge "zypper: Clean caches and don't cache packages locally" 2017-07-26 02:25:40 +00:00
Markos Chandras
81e72d4045 elements: zypper-minimal: Install tar package
tar is an essential package but nothing pulls it explicitly. This causes
some issues in the openSUSE CI jobs like the following one

"Failed to execute tar: No such file or directory", "Failed to write
file: Broken pipe", "Failed to retrieve image file. (Wrong URL?)",
"Exiting."], "stdout": "", "stdout_lines": []}

Just like 'sed', add 'tar' to the list of packages for the openSUSE
minimal builds.

Change-Id: Ia36e3d9fd6b78862a6831ba80b43d4614a349ca0
2017-07-25 16:27:25 +01:00
Jenkins
a6da39acb8 Merge "Move setfiles to outside chroot with runcon" 2017-07-24 02:04:21 +00:00
Ian Wienand
5089e4e541 Move setfiles to outside chroot with runcon
As described in the comments inline, on a selinux enabled kernel (such
as a centos build host) you need to have permissions to change the
contexts to those the kernel doesn't understand -- such as when you're
building a fedora image.

For some reason, setfiles has an arbitrary limit of 10 errors before
it stops.  I believe we previously had 9 errors (this mean 9
mis-labeled files, which were just waiting to cause problems).
Something changed with F26 setfiles and it started erroring
immediately, which lead to investigation.  Infra builds, on
non-selinux Ubuntu kernel's, would not have hit this issue.

This means we need to move this to run with a manual chroot into the
image under restorecon.

I'm really not sure why ironic-agent removes all the selinux tools
from the image, it seems like an over-optimisation (it's been like
that since Id6333ca5d99716ccad75ea1964896acf371fa72a).  Keep them so
we can run the relabel.

Change-Id: I4f5b591817ffcd776cbee0a0f9ca9f48de72aa6b
2017-07-24 10:14:07 +10:00
Dirk Mueller
bfeb9d9e99 zypper: Clean caches and don't cache packages locally
For builds inside the infra, we don't want to pack the cache
inside the image (as it might be different at the time the image
runs). In an opensuse-minimal image this saves about 10MB of image
size.

Change-Id: I5ecabd46f0a662798bda3e4468395ad8308d0055
2017-07-23 17:24:24 +02:00
Jenkins
55971717b6 Merge "elements: openstack-ci-mirrors: Use openSUSE mirrors for gating jobs" 2017-07-22 05:22:34 +00:00
Jenkins
e029af993b Merge "Remove DIB_[DISTRO]_DISTRIBUTION_MIRROR" 2017-07-22 05:22:04 +00:00
Jenkins
7a70299668 Merge "Enable console during kernel boot on Power" 2017-07-20 03:55:19 +00:00
Jenkins
d66dbc679c Merge "The correct option for label name in fat and vfat is '-n'" 2017-07-20 03:54:23 +00:00
Ian Wienand
7ffe6856d6
Add -m flag to setfiles for Fedora 26
As described in the comment and associated bugzilla, the behaviour of
setfiles has changed in Fedora 26 to require "-m" situations where
labeled file-systems are mounted below non-labeled file-systems.  Our
loopback/chroot system appears to trigger this nicely, leading to a
setfiles call that does nothing without this.

Change-Id: I276c6f6a4fb44f4bea5004f6b4214f94757728ae
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2017-07-19 08:01:19 -04:00
Ian Wienand
6ffde2e596 yum-minimal: pre-install coreutils
As described in the referenced bug, the dependency solver in yum
doesn't handle weak dependencies well and in some cases, such as
Fedora 26, can end up choosing coreutils-single (the busybox-esque
single binary) instead of actual coreutils, which then causes problems
with conflicting packages later.

Change-Id: I2907bf3b74c146986b483d52cc6ac437036330b4
2017-07-18 14:51:18 +10:00
Ian Wienand
b8ad9c2e37 Force install during pip-and-virtualenv
On a system where the packaged pip/virtualenv is up-to-date with
upstream (such as Fedora 26 ... for now), we don't reinstall, which
then violates a bunch of assumptions later on.  Force install.

Change-Id: I6ebcda0351997fa7e32f0e6e77a98b2c33764e3f
2017-07-18 14:50:09 +10:00
Ian Wienand
da90ef4743 Fix latest-limit command line
It turns out dnf argparse can't handle negative numbers without "=".
It's actually documented in the man page

    --latest-limit <number> ...  If <number> is negative skip <number>
      of latest packages. If a negative number is used use syntax
      --latest-limit=<number>

But who reads that :)  This started failing with Fedora 26

Change-Id: I884af94c07fa11b010f69863047a04711b14f21e
2017-07-18 13:17:10 +10:00
Jenkins
016606c81d Merge "opensuse-minimal: install glibc-locale" 2017-07-18 00:40:48 +00:00
Dirk Mueller
59721d3c74 opensuse-minimal: install glibc-locale
We expect LC_ALL for non-C locales to be working inside
images, so always install glibc-locale for openSUSE.

Change-Id: I8fe92773e377539070d9d9fe2960a6202bb80a18
2017-07-17 22:50:25 +02:00
Markos Chandras
6be09152c2 elements: openstack-ci-mirrors: Use openSUSE mirrors for gating jobs
In preparation for promoting the openSUSE jobs to voting ones we should
use the OpenStack mirrors. As such, the opensuse elements are modified
to make use of the DIB_DISTRIBUTION_MIRROR variable which is normally
exported by the openstack-ci-mirrors element.

Change-Id: Ie588c1c1eec13190cfb2ec718ba51f8c9878283f
2017-07-17 10:54:03 +01:00
Jenkins
c18a3ff029 Merge "Replace architecture-emulation-binaries with qemu-debootstrap" 2017-07-17 05:36:09 +00:00
Ian Wienand
3457d2f8e8 Remove DIB_[DISTRO]_DISTRIBUTION_MIRROR
We added the DIB_distro_DISTRIBUTION_MIRROR arguments with
I92964b17ec3e47cf97e3a3091f054b2a205ac768 as a way that we could
source a list of mirrors and then have the distro elements choose
which one applied to them.

However, this hasn't worked out to be so useful.  The
openstack-ci-mirrors element is working as a mirror setup script -- it
translates the openstack CI mirror list variables into the generic
"DIB_DISTRIBUTION_MIRROR" as appropriate for each distro's build.
Also, it turns out there's other things that need to be done, such as
turning off gpg checking, which mean the idea of "just export
variables" hasn't turned out as valid ... you need actual code
involved to get it right.

AFAICT we never actually documented these, and they do not seem to be
in use.  They have caused considerable confusion when dealing with new
platforms as we try to keep consistency.  Remove them.

[1] http://codesearch.openstack.org/?q=DIB_.*_DISTRIBUTION_MIRROR&i=nope&files=&repos=

Change-Id: Ifc4ab700631ffdfbe790068558f670f9a11dde5e
2017-07-17 14:47:31 +10:00
Jenkins
787e76b916 Merge "Remove additional Bumblebee repository for opensuse element" 2017-07-17 00:50:46 +00:00
Jenkins
64a8c6e1dc Merge "zypper-minimal: No point in preserving the environment here" 2017-07-17 00:26:11 +00:00
Amrith Kumar
59f416ae20 The correct option for label name in fat and vfat is '-n'
The code in mkfs correctly extends the command line with a '-n' for
vfat but does not currently do it for fat. This means that mkfs for
fat ends up with a '-L' which is what you'd do for everything like
ext[234].

The change just treats fat like vfat in the one place where this check
is required.

Change-Id: If65dfd949acdadff33a564640fb42ea73026a786
Closes-Bug: #1703063
2017-07-15 22:48:52 -04:00
Dirk Mueller
02d33f2ca7 zypper-minimal: No point in preserving the environment here
Change-Id: I46442e841d1f718b683bca4d2a348f0013306907
2017-07-13 22:50:47 +02:00
Dirk Mueller
05ba445ade Remove additional Bumblebee repository for opensuse element
The purpose of the openSUSE element is to build openSUSE distribution
based images, so an additional community repo shouldn't be pulled into
the image. In addition the dkms dependency is blacklisted for SUSE
in the dkms element anyway, so this should be a noop.

Change-Id: I0aa06d9f4f110546032f910e3361840693d02de7
2017-07-11 23:24:05 +02:00
Jenkins
0327d775f1 Merge "pip-and-virtualenv: Install python3 on openSUSE" 2017-07-11 08:11:16 +00:00
Jenkins
997a6ea6cb Merge "Add symlink test for resolv.conf restore" 2017-07-11 08:07:23 +00:00
Rafael Folco
bfdf7dc0f6 Enable console during kernel boot on Power
On Power systems console should be added the kernel command line
in the following order: 'console=tty0 console=hvc0'.
The first one is the graphical console. The last one is the serial
console. The kernel enables all the consoles pointed through the
kernel command line. However, only the last one will receive
input/output during kernel boot. All the other consoles will be
enabled after the boot.

Change-Id: I0069f608e0ab104d3778954e033fb82ed5ea7693
2017-07-07 17:55:56 +00:00
Amrith Kumar
43e32116bd fix readme.rst to reflect correct environment variable
The readme.rst incorrectly refers to the environment variable
DIB_APT_KEYS which should be DIB_ADD_APT_KEYS. See [1] for usage in
code.

This is a minor correction to the readme only, no runnable code is
modified.

[1] http://git.openstack.org/cgit/openstack/diskimage-builder/tree/diskimage_builder/elements/dpkg/extra-data.d/01-copy-apt-keys#n23

Change-Id: I04129cef9f40ec75a206c126bfd40ee61e4e6a2b
2017-07-06 22:54:08 -04:00
Ian Wienand
5fa6e3e13c Add symlink test for resolv.conf restore
We replace the base resolv.conf with an "outside" copy so that
resolving works when we're in the chroot.

Installing resolvconf package modifies the in-chroot resolv.conf to a
symlink (to /var/run) which it wants maintained in the final image.
We have the existing "immutable" check for a created resolv.conf file,
but no eqivalent for a symlink.

This adds a check to see if the resolv.conf is a symlink and leave it
alone if it is, assuming it has been re-created in the chroot.

I have tested this with ubuntu-minimal+resolvconf with
dhcp-all-interfaces and the system seems to work with resolvconf
working correctly.

Change-Id: Idd5a26e9d55979bd951577d5b098ed4bfba91ad3
2017-07-06 13:48:27 +10:00
Jenkins
e8ad2a3799 Merge "elements: pip-and-virtualenv: Use common packages for openSUSE" 2017-07-04 11:20:35 +00:00
Markos Chandras
5fe35b0d7a pip-and-virtualenv: Install python3 on openSUSE
The python3 package actually contains some core modules (like the xml
one) which are not present in the python3-base on which is pulled by
the python3-devel package. As such, it's best to have it installed
similar to python-xml for python2.

Change-Id: I5cd5d1127ae62d6753c2ace44965179c5400bb9a
2017-07-04 08:40:34 +01:00
Jenkins
fad72745d2 Merge "Support for Cloud Images on ppc64le for rhel7 and centos7" 2017-07-04 01:13:24 +00:00
Jenkins
2ed643a734 Merge "Use the dib python to do cleanup" 2017-06-29 21:22:36 +00:00
Jenkins
6b45497ff6 Merge "Remove centos and rhel elements" 2017-06-29 21:16:57 +00:00
Jenkins
f0fb835db9 Merge "Avoid hanging endlessly on unreachable cache urls" 2017-06-29 08:03:25 +00:00
Chhavi Agarwal
6d69d7909d Support for Cloud Images on ppc64le for rhel7 and centos7
In order to support {CentOS,RHEL}7 for building cloud images we need to
handle the differences in grub packaging from Ubuntu.  We also need to
populate the defualt location for cloud images for CentOS builds.

Change-Id: Ie0d82ff21a42b08c4cb94b7a5635f80bfabf684e
2017-06-29 15:44:26 +10:00
Dirk Mueller
959226c55e Avoid hanging endlessly on unreachable cache urls
When a download redirector redirects to a broken mirror, timeout
quickly rather than waiting until the overall job is being timed out.

Change-Id: If7eb63d406aaf61f71aa9203cf708c474aa63fd0
2017-06-28 22:14:55 +02:00
Markos Chandras
c46b6da65f elements: pip-and-virtualenv: Use common packages for openSUSE
The 'packages' variable already contains the packages we need so
use it instead of duplicating the packages.

Change-Id: Id22e1862f9654e66252d03a0fed9839cf004d750
2017-06-28 17:59:25 +01:00
Ian Wienand
a00d02f6a1 Remove centos and rhel elements
Several people have popped up in IRC recently with failures in these
elements.  Without Python 2.7 available in the image they are
unsupported (OpenStack hasn't supported it for a long time).  Remove
these to avoid further confusion.

The centos/centos7 DISTRO split that has happened with centos-minimal
is unfortunate but I don't think it helps to rename centos7/rhel7 ATM.
To summarise; DISTRO=centos7 means image based build,
DISTRO=centos && DIB_RELEASE=7 means the minimal build.

In the future, I think it is important that the minimal builds and
image builds set the same DISTRO.  This reflects that "upper" layers
shouldn't care about the exact building of the lower layers.  I see
CentOS 8 going one of two ways

1) the changes are so significant, we start separate centos8 /
centos8-minimal elements.  They both set DISTRO=centos8 (and
DIB_RELEASE to point-release maybe?).  This means we have to update
all "if DISTRO == centos || DISTRO == centos7" branches to also check
for "centos8".  Evenually (!)  "centos" goes away for versioned DISTRO
only

2) we restore centos element with DISTRO=centos and DIB_RELEASE=8, and
centos-minimal remains the same.  This means we have to audit all "if
DISTRO == centos" calls to make sure they're appropriate for version 8
(stick a "&& DIB_RELEASE=7" on them all basically).

I'm not sure we can fully decide until we start to see excatly how the
distro switching/matching bits look, but (2) is consistent with Ubuntu
and probably the preferred solution.

Some "rhel" parts have been cleaned up.  More could be done in
rhel-common, but given our lack of coverage of that I'd prefer to
leave it for now.

Change-Id: I6ea784116ef59ca22878c8512c963f29c815a00a
2017-06-28 12:26:24 +10:00
Tony Breeds
c383428727 As far as block-device layout is concerned ppc64le == ppc64el
Change-Id: I06a89f256f66eba2f73dd110f5d8a61e0f0a924c
2017-06-27 00:34:33 +00:00
Jenkins
c1facd7980 Merge "Fix mkfs use wrong label option for vfat" 2017-06-26 02:41:30 +00:00
Ian Wienand
b0e0dd991c Move image download tests to default skip
The image download tests have long been too unreliable for the gate.
We need to cache the base images similar to how devstack caches it's
testing images.  Let's move them to non-voting jobs for the time
being.

This means that the gate jobs are now all based on "-minimal" and are
using infra mirrors.  Unfortunately, there is still some unreliability
because we currently have issues with infra mirrors being very slow
after AFS updates, leading to job timeouts.  But we're on the right
path...

Also, I noticed we don't have tests of the "ubuntu" image-download
based tests, which were tacitly being tested by apt-sources before we
moved that to -minimal.  Add simple tests for these.

Change-Id: Ie33ee49656872467ef68d753210032156bb6b2cb
2017-06-23 10:58:47 +10:00
Xinliang Liu
178db0c97b Fix mkfs use wrong label option for vfat
For vfat type, mkfs should use '-n' option for label.
e.g.:
mkfs -t vfat -n LABEL-STRING

Change-Id: I1414c5b8e0aeb240c3e6884e35ba75dde677db0c
2017-06-22 14:50:53 +08:00
Jamie Lennox
aa03e24c30 Use the dib python to do cleanup
In a system where python2 is not installed and /usr/bin/python is not
linked then the cleanup process will fail trying to invoke the python
script. Use the previously determined DIB_PYTHON_EXEC if it's available.

Change-Id: I128292808ccef92cc1803988b35caae5aa6fa541
2017-06-22 11:20:51 +10:00
Clark Boylan
559de43694 On suse the python2 dev package is python-devel
This was previously defined as python2-devel (which is what rhel uses),
but the actual package name is python-devel. See:

  https://software.opensuse.org/package/python-devel

Change-Id: Id61e5b05772d10c32b33d3e70cb64d5ebdcba6e4
2017-06-21 15:52:05 +10:00
Ian Wienand
18a0d970fa Move ironic-agent test to fedora-minimal
I'm uncertain as to why this is using the "fedora" element for testing
... but it requires downloading the fedora .qcow on every test which
has shown to be unreliable.  An easy thing to do is to switch it to
fedora-minimal; that will only involve downloads from local mirrors in
the gate.

Add redhat-rpm-config for minimal.  I admit I have not fully gone
through why this is not pulled in.  It's been an issue since
I459f2203fa145049dda185da952813118193d573 and there's all sorts of
bugs.

Change-Id: I37458e3926dae32a259bd5aa9efc645561b029a0
2017-06-21 15:05:36 +10:00
Ian Wienand
649f0b66d9 Start at using CI mirrors for fedora/centos
fedora/centos-minimal don't obey DIB_DISTRIBUTION_MIRROR currently.  I
don't really want them too -- we want to be able to separate the
mirrors used during the build process from those embedded into the
final image.  Add DIB_YUM_MINIMAL_BOOTSTRAP_REPOS which is a directory
with repo files to use during the install.

This introduces setup-gate-mirrors.sh which is intended to setup
repo/sources/whatever files in the openstack gate that point to the
local region mirror.  It pulls the info from the mirror_info.sh script
on each CI node.

The openstack-ci-mirrors element is updated to export these variables.
elements are updated to depend on it.  Tests are restored

Change-Id: I7604fc4d41cb1483be16b8d628a24e8fc764f515
2017-06-21 12:02:27 +10:00
Ian Wienand
f0b70211c6 Use local mirror for ubuntu-minimal jobs
This adds "openstack-ci-mirrors" element which performs various
settings to get builds using local mirrors.  As a first step, we
convert ubuntu-minimal jobs

The main trick is that since infra mirrors are created with rerepo
they are not signed (they are recreated, not cloned, and not signing
is seen as a feature in that it deters external use).  So we need to
instruct debootstrap to ignore signing and also turn it off for
in-chroot apt.  Other than that, the existing DIB_DISTRIBUTION_MIRROR
works to redirect installs.

Remove "restricted" as it's not mirrored, and I don't think we want it
in here by default.

(I think DIB_DISTRIBUTION_MIRROR is a bit of an anti-pattern, because
it leaves the mirrors in the final image -- just because you use them
to build, doesn't mean you want them at runtime).  But we don't need
to fix that now, and we don't use any created images.)

This pauses fedora testing until the next change, which moves to using
local mirrors for testing on fedora/centos

Change-Id: I778bd05a1e615c27edf1c9f0a1409119a6b3a850
2017-06-21 12:01:31 +10:00
Ian Wienand
0d37351031 Move apt-sources to ubuntu-minimal / move debian to skip list
The gate is currently extremley unstable, and these two issues are
causing most of the problems.  We need to commit them atomically so we
can get anything moving again

---

The gate is very unstable downloading the ubuntu tarballs from
upstream at the moment.  Move this to ubuntu-minimal which, in a later
change will source files from our local mirror.

We need a caching mechanism for these large files to avoid this
instability.  This is future work for the various image-based jobs.

---

Move debian to default skip lists

I don't know if it's mirrors being worked hard for the Stretch
release, but this is constantly failing the gate.  I will move this to
the -nv extras job

I am working on having the voting job use local mirrors for
everything.  Unfortunately debian infra mirrors don't have stretch yet
and we need to do some fiddling to get "stable" available.  Once we
have all this, we can consider making it voting again.

Change-Id: Iaf7b3888ef06c7aef63cbf76a94b33f96bc9c5c2
2017-06-21 10:34:53 +10:00
Ian Wienand
5d5fa06e5c Sync after writing partition table
We introduced the "settle" in
I90103b59357edebbac7a641e8980cb282d37561b thinking that maybe kpartx
had not finished writing the partition.  This probably wasn't a bad
first assumption, since we used to have this -- but is seems
insufficient.

The other failiure here seems to be if kpartx hasn't actually seen the
updated partition table in the image, so it has correctly (in it's
mind) not mounted the partition.

Looking at strace of fdisk run manually on a loopback, it will do a
fsync on the raw device after writing and then a global sync as it
exits.

This replicates this; we flush and fsync in mbr.py in the exit handler
after writing the partition, before closing the file (i've updated one
of the unit tests to double-check the call).  In the partitioning.py
caller we execute a sync call too.

Since it does seem unlikely the "-s" option of kpartx is not working,
I've removed the udev settle work-around too.

Change-Id: Ia77a0ffe4c76854b326ed76490479d9c691b49aa
Partial-Bug: #1698337
2017-06-19 17:13:36 +10:00
Ian Wienand
a0f747932d Install systemd earlier for Stretch
Debian Stretch released as stable recently, and the init system is
less tightly specified in the base dependencies (for some info, see
[1]).  It seems, probably unintentionally, that in the previous
release systemd-sysv was brought in by debootstrap, but that is no
longer happening.

Add systemd as an early dependency of debian-minimal.

Remove the package-installs.yaml as that happens too late (other
things need to know the init system to write out service files, etc
and probe for systemd utils before package-installs).  As mentioned, I
do not believe the "only install systemd on testing" idea was actually
working here, because it was being brought in during the initial
debootstrap.

Update some documentation to explain what's going on

[1] https://lists.debian.org/debian-boot/2015/05/msg00156.html

Change-Id: Id67c0cf08728407d234976f9807d3bd71d12f758
2017-06-19 13:27:33 +10:00
Michael Johnson
250aeb5d21 Fix mkfs failure when loop device is not ready
There was a race in diskimage-builder where the mkfs call after a
kpartx -avs for the loop device would fail because the device was
not yet ready.  This adds a udevadm settle call after the kpartx
to make sure the udev event queue has cleared.

Change-Id: I90103b59357edebbac7a641e8980cb282d37561b
Closes-Bug: #1698337
2017-06-17 09:00:13 +10:00
Ian Wienand
f60dd38482 Add timestamp output filter
This adds a devstack-inspired output filter to standardise
timestamping.

Currently, python tools timestamp always (timestamp setup in
logging_config.py) but all the surrounding bash does not.

We have extra timestamps added in run_functests.sh for our own
purposes to get the bash timestamps; but this ends up giving us
double-timestamps for the python bits.  Additionally, callers such as
nodepool capture our output and put their own timestamps on it, and
again have the double-timestamps.

This uses a lightly modified outfilter.py from devstack to standardise
this.

All output is run through this filter, which will timestamp it.  I
have removed the places where we double-timestamp -- logging_config.py
and the prefix in dib-run-parts.

An env option is added to turn timestamps off completely (does not
seem worth taking up a command-line option for).  For callers like
nodepool, they can set this and will just have their own timestamps as
they collect the lines.

Since all logging is going through outfilter, it's easy to add a
--logfile option.  I think this will be quite handy; personally I'm
always redirecting dib runs to files for debugging.

I've also added a "quiet" option.  I think this could be useful in
run_tests.sh if we were to start logging the output of each test to
individual files.  This would be much easier to deal with than the
very large log files we get (especially if we wanted to turn on
parallel running...)

Change-Id: I202e1cb200bde17f6d7770cf1e2710bbf4cca64c
2017-06-16 15:58:50 +10:00
Jenkins
1324f5b7db Merge "Remove use of 'which'." 2017-06-11 09:30:34 +00:00
Ian Wienand
5ac8a98e9a PPC bootloader; install to boot partition
Using the newly exposed variables from the prior change, install the
ppc bootloader to the boot partition, not the underlying loopback
device.

Change-Id: I0918e8df8797d6dbabf7af618989ab7f79ee9580
2017-06-08 17:14:22 +10:00
Ian Wienand
6c394f5746 Pass all blockdevices to bootloader
Currently we only export "image-block-device" which is the loopback
device (/dev/loopX) for the underlying image.  This is the device we
install grub to (from inside the chroot ...)

This is ok for x86, but is insufficient for some platforms like PPC
which have a separate boot partition.  They do not want to install to
the loop device, but do things like dd special ELF files into special
boot partitions.

The first problem seems to be that in level1/partitioning.py we have a
whole bunch of different paths that either call partprobe on the loop
device, or kpartx.  We have _all_part_devices_exist() that gates the
kpartx for unknown reasons.  We have detach_loopback() that does not
seem to remove losetup created devices.  I don't think this does
cleanup if it uses kpartx correctly.  It is extremley unclear what's
going to be mapped where.

This moves to us *only* using kpartx to map the partitions of the loop
device.  We will *not* call partprobe and create the /dev/loopXpN
devices and will only have the devicemapper nodes kpartx creates.
This seems to be best.  Cleanup happens inside partitioning.py.
practice.  Deeper thinking about this, and more cleanup of the
variables will be welcome.

This adds "image-block-devices" (note the extra "s") which exports all
the block devices with name and path.  This is in a string format that
can be eval'd to an array (you can't export arrays).

This is then used in a follow-on
(I0918e8df8797d6dbabf7af618989ab7f79ee9580) to pick the right
partition on PPC.

Change-Id: If8e33106b4104da2d56d7941ce96ffcb014907bc
2017-06-08 17:14:22 +10:00
Ian Wienand
1d1e4ccb3e Move rollback into NodeBase object
Currently we pass a reference to a global "rollback" list to create()
to keep rollback functions.  Other nodes don't need to know about
global rollback state, and by passing by reference we're giving them
the chance to mess it up for everyone else.

Add a "add_rollback()" function in NodeBase for create() calls to
register rollback calls within themselves.  As they hit rollback
points they can add a new entry.  lambda v arguments is much of a
muchness -- but this is similar to the standard atexit() call so with
go with that pattern.  A new "rollback()" call is added that the
driver will invoke on each node as it works its way backwards in case
of failure.

On error, nodes will have rollback() called in reverse order (which
then calls registered rollbacks in reverse order).

A unit test is added to test rollback behaviour

Change-Id: I65214e72c7ef607dd08f750a6d32a0b10fe97ac3
2017-06-08 17:14:20 +10:00
Ian Wienand
09dee46579 Move global mount tracking into state
Keep track of the mount-point ordering in a state variable, rather
than a global.  This path is tested by existing unit tests.

Note a prior change inserted the MountNode objects directly into a
list in self.state, which makes sorting quite easy as it can just
implement __lt__.  Unfortunately we still json dump the state, and
thus we can't have aribtrary objects in it (future work may be to
check keys inserted into the status object...).  So we have to do a
bit of wrangling with tuple lists and comparision functions here, but
it's not too bad.

Change-Id: I0c51e0c53c4efdb7a65ab0efe09a6780cb1affa8
2017-06-08 17:13:28 +10:00
Ian Wienand
886f925b13 Use global state to check for duplicate fs labels
As we add file-systems, add them to global state and check the labels
are uniqiue.  Add a unit test and remove the old global value.

Bonus fixup to the length check, and a test for that too.

Change-Id: I0f5a96f687c92e000afc9c98a26c49c4b1d3f28d
2017-06-08 17:13:28 +10:00
Ian Wienand
b708918b85 Remove 'state' argument from later cmd_* calls
With I468dbf5134947629f125504513703d6f2cdace59 each node has a
reference to the global state object.  This means it gets pickled into
the node-list, which is loaded for later calls.  There is no need to
reload the state.json it and pass it for later cmd_* calls, as the
nodes can see it via the unpickled self.state

Change-Id: I9e2f8910f17599d92ee33e7df8e36d8ed4d44575
2017-06-08 17:13:28 +10:00
Ian Wienand
824a9e91c4 Add state to NodeBase class
Making the global state reference a defined part of the node makes
some parts of the block device processing easier and removes the need
for other global values.

The state is passed to PluginNodeBase.__init__() and expected to be
passed into all nodes as they are created.  NodeBase.__init__() is
updated with the new paramater 'state'.

The parameter is removed from the create() call as nodes can simply
reference it at any point as "self.state".

This is similar to 1cdc8b20373c5d582ea928cfd7334469ff36dbce, except it
is based on I68840594a34af28d41d9522addcfd830bd203b97 which loads the
node-list from pickled state for later cmd_* calls.  Thus we only
build the state *once*, at cmd_create() time as we build the node
list.

Change-Id: I468dbf5134947629f125504513703d6f2cdace59
2017-06-08 17:13:26 +10:00
Ian Wienand
e82e0097a9 Use picked nodes for later cmd_* calls
Currently the later cmd_* calls -- umount, cleanup, delete -- all
recreate the node graph by parsing the config file using
create_graph()

There is some need, however, to have a sense of global state when
building the node list.  The problem is, this is a one time operation
-- we do not want to rebuild that state for these later calls (see the
"loaded" checks in proposed
Ic3b805f9258128d5233b21ff25579c03487c7fcc).

An insight here seems to be that these cmd_* calls do not actually
want to re-parse the configuration file and rebuild the node list;
they just want to walk the node list in reverse with the state as
provided after cmd_create().

So, rather than re-creating the node list, we might as well just
pickle it, save it to disk along side the state dictionary dump and
reload it for cmd_*.

After this, I think we can safely have PluginBase.__init__() be passed
the state.  We will now know that this will only be called once,
during initial creation.

Change-Id: I68840594a34af28d41d9522addcfd830bd203b97
2017-06-08 17:10:10 +10:00
Ian Wienand
9a8b135267 Don't make image & loopdev functions static
You can't pickle a static method reference which complicates being
able to save the node graph when the "rollback" call-back wants to
hold references to these functions.  The outer module (localoop.py) is
small anyway, so from an organisation point of view the difference is
minimal.  Since these are really only called with parameters from the
containing class, they could be class methods with no parameters, at
the small expense of having to fiddle the mbr test-case a bit.

Change-Id: I6f9592a4295abe1b41294b79828bc2f3c2da01c6
2017-06-08 17:10:10 +10:00
Jenkins
60a5484ae8 Merge "Add env var to dump config graph" 2017-06-08 06:59:51 +00:00
Ian Wienand
d5c3863b87 Add env var to dump config graph
Make this a bit easier during debugging.  Add env var and some
developer instructions.

Change-Id: I34978ddb47d6642dfa22cae0f4c0543c0ba5475f
2017-06-08 05:04:58 +00:00
Ian Wienand
6fe1ef94f1 Use class as super() argument
Fix a few typos using the inherited class for super()

Change-Id: If9f2f423f136fb78ee93018d5c299d0dae603aad
2017-06-08 09:43:47 +10:00
Ian Wienand
90b56b3aab Move ppc block-device default to right $ARCH
The supported ppc ${ARCH} is "ppc64el" (at least in the gate testing
...) so move the file to that, so gets picked up by
block_device_create_config_file

Change-Id: I9273f35cdbfb0a62404461cbc1df9b2a92155fb0
2017-06-07 13:30:38 +10:00
Ian Wienand
89a85f6fbb Update tracing in block_device_create_config_file
Something seems to be going on with the ppc matching in the gate test.
Small updates to see what's going on...

Change-Id: Ie48cd4ce1f983a58932a577a43746240f6866936
2017-06-07 13:30:38 +10:00
Ian Wienand
7661da1341 Pad state dump
Because we append the function/line info after debug lines in the gate
logs, the pretty-print ends up not looking all that pretty.  Pad it.

Change-Id: Ice013428342614300cd51e8b7be56e79b75b31fc
2017-06-06 12:34:00 +10:00
Jenkins
ec70cb61f0 Merge "Trivial fix typos" 2017-06-05 05:54:50 +00:00
Ian Wienand
cdb1a95be1 Move "functional" unit tests under block-device
This is code motion with some small changes to make follow-on's
easier.

test_blockdevice_mbr.py is moved alongside the other tests.  It is
modified slightly to use the standard base class and remove a lot of
repeated test setup; a fixture is used for the tempdir (so it doesn't
have to be torn-down, and is removed properly on error) and the partx
args are moved into the setUp() so each test doesn't have to create
it.  No functional change.  renamed test_mbr.py for shortness.

test_blockdevice_utils.py is merged with existing test_utils.py.  No
change to the tests.

test_blockdevice.py is removed.  It isn't doing anything currently; to
work it will need to take an approach based more on mocking of calls
that require elevated permissions.  It's in history if we need it.

Change-Id: I87b1ea94afaaa0b44e6a57b9d073f95a63a04cf0
2017-06-05 12:22:52 +10:00
Vu Cong Tuan
cae44c7eea Replace assertRaisesRegexp with assertRaisesRegex
assertRaisesRegexp was renamed to assertRaisesRegex in Py3.2
For more details, please check:
https://docs.python.org/3/library/
unittest.html#unittest.TestCase.assertRaisesRegex

Change-Id: I705c958c0dbf1daa409ed29ccbc038426298c306
Closes-Bug: #1436957
2017-06-03 13:27:37 +07:00
Jenkins
5a045e036d Merge "dhcp-all-interfaces.sh - Add support for InfiniBand interface DHCP" 2017-06-02 06:11:19 +00:00
Jenkins
80cc1d0ea4 Merge "Adjust package mapping for SUSE family" 2017-06-02 02:56:16 +00:00
Dirk Mueller
d0a398c167 Adjust package mapping for SUSE family
package-installs.yaml is installing python-dev, not python2-dev,
so we need to adjust the mapping accordingly.

In addition, zypper-minimal used an dpkg specific package name,
while there is a SUSE equivalent (and zypper-minimal is anyway
SUSE family specific)

Change-Id: Ia9dd061fa46a514781808d62e5e93b03f75c6745
2017-05-31 21:09:53 +02:00
Dirk Mueller
f58bf252de Drop support for Ubuntu precise
Ubuntu 12.04 LTS reached its regular End of Life on April 28, 2017.

Depends-On: I5e145095a10db112bb27516bfe652d2cdc052a61
Change-Id: I64af4c5183d77a75dcd062895d19b0a1330c8da8
2017-05-31 14:36:30 +02:00
Jenkins
016b1f1522 Merge "Make BlockDeviceState implement a dict" 2017-05-31 10:55:32 +00:00
Jenkins
d1c49c1ae4 Merge "Refactor mount-point sorting" 2017-05-31 10:50:26 +00:00
Jenkins
b312c06dbb Merge "Decode string to bytes in dracut-regenerate" 2017-05-31 10:49:51 +00:00
Jenkins
09543cf52b Merge "Add state object, rename "results", add unit tests" 2017-05-31 05:52:24 +00:00
Jenkins
d0e0714f71 Merge "Test openSUSE 42.2/42.3 image builds" 2017-05-31 04:37:57 +00:00
Vu Cong Tuan
6a72052108 Trivial fix typos
Change-Id: Ib86aa9938fd852610ec0a6d8d868181f87bd2f24
2017-05-31 11:17:05 +07:00
Jenkins
2bdc154df5 Merge "drop deprecated map-services/packages from zypper element" 2017-05-31 02:11:21 +00:00
Jenkins
05d64b99ce Merge "Remove ccache" 2017-05-31 01:48:01 +00:00
Ian Wienand
4253cab773 Make BlockDeviceState implement a dict
While plugins treat the state as just a dictionary, it's nice for the
driver functions to keep state related functions encapsulated in the
state object singleton.  Wrap the internal state dictionary so we can
pass the BlockDeviceState directly without dereferencing.

Change-Id: Ic0193c64d645ed1312f898cbfef87841f460799c
2017-05-31 11:24:55 +10:00
Ian Wienand
35a1e7bee9 Refactor mount-point sorting
Currently we keep a global list of mount-points defined in the
configuration and automatically setup dependencies between mount nodes
based on their global "mount order" (i.e. higher directories mount
first).

The current method for achieving this is roughly to add the mount
points to a dictionary indexed my mount-point, then at "get_edge()"
call build the sorted list ... unless it has already been built
because this gets called for every node.

It seems much simpler to simply keep a sorted list of the
MountPointNode objects as we add them.  We don't need to implement a
sorting algorithm then, we can just use sort() and implement __lt__
for the nodes.

I believe the existing mount-order unit testing is sufficient; I'm
struggling to find a valid configuration where the mount-order is
*not* correctly specified in the configuration graph.

Change-Id: Idc05cdf42d95e230b9906773aa2b4a3b0f075598
2017-05-31 11:05:50 +10:00
Jenkins
edaf577bad Merge "Remove dracut-network element" 2017-05-31 00:14:01 +00:00
Mark Goddard
54765fd2f4 Remove dracut-network element
This element has not been functioning correctly for some time due to
an incorrect path to select-boot-kernel-initrd (should be /usr/local/bin).

The dracut-regenerate element can be used to regenerate dracut ramdisks
and is more flexible than this element.

Change-Id: I33d555ffd4a92b2948b2ea4a66b151f0422ccb8c
Closes-Bug: #1688546
2017-05-31 08:36:56 +10:00
Andreas Florath
b107606a75 Remove ccache
This patch removes the ccache handling from the base element.  For
mostly all systems this was never used at all.

This is working towards the removal of the base element from DIB

Change-Id: Ieb16ef612ebd98470993dcd6f55b3a22d37084ba
Signed-off-by: Andreas Florath <andreas@florath.net>
2017-05-31 08:28:09 +10:00
Mark Goddard
aa6c1d01a9 Decode string to bytes in dracut-regenerate
In python3, the standard out data returned by
subprocess.Popen.communicate() will in most cases be bytes rather than a
string and must therefore be decoded.

Without this fix we hit the following error:

TypeError: a bytes-like object is required, not 'str'

Change-Id: I6d75f867ebfdb925970c3397175214b9050d7632
Closes-Bug: #1694463
2017-05-30 16:15:06 +01:00
Dirk Mueller
5d39f83f74 Test openSUSE 42.2/42.3 image builds
Currently openSUSE 42.3 has entered feature freeze mode
so it is a good point in time to verify that 42.3 builds
are working successfully. Also test opensuse-minimal for
platforms that support it (need working zypper package)

Change-Id: I4c613e1e68cb7375c29d544bbf70b5da9bf21414
2017-05-30 13:07:04 +02:00
Ian Wienand
b85de3cd9e Add state object, rename "results", add unit tests
A couple of things going on, but I think it makes sense to do them
atomically.

The NodeBase.create() argument "results" is the global state
dictionary that will be saved to "state.json", and re-loaded in later
phases and passed to them as the argument "state".  So for
consistency, call this argument "state" (this fits with the change out
to start building the state dictionary earlier in the
PluginBase.__init__() calls).

Since the "state" is a pretty important part of how everything works,
move it into a separate object.  This is treated as essentially a
singleton.  It bundles it nicely together for some added
documentation [1].

We move instantiation of this object out of the generic
BlockDevice.__init__() call and into the actual cmd_* drivers.  This
is because there's two distinct instantiation operations -- creating a
new state (during cmd_create) and loading an existing state (other
cmd_*).  This is also safer -- since we know the cmd_* arguments are
looking for an existing state.json, we will fail if it somehow goes
missing.

To more fully unit test this, some testing plugins and new
entry-points are added.  These add known state values which we check
for.  These should be a good basis for further tests.

[1] as noted, we could probably do some fun things in the future like
make this implement a dictionary and have some saftey features like
r/o keys.

Change-Id: I90eb711b3e9b1ce139eb34bdf3cde641fd06828f
2017-05-30 20:39:00 +10:00
Jenkins
634e9ac043 Merge "Refactor: use lazy logging" 2017-05-30 09:30:28 +00:00
Jenkins
2275ccb97b Merge "allow uninstalls to fail on gentoo" 2017-05-30 07:38:56 +00:00
Matthew Thode
ce7ea9d34c
allow uninstalls to fail on gentoo
The cleanup of packages should be more opertunistic, if it's not there
then fail quietly.

Change-Id: I207a1162abc9ca5e9636b8de192f21424db0f569
2017-05-29 23:46:42 -05:00
Andreas Florath
f314df12c3 Refactor: use lazy logging
As described in pep282 [1], the variable part of a log message
should be passed in via parameter.  In this case the parameters
are evaluated only when they need to be.

This patch fixes (unifies) this for DIB.

A check using pylint was added that this kind of passing parameters to
the logging subsystem is enforced in future.  As a blueprint a similar
(stripped-down) approach from cinder [2] was used.

[1] https://www.python.org/dev/peps/pep-0282/
[2] https://github.com/openstack/cinder/blob/master/tox.ini

Change-Id: I2d7bcc863e4e9583d82d204438b3c781ac99824e
Signed-off-by: Andreas Florath <andreas@florath.net>
2017-05-30 14:39:58 +10:00
Ian Wienand
543dc1baa6 Add pylint with indent check
This is an initial creation of pylint with a basic indent checker.
Small issues corrected.  Job added to gate with
Ib554a284e92583cc1d6a5c2219b3922852ca4c73

Change-Id: I7e24d8348db3aef79e1395d12692199a1f80161a
Co-Authored-By: Andreas Florath <andreas@florath.net>
2017-05-29 16:12:35 +10:00
Jenkins
3ac57740c6 Merge "Move create_graph into config.py" 2017-05-27 10:22:08 +00:00
Jenkins
795ec082ed Merge "Create and use plugin/node abstract classes" 2017-05-27 10:22:01 +00:00
Jenkins
3b85cad420 Merge "Use networkx for digraph" 2017-05-27 10:10:31 +00:00
Jenkins
68393b6f9f Merge "Disable recommended package installations for zypper-minimal" 2017-05-26 17:34:02 +00:00
Jenkins
da7bfd2336 Merge "Add a more generic tree->graph parser" 2017-05-26 15:27:41 +00:00
Dirk Mueller
b4edb7d0eb Disable recommended package installations for zypper-minimal
This is consistent with how dpkg based images are configured
and minimizes the nodepool images drastically (avoid installing
texlive for example)

Change-Id: I98fb31bc0e06869e9770fae3dbd62f0d86acb879
2017-05-26 09:47:07 +02:00
Jenkins
da754520f6 Merge "Adding unit testing for configuration" 2017-05-26 06:45:54 +00:00
Ian Wienand
3fdd9df983 Move create_graph into config.py
This was suggested in a review comment in
I8a5d62a076a5a50597f2f1df3a8615afba6dadb2.  It works out quite nicely
because the BlockDevice() driver now doesn't need to know anything
about stevedore or plugins, and just works on the node list.  It also
simplifies the unit testing by not having to call create_graph through
a BlockDevice object.

Change-Id: I98512f6cf42e256d2ea8225a0b496d303bf357b8
2017-05-26 11:48:39 +10:00
Ian Wienand
deb832d685 Create and use plugin/node abstract classes
This completes the transitions started in
Ic5a61365ef0132476b11bdbf1dd96885e91c3cb6

The new file plugin.py is the place to start with this change.  The
abstract base classes PluginBase and NodeBase are heavily documented.
NodeBase essentially replaces Digraph.Node

The changes in level?/*.py make no functional changes, but are just
refactoring to implement the plugin and node classes consistently.
Additionally we have added asserts during parsing & generation to
ensure plugins are implemented PluginBase, and get_nodes() is always
returning NodeBase objects for the graph.

Change-Id: Ie648e9224749491260dea65d7e8b8151a6824b9c
2017-05-26 11:48:11 +10:00
Ian Wienand
75817ef205 Use networkx for digraph
This switches the code to use networkx for the digraph implementation.

Note that the old implementation specifically isn't removed in this
change -- for review clarity.  It will be replaced by a base class
that defines things properly to the API described below.

Plugins return a node object with three functions

 get_name() : return the unique name of this node

 get_nodes() : return a list of nodes for insertion into the graph.
  Usually this is just "self".  Some special things like partitioning
  add extra nodes at this point, however.

 get_edges() : return a tuple of two lists; edges_from and edges_to
  As you would expect the first is a list of node names that points to
  us, and the second is a list of node names we point to.  Usually
  this is only populated as ([self.base],[]) -- i.e. our "base" node
  points to us.  Some plugins, such as mounting, create links both to
  and from themselves, however.

Plugins have been updated, some test cases added (error cases
specifically)

Change-Id: Ic5a61365ef0132476b11bdbf1dd96885e91c3cb6
2017-05-26 11:42:10 +10:00
Ian Wienand
00da1982ce Add a more generic tree->graph parser
This moves to a more generic config parser that doesn't have plugins
parsing part of the tree.

I understand why it ended up that way; we have "partitions" key which
has special semantics compared to others keys and there was a desire
to keep it isolated from core tree->graph code.  But this isn't really
isolated; you have to reverse-engineer several module-crossing
boundaries, extras classes and repetitive recursive functions.

Ultimately, plugins should have access to the node graph, but not
participate in configuration parsing.  This way we ensure that plugins
can't invent new methods of configuration parsing.

Note: unit tests produce the same tree -> graph conversion as the old
method.  i.e. this is not intended to have a functional change.

Change-Id: I8a5d62a076a5a50597f2f1df3a8615afba6dadb2
2017-05-26 10:13:14 +10:00
Ian Wienand
7341542f2c Adding unit testing for configuration
Add a range of unit-testing for configuration parsing, graph
generation and mount-point generation.  Unfortunately there's some
global variable hacks, and some stubs, but it's a start.

Change-Id: I9e4f950c2c2ea656fc0c1a14594059fb4c62fa35
2017-05-26 09:44:19 +10:00
Dirk Mueller
f039a9b796 drop deprecated map-services/packages from zypper element
Change-Id: Ie3065dcc6aefccba93c02085e9977681d1b0535c
2017-05-25 23:43:21 +02:00
Jenkins
b7924c0c83 Merge "Produce API documentation" 2017-05-25 08:03:31 +00:00
Ian Wienand
78c0766bec Produce API documentation
There's an increasing amount of pydoc based documentation.  Output the
module reference and link it into the developers main page.

One fixup to the rst needed

Change-Id: I1d43a1fe1c735eb4559e3d2b40834d1c8115c586
2017-05-25 14:26:31 +10:00
Jenkins
0208f83a97 Merge "Set manifest permissions in the image" 2017-05-24 06:58:10 +00:00
Noam Angel
f1369a1add Set manifest permissions in the image
This is a follow-on to 57ef187632.

There's two things going on here; DIB_MANIFEST_IMAGE_DIR is *outside*
the chroot on the build host.  We copy the files here for posterity, I
guess.  MANIFEST_IMAGE_PATH is *inside* the chroot and are the files
we want to ensure are locked to root.

The prior change modified the permissions on DIB_MANIFEST_IMAGE_DIR.
So the first time you build, it works -- then the second time,
assuming you're using the same output filename, it hits the root-owned
manifest directories and causes a build failure.

I have built with this and checked that the manifest files in the
image are locked to root:

 $ virt-ls -a ./test.qcow2 -l /etc/dib-manifests
 total 32
 drwxr-xr-x  2 0 0  4096 May 24 03:39 .
 drwxr-xr-x 53 0 0  4096 May 24 03:39 ..
 -rw-------  1 0 0 15236 May 24 03:39 dib-manifest-dpkg-test
 -rw-------  1 0 0    35 May 24 03:39 dib_arguments
 -rw-------  1 0 0   137 May 24 03:39 dib_environment

Related-Bug: #1671842
Change-Id: I08319d0b5fcc461d40fe0be8427dcf0e37ad21e6
2017-05-24 15:20:55 +10:00
Jenkins
57c40a2ac4 Merge "Add dracut-regenerate elements" 2017-05-23 07:35:51 +00:00
Angel Noam
ba4f72f4f0 dhcp-all-interfaces.sh - Add support for InfiniBand interface DHCP
Change-Id: Ic2a9e2909a8086903257d43fbda97694baa339b4
2017-05-22 07:48:32 +00:00
Ian Wienand
bc58b5c515 Move parts of Partition creation into object
Move Partition() object creation into the actual Partition object,
rather than having the logic within the Partitioning() object

Change-Id: I833ed419a0fca38181a9e2db28e5af87500d8ba4
2017-05-20 06:44:39 +00:00
Ian Wienand
d013496ba0 Split partition into it's own file
Split Partition() into it's own file for clarity.  This will be
followed-on by less dependence between Partitions and Partition

Change-Id: I860f6a1787c0e4fe99f93919ac37cf7d80bfaae9
2017-05-20 06:44:39 +00:00
Ian Wienand
4e08765f87 Move exception to it's own file (again)
Moving the exception didn't cause problems in
I925ed62bdc808f0e07862f6e0905e80b50fbe942, but in later changes where
we split blockdevice.py up a bit more, we can get a bit tangled with
circular imports.

Change-Id: I8297483f64c4e1deecd5ec88ee40e9198bb83589
2017-05-20 06:44:39 +00:00
Andreas Florath
45272343c5 Add weights to digraph
Because in some cases (e.g. partitioning) the order is needed,
add weights to the digraph to get an (somewhat) stable
topological sort.

Change-Id: I5ef1acc6338ac93c593faa0eafe26cbed42ed887
Signed-off-by: Andreas Florath <andreas@florath.net>
2017-05-20 06:42:48 +00:00
Ian Wienand
9eb71a1fe0 Switch debian to deb.debian.org
Per [1] this is the "official" CDN mirror, which I think is the most
appropriate for the default.  I think this addresses the concerns
httpredir service, which I don't think ever quite got out of beta.

[1] https://wiki.debian.org/DebianGeoMirror

Change-Id: I55f2a00b8bbb0f0a20d3be229e4c2c32a7b69057
2017-05-19 20:10:40 +00:00
Jesse Keating
84d10dce57 Remove use of 'which'.
Instead, either use the bash built-in of type to ensure it exists. Since
which is an external dep, things can fail oddly in a constrained
environment.

Also add a dib-lint test for this.

Change-Id: I645029f5b5bfe1198c89ce10fd3246be8636e8af
Signed-off-by: Jesse Keating <omgjlk@us.ibm.com>
2017-05-19 12:43:36 -07:00
Yolanda Robla
f23ea63341 Add dracut-regenerate elements
This new element will allow to regenerate dracut
on the produced images, to enable different modules. It
relies on a yaml blob to specify modules and packages
needed. It defaults to installing lvm and crypt.

Change-Id: I292fb70cde41ee6053b7b81a67931bcdaaa6d664
2017-05-19 09:43:11 +02:00
Jenkins
94ab9e2e7e Merge "Set manifests to mode 600 and owner root" 2017-05-18 08:37:05 +00:00
Jenkins
ca04348393 Merge "Remove _config_error thrower" 2017-05-18 02:37:53 +00:00
Ian Wienand
b91207ae47 Remove _config_error thrower
"log and throw" is arguably an anti-pattern; the error message either
bubles-up into the exception, or the handler figures it out.  We have
an example where this logs, and then the handler in blockdevice.py
catches it and logs it again.

Less layers is better; just raise the exception, and use log.exception
to get tracebacks where handled.

Change-Id: I8efd94fbe52a3911253753f447afdb7565849185
2017-05-18 10:37:56 +10:00
Jenkins
812563f90c Merge "Remove PluginBase/NodePluginBase class" 2017-05-18 00:24:33 +00:00
Gregory Haynes
57ef187632 Set manifests to mode 600 and owner root
Manifests files can release sensitive information and therefore should
have restrictive permissions.

Change-Id: I64d6c830217a7d8b0172df2dc774079dcd1e2a68
Related-Bug: #1671842
2017-05-18 10:10:15 +10:00
Jenkins
c54e9fddfa Merge "Use fakelogger in test_blockdevice_mbr" 2017-05-17 23:59:18 +00:00
Ian Wienand
08e8471360 Use fakelogger in test_blockdevice_mbr
Capture logging instead of putting it on stdout

Change-Id: Ic0c9f25549a1984fecf8e912e5b14845e2363730
2017-05-17 18:49:12 +10:00
Ian Wienand
2d2b2725bd Remove PluginBase/NodePluginBase class
A majority of the "plugins" aren't implementing the plugin class.
Clearly we need some refactoring of the ideas here.  Remove for
simplicity.

Change-Id: If399a371b171f4fd17cfa5856fe55daca4c86e60
2017-05-17 09:03:42 +02:00
Yolanda Robla
5b305ffa7b Only unmount directories that are mounted
To avoid failures with double unmount, skip unmounting
the mountpoints that are managed by block device.

Change-Id: I228779eb9bf544a27a53e5017c87573023fd375a
2017-05-17 08:37:25 +02:00
Yolanda Robla
6d0b9abc0f Apply setfiles on all mountpoints
With new block device definition, where content of the image
can be mounted on different partitions, is not enough with
executing setfiles on root directory. Instead of that, expose
all the mountpoints on the image, and apply setfiles on them.

Change-Id: I153f979722eaec49eab93d7cd398c5589b9bfc44
2017-05-16 07:51:48 +02:00
Andreas Florath
e4e23897a1 Refactor: block-device filesystem creation, mount and fstab
This patch finalizes the block device refactoring.  It moves the three
remaining levels (filesystem creation, mount and fstab handling) into
the new python module.

Now it is possible to use any number of disk images, any number of
partitions and used them mounted to different directories.

Notes:

 * unmount_dir : modified to only unmount the subdirs mounted by
   mount_proc_sys_dev().  dib-block-device unmounts
   $TMP_MOUNT_PATH/mnt (see I85e01f3898d3c043071de5fad82307cb091a64a9)

Change-Id: I592c0b1329409307197460cfa8fd69798013f1f8
Signed-off-by: Andreas Florath <andreas@florath.net>
Closes-Bug: #1664924
2017-05-12 13:52:02 +02:00
Ian Wienand
b9c065de28 Remove args from BlockDevice() init
The args agument was only used to find the symbol for the getval
command.  Have the command pass the symbol to find in directly.  We
can therefore remove the args paramater to the BlockDevice() creation.

Change-Id: I8e357131b70a00e4a2c4792c009f6058d1d5ae9e
2017-05-12 09:36:23 +10:00
Ian Wienand
c74ba2fe74 Move to subparsers
Move argument parsing to subparsers, rather than positional arguments.
This better reflects the tool's role as a driver and allows
sub-commands to deal with arguments in a natural way.

Change-Id: Iae8c368e0f3fe47abfddb9e0a1558bd5b3423aee
2017-05-11 21:03:33 +10:00
Jenkins
b39b5cb5a5 Merge "Clear __init__.py from cmd move" 2017-05-11 10:35:22 +00:00
Ian Wienand
f76d68020b Clear __init__.py from cmd move
I accidentally dropped the clearing of this file when it moved to
cmd.py during rebase of I1919f6e865acae14ee95cd025c9c7b75ca266a9c

Change-Id: Ibe9fcde594770cb51c732cc253987308dc038083
2017-05-11 18:52:14 +10:00
Ian Wienand
0368052a2f Take --params from environment
DIB_BLOCK_DEVICE_PARAMS_YAML should be exported, and the
dib-block-device will take this as the value of --params.  Remove this
to simplify the command-line

Change-Id: I6764ed223ecd36f9d24e19f164b6a927380b410f
2017-05-11 17:36:51 +10:00
Ian Wienand
ff5b30db8a Create BlockDeviceCmd object
This creates a BlockDeviceCmd object to hold the main() function.
This doesn't really do anything different right now, but sets a base
for using argparse subparsers to handle the command-line

Change-Id: I4acf95ff4d554a3b4e7e2244ab1706631b98458f
2017-05-11 15:48:47 +10:00
Ian Wienand
a06c610a8c Move YAML parsing into cmd.py; default to env
This moves the YAML parameter parsing into the command-line driver.
It makes the argument optional so it can be taken from the environment
variable directly.  The parsed YAML is passed to the BlockDevice
object.

Change-Id: I6fa5e5b7d1fccfc7cf47d6e4a1fa6e560734680d
2017-05-11 15:22:41 +10:00
Ian Wienand
a9d8e51ec2 Move dib-block-device implementation into cmd.py
Move the command-line driver components into cmd.py.  No functional
change.

Change-Id: I1919f6e865acae14ee95cd025c9c7b75ca266a9c
2017-05-11 15:22:41 +10:00
Ian Wienand
47140293b6 Move blockdevicesetupexception.py into blockdevice.py
Less is more when it comes to code :)

Change-Id: I925ed62bdc808f0e07862f6e0905e80b50fbe942
2017-05-11 15:22:41 +10:00
Jenkins
fae8484e6b Merge "block_device: reorder imports" 2017-05-11 04:53:40 +00:00
Jenkins
bd7d7b052a Merge "Remove unused val_else_none" 2017-05-11 03:24:35 +00:00
Ian Wienand
9a8184ab4c Remove unused val_else_none
This function is unused

Change-Id: I8fe3e5452b95a639618a37a02f34b5b9f63ca43a
2017-05-11 11:53:08 +10:00
Ian Wienand
2a185ec6b6 block_device: reorder imports
Reorder imports to hacking standard (stdlib, third-party, project) [1]

[1] https://docs.openstack.org/developer/hacking/#import-order-template

Change-Id: I4aa73321e1e796ef6b8b079e42f90bf5c75388fe
2017-05-11 10:38:55 +10:00
Ian Wienand
0d8c4270c0 exec_sudo: check cmd for str, log output and raise exception
To avoid any confusion, commands passed to exec_sudo() should be a
list of "str"s.  Log a message if we see unicode issues.

This also adds a debug trace of all output.  stderr is captured.

This is modified to raise CalledProcessError on failure, like
check_call().  Calls that are ok to fail will need to explicitly catch
and ignore this.

The two calls that we expect to fail are wrapped

We wish to try rolling back if one of these command raises an
exception.  Modify the create handler to initiate rollback on all
exceptions.

Change-Id: Iee4fa41ffaf243e4728bf3a5eeec5c8fa8d2dadc
2017-05-11 09:45:25 +10:00
Ian Wienand
6f90daca7f Use check_output
The await function is essentially a non-standard check_output call.
Let's use standard calls to increase maintainability.

Change-Id: I2c25e1cd7122791fcaa86b46bd801e661471bc9e
2017-05-09 15:49:39 +10:00
Yolanda Robla
59a1fc6546 Add sort_mount_point method
As this method can be introduced without any dependency,
provide it on an independent change to simplify reviews.
This is a partial refactor based on
I592c0b1329409307197460cfa8fd69798013f1f8

Change-Id: Idaf3d2b3b3e23d0b9d6bc071d67b961a829ae422
Co-Authored-By: Andreas Florath <andreas@florath.net>
2017-05-05 17:29:53 +00:00
Yolanda Robla
fb70a49ba2 Add a test to validate we can build debian vms
This is a partial refactor from change
I592c0b1329409307197460cfa8fd69798013f1f8

Change-Id: I8822e68e41c4ebd47eea9ffed4557efc130a7bf7
Co-Authored-By: Andreas Florath <andreas@florath.net>
2017-05-05 19:17:39 +02:00
Yolanda Robla
9b75eda51a Introduce exec_sudo command
Add a new method in the block device library called
exec_sudo, so it can be reused.

This is a partial refactor of change
I592c0b1329409307197460cfa8fd69798013f1f8

Change-Id: Id621f6d029e1275a35c4fd3f19b57c8518076134
Co-Authored-By: Andreas Florath <andreas@florath.net>
2017-05-05 16:11:41 +02:00
Adam Harwell
196b44f38a Fix py3 error in block-device
This is breaking the octavia py3 gates.
Introduced by: I7aa4fe0466e44846d8fa3194575d446fe4b5b2e6

Change-Id: I3d86482a2999197a60a81d42afc5ef7a6e71e313
2017-05-04 15:55:22 -07:00
Yolanda Robla
e21935626b Refactor block-device base functions.
As part of the final steps, refactor the bits belonging
to block device and functions. This is a partial refactor
from I3600c6a3d663c697b59d91bd3fbb5e408af345e4

Change-Id: I7aa4fe0466e44846d8fa3194575d446fe4b5b2e6
Co-Authored-By: Andreas Florath <andreas@florath.net>
2017-05-04 19:54:18 +00:00
Yolanda Robla
08c36e4bf8 Add refactor of tree-like vs graph
Introducing the refactors of the block device to allow a tree-like
configuration, and start using it for the partitions level.
Based on patch I3600c6a3d663c697b59d91bd3fbb5e408af345e4

Change-Id: I58bb3c256a1dfd100d29266571c333c2d43334f7
Co-Authored-By: Andreas Florath <andreas@florath.net>
2017-05-03 05:27:43 +00:00
Ian Wienand
91ba21dd6c Set LC_ALL in disk-image-create
It seems that the redhat nodepool job is quite reliably geting a
"floating point" error during centos image build.  This happens after
03-yum-cleanup which is pruning the locales.  This might be a
red-herring, since the logs are full of

  /bin/bash: warning: setlocale: LC_ALL: cannot change locale (C.UTF-8)

I think in our recent de-puppetisation of hosts, something might have
changed that is setting LC_ALL=C.UTF-8 for the jenkins user, at least
on Ubuntu.  This is a problem for centos, as it doesn't have C.UTF-8
locale.  I then think using the invalid locale is what leads the the
floating-point error when doing some maths in dib-run-parts to
calculate runtimes.

We are currently overriding LANG, but we really want LC_ALL to ensure
this applies globally.

Change-Id: I8e7cae093c4b32e0d20b73ae0086f14c7cc6a9cb
2017-05-03 11:31:11 +10:00
Jenkins
629b81bc9b Merge "Refactor block_device: isolate the getval call" 2017-05-01 14:45:13 +00:00
Yolanda Robla
943f1ccf04 Refactor block_device: isolate the getval call
Add a new getval call that allows to retrieve values
from the block device. Also isolating the block device
information into a 'blockdev' dictionary entry, to better
return it with the getval command.

This is a refactor from the original code at
I3600c6a3d663c697b59d91bd3fbb5e408af345e4.

Change-Id: I93d33669a3a0ae644eab9f9b955bb5a9470cadeb
Co-Authored-By: Andreas Florath <andreas@florath.net>
2017-05-01 12:22:52 +02:00
Jenkins
83ea4a17b8 Merge "Clean out apt index caches at end of image build" 2017-05-01 08:09:07 +00:00
Jenkins
ead4a50c2a Merge "Make Gentoo package updates work more often" 2017-05-01 08:05:04 +00:00
Jenkins
c36ac99458 Merge "Turn off strict_id mode for Ec2 datasource" 2017-05-01 08:04:52 +00:00
Andreas Florath
803d40b0c6 Refactor block_device: passing command line parameters
The original approach was to pass each and every command
line parameter to the block device.  While the block device
functionality gets extended, this is not any longer practical.

Instead of passing in all the parameters separately this patch
collects these in a YAML file that is passed in to the block device
layer.

Change-Id: I9d07593a01441b62632234468ac25a982cf1a9f0
Signed-off-by: Andreas Florath <andreas@florath.net>
2017-04-23 09:51:18 +00:00
Matthew Thode
6f51fbe355
Make Gentoo package updates work more often
Some package updates are more complex and require things like --backtrack=99 to
be passed to emerge.  We also try harder to ensure the system is in a consistent
state as a last step.

Change-Id: Ia5d3514e8b2a6cb2d656ade997cebb798d9c0a47
2017-04-22 19:34:11 -05:00
Paul Belanger
1ce16a987b
Add yum-utils as EPEL dependency
With 8e822768f9 we added the ability to
disable the EPEL repository, however we need yum-utils to use
yum-config-manager.

Change-Id: Iea445f84494fd9a89fd93e9b35f920eb5e55211d
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2017-04-20 17:01:00 -04:00
Simon Leinen
0ff40cf63c Turn off strict_id mode for Ec2 datasource
Recent changes in the default configuration of cloud-init in Ubuntu
cause warnings when the Ec2 datasource is used on non-Amazon clouds,
see https://bugs.launchpad.net/cloud-init/+bug/1660385

We explicitly select the previous behavior when an Ec2 datasource is
desired.

Change-Id: Iebad8f6c0017fe08013dd5fe667c6132158b71cd
Closes-bug: 1683038
2017-04-19 09:47:56 +02:00
Paul Belanger
3e17ee3f48 Clean up dib-python symlink
We only need dib-python when we build the image, no need to leak it to
the final product.  Remove it in cleanup.d outside the chroot so
nothing can be using it.

Change-Id: I1e229caad7968fb3ab8e44ecdda427e174088d2d
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2017-04-12 14:33:57 +10:00
Ian Wienand
7a962e9d1c Skip python3-virtualenv on <= trusty
If DIB_PYTHON_VERSION is < 3 on the !redhat path, that means we're on
an older platform that may not have python3-virtualenv packages.  Skip
install.

Ensure the order of operations happens by forcing the installs

Also add a note about limited platform support (patches welcome :)

Change-Id: I18412767f0ebf946d557a0a126285369e96af159
2017-04-12 06:36:20 +10:00
Ian Wienand
79d4113cbe pip-and-virtualenv : install python2 & 3, and default to 2
Recent changes in project-config have shown that we leave the system
in an inconsistent state when installing from source.  On fedora, we
will have installed the python2 packages, but then used $DIB_PYTHON to
install python3 pip from source!

This tries to clarify the situation.  As described in the document,
with package installs, we just install the $DIB_PYTHON packaged
versions.

Source installs want to take over the global namespace.  This is the
price you pay for running the latest versions outside package managers
:) The only sane thing seems to be for us to normalise python2 &
python3 versions of pip, setuptools and virtualenv and then hacking
things such that "/usr/bin/pip" and "/usr/bin/virtalenv" remain
defaulted to python2 versions.

Documentation is added

Change-Id: Ibc6572b89e256d1f48b7fe7c672b8b9524dc704f
2017-04-11 18:59:11 +10:00
Ian Wienand
ffd4820d59 Install pip with python interpreter
Currently we install pip/virtualenv with "/usr/local/bin/dib-python".
This means that every time you create a virtualenv, the python
interpreter inside it is called "dib-python" which is confusing.

Add an env var DIB_PYTHON that points directly the to interpreter
available during build, for use when running scripts.

Change-Id: I88ad3c9eb958d58db4631d9b27bc2c592f970345
2017-04-11 18:59:09 +10:00
Jenkins
5c1579f0d6 Merge "Unify and fix doc of several Debian and Ubuntu elements" 2017-04-10 01:25:01 +00:00
Jenkins
a076bf66d6 Merge "Move do_extra_package_install to run in install phase" 2017-04-10 01:24:55 +00:00
Noam Angel
95503b4201 Move do_extra_package_install to run in install phase
This change move "do_extra_package_install" from pre-install to install
phase.

Extra packages are added by user request using the flag "-p", This
package should not be something the elements depend on.
The reason behind this patch is to move the extra package install to
a proper phase, Also more reasonable if base element run package update
to be before we install extra packages.

Change-Id: I68cc773aba9aa01743f0dda9f4e635e4cac2a282
2017-04-10 08:32:47 +10:00
Paul Belanger
1778fb57db Clean out apt index caches at end of image build
Apt gets confused if it talks to a mirror with an older index than the
index currently cached by apt. This can happen when image builds use a
newer index than the booted image. Avoid these problems entirely by
removing those index caches at the end of image building.

Change-Id: I245d516ee8a44831b2c29612b782bad555c48a3f
Co-Author: Clark Boylan <clark.boylan@gmail.com>
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2017-04-09 11:57:23 -04:00
Jenkins
2d178e43e1 Merge "Use DIB_PYTHON_EXEC to run commands" 2017-04-07 20:29:30 +00:00
Jenkins
347833856e Merge "Ignore missing path in unmount_dir" 2017-04-07 15:59:47 +00:00
Jenkins
3f5c558753 Merge "Fix package-installs-v2 output" 2017-04-07 13:24:17 +00:00
Jenkins
1b5045c563 Merge "Basic logging for package-installs-squash" 2017-04-07 13:24:10 +00:00
Jenkins
08ad06f86d Merge "Fix opensuse-minimal element on non-SUSE hosts" 2017-04-07 12:06:31 +00:00
Andreas Florath
ee8fa3c4bb Unify and fix doc of several Debian and Ubuntu elements
This patch removes three nearly-copies of debootstrap documentation
and fixes some documentation aspects.

Change-Id: Ief7794f5c1abad73788c063af6c862472cd34744
Signed-off-by: Andreas Florath <andreas@florath.net>
2017-04-07 05:56:50 +00:00
Ian Wienand
f068e6aa6e Fix package-installs-v2 output
The current output for package-installs-v2 is inscrutable [1]

The problem starts with process_output() which is not capturing
stderr.  This means that any stderr output is dislocated from any
stdout output around it.  This is *really* confusing as you get a
bunch of seemingly meaningless stderr output from any calls before you
see any stdout (e.g. in [1] you can see random yum error output that
should have been with the yum call)).  The simplest thing to do is to
redirect stderr to stdout which keeps everything in sync.

This causes a slight problem, however, because pkg-map outputs both
status information and errors on stderr.  To work around this but
maintain compatibility, we add a "--prefix" argument that prepends
mapped packages from pkg-map with a value we can match on.  The
existing status/debug output from pkg-map is low-value; modify the
call so that it will be traced only at higher debug levels (e.g. -x
-x).

The current loop is also calling pkg-map for every package in every
element (this is why in [1] the same message is repeated over and
over).  This is unnecessary; it only needs to pkg-map once for each
element, giving the package list as the arguments.  Create package
lists by element and pass those to pkg-map.

As a cleanup, there is no point in printing e.output if the
process_output fails for the install because we are already tracing
it; i.e. the output, even for failures, is already in the logs.
Printing it again just duplicates the output.

[2] is an extract showing what I feel is a much more understandable
log output for a fairly complex install.

[1] http://paste.openstack.org/show/595118/
[2] http://paste.openstack.org/show/595303/

Change-Id: Ia74602a5d2db032a476481caec0e45dab013d54f
2017-04-07 13:48:53 +10:00
Ian Wienand
facabe585c Basic logging for package-installs-squash
Some basic info logging and debugging for package-installs-squash.

Change-Id: If7c227cf0a53b71d334b5f5a6075120f2a7bf3a7
2017-04-07 13:48:52 +10:00
Adam Harwell
f06231ee08 Use DIB_PYTHON_EXEC to run commands
DIB_PYTHON_EXEC was added in I5fab0e192c3a2dad8f60e821c184479e24e33bcd
to export the python that disk-image-create is running under.  If dib
is installed with python3 then just calls "python" (python2) to run
sub-scripts, it fails to find itself.

This has been tested in devstack gates that use py3x

Change-Id: Ia1028972bfc0517b468b279aab9decdbcd7424ca
2017-04-07 07:40:58 +10:00
Corey O'Brien
0ea7e927de Fix typo in pip-and-virtualenv
Change-Id: I3058b45fff037106eba0267fd6629707a5ebb8b1
2017-04-06 10:19:40 -04:00
Corey O'Brien
aa90f7991a Ignore missing path in unmount_dir
If the path is missing, unmount_dir currently exits with an error which
unintentionally aborts cleanup efforts early. This change makes
unmount_dir idempotent by exiting successfully if a directory doesn't
exist.

Change-Id: I1491b4344e8569ecb2833f44baee445a89a39d61
2017-04-06 10:08:13 -04:00
Jenkins
10baebe1e4 Merge "Run dib-run-parts out of /tmp" 2017-04-05 10:18:55 +00:00
Jenkins
0279258ac6 Merge "Don't provide dib-run-parts" 2017-04-05 07:01:29 +00:00
Jenkins
221c44f24c Merge "Adding aarch64 support for CentOS7" 2017-04-05 05:10:05 +00:00
Ian Wienand
6802cf7100 Run dib-run-parts out of /tmp
The dib-run-parts element was copying our internal version of
dib-run-parts into /usr/local/bin to be used running scripts inside
the target chroot.  However, it never cleaned up after itself.  This
means all images were left with an unmanaged local install of
dib-run-parts.

This copies dib-run-parts into the hooks directory of the chroot and
runs it from there.  It is cleaned up automatically on the exit path.

The dib-run-parts element is no longer required and it has been
removed from all dependencies.  It is left with a deprecation notice
in the README.  For compatability we convert it to simply install
dib-utils.

Codesearch shows no users depending on this unintentional implicit
install.  Note os-refresh-config depends on dib-utils and thus will
have an explicitly installed version.

Partial-Bug: #1673144
Change-Id: Ia2e96c00a4246c04beb96c17f83b8aefb69219ca
2017-04-05 13:11:22 +10:00
Ian Wienand
fd424757a6 Don't provide dib-run-parts
It was an oversight during v2 development for dib to start providing
dib-run-parts.  The intention was for dib to use a vendored
dib-run-parts directly from $_LIB and have no dependencies on
dib-utils at all.  By exporting dib-run-parts, we created an
unintentional conflict with the dib-utils package which provides the
same script.

Tools that depend on dib-utils are unaffected by this
(os-refresh-config).

The only tool that installs diskimage-builder and then assumes
dib-run-parts is available in the path is instack.  I have proposed
Ibfe972208df40fa092b11b5419043524c903f1b4 to modify that to use our
internal version.

Change-Id: I149c345d38d761a49b3a6ccc4833482f09f1cd05
2017-04-05 13:11:20 +10:00
Ian Wienand
8e822768f9 Add flag to disable EPEL
Add DIB_EPEL_DISABLED flag that allows installation of the EPEL repo,
but to have it disabled by default.  This will help when you have
unavoidable EPEL dependencies, but want to make sure you only pull
specific things in with "--enablerepo" calls when installing those
packages.

Change-Id: Iedf6167a7cd69418255ebbee095aea04c50d73fd
2017-04-05 05:59:40 +10:00
Jenkins
6e7d7e77ad Merge "Use stevedore for plugin config of block device" 2017-04-04 08:13:45 +00:00
Hironori Shiina
eaa0569ac5 Fix code-block in README of rhel7 element
A code-block in README of rhel7 element is not rendered as expected.
This patch fixes it to be rendered correctly.

Change-Id: Ie8f4c05edd1dd93314290682e4b2734622894e15
2017-04-03 03:56:22 +00:00
Dirk Mueller
1bdeaadc95 Fix opensuse-minimal element on non-SUSE hosts
zypper on non-suse hosts is not parsing the pattern repodata because
those are marked as an inofficial extension to the repomd specification.
This is not a big issue as there is meanwhile in newer openSUSE
distributions a pattern *package* that depends on the same packages like
the pattern would do, so we can just replace it with that.

Change-Id: I0c8f713075bd7e5bf1d425f81933b4666654add7
Depends-On: I34e98f0f7693859ed05011b008334628adff612f
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2017-04-02 19:37:01 -04:00
Paul Belanger
87236af75c Have simple-init enable network.service
When a glean is running on centos with multiple NICs, it will try to
systemctl enable network.service multiple times for each interface.
Because of systemd magic, it is possible for the systemctl command to
fail in a race condition.

glean shouldn't be enabling network.service during boot in
pre-networking phases (Ib2b618dd975ca44e9c6b0a2c9027642ffc46b9b0).  I
have proposed I8319f1ed6498a9d447950c2b4b34bca59e7b97e4 to remove this
and document the behaviour.

This also bring across suse's version
(I20bffabd333ea290d8712ec2a467f2b2d5678f3a)

Change-Id: I89d9443cb61e287bd0d9da3f48315272218ee335
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2017-03-28 19:28:51 +11:00
Andreas Florath
57c9e0bb41 Use stevedore for plugin config of block device
This patch introduces stevedore plugin mechanism for use
with the block device layer.  This makes it possible that
other projects pass in their own block device plugins.

Change-Id: Id3ea56aaf75f5a20a4e1b6ac2a68adb12c56b574
Signed-off-by: Andreas Florath <andreas@florath.net>
2017-03-24 21:42:47 +00:00
Jenkins
f128d08cf0 Merge "Use correct Ubuntu distro url on non-x86 arches" 2017-03-24 04:42:59 +00:00
Jenkins
94e2973785 Merge "functests: skip qcow2 generically but add specific test" 2017-03-24 01:07:10 +00:00
Mikhail S Medvedev
4d4417c9a6 Use correct Ubuntu distro url on non-x86 arches
diskimage-builder usually provides defaults that work out of the box.
One default that does not work outside of x86 land is Ubuntu distro
mirror url. Considering there are only two valid default options, we can
automatically choose a better default.

This patch changes behavior only for architectures known to be using
http://ports.ubuntu.com/ubuntu-ports. All others still would use
http://archive.ubuntu.com/ubuntu as default. It provides some guarantee
that we do not introduce a regression.

Change-Id: If95a64bac0c88f30736da4bae7f1fdce126c0bf6
2017-03-23 09:27:25 -05:00
gecong1973
8ff6f51b81 Typo fix: curent => current
Change-Id: I5ce5a98c355310d6f5f6c09e19993b01ed025f0e
2017-03-23 15:02:20 +08:00
Ian Wienand
76ae374413 functests: skip qcow2 generically but add specific test
We somewhat discussed skipping qcow2 generation previously in
I9372e195913798a851c96e62eee89029e067baa1.  As recent issues with PPC
testing have shown, we are not actually testing the "vm" element and
hence the bootloader path in the functional tests.

I don't think we need to test this on every element; it overlaps
somewhat with the testing done by the nodepool jobs which build full
images and boot them.  I also didn't want to introduce a separate run
for this.  Thus it seems valuable to at least have one element
enhanced to do this installation and conversion in our default tests
for basic sanity.

This disables qcow generation by default, as per the other change, but
allows an element to drop a file that will override the output
formats.  The Xenial element is modified to produce a qcow2 using
this, and also introduces a dependency on the "vm" element so it tries
to install the bootloader.

We now exit if the .qcow2 fails to build as well.

Change-Id: I1a6acefe52f8c696c39b2d592fdc7ae32a87e6fe
2017-03-23 13:49:24 +11:00
Jenkins
2ea3a434f7 Merge "Add default PPC block-device layout" 2017-03-23 01:03:13 +00:00