Commit graph

785 commits

Author SHA1 Message Date
Matthew Thode
9755c4f9a2
update gentoo systemd profile to 17.1 from 17.0
Upstream is now publishing 17.1 profile systemd stages

Also updates the docs that were forgotten in the last patch

Change-Id: I0f2e7976845b1d3c55ffe8869eec0bc04a191252
2019-08-19 15:13:09 -05:00
Ian Wienand
f23318d579 rpm-distro: ensure we selinux relabel underlying directories
As described inline, we need to ensure the underlying directories in
the image are correctly labeled, or we get all manner of services
failing during boot with selinux in enforcing mode.  Although the
problem is generic, this first shows up in Fedora 30 as systemd has
become more strict about namespace failures (I think) [1].

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1663040#c22

Change-Id: I52c1cc719884879169b606b00651aa26f5b783f1
2019-08-16 15:44:55 +10:00
Carlos Goncalves
9eb175e440 Allow extra repositories to be added to images
This patch adds option DIB_YUM_MINIMAL_EXTRA_REPOS to yum-minimal to
allow DIB users to include extra repositories to their final image.

Change-Id: I89549f4b0f4c9470143b5064817acab5043e31c5
2019-08-15 16:20:21 +02:00
Ian Wienand
efa3f3675a yum-minimal: install fedora-release-cloud
Something (possibly [1], but that change is at best cryptic) has
changed such that we don't get correct /etc/os-release files
installed.  This flows on to grub half-installing itself, enough to
not fail the build but not enough to make something bootable.

Installing the -cloud release package gets it back, and seems like a
sane choice for dib.

[1] 617b1bed34

Change-Id: Iff0413887fad798273b2bfcb140cc07f36d54a04
2019-08-15 15:56:13 +10:00
Ian Wienand
5492843aa8 block-device-efi : expand disk size calculation
As noted in the change, 7fd52ba841
increased the size of the EFI partition considerably.  This has meant
that our padding upwards of the disk size is insufficient and EFI
builds (arm64 in particular) is failing due to out-of-disk errors
during final image operations like installing kernels.

Similar to the discussion we had in
I65fa13a088eecdfe61636678578577ea2cfb3c0c, this feels a bit ugly
because we're mixing logic here with sizes specified in block-device
config files.  But it boils down to the same problem; we are
calculating the disk size here and passing it to the block-layer, so
unless we want to make large changes to the status quo about where
these sizes are calculated, small adjustments here are the most KISS
solution.

Thus we check if we have selected the EFI bootloader element, and thus
assume there will be a large system EFI partition and expand the disk
size accordingly.

Change-Id: Ifa05366c2f2b95259f3312e4dde8c85347075ba1
2019-08-14 15:49:38 +10:00
Ian Wienand
c596fb7dbd Don't show all elements found
This debug statement lists every element found and its dependencies on
every build; it's just noise unless you're debugging the element
dependency solver itself.  Remove from output.

Change-Id: I9281b953d958a3fd5e20edbc560a341a2fcc3deb
2019-08-14 13:07:16 +10:00
Zuul
0c6a4dbd92 Merge "Fixes packages for arm64 bootloader" 2019-08-13 09:21:37 +00:00
Zuul
24e620ae6d Merge "Fix the pypi element for multiple mirror URLs" 2019-08-13 07:04:20 +00:00
Zuul
1af5e6010d Merge "Create /etc/machine-id for RHEL images" 2019-08-13 07:00:31 +00:00
Ian Wienand
bac8b4246e dracut-regenerate: catch failures and exit code
This seems to miss the exit code of the dracut process, which actualy
caused some issues in I8511669e188717494daf2bc1384a6dd346f942a4 where
it would have been much clearer to stop after the initramfs generation
failed.

Add some debug messages, and catch any errors from the final call.

Change-Id: I6f89441ec4709f5199535e15a7cc53a3a8af273d
2019-08-13 15:51:05 +10:00
Zuul
4131942356 Merge "Fixes DIB_DISTRIBUTION_MIRROR_UBUNTU_IGNORE matching when empty" 2019-08-12 13:22:44 +00:00
liyingjun
bf0d7ede55 Fixes packages for arm64 bootloader
Should install "grub2-efi-aa64 grub2-efi-aa64-modules" instead of
"grub2-efi grub2-efi-modules" for arm64

Change-Id: Iee3191b0944b3b862890d166a9d36bd592fe8f7e
Closes-bug: #1839816
2019-08-12 17:18:36 +08:00
Manuel Torrinha
a38ac762f1 Fixes DIB_DISTRIBUTION_MIRROR_UBUNTU_IGNORE matching when empty
- DIB_DISTRIBUTION_MIRROR_UBUNTU_IGNORE matches when it is empty or not
 set and DIB_DISTRIBUTION_MIRROR is being used. Checking for it being 
 set and not empty solves this.
 - Normalizing bash conditionals for readability

Closes-Bug: #1808359
Change-Id: I87853fcda4c8b29a3f1720a2778debeb3acc3a53
Signed-off-by: Manuel Torrinha <manuel.torrinha@tecnico.ulisboa.pt>
2019-08-09 10:26:48 +00:00
Michael Johnson
bac0fa3eb2 Fix the pypi element for multiple mirror URLs
The 'pypi' mirror element is generating invalid pip.conf files
when more than one "DIB_PYPI_MIRROR_URL" is specified.
This patch fixes the pip.conf file rendering to use a proper form
when there are multiple "extra_index_url" provided.

Closes-Bug: #1839558
Change-Id: Ibda0e7390955683560e09b486f636775643ff57c
2019-08-08 22:52:43 +00:00
Andreas Jaeger
9a145cf0d0 Stop regex warning
python 3.6 warns about regexes like:
DeprecationWarning: invalid escape sequence \+

I noticed that debugging a trove job and it really led me in the wrong
way. Fix this with making it a raw string.

Change-Id: I58ee1a49d62316c6c3f0588832c97f659f7e460b
2019-08-08 15:31:52 +02:00
Ian Wienand
1f2a874e8e Create /etc/machine-id for RHEL images
Per the inline comment, a machine-id is required for kernels to
install correctly (this may well be a bug, but the linked issue
remained inconclusive).

Add a call to make the machine-id before install packages.

Change-Id: If75d04376e62bfdfe14ee3ca4d0bd5c8b383c1b0
Redhat-Bug: https://bugzilla.redhat.com/show_bug.cgi?id=1737355
2019-08-07 18:18:34 +10:00
Zuul
37909a0e81 Merge "Cleanup: remove useless statement" 2019-08-05 02:46:12 +00:00
Zuul
a0c6d16e1c Merge "journal-to-console: element to send systemd journal to console" 2019-08-05 02:00:15 +00:00
Zuul
f59ffcd9f3 Merge "fix comments / spelling errors in gentoo element" 2019-07-30 07:22:26 +00:00
Zuul
3d995645e6 Merge "update version of open-iscsi that is installed on musl" 2019-07-30 02:37:18 +00:00
Matthew Thode
559d2bcf3b
fix comments / spelling errors in gentoo element
Change-Id: I41d906a99470599da3cfac7aaa0350232ce79316
2019-07-29 08:54:16 -05:00
Matthew Thode
f1d7e902e3
support alternate portage directories
The 17.1 profile changed the defaults used in portage for where we store
our repo, distfiles and binpkgs.  Some portage related variables need to
be set deterministically.  17.1 is no enabled for Systemd's profile.

Change-Id: Ib55f6875c5cb461c3c530b51d7420ce3dc8da360
2019-07-26 19:30:01 -05:00
Ian Wienand
a5bd03ec6b journal-to-console: element to send systemd journal to console
This element configures systemd to send its journal to the console,
which can then be retreived by server commands.  In the case of
nodepool, if the image failed to boot the console will be dumped into
the logs when nodepool decides the node is not responding.  Having
this can be very helpful diagnosing early boot errors.

Needed-By: https://review.opendev.org/#/c/669787/
Change-Id: I6b6df7023acb6b2f967b84840bc4b542ebc03727
2019-07-25 11:24:49 +10:00
Matthew Thode
ed95b4eba1
update version of open-iscsi that is installed on musl
Newer versions of open-iscsi seem to compile on Gentoo / musl.  Use them
if we can.  This also removes the cap on open-iscsi.

Change-Id: I596cb61494e459a419bce6a63deff89f9e78fe23
2019-07-22 14:30:32 -05:00
Zuul
f35d29c356 Merge "ironic-agent: Use targetcli & python3-devel on rhel8" 2019-07-15 00:31:01 +00:00
Zuul
fc1709b0fb Merge "set default sources conf for buster as it now has a release" 2019-07-15 00:31:00 +00:00
Zuul
c7eb556098 Merge "disable autounmask for emerge" 2019-07-15 00:25:57 +00:00
Clark Boylan
abb6aed459 Only enable dbus-daemon on fedora-29
Previously we were trying to enable dbus-daemon service on all prior to
fedora 30. Unfortunately 28 and older don't have this service so this
broke those releases and only worked for 29. Fix this by only enabling
this service on fedora 29.

Change-Id: I1bd15dcf0bbe270afccb0c0c3ea6ad08862a53f1
2019-07-12 10:21:49 -07:00
Zuul
709c9e70c9 Merge "Set router solicitation delay with using NM" 2019-07-10 19:14:00 +00:00
Clark Boylan
5b5b78bf59 Set router solicitation delay with using NM
The linux kernel and NetworkManager fight each other over control for
interface management when router advertisements are in use. Long story
short if the linux kernel configures a network interface for ipv6
before NetworkManager attempts to manage that interface then NM will
ignore the interface and not configure ipv4 on it.

This can happen because the kernel is configured to send router
advertisements solicitations which result in router advertisements which
the kernel uses to configure the interface(s). There is a default of a 1
second delay before sending the solicitation which in many cases is long
enough that NM has started before then. However, in slower environments
like those used for testing with qemu this isn't long enough.

Some testing by hand indicates that 15 seconds is about right so
increase the delay to 15 seconds via sysctl.conf.

Note this may increase boot times in ipv6 only environments (though it
is hard to be sure due to how systemd starts everything at once and does
socket activation and the like).

Change-Id: I475a253091cbaf63687b91c748c31a6753bb0f57
2019-07-10 08:33:17 -07:00
Chandan Kumar (raukadah)
08caa8034d [RHEL-8] Set _clear_old_files=0 in install-pip element
When virtualenv and setuptools gots installed from source and rpm
then their installation path lives at different places but when
the python script got called then that time it choses either of
rpm or source based path on system wide installation and leads to
different failure as their methods are not implemeted.

So by setting _clear_old_files to 0 will install
python3-virtualenv python3-pip python3-setuptools from rpms only
and avoid these failures.

Change-Id: I0c162f1fe8168513e352546ab8dd2b68fa65b88c
Signed-off-by: Chandan Kumar (raukadah) <chkumar@redhat.com>
2019-07-08 19:50:25 +05:30
Matthew Thode
de94e07a86
disable autounmask for emerge
autounmask=y (default) changes portage depsolving, causing errors
(mostly often seen in perl and binpkg related issues).

Disabling this functionality for DIB builds is OK as the enviroment is
not passed on post build and the build process is not interactive
anyway.

Change-Id: Ife9ace246bec16864ee4982bc456763af5dff2e8
Signed-off-by: Matthew Thode <mthode@mthode.org>
2019-07-05 12:41:51 -05:00
Lon Hohberger
92f87d0d0a ironic-agent: Use targetcli & python3-devel on rhel8
Change-Id: I2e596f5c3dc23b21441a4eb61c773725ebd25d34
Signed-off-by: Lon Hohberger <lhh@redhat.com>
2019-07-02 21:39:13 -04:00
Matthew Thode
d8f796e153
install gnupg2 by default in debian-minimal
debian-minimal depends on debootstrap which depends on dpkg

This needs to be installed early as dpkg installs the apt keys early via
02-add-apt-keys in pre-install.d

Change-Id: I8580849ceaa7a5152c94f29afa890ac6d6983fb1
2019-07-01 14:28:01 -05:00
Matthew Thode
03933b3ab7
set default sources conf for buster as it now has a release
Change-Id: Ica014a1a267b55e2f29a0c333213e0a4c897108b
2019-07-01 13:59:14 -05:00
Maksim Malchuk
32dd3305d2 Cleanup: remove useless statement
This change removes useless statement accidentally added in the
06576a02f0

Change-Id: I7ea4a24d8c72c9e72f5f87247403af0f9bf69b40
Signed-off-by: Maksim Malchuk <maksim.malchuk@gmail.com>
2019-07-01 10:28:12 +03:00
Zuul
091a4e2c6e Merge "debootstrap: make default network interface names configurable" 2019-06-28 06:30:46 +00:00
Zuul
2c5fc5f8f9 Merge "Enable nodepool testing for opensuse 15.1" 2019-06-28 06:26:58 +00:00
Dirk Mueller
97444ad92e Enable nodepool testing for opensuse 15.1
There are several jobs depending on working opensuse 15.1
images in nodepool, so it makes sense to ensure its working.
Also upgrade the previously marked experimental opensuse-15.0
job as it tests the xenial->opensuse combination, which is
particularly difficult to keep working and we'll need it in
the CI.

Change-Id: Icb6d998756ce5221e017959dcb59b21f0f023454
2019-06-27 19:59:45 +02:00
Zuul
f27a0dfc26 Merge "Add DIB_UBUNTU_KERNEL to ubuntu-minimal" 2019-06-25 22:48:50 +00:00
Michael Johnson
e433aebf7d Add DIB_UBUNTU_KERNEL to ubuntu-minimal
This patch adds a new environment variable to the ubuntu-minimal
element called DIB_UBUNTU_KERNEL that allows you to specify the kernel
meta package that will be using to install the kernel inside the image.
It supports "linux-image-generic" (The default), "linux-image-kvm", and
"linux-image-virtual".
This allows building images that are smaller in size (~200MB smaller
qcow2) that have only the kernel modules necessary for virtual
machines.

Change-Id: I8ce65e1d357d15e8ed8995ad1dcaea02bbd1986f
2019-06-20 10:18:23 -07:00
jacky06
8dd7ca720f Sync Sphinx requirement
1. Sync sphinx dependency with global requirements. It caps python 2 since
sphinx 2.0 no longer supports Python 2.7.
2. Update some URLs to latest
3. Remove the unnecessary space

Change-Id: I5464be9e055feecd80918f691448acf5f100e701
2019-06-18 23:29:52 +08:00
Zuul
4afcba1fea Merge "Move pypi to dib-python" 2019-06-18 06:49:37 +00:00
Zuul
e281c4dae9 Merge "Use architecture-specific grub2 RPMs on RHEL8" 2019-06-18 06:49:35 +00:00
Zuul
c46f8714e3 Merge "Remove the rhel 8 check for xfs" 2019-06-18 06:49:34 +00:00
Zuul
fe618002ff Merge "Update test coverage for openSUSE/-minimal to 15.1" 2019-06-18 06:21:21 +00:00
Dirk Mueller
a81cf9e231 Update test coverage for openSUSE/-minimal to 15.1
Use openSUSE 15.1 as default, which is the latest released stable
openSUSE release.

Remove leftovers for unmaintained openSUSE 42.2 images.

Depends-On: https://review.opendev.org/#/c/660126/
Change-Id: I0b204b7b3d7ae74b6749320b3bfe1ca89d154ebb
2019-06-13 09:20:40 +02:00
Michael Johnson
5f4b764a11 Remove the rhel 8 check for xfs
This patch removes the check and default for rhel 8 requiring
xfs filesystem as rhel 8 images can successfully be built with
ext4 filesystems.

Change-Id: I1a6bfa26324fd43ae0c77c2c977dda0dd56e26e5
2019-06-12 07:01:36 -07:00
Andreas Florath
05af0fc863 debootstrap: make default network interface names configurable
Nowadays, in the time of Predictable Network Interface Names, the
network interface names 'ethX' are not used that often any more.
Depending on the virtualization layer and the guest OS names like
'ens3', 'enp1s0' or 'enp0s31f6' are used.
This patch enables the user to set DIB_NETWORK_INTERFACE_NAMES to a
list of network interfaces which are brought up using DHCP during
(first) boot.

Change-Id: I04cc2ee710f0389a8085b1c91d9329784cb28048
Signed-off-by: Andreas Florath <Andreas.Florath@telekom.de>
2019-06-12 13:53:38 +00:00
Maksim Malchuk
ea9ab89829 Move pypi to dib-python
The latest Fedora/Ubuntu images don't ship python2 by default, so we
need to use our dib-python wrapper for this so we work in python3 only
environments.

This change also correctly creates the pip.conf and .pydistutils.cfg
files with trusted host extracted from the index-url.

Related-bug: 1577105

Change-Id: Ibb5348af3e3bbe46b19affe90a8930a4b4ad4cad
Signed-off-by: Maksim Malchuk <maksim.malchuk@gmail.com>
2019-06-10 20:43:09 +03:00
Dmitry Tantsur
928c6e61f0 ironic-agent: install mdadm on the ramdisk
The newly introduced software RAID support requires it.

Change-Id: Ic438865006f1472abc0c9f4d40cc40c91b4ada71
2019-06-07 14:05:41 +02:00
Lon Hohberger
0cf0942068 Use architecture-specific grub2 RPMs on RHEL8
RHEL8 ships a bunch of grub2-efi-X-modules in its main
repository, each of which provides grub2-efi-modules,
potentially causing nondeterminism when building images.

This changes the DIB elements to always use architecture-
specific RPMs when RHEL8 is selected.

Change-Id: If94f3721195d5ecd80036e4234a3ca223a19c349
Related: https://bugzilla.redhat.com/show_bug.cgi?id=1716672
2019-06-06 10:50:51 -04:00
Zuul
7f469e3e83 Merge "Increase size of EFI system partition (again)" 2019-05-31 09:05:17 +00:00
Zuul
d323928af0 Merge "Makes image caching more resilient" 2019-05-31 08:54:31 +00:00
Zuul
d6f43865ed Merge "fail early when lates build information can not be fetched" 2019-05-31 08:45:50 +00:00
Zuul
21676bd350 Merge "Add option to skip update packages" 2019-05-31 08:13:38 +00:00
Zuul
49930da885 Merge "Deprecate rhel7 in favor of rhel" 2019-05-31 08:01:13 +00:00
Zuul
4367dd2dd3 Merge "Add version-less RHEL element for RHEL7 and RHEL8" 2019-05-31 07:56:50 +00:00
Pierre Riteau
7fd52ba841 Increase size of EFI system partition (again)
When I said in I8594d1fe05242f246a5809740a115ab2f84ac5a3 that 12 MiB
ought to be enough, I should have expected that I would be proven wrong.
While 12 MiB is enough to fit shim-x64 and grub2-efi-x64, yum fails to
update these packages to newer versions:

Transaction check error:
  installing package shim-x64-15-2.el7.centos.x86_64 needs 7MB on the /boot/efi filesystem
  installing package grub2-efi-x64-1:2.02-0.76.el7.centos.1.x86_64 needs 3MB on the /boot/efi filesystem

Error Summary
-------------
Disk Requirements:
  At least 7MB more space needed on the /boot/efi filesystem.

It is recommended that the ESP partition be much bigger. This commit
bumps its size to 550MiB, following guidelines from Rod Smith to avoid
incompatibilities with some EFIs [1].

[1] https://www.rodsbooks.com/efi-bootloaders/principles.html

Change-Id: If9515234f1a803cda32b2482f8abe10ddf0e6d26
2019-05-31 17:10:08 +10:00
Sorin Sbarnea
fb656718fb Makes image caching more resilient
Avoids failing on the first attempt to download the image to cache as
mirrors hosting them can randomly go down, usually with a connection
refused.

Change-Id: I9de9f33c2cc16596d04b35c4eb92621e6a2c7511
2019-05-31 16:31:43 +10:00
Dirk Mueller
421a0fa541 fail early when lates build information can not be fetched
When the mirror returns a error, it was trying to interpret the error
message (e.g. <html><title>Internal server error..) as a download link.
By using -f on curl we get an empty reply and an exit code, which, as
we run in set -e mode, aborts.

Change-Id: Ibaa39aedb7db286f859c4b090114c6a233b150c7
2019-05-31 16:09:25 +10:00
Zuul
ed6dfd87e5 Merge "allow the use of non-bzip compressed stages for building gentoo" 2019-05-31 04:42:07 +00:00
Nir Magnezi
433a374748 Deprecate rhel7 in favor of rhel
The rhel7 element is deprecated and is left only for backward
compatibility.
The rhel element should be used instead. Users should set DIB_RELEASE to
'7' to indicate which release you are using.

The new element is a version-less RHEL element to handle both '7'
and '8' DIB_RELEASE, which aligns with other elements which operate in
the same way such as the Fedora element.

Change-Id: Ic39ed85cacae9942448eb18ad685763f9369c2ed
2019-05-29 12:07:44 +00:00
Nir Magnezi
ee46e2f9b7 Add version-less RHEL element for RHEL7 and RHEL8
Make a version-less RHEL element to handle both '7' and '8' DIB_RELEASE.
The element usage should align with other elements which operate in the
same way such as the Fedora element.

Additionally, this patch adds support for RHEL8 that operates with
Python 3.
As of now, users of diskimage-builder will still be able to use the
'rhel7' element, or migrate to 'rhel' and specify their respective
DIB_RELEASE value.

* mount the xfs file-system for extraction as read-only.  vaguely
  based on explaination in [1] and the fact we only read the image
  data into a tar, so can ignore this.

    XFS (dm-1): Superblock has unknown read-only compatible features (0x4) enabled.

* Use the redhat system python as the dib-python version.  dib was
  ahead of it's time making an abstracted python interpreter for
  system work ;) the system python should work for running the various
  dib element scripts.

[1] https://unix.stackexchange.com/questions/247550/unmountable-xfs-filesystem

Redhat-Bug: https://bugzilla.redhat.com/show_bug.cgi?id=1700253
Co-Authored-By: Ian Wienand <iwienand@redhat.com>
Change-Id: I90540675c70bb475d9db2ae24f81c648a31f3f95
2019-05-29 11:28:53 +03:00
Matthew Thode
afcac1922c
allow the use of non-bzip compressed stages for building gentoo
Upstream is switching to xz so we need to be able to support it.

Change-Id: I382cc3e8038e2e552c553c526a990a01e51aeb12
2019-05-24 09:32:57 -05:00
Zuul
c47a9d5001 Merge "Replace git.openstack.org URLs with opendev.org URLs" 2019-05-24 08:03:47 +00:00
melissaml
a6322c6ed0 Replace git.openstack.org URLs with opendev.org URLs
Change-Id: I03e9162d5a59a2aa1631a9ecf6f6833bb7ac6050
2019-05-16 14:45:52 +08:00
Zuul
3d3ba26edd Merge "Use megabyte granularity for image extra space" 2019-05-15 06:51:10 +00:00
Logan V
87a18f51e3 Use megabyte granularity for image extra space
I want to use the new --image-extra-size flag[1] but my use-case
calls for megabyte granularity of this value. Rather than adding
60% to an 800MB image, maybe I only want to add 100 or 200MB, etc.

[1] https://review.opendev.org/#/c/655127/

Change-Id: I8fb9685d60ebb1260d5efcf03c5c23c561c24384
2019-05-15 13:38:25 +10:00
Dirk Mueller
c7ac6ee0cb Update test coverage for openSUSE/-minimal to 15.0
Use openSUSE 15.0 as default, which is the latest released stable
openSUSE release. Switch to https for accessing download.o.org
as encrypted transfers should be used by default.

Remove leftovers for definitely unmaintained openSUSE 13.x images
and split into old/new leap style versioning scheme for clarity.

Change-Id: Iab129eeee2b1a2563f0f0d2cb17bbad57c068e38
2019-05-08 14:59:51 +00:00
Zuul
4665e79245 Merge "openssh-server: harden sshd config" 2019-05-07 19:35:42 +00:00
Paul Belanger
5d60979e93 Use fedora-release-common for fedora 30+
It looks like fedora-release on fedora 30+ has been split into sub
packages. Use fedora-release-common to avoid package conflicts.

Change-Id: I8f8711044fc4074b91939e0a6dfdac4d7a14a35b
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2019-05-07 12:08:50 +00:00
Zuul
fa34eb7fe4 Merge "Support defining the free space in the image" 2019-05-07 10:14:01 +00:00
Zuul
8bf37a064e Merge "Allow specification of filesystem journal size" 2019-05-07 10:14:00 +00:00
Zuul
8c8b856c27 Merge "Only enable dbus-daemon for fedora-29 and below" 2019-05-07 05:47:57 +00:00
Paul Belanger
38d7574127 Only enable dbus-daemon for fedora-29 and below
In fedora-30 is when we migrate to dbus-broker, fedora-29 is still using
dbus-daemon.

Change-Id: I1e1d3a3826157b8b22386c211eaa58b6439b5f3c
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2019-05-06 17:49:42 +10:00
Paul Belanger
daf5a4e4bd Switch simple-init to support python3
Depending on the version of $DIB_PYTHON_VERSION, we can either use pip /
pip3 to install glean.  This is helpful for newer OSes that might not
want to ship python2 (pip).

Change-Id: I25c5927a1eb55ee16b919dd64403184f335839b6
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2019-05-02 19:38:16 -04:00
Tristan Cacqueray
11ec95b779 openssh-server: harden sshd config
Harden sshd configuration by adding KexAlgorithms, Ciphers and MACs for sshd,
following good pratices on https://infosec.mozilla.org/guidelines/openssh

Change-Id: I3051320d867a5033e82deef10c5e723ca9829884
Co-Authored-By: Nicolas Hicher <nhicher@redhat.com>
2019-05-01 11:42:21 -04:00
Tobias Henkel
778d007150 Support defining the free space in the image
Currently diskimage-builder supports two ways to specify the image
size. One is defining a fixed image size using DIB_IMAGE_SIZE, the
other one is auto-detection while adding a security margin of 60% as
free space. This means when building larger images (e.g. >100GB) with
unknown size upfront we end up with much wasted space, IO and network
traffic when uploading the images to several cloud providers. This can
be optimized by adding a third way by defining DIB_IMAGE_EXTRA_SIZE to
specify the free space in GB. This makes it possible to easily build
images of varying sizes while still minimizing the overhead by keeping
the free space constant to e.g. 1GB.

Change-Id: I114c739d11d0cfe3b8d8abc6df5ff989edfb67f2
2019-04-29 20:18:43 +10:00
caoyuan
0329a6de5e Replace git.openstack.org URLs with opendev.org URLs
Change-Id: Iac5a9da62db84365a769ea07146281866215a9c5
2019-04-29 20:15:25 +10:00
Logan V
11142f75b4 Allow specification of filesystem journal size
In many cases, the statically sized 64MB journal is far below the
e2fstools default calculation[0] which calls for a 64MB journal only
on filesystems smaller than 16GB. On bare metal in particular, the
correct default journal size will often be in the 512MB-1GB range.

Since we cannot know what the target system is, this should be a
tunable parameter that the user can set depending on the intended
image usage.

Add a DIB_JOURNAL_SIZE envvar and --mkfs-journal-size parameter
to the image creation so users can override the default journal
size.

[0] https://git.kernel.org/pub/scm/fs/ext2/e2fsprogs.git/tree/lib/ext2fs/mkjournal.c#n333

Change-Id: I65fa13a088eecdfe61636678578577ea2cfb3c0c
2019-04-29 17:00:30 +10:00
Zuul
6a4bf78e0c Merge "Fix Fedora aarch64 image location" 2019-04-18 12:22:06 +00:00
Zuul
5b47dc3a5b Merge "debian-minimal buster support" 2019-04-18 08:32:47 +00:00
Zuul
2205741de6 Merge "Also use selinuxenabled to check selinux status" 2019-04-18 08:32:15 +00:00
Pedro Alvarez
f034dd00d9 Fix Fedora aarch64 image location
It used to be considered a 'secondary architecture' but that
is not the case anymore.

Change-Id: I8e5e9cfa915c8a3c979ff9db26477c0542d271db
2019-04-09 15:50:55 +00:00
Ian Wienand
105d201e1f debian-minimal buster support
Due to the referenced bug, many versions of debootstrap can't bring up
a buster environment.  Unfortunately, these include versions we use to
do this on Xenial/Bionic nodes.

Also, there isn't backports or security updates, so elide these for
now.

I did get a working build (I haven't gone so far as a full boot+glean)
with this, at least.

Change-Id: If2420e92cb728ab6e91b0d70547da4483679b391
Paritial-Bug: #1822927
2019-04-04 16:10:08 +11:00
Serena Ziviani
19cc00041a Also use selinuxenabled to check selinux status
Currently, the cleanup script is using the existence of the folder
/sys/fs/selinux to check if SELinux is enabled. This, however, is
misleading in case disk-image-builder is used inside a Docker
container on a selinux-enabled host. In this case, the folder exists
in the container but SELinux is disabled.

This patch addresses the problem by checking, in addition to the
check already in place, the output of the command selinuxenabled.

Change-Id: I83e58f2467e60df9f0f00f7b7a58d0e2ce357a9a
Closes-Bug: #1820077
2019-03-28 14:20:24 +01:00
Zuul
36b4bc87f9 Merge "Minor clarifications in centos7 element docs" 2019-03-28 03:50:52 +00:00
Ian Wienand
20c5c98426 Replace openstack.org git:// URLs with https://
This is a mechanically generated change to replace openstack.org
git:// URLs with https:// equivalents.

This is in aid of a planned future move of the git hosting
infrastructure to a self-hosted instance of gitea (https://gitea.io),
which does not support the git wire protocol at this stage.

This update should result in no functional change.

For more information see the thread at

 http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003825.html

Change-Id: Id26bec14c3d94e2f81b2148fc85d17f07866398c
2019-03-22 01:35:42 +00:00
Daniel Abad
965d6f97aa Minor clarifications in centos7 element docs
Change-Id: I6aef77513efa37262269ca24b296acbdc823a039
2019-03-20 17:23:34 +01:00
Ian Wienand
5284564071 Unmount internal mounts on finalise errors
This is only one line, but it takes a lot to untangle ...  basically
the current "correct" path is:

---
 mk_build_dir()
  -> sets trap trap_cleanup EXIT

 ... stuff ..

 mount_proc_dev_sys
  -> mounts $TMP_MOUNT_PATH/<proc,dev.sysfs>

 pre-finalise.d
 finalise.d

 unmount_image $TMP_BUILD_DIR/mnt # nb == $TMP_MOUNT_PATH
  -> unmount_dir()
   -> recursive unmount everything inside TMP_MOUNT_PATH

 TMP_IMAGE_PATH=$(dib-block-device getval image-path)
 export TMP_IMAGE_PATH

 dib-block-device umount
 dib-block-device cleanup

 ... actually cleanup directories ...
---

Our current failure exit trap does:

---
 dib-block-device umount
 unmount_image
 ...
---

Note this is the *opposite* of what is done in the correct exit path.
In the failure case, if a script fails in the finalise stages it leads
to /proc, /sys, /dev etc. still being mounted inside the image; the
"dib-block-device umount" call doesn't know anything about these
mounts and tries to unmount the parent directory, and we get a hard
failure with a busy mount, and all the mounts are subsequently leaked.

Note that "unmount_dir", which is ultimately called by
"unmount_image", already knows to skip those mounts that
"dib-block-device umount" manages (this is the DIB_MOUNTPOINTS list).
This is further evidence it should be called *before* the
dib-block-device umount.

Change-Id: Ibef3ce9d1167b9c4ff3d5717b113cd3ed374f5e3
2019-03-13 16:38:49 +11:00
Zuul
bdfc13a5c0 Merge "[lvm] Add Ubuntu bionic as supported distro" 2019-03-11 09:19:49 +00:00
Zuul
186db05ffb Merge "Add DIB_APT_MINIMAL_CREATE_INTERFACES toggle" 2019-03-05 07:24:24 +00:00
Logan V
8756cbea1b Add DIB_APT_MINIMAL_CREATE_INTERFACES toggle
Add a DIB_APT_MINIMAL_CREATE_INTERFACES boolean to the debootstrap
element which functions identically to
DIB_YUM_MINIMAL_CREATE_INTERFACES in the yum-minimal element.

This can be used to disable the creation of the
/etc/network/interfaces.d/eth[01] dhcp configuration files, which
are not needed on systems where cloud-init or other means are used
to configure networking.

The flag is enabled by default to keep creating the dhcp interface
files, maintaining backwards compatibility.

Change-Id: I1fdaca8350a5ceefd9e437af4fd000ce6a3ee7f3
2019-03-05 16:27:57 +11:00
Gaëtan Trellu
bbde9bb320 [lvm] Add Ubuntu bionic as supported distro
The way how LVM is created on Ubuntu Xenial and Ubuntu Bionic
is the same.

Change-Id: I16d548f6393dd3cdfd5a9befa5c0ef0f6db92df1
2019-03-04 15:05:37 -05:00
Zuul
0f8d340c6c Merge "Update gentoo-releng gpg key" 2019-02-28 00:33:06 +00:00
Matthew Thode
b2cc91d276
Update gentoo-releng gpg key
A new signing subkey is used.

Change-Id: Idc9aceba7ee144fd0307737c24991acfacf68985
2019-02-27 15:18:25 -06:00
Zuul
703549412d Merge "update spelling errors" 2019-02-25 08:49:56 +00:00
Zuul
b2e2d121f0 Merge "set rhel minor release" 2019-02-25 08:11:56 +00:00
Noam Angel
8b83196024 Add option to skip update packages
in same cases it is required to avoid update all existing packages,
doing so can result in release update which is currently not possible
unless you not include "base" element.

"base" element used for most distribution (rhel, debain), and is
necessary for most cloud operations, this patch add 
"DIB_AVOID_PACKAGES_UPDATE" parameter to skip updating all packages.

usecases for this patch can be:
 * Avoid release update when building old release ex. RHEL7.5.
 * build on network-less environment.

usage:
DIB_AVOID_PACKAGES_UPDATE=1

or
DIB_AVOID_PACKAGES_UPDATE=0

Change-Id: I71192b23c8f0bc48b348fe7377bf8a2399b53792
2019-02-25 06:31:30 +00:00
Ian Wienand
37dff9738a Fix opensuse 42.3 pip-and-virtualenv
Related to I041a141366099093805e6052b1bbf64efd277e1e, we also need to
remove this on opensuse.  The files for gate testing are added, but
the test is not added to any jobs at this point in the interests of
gate time.

Change-Id: I1af9e84d76bedcb2607717edc6d2abe2920b0584
2019-02-25 15:37:17 +11:00
Quique Llorente
5b1844acf9 Keep git after ironic-agent post
New versions of pbr depends on git [1] and IPA depends on pbr [2] so
removing git will remove pbr, IPA and friends.

[1] https://src.fedoraproject.org/rpms/python-pbr/blob/master/f/python-pbr.spec#_64
[2] https://github.com/rdo-packages/ironic-python-agent-distgit/blob/rpm-master/openstack-ironic-python-agent.spec#L85

Closes-Bug: #1816017
Depends-On: https://review.openstack.org/637668
Change-Id: I97f6b593e88e1cb81cd4bb2d77787bc012fb8271
2019-02-19 16:32:22 +11:00
Noam Angel
802dc35a61 set rhel minor release
Change-Id: I52a38c16dbbbe9fa1d4d6b6daffde01f63f664e6
2019-02-10 14:59:41 +00:00
Zuul
25ba034a0e Merge "pip-and-virtualenv: handle centos image-based builds" 2019-02-07 07:54:23 +00:00
Ian Wienand
d0906ad473 pip-and-virtualenv: handle centos image-based builds
This fixes a regression in I041a141366099093805e6052b1bbf64efd277e1e
where we starting skipping the removal of old files for image-based
builds (confusingly named centos7 rather than centos for historical
reasons).  Fix the check

Change-Id: I74688a9e91d833b5d654056431729bed0585616c
2019-02-07 10:56:29 +11:00
Zuul
dd8bbd5c4f Merge "fix systemd import-tar for gentoo" 2019-02-04 02:15:09 +00:00
Zuul
46ac931513 Merge "pip-and-virtualenv : only remove system files on centos" 2019-02-02 01:38:48 +00:00
Ian Wienand
ea1735b6a2 pip-and-virtualenv : only remove system files on centos
As described inline, we only want to remove the system package files
on centos; it causes problems on Fedora where some system tools expect
these to be there.

But there is an additional bug -- pip actually removes the system
package files anyway.  To work around this, reinstall the system
package.

Closes-Bug: #1813232
Change-Id: I041a141366099093805e6052b1bbf64efd277e1e
2019-02-01 11:01:45 +11:00
Ian Wienand
7cb5916a76 Enable dbus-broker for Fedora 29
As described in the comments, it seems the transition between
dbus-daemon -> dbus-broker in Fedora 29 has made it so the packages
can get into a state where neither service is enabled.

Explicitly install and enable dbus-broker for F29

Change-Id: I06753043a75be2f635653899c6c251b9fbdd7c67
2019-01-31 18:08:37 +11:00
Matthew Thode
f4a1c7f89f
fix systemd import-tar for gentoo
use a newer version of systemd
Fixes https://review.openstack.org/#/c/608102

Change-Id: I23fd671adb893f3abd9fbc65382f2aec5a317c24
2019-01-27 18:16:51 -06:00
Zuul
379f1bdfc0 Merge "support cracklib in pam for Gentoo's musl profile" 2019-01-21 05:03:48 +00:00
Zuul
ad23eef69c Merge "Change phase to check for dracut-regenerate in iscsi-boot" 2019-01-18 09:57:14 +00:00
Matthew Thode
1bde2591ae
support cracklib in pam for Gentoo's musl profile
Needed for any musl build, is default in other profiles

Change-Id: Ib7cae9124f5846d33c05f26befd8f13646a08610
2019-01-15 09:47:46 -06:00
Michael Johnson
cfba9ea79d Make sure $TMP_BUILD_DIR/mnt is owned by root
The path $TMP_BUILD_DIR/mnt becomes the / inside the chroot during
the chroot phases of diskimage-builder. Previously this path was being
created using the account running diskimage-builder. This account may
not be valid inside the chroot. This causes path validation, when running
on a Ubuntu bionic host, to fail.
This patch chown's the $TMP_BUILD_DIR/mnt to root.root to make sure
that / is owned by a valid account inside the chroot.

Change-Id: Ifedc136baa67c7952942aed2c8cb1041902fef91
Closes-Bug: 1811113
2019-01-09 20:08:15 -08:00
Zuul
b0d41230e4 Merge "Delete the duplicate words in 50-zipl" 2019-01-10 00:17:53 +00:00
Zuul
8d3fa3a85c Merge "simple-init: allow for NetworkManager support" 2019-01-09 03:38:25 +00:00
Zuul
18c0c42c8d Merge "package-installs: provide for skip from env var" 2019-01-09 03:34:08 +00:00
Zuul
234f000e6b Merge "Update to Fedora 29" 2019-01-08 23:56:43 +00:00
Matthew Thode
cf786bb175
change to python36 for gentoo
fixes build issues since dev-util/glib-utils was updated

Change-Id: Ie8b5c425f846619ab4fc07f5bd1902dc83172a59
2019-01-08 04:59:08 -06:00
weiyj
64a8fc7c58 update spelling errors
Change-Id: Ic206b8247acce1975409329faa29deccd4f886de
2019-01-08 14:31:06 +08:00
Zuul
31f3e9dbc3 Merge "source-repositories: Replace documentation http with https links" 2019-01-08 05:16:18 +00:00
chengebj5238
079a104c1a source-repositories: Replace documentation http with https links
Change-Id: I7db3f2a1ed4e0460db60635ab00367050b0300a5
2019-01-08 15:49:41 +11:00
zhouxinyong
e3d6b8b0ec Delete the duplicate words in 50-zipl
Change-Id: Icaa3678f8b46f2a02eb9254753ed30ab8215aa7f
2019-01-07 10:02:35 +08:00
Yolanda Robla
853e13c6c3 Change phase to check for dracut-regenerate in iscsi-boot
There is an use of get_image_element_array on the environment.d
phase, for the iscsi-boot element.
This function is not available on that step. So moving the check
at next step, extra-data-d, where it is available.

Change-Id: I89cfe565492142c2f7962109360fcbcebadfd469
2019-01-02 11:05:00 +01:00
Zuul
56c72a0139 Merge "Add an element to configure iBFT network interfaces" 2018-11-30 13:46:05 +00:00
Zuul
58790e15f2 Merge "Add missing ws separator between words" 2018-11-30 03:53:49 +00:00
Ian Wienand
8ec3750dda simple-init: allow for NetworkManager support
This plumbs through an "--use-nm" flag to glean which instructs it to
setup interface bringup with NetworkManager rather than legacy network
enablement scripts.

In this case, install the NetworkManager package.  In the non-nm case,
also install the network-scripts for Fedora 29 -- this has stopped
being installed by default (it's been deprecated since forever).

As noted in the docs, this is currently really only relevant on the
supported rpm distros which are using the ifcfg-rh NetworkManager
plugin to effectively re-use old config files.  However,
NetworkManager has similar plugins for other platforms, so support can
be expanded if changes are proposed.

Depends-On: https://review.openstack.org/618964
Change-Id: I4d76e88ce25e5675fd5ef48924acd09915a62a4b
2018-11-30 10:02:47 +11:00
Ian Wienand
c52c383f1b package-installs: provide for skip from env var
Provide a "when" option that provides for not installing packages
based on a = or != match on an environment variable.

Unit tests are added.

Change-Id: Ifa824dccaff69fd447f45d54cb4a3083bcabdd86
2018-11-30 10:02:47 +11:00
Zuul
3eab481ab8 Merge "Fix a typo in the help message of disk-image-create" 2018-11-29 11:42:19 +00:00
Ian Wienand
0da1d3a419 Fix unit tests for elements
It looks like we dropped running these probably when we moved the
elements around.  For testtools to find the test scripts we need to
add the __init__.py files to make the directories look like modules.
Also prevent copying any .pyc or cache files in as hooks.

Change-Id: I66d5f6ee62cc4d9ee14c64e819b4db57d035d09f
2018-11-28 11:04:50 +11:00
Pierre Riteau
a64aa0cb47 Fix a typo in the help message of disk-image-create
Change-Id: I092e5ea88747b80c0e59c0aea4301d19009e0241
2018-11-23 09:09:31 +00:00
zhufl
b5f247b04f Add missing ws separator between words
This is to add missing ws separator between words.

Change-Id: Ie192d296128fb785c344ac5d8a77cad59764080e
2018-11-21 16:07:59 +08:00
Dmitry Tantsur
f0f3e3bac4 Add an element to configure iBFT network interfaces
This allows nodes with remote devices configured via iBFT to be
correctly used during Ironic introspection and deployment,
at least for non-multipath configurations.

The new element is added as a dependency for ironic-agent.

Change-Id: If3dac6504d26535593f12e851092065b688ef696
2018-11-20 14:11:11 +01:00
Zuul
d9d59b70da Merge "move selinux-permissive configure to pre-install phase" 2018-11-20 10:34:42 +00:00
Noam Angel
6f1f60983f move selinux-permissive configure to pre-install phase
install-packages is running before install.d phase, there is a chance
that installing a package like "container-selinux" will failed the
build, moving "selinux-permissive" to run at pre-install stage make
more sense.

Change-Id: I32f988be725d4b385c3765c47a00cd57c53d7d71
2018-11-19 13:13:57 +11:00
Zuul
d591c53196 Merge "fix some errors for ill-syntax in README.rst" 2018-11-19 01:59:49 +00:00
Ian Wienand
5085f23fdf Update to Fedora 29
Update builds to Fedora 29.  Remove the openstack gate CI mirror
workaround for pre-28 versions as they're not building in the gate any
more.

Change-Id: Ia6a8ae8d66d69f6add39e571043328e7274ba26c
2018-11-16 09:05:08 +00:00
Zuul
04806ea971 Merge "Increase size of EFI system partition" 2018-11-15 10:14:26 +00:00
zhouxinyong
621cbfca26 fix some errors for ill-syntax in README.rst
Change-Id: I0daf36848e8ddb09fcae6cbd55f460a8e49d209c
2018-11-13 15:05:51 +08:00
Pierre Riteau
257f9f4d1d Increase size of EFI system partition
8 MiB is not enough when using the grub2 element with centos7 images,
which installs binaries from the shim-x64 and grub2-efi-x64 packages
under /boot/efi. 12 MiB ought to be enough for anybody.

Change-Id: I8594d1fe05242f246a5809740a115ab2f84ac5a3
2018-10-31 14:38:28 +00:00
Jesse Pretorius
d59a0c8786 Add ubuntu-systemd-container operating-system element
In order to allow the simple preparation of base images which
can be used for LXC/nspawn machine containers, we add this
element.

Containers inherit a kernel from the host, so there is no need
to build a kernel into the image. All the element needs is a
base init system which, in this case, is systemd.

Change-Id: I45651de2aa1b19bdeee301094f0bdffdd0a3b45c
2018-10-31 14:22:28 +11:00
Zuul
16d5c4280b Merge "Turn on quiet mode when logfile specified" 2018-10-31 00:15:27 +00:00
Zuul
f8c12712cc Merge "Native zuulv3 tests" 2018-10-29 05:45:18 +00:00
Ian Wienand
36d642a6a0 Native zuulv3 tests
This finalises the ports of the legacy jobs to zuul native jobs.

The dib-setup-gate-mirrors role preconfigures the repo templates,
etc. for the openstack-ci-mirrors element.

The dib-functests role runs the tests as specified by dib_functests,
and can run under python2 or 3.

Change-Id: Ied67a31f0d31503d13eccad8662c29740c93f33e
2018-10-29 12:46:15 +11:00
Zuul
1785bd3975 Merge "Fix epel repo rewrite, add to testing" 2018-10-28 23:41:21 +00:00
Ian Wienand
86d5534352 Turn on quiet mode when logfile specified
I'm not really sure why I originally had --logfile also log to stdout
in I202e1cb200bde17f6d7770cf1e2710bbf4cca64c, but it seem
counter-intuitive (indeed, I just tripped myself up thinking that in a
devstack job "--logfile" would put the logs into a separate file and
avoid the stdout logging, and I wrote it!).

Make it so specifying a --logfile puts dib into quiet mode for stdout.
Explicitly overriding DIB_QUIET will allow both if someone wants that.

Change-Id: I3279c9253eee1c9db69c958b87a0ce73efc0be9b
2018-10-24 12:40:09 +11:00
Ian Wienand
dfd1e4fdaf Fix epel repo rewrite, add to testing
It looks like epel-release switch from "mirrorlist" to "metalink"
(around release 7-10 Jun 2017 according to [1]).  Update our rewrite
matching to handle this "metalink" as well.

Add epel element to the centos7 (image-based) build for testing too

[1] https://koji.fedoraproject.org/koji/buildinfo?buildID=978473

Add epel element so it's tested during the centos7 functional test.

Change-Id: I2d6d4c2ec47bc69d2f16c96b5045b05c435a1af9
2018-10-24 12:39:59 +11:00
Zuul
6d3d830cab Merge "Add a post-root.d phase" 2018-10-23 09:16:35 +00:00
Zuul
668978b0c2 Merge "Remove redundant sources change/update" 2018-10-23 09:16:35 +00:00
Tobias Henkel
eff5b2312b
Add a post-root.d phase
While trying to get docker image pre-caching to work we couldn't get a
docker daeomon to run within the chrooted environment. However we got
docker running with the help of bwrap outside of the chrooted
environment. The only option so far for this is the block-device.d
phase. But this has the problem that it runs after the image size has
been calculated. This leads to broken builds if the docker images
being pulled are big.

This can be solved by adding a post-root.d phase that runs outside the
chroot but before the image size calculation.

Change-Id: I36c2a81e2d9f5069f18ce5b0d52c5f1c7212c3ae
2018-10-19 10:33:56 +02:00