Commit Graph

629 Commits

Author SHA1 Message Date
Zuul
d6f43865ed Merge "fail early when lates build information can not be fetched" 2019-05-31 08:45:50 +00:00
Zuul
21676bd350 Merge "Add option to skip update packages" 2019-05-31 08:13:38 +00:00
Zuul
49930da885 Merge "Deprecate rhel7 in favor of rhel" 2019-05-31 08:01:13 +00:00
Zuul
4367dd2dd3 Merge "Add version-less RHEL element for RHEL7 and RHEL8" 2019-05-31 07:56:50 +00:00
Dirk Mueller
421a0fa541 fail early when lates build information can not be fetched
When the mirror returns a error, it was trying to interpret the error
message (e.g. <html><title>Internal server error..) as a download link.
By using -f on curl we get an empty reply and an exit code, which, as
we run in set -e mode, aborts.

Change-Id: Ibaa39aedb7db286f859c4b090114c6a233b150c7
2019-05-31 16:09:25 +10:00
Zuul
ed6dfd87e5 Merge "allow the use of non-bzip compressed stages for building gentoo" 2019-05-31 04:42:07 +00:00
Nir Magnezi
433a374748 Deprecate rhel7 in favor of rhel
The rhel7 element is deprecated and is left only for backward
compatibility.
The rhel element should be used instead. Users should set DIB_RELEASE to
'7' to indicate which release you are using.

The new element is a version-less RHEL element to handle both '7'
and '8' DIB_RELEASE, which aligns with other elements which operate in
the same way such as the Fedora element.

Change-Id: Ic39ed85cacae9942448eb18ad685763f9369c2ed
2019-05-29 12:07:44 +00:00
Nir Magnezi
ee46e2f9b7 Add version-less RHEL element for RHEL7 and RHEL8
Make a version-less RHEL element to handle both '7' and '8' DIB_RELEASE.
The element usage should align with other elements which operate in the
same way such as the Fedora element.

Additionally, this patch adds support for RHEL8 that operates with
Python 3.
As of now, users of diskimage-builder will still be able to use the
'rhel7' element, or migrate to 'rhel' and specify their respective
DIB_RELEASE value.

* mount the xfs file-system for extraction as read-only.  vaguely
  based on explaination in [1] and the fact we only read the image
  data into a tar, so can ignore this.

    XFS (dm-1): Superblock has unknown read-only compatible features (0x4) enabled.

* Use the redhat system python as the dib-python version.  dib was
  ahead of it's time making an abstracted python interpreter for
  system work ;) the system python should work for running the various
  dib element scripts.

[1] https://unix.stackexchange.com/questions/247550/unmountable-xfs-filesystem

Redhat-Bug: https://bugzilla.redhat.com/show_bug.cgi?id=1700253
Co-Authored-By: Ian Wienand <iwienand@redhat.com>
Change-Id: I90540675c70bb475d9db2ae24f81c648a31f3f95
2019-05-29 11:28:53 +03:00
Matthew Thode
afcac1922c
allow the use of non-bzip compressed stages for building gentoo
Upstream is switching to xz so we need to be able to support it.

Change-Id: I382cc3e8038e2e552c553c526a990a01e51aeb12
2019-05-24 09:32:57 -05:00
Zuul
c47a9d5001 Merge "Replace git.openstack.org URLs with opendev.org URLs" 2019-05-24 08:03:47 +00:00
melissaml
a6322c6ed0 Replace git.openstack.org URLs with opendev.org URLs
Change-Id: I03e9162d5a59a2aa1631a9ecf6f6833bb7ac6050
2019-05-16 14:45:52 +08:00
Zuul
3d3ba26edd Merge "Use megabyte granularity for image extra space" 2019-05-15 06:51:10 +00:00
Logan V
87a18f51e3 Use megabyte granularity for image extra space
I want to use the new --image-extra-size flag[1] but my use-case
calls for megabyte granularity of this value. Rather than adding
60% to an 800MB image, maybe I only want to add 100 or 200MB, etc.

[1] https://review.opendev.org/#/c/655127/

Change-Id: I8fb9685d60ebb1260d5efcf03c5c23c561c24384
2019-05-15 13:38:25 +10:00
Dirk Mueller
c7ac6ee0cb Update test coverage for openSUSE/-minimal to 15.0
Use openSUSE 15.0 as default, which is the latest released stable
openSUSE release. Switch to https for accessing download.o.org
as encrypted transfers should be used by default.

Remove leftovers for definitely unmaintained openSUSE 13.x images
and split into old/new leap style versioning scheme for clarity.

Change-Id: Iab129eeee2b1a2563f0f0d2cb17bbad57c068e38
2019-05-08 14:59:51 +00:00
Zuul
4665e79245 Merge "openssh-server: harden sshd config" 2019-05-07 19:35:42 +00:00
Paul Belanger
5d60979e93 Use fedora-release-common for fedora 30+
It looks like fedora-release on fedora 30+ has been split into sub
packages. Use fedora-release-common to avoid package conflicts.

Change-Id: I8f8711044fc4074b91939e0a6dfdac4d7a14a35b
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2019-05-07 12:08:50 +00:00
Zuul
fa34eb7fe4 Merge "Support defining the free space in the image" 2019-05-07 10:14:01 +00:00
Zuul
8bf37a064e Merge "Allow specification of filesystem journal size" 2019-05-07 10:14:00 +00:00
Zuul
8c8b856c27 Merge "Only enable dbus-daemon for fedora-29 and below" 2019-05-07 05:47:57 +00:00
Paul Belanger
38d7574127 Only enable dbus-daemon for fedora-29 and below
In fedora-30 is when we migrate to dbus-broker, fedora-29 is still using
dbus-daemon.

Change-Id: I1e1d3a3826157b8b22386c211eaa58b6439b5f3c
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2019-05-06 17:49:42 +10:00
Paul Belanger
daf5a4e4bd Switch simple-init to support python3
Depending on the version of $DIB_PYTHON_VERSION, we can either use pip /
pip3 to install glean.  This is helpful for newer OSes that might not
want to ship python2 (pip).

Change-Id: I25c5927a1eb55ee16b919dd64403184f335839b6
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2019-05-02 19:38:16 -04:00
Tristan Cacqueray
11ec95b779 openssh-server: harden sshd config
Harden sshd configuration by adding KexAlgorithms, Ciphers and MACs for sshd,
following good pratices on https://infosec.mozilla.org/guidelines/openssh

Change-Id: I3051320d867a5033e82deef10c5e723ca9829884
Co-Authored-By: Nicolas Hicher <nhicher@redhat.com>
2019-05-01 11:42:21 -04:00
Tobias Henkel
778d007150 Support defining the free space in the image
Currently diskimage-builder supports two ways to specify the image
size. One is defining a fixed image size using DIB_IMAGE_SIZE, the
other one is auto-detection while adding a security margin of 60% as
free space. This means when building larger images (e.g. >100GB) with
unknown size upfront we end up with much wasted space, IO and network
traffic when uploading the images to several cloud providers. This can
be optimized by adding a third way by defining DIB_IMAGE_EXTRA_SIZE to
specify the free space in GB. This makes it possible to easily build
images of varying sizes while still minimizing the overhead by keeping
the free space constant to e.g. 1GB.

Change-Id: I114c739d11d0cfe3b8d8abc6df5ff989edfb67f2
2019-04-29 20:18:43 +10:00
caoyuan
0329a6de5e Replace git.openstack.org URLs with opendev.org URLs
Change-Id: Iac5a9da62db84365a769ea07146281866215a9c5
2019-04-29 20:15:25 +10:00
Logan V
11142f75b4 Allow specification of filesystem journal size
In many cases, the statically sized 64MB journal is far below the
e2fstools default calculation[0] which calls for a 64MB journal only
on filesystems smaller than 16GB. On bare metal in particular, the
correct default journal size will often be in the 512MB-1GB range.

Since we cannot know what the target system is, this should be a
tunable parameter that the user can set depending on the intended
image usage.

Add a DIB_JOURNAL_SIZE envvar and --mkfs-journal-size parameter
to the image creation so users can override the default journal
size.

[0] https://git.kernel.org/pub/scm/fs/ext2/e2fsprogs.git/tree/lib/ext2fs/mkjournal.c#n333

Change-Id: I65fa13a088eecdfe61636678578577ea2cfb3c0c
2019-04-29 17:00:30 +10:00
Zuul
6a4bf78e0c Merge "Fix Fedora aarch64 image location" 2019-04-18 12:22:06 +00:00
Zuul
5b47dc3a5b Merge "debian-minimal buster support" 2019-04-18 08:32:47 +00:00
Zuul
2205741de6 Merge "Also use selinuxenabled to check selinux status" 2019-04-18 08:32:15 +00:00
Pedro Alvarez
f034dd00d9 Fix Fedora aarch64 image location
It used to be considered a 'secondary architecture' but that
is not the case anymore.

Change-Id: I8e5e9cfa915c8a3c979ff9db26477c0542d271db
2019-04-09 15:50:55 +00:00
Ian Wienand
105d201e1f debian-minimal buster support
Due to the referenced bug, many versions of debootstrap can't bring up
a buster environment.  Unfortunately, these include versions we use to
do this on Xenial/Bionic nodes.

Also, there isn't backports or security updates, so elide these for
now.

I did get a working build (I haven't gone so far as a full boot+glean)
with this, at least.

Change-Id: If2420e92cb728ab6e91b0d70547da4483679b391
Paritial-Bug: #1822927
2019-04-04 16:10:08 +11:00
Serena Ziviani
19cc00041a Also use selinuxenabled to check selinux status
Currently, the cleanup script is using the existence of the folder
/sys/fs/selinux to check if SELinux is enabled. This, however, is
misleading in case disk-image-builder is used inside a Docker
container on a selinux-enabled host. In this case, the folder exists
in the container but SELinux is disabled.

This patch addresses the problem by checking, in addition to the
check already in place, the output of the command selinuxenabled.

Change-Id: I83e58f2467e60df9f0f00f7b7a58d0e2ce357a9a
Closes-Bug: #1820077
2019-03-28 14:20:24 +01:00
Zuul
36b4bc87f9 Merge "Minor clarifications in centos7 element docs" 2019-03-28 03:50:52 +00:00
Ian Wienand
20c5c98426 Replace openstack.org git:// URLs with https://
This is a mechanically generated change to replace openstack.org
git:// URLs with https:// equivalents.

This is in aid of a planned future move of the git hosting
infrastructure to a self-hosted instance of gitea (https://gitea.io),
which does not support the git wire protocol at this stage.

This update should result in no functional change.

For more information see the thread at

 http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003825.html

Change-Id: Id26bec14c3d94e2f81b2148fc85d17f07866398c
2019-03-22 01:35:42 +00:00
Daniel Abad
965d6f97aa Minor clarifications in centos7 element docs
Change-Id: I6aef77513efa37262269ca24b296acbdc823a039
2019-03-20 17:23:34 +01:00
Ian Wienand
5284564071 Unmount internal mounts on finalise errors
This is only one line, but it takes a lot to untangle ...  basically
the current "correct" path is:

---
 mk_build_dir()
  -> sets trap trap_cleanup EXIT

 ... stuff ..

 mount_proc_dev_sys
  -> mounts $TMP_MOUNT_PATH/<proc,dev.sysfs>

 pre-finalise.d
 finalise.d

 unmount_image $TMP_BUILD_DIR/mnt # nb == $TMP_MOUNT_PATH
  -> unmount_dir()
   -> recursive unmount everything inside TMP_MOUNT_PATH

 TMP_IMAGE_PATH=$(dib-block-device getval image-path)
 export TMP_IMAGE_PATH

 dib-block-device umount
 dib-block-device cleanup

 ... actually cleanup directories ...
---

Our current failure exit trap does:

---
 dib-block-device umount
 unmount_image
 ...
---

Note this is the *opposite* of what is done in the correct exit path.
In the failure case, if a script fails in the finalise stages it leads
to /proc, /sys, /dev etc. still being mounted inside the image; the
"dib-block-device umount" call doesn't know anything about these
mounts and tries to unmount the parent directory, and we get a hard
failure with a busy mount, and all the mounts are subsequently leaked.

Note that "unmount_dir", which is ultimately called by
"unmount_image", already knows to skip those mounts that
"dib-block-device umount" manages (this is the DIB_MOUNTPOINTS list).
This is further evidence it should be called *before* the
dib-block-device umount.

Change-Id: Ibef3ce9d1167b9c4ff3d5717b113cd3ed374f5e3
2019-03-13 16:38:49 +11:00
Zuul
bdfc13a5c0 Merge "[lvm] Add Ubuntu bionic as supported distro" 2019-03-11 09:19:49 +00:00
Zuul
186db05ffb Merge "Add DIB_APT_MINIMAL_CREATE_INTERFACES toggle" 2019-03-05 07:24:24 +00:00
Logan V
8756cbea1b Add DIB_APT_MINIMAL_CREATE_INTERFACES toggle
Add a DIB_APT_MINIMAL_CREATE_INTERFACES boolean to the debootstrap
element which functions identically to
DIB_YUM_MINIMAL_CREATE_INTERFACES in the yum-minimal element.

This can be used to disable the creation of the
/etc/network/interfaces.d/eth[01] dhcp configuration files, which
are not needed on systems where cloud-init or other means are used
to configure networking.

The flag is enabled by default to keep creating the dhcp interface
files, maintaining backwards compatibility.

Change-Id: I1fdaca8350a5ceefd9e437af4fd000ce6a3ee7f3
2019-03-05 16:27:57 +11:00
Gaëtan Trellu
bbde9bb320 [lvm] Add Ubuntu bionic as supported distro
The way how LVM is created on Ubuntu Xenial and Ubuntu Bionic
is the same.

Change-Id: I16d548f6393dd3cdfd5a9befa5c0ef0f6db92df1
2019-03-04 15:05:37 -05:00
Zuul
0f8d340c6c Merge "Update gentoo-releng gpg key" 2019-02-28 00:33:06 +00:00
Matthew Thode
b2cc91d276
Update gentoo-releng gpg key
A new signing subkey is used.

Change-Id: Idc9aceba7ee144fd0307737c24991acfacf68985
2019-02-27 15:18:25 -06:00
Zuul
703549412d Merge "update spelling errors" 2019-02-25 08:49:56 +00:00
Zuul
b2e2d121f0 Merge "set rhel minor release" 2019-02-25 08:11:56 +00:00
Noam Angel
8b83196024 Add option to skip update packages
in same cases it is required to avoid update all existing packages,
doing so can result in release update which is currently not possible
unless you not include "base" element.

"base" element used for most distribution (rhel, debain), and is
necessary for most cloud operations, this patch add 
"DIB_AVOID_PACKAGES_UPDATE" parameter to skip updating all packages.

usecases for this patch can be:
 * Avoid release update when building old release ex. RHEL7.5.
 * build on network-less environment.

usage:
DIB_AVOID_PACKAGES_UPDATE=1

or
DIB_AVOID_PACKAGES_UPDATE=0

Change-Id: I71192b23c8f0bc48b348fe7377bf8a2399b53792
2019-02-25 06:31:30 +00:00
Ian Wienand
37dff9738a Fix opensuse 42.3 pip-and-virtualenv
Related to I041a141366099093805e6052b1bbf64efd277e1e, we also need to
remove this on opensuse.  The files for gate testing are added, but
the test is not added to any jobs at this point in the interests of
gate time.

Change-Id: I1af9e84d76bedcb2607717edc6d2abe2920b0584
2019-02-25 15:37:17 +11:00
Quique Llorente
5b1844acf9 Keep git after ironic-agent post
New versions of pbr depends on git [1] and IPA depends on pbr [2] so
removing git will remove pbr, IPA and friends.

[1] https://src.fedoraproject.org/rpms/python-pbr/blob/master/f/python-pbr.spec#_64
[2] https://github.com/rdo-packages/ironic-python-agent-distgit/blob/rpm-master/openstack-ironic-python-agent.spec#L85

Closes-Bug: #1816017
Depends-On: https://review.openstack.org/637668
Change-Id: I97f6b593e88e1cb81cd4bb2d77787bc012fb8271
2019-02-19 16:32:22 +11:00
Noam Angel
802dc35a61 set rhel minor release
Change-Id: I52a38c16dbbbe9fa1d4d6b6daffde01f63f664e6
2019-02-10 14:59:41 +00:00
Zuul
25ba034a0e Merge "pip-and-virtualenv: handle centos image-based builds" 2019-02-07 07:54:23 +00:00
Ian Wienand
d0906ad473 pip-and-virtualenv: handle centos image-based builds
This fixes a regression in I041a141366099093805e6052b1bbf64efd277e1e
where we starting skipping the removal of old files for image-based
builds (confusingly named centos7 rather than centos for historical
reasons).  Fix the check

Change-Id: I74688a9e91d833b5d654056431729bed0585616c
2019-02-07 10:56:29 +11:00
Zuul
dd8bbd5c4f Merge "fix systemd import-tar for gentoo" 2019-02-04 02:15:09 +00:00