Commit Graph

83 Commits

Author SHA1 Message Date
Michael Johnson
e433aebf7d Add DIB_UBUNTU_KERNEL to ubuntu-minimal
This patch adds a new environment variable to the ubuntu-minimal
element called DIB_UBUNTU_KERNEL that allows you to specify the kernel
meta package that will be using to install the kernel inside the image.
It supports "linux-image-generic" (The default), "linux-image-kvm", and
"linux-image-virtual".
This allows building images that are smaller in size (~200MB smaller
qcow2) that have only the kernel modules necessary for virtual
machines.

Change-Id: I8ce65e1d357d15e8ed8995ad1dcaea02bbd1986f
2019-06-20 10:18:23 -07:00
Zuul
fe618002ff Merge "Update test coverage for openSUSE/-minimal to 15.1" 2019-06-18 06:21:21 +00:00
Dirk Mueller
a81cf9e231 Update test coverage for openSUSE/-minimal to 15.1
Use openSUSE 15.1 as default, which is the latest released stable
openSUSE release.

Remove leftovers for unmaintained openSUSE 42.2 images.

Depends-On: https://review.opendev.org/#/c/660126/
Change-Id: I0b204b7b3d7ae74b6749320b3bfe1ca89d154ebb
2019-06-13 09:20:40 +02:00
Zuul
7f469e3e83 Merge "Increase size of EFI system partition (again)" 2019-05-31 09:05:17 +00:00
Pierre Riteau
7fd52ba841 Increase size of EFI system partition (again)
When I said in I8594d1fe05242f246a5809740a115ab2f84ac5a3 that 12 MiB
ought to be enough, I should have expected that I would be proven wrong.
While 12 MiB is enough to fit shim-x64 and grub2-efi-x64, yum fails to
update these packages to newer versions:

Transaction check error:
  installing package shim-x64-15-2.el7.centos.x86_64 needs 7MB on the /boot/efi filesystem
  installing package grub2-efi-x64-1:2.02-0.76.el7.centos.1.x86_64 needs 3MB on the /boot/efi filesystem

Error Summary
-------------
Disk Requirements:
  At least 7MB more space needed on the /boot/efi filesystem.

It is recommended that the ESP partition be much bigger. This commit
bumps its size to 550MiB, following guidelines from Rod Smith to avoid
incompatibilities with some EFIs [1].

[1] https://www.rodsbooks.com/efi-bootloaders/principles.html

Change-Id: If9515234f1a803cda32b2482f8abe10ddf0e6d26
2019-05-31 17:10:08 +10:00
Nir Magnezi
433a374748 Deprecate rhel7 in favor of rhel
The rhel7 element is deprecated and is left only for backward
compatibility.
The rhel element should be used instead. Users should set DIB_RELEASE to
'7' to indicate which release you are using.

The new element is a version-less RHEL element to handle both '7'
and '8' DIB_RELEASE, which aligns with other elements which operate in
the same way such as the Fedora element.

Change-Id: Ic39ed85cacae9942448eb18ad685763f9369c2ed
2019-05-29 12:07:44 +00:00
Nir Magnezi
ee46e2f9b7 Add version-less RHEL element for RHEL7 and RHEL8
Make a version-less RHEL element to handle both '7' and '8' DIB_RELEASE.
The element usage should align with other elements which operate in the
same way such as the Fedora element.

Additionally, this patch adds support for RHEL8 that operates with
Python 3.
As of now, users of diskimage-builder will still be able to use the
'rhel7' element, or migrate to 'rhel' and specify their respective
DIB_RELEASE value.

* mount the xfs file-system for extraction as read-only.  vaguely
  based on explaination in [1] and the fact we only read the image
  data into a tar, so can ignore this.

    XFS (dm-1): Superblock has unknown read-only compatible features (0x4) enabled.

* Use the redhat system python as the dib-python version.  dib was
  ahead of it's time making an abstracted python interpreter for
  system work ;) the system python should work for running the various
  dib element scripts.

[1] https://unix.stackexchange.com/questions/247550/unmountable-xfs-filesystem

Redhat-Bug: https://bugzilla.redhat.com/show_bug.cgi?id=1700253
Co-Authored-By: Ian Wienand <iwienand@redhat.com>
Change-Id: I90540675c70bb475d9db2ae24f81c648a31f3f95
2019-05-29 11:28:53 +03:00
Zuul
3d3ba26edd Merge "Use megabyte granularity for image extra space" 2019-05-15 06:51:10 +00:00
Logan V
87a18f51e3 Use megabyte granularity for image extra space
I want to use the new --image-extra-size flag[1] but my use-case
calls for megabyte granularity of this value. Rather than adding
60% to an 800MB image, maybe I only want to add 100 or 200MB, etc.

[1] https://review.opendev.org/#/c/655127/

Change-Id: I8fb9685d60ebb1260d5efcf03c5c23c561c24384
2019-05-15 13:38:25 +10:00
Dirk Mueller
c7ac6ee0cb Update test coverage for openSUSE/-minimal to 15.0
Use openSUSE 15.0 as default, which is the latest released stable
openSUSE release. Switch to https for accessing download.o.org
as encrypted transfers should be used by default.

Remove leftovers for definitely unmaintained openSUSE 13.x images
and split into old/new leap style versioning scheme for clarity.

Change-Id: Iab129eeee2b1a2563f0f0d2cb17bbad57c068e38
2019-05-08 14:59:51 +00:00
Zuul
4665e79245 Merge "openssh-server: harden sshd config" 2019-05-07 19:35:42 +00:00
Tristan Cacqueray
11ec95b779 openssh-server: harden sshd config
Harden sshd configuration by adding KexAlgorithms, Ciphers and MACs for sshd,
following good pratices on https://infosec.mozilla.org/guidelines/openssh

Change-Id: I3051320d867a5033e82deef10c5e723ca9829884
Co-Authored-By: Nicolas Hicher <nhicher@redhat.com>
2019-05-01 11:42:21 -04:00
Tobias Henkel
778d007150 Support defining the free space in the image
Currently diskimage-builder supports two ways to specify the image
size. One is defining a fixed image size using DIB_IMAGE_SIZE, the
other one is auto-detection while adding a security margin of 60% as
free space. This means when building larger images (e.g. >100GB) with
unknown size upfront we end up with much wasted space, IO and network
traffic when uploading the images to several cloud providers. This can
be optimized by adding a third way by defining DIB_IMAGE_EXTRA_SIZE to
specify the free space in GB. This makes it possible to easily build
images of varying sizes while still minimizing the overhead by keeping
the free space constant to e.g. 1GB.

Change-Id: I114c739d11d0cfe3b8d8abc6df5ff989edfb67f2
2019-04-29 20:18:43 +10:00
Logan V
11142f75b4 Allow specification of filesystem journal size
In many cases, the statically sized 64MB journal is far below the
e2fstools default calculation[0] which calls for a 64MB journal only
on filesystems smaller than 16GB. On bare metal in particular, the
correct default journal size will often be in the 512MB-1GB range.

Since we cannot know what the target system is, this should be a
tunable parameter that the user can set depending on the intended
image usage.

Add a DIB_JOURNAL_SIZE envvar and --mkfs-journal-size parameter
to the image creation so users can override the default journal
size.

[0] https://git.kernel.org/pub/scm/fs/ext2/e2fsprogs.git/tree/lib/ext2fs/mkjournal.c#n333

Change-Id: I65fa13a088eecdfe61636678578577ea2cfb3c0c
2019-04-29 17:00:30 +10:00
Zuul
8d3fa3a85c Merge "simple-init: allow for NetworkManager support" 2019-01-09 03:38:25 +00:00
Zuul
18c0c42c8d Merge "package-installs: provide for skip from env var" 2019-01-09 03:34:08 +00:00
Zuul
56c72a0139 Merge "Add an element to configure iBFT network interfaces" 2018-11-30 13:46:05 +00:00
Ian Wienand
8ec3750dda simple-init: allow for NetworkManager support
This plumbs through an "--use-nm" flag to glean which instructs it to
setup interface bringup with NetworkManager rather than legacy network
enablement scripts.

In this case, install the NetworkManager package.  In the non-nm case,
also install the network-scripts for Fedora 29 -- this has stopped
being installed by default (it's been deprecated since forever).

As noted in the docs, this is currently really only relevant on the
supported rpm distros which are using the ifcfg-rh NetworkManager
plugin to effectively re-use old config files.  However,
NetworkManager has similar plugins for other platforms, so support can
be expanded if changes are proposed.

Depends-On: https://review.openstack.org/618964
Change-Id: I4d76e88ce25e5675fd5ef48924acd09915a62a4b
2018-11-30 10:02:47 +11:00
Ian Wienand
c52c383f1b package-installs: provide for skip from env var
Provide a "when" option that provides for not installing packages
based on a = or != match on an environment variable.

Unit tests are added.

Change-Id: Ifa824dccaff69fd447f45d54cb4a3083bcabdd86
2018-11-30 10:02:47 +11:00
Dmitry Tantsur
f0f3e3bac4 Add an element to configure iBFT network interfaces
This allows nodes with remote devices configured via iBFT to be
correctly used during Ironic introspection and deployment,
at least for non-multipath configurations.

The new element is added as a dependency for ironic-agent.

Change-Id: If3dac6504d26535593f12e851092065b688ef696
2018-11-20 14:11:11 +01:00
Zuul
d9d59b70da Merge "move selinux-permissive configure to pre-install phase" 2018-11-20 10:34:42 +00:00
Noam Angel
6f1f60983f move selinux-permissive configure to pre-install phase
install-packages is running before install.d phase, there is a chance
that installing a package like "container-selinux" will failed the
build, moving "selinux-permissive" to run at pre-install stage make
more sense.

Change-Id: I32f988be725d4b385c3765c47a00cd57c53d7d71
2018-11-19 13:13:57 +11:00
zhouxinyong
23d7668db1 delete the duplicate words in package-outside-debootstrap-ac93e9ce991819f1.yaml
Change-Id: I807fefffaa3aa8322cf40ef6e4494a1fd5be73c1
2018-11-13 15:01:42 +08:00
Zuul
16d5c4280b Merge "Turn on quiet mode when logfile specified" 2018-10-31 00:15:27 +00:00
Ian Wienand
86d5534352 Turn on quiet mode when logfile specified
I'm not really sure why I originally had --logfile also log to stdout
in I202e1cb200bde17f6d7770cf1e2710bbf4cca64c, but it seem
counter-intuitive (indeed, I just tripped myself up thinking that in a
devstack job "--logfile" would put the logs into a separate file and
avoid the stdout logging, and I wrote it!).

Make it so specifying a --logfile puts dib into quiet mode for stdout.
Explicitly overriding DIB_QUIET will allow both if someone wants that.

Change-Id: I3279c9253eee1c9db69c958b87a0ce73efc0be9b
2018-10-24 12:40:09 +11:00
Tobias Henkel
eff5b2312b
Add a post-root.d phase
While trying to get docker image pre-caching to work we couldn't get a
docker daeomon to run within the chrooted environment. However we got
docker running with the help of bwrap outside of the chrooted
environment. The only option so far for this is the block-device.d
phase. But this has the problem that it runs after the image size has
been calculated. This leads to broken builds if the docker images
being pulled are big.

This can be solved by adding a post-root.d phase that runs outside the
chroot but before the image size calculation.

Change-Id: I36c2a81e2d9f5069f18ce5b0d52c5f1c7212c3ae
2018-10-19 10:33:56 +02:00
Ian Wienand
fadf99af05 Add a pre-finalise.d phase
In exploring Gentoo caching, it was realised that we have no way to
bind mount the cache into the finalised image for the finalise.d
phases.

By adding a pre-finalise.d phase that runs outside the chroot, we can
mount outside things into the hierarchy at $TMP_BUILD_DIR/mnt which
are then seen by the in-chroot finalise.d phase.

This is similar to the pre-install phase

Change-Id: I9d782994843383ddf90f62c40498af9925fd9558
2018-10-15 12:45:23 +11:00
Ian Wienand
f6a2452d4c Only append DIB_BOOTLOADER_DEFAULT_CMDLINE to default grub entry
The grub.cfg has two variables [1]

 GRUB_CMDLINE_LINUX : used on all boots
 GRUB_CMDLINE_LINUX_DEFAULT : additionally used on all "normal" boots

The problem with I2298675dda1f699c572b3423e7274bc8bd7c1c9d is that it
appened the values in DIB_BOOTLOADER_DEFAULT_CMDLINE to both of these,
resulting in duplicated arguments.  I don't think we considered that
GRUB_CMDLINE_LINUX_DEFAULT actually already appends to the
GRUB_CMDLINE_LINUX values.

Make DIB_BOOTLOADER_DEFAULT_CMDLINE only append itself to
GRUB_CMDLINE_LINUX_DEFAULT.  That seems to line up sensibly with the
name of the variable.

Documentation is enhanced around this, and a releasenote added.

[1] https://help.ubuntu.com/community/Grub2/Setup

Change-Id: I76b5442a9090c19a6540ed2d4ab324546f241ebf
Closes: #1791736
2018-09-13 09:51:50 +10:00
Yolanda Robla
31383970c7 Add new modprobe element
This element will replace modprobe-blacklist element. It wil
still have the blacklist functionality, but it also adds
the feature of passing a complete file with settings to the
modprobe.d directory. Adding this functionality, that will
allow elements that depends on this module, to just copy the
specified files to the final directory.

Change-Id: I9a44f7d11520b8b1e604956d3c1db2fc7e2bf457
2018-06-28 13:55:53 +02:00
Roman Gorshunov
fde82c1f19 Added release notes for Change 568697
Change-Id: Ief646104eb473f25ce845e6168fac9a3209d033b
2018-05-29 10:13:37 -07:00
Ian Wienand
55b479b54f GPT partitioning support
This adds support for a GPT label type to the partitioning code.  This
is relatively straight-forward translation of the partition config
into a sgparted command-line and subsequent call.

A unit test is added based on a working GPT/EFI configuration and the
fedora-minimal functional test is updated to build a single-partition
GPT based using the new block-device-gpt override element.  See notes
in the sample configuration files about partition requirements and
types.

Documentation has been updated.

Co-Authored-By: Marcin Juszkiewicz <marcin.juszkiewicz@linaro.org>
Change-Id: I6b819a8071389e7e4eb4874ff7750bd192695ff2
2018-02-23 10:04:26 +11:00
Zuul
cb599b13de Merge "upgrade pip before using -c option" 2018-02-05 22:34:00 +00:00
Mark Hamzy
c7da8bc90a Set default label for XFS disks
As described, we want to set the default label for XFS disks to the
shorter value.

For example, you hit this when setting the old FS_TYPE environment
variable to 'xfs' (which sets the "root-fs-type" parameter, which gets
passed through to 'type'; but does not set a default label).

Change-Id: I41dce6e25766562db4366021309b8c2b74a8ab80
Closes-Bug: 1742170
2018-01-29 15:10:08 +11:00
Mark Hamzy
34ff72f253 upgrade pip before using -c option
The installed pip can be an older version which does not support
the -c argument. Therefore, upgrade pip before using -c.

Change-Id: If18d8ea822a62c8551c9c4d47354d58b0299fed2
Closes-Bug: 1744403
2018-01-19 16:46:59 -06:00
Zuul
247c68b5a3 Merge "Clear /etc/machine-id to avoid duplicate machine-ids" 2017-11-30 01:56:26 +00:00
Jenkins
7a76270227 Merge "Add timestamp output filter" 2017-10-09 15:56:31 +00:00
Yolanda Robla
c2dc3dc78e LVM support for dib-block-device
This provides a basic LVM support to dib-block-device.

Co-Authored-By: Ian Wienand <iwienand@redhat.com>

Change-Id: Ibd624d9f95ee68b20a15891f639ddd5b3188cdf9
2017-08-24 16:22:56 +10:00
Dave Hill
6c2b1465cc Clear /etc/machine-id to avoid duplicate machine-ids
Deploying many nodes with the generated image shouldn't have the same
/etc/machine-id so clearing it and letting systemd generate a new
id upon first boot seems to be the best way to achieve this.

Change-Id: I73d0577d31464521b3989312fd9d982a1312a268
Closes-bug: 1707526
Closes-bug: 1672461
2017-08-06 13:56:58 -04:00
Paul Belanger
7cbbee7ea3 Bump fedora/fedora-minimal DIB_RELEASE 26
Fedora 26 is now the latest release:

  https://fedoraproject.org/wiki/Releases/26/Schedule

We are building and using these in infra now

Change-Id: I012c2d28255be274e88abc2751d968bafaf76fbb
Depends-On: Ieba5f69020a13681074f72cfca2955071801b63a
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2017-08-04 14:18:05 +10:00
Dirk Mueller
1c4c4fd734 Switch openSUSE to 42.3 by default
This is the latest stable release, so we should default to it.

Change-Id: I05643787002d339ccbf7a718847fe4ed6f39eacc
2017-07-26 08:56:02 +02:00
Ian Wienand
3457d2f8e8 Remove DIB_[DISTRO]_DISTRIBUTION_MIRROR
We added the DIB_distro_DISTRIBUTION_MIRROR arguments with
I92964b17ec3e47cf97e3a3091f054b2a205ac768 as a way that we could
source a list of mirrors and then have the distro elements choose
which one applied to them.

However, this hasn't worked out to be so useful.  The
openstack-ci-mirrors element is working as a mirror setup script -- it
translates the openstack CI mirror list variables into the generic
"DIB_DISTRIBUTION_MIRROR" as appropriate for each distro's build.
Also, it turns out there's other things that need to be done, such as
turning off gpg checking, which mean the idea of "just export
variables" hasn't turned out as valid ... you need actual code
involved to get it right.

AFAICT we never actually documented these, and they do not seem to be
in use.  They have caused considerable confusion when dealing with new
platforms as we try to keep consistency.  Remove them.

[1] http://codesearch.openstack.org/?q=DIB_.*_DISTRIBUTION_MIRROR&i=nope&files=&repos=

Change-Id: Ifc4ab700631ffdfbe790068558f670f9a11dde5e
2017-07-17 14:47:31 +10:00
Ian Wienand
a00d02f6a1 Remove centos and rhel elements
Several people have popped up in IRC recently with failures in these
elements.  Without Python 2.7 available in the image they are
unsupported (OpenStack hasn't supported it for a long time).  Remove
these to avoid further confusion.

The centos/centos7 DISTRO split that has happened with centos-minimal
is unfortunate but I don't think it helps to rename centos7/rhel7 ATM.
To summarise; DISTRO=centos7 means image based build,
DISTRO=centos && DIB_RELEASE=7 means the minimal build.

In the future, I think it is important that the minimal builds and
image builds set the same DISTRO.  This reflects that "upper" layers
shouldn't care about the exact building of the lower layers.  I see
CentOS 8 going one of two ways

1) the changes are so significant, we start separate centos8 /
centos8-minimal elements.  They both set DISTRO=centos8 (and
DIB_RELEASE to point-release maybe?).  This means we have to update
all "if DISTRO == centos || DISTRO == centos7" branches to also check
for "centos8".  Evenually (!)  "centos" goes away for versioned DISTRO
only

2) we restore centos element with DISTRO=centos and DIB_RELEASE=8, and
centos-minimal remains the same.  This means we have to audit all "if
DISTRO == centos" calls to make sure they're appropriate for version 8
(stick a "&& DIB_RELEASE=7" on them all basically).

I'm not sure we can fully decide until we start to see excatly how the
distro switching/matching bits look, but (2) is consistent with Ubuntu
and probably the preferred solution.

Some "rhel" parts have been cleaned up.  More could be done in
rhel-common, but given our lack of coverage of that I'd prefer to
leave it for now.

Change-Id: I6ea784116ef59ca22878c8512c963f29c815a00a
2017-06-28 12:26:24 +10:00
Ian Wienand
f60dd38482 Add timestamp output filter
This adds a devstack-inspired output filter to standardise
timestamping.

Currently, python tools timestamp always (timestamp setup in
logging_config.py) but all the surrounding bash does not.

We have extra timestamps added in run_functests.sh for our own
purposes to get the bash timestamps; but this ends up giving us
double-timestamps for the python bits.  Additionally, callers such as
nodepool capture our output and put their own timestamps on it, and
again have the double-timestamps.

This uses a lightly modified outfilter.py from devstack to standardise
this.

All output is run through this filter, which will timestamp it.  I
have removed the places where we double-timestamp -- logging_config.py
and the prefix in dib-run-parts.

An env option is added to turn timestamps off completely (does not
seem worth taking up a command-line option for).  For callers like
nodepool, they can set this and will just have their own timestamps as
they collect the lines.

Since all logging is going through outfilter, it's easy to add a
--logfile option.  I think this will be quite handy; personally I'm
always redirecting dib runs to files for debugging.

I've also added a "quiet" option.  I think this could be useful in
run_tests.sh if we were to start logging the output of each test to
individual files.  This would be much easier to deal with than the
very large log files we get (especially if we wanted to turn on
parallel running...)

Change-Id: I202e1cb200bde17f6d7770cf1e2710bbf4cca64c
2017-06-16 15:58:50 +10:00
Mark Goddard
54765fd2f4 Remove dracut-network element
This element has not been functioning correctly for some time due to
an incorrect path to select-boot-kernel-initrd (should be /usr/local/bin).

The dracut-regenerate element can be used to regenerate dracut ramdisks
and is more flexible than this element.

Change-Id: I33d555ffd4a92b2948b2ea4a66b151f0422ccb8c
Closes-Bug: #1688546
2017-05-31 08:36:56 +10:00
Andreas Florath
e4e23897a1 Refactor: block-device filesystem creation, mount and fstab
This patch finalizes the block device refactoring.  It moves the three
remaining levels (filesystem creation, mount and fstab handling) into
the new python module.

Now it is possible to use any number of disk images, any number of
partitions and used them mounted to different directories.

Notes:

 * unmount_dir : modified to only unmount the subdirs mounted by
   mount_proc_sys_dev().  dib-block-device unmounts
   $TMP_MOUNT_PATH/mnt (see I85e01f3898d3c043071de5fad82307cb091a64a9)

Change-Id: I592c0b1329409307197460cfa8fd69798013f1f8
Signed-off-by: Andreas Florath <andreas@florath.net>
Closes-Bug: #1664924
2017-05-12 13:52:02 +02:00
Jenkins
a076bf66d6 Merge "Move do_extra_package_install to run in install phase" 2017-04-10 01:24:55 +00:00
Noam Angel
95503b4201 Move do_extra_package_install to run in install phase
This change move "do_extra_package_install" from pre-install to install
phase.

Extra packages are added by user request using the flag "-p", This
package should not be something the elements depend on.
The reason behind this patch is to move the extra package install to
a proper phase, Also more reasonable if base element run package update
to be before we install extra packages.

Change-Id: I68cc773aba9aa01743f0dda9f4e635e4cac2a282
2017-04-10 08:32:47 +10:00
Ian Wienand
6802cf7100 Run dib-run-parts out of /tmp
The dib-run-parts element was copying our internal version of
dib-run-parts into /usr/local/bin to be used running scripts inside
the target chroot.  However, it never cleaned up after itself.  This
means all images were left with an unmanaged local install of
dib-run-parts.

This copies dib-run-parts into the hooks directory of the chroot and
runs it from there.  It is cleaned up automatically on the exit path.

The dib-run-parts element is no longer required and it has been
removed from all dependencies.  It is left with a deprecation notice
in the README.  For compatability we convert it to simply install
dib-utils.

Codesearch shows no users depending on this unintentional implicit
install.  Note os-refresh-config depends on dib-utils and thus will
have an explicitly installed version.

Partial-Bug: #1673144
Change-Id: Ia2e96c00a4246c04beb96c17f83b8aefb69219ca
2017-04-05 13:11:22 +10:00
Ian Wienand
fd424757a6 Don't provide dib-run-parts
It was an oversight during v2 development for dib to start providing
dib-run-parts.  The intention was for dib to use a vendored
dib-run-parts directly from $_LIB and have no dependencies on
dib-utils at all.  By exporting dib-run-parts, we created an
unintentional conflict with the dib-utils package which provides the
same script.

Tools that depend on dib-utils are unaffected by this
(os-refresh-config).

The only tool that installs diskimage-builder and then assumes
dib-run-parts is available in the path is instack.  I have proposed
Ibfe972208df40fa092b11b5419043524c903f1b4 to modify that to use our
internal version.

Change-Id: I149c345d38d761a49b3a6ccc4833482f09f1cd05
2017-04-05 13:11:20 +10:00
Jenkins
8b19f7f99d Merge "Fix dib-init-system for Debian Jessie and Debian Stretch" 2017-03-13 03:29:41 +00:00