Some phases of diskimage-builder run outside the chroot environment,
such as the extra-data.d scripts, and don't have access to dib-python.
This means these scripts may choose the wrong python version by using
"#!/usr/bin/env python" to execute. The svc-map element is an example.
This patch creates a temporary directory and symbolic link for the
correct version of python, then manipulates the environment PATH
to preference the symbolic link "python" command.
This will allow elements with these scripts to work correctly with
the version of python diskimage-builder is running under.
Change-Id: I289d621e1bfbba0eb174dff977d1a5c92c04e4fa
Co-Authored-By: Ian Wienand <iwienand@redhat.com>
As described inline, Bionic hosts will build invalid Trusty images.
Hack around this by disabling metadata_csum in the ext4 mkfs.
Change-Id: Ibd67d58ca830a9e60605d0700ee2b17906c804e6
The base URL of EPEL repository installed by the epel-release package in
CentOS 8 at least now defaults to https.
The error seen when building an CentOS 8 image was:
"Error: Cannot find a valid baseurl for repo: epel"
This patch fixes it so that it will always match regardless of being
http or https.
Change-Id: I9ec5536ee72047c929a1ef6950ff4e9092842a4c
The ndisc6 package is not yet available in EPEL 8.
See: https://bugzilla.redhat.com/show_bug.cgi?id=1779134
Until the package is available set the pkg-map to "" for
the ndisc6 package when distro is redhat and install the
package using || true in the element script instead so
that CentOS 8 build's do not fail because of the missing
package.
Once the package is in EPEL 8 this change can be reverted.
Related-Bug: #1754219
Change-Id: Icd4bad8852ce5ba40fb0e7b0d335191efbe88c67
After the introduction of 'Add output for mis-configured element
scripts' we started seeing CI failures in tripleo where
instack-undercloud is being used (rocky/queens):
/usr/lib/python2.7/site-packages/diskimage_builder/lib/dib-run-parts: line 108: DIB_DEBUG_TRACE: unbound variable
INFO: 2019-12-02 16:24:33,423 -- ############### End stdout/stderr logging ###############
ERROR: 2019-12-02 16:24:33,423 -- Hook FAILED.
Let's make sure that by default the env variable is set
to 0.
Change-Id: I38c76c0edee436f1e7dd0c9a868cea1e6ee3271d
Closes-Bug: #1854904
Without this change, operating system elements that use the
zypper-minimal element always must use download.opensuse.org as their
repository source. This change makes ZYPPER_REPOS overrideable, which
allows the user to create custom operating system elements that can use
private repositories as their source for base packages. For example,
with only this change, it is possible to create a sles-minimal element
that generates a SLE 15 SP1 image just by overriding DIB_ZYPPER_REPOS
and DIB_OPENSUSE_PATTERNS.
Change-Id: I46e40fbe4408d4204056a27b182b21213f1176ff
On openSUSE Tumbleweed, the login.defs config file was moved under
/usr[1]. This change allows the login.defs config change to work for
both old and new locations.
[1] https://build.opensuse.org/request/show/736424
Change-Id: Ia5eff5e7b0709836278361b1b8daa788619eff75
If rdisc6 is available, a node using this element will loops until
DIB_DHCP_TIMEOUT is reached because of a missing 'break' when rdisc6
return code is 0.
This will mark the dhcp-interface@.service unit as failed (because it
has the same timeout) and not bring any network interface online.
Change-Id: I034dcda94d765f236950ebcbee36789f5bdc515f
Closes-Bug: #1854717
Signed-off-by: Hervé Rousseau <hroussea@cern.ch>
Support for easy_install codepaths is increasingly broken, and now
putting allow-hosts in this file breaks most recent pip. Just stop
installing the file - people should be using pip anyway.
Change-Id: I0a6b2432f81d80fbcbb336403fe555003880fa9f
When running under nodepool in a foreground, non-daemonized situation
without a tty (i.e. within a container) we're seeing this "wait" hang
indefinitely.
It is probably related to "outfilter.py" and output file descriptors,
although TBH we haven't completely root-caused it. I won't claim this
is a great solution, but it should hopefully let the dib process
finish and just die, where outfilter will disappear.
Change-Id: If78da54df3d4c240fee16aee4413ec554b37c1d6
The current implementation evauates the dib-init-system
script too early. Also it looks that there is no simple
way of getting the info about the init system automatically:
another element can install (later on) a different
init system. Therefore the only reliable way of setting
this is manual.
Change-Id: I6e9ffa1bdb3154f488f4fd335b197699b86aacd4
Signed-off-by: Andreas Florath <andreas@florath.net>
I commonly get asked for help when people are attempting to create
local image elements and they cannot get them to work.
diskimage-builder silently ignores element scripts that it doesn't
find to it's liking, such as non-executable or files with extensions
(.sh is a common mistake).
This patch extends the '-x' tracing flag down to dib-run-parts and
will cause it to print out helpful messages when these files would
otherwise be silently ignored.
Examples:
Ignoring non-executable files: 10-do-not-run-me
Ignoring non-conforming filenames: 10-I-can-run.sh
I am not enabling these by default as they can create extra noise
and require additional filesystem IO to produce.
Change-Id: Ic804efca3015c199440b4b10da951d71a815c64f
When the rdisc6 utility is available probe for router
advertisement. configure eni and rhel-netscripts interfaces
to do IPv6 address configuration according to the flags
in the RA recived from the router.
The systemd service file timeout is DIB_DHCP_TIMEOUT * 2,
so that DHCPv4 can timout, and dhcpv6 run before the service
times out.
Retries are commented in dhclient.conf, without it we end up
trying DIB_DHCP_TIMEOUT * 60 before the client move on to
IPv6.
WHEN:
Stateful address conf. : No
Stateful other conf. : No
THEN:
Do not run dhclient at all, autoconfiguration via
SLAAC only.
WHEN:
Stateful address conf. : No
Stateful other conf. : Yes
THEN:
Run "dhclient -6 -S", The ``-S`` option makes the
dhcp client not request an address, only other
options such as DNS servers and NTP servers from
DHCPv6 server.
WHEN:
Stateful address conf. : Yes
Stateful other conf. : Yes
THEN:
The dhcp client should request an address _and_ other
options such as DNS servers and NTP servers from
DHCPv6 server.
NOTE: No IPv6 support added for suse-netscripts
Closes-Bug: 1754219
Change-Id: Icdc79875c33f894ab7eaec8afdfb33a731efff99
Currently DIB_ADD_APT_KEYS only supports GPG armor keys, while
default Debuntu apt gpg keys are in keyring format.
Change-Id: I361c375e25b03a08b19052b10c6733939c8df921
This reverts commit a3e9e7f89e.
We still have some issues with vhd creation on RAX
In short, it appears that images fail to resize unless they have a
specific "creator" field. Revert this while we consider the options.
[1] https://bugs.launchpad.net/nova/+bug/862653
Change-Id: I2b6a3bfbfe28432fbb6a2ce4a0211939d224b8d5
The "ironic-agent" is copied to ironic-python-agent-builder and
hence it is deprecated from DIB.
Remove from functional testing
Change-Id: Ibc4f75b9d7e2a31994fc86d05bd57975f00fb74f
Task: 36198
Story: 2005114
This package is not installed by default on Debuntu, but is on RH
platforms. This is causing a build breakage as DIB_PYTHON_VIRTUALENV
tries to use this (I3414fb9e503f94ff744b560eff9ec0f4afdbb50e).
Add the package.
Change-Id: I9a551c57dd128bbb4b095c847f634c777b2cb553
To ensure dracut does not load nouveau we need to explicitly disable it via
omit_drivers.
This change adds a method to drop in arbitary dracut conf files to an element
which are picked up by dracut-regenerate and included in the chroot where we
run dracut.
The disable-nouveau element just adds a conf file with
`omit_drivers += " nouveau"`
The default dracut conf files in /usr/lib include a similar file to omit the
nvidia kernel modules.
Change-Id: I6375e4843fd08d1410141fbbd8658042dcd5ad05
Closes-bug: 1842664
Seeing this at the end of the tripleo overcloud full build:
99-selinux-fixfiles-restore: line 69: [: too many arguments
Change-Id: I8fb10f3d3d38723b41190ae1898757e6df073945
The vhdutil utility is completely dead; the whole subsystem it relies
on was removed with [1] so it's not even vaguely possible to keep it
up-to-date.
I took the .raw images on a nb and used the qemu-img there (so Xenial)
and generated some VPC images; uploaded them to rackspace and the all
seemed to boot fine. If there was a problem, maybe it's been fixed on
either the qemu or RAX side in the previous few years.
Thus swith to qemu-img to generate the vhd images too.
[1] https://xenbits.xen.org/gitweb/?p=xen.git;a=commit;h=5c883cf036cf5ab8b1b79390549e2475f7a568dd
Change-Id: I3099d2ebb958370fcec623087a093b2c8dbdc6c4
Add option to set the suite subpath after the release name for the
security mirror URL independently in the debian-minimal element,
since this can differ between mirrors.
Change-Id: I4cc8f54fba012986423e30e19bff276208b8ad62
With the introduction of centos 8 we have constructs like
if [[ $DISTRO =~ (centos|fedora) && $DIB_RELEASE -ge 8 ]]
This is intended to match the "centos7" element (from the =~) but it
was missed that this is setting the DIB_RELEASE to "GenericCloud".
I think it makes more sense for this to be a numeric release, and
makes constructs like above work. There really isn't any other type
of image to choose here; thus we move it into a new, centos7
specific variable.
Note that when the centos 8 images are available, we want to move to a
generic "centos" element that will handle both 7 and 8 together (same
as rhel) based on DIB_RELEASE and deprecate centos7; this works with
that environment too.
Change-Id: I2e6b7848070d6452c0563e2a122447627c6e6bf7
It turns out that this breaks ipv6 config with NM. Instead what we want
is for glean to not up interfaces on boot (see the depends-on).
Change-Id: I6c5bc76c433e29f02d3266ab8f669015125ec954
Depends-On: https://review.opendev.org/#/c/688031
This adds CentOS 8 into functional and boot tests.
This completes centos-minimal support, documentation is updated and a
release note is added.
Change-Id: I435c2967b4f49faeb6d6edf189907b9f96e80357
As described inline, NetworkManager and dhcp-client make up the basic
networking for centos 8 installs; bring them into the base image.
Although in infra we then use simple-init, some other users find this
helpful.
Change-Id: Ib9f32e73bf9109cc1b659fe1deceb1a15301ffeb
By default network-scripts package isn't installed, so the directories
for these files don't exist either. Skip by default for Centos 8.
Change-Id: I194ec3735e17f27e586386541dc51f775b01e510
Use the wrapper calls from Ia267a60eecfa8f4071dd477d86daebe07e9a7e38
to install glean.
Using this wrapper means we cover all cases without more and more
branches; it should work for python2, python3 and also the special
case of RHEL/CentOS where dib-python points to the special
/usr/libexec/platform-python (which is python3.6 with inbuilt pip)
Change-Id: If624e8bb66ce0761fc0d5f34c2bed8b93a7daeee
NetworkManager with simple-init has proven to be stable in OpenStack
infra, switch to it by default for CentOS and Fedora. For CentOS 8
and Fedora, add a check to make it the only option. Thus only CenOS 7
remains optionally using the legacy scripts; this is likely not used
anywhere (infra is really the primary user, where NetworkManager is
already used); we can likely remove this variable (and hence path) in
a future cleanup.
In the setup, remove rhel7 element which was never really tested.
Reorganise the fallthrough to call out the default paths as doing
nothing.
Change-Id: Ic996956da4b85f7d95179b8df9881d5f52c091af
Currently, the serial console is hardcoded to ttyS0 in the bootloader
element. This is a challenge for users that want to build images for
some baremetal servers. Supermicro servers, for example, use ttyS1 for
the serial over lan interface.
This patch adds a new environment variable DIB_BOOTLOADER_SERIAL_CONSOLE
that can be set to override the default.
Change-Id: Ie8173be8690ac0b7164ce9e5b66d3c1c18f844d6
Add option to set the security mirror URL independently in the
debian-minimal element, since this can not be overriden by the
standard DIB_DISTRIBUTION_MIRROR variable.
Change-Id: I145844a410d06a479e68db1bf6d5d0159389305c
As described inline, deprecate the "source" install for CentOS 8.
Overwriting the packaged tools has long been a pain-point in our
images, and the best outcome is just not to play the game [1].
However, the landscape remains complicated. For example, RHEL/CentOS
8 introduces the separate "platform-python" binary, which seems like
the right tool to install platform tools like "glean" (simple-init)
with. However, platform-python doesn't have virtualenv (only the
inbuilt venv).
So that every element doesn't have to hard-code in workarounds for
these various layouts, create two new variables DIB_PYTHON_PIP and
DIB_PYTHON_VIRTUALENV to just "do the right thing". If you need is
"install a pip package" or "create a virtualenv" this should work on
all the platforms we support. If you know more specifically what you
want (e.g. must be a python3 virtualenv) then nothing stops elements
calling that directly (e.g. python3 -m virtualenv create); these are
just helper wrappers for base elements that need to be broadly
compatible.
[1] http://lists.openstack.org/pipermail/openstack-infra/2019-September/006483.html
Change-Id: Ia267a60eecfa8f4071dd477d86daebe07e9a7e38
Don't install the "yum" package, which is a backwards compat around
dnf. With 687003f we should not need the backwards compat links any
more.
Add libcurl to avoid conficts with in the curl "-minimal" packages
that happens on CentOS 8. But skip it on Fedora, because it seems to
create more problems there (not going to pretend it isn't all a
hack ... but it seems to work).
Change-Id: I1de2703eb5075a0a22837b6898bd8eb960d080dd
A few places we either assume centos uses "yum" directly, or have
switching based on the distro type.
In both cases, we can use ${YUM} directly to avoid ambiguity
Change-Id: I71095a9bd1862f8956b5982fbbb3e1d213926c14
The libselinux packages etc don't exist for Python 2 on Centos 8 [1].
Ensure the package map installs the python3 versions.
We could probably invert the logic now, and make it so Centos 7 is the
"special" version that overrides things to install python2. Left
alone for now to avoid changing too much at once.
[1] https://bugs.centos.org/view.php?id=16458
Change-Id: I944cf4f2902c28728aa5bb9e2a00b3eef122d52e
CentOS 8 has the "new" split-up locales packages. Fedora 24 is now
long gone, so take out the old branch and apply the lang package
install to Centos 8 as well.
The manual locale cleanup is not necessary on Centos 8; skip it.
Change-Id: Ib65fc15fe471348793fd6efb034517f11abd905e
The repo format has slightly changed for CentOS 8 (s/os/baseos/).
Make the chroot builder look for a more specific repos.d directory
first named for the distro variable, then fall back to to top-level
dir (this avoids having to constantly change fedora).
Update the gate mirror setup and roles for new Centos 8 paths too.
Change-Id: I5b7f0c3624cac1d7aa7ed8bf6286b85d808b9c9a
This is no longer a valid option for dnf, and it puts out a lot of
warnings constantly about the invalid entry [1]. Remove it.
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1653831
Change-Id: Iba0585cab3e5e78e9324196f276b2341e7bb9e3c
Install the Python 3 libselinux packages for Fedora platforms. I
think this is the right choice; Fedora is a Python-3 only distro so we
shouldn't default to installing the python2 libraries.
This has a practical effect if you're using Ansible with
ansible_python_interpreter=/usr/bin/python3 as it needs these
packages.
There is some small chance of breakage if you're using Ansible still
with Python 2, I guess. In infra I notice we bring this in with
"zuul-worker" project-config element. On balance, I think that if you
need the Python 2 packages for some reason, it should be a special
install and not part of redhat-common.
Change-Id: Ibcec0b3660d01b861838c2ae87ca43d98953ce32
Two bugs are addressed.
1) The sysprep element was broken in that it only truncates
/etc/machine-id, but not /var/lib/dbus/machine-id. systemd will
not generate a new machine-id if /var/lib/dbus/machine-id is
present[1], it will simply copy it to /etc/machine-id.
We observed machine-ids being packaged in /var/lib/dbus/machine-id
on several distros: Ubuntu Bionic, Fedora 29, Debian Stretch.
CentOS 7 and Ubuntu Xenial do not contain packaged machine-id as
far as I can tell.
All test builds were performed using -minimal elements.
2) A second bug existed where debian-minimal did not run the sysprep
element at all, so a stretch image I tested contained a populated
/etc/machine-id AND a populated /var/lib/dbus/machine-id.
[1] https://www.freedesktop.org/software/systemd/man/machine-id.html#Initialization
Change-Id: Ibb28b6e90d966a845de38a2cd5a1e8babd2604bc
Similar to https://review.opendev.org/#/c/663693/, the x64 packages
should be used for x86 architectures.
Change-Id: I5e8a4d58e96d65eb60fc539b8a1d56853b12faac
Closes-Bug: 1843820
linux-firmware and linux-firmware-whence (meta package for mostly iwl
firmwares) packages account for approx. 289 M install size on a F30
system, and linux-firmware for approx. 176 M on CentOS 7. Users needing
these firmwares are eventually baremetal users and are not looking for a
very minimal operating system base install like virtual image users are.
Thus, a non-minimal OS element is better suited for them. Alternatively,
it could be later considered a dedicated firmware element.
This is inline with I8ce65e1d357d15e8ed8995ad1dcaea02bbd1986f.
Change-Id: If104fc3c1e9349b8d501a2351fff1ab4c0dbc6a4
This is consistent with the previous simplication of
build targets in the opendev environment to refer to
"opensuse15" being the alias of "latest stable openSUSE Leap 15.x".
Change-Id: I904a3ca0d6dbddd2bb1a673836ab6a0ad249526d
Add a new environment variable $DIB_GZIP_BIN allowing builders to
specify a different gzip (such as pigz) to be used when compressing
tgz images.
Change-Id: Ifb617568140a149e2fda241e07ff8a59429e6697
We have an application breaking because /usr/share/cracklib is being
deleted from the image. The application installs its dependencies,
including cracklib, but since yum shows that cracklib is already
installed, it does not reinstall it.
Change-Id: Id6fccf76c706dbc6c2124abcfd12c1f10cef5e09
Newer openSUSE distributions install an absolute link to /run/netconfig
as /etc/resolv.conf in $TARGET_ROOT. as that points outside
TARGET_ROOT, we unintentionally wipe the system resolv.conf here
and break our ability to finish building the image.
Change-Id: I9d5aaa9fad2f81dcabfe19e2f1e6b6e50af597d7
This is a follow-on to I475a253091cbaf63687b91c748c31a6753bb0f57 as we
are still seeing issues on some clouds with unconfigured networking.
We increase the timeout, but also make it configurable so we can
fiddle it without a dib release in the gate.
To follow-on from the experimentation done by clarkb, I can confirm by
emperical testing on a Centos 7 image (from today, today being this
change's date) that setting
net.ipv6.conf.all.autoconf=0
by itself is "fatal" and the interfaces do not come up; i.e. nm does
not by default seem to re-enable ipv6 for the interface. However,
explicitly adding:
IPV6INIT=yes
IPV6_AUTOCONF=yes
to the interface file *does* seem to make it work, even if
"all.autoconf=0" is set (then again, there's also bugs about the
effect of this [1]). However, no extant distribution (I can currently
find) does anything like this by default.
If this continues, this may be an option. Another might be to avoid
the use of the nm-settings-ifcfg-rh profiles and move directly to nm
ini files with glean.
[1] https://bugzilla.kernel.org/show_bug.cgi?id=11655
Change-Id: I869ebffc8cde3bbff573f6583fd9dd02a5598590
Upstream is now publishing 17.1 profile systemd stages
Also updates the docs that were forgotten in the last patch
Change-Id: I0f2e7976845b1d3c55ffe8869eec0bc04a191252
As described inline, we need to ensure the underlying directories in
the image are correctly labeled, or we get all manner of services
failing during boot with selinux in enforcing mode. Although the
problem is generic, this first shows up in Fedora 30 as systemd has
become more strict about namespace failures (I think) [1].
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1663040#c22
Change-Id: I52c1cc719884879169b606b00651aa26f5b783f1
This patch adds option DIB_YUM_MINIMAL_EXTRA_REPOS to yum-minimal to
allow DIB users to include extra repositories to their final image.
Change-Id: I89549f4b0f4c9470143b5064817acab5043e31c5
Something (possibly [1], but that change is at best cryptic) has
changed such that we don't get correct /etc/os-release files
installed. This flows on to grub half-installing itself, enough to
not fail the build but not enough to make something bootable.
Installing the -cloud release package gets it back, and seems like a
sane choice for dib.
[1] 617b1bed34
Change-Id: Iff0413887fad798273b2bfcb140cc07f36d54a04
As noted in the change, 7fd52ba841
increased the size of the EFI partition considerably. This has meant
that our padding upwards of the disk size is insufficient and EFI
builds (arm64 in particular) is failing due to out-of-disk errors
during final image operations like installing kernels.
Similar to the discussion we had in
I65fa13a088eecdfe61636678578577ea2cfb3c0c, this feels a bit ugly
because we're mixing logic here with sizes specified in block-device
config files. But it boils down to the same problem; we are
calculating the disk size here and passing it to the block-layer, so
unless we want to make large changes to the status quo about where
these sizes are calculated, small adjustments here are the most KISS
solution.
Thus we check if we have selected the EFI bootloader element, and thus
assume there will be a large system EFI partition and expand the disk
size accordingly.
Change-Id: Ifa05366c2f2b95259f3312e4dde8c85347075ba1
This debug statement lists every element found and its dependencies on
every build; it's just noise unless you're debugging the element
dependency solver itself. Remove from output.
Change-Id: I9281b953d958a3fd5e20edbc560a341a2fcc3deb
This seems to miss the exit code of the dracut process, which actualy
caused some issues in I8511669e188717494daf2bc1384a6dd346f942a4 where
it would have been much clearer to stop after the initramfs generation
failed.
Add some debug messages, and catch any errors from the final call.
Change-Id: I6f89441ec4709f5199535e15a7cc53a3a8af273d
Should install "grub2-efi-aa64 grub2-efi-aa64-modules" instead of
"grub2-efi grub2-efi-modules" for arm64
Change-Id: Iee3191b0944b3b862890d166a9d36bd592fe8f7e
Closes-bug: #1839816
- DIB_DISTRIBUTION_MIRROR_UBUNTU_IGNORE matches when it is empty or not
set and DIB_DISTRIBUTION_MIRROR is being used. Checking for it being
set and not empty solves this.
- Normalizing bash conditionals for readability
Closes-Bug: #1808359
Change-Id: I87853fcda4c8b29a3f1720a2778debeb3acc3a53
Signed-off-by: Manuel Torrinha <manuel.torrinha@tecnico.ulisboa.pt>
The 'pypi' mirror element is generating invalid pip.conf files
when more than one "DIB_PYPI_MIRROR_URL" is specified.
This patch fixes the pip.conf file rendering to use a proper form
when there are multiple "extra_index_url" provided.
Closes-Bug: #1839558
Change-Id: Ibda0e7390955683560e09b486f636775643ff57c
python 3.6 warns about regexes like:
DeprecationWarning: invalid escape sequence \+
I noticed that debugging a trove job and it really led me in the wrong
way. Fix this with making it a raw string.
Change-Id: I58ee1a49d62316c6c3f0588832c97f659f7e460b
Per the inline comment, a machine-id is required for kernels to
install correctly (this may well be a bug, but the linked issue
remained inconclusive).
Add a call to make the machine-id before install packages.
Change-Id: If75d04376e62bfdfe14ee3ca4d0bd5c8b383c1b0
Redhat-Bug: https://bugzilla.redhat.com/show_bug.cgi?id=1737355
The 17.1 profile changed the defaults used in portage for where we store
our repo, distfiles and binpkgs. Some portage related variables need to
be set deterministically. 17.1 is no enabled for Systemd's profile.
Change-Id: Ib55f6875c5cb461c3c530b51d7420ce3dc8da360
This element configures systemd to send its journal to the console,
which can then be retreived by server commands. In the case of
nodepool, if the image failed to boot the console will be dumped into
the logs when nodepool decides the node is not responding. Having
this can be very helpful diagnosing early boot errors.
Needed-By: https://review.opendev.org/#/c/669787/
Change-Id: I6b6df7023acb6b2f967b84840bc4b542ebc03727
Newer versions of open-iscsi seem to compile on Gentoo / musl. Use them
if we can. This also removes the cap on open-iscsi.
Change-Id: I596cb61494e459a419bce6a63deff89f9e78fe23
Previously we were trying to enable dbus-daemon service on all prior to
fedora 30. Unfortunately 28 and older don't have this service so this
broke those releases and only worked for 29. Fix this by only enabling
this service on fedora 29.
Change-Id: I1bd15dcf0bbe270afccb0c0c3ea6ad08862a53f1
The linux kernel and NetworkManager fight each other over control for
interface management when router advertisements are in use. Long story
short if the linux kernel configures a network interface for ipv6
before NetworkManager attempts to manage that interface then NM will
ignore the interface and not configure ipv4 on it.
This can happen because the kernel is configured to send router
advertisements solicitations which result in router advertisements which
the kernel uses to configure the interface(s). There is a default of a 1
second delay before sending the solicitation which in many cases is long
enough that NM has started before then. However, in slower environments
like those used for testing with qemu this isn't long enough.
Some testing by hand indicates that 15 seconds is about right so
increase the delay to 15 seconds via sysctl.conf.
Note this may increase boot times in ipv6 only environments (though it
is hard to be sure due to how systemd starts everything at once and does
socket activation and the like).
Change-Id: I475a253091cbaf63687b91c748c31a6753bb0f57
When virtualenv and setuptools gots installed from source and rpm
then their installation path lives at different places but when
the python script got called then that time it choses either of
rpm or source based path on system wide installation and leads to
different failure as their methods are not implemeted.
So by setting _clear_old_files to 0 will install
python3-virtualenv python3-pip python3-setuptools from rpms only
and avoid these failures.
Change-Id: I0c162f1fe8168513e352546ab8dd2b68fa65b88c
Signed-off-by: Chandan Kumar (raukadah) <chkumar@redhat.com>
autounmask=y (default) changes portage depsolving, causing errors
(mostly often seen in perl and binpkg related issues).
Disabling this functionality for DIB builds is OK as the enviroment is
not passed on post build and the build process is not interactive
anyway.
Change-Id: Ife9ace246bec16864ee4982bc456763af5dff2e8
Signed-off-by: Matthew Thode <mthode@mthode.org>
debian-minimal depends on debootstrap which depends on dpkg
This needs to be installed early as dpkg installs the apt keys early via
02-add-apt-keys in pre-install.d
Change-Id: I8580849ceaa7a5152c94f29afa890ac6d6983fb1
This change removes useless statement accidentally added in the
06576a02f0
Change-Id: I7ea4a24d8c72c9e72f5f87247403af0f9bf69b40
Signed-off-by: Maksim Malchuk <maksim.malchuk@gmail.com>
There are several jobs depending on working opensuse 15.1
images in nodepool, so it makes sense to ensure its working.
Also upgrade the previously marked experimental opensuse-15.0
job as it tests the xenial->opensuse combination, which is
particularly difficult to keep working and we'll need it in
the CI.
Change-Id: Icb6d998756ce5221e017959dcb59b21f0f023454
This patch adds a new environment variable to the ubuntu-minimal
element called DIB_UBUNTU_KERNEL that allows you to specify the kernel
meta package that will be using to install the kernel inside the image.
It supports "linux-image-generic" (The default), "linux-image-kvm", and
"linux-image-virtual".
This allows building images that are smaller in size (~200MB smaller
qcow2) that have only the kernel modules necessary for virtual
machines.
Change-Id: I8ce65e1d357d15e8ed8995ad1dcaea02bbd1986f
1. Sync sphinx dependency with global requirements. It caps python 2 since
sphinx 2.0 no longer supports Python 2.7.
2. Update some URLs to latest
3. Remove the unnecessary space
Change-Id: I5464be9e055feecd80918f691448acf5f100e701
Use openSUSE 15.1 as default, which is the latest released stable
openSUSE release.
Remove leftovers for unmaintained openSUSE 42.2 images.
Depends-On: https://review.opendev.org/#/c/660126/
Change-Id: I0b204b7b3d7ae74b6749320b3bfe1ca89d154ebb
This patch removes the check and default for rhel 8 requiring
xfs filesystem as rhel 8 images can successfully be built with
ext4 filesystems.
Change-Id: I1a6bfa26324fd43ae0c77c2c977dda0dd56e26e5
Nowadays, in the time of Predictable Network Interface Names, the
network interface names 'ethX' are not used that often any more.
Depending on the virtualization layer and the guest OS names like
'ens3', 'enp1s0' or 'enp0s31f6' are used.
This patch enables the user to set DIB_NETWORK_INTERFACE_NAMES to a
list of network interfaces which are brought up using DHCP during
(first) boot.
Change-Id: I04cc2ee710f0389a8085b1c91d9329784cb28048
Signed-off-by: Andreas Florath <Andreas.Florath@telekom.de>
The latest Fedora/Ubuntu images don't ship python2 by default, so we
need to use our dib-python wrapper for this so we work in python3 only
environments.
This change also correctly creates the pip.conf and .pydistutils.cfg
files with trusted host extracted from the index-url.
Related-bug: 1577105
Change-Id: Ibb5348af3e3bbe46b19affe90a8930a4b4ad4cad
Signed-off-by: Maksim Malchuk <maksim.malchuk@gmail.com>
RHEL8 ships a bunch of grub2-efi-X-modules in its main
repository, each of which provides grub2-efi-modules,
potentially causing nondeterminism when building images.
This changes the DIB elements to always use architecture-
specific RPMs when RHEL8 is selected.
Change-Id: If94f3721195d5ecd80036e4234a3ca223a19c349
Related: https://bugzilla.redhat.com/show_bug.cgi?id=1716672
When I said in I8594d1fe05242f246a5809740a115ab2f84ac5a3 that 12 MiB
ought to be enough, I should have expected that I would be proven wrong.
While 12 MiB is enough to fit shim-x64 and grub2-efi-x64, yum fails to
update these packages to newer versions:
Transaction check error:
installing package shim-x64-15-2.el7.centos.x86_64 needs 7MB on the /boot/efi filesystem
installing package grub2-efi-x64-1:2.02-0.76.el7.centos.1.x86_64 needs 3MB on the /boot/efi filesystem
Error Summary
-------------
Disk Requirements:
At least 7MB more space needed on the /boot/efi filesystem.
It is recommended that the ESP partition be much bigger. This commit
bumps its size to 550MiB, following guidelines from Rod Smith to avoid
incompatibilities with some EFIs [1].
[1] https://www.rodsbooks.com/efi-bootloaders/principles.html
Change-Id: If9515234f1a803cda32b2482f8abe10ddf0e6d26
Avoids failing on the first attempt to download the image to cache as
mirrors hosting them can randomly go down, usually with a connection
refused.
Change-Id: I9de9f33c2cc16596d04b35c4eb92621e6a2c7511
When the mirror returns a error, it was trying to interpret the error
message (e.g. <html><title>Internal server error..) as a download link.
By using -f on curl we get an empty reply and an exit code, which, as
we run in set -e mode, aborts.
Change-Id: Ibaa39aedb7db286f859c4b090114c6a233b150c7
The rhel7 element is deprecated and is left only for backward
compatibility.
The rhel element should be used instead. Users should set DIB_RELEASE to
'7' to indicate which release you are using.
The new element is a version-less RHEL element to handle both '7'
and '8' DIB_RELEASE, which aligns with other elements which operate in
the same way such as the Fedora element.
Change-Id: Ic39ed85cacae9942448eb18ad685763f9369c2ed
Make a version-less RHEL element to handle both '7' and '8' DIB_RELEASE.
The element usage should align with other elements which operate in the
same way such as the Fedora element.
Additionally, this patch adds support for RHEL8 that operates with
Python 3.
As of now, users of diskimage-builder will still be able to use the
'rhel7' element, or migrate to 'rhel' and specify their respective
DIB_RELEASE value.
* mount the xfs file-system for extraction as read-only. vaguely
based on explaination in [1] and the fact we only read the image
data into a tar, so can ignore this.
XFS (dm-1): Superblock has unknown read-only compatible features (0x4) enabled.
* Use the redhat system python as the dib-python version. dib was
ahead of it's time making an abstracted python interpreter for
system work ;) the system python should work for running the various
dib element scripts.
[1] https://unix.stackexchange.com/questions/247550/unmountable-xfs-filesystem
Redhat-Bug: https://bugzilla.redhat.com/show_bug.cgi?id=1700253
Co-Authored-By: Ian Wienand <iwienand@redhat.com>
Change-Id: I90540675c70bb475d9db2ae24f81c648a31f3f95
I want to use the new --image-extra-size flag[1] but my use-case
calls for megabyte granularity of this value. Rather than adding
60% to an 800MB image, maybe I only want to add 100 or 200MB, etc.
[1] https://review.opendev.org/#/c/655127/
Change-Id: I8fb9685d60ebb1260d5efcf03c5c23c561c24384
Use openSUSE 15.0 as default, which is the latest released stable
openSUSE release. Switch to https for accessing download.o.org
as encrypted transfers should be used by default.
Remove leftovers for definitely unmaintained openSUSE 13.x images
and split into old/new leap style versioning scheme for clarity.
Change-Id: Iab129eeee2b1a2563f0f0d2cb17bbad57c068e38
It looks like fedora-release on fedora 30+ has been split into sub
packages. Use fedora-release-common to avoid package conflicts.
Change-Id: I8f8711044fc4074b91939e0a6dfdac4d7a14a35b
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
In fedora-30 is when we migrate to dbus-broker, fedora-29 is still using
dbus-daemon.
Change-Id: I1e1d3a3826157b8b22386c211eaa58b6439b5f3c
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
Depending on the version of $DIB_PYTHON_VERSION, we can either use pip /
pip3 to install glean. This is helpful for newer OSes that might not
want to ship python2 (pip).
Change-Id: I25c5927a1eb55ee16b919dd64403184f335839b6
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
Harden sshd configuration by adding KexAlgorithms, Ciphers and MACs for sshd,
following good pratices on https://infosec.mozilla.org/guidelines/openssh
Change-Id: I3051320d867a5033e82deef10c5e723ca9829884
Co-Authored-By: Nicolas Hicher <nhicher@redhat.com>
Currently diskimage-builder supports two ways to specify the image
size. One is defining a fixed image size using DIB_IMAGE_SIZE, the
other one is auto-detection while adding a security margin of 60% as
free space. This means when building larger images (e.g. >100GB) with
unknown size upfront we end up with much wasted space, IO and network
traffic when uploading the images to several cloud providers. This can
be optimized by adding a third way by defining DIB_IMAGE_EXTRA_SIZE to
specify the free space in GB. This makes it possible to easily build
images of varying sizes while still minimizing the overhead by keeping
the free space constant to e.g. 1GB.
Change-Id: I114c739d11d0cfe3b8d8abc6df5ff989edfb67f2
In many cases, the statically sized 64MB journal is far below the
e2fstools default calculation[0] which calls for a 64MB journal only
on filesystems smaller than 16GB. On bare metal in particular, the
correct default journal size will often be in the 512MB-1GB range.
Since we cannot know what the target system is, this should be a
tunable parameter that the user can set depending on the intended
image usage.
Add a DIB_JOURNAL_SIZE envvar and --mkfs-journal-size parameter
to the image creation so users can override the default journal
size.
[0] https://git.kernel.org/pub/scm/fs/ext2/e2fsprogs.git/tree/lib/ext2fs/mkjournal.c#n333
Change-Id: I65fa13a088eecdfe61636678578577ea2cfb3c0c
Due to the referenced bug, many versions of debootstrap can't bring up
a buster environment. Unfortunately, these include versions we use to
do this on Xenial/Bionic nodes.
Also, there isn't backports or security updates, so elide these for
now.
I did get a working build (I haven't gone so far as a full boot+glean)
with this, at least.
Change-Id: If2420e92cb728ab6e91b0d70547da4483679b391
Paritial-Bug: #1822927
Currently, the cleanup script is using the existence of the folder
/sys/fs/selinux to check if SELinux is enabled. This, however, is
misleading in case disk-image-builder is used inside a Docker
container on a selinux-enabled host. In this case, the folder exists
in the container but SELinux is disabled.
This patch addresses the problem by checking, in addition to the
check already in place, the output of the command selinuxenabled.
Change-Id: I83e58f2467e60df9f0f00f7b7a58d0e2ce357a9a
Closes-Bug: #1820077
This is a mechanically generated change to replace openstack.org
git:// URLs with https:// equivalents.
This is in aid of a planned future move of the git hosting
infrastructure to a self-hosted instance of gitea (https://gitea.io),
which does not support the git wire protocol at this stage.
This update should result in no functional change.
For more information see the thread at
http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003825.html
Change-Id: Id26bec14c3d94e2f81b2148fc85d17f07866398c
This is only one line, but it takes a lot to untangle ... basically
the current "correct" path is:
---
mk_build_dir()
-> sets trap trap_cleanup EXIT
... stuff ..
mount_proc_dev_sys
-> mounts $TMP_MOUNT_PATH/<proc,dev.sysfs>
pre-finalise.d
finalise.d
unmount_image $TMP_BUILD_DIR/mnt # nb == $TMP_MOUNT_PATH
-> unmount_dir()
-> recursive unmount everything inside TMP_MOUNT_PATH
TMP_IMAGE_PATH=$(dib-block-device getval image-path)
export TMP_IMAGE_PATH
dib-block-device umount
dib-block-device cleanup
... actually cleanup directories ...
---
Our current failure exit trap does:
---
dib-block-device umount
unmount_image
...
---
Note this is the *opposite* of what is done in the correct exit path.
In the failure case, if a script fails in the finalise stages it leads
to /proc, /sys, /dev etc. still being mounted inside the image; the
"dib-block-device umount" call doesn't know anything about these
mounts and tries to unmount the parent directory, and we get a hard
failure with a busy mount, and all the mounts are subsequently leaked.
Note that "unmount_dir", which is ultimately called by
"unmount_image", already knows to skip those mounts that
"dib-block-device umount" manages (this is the DIB_MOUNTPOINTS list).
This is further evidence it should be called *before* the
dib-block-device umount.
Change-Id: Ibef3ce9d1167b9c4ff3d5717b113cd3ed374f5e3
Add a DIB_APT_MINIMAL_CREATE_INTERFACES boolean to the debootstrap
element which functions identically to
DIB_YUM_MINIMAL_CREATE_INTERFACES in the yum-minimal element.
This can be used to disable the creation of the
/etc/network/interfaces.d/eth[01] dhcp configuration files, which
are not needed on systems where cloud-init or other means are used
to configure networking.
The flag is enabled by default to keep creating the dhcp interface
files, maintaining backwards compatibility.
Change-Id: I1fdaca8350a5ceefd9e437af4fd000ce6a3ee7f3
in same cases it is required to avoid update all existing packages,
doing so can result in release update which is currently not possible
unless you not include "base" element.
"base" element used for most distribution (rhel, debain), and is
necessary for most cloud operations, this patch add
"DIB_AVOID_PACKAGES_UPDATE" parameter to skip updating all packages.
usecases for this patch can be:
* Avoid release update when building old release ex. RHEL7.5.
* build on network-less environment.
usage:
DIB_AVOID_PACKAGES_UPDATE=1
or
DIB_AVOID_PACKAGES_UPDATE=0
Change-Id: I71192b23c8f0bc48b348fe7377bf8a2399b53792
Related to I041a141366099093805e6052b1bbf64efd277e1e, we also need to
remove this on opensuse. The files for gate testing are added, but
the test is not added to any jobs at this point in the interests of
gate time.
Change-Id: I1af9e84d76bedcb2607717edc6d2abe2920b0584
This fixes a regression in I041a141366099093805e6052b1bbf64efd277e1e
where we starting skipping the removal of old files for image-based
builds (confusingly named centos7 rather than centos for historical
reasons). Fix the check
Change-Id: I74688a9e91d833b5d654056431729bed0585616c
As described inline, we only want to remove the system package files
on centos; it causes problems on Fedora where some system tools expect
these to be there.
But there is an additional bug -- pip actually removes the system
package files anyway. To work around this, reinstall the system
package.
Closes-Bug: #1813232
Change-Id: I041a141366099093805e6052b1bbf64efd277e1e
As described in the comments, it seems the transition between
dbus-daemon -> dbus-broker in Fedora 29 has made it so the packages
can get into a state where neither service is enabled.
Explicitly install and enable dbus-broker for F29
Change-Id: I06753043a75be2f635653899c6c251b9fbdd7c67
The path $TMP_BUILD_DIR/mnt becomes the / inside the chroot during
the chroot phases of diskimage-builder. Previously this path was being
created using the account running diskimage-builder. This account may
not be valid inside the chroot. This causes path validation, when running
on a Ubuntu bionic host, to fail.
This patch chown's the $TMP_BUILD_DIR/mnt to root.root to make sure
that / is owned by a valid account inside the chroot.
Change-Id: Ifedc136baa67c7952942aed2c8cb1041902fef91
Closes-Bug: 1811113
There is an use of get_image_element_array on the environment.d
phase, for the iscsi-boot element.
This function is not available on that step. So moving the check
at next step, extra-data-d, where it is available.
Change-Id: I89cfe565492142c2f7962109360fcbcebadfd469
This plumbs through an "--use-nm" flag to glean which instructs it to
setup interface bringup with NetworkManager rather than legacy network
enablement scripts.
In this case, install the NetworkManager package. In the non-nm case,
also install the network-scripts for Fedora 29 -- this has stopped
being installed by default (it's been deprecated since forever).
As noted in the docs, this is currently really only relevant on the
supported rpm distros which are using the ifcfg-rh NetworkManager
plugin to effectively re-use old config files. However,
NetworkManager has similar plugins for other platforms, so support can
be expanded if changes are proposed.
Depends-On: https://review.openstack.org/618964
Change-Id: I4d76e88ce25e5675fd5ef48924acd09915a62a4b
Provide a "when" option that provides for not installing packages
based on a = or != match on an environment variable.
Unit tests are added.
Change-Id: Ifa824dccaff69fd447f45d54cb4a3083bcabdd86
It looks like we dropped running these probably when we moved the
elements around. For testtools to find the test scripts we need to
add the __init__.py files to make the directories look like modules.
Also prevent copying any .pyc or cache files in as hooks.
Change-Id: I66d5f6ee62cc4d9ee14c64e819b4db57d035d09f
This allows nodes with remote devices configured via iBFT to be
correctly used during Ironic introspection and deployment,
at least for non-multipath configurations.
The new element is added as a dependency for ironic-agent.
Change-Id: If3dac6504d26535593f12e851092065b688ef696
install-packages is running before install.d phase, there is a chance
that installing a package like "container-selinux" will failed the
build, moving "selinux-permissive" to run at pre-install stage make
more sense.
Change-Id: I32f988be725d4b385c3765c47a00cd57c53d7d71
Update builds to Fedora 29. Remove the openstack gate CI mirror
workaround for pre-28 versions as they're not building in the gate any
more.
Change-Id: Ia6a8ae8d66d69f6add39e571043328e7274ba26c
8 MiB is not enough when using the grub2 element with centos7 images,
which installs binaries from the shim-x64 and grub2-efi-x64 packages
under /boot/efi. 12 MiB ought to be enough for anybody.
Change-Id: I8594d1fe05242f246a5809740a115ab2f84ac5a3
In order to allow the simple preparation of base images which
can be used for LXC/nspawn machine containers, we add this
element.
Containers inherit a kernel from the host, so there is no need
to build a kernel into the image. All the element needs is a
base init system which, in this case, is systemd.
Change-Id: I45651de2aa1b19bdeee301094f0bdffdd0a3b45c
This finalises the ports of the legacy jobs to zuul native jobs.
The dib-setup-gate-mirrors role preconfigures the repo templates,
etc. for the openstack-ci-mirrors element.
The dib-functests role runs the tests as specified by dib_functests,
and can run under python2 or 3.
Change-Id: Ied67a31f0d31503d13eccad8662c29740c93f33e
I'm not really sure why I originally had --logfile also log to stdout
in I202e1cb200bde17f6d7770cf1e2710bbf4cca64c, but it seem
counter-intuitive (indeed, I just tripped myself up thinking that in a
devstack job "--logfile" would put the logs into a separate file and
avoid the stdout logging, and I wrote it!).
Make it so specifying a --logfile puts dib into quiet mode for stdout.
Explicitly overriding DIB_QUIET will allow both if someone wants that.
Change-Id: I3279c9253eee1c9db69c958b87a0ce73efc0be9b
It looks like epel-release switch from "mirrorlist" to "metalink"
(around release 7-10 Jun 2017 according to [1]). Update our rewrite
matching to handle this "metalink" as well.
Add epel element to the centos7 (image-based) build for testing too
[1] https://koji.fedoraproject.org/koji/buildinfo?buildID=978473
Add epel element so it's tested during the centos7 functional test.
Change-Id: I2d6d4c2ec47bc69d2f16c96b5045b05c435a1af9
While trying to get docker image pre-caching to work we couldn't get a
docker daeomon to run within the chrooted environment. However we got
docker running with the help of bwrap outside of the chrooted
environment. The only option so far for this is the block-device.d
phase. But this has the problem that it runs after the image size has
been calculated. This leads to broken builds if the docker images
being pulled are big.
This can be solved by adding a post-root.d phase that runs outside the
chroot but before the image size calculation.
Change-Id: I36c2a81e2d9f5069f18ce5b0d52c5f1c7212c3ae
This updates diskimage-builder to support current Fedora releases (27
and 28) and removes support for Fedora 26 which is EOL as of June
2018.
Change-Id: I602b22ed4d5397b39dc1eef67964f6fbdcd93060
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
We want to set DIB_EPEL_MIRROR for the epel element, which means we
need access to the CI mirror info script in the chroot. Copy it into
the temp directory with extra-data.d and update the environment file
to find it.
Change-Id: Ia12f0cbdeb194eef3155497ceb5ffc4a452aad76
When using the upstream cloud images with the "ubuntu" element, they
have universe and multiverse enabled which we don't mirror.
To use the infra mirrors as a DIB_DISTRIBUTION_MIRROR with this
element, we need to be able to skip redirecting to universe and
multiverse, and additionally enable insecure repos (as we don't gpg
sign our mirrors).
Add and document two new variables with the ubuntu element to do this.
This is then setup by the openstack-ci-mirrors element so that we use
local mirrors duing dib functional testing for the "ubuntu" element.
Change-Id: I6ffbde07fa0e103641ee5c5f9d9e854e5b2168dc
openssl/cryptography versions are updated/stable
musl profiles need newer versions of open-iscsi masked as upstream
doesn't want to work with multiple libcs
Change-Id: If5baf339516390ae332015928557c6bb734486c2
This is a lot of very low value noise in the logs as these iterate
through all the elements (often doing nothing). Turn it down and add
an echo so we just see what elements it is working on.
Change-Id: I0687de4722766189db9d4a7bd7d3cfb45d387b62
To facilitate this I've created two new environment variables to set
the environment and default options for package actions.
eval is needed for the export as it preserves quotes.
Change-Id: Ib03651ee8dacd48cd1c135afd57cd31101356056
Signed-off-by: Matthew Thode <mthode@mthode.org>
In exploring Gentoo caching, it was realised that we have no way to
bind mount the cache into the finalised image for the finalise.d
phases.
By adding a pre-finalise.d phase that runs outside the chroot, we can
mount outside things into the hierarchy at $TMP_BUILD_DIR/mnt which
are then seen by the in-chroot finalise.d phase.
This is similar to the pre-install phase
Change-Id: I9d782994843383ddf90f62c40498af9925fd9558
Some minor things after looking at these parts.
The dib-run-parts element doesn't do any of the copying any more, so
these comments are wrong.
The reason for the multiple mounts in the bind mount was non-obvious
to modern eyes (as util-linux has handled this for some time).
Formatting fix for the rst
Change-Id: Idb4c9ff32c49aced2c68a5c905bf7a8b2832a5a2
Currently there's more-or-less an assumption that a kernel is
installed, so module blacklists are simply echoed into the modprobe
blacklist. This may not be the case with some ongoing container work.
Although we don't need to blacklist modules for containers, it also
doesn't hurt. Move the debootstrap element to the new modprobe
element, and allow it to create the blacklist directory.
Change-Id: I0f057caf473951df56a2af9633e3a5b53e0809b1
With the check added in commit 7566819139,
diskimage-builder fails to build RPM-based images if kauditd is not
running. However, this is only valid for environments where SELinux is
enabled. If SELinux is disabled (which is identified by an empty _runcon
variable), proceed with running setfiles.
Change-Id: I1b056f20a3a55f7333391207d9e1049d25ece041
Closes-Bug: #1779273
Diskimage-builder fails to build ubuntu-minimal images when run on
a Ubuntu bionic-beaver (18.04) instance.
The user gets "Couldn't create tempfiles for splitting up" when
apt-get update is run in the ubuntu-minimal element root.d.
The issue is that the /tmp inside the chroot is not getting the
proper permissions applied from the base-files package. This is likely
because the pip-cache element has already created the directory before
the base-files package is installed.
This patch changes the order of pip-cache to root.d/11-pip-cache so that
it runs after teh base OS root.d elements run.
Change-Id: I6fd1cb2a23422206884165eb502b260f0c1e52f7
The ubuntu-minimal README states that the latest Ubuntu LTS
is the default, but currently that is not true. This patch
changes the default to the current LTS.
Change-Id: I10f28314d1a5969c20094194637cfe31219d228c
The apt sources are set out in root.d/75-ubuntu-minimal-baseinstall
and the cache is updated, cleaned and a dist-upgrade is done there.
As such, this file is unnecessary.
Change-Id: Idab5ede3f235bc204c4bdebf40fbcf4a12e5bc2f
The ubuntu, and ubuntu-minimal elements both make use of a common set
of environment settings to determine the distribution name.
The ubuntu-minimal element also does a few extra things which would
appear to apply to both sets and bring in extra architecture support.
As such, these are included in the common element.
This intends to be part of a series of patches which will eventually
create a new element to build a minimal ubuntu-systemd-container
element which can be used for lxc/nspawn containers.
Change-Id: Ia4e620f7d3fa6215484a8d218cea2f28bd1ffaee
The grub.cfg has two variables [1]
GRUB_CMDLINE_LINUX : used on all boots
GRUB_CMDLINE_LINUX_DEFAULT : additionally used on all "normal" boots
The problem with I2298675dda1f699c572b3423e7274bc8bd7c1c9d is that it
appened the values in DIB_BOOTLOADER_DEFAULT_CMDLINE to both of these,
resulting in duplicated arguments. I don't think we considered that
GRUB_CMDLINE_LINUX_DEFAULT actually already appends to the
GRUB_CMDLINE_LINUX values.
Make DIB_BOOTLOADER_DEFAULT_CMDLINE only append itself to
GRUB_CMDLINE_LINUX_DEFAULT. That seems to line up sensibly with the
name of the variable.
Documentation is enhanced around this, and a releasenote added.
[1] https://help.ubuntu.com/community/Grub2/Setup
Change-Id: I76b5442a9090c19a6540ed2d4ab324546f241ebf
Closes: #1791736
Without this fix, building a CentOS image on Ubuntu where audit=0 is passed
as a kernel boot parameter will lead to the following error:
disk-image-create centos7 dhcp-all-interfaces cloud-init-nocloud \
devuser yum epel baremetal
... dib-run-parts Running tmpdir/hooks/cleanup.d/99-selinux-fixfiles-restore
... Error connecting to audit system.
Change-Id: I229d9b72f88bffddca42da57f01c27e902427071
Due to the arm naming convention, building centos images for arm64 and
aarch64 does not yield the same result. In order to locate grub2 on
aarch64 the correct mapping is added.
Change-Id: I1bb227b2523e420e394fec8c52c6c79fcdd31c53
Closes-Bug:#1789414
Signed-off-by: Charalampos Kominos <Charalampos.Kominos@enea.com>
This replaces the deprecated (in python 3.2) unittest.TestCase method
assertRaisesRegexp() with assertRaisesRegex().
Change-Id: I0ac482741ad4adc1127dd9e9f64c1c8101f370e5
Signed-off-by: Chuck Short <chucks@redhat.com>
Currently there is a bug, that tries to detach the device from a
partition at the first try, without considering that there may be
other partitions and volumes on it. Ensure that the detach is done
properly, and add a test to ensure that this happens correctly.
Change-Id: I35c5a473509f17a70270a2cbf5bf579faaeb123a
Fixes-Bug: #1777861
A recap -- we run umount phase then cleanup phase.
Currently we register a object to do the final LVM cleanup based on
the parent PV. In light of I697bfbf042816c5ddf170bde9534cc4f0c7279ff,
I believe this should just be done in the cleanup phase. Note there
was probably additional confusion because the partition removal was
done in the cleanup phase until
I7af3c5cf66afd81a481f454b5207af552ad52a32, where is was moved into the
umount phase.
Thus it is moved into the cleanup() function and this should now run,
per the comment, after everything is unmounted in umount phase.
This also exposes that we didn't have the cleanup phase in the unit
tests (because it wasn't doing anything I guess). Add it.
Change-Id: I1c5f4ffc9619c774f78d21b918a81647b3dc28f5
This ensures nouveau is not loaded at boot, which is required when installing
NVIDIA GPU drivers and to avoid issues with PCI passthrough of NVIDIA GPUs.
The option to disable kernel modesets ensures that it can be unloaded again if
it happens to be loaded after boot (e.g manually or implicitly by X).
bp tripleo-vgpu
Change-Id: I60815de86e7b22dfb39555af9d2d53564841e2ab
Related-bug: 1774674
modprobe element currently fails when DIB_MODPROBE_BLACKLIST is not set.
As there are now two methods to control blacklisting this should be optional.
Change-Id: Ibf3c31a95177ba88c1b93228490c7f36f5b70b57
In some cases cache-url can get pulled in without curl, causing it to
fail.
Co-Authored-By: Adam Harwell <flux.adam@gmail.com>
Change-Id: Ibd66c2ca4f8cc423783555d8a99b1184f43adff2
This review squashes:
Iac9afc7766d3640815dc20cfd6de1245d36a09cc
Ie894b5801bd7b3815432882cd626941e89d9f9a1
We need to do this as we can't fix pylint without networkx as that
failes requirements-chak due to us having a cap on networkx and we can't
uncap networkx as part of tripleo-buildimage installs without
constratints which gets us 2.1 and DIB desn't support 2.x
This is the commit message Iac9afc7766d3640815dc20cfd6de1245d36a09cc
---
One of the pylint dependencies has updated to be python3 only; this
version of pylint correctly caps things so it still works with
python2.
This also exposes that we need to uncap networkx due to
I34045f87ca19c2f184b040f4d89347374cce518b. We should remain on
version 1 for now thanks to upper-constraints, but we need to maintain
the lower-constraint.
---
This is the commit message Ie894b5801bd7b3815432882cd626941e89d9f9a1
---
Support different versions of networkx
Since the entry of networkx 2.0 nodes has a different
behaviour. Checking if dg.nodes is iterable is enough to add
compatibility for new/older versions.
---
Change-Id: I82dc61fac6c156a4f0d574290c7632077aa53195
Without this fix, a LVM based ubuntu-minimal image will fail
booting due to the fact that the boot process will not be able
to retrieve the root filesystem using LABEL=(cloud)img-rootfs
Change-Id: If4ecf65868563f7b799160a58af6312bedf956bf
This patch adds an expected semicolon to an end of statement in
dhclient.conf for dhcp-all-interfaces element. Without this fix, an
error occurs when an image is booted with a message,
'semicolon expected.'.
Change-Id: I8311dbc67cc2815223111da01e7a7517c7d6f059
When building with debootstrap, debootstrap will use the key to check
that everything is properly signed. It will not `apt-key add` the key
into the final environment, however.
Early adding the key after debootstrap before we need to read from the
private repo again prevents unsigned issues. This also maintains the
integrity of the packages in the environment throughout the build.
Change-Id: I5ca75ae4620c9fb26b512cb30f8cd79fa7a0373a
Similar to I697bfbf042816c5ddf170bde9534cc4f0c7279ff, the order of
things called is "dib-block-device umount" *then* "dib-block-device
cleanup".
Because we're doing the "kpartx -d" here in cleanup, it means that the
loop-device is removed in umount phase from level0/localloop.py, then
afterwards we try and remove the partitions.
Change-Id: I7af3c5cf66afd81a481f454b5207af552ad52a32
TODO: a test case to ensure the ordering
One call in localloop requires the output of the command, so modify
exec_sudo to buffer up output and return it. This is modelled on the
same thing in package-installs-v2 which seems to work. Rather than
return a subprocess exception, return a dib exception which everything
should have imported anyway.
The overall reason for this is to make our external calls more
consistent for mocking in unit testing.
Change-Id: I10d23b873dee9f775daef2a4c8be5671d02c386e
This element will replace modprobe-blacklist element. It wil
still have the blacklist functionality, but it also adds
the feature of passing a complete file with settings to the
modprobe.d directory. Adding this functionality, that will
allow elements that depends on this module, to just copy the
specified files to the final directory.
Change-Id: I9a44f7d11520b8b1e604956d3c1db2fc7e2bf457
As described in blockdevice.py detachment and (most) resources
release must be done in the umount phase of a block device module.
Until now these jobs were done in the lvm cleanup() phase - which
is too late - especially when using nested LVMs.
This patch moves the functionality of the cleanup() phase to the
umount() phase for the lvm module.
It includes a test case that fails without applying the provided
source code changes.
Change-Id: I697bfbf042816c5ddf170bde9534cc4f0c7279ff
Signed-off-by: Andreas Florath <andreas@florath.net>
Our sgdisk calls are putting extra double-quotes around the names of
partitions. This confuses sfdisk, which confuses growpart, which
confuses growroot ... and you don't get your partition grown for EFI
boot.
Ensure we just bunch arguments into the list directly (for Popen)
rather than string split and have to worry about quoting. Add a check
for this to our GPT unit test, extending it to include a space in the
name of the root partition.
Change-Id: I0a8cb69bb4c9c0865fbaa63ba0d7210028da552e
The existing directories are needed for stage building (a part of the
Gentoo build process). Normally these directories are empty, but there
are times where overrides need to be defined. This commit handles
existing overrides for keywords. For historical reasons the overrides
were able to be put in different files and directories, this
centralizes them.
This also updates the version of openssl/cryptography that works with
or without bindist.
Change-Id: I62c934ed305a711a4a9a3ef01fa55ad142aebb78
This patch adds an element that handles the configuration for
creating a disk capable of being a remote root filesystem through
iSCSI on CentOS images.
Tested on Fujitsu Server and boot with BIOS and UEFI mode successfully.
- Tested Boot-From-Volume + EFI for centos7 with following elements:
"centos7 vm devuser cloud-init-datasources dhcp-all-interfaces
iscsi-boot dracut-regenerate block-device-efi"
Co-authored-By: Nguyen Van Trung <trungnv@vn.fujitsu.com>
Change-Id: Ia1f23d722dced6f254fd7aee86abe8066a72fa42
According to http://bit.ly/2HA4oDO and
the official Ubuntu manual
http://manpages.ubuntu.com/manpages/xenial/man5/interfaces.5.html
source-dir support has been removed from Ubuntu >= 16.04/Xenial
Once an image is generated and booted, moving the dhcp interface(s)
declaration(s) from /etc/network/interfaces into specific subentries
of /etc/network/interfaces.d and calling 'service networking restart'
just make your instance unreachable and all interfaces are left
unconfigured.
This patchset fixes this issue
Change-Id: I6b6b99c81490c874c5db5405c2fbf3c180c87464
When booting on UEFI, there was an issue mounting the vfat
filesystem. It was caused because the mount was defined in
/etc/fstab in lowercase, but the disk had it labeled in upper
case, and system could not find it. Conver the label to upper
case in case of fat/vfat.
Change-Id: Id3dee735e6f8fb221d199c4aba648f3e9a6e4206
When building the image on a non-efi environment, it generates
linux16/initrd16 entries. But to boot from UEFI they need to have
linuxefi/initrdefi entries.
Use sed to replace those entries, in case we have an EFI image.
Change-Id: I47c96450e10f34b91bcc32888532bd7ab87cf316
setfiles isn't supported on the vfat /boot/efi partition. Add it to
the skip list.
Tested on Fujitsu Server successfully.
Change-Id: Iab262c4bdb0ecc25ca6b77ee4aff1ce442c0c578
This patch adds an element that handles the configuration for
creating a disk capable of being a remote root filesystem through
iSCSI on Ubuntu and Debian images.
Change-Id: Ibf9e39d2bdab530106015f156d23d28029d12b0d
Closes-bug: #1716794
When using uefi in rhel, the package mapping is incorrect.
We need to add specific grub-efi* mappings to use grub2-efi
Change-Id: I2db96ae85fd5e4638c794015b2f8164c018420e3
We need to handle openSUSE Leap 15 when installing pip and virtualenv
packages. This fixes the following problem when the pip-and-virtualenv
elements is used:
2018-05-31 09:42:12.014 | + [[ opensuse = opensuse ]]
2018-05-31 09:42:12.014 | /tmp/in_target.d/install.d/04-install-pip: line 57: packages: unbound variable
Change-Id: Id7911b0a0836fa8dcc003e23fa515b78fba67126
Redirecting our output through outfilter.py is inherently a bit racy,
since the disk-image-create process will exit, and then you might get
outfilter.py flushing any remaining output as it closes.
On an interactive prompt this might lead to final output overwriting
the prompt, etc. This can be a bit confusing when you start running
things in a loop.
If we save the original fd, then on the exit path close the redirected
fd's and wait a little bit for final output (as a result of the
close), we get a more consistent output.
Change-Id: I8efe57ab421c1941e99bdecab62c6e21a87e4584
Strip everything before "site-packages" in the output filename for the
PS4 prompt. This makes the line in debug logs significantly shorter
as we don't have the full virtualenv path every single time. The
important thing -- the file being called in the lib/ dir, is retained.
Change-Id: I00706b6f6c0425c7795f997c08ceda3374dc84b5
When switching to using log-file capture, we're getting
[gentoo/build-succeeds] outfile.write(ts_line.encode('utf-8'))
[gentoo/build-succeeds] UnicodeEncodeError: 'utf-8' codec can't
encode character 'udcc5' in position 59: surrogates not allowed
Use surrogateescape [1] on the output to avoid this
[1] https://www.python.org/dev/peps/pep-0383/
Change-Id: I2c2c537296edfa5a8fe661a41bd5bfb3bfcf57e3
Patch allows to rebuild arbitrary images, which location, filename and
sha256sum are specified in variables, not only hardcoded $DIB_RELEASE/current.
Change-Id: I05418932a0c40d885fe00a49f1f49d7e86c67518
Add a bionic test in replacement of trusty. We are already building
bionic images in the gate, so this seems like a good time to switch.
Change-Id: I20d4c25e9b79e7326c86767c36be8615ba0888a3
Removing no longer working and no longer maintained ubuntu-core element, which
intent is unclear, and not documented.
Change-Id: Id847591d04fd7cd32c8903967da01ee0d303b267
Closes-Bug: 1771614
Without this change DIB appends a second command line entry to the GRUB
config. This causes the original command line entry to be ignored
when Linux is booted.
The expected behaviour is that DIB appends to the existing entry as
it does for Ubuntu and SUSE.
Following discussion on the review, this also removes the distro specific
switch statement, as update-grub just calls grub-mkconfig, meaning that
there was nothing distro specific in the first place.
Change-Id: I2298675dda1f699c572b3423e7274bc8bd7c1c9d
Closes-Bug: #1771366
Since CentOS-7.5, a new yum variable is needed for SIG repositories.
This change replicates the %post task of the centos-release package
to setup the contentdir yum var. This should fix issues when
repository url uses $contentdir and yum fail with:
http://mirror.centos.org/%24contentdir ... [Errno 14] HTTP Error 404 - Not Found
For more details see:
https://lists.centos.org/pipermail/centos-devel/2018-March/016542.html
This change also drops support for fedora without dnf.
Change-Id: I1819a48b94670577b0c5e29b24cebfb20ea07d28
This commit addresses the issue described in bug 1768354 when using the
apt-sources element and adding a key to a custom repo, subsequent deb
package installs fail due no update of the repo before package install
Change-Id: I968b3422fab2fb2305426d49215391d8ba7499df
Closes-Bug: 1768354
The pip-and-virtualenv element provides pip inside the created images.
When this pip is used inside the chroot of the image it, by default, creates
a cache directory with the http packages and the resulting wheels.
In the case of the octavia ubuntu-minimal image this cache is using ~50MB of
cache that is not needed after the image finished building.
This package creates a finalise.d task that removes the cache from the
default location.
Change-Id: I4715437b068d04993ef755bd1e27963db1d22417
When RDO projects repository is installed, the python-setuptools
package is obsoleted by python2-setuptools, this makes the install-pip
script failed:
Package python-setuptools-0.9.8-7.el7.noarch is obsoleted by
python2-setuptools-22.0.5-1.el7.noarch which is already installed
Then the "rpm -ql python-setuptools | xargs rm -rf" exit 1. Check if
we have a record of the updated package obsoleting then old one; if
so, use it.
Change-Id: I2b0051bd9e81908c187098a7b82e120b999b111d
We no longer install wget / yum-utils for centos-minimal, this fixes
that.
Change-Id: I8d89026bd48cf7398cc1cbe41e3b7f00f682dbb8
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
This currently breaks glean on rackspace, revert until we can figure
out why that is.
This reverts commit 43bc352c59.
Change-Id: Iae88a3b0457bab0b8f0fd1febf58732ca95e5dc9
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
The release of pip10 has shown up a few issues here
Firstly, pip10 now refuses to overwrite distutils installed packages,
which includes "python-virtualenv" on centos. History has shown us
that we want the packages installed and overwritten, to avoid the
packages coming back and messing things up.
Pre-install all the packages, then list the files in the packages with
"rpm" directly and remove them. This way pip is happy to install.
We need to take better account of the package names for this; on
Fedora things have switch to "python2-virtualenv" instead of
"python-virtualenv" and we can't use an alias to list the package
contents.
This also highlighted that python2-pip is in EPEL for centos, so
enable that when we install it. Make the epel element a no-op for non
centos/rhe distros.
There is a related change in recent fedora that python3 now installs
binaries into /usr/local/bin. There are commented swizzles in here to
ensure we retain the status quo of "pip" and "virtualenv" both being
python2 based, with the python3 versions being called explicitly
"pip3" and "virtualenv3" respectively.
Change-Id: I2ffdd9f615ae6b00428c17249e4f216774991b99
We added this sed in I422490ebe9a9c655552685bc2ff342d288335a9c to
avoid installing python2 packages on python3-only systems and thus
dragging in all of python2.
We made a similar change to python-pip in
I7d8ba9300039cce90965410a4e16ca9e711904c3; however we realised that
the gate (and other consumers) were relying on this element having
installed the python2 & 3 packages for consistency -- otherwise jobs
would install the python-pip packages and overwrite the
pip-from-source and mess everything up. We reverted that in
I419dbdf4682394db68974944af1e5c432f3e0565 and added some clearer notes
that this element brings in python2 & 3, and if you want something
that doesn't do that then this element isn't for you.
However, we never fixed up the virtualenv package install -- currently
our Xenial images have a global virtualenv installed from source, but
the python-virtualenv packages aren't installed. Thus if a job does
"apt-get install python-virtualenv" it overwrites the from-source
virtualenv with older parts and again messes everything up.
Probably most jobs just call "virtualenv" and assume it is there;
however in bringing up some rspec test for puppet I have hit this
issue as some modules specify dependencies on the virtualenv packages.
Thus install the python-virtualenv AND python3-virtualenv packages in
this element.
Change-Id: Ia84c38dc3c40a6080e144b563e10abca7dac2881
The behavior of test -e and [[ -e against broken symlinks is to fail
even if the symlink exists. However we want to test if the link exists
or if there is a file in that location. Therefore switch from test -e to
test -L and test -f to check if the file or link exists regardless of
link target validity.
Change-Id: I84a9b6731eccf950707be50aef464a2de1e33e8e
On initial boot when networking is brought up by cloud-init this
is the timeout that dhclient adheres to. Centos configures
"timeout 300" (for an EC2 bug) in their cloud image, which results
in a 5 minutes delay to boot in cases where no dhcp available (e.g. IPv6
SLAAC). To reduce this boot delay and to provide consistency with
places where we have set other dhcp timeouts set this to DIB_DHCP_TIMEOUT.
Change-Id: I119a002070501c3dfe7c6730b07ee25f422b85b0
Related-Bug: #1758324
systemd-resolved has a new behaviour in bionic, in that if there is no
/etc/resolv.conf file when it installs, it assumes it is a fresh
system and makes /etc/resolf.conf a symlink into its compatability
files.
dib ends up saving & restoring whatever /etc/resolv.conf we have after
the inital chroot creation, which may not be what we want -- in the
above case it restores the system-resolved symlink. For
openstack-infra, we use unbound and want simply "127.0.0.1" in a
/etc/resolv.conf file [1].
Formalise the ability to save specific contents into the final image.
Add documentation, and a note in the code that it's an external
interface.
I would have preferred to namespace the .ORIG file with DIB_ or
similar, but this unofficial interface has already escaped into the
wild. Leave it as is for simplicity.
[1] Note that systemd-resolved will obey /etc/resolv.conf as you would
expect, if file exists.
Change-Id: Ie0e97d8072e2b21a54b053fa6fb07b62960c686d
We exit in several places and don't restore tracing. Previously in
nodepool we relied on the default fallback, which did restore the
tracing. Since we now use the MBR config file, we take the different
exit path without it and the debugging output is incomplete.
Change-Id: I586fc95517926025705ce376ec5c4aaf4122773f
Many elements install additional distribution packages.
In addition the user can provide a set of packages to be installed
via the '-p' switch.
Some of them influence the boot process and therefore the initramfs
needs to be updated. Because the package manager during the image
creation process is configured not to run package scripts, this needs
to be done explicitly.
This issue was found during development and debugging of the
block-device LVM plugin: Even when the e.g. the lvm2 package
was installed in the image, it was missing in the initramfs
because of the missing update.
Change-Id: I7c92033b3ca80cdd23d081002059d83ca3f53bdb
Signed-off-by: Andreas Florath <andreas@florath.net>
Default the GENTOO_PORTAGE_CLEANUP to True. By default we should not
ship package info, this bloats the image and is usually outdated by the
time it'd be consumed.
Change-Id: I14c2530d91807cbc6a3806e01c7e4f6f472b190d
The debian element depends on debian-minimal now which provides
operating-system. This means that the debian element can no longer
provide operating-system and doing so results in an error when using the
debian element.
The fix is simple just rely on the fact that debian-minimal provides
operating-system and remove this element-provides from debian.
Fixes-Bug: 1758000
Change-Id: I524feeb82c19046ec987eb1186c7f4568309e559
The devuser element can set up passwordless sudo, which requiers the
/etc/sudoers.d directory, which requires the sudo package, so we ensure
the sudo package is installed.
Change-Id: I80d6c669d4ac0d97b49d01cb621bf05b8e7f8ef1
There was a typo in I6b819a8071389e7e4eb4874ff7750bd192695ff2 that
modified this default partition type from "0x83" to just 83. We are
now seeing failures relating to this as sfdisk checks for a "disk
manager" when it see Id 0x53 (== 83)
Device Boot Start End Blocks Id System
/dev/vda1 * 2048 26664575 13331264 53 OnTrack DM6 Aux3
Restore to 0x83
Change-Id: Ib43038d2d740fbe01a21a13dd56367f7bc97f869
Hpsum utiltity of proliant-tools requires net-tools to be installed
as part of base image. This commit adds support for installation of
net-tools for all distros.
Change-Id: I2a1e81059ed1aee975db78cfa5e61bbf1b98e06f
Closes-bug: 1751777
When using the package-installs element there can be some encoding
problems if the package installation emits unparsable output
[1]. However in this case we just want to forward the output to the
console which normally can handle this correctly. In order to fix this
switch off universal_newlines processing such that we just operate on
bytes.
Further we have to decode the lines without setting the locale and
ignoring errors. This is required because print encodes without
setting the locale and thus we need to filter/modify the stream such
that it doesn't crash.
[1] Traceback:
2018-03-01 09:58:00.515 | Traceback (most recent call last):
2018-03-01 09:58:00.515 | File "/usr/local/bin/package-installs-v2", line 137, in <module>
2018-03-01 09:58:00.515 | main()
2018-03-01 09:58:00.515 | File "/usr/local/bin/package-installs-v2", line 130, in main
2018-03-01 09:58:00.515 | process_output(install_args, follow=True)
2018-03-01 09:58:00.515 | for line in iter(proc.stdout.readline, ''):
2018-03-01 09:58:00.515 | File "/usr/lib/python3.5/encodings/ascii.py", line 26, in decode
2018-03-01 09:58:00.515 | return codecs.ascii_decode(input, self.errors)[0]
2018-03-01 09:58:00.515 | UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 34: ordinal not in range(128)
Change-Id: Ie4af9b4523459a630cfb98d09093bfe9ef7aa61e
Currently rhel7 image creation fails because it tries to copy
default bootloaders which is ubuntu way. This commit updates `iso`
element to correct the path of bootloaders required for rhel image.
Change-Id: I526d75b2db609fc77be0fc778b4d00f2d3df38ec
Closes-bug: 1750725
For 'satellite' mode of registration, rpm for rhel SSL certificate is
hard coded to 'katello-ca-consumer-latest.noarch.rpm'. This commit adds
functionality that provides an option to set this as defined in their
satellite server.
Change-Id: Ib176cfa209f5ac8a4b5da71419327b4237330904
Closes-Bug: 1749947
Install hwe kernel for ubuntu-minimal. As noted this is currently
Xenial specific; we need this for initial bring-up so let's tackle
future releases as things progress.
Ensure we use ttyAMA0 for arm64 console too.
Change-Id: Ic607cf8369666dc24929aff6f2ef8a72e7980599
In the prior change we added block-device-[mbr|gpt|efi] elements to
create appropriate disk-layouts.
This adds an environment flag to each so the bootloader can install
the right thing. The EFI install path is updated to work with this
(this part a copy of I572937945adbb5adaa5cb09200752e323c2c9531)
We do some basic sanity checking in the block-device elements;
e.g. mbr is not suitable for aarch64, and efi is not suitable for
power.
This updates the bootloader to install EFI where appropriate
Co-Authored-By: Marcin Juszkiewicz <marcin.juszkiewicz@linaro.org>
Change-Id: Ib80acbfd9a12efd976c3fa15a5d1081eb0799305
This moves the block-device default out of the "vm" element and into a
selection of other elements. There's "mbr" which retains the status
quo. There's an EFI version that has the boot/grub partitions as
required. In between there's the GPT only version, which is useful
for architectures like power without EFI, but still want possible
larger disks using GPT.
Change-Id: I4a566a97d073fc0dda0ab2494ac988fe015800a9
The current check only validates that an element that specifies
"element-provides" doesn't conflict with a "real" element. We also
want to check this against the provides of other elements.
A real example is with a "block-device" element. There is no actual
"block-device" element; we can have multiple elements provide it
(block-device-[gpt,mbr,efi], say) but we only want one of them at a
time.
Update the unit test for this.
Change-Id: I59d4aa5f6f09e2892b213e154befa10d85e95ca3
This adds support for a GPT label type to the partitioning code. This
is relatively straight-forward translation of the partition config
into a sgparted command-line and subsequent call.
A unit test is added based on a working GPT/EFI configuration and the
fedora-minimal functional test is updated to build a single-partition
GPT based using the new block-device-gpt override element. See notes
in the sample configuration files about partition requirements and
types.
Documentation has been updated.
Co-Authored-By: Marcin Juszkiewicz <marcin.juszkiewicz@linaro.org>
Change-Id: I6b819a8071389e7e4eb4874ff7750bd192695ff2
In slow networks like Infiniband it takes much time for the
interface to get the carrier. This patch enables this service
to run more then 20 seconds and limited by DIB_DHCP_TIMEOUT.
Change-Id: I8a6015567ac25e37b5a5aba4b1fda71170cc144a
Like we did in https://review.openstack.org/475206 we need to install
systemd sooner because of the new world order of containers.
Change-Id: Ia60d751fee3af6f8d72ad664107acb337360feca
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
As described, we want to set the default label for XFS disks to the
shorter value.
For example, you hit this when setting the old FS_TYPE environment
variable to 'xfs' (which sets the "root-fs-type" parameter, which gets
passed through to 'type'; but does not set a default label).
Change-Id: I41dce6e25766562db4366021309b8c2b74a8ab80
Closes-Bug: 1742170
While Debian-based distros use the label of ppc64el for ppc64 little
endian, Fedora uses ppc64le.
The ironic-agent was doing arch specific package install of lshw over
dmidecode for ppc64 and ppc64el but was attempting to install dmidecode
on Fedora ppc64le which caused the test to fail due to a missing
package.
This change just adds ppc64le to the arch-specific package installation
description for the ironic-agent element.
Change-Id: I38c3c1480bbbb2df817856614e6b740a0c02723a
Closes-Bug: 1744944
This updates diskimage-builder to support current Fedora releases (26
and 27) and removes support for Fedora 25 which is EOL as of December
12, 2017.
Change-Id: I227a607c6c468cc8b7bb154a189e9c8ce2021192
This small change avoids running fstrim on vfat partitions.
The mount order test-case has been updated to also test the mkfs
creation components, and the input config modified to have a vfat
partition to cover this path.
Change-Id: I8952e748d4bdc12a5769706de9057c1e97d95e37
The installed pip can be an older version which does not support
the -c argument. Therefore, upgrade pip before using -c.
Change-Id: If18d8ea822a62c8551c9c4d47354d58b0299fed2
Closes-Bug: 1744403
This reverts commit ab89c7d69c.
This commit checked for DIB_PYTHON_VERSION and only installed the v3
packages. This is unfortunately backwards-incompatible, as consumers
such as the openstack gate are relying on this package installing pip
& virtualenv packages for python2 AND python3.
This was sort-of expressed in the docs, where it discusses what the
resulting setup of the system will be, but I've added a note to make
it clearer.
If we want to change this, I think we'll need either a new element, or
a non-defaulting flag.
Change-Id: I419dbdf4682394db68974944af1e5c432f3e0565
It turns out make has always been a tacit dependency of openssl as it
ships a Makefile for certificates [1]. This just recently changed to
be a hard dependency in F27, so this now fails as openssl is a
dependency of protected packages such as dnf. Since it's always been
wrong to remove it, we take it out of the purge list.
[1] https://bugzilla.redhat.com/show_bug.cgi?id=783446
Change-Id: I69efb3a56878ab97c4587bbbf5356bea752f2846
There's a patch in flight in ironic-python-agent to switch the
default hardware manager to use lshw instead of dmidecode. [0]
This would require lshw to be installed regardless of
architecture. This patch removes the architecture rules from
lshw in the package-installs list.
[0] Ie370331df6bb5ef131c5cb60f458877e2a7ad71a
Change-Id: Idaf05b8efce28cd0cbf339cf693db4f55a693d9b
Partial-Bug: #1715790
zypper only supports the --no-recommends option during installs, giving
the option during removals results in an error.
When setting ACTION=remove, remove --no-recommends from EXTRA_ARGS, and
set --clean-deps to also remove no-longer-needed dependencies.
Rename EXTRA_ARGS to ACTION_ARGS for increased readability.
Change-Id: Ifbd168992b1a20658b6b4a99ba175234f6c78f6d
When "epel" element is used during a build process
with "rhel7" distribution, the build failed
because the "epel-release-7*" package cannot be
installed.
The reason is because the URL is not correct, it
should be:
URL=$BASE_URL/$RELEASE/x86_64/Packages/e/
Change-Id: I90c26892361f7611645b85f2eddc949b2f0d76fc
Closes-Bug: #1735547
At the moment all musl needs in addition to an official stage4 file is a
few keywords and use flag changes.
Change-Id: Ibf4a6d616aca1aef876967e2aa34170c96ac9ef8
This is intended to eventually support building musl-libc based images,
which need the musl overlay.
Change-Id: I8f5429ffa64e74c860772d9a00ff0b7eebb7721a
As described, Fedora 27 has a curl-minimal package that comes in to
satisfy the rpm package dependency. It conflicts with the "real" curl
package -- which is so commonly installed (by infra elements, etc)
that this becomes an annoying problem. Just pre-install the full curl
package.
Fedora 24 is old enough to not worry about, so remove some old
workarounds to make the flow a little simpler.
Change-Id: I67baf96377109ac4521ba00243a0d91b35fafba0
The current implementation - as introduced in
Iee44703297a15b14c715f4bfb7bae67f613aceee - has some shortcomings / bugs,
like:
* the 'grep' check is too sloppy
* when /dev/pts is already mounted multiple times the current implementation
fails:
$ mount | grep devpts | sed 's/.*(\(.*\))/\1/'
rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000
rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000
rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000
* code duplication
* Using the undocumented and non-robust output
of 'mount'.
This patch fixed the above problems.
Change-Id: Ib0c7358772480c56d405659a6a32afd60c311686
Signed-off-by: Andreas Florath <andreas@florath.net>
We oneshot emerge without calculating dependencies a few things to solve
for possible dependency loops.
Python 3.5 also became stable, so don't need to do special things for
it.
Matched the uninstall with the install lines (no need for a full if
statement).
Change-Id: I7c5e546612ac47d659e73a46a52e34d39ca81949
We should always refresh the Tumbleweed repositories and the 'update'
one for Leap in order to always have the latest information from the
repositories.
Change-Id: I85db9d8bb7fa153f01222129e9b36fecc2632f57
This is a continuation for f2cc647dae ("diskimage_builder: lib:
common-functions: Fix options for devpts mount"). We also need to
respect the devpts mount options when the dib elements are mounting
this virtual filesystems themselves.
Change-Id: Iee44703297a15b14c715f4bfb7bae67f613aceee
We want to install python3-pip, not python-pip when we are building a
py3k image less we pull in python2. Once we stop installing python2 we
have to stop calling python2 during pip install.
Change-Id: I7d8ba9300039cce90965410a4e16ca9e711904c3
Currently in Leap 42.x the bootup scripts don't actually make use of
locale.conf yet, so we need to set it in /etc/sysconfig/language. For
future distro compatibility the setting in locale.conf is kept in sync.
Also fix default timezone link.
Change-Id: I59e5dccad8a5ae132d3039851e7aa1db86a609d7
s390x architecture uses zipl as bootloader. When used in combination
with the vm element it replaces the existing bootloader element.
It's mandatory for s390x vm images.
Use cases
---------
* Allow users to create s390x images that run on nova with s390x
libvirt/kvm backend
* Building nodepool images for s390x third party CI
Supported Distros
-----------------
The following listing shows all Distros that officially support
s390x and how those Distros are supported in DIB with this patch.
* SLES - not supported (SLES is not supported in DIB)
* RHEL - not suppoprted (RHEL is not supported as KVM guest on s390x,
therefore there's no rhel7 qcow image for s390x available
like it is for other archictectures)
* Ubuntu - supported
Ubuntu images can for example be built using the following commands:
$ disk-image-create ubuntu-minimal zipl vm
$ disk-image-create ubuntu-minimal zipl
$ disk-image-create ubuntu zipl vm
Testing
-------
Cross architecture building of s390x images is not supported so far.
The plan is to set up a ThirdParty CI that builds the image for s390x and
provides the logs.
Co-Authored-By: Andreas Scheuring <andreas.scheuring@de.ibm.com>
Co-Authored-By: Holger Smolinsky <holger@smolinski.name>
Co-Authored-By: Zhiguo Deng <bjzgdeng@linux.vnet.ibm.com>
Co-Authored-By: Arne Recknagel <arne.recknagel@hotmail.com>
Closes-Bug: #1730641
Change-Id: I576e7edda68da12e97c60af38f457915efe7b934
Commit cebfcf85f9 ("Use -t devpts for
/dev/pts mounts") switched from using '--bind' to '-t devpts' for
mounting the /dev/pts virtual filesystem. However, mounting devpts to
another location also affects the host's /dev/pts mountpoint. Since we
are now mounting devpts without options we end up with the following one
on openSUSE
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,mode=600,ptmxmode=000)
instead of the one we want
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
The missing gid=5 options results to boot problems for virtual machines
So in order to fix that, we need to use the existing devpts options for
/dev/pts so we don't lose them in the new mount.
Change-Id: I17f2c2bb96b807f8dbc07185ae0147bff3230f92
In a couple of places we use flock for critical sections, but we leave
lockfiles around in various locations which can be confusing.
Introduce DIB_LOCKFILES global (under ~/.cache/dib/lockfiles) and
write lockfiles in there.
Fix up removal of the lockfile in the yum path; we just want to make
sure we cleanup the .rpmmacros file, but we don't need to remove the
lockfile as well.
Co-Authored-By: Andreas Florath <andreas@florath.net>
Change-Id: Ie810b2836be521325afe923708d046112e1e1e20
Create a new service, that will be launched after ironic
agent has been exited. This will launch an script that will
take the rescue password, and create the rescue user with
that credentials.
Depends-On: I7898ff22800dedba73d7fbfb3801378867abe183
Change-Id: Ic3a241e2789a122d3d966e7e2148306fd0cf6aed
Partial-Bug: 1526449
Currently a bind is used when mounting /dev/pts in chroot.
This leads to problems - especially when running DIB in parallel:
It was observed that the /dev/pts mount vanishes from the host
system.
This patch uses '-t devpts' - as it is done for /sys and /proc -
for handling /dev/pts.
Change-Id: Id7775ae6fca6502af800e7b73a00862ef320206b
Signed-off-by: Andreas Florath <andreas@florath.net>
On ubuntu we detect that in python3 we need to install
python3-virtualenv, but append this to the packages to install rather
than replace python-virtualenv which results in both being installed
(and therefore grabbing python2).
Change-Id: I422490ebe9a9c655552685bc2ff342d288335a9c
Closes-Bug: #1724656
This patch removes the unneeded dd calls in the lvm block device
plugin.
After removing the underlying block device, there is the need to call
'pvscan --cache'. This is done by a dedicated LVM cleanup node which
is cleaned up after the the underlying block device.
Change-Id: Id8eaede77fbdc107d2ba1035cd6b8eb5c10160c3
Signed-off-by: Andreas Florath <andreas@florath.net>
There have been a few changes over the past few months, here we make the
following changes.
* change from backtrack=99 to complete-graph as a more correct flag
* make python version selection more in line with what gentoo supports
* set up python before stuff gets pip installed
* ensure we have the proper pip so we can install pip packages as root
* ensure we have the proper use flags for the disk formatting changes
* set DIB_RELEASE like other distros
* fix openssh-server element for gentoo
Change-Id: I17202de3016616ce34c8cbead7d0fb047a64e96b
This commits make update to ssacli version to point to latest
ssacli release that has support for HPE P/E-Class SR Gen10 controllers.
Change-Id: Ia9a0eaec78d601f56b4036e57601554b87f21acc
Closes-Bug: 1721185
The call to fstrim in disk-image-create is currently useless, because
at the time this is called, the file systems were already umounted by
the block device layer.
The current implementation of the block-device mount plugin does not
call fstrim at all: resulting in larger image sizes.
This patch removes the useless fstrim call from the disk-image-create
script and moves this into the block-device mount.py.
The resulting image might be much smaller. Example: Ubuntu Xenial
with some elements; once with and once without this patch:
-rw-r--r-- 1 dib dib 475661824 Sep 16 06:43 ubuntu-xenial-without-fstrim.qcow2
-rw-r--r-- 1 dib dib 364249088 Sep 16 09:30 ubuntu-xenial-with-fstrim.qcow2
Change-Id: I4e21ae50c5e6e26dc9f50f004ed6413132c81047
Signed-off-by: Andreas Florath <andreas@florath.net>
This reverts commit a47ff0dd4a.
Since this merged, a global-requirements pin to keep networkx <2.0 has
also merged. The plan is:
1. revert our 2.0 support and
1a. take the <2.0 pin from global requirements
2. figure out how to use constraints properly in our testing
3. restore this, with a depends-on for a 2.0 bump in requirements
(which will self-test, see 3.)
4. when other projects are ready for a global 2.0 bump, merge
in a controlled fashion
This reverts the 2.0 support, and adds the pin for networkx <2.0
Change-Id: I18f6a1115da779581245e3dd423fd90516974a33
Networkx 2.0 released recently. The main difference for us is that
"node" is no longer a dictionary and should be accessed via "nodes",
and the topological_sort returns an interator
Closes-Bug: 1712693
Change-Id: I78e89f2261b8b8d28c68b517c1e61691ab40016c
A small update was made to 4.4.0-96.119 that dropped the
initramfs-tools dependency from the kernel [1]. This had the
unfortunate affect of removing the initramfs from ubuntu-minimal and
making it unbootable, since we specify the root device via LABEL=.
Add the package explicitly alongside the kernel.
Also, small fix to pass unit tests
[1] https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/1700972
Change-Id: I57a0f08cd5e082ecdf8dba0ab34fb3062c50836d
We intended to do an in-place sort of the mount-point list, but
sorted() returns a new list that wasn't captured. Move to the .sort()
function.
It seems the existing unit-test missed this. Add a new test taken
from the bug which does exhibit a sorting issue. Also added a
unit-test of just the comparitor for sanity.
Closes-Bug: 1699437
Change-Id: I8101e4a1804a4af7dbda20d48bf362c3f4ad2742
This commit adds change in 'proliant-tools' element to
install a package 'unzip' which is required to perform
SUM based firmware update for HPE Proliant servers.
Change-Id: Ib8f6d18402439edd93d100cc7a4fb2094c863715
As described in the comment, we need to create the /etc/machine-id for
the image-based build when systemd isn't updated (as is usually the
case for a new distro)
Work on clearing this out continues, but this brings it to parity with
fedora-minimal.
Change-Id: Icbbbabb4114d4d95909648d8e39a6bae6d2a7b7b
Depends-On: I761e425f8a658669d9b8a70ce4260cec263ea51a
The URL we are using seems to have disappeared. Update this to
download.fedoraproject.org. The new URL requires a "subrelease" now,
add it, along with a note on where it comes from.
Change-Id: I761e425f8a658669d9b8a70ce4260cec263ea51a
This element was assuming that yaml was included as package,
but there are systems not including it. So properly add yaml
as a dependency.
Change-Id: I72da2776674a3963657052b9a9715abcb4fab1e2
Partially-Fixes-Bug: #1715686
When using combined with rhel7 image, the unregister of repos
has already happened, because it is executed under 60- ordering.
As dracut-regenerate may need to install extra packages for it,
it causes this step to fail, because it cannot find repos where
to pull the packages from.
Change-Id: I35e37df7990ad76a5004cb90fdd863ec743a5483
Per the bug report, these seem to be causing issues with maintaining
file capabilities. They aren't necessary so let's just remove them.
Change-Id: I06c90fdc85655986142b936cadbe04d75dd27427
Closes-Bug: 1714604
Avoid incorrect use of [ with =~ matching
I guess this doesn't trip "-e" because it's in an if-conditional. I'm
looking at making bashate detect this; maybe we can run bashate over
things we know are scripts
Change-Id: Ia3fe2b978fae5bdaadbb1789058180d3ad950d00
In Ubuntu/Debian, the default dependencies cannot be relied
upon as we enter into a cyclical dependency relationship which
prevents the unit from starting.
Added the required configuration to the systemd unit file.
This issue has also been observed in glean[0], which has a nearly
identical unit file for interface start-up.
[0]: https://review.openstack.org/#/c/485748
Closes-Bug: #1708685
Change-Id: I23ac9510d1a21c7073bd33f76ba66fa04a8be035
This provides a basic LVM support to dib-block-device.
Co-Authored-By: Ian Wienand <iwienand@redhat.com>
Change-Id: Ibd624d9f95ee68b20a15891f639ddd5b3188cdf9
Under certain environments, this timeout was causing failures
because it was too short. Increasing to 10, to give time to
perform the specified tasks.
Change-Id: I01dd3553f38e1137b2fcb04b4ee12202be3ad1a8
Many programs rely upon /etc/protocols to be present
however the default debian image that is generated lacks
/etc/protocols. This is observable when building an image
for use with ironic via the ironic-agent element, since
the IPA agent fails to start as python needs /etc/protocols
to open a socket connection.
Added to debian-minimal as it is inherited into the debian
element.
Change-Id: Icc81635870961943707cf6b3f61a9ddbd51cb8fd
Closes-Bug: #1708531
There is some confusion in the readme's over what is happening. The
original change (Iaf46c8e61bf1cac9a096cbfd75d6d6a9111b701e) split out
debian-minimal and made debian "... simply be a collection of the
extra things we do to make it look like a cloud-init based cloud
image"
Make this clearer in the documentation
Change-Id: Ibe6fad9c67b70a5e31e43e06419968135174fef3
Deploying many nodes with the generated image shouldn't have the same
/etc/machine-id so clearing it and letting systemd generate a new
id upon first boot seems to be the best way to achieve this.
Change-Id: I73d0577d31464521b3989312fd9d982a1312a268
Closes-bug: 1707526
Closes-bug: 1672461
Fedora 26 is now the latest release:
https://fedoraproject.org/wiki/Releases/26/Schedule
We are building and using these in infra now
Change-Id: I012c2d28255be274e88abc2751d968bafaf76fbb
Depends-On: Ieba5f69020a13681074f72cfca2955071801b63a
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
Change I008f8bbc9c8414ce948c601e3907e27764e15a52 has shown that we
build redhat images without the "semange" tool available, which comes
from the policycoreutils-python package (see also
I3f9e2c322d042a5dddba33451c0fc21a4d32a88a).
I403e7806ae10d5dd96d0727832f4da20e34b94c7 added some of the selinux
libraries to yum-minimal for ansible support, but not to others.
Given both these changes, it seems that selinux[-targeted],
libselinux[-python] and policycoreutils[-python] can reasonably
considered part of all base images. Move the selinux related packages
into redhat-common.
This also adds it explicitly to install_test_deps.sh. It was actually
being dragged in by the docker install, but is a required component
for building (should be in bindep, but not there with that yet).
Change-Id: Idd4ae71ee6deee84604823b6b5dc4a845f316e01
Related-Bug: #1707788
The MBR Partition Table Entry (PTE) allows one to specify many
possible partition types and one of the benefits of this is being able
to specify the CHS variant or the LBA variant.
By default, LBA only creates partitions of type 0x83 (of course,
that's only because the documentation doesn't tell you how to make it
do anything else).
I will take up Ian's suggestion in patch set 2 for a more rigorous
test in an independent patch set.
Change-Id: If3068535980eac2e58d4025444c65147a8c7fedc
Closes-Bug:#1703352
Currently, the cleanup script is using existence of
semanage binary to check if selinux is enabled. However
this is misleading and can lead to problems when selinux
is disabled in a system where the binary exist.
This patch changes the detection logic to use /sys/fs/selinux
directory which is a in-memory filesystem created only when
selinux is really enabled.
Change-Id: I008f8bbc9c8414ce948c601e3907e27764e15a52
Related-Bug: 1706386
