As described inline, we only want to remove the system package files
on centos; it causes problems on Fedora where some system tools expect
these to be there.
But there is an additional bug -- pip actually removes the system
package files anyway. To work around this, reinstall the system
package.
Closes-Bug: #1813232
Change-Id: I041a141366099093805e6052b1bbf64efd277e1e
As described in the comments, it seems the transition between
dbus-daemon -> dbus-broker in Fedora 29 has made it so the packages
can get into a state where neither service is enabled.
Explicitly install and enable dbus-broker for F29
Change-Id: I06753043a75be2f635653899c6c251b9fbdd7c67
There is an use of get_image_element_array on the environment.d
phase, for the iscsi-boot element.
This function is not available on that step. So moving the check
at next step, extra-data-d, where it is available.
Change-Id: I89cfe565492142c2f7962109360fcbcebadfd469
This plumbs through an "--use-nm" flag to glean which instructs it to
setup interface bringup with NetworkManager rather than legacy network
enablement scripts.
In this case, install the NetworkManager package. In the non-nm case,
also install the network-scripts for Fedora 29 -- this has stopped
being installed by default (it's been deprecated since forever).
As noted in the docs, this is currently really only relevant on the
supported rpm distros which are using the ifcfg-rh NetworkManager
plugin to effectively re-use old config files. However,
NetworkManager has similar plugins for other platforms, so support can
be expanded if changes are proposed.
Depends-On: https://review.openstack.org/618964
Change-Id: I4d76e88ce25e5675fd5ef48924acd09915a62a4b
Provide a "when" option that provides for not installing packages
based on a = or != match on an environment variable.
Unit tests are added.
Change-Id: Ifa824dccaff69fd447f45d54cb4a3083bcabdd86
It looks like we dropped running these probably when we moved the
elements around. For testtools to find the test scripts we need to
add the __init__.py files to make the directories look like modules.
Also prevent copying any .pyc or cache files in as hooks.
Change-Id: I66d5f6ee62cc4d9ee14c64e819b4db57d035d09f
This allows nodes with remote devices configured via iBFT to be
correctly used during Ironic introspection and deployment,
at least for non-multipath configurations.
The new element is added as a dependency for ironic-agent.
Change-Id: If3dac6504d26535593f12e851092065b688ef696
install-packages is running before install.d phase, there is a chance
that installing a package like "container-selinux" will failed the
build, moving "selinux-permissive" to run at pre-install stage make
more sense.
Change-Id: I32f988be725d4b385c3765c47a00cd57c53d7d71
Update builds to Fedora 29. Remove the openstack gate CI mirror
workaround for pre-28 versions as they're not building in the gate any
more.
Change-Id: Ia6a8ae8d66d69f6add39e571043328e7274ba26c
8 MiB is not enough when using the grub2 element with centos7 images,
which installs binaries from the shim-x64 and grub2-efi-x64 packages
under /boot/efi. 12 MiB ought to be enough for anybody.
Change-Id: I8594d1fe05242f246a5809740a115ab2f84ac5a3
In order to allow the simple preparation of base images which
can be used for LXC/nspawn machine containers, we add this
element.
Containers inherit a kernel from the host, so there is no need
to build a kernel into the image. All the element needs is a
base init system which, in this case, is systemd.
Change-Id: I45651de2aa1b19bdeee301094f0bdffdd0a3b45c
This finalises the ports of the legacy jobs to zuul native jobs.
The dib-setup-gate-mirrors role preconfigures the repo templates,
etc. for the openstack-ci-mirrors element.
The dib-functests role runs the tests as specified by dib_functests,
and can run under python2 or 3.
Change-Id: Ied67a31f0d31503d13eccad8662c29740c93f33e
It looks like epel-release switch from "mirrorlist" to "metalink"
(around release 7-10 Jun 2017 according to [1]). Update our rewrite
matching to handle this "metalink" as well.
Add epel element to the centos7 (image-based) build for testing too
[1] https://koji.fedoraproject.org/koji/buildinfo?buildID=978473
Add epel element so it's tested during the centos7 functional test.
Change-Id: I2d6d4c2ec47bc69d2f16c96b5045b05c435a1af9
This updates diskimage-builder to support current Fedora releases (27
and 28) and removes support for Fedora 26 which is EOL as of June
2018.
Change-Id: I602b22ed4d5397b39dc1eef67964f6fbdcd93060
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
We want to set DIB_EPEL_MIRROR for the epel element, which means we
need access to the CI mirror info script in the chroot. Copy it into
the temp directory with extra-data.d and update the environment file
to find it.
Change-Id: Ia12f0cbdeb194eef3155497ceb5ffc4a452aad76
When using the upstream cloud images with the "ubuntu" element, they
have universe and multiverse enabled which we don't mirror.
To use the infra mirrors as a DIB_DISTRIBUTION_MIRROR with this
element, we need to be able to skip redirecting to universe and
multiverse, and additionally enable insecure repos (as we don't gpg
sign our mirrors).
Add and document two new variables with the ubuntu element to do this.
This is then setup by the openstack-ci-mirrors element so that we use
local mirrors duing dib functional testing for the "ubuntu" element.
Change-Id: I6ffbde07fa0e103641ee5c5f9d9e854e5b2168dc
openssl/cryptography versions are updated/stable
musl profiles need newer versions of open-iscsi masked as upstream
doesn't want to work with multiple libcs
Change-Id: If5baf339516390ae332015928557c6bb734486c2
This is a lot of very low value noise in the logs as these iterate
through all the elements (often doing nothing). Turn it down and add
an echo so we just see what elements it is working on.
Change-Id: I0687de4722766189db9d4a7bd7d3cfb45d387b62
To facilitate this I've created two new environment variables to set
the environment and default options for package actions.
eval is needed for the export as it preserves quotes.
Change-Id: Ib03651ee8dacd48cd1c135afd57cd31101356056
Signed-off-by: Matthew Thode <mthode@mthode.org>
Currently there's more-or-less an assumption that a kernel is
installed, so module blacklists are simply echoed into the modprobe
blacklist. This may not be the case with some ongoing container work.
Although we don't need to blacklist modules for containers, it also
doesn't hurt. Move the debootstrap element to the new modprobe
element, and allow it to create the blacklist directory.
Change-Id: I0f057caf473951df56a2af9633e3a5b53e0809b1
With the check added in commit 7566819139,
diskimage-builder fails to build RPM-based images if kauditd is not
running. However, this is only valid for environments where SELinux is
enabled. If SELinux is disabled (which is identified by an empty _runcon
variable), proceed with running setfiles.
Change-Id: I1b056f20a3a55f7333391207d9e1049d25ece041
Closes-Bug: #1779273
Diskimage-builder fails to build ubuntu-minimal images when run on
a Ubuntu bionic-beaver (18.04) instance.
The user gets "Couldn't create tempfiles for splitting up" when
apt-get update is run in the ubuntu-minimal element root.d.
The issue is that the /tmp inside the chroot is not getting the
proper permissions applied from the base-files package. This is likely
because the pip-cache element has already created the directory before
the base-files package is installed.
This patch changes the order of pip-cache to root.d/11-pip-cache so that
it runs after teh base OS root.d elements run.
Change-Id: I6fd1cb2a23422206884165eb502b260f0c1e52f7
The ubuntu-minimal README states that the latest Ubuntu LTS
is the default, but currently that is not true. This patch
changes the default to the current LTS.
Change-Id: I10f28314d1a5969c20094194637cfe31219d228c
The apt sources are set out in root.d/75-ubuntu-minimal-baseinstall
and the cache is updated, cleaned and a dist-upgrade is done there.
As such, this file is unnecessary.
Change-Id: Idab5ede3f235bc204c4bdebf40fbcf4a12e5bc2f
The ubuntu, and ubuntu-minimal elements both make use of a common set
of environment settings to determine the distribution name.
The ubuntu-minimal element also does a few extra things which would
appear to apply to both sets and bring in extra architecture support.
As such, these are included in the common element.
This intends to be part of a series of patches which will eventually
create a new element to build a minimal ubuntu-systemd-container
element which can be used for lxc/nspawn containers.
Change-Id: Ia4e620f7d3fa6215484a8d218cea2f28bd1ffaee
The grub.cfg has two variables [1]
GRUB_CMDLINE_LINUX : used on all boots
GRUB_CMDLINE_LINUX_DEFAULT : additionally used on all "normal" boots
The problem with I2298675dda1f699c572b3423e7274bc8bd7c1c9d is that it
appened the values in DIB_BOOTLOADER_DEFAULT_CMDLINE to both of these,
resulting in duplicated arguments. I don't think we considered that
GRUB_CMDLINE_LINUX_DEFAULT actually already appends to the
GRUB_CMDLINE_LINUX values.
Make DIB_BOOTLOADER_DEFAULT_CMDLINE only append itself to
GRUB_CMDLINE_LINUX_DEFAULT. That seems to line up sensibly with the
name of the variable.
Documentation is enhanced around this, and a releasenote added.
[1] https://help.ubuntu.com/community/Grub2/Setup
Change-Id: I76b5442a9090c19a6540ed2d4ab324546f241ebf
Closes: #1791736
Without this fix, building a CentOS image on Ubuntu where audit=0 is passed
as a kernel boot parameter will lead to the following error:
disk-image-create centos7 dhcp-all-interfaces cloud-init-nocloud \
devuser yum epel baremetal
... dib-run-parts Running tmpdir/hooks/cleanup.d/99-selinux-fixfiles-restore
... Error connecting to audit system.
Change-Id: I229d9b72f88bffddca42da57f01c27e902427071
Due to the arm naming convention, building centos images for arm64 and
aarch64 does not yield the same result. In order to locate grub2 on
aarch64 the correct mapping is added.
Change-Id: I1bb227b2523e420e394fec8c52c6c79fcdd31c53
Closes-Bug:#1789414
Signed-off-by: Charalampos Kominos <Charalampos.Kominos@enea.com>
This ensures nouveau is not loaded at boot, which is required when installing
NVIDIA GPU drivers and to avoid issues with PCI passthrough of NVIDIA GPUs.
The option to disable kernel modesets ensures that it can be unloaded again if
it happens to be loaded after boot (e.g manually or implicitly by X).
bp tripleo-vgpu
Change-Id: I60815de86e7b22dfb39555af9d2d53564841e2ab
Related-bug: 1774674
modprobe element currently fails when DIB_MODPROBE_BLACKLIST is not set.
As there are now two methods to control blacklisting this should be optional.
Change-Id: Ibf3c31a95177ba88c1b93228490c7f36f5b70b57
In some cases cache-url can get pulled in without curl, causing it to
fail.
Co-Authored-By: Adam Harwell <flux.adam@gmail.com>
Change-Id: Ibd66c2ca4f8cc423783555d8a99b1184f43adff2
Without this fix, a LVM based ubuntu-minimal image will fail
booting due to the fact that the boot process will not be able
to retrieve the root filesystem using LABEL=(cloud)img-rootfs
Change-Id: If4ecf65868563f7b799160a58af6312bedf956bf
This patch adds an expected semicolon to an end of statement in
dhclient.conf for dhcp-all-interfaces element. Without this fix, an
error occurs when an image is booted with a message,
'semicolon expected.'.
Change-Id: I8311dbc67cc2815223111da01e7a7517c7d6f059
When building with debootstrap, debootstrap will use the key to check
that everything is properly signed. It will not `apt-key add` the key
into the final environment, however.
Early adding the key after debootstrap before we need to read from the
private repo again prevents unsigned issues. This also maintains the
integrity of the packages in the environment throughout the build.
Change-Id: I5ca75ae4620c9fb26b512cb30f8cd79fa7a0373a
This element will replace modprobe-blacklist element. It wil
still have the blacklist functionality, but it also adds
the feature of passing a complete file with settings to the
modprobe.d directory. Adding this functionality, that will
allow elements that depends on this module, to just copy the
specified files to the final directory.
Change-Id: I9a44f7d11520b8b1e604956d3c1db2fc7e2bf457
The existing directories are needed for stage building (a part of the
Gentoo build process). Normally these directories are empty, but there
are times where overrides need to be defined. This commit handles
existing overrides for keywords. For historical reasons the overrides
were able to be put in different files and directories, this
centralizes them.
This also updates the version of openssl/cryptography that works with
or without bindist.
Change-Id: I62c934ed305a711a4a9a3ef01fa55ad142aebb78
This patch adds an element that handles the configuration for
creating a disk capable of being a remote root filesystem through
iSCSI on CentOS images.
Tested on Fujitsu Server and boot with BIOS and UEFI mode successfully.
- Tested Boot-From-Volume + EFI for centos7 with following elements:
"centos7 vm devuser cloud-init-datasources dhcp-all-interfaces
iscsi-boot dracut-regenerate block-device-efi"
Co-authored-By: Nguyen Van Trung <trungnv@vn.fujitsu.com>
Change-Id: Ia1f23d722dced6f254fd7aee86abe8066a72fa42
According to http://bit.ly/2HA4oDO and
the official Ubuntu manual
http://manpages.ubuntu.com/manpages/xenial/man5/interfaces.5.html
source-dir support has been removed from Ubuntu >= 16.04/Xenial
Once an image is generated and booted, moving the dhcp interface(s)
declaration(s) from /etc/network/interfaces into specific subentries
of /etc/network/interfaces.d and calling 'service networking restart'
just make your instance unreachable and all interfaces are left
unconfigured.
This patchset fixes this issue
Change-Id: I6b6b99c81490c874c5db5405c2fbf3c180c87464
When building the image on a non-efi environment, it generates
linux16/initrd16 entries. But to boot from UEFI they need to have
linuxefi/initrdefi entries.
Use sed to replace those entries, in case we have an EFI image.
Change-Id: I47c96450e10f34b91bcc32888532bd7ab87cf316
setfiles isn't supported on the vfat /boot/efi partition. Add it to
the skip list.
Tested on Fujitsu Server successfully.
Change-Id: Iab262c4bdb0ecc25ca6b77ee4aff1ce442c0c578
This patch adds an element that handles the configuration for
creating a disk capable of being a remote root filesystem through
iSCSI on Ubuntu and Debian images.
Change-Id: Ibf9e39d2bdab530106015f156d23d28029d12b0d
Closes-bug: #1716794
When using uefi in rhel, the package mapping is incorrect.
We need to add specific grub-efi* mappings to use grub2-efi
Change-Id: I2db96ae85fd5e4638c794015b2f8164c018420e3
We need to handle openSUSE Leap 15 when installing pip and virtualenv
packages. This fixes the following problem when the pip-and-virtualenv
elements is used:
2018-05-31 09:42:12.014 | + [[ opensuse = opensuse ]]
2018-05-31 09:42:12.014 | /tmp/in_target.d/install.d/04-install-pip: line 57: packages: unbound variable
Change-Id: Id7911b0a0836fa8dcc003e23fa515b78fba67126
Patch allows to rebuild arbitrary images, which location, filename and
sha256sum are specified in variables, not only hardcoded $DIB_RELEASE/current.
Change-Id: I05418932a0c40d885fe00a49f1f49d7e86c67518
Add a bionic test in replacement of trusty. We are already building
bionic images in the gate, so this seems like a good time to switch.
Change-Id: I20d4c25e9b79e7326c86767c36be8615ba0888a3
Removing no longer working and no longer maintained ubuntu-core element, which
intent is unclear, and not documented.
Change-Id: Id847591d04fd7cd32c8903967da01ee0d303b267
Closes-Bug: 1771614
Without this change DIB appends a second command line entry to the GRUB
config. This causes the original command line entry to be ignored
when Linux is booted.
The expected behaviour is that DIB appends to the existing entry as
it does for Ubuntu and SUSE.
Following discussion on the review, this also removes the distro specific
switch statement, as update-grub just calls grub-mkconfig, meaning that
there was nothing distro specific in the first place.
Change-Id: I2298675dda1f699c572b3423e7274bc8bd7c1c9d
Closes-Bug: #1771366
Since CentOS-7.5, a new yum variable is needed for SIG repositories.
This change replicates the %post task of the centos-release package
to setup the contentdir yum var. This should fix issues when
repository url uses $contentdir and yum fail with:
http://mirror.centos.org/%24contentdir ... [Errno 14] HTTP Error 404 - Not Found
For more details see:
https://lists.centos.org/pipermail/centos-devel/2018-March/016542.html
This change also drops support for fedora without dnf.
Change-Id: I1819a48b94670577b0c5e29b24cebfb20ea07d28
This commit addresses the issue described in bug 1768354 when using the
apt-sources element and adding a key to a custom repo, subsequent deb
package installs fail due no update of the repo before package install
Change-Id: I968b3422fab2fb2305426d49215391d8ba7499df
Closes-Bug: 1768354
The pip-and-virtualenv element provides pip inside the created images.
When this pip is used inside the chroot of the image it, by default, creates
a cache directory with the http packages and the resulting wheels.
In the case of the octavia ubuntu-minimal image this cache is using ~50MB of
cache that is not needed after the image finished building.
This package creates a finalise.d task that removes the cache from the
default location.
Change-Id: I4715437b068d04993ef755bd1e27963db1d22417
When RDO projects repository is installed, the python-setuptools
package is obsoleted by python2-setuptools, this makes the install-pip
script failed:
Package python-setuptools-0.9.8-7.el7.noarch is obsoleted by
python2-setuptools-22.0.5-1.el7.noarch which is already installed
Then the "rpm -ql python-setuptools | xargs rm -rf" exit 1. Check if
we have a record of the updated package obsoleting then old one; if
so, use it.
Change-Id: I2b0051bd9e81908c187098a7b82e120b999b111d
We no longer install wget / yum-utils for centos-minimal, this fixes
that.
Change-Id: I8d89026bd48cf7398cc1cbe41e3b7f00f682dbb8
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
This currently breaks glean on rackspace, revert until we can figure
out why that is.
This reverts commit 43bc352c59.
Change-Id: Iae88a3b0457bab0b8f0fd1febf58732ca95e5dc9
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
The release of pip10 has shown up a few issues here
Firstly, pip10 now refuses to overwrite distutils installed packages,
which includes "python-virtualenv" on centos. History has shown us
that we want the packages installed and overwritten, to avoid the
packages coming back and messing things up.
Pre-install all the packages, then list the files in the packages with
"rpm" directly and remove them. This way pip is happy to install.
We need to take better account of the package names for this; on
Fedora things have switch to "python2-virtualenv" instead of
"python-virtualenv" and we can't use an alias to list the package
contents.
This also highlighted that python2-pip is in EPEL for centos, so
enable that when we install it. Make the epel element a no-op for non
centos/rhe distros.
There is a related change in recent fedora that python3 now installs
binaries into /usr/local/bin. There are commented swizzles in here to
ensure we retain the status quo of "pip" and "virtualenv" both being
python2 based, with the python3 versions being called explicitly
"pip3" and "virtualenv3" respectively.
Change-Id: I2ffdd9f615ae6b00428c17249e4f216774991b99
We added this sed in I422490ebe9a9c655552685bc2ff342d288335a9c to
avoid installing python2 packages on python3-only systems and thus
dragging in all of python2.
We made a similar change to python-pip in
I7d8ba9300039cce90965410a4e16ca9e711904c3; however we realised that
the gate (and other consumers) were relying on this element having
installed the python2 & 3 packages for consistency -- otherwise jobs
would install the python-pip packages and overwrite the
pip-from-source and mess everything up. We reverted that in
I419dbdf4682394db68974944af1e5c432f3e0565 and added some clearer notes
that this element brings in python2 & 3, and if you want something
that doesn't do that then this element isn't for you.
However, we never fixed up the virtualenv package install -- currently
our Xenial images have a global virtualenv installed from source, but
the python-virtualenv packages aren't installed. Thus if a job does
"apt-get install python-virtualenv" it overwrites the from-source
virtualenv with older parts and again messes everything up.
Probably most jobs just call "virtualenv" and assume it is there;
however in bringing up some rspec test for puppet I have hit this
issue as some modules specify dependencies on the virtualenv packages.
Thus install the python-virtualenv AND python3-virtualenv packages in
this element.
Change-Id: Ia84c38dc3c40a6080e144b563e10abca7dac2881
On initial boot when networking is brought up by cloud-init this
is the timeout that dhclient adheres to. Centos configures
"timeout 300" (for an EC2 bug) in their cloud image, which results
in a 5 minutes delay to boot in cases where no dhcp available (e.g. IPv6
SLAAC). To reduce this boot delay and to provide consistency with
places where we have set other dhcp timeouts set this to DIB_DHCP_TIMEOUT.
Change-Id: I119a002070501c3dfe7c6730b07ee25f422b85b0
Related-Bug: #1758324
Many elements install additional distribution packages.
In addition the user can provide a set of packages to be installed
via the '-p' switch.
Some of them influence the boot process and therefore the initramfs
needs to be updated. Because the package manager during the image
creation process is configured not to run package scripts, this needs
to be done explicitly.
This issue was found during development and debugging of the
block-device LVM plugin: Even when the e.g. the lvm2 package
was installed in the image, it was missing in the initramfs
because of the missing update.
Change-Id: I7c92033b3ca80cdd23d081002059d83ca3f53bdb
Signed-off-by: Andreas Florath <andreas@florath.net>
Default the GENTOO_PORTAGE_CLEANUP to True. By default we should not
ship package info, this bloats the image and is usually outdated by the
time it'd be consumed.
Change-Id: I14c2530d91807cbc6a3806e01c7e4f6f472b190d
The debian element depends on debian-minimal now which provides
operating-system. This means that the debian element can no longer
provide operating-system and doing so results in an error when using the
debian element.
The fix is simple just rely on the fact that debian-minimal provides
operating-system and remove this element-provides from debian.
Fixes-Bug: 1758000
Change-Id: I524feeb82c19046ec987eb1186c7f4568309e559
The devuser element can set up passwordless sudo, which requiers the
/etc/sudoers.d directory, which requires the sudo package, so we ensure
the sudo package is installed.
Change-Id: I80d6c669d4ac0d97b49d01cb621bf05b8e7f8ef1
Hpsum utiltity of proliant-tools requires net-tools to be installed
as part of base image. This commit adds support for installation of
net-tools for all distros.
Change-Id: I2a1e81059ed1aee975db78cfa5e61bbf1b98e06f
Closes-bug: 1751777
When using the package-installs element there can be some encoding
problems if the package installation emits unparsable output
[1]. However in this case we just want to forward the output to the
console which normally can handle this correctly. In order to fix this
switch off universal_newlines processing such that we just operate on
bytes.
Further we have to decode the lines without setting the locale and
ignoring errors. This is required because print encodes without
setting the locale and thus we need to filter/modify the stream such
that it doesn't crash.
[1] Traceback:
2018-03-01 09:58:00.515 | Traceback (most recent call last):
2018-03-01 09:58:00.515 | File "/usr/local/bin/package-installs-v2", line 137, in <module>
2018-03-01 09:58:00.515 | main()
2018-03-01 09:58:00.515 | File "/usr/local/bin/package-installs-v2", line 130, in main
2018-03-01 09:58:00.515 | process_output(install_args, follow=True)
2018-03-01 09:58:00.515 | for line in iter(proc.stdout.readline, ''):
2018-03-01 09:58:00.515 | File "/usr/lib/python3.5/encodings/ascii.py", line 26, in decode
2018-03-01 09:58:00.515 | return codecs.ascii_decode(input, self.errors)[0]
2018-03-01 09:58:00.515 | UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 34: ordinal not in range(128)
Change-Id: Ie4af9b4523459a630cfb98d09093bfe9ef7aa61e
Currently rhel7 image creation fails because it tries to copy
default bootloaders which is ubuntu way. This commit updates `iso`
element to correct the path of bootloaders required for rhel image.
Change-Id: I526d75b2db609fc77be0fc778b4d00f2d3df38ec
Closes-bug: 1750725
