Provide a "when" option that provides for not installing packages
based on a = or != match on an environment variable.
Unit tests are added.
Change-Id: Ifa824dccaff69fd447f45d54cb4a3083bcabdd86
It looks like we dropped running these probably when we moved the
elements around. For testtools to find the test scripts we need to
add the __init__.py files to make the directories look like modules.
Also prevent copying any .pyc or cache files in as hooks.
Change-Id: I66d5f6ee62cc4d9ee14c64e819b4db57d035d09f
This allows nodes with remote devices configured via iBFT to be
correctly used during Ironic introspection and deployment,
at least for non-multipath configurations.
The new element is added as a dependency for ironic-agent.
Change-Id: If3dac6504d26535593f12e851092065b688ef696
install-packages is running before install.d phase, there is a chance
that installing a package like "container-selinux" will failed the
build, moving "selinux-permissive" to run at pre-install stage make
more sense.
Change-Id: I32f988be725d4b385c3765c47a00cd57c53d7d71
Update builds to Fedora 29. Remove the openstack gate CI mirror
workaround for pre-28 versions as they're not building in the gate any
more.
Change-Id: Ia6a8ae8d66d69f6add39e571043328e7274ba26c
8 MiB is not enough when using the grub2 element with centos7 images,
which installs binaries from the shim-x64 and grub2-efi-x64 packages
under /boot/efi. 12 MiB ought to be enough for anybody.
Change-Id: I8594d1fe05242f246a5809740a115ab2f84ac5a3
In order to allow the simple preparation of base images which
can be used for LXC/nspawn machine containers, we add this
element.
Containers inherit a kernel from the host, so there is no need
to build a kernel into the image. All the element needs is a
base init system which, in this case, is systemd.
Change-Id: I45651de2aa1b19bdeee301094f0bdffdd0a3b45c
This finalises the ports of the legacy jobs to zuul native jobs.
The dib-setup-gate-mirrors role preconfigures the repo templates,
etc. for the openstack-ci-mirrors element.
The dib-functests role runs the tests as specified by dib_functests,
and can run under python2 or 3.
Change-Id: Ied67a31f0d31503d13eccad8662c29740c93f33e
It looks like epel-release switch from "mirrorlist" to "metalink"
(around release 7-10 Jun 2017 according to [1]). Update our rewrite
matching to handle this "metalink" as well.
Add epel element to the centos7 (image-based) build for testing too
[1] https://koji.fedoraproject.org/koji/buildinfo?buildID=978473
Add epel element so it's tested during the centos7 functional test.
Change-Id: I2d6d4c2ec47bc69d2f16c96b5045b05c435a1af9
This updates diskimage-builder to support current Fedora releases (27
and 28) and removes support for Fedora 26 which is EOL as of June
2018.
Change-Id: I602b22ed4d5397b39dc1eef67964f6fbdcd93060
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
We want to set DIB_EPEL_MIRROR for the epel element, which means we
need access to the CI mirror info script in the chroot. Copy it into
the temp directory with extra-data.d and update the environment file
to find it.
Change-Id: Ia12f0cbdeb194eef3155497ceb5ffc4a452aad76
When using the upstream cloud images with the "ubuntu" element, they
have universe and multiverse enabled which we don't mirror.
To use the infra mirrors as a DIB_DISTRIBUTION_MIRROR with this
element, we need to be able to skip redirecting to universe and
multiverse, and additionally enable insecure repos (as we don't gpg
sign our mirrors).
Add and document two new variables with the ubuntu element to do this.
This is then setup by the openstack-ci-mirrors element so that we use
local mirrors duing dib functional testing for the "ubuntu" element.
Change-Id: I6ffbde07fa0e103641ee5c5f9d9e854e5b2168dc
openssl/cryptography versions are updated/stable
musl profiles need newer versions of open-iscsi masked as upstream
doesn't want to work with multiple libcs
Change-Id: If5baf339516390ae332015928557c6bb734486c2
This is a lot of very low value noise in the logs as these iterate
through all the elements (often doing nothing). Turn it down and add
an echo so we just see what elements it is working on.
Change-Id: I0687de4722766189db9d4a7bd7d3cfb45d387b62
To facilitate this I've created two new environment variables to set
the environment and default options for package actions.
eval is needed for the export as it preserves quotes.
Change-Id: Ib03651ee8dacd48cd1c135afd57cd31101356056
Signed-off-by: Matthew Thode <mthode@mthode.org>
Currently there's more-or-less an assumption that a kernel is
installed, so module blacklists are simply echoed into the modprobe
blacklist. This may not be the case with some ongoing container work.
Although we don't need to blacklist modules for containers, it also
doesn't hurt. Move the debootstrap element to the new modprobe
element, and allow it to create the blacklist directory.
Change-Id: I0f057caf473951df56a2af9633e3a5b53e0809b1
With the check added in commit 7566819139,
diskimage-builder fails to build RPM-based images if kauditd is not
running. However, this is only valid for environments where SELinux is
enabled. If SELinux is disabled (which is identified by an empty _runcon
variable), proceed with running setfiles.
Change-Id: I1b056f20a3a55f7333391207d9e1049d25ece041
Closes-Bug: #1779273
Diskimage-builder fails to build ubuntu-minimal images when run on
a Ubuntu bionic-beaver (18.04) instance.
The user gets "Couldn't create tempfiles for splitting up" when
apt-get update is run in the ubuntu-minimal element root.d.
The issue is that the /tmp inside the chroot is not getting the
proper permissions applied from the base-files package. This is likely
because the pip-cache element has already created the directory before
the base-files package is installed.
This patch changes the order of pip-cache to root.d/11-pip-cache so that
it runs after teh base OS root.d elements run.
Change-Id: I6fd1cb2a23422206884165eb502b260f0c1e52f7
The ubuntu-minimal README states that the latest Ubuntu LTS
is the default, but currently that is not true. This patch
changes the default to the current LTS.
Change-Id: I10f28314d1a5969c20094194637cfe31219d228c
The apt sources are set out in root.d/75-ubuntu-minimal-baseinstall
and the cache is updated, cleaned and a dist-upgrade is done there.
As such, this file is unnecessary.
Change-Id: Idab5ede3f235bc204c4bdebf40fbcf4a12e5bc2f
The ubuntu, and ubuntu-minimal elements both make use of a common set
of environment settings to determine the distribution name.
The ubuntu-minimal element also does a few extra things which would
appear to apply to both sets and bring in extra architecture support.
As such, these are included in the common element.
This intends to be part of a series of patches which will eventually
create a new element to build a minimal ubuntu-systemd-container
element which can be used for lxc/nspawn containers.
Change-Id: Ia4e620f7d3fa6215484a8d218cea2f28bd1ffaee
The grub.cfg has two variables [1]
GRUB_CMDLINE_LINUX : used on all boots
GRUB_CMDLINE_LINUX_DEFAULT : additionally used on all "normal" boots
The problem with I2298675dda1f699c572b3423e7274bc8bd7c1c9d is that it
appened the values in DIB_BOOTLOADER_DEFAULT_CMDLINE to both of these,
resulting in duplicated arguments. I don't think we considered that
GRUB_CMDLINE_LINUX_DEFAULT actually already appends to the
GRUB_CMDLINE_LINUX values.
Make DIB_BOOTLOADER_DEFAULT_CMDLINE only append itself to
GRUB_CMDLINE_LINUX_DEFAULT. That seems to line up sensibly with the
name of the variable.
Documentation is enhanced around this, and a releasenote added.
[1] https://help.ubuntu.com/community/Grub2/Setup
Change-Id: I76b5442a9090c19a6540ed2d4ab324546f241ebf
Closes: #1791736
Without this fix, building a CentOS image on Ubuntu where audit=0 is passed
as a kernel boot parameter will lead to the following error:
disk-image-create centos7 dhcp-all-interfaces cloud-init-nocloud \
devuser yum epel baremetal
... dib-run-parts Running tmpdir/hooks/cleanup.d/99-selinux-fixfiles-restore
... Error connecting to audit system.
Change-Id: I229d9b72f88bffddca42da57f01c27e902427071
Due to the arm naming convention, building centos images for arm64 and
aarch64 does not yield the same result. In order to locate grub2 on
aarch64 the correct mapping is added.
Change-Id: I1bb227b2523e420e394fec8c52c6c79fcdd31c53
Closes-Bug:#1789414
Signed-off-by: Charalampos Kominos <Charalampos.Kominos@enea.com>
This ensures nouveau is not loaded at boot, which is required when installing
NVIDIA GPU drivers and to avoid issues with PCI passthrough of NVIDIA GPUs.
The option to disable kernel modesets ensures that it can be unloaded again if
it happens to be loaded after boot (e.g manually or implicitly by X).
bp tripleo-vgpu
Change-Id: I60815de86e7b22dfb39555af9d2d53564841e2ab
Related-bug: 1774674
modprobe element currently fails when DIB_MODPROBE_BLACKLIST is not set.
As there are now two methods to control blacklisting this should be optional.
Change-Id: Ibf3c31a95177ba88c1b93228490c7f36f5b70b57
In some cases cache-url can get pulled in without curl, causing it to
fail.
Co-Authored-By: Adam Harwell <flux.adam@gmail.com>
Change-Id: Ibd66c2ca4f8cc423783555d8a99b1184f43adff2
Without this fix, a LVM based ubuntu-minimal image will fail
booting due to the fact that the boot process will not be able
to retrieve the root filesystem using LABEL=(cloud)img-rootfs
Change-Id: If4ecf65868563f7b799160a58af6312bedf956bf
This patch adds an expected semicolon to an end of statement in
dhclient.conf for dhcp-all-interfaces element. Without this fix, an
error occurs when an image is booted with a message,
'semicolon expected.'.
Change-Id: I8311dbc67cc2815223111da01e7a7517c7d6f059
When building with debootstrap, debootstrap will use the key to check
that everything is properly signed. It will not `apt-key add` the key
into the final environment, however.
Early adding the key after debootstrap before we need to read from the
private repo again prevents unsigned issues. This also maintains the
integrity of the packages in the environment throughout the build.
Change-Id: I5ca75ae4620c9fb26b512cb30f8cd79fa7a0373a
This element will replace modprobe-blacklist element. It wil
still have the blacklist functionality, but it also adds
the feature of passing a complete file with settings to the
modprobe.d directory. Adding this functionality, that will
allow elements that depends on this module, to just copy the
specified files to the final directory.
Change-Id: I9a44f7d11520b8b1e604956d3c1db2fc7e2bf457
The existing directories are needed for stage building (a part of the
Gentoo build process). Normally these directories are empty, but there
are times where overrides need to be defined. This commit handles
existing overrides for keywords. For historical reasons the overrides
were able to be put in different files and directories, this
centralizes them.
This also updates the version of openssl/cryptography that works with
or without bindist.
Change-Id: I62c934ed305a711a4a9a3ef01fa55ad142aebb78
This patch adds an element that handles the configuration for
creating a disk capable of being a remote root filesystem through
iSCSI on CentOS images.
Tested on Fujitsu Server and boot with BIOS and UEFI mode successfully.
- Tested Boot-From-Volume + EFI for centos7 with following elements:
"centos7 vm devuser cloud-init-datasources dhcp-all-interfaces
iscsi-boot dracut-regenerate block-device-efi"
Co-authored-By: Nguyen Van Trung <trungnv@vn.fujitsu.com>
Change-Id: Ia1f23d722dced6f254fd7aee86abe8066a72fa42
According to http://bit.ly/2HA4oDO and
the official Ubuntu manual
http://manpages.ubuntu.com/manpages/xenial/man5/interfaces.5.html
source-dir support has been removed from Ubuntu >= 16.04/Xenial
Once an image is generated and booted, moving the dhcp interface(s)
declaration(s) from /etc/network/interfaces into specific subentries
of /etc/network/interfaces.d and calling 'service networking restart'
just make your instance unreachable and all interfaces are left
unconfigured.
This patchset fixes this issue
Change-Id: I6b6b99c81490c874c5db5405c2fbf3c180c87464
When building the image on a non-efi environment, it generates
linux16/initrd16 entries. But to boot from UEFI they need to have
linuxefi/initrdefi entries.
Use sed to replace those entries, in case we have an EFI image.
Change-Id: I47c96450e10f34b91bcc32888532bd7ab87cf316
setfiles isn't supported on the vfat /boot/efi partition. Add it to
the skip list.
Tested on Fujitsu Server successfully.
Change-Id: Iab262c4bdb0ecc25ca6b77ee4aff1ce442c0c578
This patch adds an element that handles the configuration for
creating a disk capable of being a remote root filesystem through
iSCSI on Ubuntu and Debian images.
Change-Id: Ibf9e39d2bdab530106015f156d23d28029d12b0d
Closes-bug: #1716794
When using uefi in rhel, the package mapping is incorrect.
We need to add specific grub-efi* mappings to use grub2-efi
Change-Id: I2db96ae85fd5e4638c794015b2f8164c018420e3
We need to handle openSUSE Leap 15 when installing pip and virtualenv
packages. This fixes the following problem when the pip-and-virtualenv
elements is used:
2018-05-31 09:42:12.014 | + [[ opensuse = opensuse ]]
2018-05-31 09:42:12.014 | /tmp/in_target.d/install.d/04-install-pip: line 57: packages: unbound variable
Change-Id: Id7911b0a0836fa8dcc003e23fa515b78fba67126
Patch allows to rebuild arbitrary images, which location, filename and
sha256sum are specified in variables, not only hardcoded $DIB_RELEASE/current.
Change-Id: I05418932a0c40d885fe00a49f1f49d7e86c67518
Add a bionic test in replacement of trusty. We are already building
bionic images in the gate, so this seems like a good time to switch.
Change-Id: I20d4c25e9b79e7326c86767c36be8615ba0888a3
Removing no longer working and no longer maintained ubuntu-core element, which
intent is unclear, and not documented.
Change-Id: Id847591d04fd7cd32c8903967da01ee0d303b267
Closes-Bug: 1771614
Without this change DIB appends a second command line entry to the GRUB
config. This causes the original command line entry to be ignored
when Linux is booted.
The expected behaviour is that DIB appends to the existing entry as
it does for Ubuntu and SUSE.
Following discussion on the review, this also removes the distro specific
switch statement, as update-grub just calls grub-mkconfig, meaning that
there was nothing distro specific in the first place.
Change-Id: I2298675dda1f699c572b3423e7274bc8bd7c1c9d
Closes-Bug: #1771366
Since CentOS-7.5, a new yum variable is needed for SIG repositories.
This change replicates the %post task of the centos-release package
to setup the contentdir yum var. This should fix issues when
repository url uses $contentdir and yum fail with:
http://mirror.centos.org/%24contentdir ... [Errno 14] HTTP Error 404 - Not Found
For more details see:
https://lists.centos.org/pipermail/centos-devel/2018-March/016542.html
This change also drops support for fedora without dnf.
Change-Id: I1819a48b94670577b0c5e29b24cebfb20ea07d28
This commit addresses the issue described in bug 1768354 when using the
apt-sources element and adding a key to a custom repo, subsequent deb
package installs fail due no update of the repo before package install
Change-Id: I968b3422fab2fb2305426d49215391d8ba7499df
Closes-Bug: 1768354
The pip-and-virtualenv element provides pip inside the created images.
When this pip is used inside the chroot of the image it, by default, creates
a cache directory with the http packages and the resulting wheels.
In the case of the octavia ubuntu-minimal image this cache is using ~50MB of
cache that is not needed after the image finished building.
This package creates a finalise.d task that removes the cache from the
default location.
Change-Id: I4715437b068d04993ef755bd1e27963db1d22417
When RDO projects repository is installed, the python-setuptools
package is obsoleted by python2-setuptools, this makes the install-pip
script failed:
Package python-setuptools-0.9.8-7.el7.noarch is obsoleted by
python2-setuptools-22.0.5-1.el7.noarch which is already installed
Then the "rpm -ql python-setuptools | xargs rm -rf" exit 1. Check if
we have a record of the updated package obsoleting then old one; if
so, use it.
Change-Id: I2b0051bd9e81908c187098a7b82e120b999b111d
We no longer install wget / yum-utils for centos-minimal, this fixes
that.
Change-Id: I8d89026bd48cf7398cc1cbe41e3b7f00f682dbb8
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
This currently breaks glean on rackspace, revert until we can figure
out why that is.
This reverts commit 43bc352c59.
Change-Id: Iae88a3b0457bab0b8f0fd1febf58732ca95e5dc9
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
The release of pip10 has shown up a few issues here
Firstly, pip10 now refuses to overwrite distutils installed packages,
which includes "python-virtualenv" on centos. History has shown us
that we want the packages installed and overwritten, to avoid the
packages coming back and messing things up.
Pre-install all the packages, then list the files in the packages with
"rpm" directly and remove them. This way pip is happy to install.
We need to take better account of the package names for this; on
Fedora things have switch to "python2-virtualenv" instead of
"python-virtualenv" and we can't use an alias to list the package
contents.
This also highlighted that python2-pip is in EPEL for centos, so
enable that when we install it. Make the epel element a no-op for non
centos/rhe distros.
There is a related change in recent fedora that python3 now installs
binaries into /usr/local/bin. There are commented swizzles in here to
ensure we retain the status quo of "pip" and "virtualenv" both being
python2 based, with the python3 versions being called explicitly
"pip3" and "virtualenv3" respectively.
Change-Id: I2ffdd9f615ae6b00428c17249e4f216774991b99
We added this sed in I422490ebe9a9c655552685bc2ff342d288335a9c to
avoid installing python2 packages on python3-only systems and thus
dragging in all of python2.
We made a similar change to python-pip in
I7d8ba9300039cce90965410a4e16ca9e711904c3; however we realised that
the gate (and other consumers) were relying on this element having
installed the python2 & 3 packages for consistency -- otherwise jobs
would install the python-pip packages and overwrite the
pip-from-source and mess everything up. We reverted that in
I419dbdf4682394db68974944af1e5c432f3e0565 and added some clearer notes
that this element brings in python2 & 3, and if you want something
that doesn't do that then this element isn't for you.
However, we never fixed up the virtualenv package install -- currently
our Xenial images have a global virtualenv installed from source, but
the python-virtualenv packages aren't installed. Thus if a job does
"apt-get install python-virtualenv" it overwrites the from-source
virtualenv with older parts and again messes everything up.
Probably most jobs just call "virtualenv" and assume it is there;
however in bringing up some rspec test for puppet I have hit this
issue as some modules specify dependencies on the virtualenv packages.
Thus install the python-virtualenv AND python3-virtualenv packages in
this element.
Change-Id: Ia84c38dc3c40a6080e144b563e10abca7dac2881
On initial boot when networking is brought up by cloud-init this
is the timeout that dhclient adheres to. Centos configures
"timeout 300" (for an EC2 bug) in their cloud image, which results
in a 5 minutes delay to boot in cases where no dhcp available (e.g. IPv6
SLAAC). To reduce this boot delay and to provide consistency with
places where we have set other dhcp timeouts set this to DIB_DHCP_TIMEOUT.
Change-Id: I119a002070501c3dfe7c6730b07ee25f422b85b0
Related-Bug: #1758324
Many elements install additional distribution packages.
In addition the user can provide a set of packages to be installed
via the '-p' switch.
Some of them influence the boot process and therefore the initramfs
needs to be updated. Because the package manager during the image
creation process is configured not to run package scripts, this needs
to be done explicitly.
This issue was found during development and debugging of the
block-device LVM plugin: Even when the e.g. the lvm2 package
was installed in the image, it was missing in the initramfs
because of the missing update.
Change-Id: I7c92033b3ca80cdd23d081002059d83ca3f53bdb
Signed-off-by: Andreas Florath <andreas@florath.net>
Default the GENTOO_PORTAGE_CLEANUP to True. By default we should not
ship package info, this bloats the image and is usually outdated by the
time it'd be consumed.
Change-Id: I14c2530d91807cbc6a3806e01c7e4f6f472b190d
The debian element depends on debian-minimal now which provides
operating-system. This means that the debian element can no longer
provide operating-system and doing so results in an error when using the
debian element.
The fix is simple just rely on the fact that debian-minimal provides
operating-system and remove this element-provides from debian.
Fixes-Bug: 1758000
Change-Id: I524feeb82c19046ec987eb1186c7f4568309e559
The devuser element can set up passwordless sudo, which requiers the
/etc/sudoers.d directory, which requires the sudo package, so we ensure
the sudo package is installed.
Change-Id: I80d6c669d4ac0d97b49d01cb621bf05b8e7f8ef1
Hpsum utiltity of proliant-tools requires net-tools to be installed
as part of base image. This commit adds support for installation of
net-tools for all distros.
Change-Id: I2a1e81059ed1aee975db78cfa5e61bbf1b98e06f
Closes-bug: 1751777
When using the package-installs element there can be some encoding
problems if the package installation emits unparsable output
[1]. However in this case we just want to forward the output to the
console which normally can handle this correctly. In order to fix this
switch off universal_newlines processing such that we just operate on
bytes.
Further we have to decode the lines without setting the locale and
ignoring errors. This is required because print encodes without
setting the locale and thus we need to filter/modify the stream such
that it doesn't crash.
[1] Traceback:
2018-03-01 09:58:00.515 | Traceback (most recent call last):
2018-03-01 09:58:00.515 | File "/usr/local/bin/package-installs-v2", line 137, in <module>
2018-03-01 09:58:00.515 | main()
2018-03-01 09:58:00.515 | File "/usr/local/bin/package-installs-v2", line 130, in main
2018-03-01 09:58:00.515 | process_output(install_args, follow=True)
2018-03-01 09:58:00.515 | for line in iter(proc.stdout.readline, ''):
2018-03-01 09:58:00.515 | File "/usr/lib/python3.5/encodings/ascii.py", line 26, in decode
2018-03-01 09:58:00.515 | return codecs.ascii_decode(input, self.errors)[0]
2018-03-01 09:58:00.515 | UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 34: ordinal not in range(128)
Change-Id: Ie4af9b4523459a630cfb98d09093bfe9ef7aa61e
Currently rhel7 image creation fails because it tries to copy
default bootloaders which is ubuntu way. This commit updates `iso`
element to correct the path of bootloaders required for rhel image.
Change-Id: I526d75b2db609fc77be0fc778b4d00f2d3df38ec
Closes-bug: 1750725
For 'satellite' mode of registration, rpm for rhel SSL certificate is
hard coded to 'katello-ca-consumer-latest.noarch.rpm'. This commit adds
functionality that provides an option to set this as defined in their
satellite server.
Change-Id: Ib176cfa209f5ac8a4b5da71419327b4237330904
Closes-Bug: 1749947
Install hwe kernel for ubuntu-minimal. As noted this is currently
Xenial specific; we need this for initial bring-up so let's tackle
future releases as things progress.
Ensure we use ttyAMA0 for arm64 console too.
Change-Id: Ic607cf8369666dc24929aff6f2ef8a72e7980599
In the prior change we added block-device-[mbr|gpt|efi] elements to
create appropriate disk-layouts.
This adds an environment flag to each so the bootloader can install
the right thing. The EFI install path is updated to work with this
(this part a copy of I572937945adbb5adaa5cb09200752e323c2c9531)
We do some basic sanity checking in the block-device elements;
e.g. mbr is not suitable for aarch64, and efi is not suitable for
power.
This updates the bootloader to install EFI where appropriate
Co-Authored-By: Marcin Juszkiewicz <marcin.juszkiewicz@linaro.org>
Change-Id: Ib80acbfd9a12efd976c3fa15a5d1081eb0799305
This moves the block-device default out of the "vm" element and into a
selection of other elements. There's "mbr" which retains the status
quo. There's an EFI version that has the boot/grub partitions as
required. In between there's the GPT only version, which is useful
for architectures like power without EFI, but still want possible
larger disks using GPT.
Change-Id: I4a566a97d073fc0dda0ab2494ac988fe015800a9
This adds support for a GPT label type to the partitioning code. This
is relatively straight-forward translation of the partition config
into a sgparted command-line and subsequent call.
A unit test is added based on a working GPT/EFI configuration and the
fedora-minimal functional test is updated to build a single-partition
GPT based using the new block-device-gpt override element. See notes
in the sample configuration files about partition requirements and
types.
Documentation has been updated.
Co-Authored-By: Marcin Juszkiewicz <marcin.juszkiewicz@linaro.org>
Change-Id: I6b819a8071389e7e4eb4874ff7750bd192695ff2
In slow networks like Infiniband it takes much time for the
interface to get the carrier. This patch enables this service
to run more then 20 seconds and limited by DIB_DHCP_TIMEOUT.
Change-Id: I8a6015567ac25e37b5a5aba4b1fda71170cc144a
Like we did in https://review.openstack.org/475206 we need to install
systemd sooner because of the new world order of containers.
Change-Id: Ia60d751fee3af6f8d72ad664107acb337360feca
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
While Debian-based distros use the label of ppc64el for ppc64 little
endian, Fedora uses ppc64le.
The ironic-agent was doing arch specific package install of lshw over
dmidecode for ppc64 and ppc64el but was attempting to install dmidecode
on Fedora ppc64le which caused the test to fail due to a missing
package.
This change just adds ppc64le to the arch-specific package installation
description for the ironic-agent element.
Change-Id: I38c3c1480bbbb2df817856614e6b740a0c02723a
Closes-Bug: 1744944
This updates diskimage-builder to support current Fedora releases (26
and 27) and removes support for Fedora 25 which is EOL as of December
12, 2017.
Change-Id: I227a607c6c468cc8b7bb154a189e9c8ce2021192
The installed pip can be an older version which does not support
the -c argument. Therefore, upgrade pip before using -c.
Change-Id: If18d8ea822a62c8551c9c4d47354d58b0299fed2
Closes-Bug: 1744403
This reverts commit ab89c7d69c.
This commit checked for DIB_PYTHON_VERSION and only installed the v3
packages. This is unfortunately backwards-incompatible, as consumers
such as the openstack gate are relying on this package installing pip
& virtualenv packages for python2 AND python3.
This was sort-of expressed in the docs, where it discusses what the
resulting setup of the system will be, but I've added a note to make
it clearer.
If we want to change this, I think we'll need either a new element, or
a non-defaulting flag.
Change-Id: I419dbdf4682394db68974944af1e5c432f3e0565
It turns out make has always been a tacit dependency of openssl as it
ships a Makefile for certificates [1]. This just recently changed to
be a hard dependency in F27, so this now fails as openssl is a
dependency of protected packages such as dnf. Since it's always been
wrong to remove it, we take it out of the purge list.
[1] https://bugzilla.redhat.com/show_bug.cgi?id=783446
Change-Id: I69efb3a56878ab97c4587bbbf5356bea752f2846
There's a patch in flight in ironic-python-agent to switch the
default hardware manager to use lshw instead of dmidecode. [0]
This would require lshw to be installed regardless of
architecture. This patch removes the architecture rules from
lshw in the package-installs list.
[0] Ie370331df6bb5ef131c5cb60f458877e2a7ad71a
Change-Id: Idaf05b8efce28cd0cbf339cf693db4f55a693d9b
Partial-Bug: #1715790
zypper only supports the --no-recommends option during installs, giving
the option during removals results in an error.
When setting ACTION=remove, remove --no-recommends from EXTRA_ARGS, and
set --clean-deps to also remove no-longer-needed dependencies.
Rename EXTRA_ARGS to ACTION_ARGS for increased readability.
Change-Id: Ifbd168992b1a20658b6b4a99ba175234f6c78f6d
When "epel" element is used during a build process
with "rhel7" distribution, the build failed
because the "epel-release-7*" package cannot be
installed.
The reason is because the URL is not correct, it
should be:
URL=$BASE_URL/$RELEASE/x86_64/Packages/e/
Change-Id: I90c26892361f7611645b85f2eddc949b2f0d76fc
Closes-Bug: #1735547
At the moment all musl needs in addition to an official stage4 file is a
few keywords and use flag changes.
Change-Id: Ibf4a6d616aca1aef876967e2aa34170c96ac9ef8
This is intended to eventually support building musl-libc based images,
which need the musl overlay.
Change-Id: I8f5429ffa64e74c860772d9a00ff0b7eebb7721a
As described, Fedora 27 has a curl-minimal package that comes in to
satisfy the rpm package dependency. It conflicts with the "real" curl
package -- which is so commonly installed (by infra elements, etc)
that this becomes an annoying problem. Just pre-install the full curl
package.
Fedora 24 is old enough to not worry about, so remove some old
workarounds to make the flow a little simpler.
Change-Id: I67baf96377109ac4521ba00243a0d91b35fafba0
The current implementation - as introduced in
Iee44703297a15b14c715f4bfb7bae67f613aceee - has some shortcomings / bugs,
like:
* the 'grep' check is too sloppy
* when /dev/pts is already mounted multiple times the current implementation
fails:
$ mount | grep devpts | sed 's/.*(\(.*\))/\1/'
rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000
rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000
rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000
* code duplication
* Using the undocumented and non-robust output
of 'mount'.
This patch fixed the above problems.
Change-Id: Ib0c7358772480c56d405659a6a32afd60c311686
Signed-off-by: Andreas Florath <andreas@florath.net>
We oneshot emerge without calculating dependencies a few things to solve
for possible dependency loops.
Python 3.5 also became stable, so don't need to do special things for
it.
Matched the uninstall with the install lines (no need for a full if
statement).
Change-Id: I7c5e546612ac47d659e73a46a52e34d39ca81949
We should always refresh the Tumbleweed repositories and the 'update'
one for Leap in order to always have the latest information from the
repositories.
Change-Id: I85db9d8bb7fa153f01222129e9b36fecc2632f57
This is a continuation for f2cc647dae ("diskimage_builder: lib:
common-functions: Fix options for devpts mount"). We also need to
respect the devpts mount options when the dib elements are mounting
this virtual filesystems themselves.
Change-Id: Iee44703297a15b14c715f4bfb7bae67f613aceee
We want to install python3-pip, not python-pip when we are building a
py3k image less we pull in python2. Once we stop installing python2 we
have to stop calling python2 during pip install.
Change-Id: I7d8ba9300039cce90965410a4e16ca9e711904c3
Currently in Leap 42.x the bootup scripts don't actually make use of
locale.conf yet, so we need to set it in /etc/sysconfig/language. For
future distro compatibility the setting in locale.conf is kept in sync.
Also fix default timezone link.
Change-Id: I59e5dccad8a5ae132d3039851e7aa1db86a609d7
s390x architecture uses zipl as bootloader. When used in combination
with the vm element it replaces the existing bootloader element.
It's mandatory for s390x vm images.
Use cases
---------
* Allow users to create s390x images that run on nova with s390x
libvirt/kvm backend
* Building nodepool images for s390x third party CI
Supported Distros
-----------------
The following listing shows all Distros that officially support
s390x and how those Distros are supported in DIB with this patch.
* SLES - not supported (SLES is not supported in DIB)
* RHEL - not suppoprted (RHEL is not supported as KVM guest on s390x,
therefore there's no rhel7 qcow image for s390x available
like it is for other archictectures)
* Ubuntu - supported
Ubuntu images can for example be built using the following commands:
$ disk-image-create ubuntu-minimal zipl vm
$ disk-image-create ubuntu-minimal zipl
$ disk-image-create ubuntu zipl vm
Testing
-------
Cross architecture building of s390x images is not supported so far.
The plan is to set up a ThirdParty CI that builds the image for s390x and
provides the logs.
Co-Authored-By: Andreas Scheuring <andreas.scheuring@de.ibm.com>
Co-Authored-By: Holger Smolinsky <holger@smolinski.name>
Co-Authored-By: Zhiguo Deng <bjzgdeng@linux.vnet.ibm.com>
Co-Authored-By: Arne Recknagel <arne.recknagel@hotmail.com>
Closes-Bug: #1730641
Change-Id: I576e7edda68da12e97c60af38f457915efe7b934
In a couple of places we use flock for critical sections, but we leave
lockfiles around in various locations which can be confusing.
Introduce DIB_LOCKFILES global (under ~/.cache/dib/lockfiles) and
write lockfiles in there.
Fix up removal of the lockfile in the yum path; we just want to make
sure we cleanup the .rpmmacros file, but we don't need to remove the
lockfile as well.
Co-Authored-By: Andreas Florath <andreas@florath.net>
Change-Id: Ie810b2836be521325afe923708d046112e1e1e20
Create a new service, that will be launched after ironic
agent has been exited. This will launch an script that will
take the rescue password, and create the rescue user with
that credentials.
Depends-On: I7898ff22800dedba73d7fbfb3801378867abe183
Change-Id: Ic3a241e2789a122d3d966e7e2148306fd0cf6aed
Partial-Bug: 1526449
Currently a bind is used when mounting /dev/pts in chroot.
This leads to problems - especially when running DIB in parallel:
It was observed that the /dev/pts mount vanishes from the host
system.
This patch uses '-t devpts' - as it is done for /sys and /proc -
for handling /dev/pts.
Change-Id: Id7775ae6fca6502af800e7b73a00862ef320206b
Signed-off-by: Andreas Florath <andreas@florath.net>
On ubuntu we detect that in python3 we need to install
python3-virtualenv, but append this to the packages to install rather
than replace python-virtualenv which results in both being installed
(and therefore grabbing python2).
Change-Id: I422490ebe9a9c655552685bc2ff342d288335a9c
Closes-Bug: #1724656
There have been a few changes over the past few months, here we make the
following changes.
* change from backtrack=99 to complete-graph as a more correct flag
* make python version selection more in line with what gentoo supports
* set up python before stuff gets pip installed
* ensure we have the proper pip so we can install pip packages as root
* ensure we have the proper use flags for the disk formatting changes
* set DIB_RELEASE like other distros
* fix openssh-server element for gentoo
Change-Id: I17202de3016616ce34c8cbead7d0fb047a64e96b
This commits make update to ssacli version to point to latest
ssacli release that has support for HPE P/E-Class SR Gen10 controllers.
Change-Id: Ia9a0eaec78d601f56b4036e57601554b87f21acc
Closes-Bug: 1721185
A small update was made to 4.4.0-96.119 that dropped the
initramfs-tools dependency from the kernel [1]. This had the
unfortunate affect of removing the initramfs from ubuntu-minimal and
making it unbootable, since we specify the root device via LABEL=.
Add the package explicitly alongside the kernel.
Also, small fix to pass unit tests
[1] https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/1700972
Change-Id: I57a0f08cd5e082ecdf8dba0ab34fb3062c50836d
This commit adds change in 'proliant-tools' element to
install a package 'unzip' which is required to perform
SUM based firmware update for HPE Proliant servers.
Change-Id: Ib8f6d18402439edd93d100cc7a4fb2094c863715
As described in the comment, we need to create the /etc/machine-id for
the image-based build when systemd isn't updated (as is usually the
case for a new distro)
Work on clearing this out continues, but this brings it to parity with
fedora-minimal.
Change-Id: Icbbbabb4114d4d95909648d8e39a6bae6d2a7b7b
Depends-On: I761e425f8a658669d9b8a70ce4260cec263ea51a
The URL we are using seems to have disappeared. Update this to
download.fedoraproject.org. The new URL requires a "subrelease" now,
add it, along with a note on where it comes from.
Change-Id: I761e425f8a658669d9b8a70ce4260cec263ea51a
This element was assuming that yaml was included as package,
but there are systems not including it. So properly add yaml
as a dependency.
Change-Id: I72da2776674a3963657052b9a9715abcb4fab1e2
Partially-Fixes-Bug: #1715686
When using combined with rhel7 image, the unregister of repos
has already happened, because it is executed under 60- ordering.
As dracut-regenerate may need to install extra packages for it,
it causes this step to fail, because it cannot find repos where
to pull the packages from.
Change-Id: I35e37df7990ad76a5004cb90fdd863ec743a5483
Per the bug report, these seem to be causing issues with maintaining
file capabilities. They aren't necessary so let's just remove them.
Change-Id: I06c90fdc85655986142b936cadbe04d75dd27427
Closes-Bug: 1714604
Avoid incorrect use of [ with =~ matching
I guess this doesn't trip "-e" because it's in an if-conditional. I'm
looking at making bashate detect this; maybe we can run bashate over
things we know are scripts
Change-Id: Ia3fe2b978fae5bdaadbb1789058180d3ad950d00
In Ubuntu/Debian, the default dependencies cannot be relied
upon as we enter into a cyclical dependency relationship which
prevents the unit from starting.
Added the required configuration to the systemd unit file.
This issue has also been observed in glean[0], which has a nearly
identical unit file for interface start-up.
[0]: https://review.openstack.org/#/c/485748
Closes-Bug: #1708685
Change-Id: I23ac9510d1a21c7073bd33f76ba66fa04a8be035
Many programs rely upon /etc/protocols to be present
however the default debian image that is generated lacks
/etc/protocols. This is observable when building an image
for use with ironic via the ironic-agent element, since
the IPA agent fails to start as python needs /etc/protocols
to open a socket connection.
Added to debian-minimal as it is inherited into the debian
element.
Change-Id: Icc81635870961943707cf6b3f61a9ddbd51cb8fd
Closes-Bug: #1708531
There is some confusion in the readme's over what is happening. The
original change (Iaf46c8e61bf1cac9a096cbfd75d6d6a9111b701e) split out
debian-minimal and made debian "... simply be a collection of the
extra things we do to make it look like a cloud-init based cloud
image"
Make this clearer in the documentation
Change-Id: Ibe6fad9c67b70a5e31e43e06419968135174fef3
Deploying many nodes with the generated image shouldn't have the same
/etc/machine-id so clearing it and letting systemd generate a new
id upon first boot seems to be the best way to achieve this.
Change-Id: I73d0577d31464521b3989312fd9d982a1312a268
Closes-bug: 1707526
Closes-bug: 1672461
Fedora 26 is now the latest release:
https://fedoraproject.org/wiki/Releases/26/Schedule
We are building and using these in infra now
Change-Id: I012c2d28255be274e88abc2751d968bafaf76fbb
Depends-On: Ieba5f69020a13681074f72cfca2955071801b63a
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
Change I008f8bbc9c8414ce948c601e3907e27764e15a52 has shown that we
build redhat images without the "semange" tool available, which comes
from the policycoreutils-python package (see also
I3f9e2c322d042a5dddba33451c0fc21a4d32a88a).
I403e7806ae10d5dd96d0727832f4da20e34b94c7 added some of the selinux
libraries to yum-minimal for ansible support, but not to others.
Given both these changes, it seems that selinux[-targeted],
libselinux[-python] and policycoreutils[-python] can reasonably
considered part of all base images. Move the selinux related packages
into redhat-common.
This also adds it explicitly to install_test_deps.sh. It was actually
being dragged in by the docker install, but is a required component
for building (should be in bindep, but not there with that yet).
Change-Id: Idd4ae71ee6deee84604823b6b5dc4a845f316e01
Related-Bug: #1707788
Currently, the cleanup script is using existence of
semanage binary to check if selinux is enabled. However
this is misleading and can lead to problems when selinux
is disabled in a system where the binary exist.
This patch changes the detection logic to use /sys/fs/selinux
directory which is a in-memory filesystem created only when
selinux is really enabled.
Change-Id: I008f8bbc9c8414ce948c601e3907e27764e15a52
Related-Bug: 1706386
tar is an essential package but nothing pulls it explicitly. This causes
some issues in the openSUSE CI jobs like the following one
"Failed to execute tar: No such file or directory", "Failed to write
file: Broken pipe", "Failed to retrieve image file. (Wrong URL?)",
"Exiting."], "stdout": "", "stdout_lines": []}
Just like 'sed', add 'tar' to the list of packages for the openSUSE
minimal builds.
Change-Id: Ia36e3d9fd6b78862a6831ba80b43d4614a349ca0
As described in the comments inline, on a selinux enabled kernel (such
as a centos build host) you need to have permissions to change the
contexts to those the kernel doesn't understand -- such as when you're
building a fedora image.
For some reason, setfiles has an arbitrary limit of 10 errors before
it stops. I believe we previously had 9 errors (this mean 9
mis-labeled files, which were just waiting to cause problems).
Something changed with F26 setfiles and it started erroring
immediately, which lead to investigation. Infra builds, on
non-selinux Ubuntu kernel's, would not have hit this issue.
This means we need to move this to run with a manual chroot into the
image under restorecon.
I'm really not sure why ironic-agent removes all the selinux tools
from the image, it seems like an over-optimisation (it's been like
that since Id6333ca5d99716ccad75ea1964896acf371fa72a). Keep them so
we can run the relabel.
Change-Id: I4f5b591817ffcd776cbee0a0f9ca9f48de72aa6b
For builds inside the infra, we don't want to pack the cache
inside the image (as it might be different at the time the image
runs). In an opensuse-minimal image this saves about 10MB of image
size.
Change-Id: I5ecabd46f0a662798bda3e4468395ad8308d0055
As described in the comment and associated bugzilla, the behaviour of
setfiles has changed in Fedora 26 to require "-m" situations where
labeled file-systems are mounted below non-labeled file-systems. Our
loopback/chroot system appears to trigger this nicely, leading to a
setfiles call that does nothing without this.
Change-Id: I276c6f6a4fb44f4bea5004f6b4214f94757728ae
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
As described in the referenced bug, the dependency solver in yum
doesn't handle weak dependencies well and in some cases, such as
Fedora 26, can end up choosing coreutils-single (the busybox-esque
single binary) instead of actual coreutils, which then causes problems
with conflicting packages later.
Change-Id: I2907bf3b74c146986b483d52cc6ac437036330b4
On a system where the packaged pip/virtualenv is up-to-date with
upstream (such as Fedora 26 ... for now), we don't reinstall, which
then violates a bunch of assumptions later on. Force install.
Change-Id: I6ebcda0351997fa7e32f0e6e77a98b2c33764e3f
It turns out dnf argparse can't handle negative numbers without "=".
It's actually documented in the man page
--latest-limit <number> ... If <number> is negative skip <number>
of latest packages. If a negative number is used use syntax
--latest-limit=<number>
But who reads that :) This started failing with Fedora 26
Change-Id: I884af94c07fa11b010f69863047a04711b14f21e
We expect LC_ALL for non-C locales to be working inside
images, so always install glibc-locale for openSUSE.
Change-Id: I8fe92773e377539070d9d9fe2960a6202bb80a18
In preparation for promoting the openSUSE jobs to voting ones we should
use the OpenStack mirrors. As such, the opensuse elements are modified
to make use of the DIB_DISTRIBUTION_MIRROR variable which is normally
exported by the openstack-ci-mirrors element.
Change-Id: Ie588c1c1eec13190cfb2ec718ba51f8c9878283f
We added the DIB_distro_DISTRIBUTION_MIRROR arguments with
I92964b17ec3e47cf97e3a3091f054b2a205ac768 as a way that we could
source a list of mirrors and then have the distro elements choose
which one applied to them.
However, this hasn't worked out to be so useful. The
openstack-ci-mirrors element is working as a mirror setup script -- it
translates the openstack CI mirror list variables into the generic
"DIB_DISTRIBUTION_MIRROR" as appropriate for each distro's build.
Also, it turns out there's other things that need to be done, such as
turning off gpg checking, which mean the idea of "just export
variables" hasn't turned out as valid ... you need actual code
involved to get it right.
AFAICT we never actually documented these, and they do not seem to be
in use. They have caused considerable confusion when dealing with new
platforms as we try to keep consistency. Remove them.
[1] http://codesearch.openstack.org/?q=DIB_.*_DISTRIBUTION_MIRROR&i=nope&files=&repos=
Change-Id: Ifc4ab700631ffdfbe790068558f670f9a11dde5e
The purpose of the openSUSE element is to build openSUSE distribution
based images, so an additional community repo shouldn't be pulled into
the image. In addition the dkms dependency is blacklisted for SUSE
in the dkms element anyway, so this should be a noop.
Change-Id: I0aa06d9f4f110546032f910e3361840693d02de7
On Power systems console should be added the kernel command line
in the following order: 'console=tty0 console=hvc0'.
The first one is the graphical console. The last one is the serial
console. The kernel enables all the consoles pointed through the
kernel command line. However, only the last one will receive
input/output during kernel boot. All the other consoles will be
enabled after the boot.
Change-Id: I0069f608e0ab104d3778954e033fb82ed5ea7693
The python3 package actually contains some core modules (like the xml
one) which are not present in the python3-base on which is pulled by
the python3-devel package. As such, it's best to have it installed
similar to python-xml for python2.
Change-Id: I5cd5d1127ae62d6753c2ace44965179c5400bb9a
In order to support {CentOS,RHEL}7 for building cloud images we need to
handle the differences in grub packaging from Ubuntu. We also need to
populate the defualt location for cloud images for CentOS builds.
Change-Id: Ie0d82ff21a42b08c4cb94b7a5635f80bfabf684e
When a download redirector redirects to a broken mirror, timeout
quickly rather than waiting until the overall job is being timed out.
Change-Id: If7eb63d406aaf61f71aa9203cf708c474aa63fd0
The 'packages' variable already contains the packages we need so
use it instead of duplicating the packages.
Change-Id: Id22e1862f9654e66252d03a0fed9839cf004d750
Several people have popped up in IRC recently with failures in these
elements. Without Python 2.7 available in the image they are
unsupported (OpenStack hasn't supported it for a long time). Remove
these to avoid further confusion.
The centos/centos7 DISTRO split that has happened with centos-minimal
is unfortunate but I don't think it helps to rename centos7/rhel7 ATM.
To summarise; DISTRO=centos7 means image based build,
DISTRO=centos && DIB_RELEASE=7 means the minimal build.
In the future, I think it is important that the minimal builds and
image builds set the same DISTRO. This reflects that "upper" layers
shouldn't care about the exact building of the lower layers. I see
CentOS 8 going one of two ways
1) the changes are so significant, we start separate centos8 /
centos8-minimal elements. They both set DISTRO=centos8 (and
DIB_RELEASE to point-release maybe?). This means we have to update
all "if DISTRO == centos || DISTRO == centos7" branches to also check
for "centos8". Evenually (!) "centos" goes away for versioned DISTRO
only
2) we restore centos element with DISTRO=centos and DIB_RELEASE=8, and
centos-minimal remains the same. This means we have to audit all "if
DISTRO == centos" calls to make sure they're appropriate for version 8
(stick a "&& DIB_RELEASE=7" on them all basically).
I'm not sure we can fully decide until we start to see excatly how the
distro switching/matching bits look, but (2) is consistent with Ubuntu
and probably the preferred solution.
Some "rhel" parts have been cleaned up. More could be done in
rhel-common, but given our lack of coverage of that I'd prefer to
leave it for now.
Change-Id: I6ea784116ef59ca22878c8512c963f29c815a00a
The image download tests have long been too unreliable for the gate.
We need to cache the base images similar to how devstack caches it's
testing images. Let's move them to non-voting jobs for the time
being.
This means that the gate jobs are now all based on "-minimal" and are
using infra mirrors. Unfortunately, there is still some unreliability
because we currently have issues with infra mirrors being very slow
after AFS updates, leading to job timeouts. But we're on the right
path...
Also, I noticed we don't have tests of the "ubuntu" image-download
based tests, which were tacitly being tested by apt-sources before we
moved that to -minimal. Add simple tests for these.
Change-Id: Ie33ee49656872467ef68d753210032156bb6b2cb
This was previously defined as python2-devel (which is what rhel uses),
but the actual package name is python-devel. See:
https://software.opensuse.org/package/python-devel
Change-Id: Id61e5b05772d10c32b33d3e70cb64d5ebdcba6e4
I'm uncertain as to why this is using the "fedora" element for testing
... but it requires downloading the fedora .qcow on every test which
has shown to be unreliable. An easy thing to do is to switch it to
fedora-minimal; that will only involve downloads from local mirrors in
the gate.
Add redhat-rpm-config for minimal. I admit I have not fully gone
through why this is not pulled in. It's been an issue since
I459f2203fa145049dda185da952813118193d573 and there's all sorts of
bugs.
Change-Id: I37458e3926dae32a259bd5aa9efc645561b029a0
fedora/centos-minimal don't obey DIB_DISTRIBUTION_MIRROR currently. I
don't really want them too -- we want to be able to separate the
mirrors used during the build process from those embedded into the
final image. Add DIB_YUM_MINIMAL_BOOTSTRAP_REPOS which is a directory
with repo files to use during the install.
This introduces setup-gate-mirrors.sh which is intended to setup
repo/sources/whatever files in the openstack gate that point to the
local region mirror. It pulls the info from the mirror_info.sh script
on each CI node.
The openstack-ci-mirrors element is updated to export these variables.
elements are updated to depend on it. Tests are restored
Change-Id: I7604fc4d41cb1483be16b8d628a24e8fc764f515
This adds "openstack-ci-mirrors" element which performs various
settings to get builds using local mirrors. As a first step, we
convert ubuntu-minimal jobs
The main trick is that since infra mirrors are created with rerepo
they are not signed (they are recreated, not cloned, and not signing
is seen as a feature in that it deters external use). So we need to
instruct debootstrap to ignore signing and also turn it off for
in-chroot apt. Other than that, the existing DIB_DISTRIBUTION_MIRROR
works to redirect installs.
Remove "restricted" as it's not mirrored, and I don't think we want it
in here by default.
(I think DIB_DISTRIBUTION_MIRROR is a bit of an anti-pattern, because
it leaves the mirrors in the final image -- just because you use them
to build, doesn't mean you want them at runtime). But we don't need
to fix that now, and we don't use any created images.)
This pauses fedora testing until the next change, which moves to using
local mirrors for testing on fedora/centos
Change-Id: I778bd05a1e615c27edf1c9f0a1409119a6b3a850
The gate is currently extremley unstable, and these two issues are
causing most of the problems. We need to commit them atomically so we
can get anything moving again
---
The gate is very unstable downloading the ubuntu tarballs from
upstream at the moment. Move this to ubuntu-minimal which, in a later
change will source files from our local mirror.
We need a caching mechanism for these large files to avoid this
instability. This is future work for the various image-based jobs.
---
Move debian to default skip lists
I don't know if it's mirrors being worked hard for the Stretch
release, but this is constantly failing the gate. I will move this to
the -nv extras job
I am working on having the voting job use local mirrors for
everything. Unfortunately debian infra mirrors don't have stretch yet
and we need to do some fiddling to get "stable" available. Once we
have all this, we can consider making it voting again.
Change-Id: Iaf7b3888ef06c7aef63cbf76a94b33f96bc9c5c2
Debian Stretch released as stable recently, and the init system is
less tightly specified in the base dependencies (for some info, see
[1]). It seems, probably unintentionally, that in the previous
release systemd-sysv was brought in by debootstrap, but that is no
longer happening.
Add systemd as an early dependency of debian-minimal.
Remove the package-installs.yaml as that happens too late (other
things need to know the init system to write out service files, etc
and probe for systemd utils before package-installs). As mentioned, I
do not believe the "only install systemd on testing" idea was actually
working here, because it was being brought in during the initial
debootstrap.
Update some documentation to explain what's going on
[1] https://lists.debian.org/debian-boot/2015/05/msg00156.html
Change-Id: Id67c0cf08728407d234976f9807d3bd71d12f758
Using the newly exposed variables from the prior change, install the
ppc bootloader to the boot partition, not the underlying loopback
device.
Change-Id: I0918e8df8797d6dbabf7af618989ab7f79ee9580
Currently we only export "image-block-device" which is the loopback
device (/dev/loopX) for the underlying image. This is the device we
install grub to (from inside the chroot ...)
This is ok for x86, but is insufficient for some platforms like PPC
which have a separate boot partition. They do not want to install to
the loop device, but do things like dd special ELF files into special
boot partitions.
The first problem seems to be that in level1/partitioning.py we have a
whole bunch of different paths that either call partprobe on the loop
device, or kpartx. We have _all_part_devices_exist() that gates the
kpartx for unknown reasons. We have detach_loopback() that does not
seem to remove losetup created devices. I don't think this does
cleanup if it uses kpartx correctly. It is extremley unclear what's
going to be mapped where.
This moves to us *only* using kpartx to map the partitions of the loop
device. We will *not* call partprobe and create the /dev/loopXpN
devices and will only have the devicemapper nodes kpartx creates.
This seems to be best. Cleanup happens inside partitioning.py.
practice. Deeper thinking about this, and more cleanup of the
variables will be welcome.
This adds "image-block-devices" (note the extra "s") which exports all
the block devices with name and path. This is in a string format that
can be eval'd to an array (you can't export arrays).
This is then used in a follow-on
(I0918e8df8797d6dbabf7af618989ab7f79ee9580) to pick the right
partition on PPC.
Change-Id: If8e33106b4104da2d56d7941ce96ffcb014907bc
The supported ppc ${ARCH} is "ppc64el" (at least in the gate testing
...) so move the file to that, so gets picked up by
block_device_create_config_file
Change-Id: I9273f35cdbfb0a62404461cbc1df9b2a92155fb0
package-installs.yaml is installing python-dev, not python2-dev,
so we need to adjust the mapping accordingly.
In addition, zypper-minimal used an dpkg specific package name,
while there is a SUSE equivalent (and zypper-minimal is anyway
SUSE family specific)
Change-Id: Ia9dd061fa46a514781808d62e5e93b03f75c6745
Ubuntu 12.04 LTS reached its regular End of Life on April 28, 2017.
Depends-On: I5e145095a10db112bb27516bfe652d2cdc052a61
Change-Id: I64af4c5183d77a75dcd062895d19b0a1330c8da8
This element has not been functioning correctly for some time due to
an incorrect path to select-boot-kernel-initrd (should be /usr/local/bin).
The dracut-regenerate element can be used to regenerate dracut ramdisks
and is more flexible than this element.
Change-Id: I33d555ffd4a92b2948b2ea4a66b151f0422ccb8c
Closes-Bug: #1688546
This patch removes the ccache handling from the base element. For
mostly all systems this was never used at all.
This is working towards the removal of the base element from DIB
Change-Id: Ieb16ef612ebd98470993dcd6f55b3a22d37084ba
Signed-off-by: Andreas Florath <andreas@florath.net>
In python3, the standard out data returned by
subprocess.Popen.communicate() will in most cases be bytes rather than a
string and must therefore be decoded.
Without this fix we hit the following error:
TypeError: a bytes-like object is required, not 'str'
Change-Id: I6d75f867ebfdb925970c3397175214b9050d7632
Closes-Bug: #1694463
Currently openSUSE 42.3 has entered feature freeze mode
so it is a good point in time to verify that 42.3 builds
are working successfully. Also test opensuse-minimal for
platforms that support it (need working zypper package)
Change-Id: I4c613e1e68cb7375c29d544bbf70b5da9bf21414
This is consistent with how dpkg based images are configured
and minimizes the nodepool images drastically (avoid installing
texlive for example)
Change-Id: I98fb31bc0e06869e9770fae3dbd62f0d86acb879
This is a follow-on to 57ef187632.
There's two things going on here; DIB_MANIFEST_IMAGE_DIR is *outside*
the chroot on the build host. We copy the files here for posterity, I
guess. MANIFEST_IMAGE_PATH is *inside* the chroot and are the files
we want to ensure are locked to root.
The prior change modified the permissions on DIB_MANIFEST_IMAGE_DIR.
So the first time you build, it works -- then the second time,
assuming you're using the same output filename, it hits the root-owned
manifest directories and causes a build failure.
I have built with this and checked that the manifest files in the
image are locked to root:
$ virt-ls -a ./test.qcow2 -l /etc/dib-manifests
total 32
drwxr-xr-x 2 0 0 4096 May 24 03:39 .
drwxr-xr-x 53 0 0 4096 May 24 03:39 ..
-rw------- 1 0 0 15236 May 24 03:39 dib-manifest-dpkg-test
-rw------- 1 0 0 35 May 24 03:39 dib_arguments
-rw------- 1 0 0 137 May 24 03:39 dib_environment
Related-Bug: #1671842
Change-Id: I08319d0b5fcc461d40fe0be8427dcf0e37ad21e6
Per [1] this is the "official" CDN mirror, which I think is the most
appropriate for the default. I think this addresses the concerns
httpredir service, which I don't think ever quite got out of beta.
[1] https://wiki.debian.org/DebianGeoMirror
Change-Id: I55f2a00b8bbb0f0a20d3be229e4c2c32a7b69057
Instead, either use the bash built-in of type to ensure it exists. Since
which is an external dep, things can fail oddly in a constrained
environment.
Also add a dib-lint test for this.
Change-Id: I645029f5b5bfe1198c89ce10fd3246be8636e8af
Signed-off-by: Jesse Keating <omgjlk@us.ibm.com>
This new element will allow to regenerate dracut
on the produced images, to enable different modules. It
relies on a yaml blob to specify modules and packages
needed. It defaults to installing lvm and crypt.
Change-Id: I292fb70cde41ee6053b7b81a67931bcdaaa6d664
Manifests files can release sensitive information and therefore should
have restrictive permissions.
Change-Id: I64d6c830217a7d8b0172df2dc774079dcd1e2a68
Related-Bug: #1671842
With new block device definition, where content of the image
can be mounted on different partitions, is not enough with
executing setfiles on root directory. Instead of that, expose
all the mountpoints on the image, and apply setfiles on them.
Change-Id: I153f979722eaec49eab93d7cd398c5589b9bfc44
This patch finalizes the block device refactoring. It moves the three
remaining levels (filesystem creation, mount and fstab handling) into
the new python module.
Now it is possible to use any number of disk images, any number of
partitions and used them mounted to different directories.
Notes:
* unmount_dir : modified to only unmount the subdirs mounted by
mount_proc_sys_dev(). dib-block-device unmounts
$TMP_MOUNT_PATH/mnt (see I85e01f3898d3c043071de5fad82307cb091a64a9)
Change-Id: I592c0b1329409307197460cfa8fd69798013f1f8
Signed-off-by: Andreas Florath <andreas@florath.net>
Closes-Bug: #1664924
This is a partial refactor from change
I592c0b1329409307197460cfa8fd69798013f1f8
Change-Id: I8822e68e41c4ebd47eea9ffed4557efc130a7bf7
Co-Authored-By: Andreas Florath <andreas@florath.net>
Some package updates are more complex and require things like --backtrack=99 to
be passed to emerge. We also try harder to ensure the system is in a consistent
state as a last step.
Change-Id: Ia5d3514e8b2a6cb2d656ade997cebb798d9c0a47
With 8e822768f9 we added the ability to
disable the EPEL repository, however we need yum-utils to use
yum-config-manager.
Change-Id: Iea445f84494fd9a89fd93e9b35f920eb5e55211d
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
Recent changes in the default configuration of cloud-init in Ubuntu
cause warnings when the Ec2 datasource is used on non-Amazon clouds,
see https://bugs.launchpad.net/cloud-init/+bug/1660385
We explicitly select the previous behavior when an Ec2 datasource is
desired.
Change-Id: Iebad8f6c0017fe08013dd5fe667c6132158b71cd
Closes-bug: 1683038
We only need dib-python when we build the image, no need to leak it to
the final product. Remove it in cleanup.d outside the chroot so
nothing can be using it.
Change-Id: I1e229caad7968fb3ab8e44ecdda427e174088d2d
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
If DIB_PYTHON_VERSION is < 3 on the !redhat path, that means we're on
an older platform that may not have python3-virtualenv packages. Skip
install.
Ensure the order of operations happens by forcing the installs
Also add a note about limited platform support (patches welcome :)
Change-Id: I18412767f0ebf946d557a0a126285369e96af159
Recent changes in project-config have shown that we leave the system
in an inconsistent state when installing from source. On fedora, we
will have installed the python2 packages, but then used $DIB_PYTHON to
install python3 pip from source!
This tries to clarify the situation. As described in the document,
with package installs, we just install the $DIB_PYTHON packaged
versions.
Source installs want to take over the global namespace. This is the
price you pay for running the latest versions outside package managers
:) The only sane thing seems to be for us to normalise python2 &
python3 versions of pip, setuptools and virtualenv and then hacking
things such that "/usr/bin/pip" and "/usr/bin/virtalenv" remain
defaulted to python2 versions.
Documentation is added
Change-Id: Ibc6572b89e256d1f48b7fe7c672b8b9524dc704f
Currently we install pip/virtualenv with "/usr/local/bin/dib-python".
This means that every time you create a virtualenv, the python
interpreter inside it is called "dib-python" which is confusing.
Add an env var DIB_PYTHON that points directly the to interpreter
available during build, for use when running scripts.
Change-Id: I88ad3c9eb958d58db4631d9b27bc2c592f970345
Apt gets confused if it talks to a mirror with an older index than the
index currently cached by apt. This can happen when image builds use a
newer index than the booted image. Avoid these problems entirely by
removing those index caches at the end of image building.
Change-Id: I245d516ee8a44831b2c29612b782bad555c48a3f
Co-Author: Clark Boylan <clark.boylan@gmail.com>
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
This patch removes three nearly-copies of debootstrap documentation
and fixes some documentation aspects.
Change-Id: Ief7794f5c1abad73788c063af6c862472cd34744
Signed-off-by: Andreas Florath <andreas@florath.net>
The current output for package-installs-v2 is inscrutable [1]
The problem starts with process_output() which is not capturing
stderr. This means that any stderr output is dislocated from any
stdout output around it. This is *really* confusing as you get a
bunch of seemingly meaningless stderr output from any calls before you
see any stdout (e.g. in [1] you can see random yum error output that
should have been with the yum call)). The simplest thing to do is to
redirect stderr to stdout which keeps everything in sync.
This causes a slight problem, however, because pkg-map outputs both
status information and errors on stderr. To work around this but
maintain compatibility, we add a "--prefix" argument that prepends
mapped packages from pkg-map with a value we can match on. The
existing status/debug output from pkg-map is low-value; modify the
call so that it will be traced only at higher debug levels (e.g. -x
-x).
The current loop is also calling pkg-map for every package in every
element (this is why in [1] the same message is repeated over and
over). This is unnecessary; it only needs to pkg-map once for each
element, giving the package list as the arguments. Create package
lists by element and pass those to pkg-map.
As a cleanup, there is no point in printing e.output if the
process_output fails for the install because we are already tracing
it; i.e. the output, even for failures, is already in the logs.
Printing it again just duplicates the output.
[2] is an extract showing what I feel is a much more understandable
log output for a fairly complex install.
[1] http://paste.openstack.org/show/595118/
[2] http://paste.openstack.org/show/595303/
Change-Id: Ia74602a5d2db032a476481caec0e45dab013d54f
The dib-run-parts element was copying our internal version of
dib-run-parts into /usr/local/bin to be used running scripts inside
the target chroot. However, it never cleaned up after itself. This
means all images were left with an unmanaged local install of
dib-run-parts.
This copies dib-run-parts into the hooks directory of the chroot and
runs it from there. It is cleaned up automatically on the exit path.
The dib-run-parts element is no longer required and it has been
removed from all dependencies. It is left with a deprecation notice
in the README. For compatability we convert it to simply install
dib-utils.
Codesearch shows no users depending on this unintentional implicit
install. Note os-refresh-config depends on dib-utils and thus will
have an explicitly installed version.
Partial-Bug: #1673144
Change-Id: Ia2e96c00a4246c04beb96c17f83b8aefb69219ca
It was an oversight during v2 development for dib to start providing
dib-run-parts. The intention was for dib to use a vendored
dib-run-parts directly from $_LIB and have no dependencies on
dib-utils at all. By exporting dib-run-parts, we created an
unintentional conflict with the dib-utils package which provides the
same script.
Tools that depend on dib-utils are unaffected by this
(os-refresh-config).
The only tool that installs diskimage-builder and then assumes
dib-run-parts is available in the path is instack. I have proposed
Ibfe972208df40fa092b11b5419043524c903f1b4 to modify that to use our
internal version.
Change-Id: I149c345d38d761a49b3a6ccc4833482f09f1cd05
Add DIB_EPEL_DISABLED flag that allows installation of the EPEL repo,
but to have it disabled by default. This will help when you have
unavoidable EPEL dependencies, but want to make sure you only pull
specific things in with "--enablerepo" calls when installing those
packages.
Change-Id: Iedf6167a7cd69418255ebbee095aea04c50d73fd
A code-block in README of rhel7 element is not rendered as expected.
This patch fixes it to be rendered correctly.
Change-Id: Ie8f4c05edd1dd93314290682e4b2734622894e15
zypper on non-suse hosts is not parsing the pattern repodata because
those are marked as an inofficial extension to the repomd specification.
This is not a big issue as there is meanwhile in newer openSUSE
distributions a pattern *package* that depends on the same packages like
the pattern would do, so we can just replace it with that.
Change-Id: I0c8f713075bd7e5bf1d425f81933b4666654add7
Depends-On: I34e98f0f7693859ed05011b008334628adff612f
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
When a glean is running on centos with multiple NICs, it will try to
systemctl enable network.service multiple times for each interface.
Because of systemd magic, it is possible for the systemctl command to
fail in a race condition.
glean shouldn't be enabling network.service during boot in
pre-networking phases (Ib2b618dd975ca44e9c6b0a2c9027642ffc46b9b0). I
have proposed I8319f1ed6498a9d447950c2b4b34bca59e7b97e4 to remove this
and document the behaviour.
This also bring across suse's version
(I20bffabd333ea290d8712ec2a467f2b2d5678f3a)
Change-Id: I89d9443cb61e287bd0d9da3f48315272218ee335
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
diskimage-builder usually provides defaults that work out of the box.
One default that does not work outside of x86 land is Ubuntu distro
mirror url. Considering there are only two valid default options, we can
automatically choose a better default.
This patch changes behavior only for architectures known to be using
http://ports.ubuntu.com/ubuntu-ports. All others still would use
http://archive.ubuntu.com/ubuntu as default. It provides some guarantee
that we do not introduce a regression.
Change-Id: If95a64bac0c88f30736da4bae7f1fdce126c0bf6
We somewhat discussed skipping qcow2 generation previously in
I9372e195913798a851c96e62eee89029e067baa1. As recent issues with PPC
testing have shown, we are not actually testing the "vm" element and
hence the bootloader path in the functional tests.
I don't think we need to test this on every element; it overlaps
somewhat with the testing done by the nodepool jobs which build full
images and boot them. I also didn't want to introduce a separate run
for this. Thus it seems valuable to at least have one element
enhanced to do this installation and conversion in our default tests
for basic sanity.
This disables qcow generation by default, as per the other change, but
allows an element to drop a file that will override the output
formats. The Xenial element is modified to produce a qcow2 using
this, and also introduces a dependency on the "vm" element so it tries
to install the bootloader.
We now exit if the .qcow2 fails to build as well.
Change-Id: I1a6acefe52f8c696c39b2d592fdc7ae32a87e6fe
Add a default PPC block-device layout. I've extracted this into
separate yaml files for ease of editing and to facilitate things like
longer comments.
This is not sufficient to get PPC images working, but it is required.
Change-Id: I09e5d1ed92260bdb632333f5203dd7e70d512dc8
Sphix 1.5 (I9e7261c4124b71eeb6bddd9e21747b61bbdc16fa) includes
"warning-is-error" which supersedes pbr's warnerrors. Enable this and
fix up the resulting failures
- trailing lines for lists in element_deps directive
- missing README's that are linked
- syntax error and highlighting in building instructions
Change-Id: I6549551b4a9bf47076c9811a7a38a666cbea2a50
99-squash-package-install in the package-installs element does not
know which python environments the requirements were installed into.
This can cause it to select the wrong python to run the
package-installs-squash script.
Co-Authored-By: Adam Harwell <flux.adam@gmail.com>
Change-Id: I5fab0e192c3a2dad8f60e821c184479e24e33bcd
On Debian Jessie and Debian Stretch systemctl is in /bin.
If the package systemd-sysv is not installed the script
dib-init-system did not find the init system.
This patch fixes the problem: it also looks in /bin
for systemctl and if found decides for systemd.
Change-Id: I5a18052a070bad5e16b14672237a1e2b38513949
Signed-off-by: Andreas Florath <andreas@florath.net>
The architecture-emulation-binaries element does for a limited number
of architectures on Ubuntu exactly the same as `qemu-debootstrap`.
`qemu-debootstrap` comes in the qemu-user-static package that is
anyhow used when cross-building an image.
This patch replaces the complete architecture-emulation-binaries
element with a call to qemu-debootstrap.
Change-Id: Ib9667307bfd3ff7592444a2ec5b04aa5365a1872
Signed-off-by: Andreas Florath <andreas@florath.net>
Depending on the types of deployment (security, nfv...) some extra
kernel flags are needed on the images. This change exposes the
DIB_BOOTLOADER_DEFAULT_CMDLINE parameter, defaulting to the
existing 'nofb nomodeset vga=normal', that will allow to modify
these boot params.
Change-Id: I67d191fa5ca44a57f776cb9739a02dd71212969c
Closes-Bug: #1668890
Looks that the special handling for Ubuntu is not needed any longer
(its a pity that there are no detailed comments...).
The grub2 element is a second implementation of the bootstrap element
- but because there are some features that come only here, e.g. efi
boot, it should be working as long as this is not implemented in the
bootloader element.
Change-Id: I74269116ea30b84f3259805720d5cd1616f960c5
Signed-off-by: Andreas Florath <andreas@florath.net>
Closes-Bug: #1627402
Currently there is no description of dependencies in the generated
documentation of the elements: therefore a user of an element does not
know which other elements are automatically included and e.g. which
configuration options are available. In addition there are some
copy&pastes of parts of the README.rst scattered thought different
Ubuntu and Debian specific elements.
This patch adds a semi-automatic generation of dependency information
of all elements. Nevertheless these are not automatically included.
The author of the element's README.rst can decide if and where the
dependency information should appear and can use the descriptor
.. element_deps::
for this.
This patch adds the dependency information for some Debian and
Ubuntu patches - and creates the base for later removing the
duplicated parts.
A call is added to element_dependencies._find_all_elements() to
populate reverse dependencies for Element objects.
(This is a reworking of I31d2b6050b6c46fefe37378698e9a330025db430 for
the feature/v2 branch)
Change-Id: Iebb83916fed71565071246baa550849eef40560b
With the old configuration structure it was only possible
to use one image and one partition layout. The new
block-device configuration uses a list at top level;
therefore it is possible to use multiple instances
of each element type.
Change-Id: I9db4327486b676887d6ce09609994116dbebfc89
Signed-off-by: Andreas Florath <andreas@florath.net>
During the creation of a disk image (e.g. for a VM), there is the need
to create, setup, configure and afterwards detach some kind of storage
where the newly installed OS can be copied to or directly installed
in.
This patch implements partitioning handling.
Change-Id: I0ca6a4ae3a2684d473b44e5f332ee4225ee30f8c
Signed-off-by: Andreas Florath <andreas@florath.net>
Move dib-run-parts from dib-utils into diskimage-builder directly.
For calling outside the chroot, we provide a standard entry-point
script. However, as noted in the warning comment, the underlying
script is still copied directly into the chroot by the dib-run-parts
element. I believe this to be the KISS approach.
This removes the dependency on dib-utils. We have discussed this
previously and nobody seemed to think retiring dib-utils was going to
be an issue.
This also updates the documentation to not mention dib-utils, or using
disk-image-create via $PATH setup, but rather gives instructions on
installing from pip with a virtualenv.
Change-Id: Ic1e22ba498d2c368da7d72e2e2b70ff34324feb8
Currently we have all our elements and library files in a top-level
directory and install them into
<root>/share/diskimage-builder/[elements|lib] (where root is either /
or the root of a virtualenv).
The problem with this is that editable/development installs (pip -e)
do *not* install data_files. Thus we have no canonical location to
look for elements -- leading to the various odd things we do such as a
whole bunch of guessing at the top of disk-image-create and having a
special test-loader in tests/test_elements.py so we can run python
unit tests on those elements that have it.
data_files is really the wrong thing to use for what are essentially
assets of the program. data_files install works well for things like
config-files, init.d files or dropping documentation files.
By moving the elements under the diskimage_builder package, we always
know where they are relative to where we import from. In fact,
pkg_resources has an api for this which we wrap in the new
diskimage_builder/paths.py helper [1].
We use this helper to find the correct path in the couple of places we
need to find the base-elements dir, and for the paths to import the
library shell functions.
Elements such as svc-map and pkg-map include python unit-tests, which
we do not need tests/test_elements.py to special-case load any more.
They just get found automatically by the normal subunit loader.
I have a follow-on change (I69ca3d26fede0506a6353c077c69f735c8d84d28)
to move disk-image-create to a regular python entry-point.
Unfortunately, this has to move to work with setuptools. You'd think
a symlink under diskimage_builder/[elements|lib] would work, but it
doesn't.
[1] this API handles stuff like getting files out of .zip archive
modules, which we don't do. Essentially for us it's returning
__file__.
Change-Id: I5e3e3c97f385b1a4ff2031a161a55b231895df5b
