We need to handle openSUSE Leap 15 when installing pip and virtualenv
packages. This fixes the following problem when the pip-and-virtualenv
elements is used:
2018-05-31 09:42:12.014 | + [[ opensuse = opensuse ]]
2018-05-31 09:42:12.014 | /tmp/in_target.d/install.d/04-install-pip: line 57: packages: unbound variable
Change-Id: Id7911b0a0836fa8dcc003e23fa515b78fba67126
Redirecting our output through outfilter.py is inherently a bit racy,
since the disk-image-create process will exit, and then you might get
outfilter.py flushing any remaining output as it closes.
On an interactive prompt this might lead to final output overwriting
the prompt, etc. This can be a bit confusing when you start running
things in a loop.
If we save the original fd, then on the exit path close the redirected
fd's and wait a little bit for final output (as a result of the
close), we get a more consistent output.
Change-Id: I8efe57ab421c1941e99bdecab62c6e21a87e4584
Strip everything before "site-packages" in the output filename for the
PS4 prompt. This makes the line in debug logs significantly shorter
as we don't have the full virtualenv path every single time. The
important thing -- the file being called in the lib/ dir, is retained.
Change-Id: I00706b6f6c0425c7795f997c08ceda3374dc84b5
When switching to using log-file capture, we're getting
[gentoo/build-succeeds] outfile.write(ts_line.encode('utf-8'))
[gentoo/build-succeeds] UnicodeEncodeError: 'utf-8' codec can't
encode character 'udcc5' in position 59: surrogates not allowed
Use surrogateescape [1] on the output to avoid this
[1] https://www.python.org/dev/peps/pep-0383/
Change-Id: I2c2c537296edfa5a8fe661a41bd5bfb3bfcf57e3
Patch allows to rebuild arbitrary images, which location, filename and
sha256sum are specified in variables, not only hardcoded $DIB_RELEASE/current.
Change-Id: I05418932a0c40d885fe00a49f1f49d7e86c67518
Add a bionic test in replacement of trusty. We are already building
bionic images in the gate, so this seems like a good time to switch.
Change-Id: I20d4c25e9b79e7326c86767c36be8615ba0888a3
Removing no longer working and no longer maintained ubuntu-core element, which
intent is unclear, and not documented.
Change-Id: Id847591d04fd7cd32c8903967da01ee0d303b267
Closes-Bug: 1771614
Without this change DIB appends a second command line entry to the GRUB
config. This causes the original command line entry to be ignored
when Linux is booted.
The expected behaviour is that DIB appends to the existing entry as
it does for Ubuntu and SUSE.
Following discussion on the review, this also removes the distro specific
switch statement, as update-grub just calls grub-mkconfig, meaning that
there was nothing distro specific in the first place.
Change-Id: I2298675dda1f699c572b3423e7274bc8bd7c1c9d
Closes-Bug: #1771366
Since CentOS-7.5, a new yum variable is needed for SIG repositories.
This change replicates the %post task of the centos-release package
to setup the contentdir yum var. This should fix issues when
repository url uses $contentdir and yum fail with:
http://mirror.centos.org/%24contentdir ... [Errno 14] HTTP Error 404 - Not Found
For more details see:
https://lists.centos.org/pipermail/centos-devel/2018-March/016542.html
This change also drops support for fedora without dnf.
Change-Id: I1819a48b94670577b0c5e29b24cebfb20ea07d28
This commit addresses the issue described in bug 1768354 when using the
apt-sources element and adding a key to a custom repo, subsequent deb
package installs fail due no update of the repo before package install
Change-Id: I968b3422fab2fb2305426d49215391d8ba7499df
Closes-Bug: 1768354
The pip-and-virtualenv element provides pip inside the created images.
When this pip is used inside the chroot of the image it, by default, creates
a cache directory with the http packages and the resulting wheels.
In the case of the octavia ubuntu-minimal image this cache is using ~50MB of
cache that is not needed after the image finished building.
This package creates a finalise.d task that removes the cache from the
default location.
Change-Id: I4715437b068d04993ef755bd1e27963db1d22417
When RDO projects repository is installed, the python-setuptools
package is obsoleted by python2-setuptools, this makes the install-pip
script failed:
Package python-setuptools-0.9.8-7.el7.noarch is obsoleted by
python2-setuptools-22.0.5-1.el7.noarch which is already installed
Then the "rpm -ql python-setuptools | xargs rm -rf" exit 1. Check if
we have a record of the updated package obsoleting then old one; if
so, use it.
Change-Id: I2b0051bd9e81908c187098a7b82e120b999b111d
We no longer install wget / yum-utils for centos-minimal, this fixes
that.
Change-Id: I8d89026bd48cf7398cc1cbe41e3b7f00f682dbb8
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
This currently breaks glean on rackspace, revert until we can figure
out why that is.
This reverts commit 43bc352c59.
Change-Id: Iae88a3b0457bab0b8f0fd1febf58732ca95e5dc9
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
The release of pip10 has shown up a few issues here
Firstly, pip10 now refuses to overwrite distutils installed packages,
which includes "python-virtualenv" on centos. History has shown us
that we want the packages installed and overwritten, to avoid the
packages coming back and messing things up.
Pre-install all the packages, then list the files in the packages with
"rpm" directly and remove them. This way pip is happy to install.
We need to take better account of the package names for this; on
Fedora things have switch to "python2-virtualenv" instead of
"python-virtualenv" and we can't use an alias to list the package
contents.
This also highlighted that python2-pip is in EPEL for centos, so
enable that when we install it. Make the epel element a no-op for non
centos/rhe distros.
There is a related change in recent fedora that python3 now installs
binaries into /usr/local/bin. There are commented swizzles in here to
ensure we retain the status quo of "pip" and "virtualenv" both being
python2 based, with the python3 versions being called explicitly
"pip3" and "virtualenv3" respectively.
Change-Id: I2ffdd9f615ae6b00428c17249e4f216774991b99
We added this sed in I422490ebe9a9c655552685bc2ff342d288335a9c to
avoid installing python2 packages on python3-only systems and thus
dragging in all of python2.
We made a similar change to python-pip in
I7d8ba9300039cce90965410a4e16ca9e711904c3; however we realised that
the gate (and other consumers) were relying on this element having
installed the python2 & 3 packages for consistency -- otherwise jobs
would install the python-pip packages and overwrite the
pip-from-source and mess everything up. We reverted that in
I419dbdf4682394db68974944af1e5c432f3e0565 and added some clearer notes
that this element brings in python2 & 3, and if you want something
that doesn't do that then this element isn't for you.
However, we never fixed up the virtualenv package install -- currently
our Xenial images have a global virtualenv installed from source, but
the python-virtualenv packages aren't installed. Thus if a job does
"apt-get install python-virtualenv" it overwrites the from-source
virtualenv with older parts and again messes everything up.
Probably most jobs just call "virtualenv" and assume it is there;
however in bringing up some rspec test for puppet I have hit this
issue as some modules specify dependencies on the virtualenv packages.
Thus install the python-virtualenv AND python3-virtualenv packages in
this element.
Change-Id: Ia84c38dc3c40a6080e144b563e10abca7dac2881
The behavior of test -e and [[ -e against broken symlinks is to fail
even if the symlink exists. However we want to test if the link exists
or if there is a file in that location. Therefore switch from test -e to
test -L and test -f to check if the file or link exists regardless of
link target validity.
Change-Id: I84a9b6731eccf950707be50aef464a2de1e33e8e
On initial boot when networking is brought up by cloud-init this
is the timeout that dhclient adheres to. Centos configures
"timeout 300" (for an EC2 bug) in their cloud image, which results
in a 5 minutes delay to boot in cases where no dhcp available (e.g. IPv6
SLAAC). To reduce this boot delay and to provide consistency with
places where we have set other dhcp timeouts set this to DIB_DHCP_TIMEOUT.
Change-Id: I119a002070501c3dfe7c6730b07ee25f422b85b0
Related-Bug: #1758324
systemd-resolved has a new behaviour in bionic, in that if there is no
/etc/resolv.conf file when it installs, it assumes it is a fresh
system and makes /etc/resolf.conf a symlink into its compatability
files.
dib ends up saving & restoring whatever /etc/resolv.conf we have after
the inital chroot creation, which may not be what we want -- in the
above case it restores the system-resolved symlink. For
openstack-infra, we use unbound and want simply "127.0.0.1" in a
/etc/resolv.conf file [1].
Formalise the ability to save specific contents into the final image.
Add documentation, and a note in the code that it's an external
interface.
I would have preferred to namespace the .ORIG file with DIB_ or
similar, but this unofficial interface has already escaped into the
wild. Leave it as is for simplicity.
[1] Note that systemd-resolved will obey /etc/resolv.conf as you would
expect, if file exists.
Change-Id: Ie0e97d8072e2b21a54b053fa6fb07b62960c686d
We exit in several places and don't restore tracing. Previously in
nodepool we relied on the default fallback, which did restore the
tracing. Since we now use the MBR config file, we take the different
exit path without it and the debugging output is incomplete.
Change-Id: I586fc95517926025705ce376ec5c4aaf4122773f
Many elements install additional distribution packages.
In addition the user can provide a set of packages to be installed
via the '-p' switch.
Some of them influence the boot process and therefore the initramfs
needs to be updated. Because the package manager during the image
creation process is configured not to run package scripts, this needs
to be done explicitly.
This issue was found during development and debugging of the
block-device LVM plugin: Even when the e.g. the lvm2 package
was installed in the image, it was missing in the initramfs
because of the missing update.
Change-Id: I7c92033b3ca80cdd23d081002059d83ca3f53bdb
Signed-off-by: Andreas Florath <andreas@florath.net>
Default the GENTOO_PORTAGE_CLEANUP to True. By default we should not
ship package info, this bloats the image and is usually outdated by the
time it'd be consumed.
Change-Id: I14c2530d91807cbc6a3806e01c7e4f6f472b190d
The debian element depends on debian-minimal now which provides
operating-system. This means that the debian element can no longer
provide operating-system and doing so results in an error when using the
debian element.
The fix is simple just rely on the fact that debian-minimal provides
operating-system and remove this element-provides from debian.
Fixes-Bug: 1758000
Change-Id: I524feeb82c19046ec987eb1186c7f4568309e559
The devuser element can set up passwordless sudo, which requiers the
/etc/sudoers.d directory, which requires the sudo package, so we ensure
the sudo package is installed.
Change-Id: I80d6c669d4ac0d97b49d01cb621bf05b8e7f8ef1
There was a typo in I6b819a8071389e7e4eb4874ff7750bd192695ff2 that
modified this default partition type from "0x83" to just 83. We are
now seeing failures relating to this as sfdisk checks for a "disk
manager" when it see Id 0x53 (== 83)
Device Boot Start End Blocks Id System
/dev/vda1 * 2048 26664575 13331264 53 OnTrack DM6 Aux3
Restore to 0x83
Change-Id: Ib43038d2d740fbe01a21a13dd56367f7bc97f869
Hpsum utiltity of proliant-tools requires net-tools to be installed
as part of base image. This commit adds support for installation of
net-tools for all distros.
Change-Id: I2a1e81059ed1aee975db78cfa5e61bbf1b98e06f
Closes-bug: 1751777
When using the package-installs element there can be some encoding
problems if the package installation emits unparsable output
[1]. However in this case we just want to forward the output to the
console which normally can handle this correctly. In order to fix this
switch off universal_newlines processing such that we just operate on
bytes.
Further we have to decode the lines without setting the locale and
ignoring errors. This is required because print encodes without
setting the locale and thus we need to filter/modify the stream such
that it doesn't crash.
[1] Traceback:
2018-03-01 09:58:00.515 | Traceback (most recent call last):
2018-03-01 09:58:00.515 | File "/usr/local/bin/package-installs-v2", line 137, in <module>
2018-03-01 09:58:00.515 | main()
2018-03-01 09:58:00.515 | File "/usr/local/bin/package-installs-v2", line 130, in main
2018-03-01 09:58:00.515 | process_output(install_args, follow=True)
2018-03-01 09:58:00.515 | for line in iter(proc.stdout.readline, ''):
2018-03-01 09:58:00.515 | File "/usr/lib/python3.5/encodings/ascii.py", line 26, in decode
2018-03-01 09:58:00.515 | return codecs.ascii_decode(input, self.errors)[0]
2018-03-01 09:58:00.515 | UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 34: ordinal not in range(128)
Change-Id: Ie4af9b4523459a630cfb98d09093bfe9ef7aa61e
Currently rhel7 image creation fails because it tries to copy
default bootloaders which is ubuntu way. This commit updates `iso`
element to correct the path of bootloaders required for rhel image.
Change-Id: I526d75b2db609fc77be0fc778b4d00f2d3df38ec
Closes-bug: 1750725
For 'satellite' mode of registration, rpm for rhel SSL certificate is
hard coded to 'katello-ca-consumer-latest.noarch.rpm'. This commit adds
functionality that provides an option to set this as defined in their
satellite server.
Change-Id: Ib176cfa209f5ac8a4b5da71419327b4237330904
Closes-Bug: 1749947
Install hwe kernel for ubuntu-minimal. As noted this is currently
Xenial specific; we need this for initial bring-up so let's tackle
future releases as things progress.
Ensure we use ttyAMA0 for arm64 console too.
Change-Id: Ic607cf8369666dc24929aff6f2ef8a72e7980599
In the prior change we added block-device-[mbr|gpt|efi] elements to
create appropriate disk-layouts.
This adds an environment flag to each so the bootloader can install
the right thing. The EFI install path is updated to work with this
(this part a copy of I572937945adbb5adaa5cb09200752e323c2c9531)
We do some basic sanity checking in the block-device elements;
e.g. mbr is not suitable for aarch64, and efi is not suitable for
power.
This updates the bootloader to install EFI where appropriate
Co-Authored-By: Marcin Juszkiewicz <marcin.juszkiewicz@linaro.org>
Change-Id: Ib80acbfd9a12efd976c3fa15a5d1081eb0799305
This moves the block-device default out of the "vm" element and into a
selection of other elements. There's "mbr" which retains the status
quo. There's an EFI version that has the boot/grub partitions as
required. In between there's the GPT only version, which is useful
for architectures like power without EFI, but still want possible
larger disks using GPT.
Change-Id: I4a566a97d073fc0dda0ab2494ac988fe015800a9
The current check only validates that an element that specifies
"element-provides" doesn't conflict with a "real" element. We also
want to check this against the provides of other elements.
A real example is with a "block-device" element. There is no actual
"block-device" element; we can have multiple elements provide it
(block-device-[gpt,mbr,efi], say) but we only want one of them at a
time.
Update the unit test for this.
Change-Id: I59d4aa5f6f09e2892b213e154befa10d85e95ca3
This adds support for a GPT label type to the partitioning code. This
is relatively straight-forward translation of the partition config
into a sgparted command-line and subsequent call.
A unit test is added based on a working GPT/EFI configuration and the
fedora-minimal functional test is updated to build a single-partition
GPT based using the new block-device-gpt override element. See notes
in the sample configuration files about partition requirements and
types.
Documentation has been updated.
Co-Authored-By: Marcin Juszkiewicz <marcin.juszkiewicz@linaro.org>
Change-Id: I6b819a8071389e7e4eb4874ff7750bd192695ff2
In slow networks like Infiniband it takes much time for the
interface to get the carrier. This patch enables this service
to run more then 20 seconds and limited by DIB_DHCP_TIMEOUT.
Change-Id: I8a6015567ac25e37b5a5aba4b1fda71170cc144a
Like we did in https://review.openstack.org/475206 we need to install
systemd sooner because of the new world order of containers.
Change-Id: Ia60d751fee3af6f8d72ad664107acb337360feca
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
As described, we want to set the default label for XFS disks to the
shorter value.
For example, you hit this when setting the old FS_TYPE environment
variable to 'xfs' (which sets the "root-fs-type" parameter, which gets
passed through to 'type'; but does not set a default label).
Change-Id: I41dce6e25766562db4366021309b8c2b74a8ab80
Closes-Bug: 1742170
While Debian-based distros use the label of ppc64el for ppc64 little
endian, Fedora uses ppc64le.
The ironic-agent was doing arch specific package install of lshw over
dmidecode for ppc64 and ppc64el but was attempting to install dmidecode
on Fedora ppc64le which caused the test to fail due to a missing
package.
This change just adds ppc64le to the arch-specific package installation
description for the ironic-agent element.
Change-Id: I38c3c1480bbbb2df817856614e6b740a0c02723a
Closes-Bug: 1744944
This updates diskimage-builder to support current Fedora releases (26
and 27) and removes support for Fedora 25 which is EOL as of December
12, 2017.
Change-Id: I227a607c6c468cc8b7bb154a189e9c8ce2021192
This small change avoids running fstrim on vfat partitions.
The mount order test-case has been updated to also test the mkfs
creation components, and the input config modified to have a vfat
partition to cover this path.
Change-Id: I8952e748d4bdc12a5769706de9057c1e97d95e37
The installed pip can be an older version which does not support
the -c argument. Therefore, upgrade pip before using -c.
Change-Id: If18d8ea822a62c8551c9c4d47354d58b0299fed2
Closes-Bug: 1744403
This reverts commit ab89c7d69c.
This commit checked for DIB_PYTHON_VERSION and only installed the v3
packages. This is unfortunately backwards-incompatible, as consumers
such as the openstack gate are relying on this package installing pip
& virtualenv packages for python2 AND python3.
This was sort-of expressed in the docs, where it discusses what the
resulting setup of the system will be, but I've added a note to make
it clearer.
If we want to change this, I think we'll need either a new element, or
a non-defaulting flag.
Change-Id: I419dbdf4682394db68974944af1e5c432f3e0565
It turns out make has always been a tacit dependency of openssl as it
ships a Makefile for certificates [1]. This just recently changed to
be a hard dependency in F27, so this now fails as openssl is a
dependency of protected packages such as dnf. Since it's always been
wrong to remove it, we take it out of the purge list.
[1] https://bugzilla.redhat.com/show_bug.cgi?id=783446
Change-Id: I69efb3a56878ab97c4587bbbf5356bea752f2846
There's a patch in flight in ironic-python-agent to switch the
default hardware manager to use lshw instead of dmidecode. [0]
This would require lshw to be installed regardless of
architecture. This patch removes the architecture rules from
lshw in the package-installs list.
[0] Ie370331df6bb5ef131c5cb60f458877e2a7ad71a
Change-Id: Idaf05b8efce28cd0cbf339cf693db4f55a693d9b
Partial-Bug: #1715790
zypper only supports the --no-recommends option during installs, giving
the option during removals results in an error.
When setting ACTION=remove, remove --no-recommends from EXTRA_ARGS, and
set --clean-deps to also remove no-longer-needed dependencies.
Rename EXTRA_ARGS to ACTION_ARGS for increased readability.
Change-Id: Ifbd168992b1a20658b6b4a99ba175234f6c78f6d
When "epel" element is used during a build process
with "rhel7" distribution, the build failed
because the "epel-release-7*" package cannot be
installed.
The reason is because the URL is not correct, it
should be:
URL=$BASE_URL/$RELEASE/x86_64/Packages/e/
Change-Id: I90c26892361f7611645b85f2eddc949b2f0d76fc
Closes-Bug: #1735547
