The release of pip10 has shown up a few issues here
Firstly, pip10 now refuses to overwrite distutils installed packages,
which includes "python-virtualenv" on centos. History has shown us
that we want the packages installed and overwritten, to avoid the
packages coming back and messing things up.
Pre-install all the packages, then list the files in the packages with
"rpm" directly and remove them. This way pip is happy to install.
We need to take better account of the package names for this; on
Fedora things have switch to "python2-virtualenv" instead of
"python-virtualenv" and we can't use an alias to list the package
contents.
This also highlighted that python2-pip is in EPEL for centos, so
enable that when we install it. Make the epel element a no-op for non
centos/rhe distros.
There is a related change in recent fedora that python3 now installs
binaries into /usr/local/bin. There are commented swizzles in here to
ensure we retain the status quo of "pip" and "virtualenv" both being
python2 based, with the python3 versions being called explicitly
"pip3" and "virtualenv3" respectively.
Change-Id: I2ffdd9f615ae6b00428c17249e4f216774991b99
We added this sed in I422490ebe9a9c655552685bc2ff342d288335a9c to
avoid installing python2 packages on python3-only systems and thus
dragging in all of python2.
We made a similar change to python-pip in
I7d8ba9300039cce90965410a4e16ca9e711904c3; however we realised that
the gate (and other consumers) were relying on this element having
installed the python2 & 3 packages for consistency -- otherwise jobs
would install the python-pip packages and overwrite the
pip-from-source and mess everything up. We reverted that in
I419dbdf4682394db68974944af1e5c432f3e0565 and added some clearer notes
that this element brings in python2 & 3, and if you want something
that doesn't do that then this element isn't for you.
However, we never fixed up the virtualenv package install -- currently
our Xenial images have a global virtualenv installed from source, but
the python-virtualenv packages aren't installed. Thus if a job does
"apt-get install python-virtualenv" it overwrites the from-source
virtualenv with older parts and again messes everything up.
Probably most jobs just call "virtualenv" and assume it is there;
however in bringing up some rspec test for puppet I have hit this
issue as some modules specify dependencies on the virtualenv packages.
Thus install the python-virtualenv AND python3-virtualenv packages in
this element.
Change-Id: Ia84c38dc3c40a6080e144b563e10abca7dac2881
The behavior of test -e and [[ -e against broken symlinks is to fail
even if the symlink exists. However we want to test if the link exists
or if there is a file in that location. Therefore switch from test -e to
test -L and test -f to check if the file or link exists regardless of
link target validity.
Change-Id: I84a9b6731eccf950707be50aef464a2de1e33e8e
Create a tox environment for running the unit tests against the lower
bounds of the dependencies.
Create a lower-constraints.txt to be used to enforce the lower bounds
in those tests.
Add openstack-tox-lower-constraints job to the zuul configuration.
See http://lists.openstack.org/pipermail/openstack-dev/2018-March/128352.html
for more details.
Change-Id: I911c66b2e9971a3e134c482a4b4ffa529d358a76
Depends-On: https://review.openstack.org/555034
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
On initial boot when networking is brought up by cloud-init this
is the timeout that dhclient adheres to. Centos configures
"timeout 300" (for an EC2 bug) in their cloud image, which results
in a 5 minutes delay to boot in cases where no dhcp available (e.g. IPv6
SLAAC). To reduce this boot delay and to provide consistency with
places where we have set other dhcp timeouts set this to DIB_DHCP_TIMEOUT.
Change-Id: I119a002070501c3dfe7c6730b07ee25f422b85b0
Related-Bug: #1758324
systemd-resolved has a new behaviour in bionic, in that if there is no
/etc/resolv.conf file when it installs, it assumes it is a fresh
system and makes /etc/resolf.conf a symlink into its compatability
files.
dib ends up saving & restoring whatever /etc/resolv.conf we have after
the inital chroot creation, which may not be what we want -- in the
above case it restores the system-resolved symlink. For
openstack-infra, we use unbound and want simply "127.0.0.1" in a
/etc/resolv.conf file [1].
Formalise the ability to save specific contents into the final image.
Add documentation, and a note in the code that it's an external
interface.
I would have preferred to namespace the .ORIG file with DIB_ or
similar, but this unofficial interface has already escaped into the
wild. Leave it as is for simplicity.
[1] Note that systemd-resolved will obey /etc/resolv.conf as you would
expect, if file exists.
Change-Id: Ie0e97d8072e2b21a54b053fa6fb07b62960c686d
We exit in several places and don't restore tracing. Previously in
nodepool we relied on the default fallback, which did restore the
tracing. Since we now use the MBR config file, we take the different
exit path without it and the debugging output is incomplete.
Change-Id: I586fc95517926025705ce376ec5c4aaf4122773f
Many elements install additional distribution packages.
In addition the user can provide a set of packages to be installed
via the '-p' switch.
Some of them influence the boot process and therefore the initramfs
needs to be updated. Because the package manager during the image
creation process is configured not to run package scripts, this needs
to be done explicitly.
This issue was found during development and debugging of the
block-device LVM plugin: Even when the e.g. the lvm2 package
was installed in the image, it was missing in the initramfs
because of the missing update.
Change-Id: I7c92033b3ca80cdd23d081002059d83ca3f53bdb
Signed-off-by: Andreas Florath <andreas@florath.net>
Default the GENTOO_PORTAGE_CLEANUP to True. By default we should not
ship package info, this bloats the image and is usually outdated by the
time it'd be consumed.
Change-Id: I14c2530d91807cbc6a3806e01c7e4f6f472b190d
The debian element depends on debian-minimal now which provides
operating-system. This means that the debian element can no longer
provide operating-system and doing so results in an error when using the
debian element.
The fix is simple just rely on the fact that debian-minimal provides
operating-system and remove this element-provides from debian.
Fixes-Bug: 1758000
Change-Id: I524feeb82c19046ec987eb1186c7f4568309e559
This reverts commit ca8a89b1fb.
tripleo job is back to business again per latest runs
(see changes #553658 and #553701):
tripleo-buildimage-overcloud-full-centos-7 SUCCESS in 27m 39s
Change-Id: I111ed21f537b747877a34931a4085671cb9bc338
The devuser element can set up passwordless sudo, which requiers the
/etc/sudoers.d directory, which requires the sudo package, so we ensure
the sudo package is installed.
Change-Id: I80d6c669d4ac0d97b49d01cb621bf05b8e7f8ef1
There was a typo in I6b819a8071389e7e4eb4874ff7750bd192695ff2 that
modified this default partition type from "0x83" to just 83. We are
now seeing failures relating to this as sfdisk checks for a "disk
manager" when it see Id 0x53 (== 83)
Device Boot Start End Blocks Id System
/dev/vda1 * 2048 26664575 13331264 53 OnTrack DM6 Aux3
Restore to 0x83
Change-Id: Ib43038d2d740fbe01a21a13dd56367f7bc97f869
Due to the issue in I72cab0131ccffb1989d6c1324d0c9b92166782da the
tripleo job is failing. Unfortunately due to known issues there, the
tripleo gate is also blocked. Remove this job for now so we can get
dib moving.
Change-Id: I07173cc11a98a946e3ee1418288136aa1f791f85
Hpsum utiltity of proliant-tools requires net-tools to be installed
as part of base image. This commit adds support for installation of
net-tools for all distros.
Change-Id: I2a1e81059ed1aee975db78cfa5e61bbf1b98e06f
Closes-bug: 1751777
Whilst working on the Reproducible Builds effort [0], the team noticed
that diskimage-builder could not be built reproducibly as it iterates
over XML elements in a non-deterministic order when generating the
documentation. This patch sorts the elements, and is a forward of the
patch sent to the Debian BTS [1].
[0] https://reproducible-builds.org/
[1] https://bugs.debian.org/892020
Change-Id: I40011514a135bc50383b072ab83c7f91689bad5a
When using the package-installs element there can be some encoding
problems if the package installation emits unparsable output
[1]. However in this case we just want to forward the output to the
console which normally can handle this correctly. In order to fix this
switch off universal_newlines processing such that we just operate on
bytes.
Further we have to decode the lines without setting the locale and
ignoring errors. This is required because print encodes without
setting the locale and thus we need to filter/modify the stream such
that it doesn't crash.
[1] Traceback:
2018-03-01 09:58:00.515 | Traceback (most recent call last):
2018-03-01 09:58:00.515 | File "/usr/local/bin/package-installs-v2", line 137, in <module>
2018-03-01 09:58:00.515 | main()
2018-03-01 09:58:00.515 | File "/usr/local/bin/package-installs-v2", line 130, in main
2018-03-01 09:58:00.515 | process_output(install_args, follow=True)
2018-03-01 09:58:00.515 | for line in iter(proc.stdout.readline, ''):
2018-03-01 09:58:00.515 | File "/usr/lib/python3.5/encodings/ascii.py", line 26, in decode
2018-03-01 09:58:00.515 | return codecs.ascii_decode(input, self.errors)[0]
2018-03-01 09:58:00.515 | UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 34: ordinal not in range(128)
Change-Id: Ie4af9b4523459a630cfb98d09093bfe9ef7aa61e
Currently rhel7 image creation fails because it tries to copy
default bootloaders which is ubuntu way. This commit updates `iso`
element to correct the path of bootloaders required for rhel image.
Change-Id: I526d75b2db609fc77be0fc778b4d00f2d3df38ec
Closes-bug: 1750725
