On a system where the packaged pip/virtualenv is up-to-date with
upstream (such as Fedora 26 ... for now), we don't reinstall, which
then violates a bunch of assumptions later on. Force install.
Change-Id: I6ebcda0351997fa7e32f0e6e77a98b2c33764e3f
The python3 package actually contains some core modules (like the xml
one) which are not present in the python3-base on which is pulled by
the python3-devel package. As such, it's best to have it installed
similar to python-xml for python2.
Change-Id: I5cd5d1127ae62d6753c2ace44965179c5400bb9a
The 'packages' variable already contains the packages we need so
use it instead of duplicating the packages.
Change-Id: Id22e1862f9654e66252d03a0fed9839cf004d750
Several people have popped up in IRC recently with failures in these
elements. Without Python 2.7 available in the image they are
unsupported (OpenStack hasn't supported it for a long time). Remove
these to avoid further confusion.
The centos/centos7 DISTRO split that has happened with centos-minimal
is unfortunate but I don't think it helps to rename centos7/rhel7 ATM.
To summarise; DISTRO=centos7 means image based build,
DISTRO=centos && DIB_RELEASE=7 means the minimal build.
In the future, I think it is important that the minimal builds and
image builds set the same DISTRO. This reflects that "upper" layers
shouldn't care about the exact building of the lower layers. I see
CentOS 8 going one of two ways
1) the changes are so significant, we start separate centos8 /
centos8-minimal elements. They both set DISTRO=centos8 (and
DIB_RELEASE to point-release maybe?). This means we have to update
all "if DISTRO == centos || DISTRO == centos7" branches to also check
for "centos8". Evenually (!) "centos" goes away for versioned DISTRO
only
2) we restore centos element with DISTRO=centos and DIB_RELEASE=8, and
centos-minimal remains the same. This means we have to audit all "if
DISTRO == centos" calls to make sure they're appropriate for version 8
(stick a "&& DIB_RELEASE=7" on them all basically).
I'm not sure we can fully decide until we start to see excatly how the
distro switching/matching bits look, but (2) is consistent with Ubuntu
and probably the preferred solution.
Some "rhel" parts have been cleaned up. More could be done in
rhel-common, but given our lack of coverage of that I'd prefer to
leave it for now.
Change-Id: I6ea784116ef59ca22878c8512c963f29c815a00a
This was previously defined as python2-devel (which is what rhel uses),
but the actual package name is python-devel. See:
https://software.opensuse.org/package/python-devel
Change-Id: Id61e5b05772d10c32b33d3e70cb64d5ebdcba6e4
fedora/centos-minimal don't obey DIB_DISTRIBUTION_MIRROR currently. I
don't really want them too -- we want to be able to separate the
mirrors used during the build process from those embedded into the
final image. Add DIB_YUM_MINIMAL_BOOTSTRAP_REPOS which is a directory
with repo files to use during the install.
This introduces setup-gate-mirrors.sh which is intended to setup
repo/sources/whatever files in the openstack gate that point to the
local region mirror. It pulls the info from the mirror_info.sh script
on each CI node.
The openstack-ci-mirrors element is updated to export these variables.
elements are updated to depend on it. Tests are restored
Change-Id: I7604fc4d41cb1483be16b8d628a24e8fc764f515
This adds "openstack-ci-mirrors" element which performs various
settings to get builds using local mirrors. As a first step, we
convert ubuntu-minimal jobs
The main trick is that since infra mirrors are created with rerepo
they are not signed (they are recreated, not cloned, and not signing
is seen as a feature in that it deters external use). So we need to
instruct debootstrap to ignore signing and also turn it off for
in-chroot apt. Other than that, the existing DIB_DISTRIBUTION_MIRROR
works to redirect installs.
Remove "restricted" as it's not mirrored, and I don't think we want it
in here by default.
(I think DIB_DISTRIBUTION_MIRROR is a bit of an anti-pattern, because
it leaves the mirrors in the final image -- just because you use them
to build, doesn't mean you want them at runtime). But we don't need
to fix that now, and we don't use any created images.)
This pauses fedora testing until the next change, which moves to using
local mirrors for testing on fedora/centos
Change-Id: I778bd05a1e615c27edf1c9f0a1409119a6b3a850
package-installs.yaml is installing python-dev, not python2-dev,
so we need to adjust the mapping accordingly.
In addition, zypper-minimal used an dpkg specific package name,
while there is a SUSE equivalent (and zypper-minimal is anyway
SUSE family specific)
Change-Id: Ia9dd061fa46a514781808d62e5e93b03f75c6745
If DIB_PYTHON_VERSION is < 3 on the !redhat path, that means we're on
an older platform that may not have python3-virtualenv packages. Skip
install.
Ensure the order of operations happens by forcing the installs
Also add a note about limited platform support (patches welcome :)
Change-Id: I18412767f0ebf946d557a0a126285369e96af159
Recent changes in project-config have shown that we leave the system
in an inconsistent state when installing from source. On fedora, we
will have installed the python2 packages, but then used $DIB_PYTHON to
install python3 pip from source!
This tries to clarify the situation. As described in the document,
with package installs, we just install the $DIB_PYTHON packaged
versions.
Source installs want to take over the global namespace. This is the
price you pay for running the latest versions outside package managers
:) The only sane thing seems to be for us to normalise python2 &
python3 versions of pip, setuptools and virtualenv and then hacking
things such that "/usr/bin/pip" and "/usr/bin/virtalenv" remain
defaulted to python2 versions.
Documentation is added
Change-Id: Ibc6572b89e256d1f48b7fe7c672b8b9524dc704f
Currently we install pip/virtualenv with "/usr/local/bin/dib-python".
This means that every time you create a virtualenv, the python
interpreter inside it is called "dib-python" which is confusing.
Add an env var DIB_PYTHON that points directly the to interpreter
available during build, for use when running scripts.
Change-Id: I88ad3c9eb958d58db4631d9b27bc2c592f970345
Currently we have all our elements and library files in a top-level
directory and install them into
<root>/share/diskimage-builder/[elements|lib] (where root is either /
or the root of a virtualenv).
The problem with this is that editable/development installs (pip -e)
do *not* install data_files. Thus we have no canonical location to
look for elements -- leading to the various odd things we do such as a
whole bunch of guessing at the top of disk-image-create and having a
special test-loader in tests/test_elements.py so we can run python
unit tests on those elements that have it.
data_files is really the wrong thing to use for what are essentially
assets of the program. data_files install works well for things like
config-files, init.d files or dropping documentation files.
By moving the elements under the diskimage_builder package, we always
know where they are relative to where we import from. In fact,
pkg_resources has an api for this which we wrap in the new
diskimage_builder/paths.py helper [1].
We use this helper to find the correct path in the couple of places we
need to find the base-elements dir, and for the paths to import the
library shell functions.
Elements such as svc-map and pkg-map include python unit-tests, which
we do not need tests/test_elements.py to special-case load any more.
They just get found automatically by the normal subunit loader.
I have a follow-on change (I69ca3d26fede0506a6353c077c69f735c8d84d28)
to move disk-image-create to a regular python entry-point.
Unfortunately, this has to move to work with setuptools. You'd think
a symlink under diskimage_builder/[elements|lib] would work, but it
doesn't.
[1] this API handles stuff like getting files out of .zip archive
modules, which we don't do. Essentially for us it's returning
__file__.
Change-Id: I5e3e3c97f385b1a4ff2031a161a55b231895df5b
