Commit Graph

3813 Commits

Author SHA1 Message Date
Ian Wienand
6ee2995214 dib-lint: use yamllint to parse YAML files
This gives us better linting of YAML files that just opening them.
This would have detected the duplicate keys in
I34e27d821fbefe274e7b007f37b0bd34db2e1d26.

The .yamllint is taken from zuul-jobs where it is also used as a
fairly sane set of default rules.

A few minor newline fixes are added.

Change-Id: I96d6644ae24f7deb84fa50fefbda0f0d33e0e009
2020-05-26 12:04:09 +10:00
Ian Wienand
6e549c33ac Revert "ubuntu-minimal : only install 16.04 HWE kernel on xenial"
This reverts commit 14ff8f942c.

This seems to not be installing the kernel at all, and needs further
investigation.

Change-Id: Ifd809d4b67aff5d80f979235db246a16af0375b3
2020-05-25 18:48:51 +10:00
Ian Wienand
7539e241da ubuntu-minimal: Add Ubuntu Focal test build
Add test builds for Focal on x86 and ARM64

Change-Id: Idb23f0e00d37c7447441ea002aad078e8c61f969
2020-05-21 14:03:54 +10:00
Ian Wienand
14ff8f942c ubuntu-minimal : only install 16.04 HWE kernel on xenial
Only install the HWE kernel by default for Xenial.  This was actually
installing the 16.04 HWE kernel on Bionic by accident, since it seems
to have that package; however it was breaking Focal.

On the other distros, just install the default generic kernel.  Let's
KISS for now if we can ...

Change-Id: I34e27d821fbefe274e7b007f37b0bd34db2e1d26
2020-05-21 14:03:54 +10:00
Ian Wienand
c5acc91574 ubuntu-minimal: fix HWE install for focal
On Focal, install the 20.04 HWE kernel if linux-image-hwe is
specified.

Change-Id: I5d536e8f64e7b987415849d8cd4da7959bba7af7
2020-05-21 14:03:54 +10:00
Ian Wienand
138d3f9b81 package-installs: allow when filter to be a list
Allow the "when:" statements to be a list of values, which are
effectively anded together to filter the package install.

Change-Id: Ia6f10741fa6be24b11d6991c8a6b6e07951ff68d
2020-05-21 14:03:49 +10:00
Ian Wienand
9080d04923 block device: update variable name
New versions of flake8 fail as "l" can be confused for "1" apparently
(E741) ... not sure I totally agree but since it's only one instance,
update it.

Change-Id: Ic5c47867facd56b53cc6534da4ae3a345c516202
2020-05-13 06:22:02 +10:00
Zuul
d3c7d2d4b1 Merge "functests: use ensure-virtualenv" 2020-05-05 08:40:55 +00:00
Ian Wienand
695ea4021e functests: use ensure-virtualenv
This ensures the virtualenv tool is available on all testing nodes.

Depends-On: https://review.opendev.org/723309
Change-Id: Ic52a380dab9faa0f65d3c1ebe0bff6f69736e3cd
2020-04-28 10:29:26 +10:00
Ian Wienand
687919e2bd Add sibling container builds to experimental queue
These are required for the experimental container boot jobs

Change-Id: I16c27311adab09eafef747c1bba868115c000928
2020-04-27 08:52:31 +10:00
Zuul
8781c5d139 Merge "Add a focal test" 2020-04-24 16:20:23 +00:00
Zuul
ba518d2553 Merge "Restore SUSE tests to gate" 2020-04-24 16:20:21 +00:00
Zuul
b81e94e8f1 Merge "Switch functional tests to containers" 2020-04-24 16:14:01 +00:00
Zuul
8634b42e4d Merge "Bionic functional tests should be voting" 2020-04-24 09:41:26 +00:00
Ian Wienand
dfdf04bf2d Add a focal test
This adds a boot test for the newly released Ubuntu Focal release.

Depends-On: https://review.opendev.org/722556
Change-Id: If8976a3f99a02c8d85cd3818d8c77c75b60816d8
2020-04-24 08:01:39 +00:00
Zuul
534b799de8 Merge "pip-and-virtualenv : fix fedora 30 install" 2020-04-24 05:34:35 +00:00
Zuul
0d0a5909e7 Merge "yum-minimal: strip env vars in chroot calls" 2020-04-24 05:34:33 +00:00
Zuul
65fc9e1f7d Merge "Fix args to debuntu functional tests" 2020-04-24 05:31:06 +00:00
Ian Wienand
5fdc31ce8a Restore SUSE tests to gate
We can build SUSE in the container after the switch in
Ieaf07438c145ba609d4d7b9a055e91b894eca6d9.

Let's start non-voting and promote when we feel it's stable.

Change-Id: Ibe997d3167dc918fcdb28cf065c50049943dd86a
2020-04-24 15:10:24 +10:00
Ian Wienand
5d899d97a0 Switch functional tests to containers
This switches the functional tests to container based build jobs.
This is mostly transparent for the existing build and boot jobs.

The release test is a bit interesting here.  The nodepool
container-based release job doesn't work in this context because it
doesn't have a way to also include dib from the zuul checkout into the
container -- ergo it's pointless to run in the dib gate.

We should probably reorganise the nodepool siblings job so it can
check some projects out from tags, then we could run something similar
to this with nodepool/openstacksdk/glean etc checked out from tags if
we want.

We can handle this in a future update so we can test these images
which we want to build in our production container asap.

Depends-On: https://review.opendev.org/722556
Depends-On: https://review.opendev.org/722549
Change-Id: Ieaf07438c145ba609d4d7b9a055e91b894eca6d9
2020-04-24 14:44:51 +10:00
Ian Wienand
90a21b84b5 Bionic functional tests should be voting
We now have a bionic(-ish ... buster debian image anyway) builder and
have moved some of the important tests (fedora, pip-and-virtualenv) to
here because they don't work on Xenial any more.  This should now be
voting so we don't miss anything.

Change-Id: I16d313e1dfa95707d996456da244667c38d6343f
2020-04-24 12:43:28 +10:00
Ian Wienand
a8a5b1ac77 pip-and-virtualenv : fix fedora 30 install
This should be installing the python2 and python3 packages (that's
what pip-and-virtualenv is designed to do), but we dropped the +=
accidentally in ee9ad32b6f.

However, we've moved on anyway and after
I7a6a342461d6001c25e55638ba9b7438c28f2519 F31 doesn't support this
element.  fedora-latest is already updated to f31 in the opendev gate.
Remove the testing as it is no longer relevant.

Change-Id: Id696a90baa1eb05cb4c08501f8dac3665d395682
2020-04-24 12:42:35 +10:00
Ian Wienand
df3ad26f58 yum-minimal: strip env vars in chroot calls
This showed up with dnf in containers when TMPDIR was set; dnf started
trying to write to this directory while in the chroot.

We already do stripping like this in run_in_target -- but this is a
bit of a unique place because it's actually setting up the initial
chroot so the target doesn't actually exist yet; so we just hard-code
it in place here.

Change-Id: If7310cb820846da903bf60daa4486c8bf7cb0136
2020-04-24 12:38:42 +10:00
Ian Wienand
99d5a8d219 Fix args to debuntu functional tests
Ia4dde1fb01da284a5e681237ab88c68fb9afcbef dropped these from the
generic job in nodepool, because they shouldn't apply to other distros
using the job like CentOS etc.  Add the required args back here to the
debuntu tests.

Change-Id: I3f9ae58f3aa534d65344d79d44b0b74368e329a8
2020-04-24 12:34:59 +10:00
Zuul
eebb68b385 Merge "pip-and-virtualenv: drop f31 & tumbleweed, rework suse 15 install" 2020-04-23 23:52:10 +00:00
Ian Wienand
262281e5f7 Remove Trusty testing
Opendev has dropped trusty from the mirrors.  With no testing the
distribution is effectively unsupported, add a release note.  Update a
few other random doc bits (that are not really that up to date
anyway).

Change-Id: I5bd0d0a94477cf8d84cef72f5d4b2e9e15ab9fd2
2020-04-23 10:00:13 +10:00
Ian Wienand
ac7fba7040 pip-and-virtualenv: drop f31 & tumbleweed, rework suse 15 install
This is an alternative approach to commit
68bb43535e.  I think this proposes a
better overall solution that the prior change which had the Python 3
packages being installed, but did not specify the _do_py3 flag to do
the installation steps that redirect the various tool installations.

Fedora 31+ doesn't have python2, and Tumbleweed does have some Python
2 support but there seems to be no reason to bother updating this
element for either with infra very close to removing this completely
[1].  Error out on these platforms, and add a release note.

The 15 path should include the python2 and python3 packages, along
with the flags to do the "cleanup"; i.e. forced removal of distutils
packages that pip 10+ won't touch.  As mentioned in the original
change, the six package causes problems here, but we can clear that
too by explicitly listing it instead of letting it come in via
dependencies.  Again, this element will be removed from the infra 15
builds ASAP; but we can release with this to provide a roll-back point
if we need to revert the removal to fix things temporarily.

Add it to the testing path as well.

[1] https://docs.opendev.org/opendev/infra-specs/latest/specs/cleanup-test-node-python.html

Change-Id: I7a6a342461d6001c25e55638ba9b7438c28f2519
2020-04-23 08:10:26 +10:00
Zuul
70d5f754a1 Merge "Remove Babel and any signs of translations" 2020-04-22 11:51:26 +00:00
Zuul
b262e88443 Merge "Make ipa centos8 dib job voting" 2020-04-22 01:44:27 +00:00
Dmitry Tantsur
97cadc4249 Remove Babel and any signs of translations
DIB does not currently have any translations or translatable strings,
so removing local configuration from setup.cfg.

Additionally, Babel is only required if you import it directly,
removing it as well.

Change-Id: I036de97641df38eef408290b50cecfe216c69f6d
2020-04-17 17:05:15 +02:00
Zuul
0445a5ce2b Merge "Add centos aarch64 tests" 2020-04-17 01:18:54 +00:00
Ian Wienand
3f823488c5 Add centos aarch64 tests
Change-Id: I2fffeeffa82c505a34c5c9972724b557f3b58975
2020-04-16 14:15:50 +10:00
Marcin Juszkiewicz
18e35893f7 Do not try to use MBR on AArch64
When I tried to build CentOS8 image for AArch64 I got error saying that
MBR is not supported. So make sure that it will not be used by default.

Change-Id: Ib67ab7f808d727c3c61932c540d398dbe723972f
2020-04-14 10:12:57 +00:00
Riccardo Pittau
716286117c Make ipa centos8 dib job voting
The ironic-python-agent-builder dib job based on centos8 is stable
since a while now.
We should make it voting to catch any possible issue on dib side.

Change-Id: I42dd81fa17de1130a3af7c031f9479e18790c350
2020-04-06 11:55:54 +02:00
Dirk Mueller
68bb43535e opensuse: fix python 2.x install
openSUSE Tumbleweed is dropping python2-* packages so we need
to stop intalling them. We can also stop installing those
for Leap 15. which avoids a pip uninstall issue (as python2-six
was still built with distribute).

Change-Id: Ie93c8addb26aab3d0154c4b5b52423799abede91
2020-04-03 18:22:33 +02:00
Ian Wienand
434d2db5d4 Debuntu: add apt-transport-https
I don't see anywhere we bring this in, especially on a minimal build.
In 2020 it seems like a base dependency, put it alongside
software-properties-common that installs the other apt helper bits.

Change-Id: I5b079eac4912cb4a164e9aa6158ed106a28f576c
2020-04-02 10:11:35 +11:00
Zuul
ac250e5ab0 Merge "centos 8 image build: fix mirror" 2020-03-31 05:03:06 +00:00
Zuul
f6aeae70c1 Merge "run_functests: handle build without tar" 2020-03-31 05:03:05 +00:00
Zuul
dc67ccfe12 Merge "Mellanox element: removed ibutils,libibcm,libmlx4-dev" 2020-03-30 22:10:13 +00:00
Ian Wienand
56dc0a2c31 centos 8 image build: fix mirror
We're ending up with "centoscentos" in the mirror location and the
build fails; strip out the $contentdir from the original too.

Change-Id: If09dbbd8028ea510d2ab0d3d8afe484cea611df5
2020-03-31 08:57:15 +11:00
Ian Wienand
92b5b84c0d run_functests: handle build without tar
It seems this code does not consider that you might set
test-output-formats with only "qcow2" -- it currently also tries to
inspect a .tar output.

Skip if tar isn't in the output format list, and fall through to
passing so both paths are considered success, only exiting if there is
a failure.

Change-Id: Ia21dbfd298d04b04ec02dfe76a0acd7c9a7a31f2
2020-03-31 08:56:26 +11:00
Zuul
21f9ea4f78 Merge "Add Fedora 31 support and test jobs" 2020-03-30 01:26:32 +00:00
Lee Yarwood
ee9ad32b6f Add Fedora 31 support and test jobs
Change-Id: Iad0261ac8db001ffa4d3a4dd6df05fe923402a69
2020-03-30 08:47:00 +11:00
Zuul
56d9956cbd Merge "Add python-stow-versions element" 2020-03-27 13:49:21 +00:00
Lenny Verkhovsky
2f4e1886e5 Mellanox element: removed ibutils,libibcm,libmlx4-dev
Those packages will be installed as part of dependencies

Change-Id: I640dc55c818dd8538bfcd199c33f9af93aae7608
2020-03-23 00:06:53 +02:00
Harald Jensås
1ac31afd62 Use rpm -e instead of dnf for cleaning old kernels
If the running kernel of the system building the image
matches the kernel that is to be removed dnf will fail.

Repalce use use of dnf with rpm -e.

Closes-Bug: #1623409
Change-Id: Ie2481ea8a02b7b0720e46fa179f24badf4aa25c5
2020-03-19 22:35:23 +01:00
Dmitriy Rabotyagov
1de6fe4ba7 Add python-stow-versions element
This element is designed to install latest minor versions of different
python releases, like py27, py35, py36, py37, py38
into stow directory, and later easily enable them with stow.

Change-Id: Iab6d20e7643e549b53c629fb430e58b1c5e72991
2020-03-19 21:21:01 +02:00
Monty Taylor
c113703050 Add support for build-only packages
Sometimes an element needs packages installed so that it can
perform tasks but those package are not appropriate for the
final image content. Add a "build-only" flag to package-install-squash
which will cause package to be installed at the beginning of the
phase and then uninstalled at the end of the phase.

Change-Id: Ie01b795991710c93f6b669c8f14b57eb4412c1d5
2020-03-18 14:40:44 -05:00
Ian Wienand
64d2ea26e7 bindep: remove lsb-release
This came in with edc06a20e5 although
it's not totally clear why.

We found that we were sourcing devstack scripts during build in infra,
and inadvertently relying on this package being around on the host.
This was incorrect but can this type of problem is hidden if the
package is installed unnecessarily.

I do not think this actually required now; remove from bindep to
minimise chance for such false or hidden dependencies.

Change-Id: I7895d4c5019511eb2169915cd8092e707e8ee732
Story: #2007407
Task: #39068
2020-03-16 10:33:40 +11:00
Monty Taylor
1038f9c8ba Remove hacking from requirements
putting it in requirements means it's installed for anything that
installs diskimage-builder. Not just for tests, but in actual prod
installations. Because of the way flake8 plugins work, this means
then that hacking is injected into all flake8 invocations in that
environment.

This is currently causing issues in nodepool where we do not
actually use hacking but it's still spewing spurious messages to
the screen because it is incorrect finding our apache header to be
incorrect.

Just depend on flake8, since that's what dib-lint depends on.

Change-Id: I8d2b2076b604d596906a3b12a5518e2a56c0fe88
2020-03-12 16:20:37 -05:00