Commit graph

4005 commits

Author SHA1 Message Date
Zuul
538fd6291e Merge "Revert "Make ipa centos8 job non-voting"" 2020-07-06 05:37:57 +00:00
Zuul
7e00908a28 Merge "add more python selections to gentoo" 2020-07-06 02:59:59 +00:00
Zuul
5d16300f00 Merge "Download latest CentOS cloud image" 2020-07-05 21:06:43 +00:00
Zuul
9ab6811e84 Merge "Add support for CentOS 8 Stream" 2020-07-05 21:06:41 +00:00
yatin
68bc561c09 Revert "Make ipa centos8 job non-voting"
This reverts commit 868a6f0c8a.

Change-Id: I1cc72314716690874eca57526eac224c4e0b24cf
Depends-On: https://review.opendev.org/#/c/738438/
2020-07-01 03:57:30 +00:00
yatinkarel
e999368d2c Disable all enabled epel repos in CentOS8
EPEL centos8 have different epel repos like
epel, epel-modular, epel-playground etc, so
we need to disable all not just epel.

Also ensure other epel repositories in CentOS7
like epel-testing are also disabled.

Related-Bug: #1885315
Change-Id: I02b3b83fa2047702d5f069d3ca1c9c0bcc1dab52
2020-06-30 10:15:30 +05:30
yatinkarel
868a6f0c8a Make ipa centos8 job non-voting
Until the fix for the job [1] lands make the
job non-voting. The fix needs patch in dib[2].

[1] https://review.opendev.org/#/c/738438/
[2] https://review.opendev.org/#/c/738435/

Depends-On: https://review.opendev.org/#/c/738434/
Change-Id: I91cd22f23657a2059334a4249f45f9afbd96b74e
2020-06-29 22:45:05 +05:30
Matthew Thode
64787d8ec2
add musl profile to gentoo
Change-Id: Id954ffe4d9b019c7e7bb648725ff7f976c929b4d
2020-06-22 18:37:30 -05:00
Carlos Goncalves
1f9619cdb5 Download latest CentOS cloud image
DIB was retrieving the oldest cloud image file which, presently, means
retrieving CentOS 8.1 instead of CentOS 8.2. Even though DIB runs a
system update and so catches up to latest, this takes bandwidth, time
and final image space (8.1 + system update = 765M qcow2, vs 8.2 + system
update = 518M qcow2).

This patch fixes that by taking the first image name in a descending
order list.

Change-Id: I648fe19f1f76c03c97492b6ac7be6381f6f9261b
2020-06-22 11:03:36 +02:00
Carlos Goncalves
367dfc9294 Add support for CentOS 8 Stream
This patch adds support for CentOS 8 Stream [1] to the centos-minimal
element. Users should set DIB_RELEASE=8-stream.

[1] https://www.centos.org/stream/

Change-Id: Id0825de735ab957c10daf35fb3c641f850cc6847
2020-06-22 10:36:30 +02:00
Zuul
6822090e22 Merge "update grub cmdline to current kernel parameters" 2020-06-17 12:15:18 +00:00
Zuul
24f2dbb59a Merge "Add .eggs to gitignore" 2020-06-16 11:43:38 +00:00
Matthew Thode
c7b6f0d6a1
add more python selections to gentoo
Change-Id: If4420e9ae7870739aac8a673ff290f9ed1391acf
2020-06-14 01:16:18 -05:00
Matthew Thode
52a8b2fa01
update grub cmdline to current kernel parameters
vga=normal is depreciated, use gfxpayload=text instead

Change-Id: I2e5be3b07acce14a9f3d47e24938d9da3c5b6c48
2020-06-14 00:39:42 -05:00
Chris MacNaughton
8f57ed9b9e
Stop bringing the test environment into the mocks
Change-Id: I1780f186cf107242cacd6d3da7a5bc81a330b536
Closes-Bug: #1883224
2020-06-12 10:37:36 +02:00
Zuul
2e6a7f949c Merge "Remove virtualenv activation" 2020-06-12 01:17:17 +00:00
Dmitry Tantsur
e793cc4038 Remove virtualenv activation
Since the original merge of this code
(04208e7c79) several things have
changed; particularly now we ship dib-run-parts as part of dib, not as
a separate package.

We setup $_LIB to point to the shipped library diretory via
pkg_resources lookups.  We now call dib-run-parts (as mentioned,
shipped as a dib library now), source scripts, etc. via $_LIB and thus
do not rely on $PATH.  Consequently we don't need this activation
part.

Which is helpful, because "venv" (as opposed to virtualenv) doesn't
have activate_this.py.  So this fixes installation under that for
Python 3.

We update the functional tests to use the virtualenv_command exported
by the ensure-pip role, which will test the venv path.  There is no
need for dib_python as we are Python 3 only now.

Change-Id: Iede929ea2d278008220aac8b1d678ba41eba0d8a
2020-06-11 16:49:15 +10:00
Zuul
3614900572 Merge "Fix yumdownloader cache dir" 2020-06-10 16:24:53 +00:00
Zuul
a0714c2300 Merge "Debuntu: add apt-transport-https" 2020-06-09 10:33:56 +00:00
Zuul
caf72a4c56 Merge "Drop six usage" 2020-06-09 10:28:14 +00:00
Zuul
68017d0046 Merge "Do not fail in a venv when activate_this.py is not found" 2020-06-09 09:54:07 +00:00
Riccardo Pittau
958fb1b6a7 Add .eggs to gitignore
Change-Id: I45da9a837be090983bd8f265b8f104909256b78b
2020-06-09 11:02:00 +02:00
melissaml
abba0b5574 Switch to newer openstackdocstheme and reno versions
Switch to openstackdocstheme 2.2.1 and reno 3.1.0 versions. Using
these versions will allow especially:
* Linking from HTML to PDF document
* Allow parallel building of documents
* Fix some rendering problems

Update Sphinx version as well.

Disable openstackdocs_auto_name to use 'project' variable as name.

Change pygments_style to 'native' since old theme version always used
'native' and the theme now respects the setting and using 'sphinx' can
lead to some strange rendering.

openstackdocstheme renames some variables, so follow the renames
before the next release removes them. A couple of variables are also
not needed anymore, remove them.

See also
http://lists.openstack.org/pipermail/openstack-discuss/2020-May/014971.html

Change-Id: If3e8d8ac29a728aa41562b31976ebe9bfa5df66f
2020-06-09 10:23:37 +02:00
Simon Westphahl
4a424ecabb Use kpartx option to update partition mappings
Fix cases of 'mkfs' failing because the partitions never showed up. Partition
mappings will now be updated instead of just adding them with 'kpartx'. That
means that 'kpartx' will also remove devmappings for deleted partitions.

Traceback of failing mkfs call:

2020-05-11 22:03:25.523 | INFO diskimage_builder.block_device.utils [-] Calling [sudo sync]
2020-05-11 22:03:25.539 | INFO diskimage_builder.block_device.utils [-] Calling [sudo kpartx -avs /dev/loop0]
2020-05-11 22:03:25.581 | INFO diskimage_builder.block_device.utils [-] Calling [sudo mkfs -t ext4 -i 4096 -J size=64 -L cloudimg-rootfs -U 21c6f9eb-4d52-4e5c-b9b7-796735de8909 -q /dev/mapper/loop0p1]
2020-05-11 22:03:25.700 | ERROR diskimage_builder.block_device.blockdevice [-] Create failed; rollback initiated
2020-05-11 22:03:25.700 | Traceback (most recent call last):
2020-05-11 22:03:25.700 |   File "/home/zuul/dib/lib/python3.6/site-packages/diskimage_builder/block_device/blockdevice.py", line 406, in cmd_create
2020-05-11 22:03:25.700 |     node.create()
2020-05-11 22:03:25.700 |   File "/home/zuul/dib/lib/python3.6/site-packages/diskimage_builder/block_device/level2/mkfs.py", line 133, in create
2020-05-11 22:03:25.700 |     exec_sudo(cmd)
2020-05-11 22:03:25.700 |   File "/home/zuul/dib/lib/python3.6/site-packages/diskimage_builder/block_device/utils.py", line 143, in exec_sudo
2020-05-11 22:03:25.700 |     raise e
2020-05-11 22:03:25.700 | diskimage_builder.block_device.exception.BlockDeviceSetupException: exec_sudo failed
2020-05-11 22:03:25.700 | INFO diskimage_builder.block_device.level0.localloop [-] loopdev detach
2020-05-11 22:03:25.701 | INFO diskimage_builder.block_device.utils [-] Calling [sudo losetup -d /dev/loop0]
2020-05-11 22:03:25.732 | INFO diskimage_builder.block_device.level0.localloop [-] Remove image file [/tmp/dib_image.muyw7t1h/image0.raw]
2020-05-11 22:03:25.734 | ERROR diskimage_builder.block_device.blockdevice [-] Rollback complete, exiting
2020-05-11 22:03:25.740 | Traceback (most recent call last):
2020-05-11 22:03:25.740 |   File "/home/zuul/dib/bin/dib-block-device", line 8, in <module>
2020-05-11 22:03:25.740 |     sys.exit(main())
2020-05-11 22:03:25.740 |   File "/home/zuul/dib/lib/python3.6/site-packages/diskimage_builder/block_device/cmd.py", line 120, in main
2020-05-11 22:03:25.740 |     return bdc.main()
2020-05-11 22:03:25.740 |   File "/home/zuul/dib/lib/python3.6/site-packages/diskimage_builder/block_device/cmd.py", line 115, in main
2020-05-11 22:03:25.740 |     self.args.func()
2020-05-11 22:03:25.740 |   File "/home/zuul/dib/lib/python3.6/site-packages/diskimage_builder/block_device/cmd.py", line 36, in cmd_create
2020-05-11 22:03:25.740 |     self.bd.cmd_create()
2020-05-11 22:03:25.740 |   File "/home/zuul/dib/lib/python3.6/site-packages/diskimage_builder/block_device/blockdevice.py", line 406, in cmd_create
2020-05-11 22:03:25.740 |     node.create()
2020-05-11 22:03:25.740 |   File "/home/zuul/dib/lib/python3.6/site-packages/diskimage_builder/block_device/level2/mkfs.py", line 133, in create
2020-05-11 22:03:25.740 |     exec_sudo(cmd)
2020-05-11 22:03:25.740 |   File "/home/zuul/dib/lib/python3.6/site-packages/diskimage_builder/block_device/utils.py", line 143, in exec_sudo
2020-05-11 22:03:25.740 |     raise e
2020-05-11 22:03:25.740 | diskimage_builder.block_device.exception.BlockDeviceSetupException: exec_sudo failed

Change-Id: I374f7f22f9e93ef35eb5813712ca59e75f0733e8
Related-Bug: #1698337
2020-06-09 09:07:55 +02:00
Zuul
1a95c3f93f Merge "Cleanup py27 and docs support" 2020-06-09 05:56:18 +00:00
Zuul
14d3a7ee4f Merge "Add back pep8 and tarball jobs" 2020-06-09 05:25:13 +00:00
Dmitry Tantsur
1e1e36df64 Do not fail in a venv when activate_this.py is not found
The standard Python venv module does not have this script, so currently
DIB unconditionally fails. While a real fix will be provided in
https://review.opendev.org/#/c/704478/ this change at least allows
users to try work around the problem.

Change-Id: I45b79d4d283f2b3ea909612e652672dcb6092488
2020-06-09 12:53:21 +10:00
Carlos Goncalves
9d9dd9249c Fix yumdownloader cache dir
TMPDIR env is not being honored as cache dir in EL 8.

Change-Id: I8281675ec5f0951b3e190a8d6727744a1a5cd8d7
2020-06-07 21:58:49 +02:00
Albin Vass
913937bb10 Update readme to clarify an ubuntu bionic image is built
Change-Id: Ia0263c743a5266bac9465675345fe974c1a1d6bc
2020-06-07 19:53:46 +02:00
Andreas Jaeger
4493208048 Drop six usage
With python3, six is not needed anymore, drop it.

Change-Id: I70bb679270605ac32ca0cceb9414ea3a210e5842
2020-06-05 12:04:37 +02:00
Andreas Jaeger
e7885499dd Cleanup py27 and docs support
This repo is now testing only with Python 3, so let's make
a few cleanups:
- Remove python 2.7 stanza from setup.py
- Remove obsolete sections from setup.cfg
- Update classifiers
- Update requirements, no need for python_version anymore
- Cleanup doc/source/conf.py to remove now obsolete content,
  use sphinxcontrib.apidoc to build module index again (this
  seems to have been broken when switching to sphinx-build).
- Remove install_command from tox.ini, the default is fine
- Use TOX_CONSTRAINTS_FILE instead of obsolete UPPER_CONSTRAINTS_FILE.
- Update tox.ini for python3 only support.

Change-Id: Id8738ecfb0f578d2a7953c63ffe10779f835bcaf
2020-06-05 12:04:30 +02:00
Andreas Jaeger
7c73557d72 Add back pep8 and tarball jobs
Use the victoria python3 template to add py36, py38, pep8 and tarball
publishing back in.

Dependency added to fix ironic-python-agent-check-image-dib-centos8 job.

Depends-On: https://review.opendev.org/733794
Change-Id: I2a789dbfbdc3e0104fa9715dcfe5233453ccbf9c
2020-06-05 12:01:52 +02:00
Monty Taylor
3477dfad00 Drop support for python2
python2 is EOL. OpenStack and Zuul both are python3 only.

Drop the Python-2 CentOS 7 build host testing, including the tripleo
centos-7 train job.

Drop the old openstack-python-jobs template and move to specific
version templates for what we support.

Drop upper-constraints from the py35 environment, as master of
requirements is now using python 3.6-only libraries.

Drop the Xenial tests for similar reasons (rather than making some
sort of forced install); we are keeping Python 3.5 support just for
Zuul compatability at this point so a tox run is enough.

Depends-On: https://review.opendev.org/729787
Depends-On: https://review.opendev.org/729508

Change-Id: I6d90754cb4b7217b10ae777d414c03466b751761
2020-05-29 09:47:18 +10:00
Ian Wienand
de57271ed2 Prepare to drop Python 2 support
Pin func test requirements to stable/train; the last stable release
with Python 2 support.  Switch to the python-jobs-no-constraints
template to avoid using master constraints file.

Only build focal on bionic, other debootstraps are too old.

Remove pip-and-virtualenv testing as we are moving to plain images.

The tripleo-buildimage-jobs are unstable; see linked bug.

Add a note that this is the last Python 2 release.

Change-Id: Ibde7a564dd41cc2d6e80e2dffe5a95a57bbf8ada
2020-05-29 09:44:54 +10:00
Ian Wienand
0c94eef7be Revert "dib-lint: use yamllint to parse YAML files"
This reverts commit 6ee2995214 and
e85c2a6f03.

I missed that if you pip install and then run dib-lint, it's not going
to pick up the .yamllint file shipped here.  Thus it gives spurious
errors.

The reason for this was simply better duplicate key detection in yaml
files, which caused us problems with the kernel installs.  However, at
this point it seems just the old "does it load" test from pyyaml will
be enough.

Change-Id: I87a9fc9bb119cfeffad48fc0fa0df31f0181825d
2020-05-28 16:44:49 +10:00
Ian Wienand
d2eaaaf622 Merge "Pre-install xz package in opensuse chroot" 2020-05-28 02:06:30 +00:00
Ian Wienand
d71284ccbd Merge "use stage3 instead of stage4 for gentoo builds" 2020-05-28 02:02:56 +00:00
Matthew Thode
041bdd331a
use stage3 instead of stage4 for gentoo builds
The main reason for using the stage4 is now gone (kernel compile).
Install and use the distro provided binary kernel package.

In addition to this, set the locale and timezone, beyond that very
little was done in the gentoo stage4.

Change-Id: I541b7d9b807e2357398ae1c249b1978958dd1137
Signed-off-by: Matthew Thode <mthode@mthode.org>
2020-05-27 18:50:16 -05:00
Colleen Murphy
19b3586d08 Pre-install xz package in opensuse chroot
As of recently, opensuse-minimal images fail to build because of an
error installing the kernel-default package:

> Problem: kernel-default-5.6.12-1.3.x86_64 requires mkinitrd >= 2.7.1, but this requirement cannot be provided
>   not installable providers: dracut-050+suse.61.g0fe0e854-1.1.i586[repo-oss]
>                    dracut-050+suse.61.g0fe0e854-1.1.x86_64[repo-oss]

The problem is there is a recently added package `busybox-links` which
provides a subpackage `busybox-xz` which provides the /usr/bin/xz
utility. Since this is available, the `aaa_base` package installs it
during the root.d base installation phase to fulfill it's dependency on
/usr/bin/xz. On the other hand, the dracut package explicitly requires
the `xz` package, and this is not co-installable with the `busybox-xz`
package, so the dracut package is not installable during the install.d
phase. This change explicitly adds the `xz` package to the initial
chroot provisioning phase so that the /usr/bin/xz requirement is already
fulfilled and `busybox-xz` does not get installed.

Change-Id: Iba8c301eb496657873963e1aa99736aacf87cb00
2020-05-27 10:45:03 -07:00
Monty Taylor
e85c2a6f03 Add dependency on yamllint
dib-lint requires yamllint now, so we need to depend on it.

Change-Id: Iccd2cdbb04bd9b429d8d19b31ff3b10cdd568e15
2020-05-27 10:28:19 -05:00
Ian Wienand
339d713527 Revert "Revert "ubuntu-minimal : only install 16.04 HWE kernel on xenial""
This reverts commit 6e549c33ac.

It uses the new multiple-parameter matching format from
Idff7b067ad4255e6fc4138f7eff313a81b75c8ba to actually do what it says.

Change-Id: I4656ff1a5c46bcfbd8587f2f541825f4ad08820f
2020-05-27 06:18:02 +10:00
Ian Wienand
b71d1c60d2 package-installs : allow a list of parameters
The change Ia6f10741fa6be24b11d6991c8a6b6e07951ff68d introduced having
"when:" as a list of values.  However, this was actually not
sufficient to express the logic required for arm64/x86_64/xenial
kernel matching we wanted.

Because the package name is a key, we can't have multiple entires in
the package-map YAML files.  This means we can't do more advanced
matching and thus we need to be able to match through multiple
parameters.  Similar to Ia6f10741fa6be24b11d6991c8a6b6e07951ff68d we
modify the matching rules to allow a list.

A an example of using this is provided in the README.rst, and this
same example worked through by the unit tests.

This also slightly updates the matching logic to be more sequential.
After each check we either continue on or log the failure and continue
to the next check (rather than set a list of flags then check that at
the end).  This makes it much easier to understand what is being
matched in the logging output from the tool.

Change-Id: Idff7b067ad4255e6fc4138f7eff313a81b75c8ba
2020-05-27 06:17:57 +10:00
Ian Wienand
6ee2995214 dib-lint: use yamllint to parse YAML files
This gives us better linting of YAML files that just opening them.
This would have detected the duplicate keys in
I34e27d821fbefe274e7b007f37b0bd34db2e1d26.

The .yamllint is taken from zuul-jobs where it is also used as a
fairly sane set of default rules.

A few minor newline fixes are added.

Change-Id: I96d6644ae24f7deb84fa50fefbda0f0d33e0e009
2020-05-26 12:04:09 +10:00
Ian Wienand
6e549c33ac Revert "ubuntu-minimal : only install 16.04 HWE kernel on xenial"
This reverts commit 14ff8f942c.

This seems to not be installing the kernel at all, and needs further
investigation.

Change-Id: Ifd809d4b67aff5d80f979235db246a16af0375b3
2020-05-25 18:48:51 +10:00
Ian Wienand
7539e241da ubuntu-minimal: Add Ubuntu Focal test build
Add test builds for Focal on x86 and ARM64

Change-Id: Idb23f0e00d37c7447441ea002aad078e8c61f969
2020-05-21 14:03:54 +10:00
Ian Wienand
14ff8f942c ubuntu-minimal : only install 16.04 HWE kernel on xenial
Only install the HWE kernel by default for Xenial.  This was actually
installing the 16.04 HWE kernel on Bionic by accident, since it seems
to have that package; however it was breaking Focal.

On the other distros, just install the default generic kernel.  Let's
KISS for now if we can ...

Change-Id: I34e27d821fbefe274e7b007f37b0bd34db2e1d26
2020-05-21 14:03:54 +10:00
Ian Wienand
c5acc91574 ubuntu-minimal: fix HWE install for focal
On Focal, install the 20.04 HWE kernel if linux-image-hwe is
specified.

Change-Id: I5d536e8f64e7b987415849d8cd4da7959bba7af7
2020-05-21 14:03:54 +10:00
Ian Wienand
138d3f9b81 package-installs: allow when filter to be a list
Allow the "when:" statements to be a list of values, which are
effectively anded together to filter the package install.

Change-Id: Ia6f10741fa6be24b11d6991c8a6b6e07951ff68d
2020-05-21 14:03:49 +10:00
Ian Wienand
9080d04923 block device: update variable name
New versions of flake8 fail as "l" can be confused for "1" apparently
(E741) ... not sure I totally agree but since it's only one instance,
update it.

Change-Id: Ic5c47867facd56b53cc6534da4ae3a345c516202
2020-05-13 06:22:02 +10:00
Zuul
d3c7d2d4b1 Merge "functests: use ensure-virtualenv" 2020-05-05 08:40:55 +00:00