Commit Graph

165 Commits

Author SHA1 Message Date
Ian Wienand
3833c2e59c containerfile: fix tar extraction
Ic68e8c5b839cbc2852326747c68ef89f630f26a3 removed the sudo from the
tar extraction here, meaning that production is failing to create the
chroot.  This is hidden in testing because
DIB_CONTAINERFILE_PODMAN_ROOT is set.  Make the sudo here
unconditional.

Change-Id: I6e36e3fc65981f85fad12ea2cd10780fde9c37da
2021-11-10 11:42:49 +11:00
Zuul
82aa8c516d Merge "Add DIB_YUM_REPO_PACKAGE as an alternative to DIB_YUM_REPO_CONF" 2021-10-04 06:51:31 +00:00
Steve Baker
296c81b9ca Add DIB_YUM_REPO_PACKAGE as an alternative to DIB_YUM_REPO_CONF
A custom yum repository can now be configured by defining
`DIB_YUM_REPO_PACKAGE` as a yum available package or a URL to an rpm file.
This package can install repo files with any associated keys and
certificates.

A good example of such a package upstream is rdo-release[1] which
includes multiple repo files, the repo keys, and a root certificate.
This makes these repos impractical to install via DIB_YUM_REPO_CONF.

Downstream, repo packages like this a frequently used to bootstrap
development builds of RHEL with development repos.

[1] https://www.rdoproject.org/repos/rdo-release.rpm

Change-Id: I2832e723998c9bd7635cdf7541a4c20eff6294d2
2021-09-13 09:32:53 +12:00
Ian Wienand
5f47584196 yum-minimal: use DNF tools on host
The latest Debian bullseye release doesn't provide yum any more, only
DNF.  This breaks the minimal builds that are using on-host yum tools
to start the chroot.  Probe for yumdownloader, and if it's not there,
use DNF.

Note this requires "dnf download" which may not be packaged.  See
I21cfbd3935e48be4b92591ea36c7eed301230753 for a sample work-around
that installs this plugin in the nodepool-builder container.

Change-Id: Ia7f1e4d115cc67c378d865d91af94a07b8cdc6cc
2021-08-30 14:14:32 +10:00
Xinliang Liu
a6ee4d0c21 Introduce openEuler distro
Add openeuler-minimal element and add CI functional tests for both
x86_64 and arm64.

OpenEuler is an open source community driven YUM/DNF distro like
Fedora. It references Fedora and CentOS a lot for the rpm packages
building. So somewhat it can be treated as a redhat family distro
and reuse the YUM/DNF related elements to help build openEuler images.

For more info about openEuler, see: https://openeuler.org/en

Depends-On: https://review.opendev.org/c/zuul/zuul-jobs/+/803413
Change-Id: I3e06e49b524364c3a4edeba8bce7a8c06b9c7b76
2021-08-04 03:06:55 +00:00
Jay Faulkner
91da6ab885 Permit specification of extra bootstrap packages
This change permits the yum-minimal element to be used in downstream
custom distributions, which may have additional packages containing repo
config or GPG keys needed.

This could also be utilized at a later time to move the
distribution-specific logic in this method to each distribution element
separately.

Change-Id: Ic1434bb2fe7301086cf11ba6bd7f2ee187c5e6c8
2021-08-02 11:57:11 -07:00
Zuul
d286f64a76 Merge "Add element block-device-efi-lvm" 2021-07-08 01:08:26 +00:00
Ian Wienand
12b60c4088 Mount /sys RO
As noted inline, this works around potential issues by being a strong
indication you are in a container (e.g. [1]).  Since nothing should be
changing anything on the host/build system, this is a generically
safer way to operate.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1975588

Change-Id: Ic6802c4ffc2e825f129af10717860a2d1770fe80
2021-07-05 11:45:02 +10:00
Steve Baker
ab8d2910c4 Add element block-device-efi-lvm
Element block-device-efi-lvm has been added which is like
block-device-efi but defines an LVM logical group in the root
partition. Three logical volumes are defined in that group, mounted to
/, /var, and /home.

This volume layout will not meet all requirements, but this is more of
an example demonstrating the capability to encourage more usage of
this existing feature.

This is based on the overcloud-partition-uefi element in
tripleo-image-elements, and I believe this capability is too useful to
have the only working example buried in a related project repo.

This change also fixes the element string matching in
_arg_defaults_hack, the 'vm' test was also matching against 'lvm' and
'block-device-efi-lvm' elements. Also the 'block-device-' test now
properly tests for this being the prefix of the block-device element.

This change also makes block-device-efi fsck-passno compliant with the
documentation[1] so that / has value 1 and all other mounts are set to
2.

[1] https://www.man7.org/linux/man-pages/man5/fstab.5.html

Change-Id: If86a0e49186ce5a65cc0084101d31ce59a97b854
Blueprint: whole-disk-default
2021-06-01 17:27:28 +12:00
Ian Wienand
f6748a4cd4 bootloader: remove extlinux/syslinux path
This is a first pass through the bootloader, that removes the extlinux
and syslinux install/cleanup path.

Change-Id: Ifb107796cdb6748430a124bf13ced93db9689bff
2021-05-13 10:33:06 +10:00
Xinliang Liu
0c51b49414 Add aarch64 support for rhel
Change-Id: I86ccc56e37b214a45ba620b731b51f58d73471f8
2021-03-08 07:00:15 +00:00
Dmitry Tantsur
fd1848282f Remove the deprecated ironic-agent element
Despite having several issues (like missing firmware), it is still
used by people. It seems that the only way to stop that is to remove it.

Change-Id: I4baed8e8ab663c624dcc8d06ff0293d57b082abb
2021-01-21 14:06:08 +01:00
Bob Fournier
5d23d8e6b0 Add support for vlan interfaces in dhcp-all-interfaces.sh
Provide ability to run dhcp client on VLAN interfaces created on
top of an Ethernet interface.  See also
<https://storyboard.openstack.org/#!/story/2008298>`__ for further details.

Change-Id: Ic3ffd7b8e23b1e996cfe6c79ce0ff47e521f30be
2020-11-03 10:04:17 -05:00
Zuul
580256f374 Merge "Disable growpart in cloud-init-disable-resizefs" 2020-10-07 08:41:03 +00:00
Michal Nasiadka
520e12ec29 Disable growpart in cloud-init-disable-resizefs
This disables growpart module in cloud-init, not resizing / partition
to maximum disk free size by default.

Change-Id: I69984a9141fa8abb12dc5d51bd334f9280deca67
2020-10-06 14:03:57 +02:00
Zuul
cbeca0babd Merge "Add support for CentOS 8 Stream cloud image" 2020-09-24 14:55:23 +00:00
Zuul
7ef82de0db Merge "rhel-common: Provide method to select module streams" 2020-09-24 14:32:19 +00:00
Dmitry Tantsur
77fac28078 Rename duplicating 10-debian-minimal.bash
This file is present in both dpkg and debian-minimal element,
causing a failure to build anything with debian.

Change-Id: I8213d581a79bb432281f31955a44418e4047d9e1
2020-09-18 14:52:56 +10:00
Lon Hohberger
5299371957 rhel-common: Provide method to select module streams
Some OpenStack releases on RHEL require specific modules
in order to function correctly.  This adds the ability
to set DIB_DNF_MODULE_STREAMS which then are selected
prior to package installation.

Change-Id: I78d7bcf214a45245e2073428120fcbdd968e1acd
Signed-off-by: Lon Hohberger <lhh@redhat.com>
2020-09-16 08:43:30 -04:00
Zuul
3d4e8d749d Merge "Move centos python3 installation after RHEL subscription" 2020-09-16 08:50:23 +00:00
Noam Angel
e9b1997267 Move centos python3 installation after RHEL subscription
I088fc4284e889147ca9a375d4a159264cff53484 tried to slot the python3
install between the 00-dnf-update and before 01-package-installs;
however it also needs to run *after* the RHEL subscription
00-rhel-registration.

Thus a better place for it is 01-00-centos-python3, which will order
it after subscription and package updates, but before any use of
package-installs.

To avoid confusion over naming, move 00-0-dnf-update back to just
00-dnf-update.

Change-Id: Ib7c82895769e4889d47e10c4b37e06a42c053903
2020-09-07 11:22:55 +10:00
Matthew Thode
8cc08418d7 Copy apt gpg keys directly into trusted.gpg.d
This avoids having to have gnupg2/apt-key dependencies in the base,
and is now well supported by modern Debuntu.

Signed-off-by: Matthew Thode <mthode@mthode.org>
Change-Id: I7065b2fab6125d9635ef99ff65d374b8b6b4c3a2
2020-08-28 15:58:07 +10:00
Zuul
5df71c0082 Merge "Makes EFI images bootable by bios" 2020-08-18 07:59:01 +00:00
Zuul
f49d6c996b Merge "Fedora 32 support" 2020-08-18 07:43:50 +00:00
Pierre Crégut
6e71bd4239 Makes EFI images bootable by bios
For Bios and EFI compatibility, grub must be installed twice.
This patch adds the bios version when EFI is selected. The GPT EFI  block partitioning
already adds the bios partition, but the bootloader only called grub once.

Change-Id: Iee6c8b3b97b3cfff4562bcb30a50800f5ade894a
Closes-Bug: #1889089
2020-08-18 14:41:21 +10:00
Ian Wienand
7b6819213e Fedora 32 support
Update for Fedora 32 support.

Change-Id: I51c5645856a76e2877c013d72e9849a758ba12ff
2020-08-17 19:40:02 +10:00
Julia Kreger
46d12ae7d3 Handle NetworkManager for dhcp-all-interfaces
NetworkManager takes a distinctly differnet network management
approach and the bulk of the dhcp-all-interfaces code is largely
targetted at distribution specific configuration. Some which may
or may not override settings, or only partially assert desirable
settings.

As such, we need to set appropriate configuration, such as the
correct client to be used, and timeouts based upon user supplied
settings.

By default this change sets the client to be dhclient on redhat
styled machines, as the packaging default, while it works for
ramdisk usage, it does not reset the interface between retries,
which can be critical if the infrastucture operator is attempting
to configure LACP trunks to the end node.

Change-Id: I0e0cfbdbf7ef2b2861b934ccd7dab9d83a35c8f0
Story: 2008001
Task: 40648
2020-08-11 08:12:31 -07:00
Carlos Goncalves
e4b6a2faef Add support for CentOS 8 Stream cloud image
This patch adds support for CentOS 8 Stream [1] to the centos element
(cloud image). Users should set DIB_RELEASE=8-stream.

[1] https://www.centos.org/stream/

Change-Id: Ib8f542031c46326ffed812fa60cbc9e56db9d6fd
2020-08-10 11:33:38 +02:00
Ian Wienand
8662297517 Deprecate dib-python; remove from in-tree elements
We are at the point that all distributions we are building have Python
3, so any tools running in the chroot can assume Python 3 exists.
This makes dib-python redundant; mark it as deprecated and start to
remove it from elements where it is no longer required.

Change-Id: I5d852843ec65d3b04444b77c54c5b82424455cd8
2020-08-07 10:38:16 +10:00
Zuul
b6d4bef9eb Merge "Use kpartx option to update partition mappings" 2020-07-07 08:27:29 +00:00
Zuul
dba1d390da Merge "Switch to newer openstackdocstheme and reno versions" 2020-07-06 18:52:41 +00:00
Carlos Goncalves
367dfc9294 Add support for CentOS 8 Stream
This patch adds support for CentOS 8 Stream [1] to the centos-minimal
element. Users should set DIB_RELEASE=8-stream.

[1] https://www.centos.org/stream/

Change-Id: Id0825de735ab957c10daf35fb3c641f850cc6847
2020-06-22 10:36:30 +02:00
Zuul
2e6a7f949c Merge "Remove virtualenv activation" 2020-06-12 01:17:17 +00:00
Dmitry Tantsur
e793cc4038 Remove virtualenv activation
Since the original merge of this code
(04208e7c79) several things have
changed; particularly now we ship dib-run-parts as part of dib, not as
a separate package.

We setup $_LIB to point to the shipped library diretory via
pkg_resources lookups.  We now call dib-run-parts (as mentioned,
shipped as a dib library now), source scripts, etc. via $_LIB and thus
do not rely on $PATH.  Consequently we don't need this activation
part.

Which is helpful, because "venv" (as opposed to virtualenv) doesn't
have activate_this.py.  So this fixes installation under that for
Python 3.

We update the functional tests to use the virtualenv_command exported
by the ensure-pip role, which will test the venv path.  There is no
need for dib_python as we are Python 3 only now.

Change-Id: Iede929ea2d278008220aac8b1d678ba41eba0d8a
2020-06-11 16:49:15 +10:00
Zuul
a0714c2300 Merge "Debuntu: add apt-transport-https" 2020-06-09 10:33:56 +00:00
melissaml
abba0b5574 Switch to newer openstackdocstheme and reno versions
Switch to openstackdocstheme 2.2.1 and reno 3.1.0 versions. Using
these versions will allow especially:
* Linking from HTML to PDF document
* Allow parallel building of documents
* Fix some rendering problems

Update Sphinx version as well.

Disable openstackdocs_auto_name to use 'project' variable as name.

Change pygments_style to 'native' since old theme version always used
'native' and the theme now respects the setting and using 'sphinx' can
lead to some strange rendering.

openstackdocstheme renames some variables, so follow the renames
before the next release removes them. A couple of variables are also
not needed anymore, remove them.

See also
http://lists.openstack.org/pipermail/openstack-discuss/2020-May/014971.html

Change-Id: If3e8d8ac29a728aa41562b31976ebe9bfa5df66f
2020-06-09 10:23:37 +02:00
Simon Westphahl
4a424ecabb Use kpartx option to update partition mappings
Fix cases of 'mkfs' failing because the partitions never showed up. Partition
mappings will now be updated instead of just adding them with 'kpartx'. That
means that 'kpartx' will also remove devmappings for deleted partitions.

Traceback of failing mkfs call:

2020-05-11 22:03:25.523 | INFO diskimage_builder.block_device.utils [-] Calling [sudo sync]
2020-05-11 22:03:25.539 | INFO diskimage_builder.block_device.utils [-] Calling [sudo kpartx -avs /dev/loop0]
2020-05-11 22:03:25.581 | INFO diskimage_builder.block_device.utils [-] Calling [sudo mkfs -t ext4 -i 4096 -J size=64 -L cloudimg-rootfs -U 21c6f9eb-4d52-4e5c-b9b7-796735de8909 -q /dev/mapper/loop0p1]
2020-05-11 22:03:25.700 | ERROR diskimage_builder.block_device.blockdevice [-] Create failed; rollback initiated
2020-05-11 22:03:25.700 | Traceback (most recent call last):
2020-05-11 22:03:25.700 |   File "/home/zuul/dib/lib/python3.6/site-packages/diskimage_builder/block_device/blockdevice.py", line 406, in cmd_create
2020-05-11 22:03:25.700 |     node.create()
2020-05-11 22:03:25.700 |   File "/home/zuul/dib/lib/python3.6/site-packages/diskimage_builder/block_device/level2/mkfs.py", line 133, in create
2020-05-11 22:03:25.700 |     exec_sudo(cmd)
2020-05-11 22:03:25.700 |   File "/home/zuul/dib/lib/python3.6/site-packages/diskimage_builder/block_device/utils.py", line 143, in exec_sudo
2020-05-11 22:03:25.700 |     raise e
2020-05-11 22:03:25.700 | diskimage_builder.block_device.exception.BlockDeviceSetupException: exec_sudo failed
2020-05-11 22:03:25.700 | INFO diskimage_builder.block_device.level0.localloop [-] loopdev detach
2020-05-11 22:03:25.701 | INFO diskimage_builder.block_device.utils [-] Calling [sudo losetup -d /dev/loop0]
2020-05-11 22:03:25.732 | INFO diskimage_builder.block_device.level0.localloop [-] Remove image file [/tmp/dib_image.muyw7t1h/image0.raw]
2020-05-11 22:03:25.734 | ERROR diskimage_builder.block_device.blockdevice [-] Rollback complete, exiting
2020-05-11 22:03:25.740 | Traceback (most recent call last):
2020-05-11 22:03:25.740 |   File "/home/zuul/dib/bin/dib-block-device", line 8, in <module>
2020-05-11 22:03:25.740 |     sys.exit(main())
2020-05-11 22:03:25.740 |   File "/home/zuul/dib/lib/python3.6/site-packages/diskimage_builder/block_device/cmd.py", line 120, in main
2020-05-11 22:03:25.740 |     return bdc.main()
2020-05-11 22:03:25.740 |   File "/home/zuul/dib/lib/python3.6/site-packages/diskimage_builder/block_device/cmd.py", line 115, in main
2020-05-11 22:03:25.740 |     self.args.func()
2020-05-11 22:03:25.740 |   File "/home/zuul/dib/lib/python3.6/site-packages/diskimage_builder/block_device/cmd.py", line 36, in cmd_create
2020-05-11 22:03:25.740 |     self.bd.cmd_create()
2020-05-11 22:03:25.740 |   File "/home/zuul/dib/lib/python3.6/site-packages/diskimage_builder/block_device/blockdevice.py", line 406, in cmd_create
2020-05-11 22:03:25.740 |     node.create()
2020-05-11 22:03:25.740 |   File "/home/zuul/dib/lib/python3.6/site-packages/diskimage_builder/block_device/level2/mkfs.py", line 133, in create
2020-05-11 22:03:25.740 |     exec_sudo(cmd)
2020-05-11 22:03:25.740 |   File "/home/zuul/dib/lib/python3.6/site-packages/diskimage_builder/block_device/utils.py", line 143, in exec_sudo
2020-05-11 22:03:25.740 |     raise e
2020-05-11 22:03:25.740 | diskimage_builder.block_device.exception.BlockDeviceSetupException: exec_sudo failed

Change-Id: I374f7f22f9e93ef35eb5813712ca59e75f0733e8
Related-Bug: #1698337
2020-06-09 09:07:55 +02:00
Ian Wienand
de57271ed2 Prepare to drop Python 2 support
Pin func test requirements to stable/train; the last stable release
with Python 2 support.  Switch to the python-jobs-no-constraints
template to avoid using master constraints file.

Only build focal on bionic, other debootstraps are too old.

Remove pip-and-virtualenv testing as we are moving to plain images.

The tripleo-buildimage-jobs are unstable; see linked bug.

Add a note that this is the last Python 2 release.

Change-Id: Ibde7a564dd41cc2d6e80e2dffe5a95a57bbf8ada
2020-05-29 09:44:54 +10:00
Ian Wienand
0c94eef7be Revert "dib-lint: use yamllint to parse YAML files"
This reverts commit 6ee2995214 and
e85c2a6f03.

I missed that if you pip install and then run dib-lint, it's not going
to pick up the .yamllint file shipped here.  Thus it gives spurious
errors.

The reason for this was simply better duplicate key detection in yaml
files, which caused us problems with the kernel installs.  However, at
this point it seems just the old "does it load" test from pyyaml will
be enough.

Change-Id: I87a9fc9bb119cfeffad48fc0fa0df31f0181825d
2020-05-28 16:44:49 +10:00
Monty Taylor
e85c2a6f03 Add dependency on yamllint
dib-lint requires yamllint now, so we need to depend on it.

Change-Id: Iccd2cdbb04bd9b429d8d19b31ff3b10cdd568e15
2020-05-27 10:28:19 -05:00
Ian Wienand
7539e241da ubuntu-minimal: Add Ubuntu Focal test build
Add test builds for Focal on x86 and ARM64

Change-Id: Idb23f0e00d37c7447441ea002aad078e8c61f969
2020-05-21 14:03:54 +10:00
Ian Wienand
138d3f9b81 package-installs: allow when filter to be a list
Allow the "when:" statements to be a list of values, which are
effectively anded together to filter the package install.

Change-Id: Ia6f10741fa6be24b11d6991c8a6b6e07951ff68d
2020-05-21 14:03:49 +10:00
Zuul
eebb68b385 Merge "pip-and-virtualenv: drop f31 & tumbleweed, rework suse 15 install" 2020-04-23 23:52:10 +00:00
Ian Wienand
262281e5f7 Remove Trusty testing
Opendev has dropped trusty from the mirrors.  With no testing the
distribution is effectively unsupported, add a release note.  Update a
few other random doc bits (that are not really that up to date
anyway).

Change-Id: I5bd0d0a94477cf8d84cef72f5d4b2e9e15ab9fd2
2020-04-23 10:00:13 +10:00
Ian Wienand
ac7fba7040 pip-and-virtualenv: drop f31 & tumbleweed, rework suse 15 install
This is an alternative approach to commit
68bb43535e.  I think this proposes a
better overall solution that the prior change which had the Python 3
packages being installed, but did not specify the _do_py3 flag to do
the installation steps that redirect the various tool installations.

Fedora 31+ doesn't have python2, and Tumbleweed does have some Python
2 support but there seems to be no reason to bother updating this
element for either with infra very close to removing this completely
[1].  Error out on these platforms, and add a release note.

The 15 path should include the python2 and python3 packages, along
with the flags to do the "cleanup"; i.e. forced removal of distutils
packages that pip 10+ won't touch.  As mentioned in the original
change, the six package causes problems here, but we can clear that
too by explicitly listing it instead of letting it come in via
dependencies.  Again, this element will be removed from the infra 15
builds ASAP; but we can release with this to provide a roll-back point
if we need to revert the removal to fix things temporarily.

Add it to the testing path as well.

[1] https://docs.opendev.org/opendev/infra-specs/latest/specs/cleanup-test-node-python.html

Change-Id: I7a6a342461d6001c25e55638ba9b7438c28f2519
2020-04-23 08:10:26 +10:00
Ian Wienand
434d2db5d4 Debuntu: add apt-transport-https
I don't see anywhere we bring this in, especially on a minimal build.
In 2020 it seems like a base dependency, put it alongside
software-properties-common that installs the other apt helper bits.

Change-Id: I5b079eac4912cb4a164e9aa6158ed106a28f576c
2020-04-02 10:11:35 +11:00
Harald Jensås
1ac31afd62 Use rpm -e instead of dnf for cleaning old kernels
If the running kernel of the system building the image
matches the kernel that is to be removed dnf will fail.

Repalce use use of dnf with rpm -e.

Closes-Bug: #1623409
Change-Id: Ie2481ea8a02b7b0720e46fa179f24badf4aa25c5
2020-03-19 22:35:23 +01:00
Zuul
61b72ca2c2 Merge "Add ensure-venv element, install glean with it" 2020-03-10 05:08:54 +00:00
Ian Wienand
82cfcfe551 Add ensure-venv element, install glean with it
All the platforms we care about now have python3 with venv (even
centos7 now) packaged somehow.  Add an ensure-venv element to make
sure that "python3 -m venv" works.  Any other elements that wish to
install non-distribution-packaged Python utilities can use this to
keep them separate from the main system installs.

Port glean to use this, and drop its dependency on pip-and-virtualenv.

Change-Id: Ic16f134fe34293bb68e7c632dd320f523366320d
2020-03-10 11:57:43 +11:00
Carlos Goncalves
8226384cf0 Add CentOS 8 support
* Add "centos" element, a CentOS version-independent element. This is in
  line with the same work done for RHEL in Stein cycle.
* Deprecate the centos7 element. CentOS 7 support itself it not
  deprecated though. The new "centos" element provides the same support
  level as the "centos7" element.
* Add functional testing

The default CentOS version is 8. You can adjust it using the DIB_RELEASE
environment variable.

Change-Id: I373ba2296c4613765676e59aabd9c651345298d1
2020-02-19 10:44:56 +01:00