Commit Graph

553 Commits

Author SHA1 Message Date
Jesse Pretorius
30493f7655 Move common ubuntu environment setting to ubuntu-common element
The ubuntu, and ubuntu-minimal elements both make use of a common set
of environment settings to determine the distribution name.

The ubuntu-minimal element also does a few extra things which would
appear to apply to both sets and bring in extra architecture support.
As such, these are included in the common element.

This intends to be part of a series of patches which will eventually
create a new element to build a minimal ubuntu-systemd-container
element which can be used for lxc/nspawn containers.

Change-Id: Ia4e620f7d3fa6215484a8d218cea2f28bd1ffaee
2018-09-20 00:04:55 +00:00
Zuul
b86af3dc6a Merge " IPA requires iptables" 2018-09-13 02:05:42 +00:00
Ian Wienand
f6a2452d4c Only append DIB_BOOTLOADER_DEFAULT_CMDLINE to default grub entry
The grub.cfg has two variables [1]

 GRUB_CMDLINE_LINUX : used on all boots
 GRUB_CMDLINE_LINUX_DEFAULT : additionally used on all "normal" boots

The problem with I2298675dda1f699c572b3423e7274bc8bd7c1c9d is that it
appened the values in DIB_BOOTLOADER_DEFAULT_CMDLINE to both of these,
resulting in duplicated arguments.  I don't think we considered that
GRUB_CMDLINE_LINUX_DEFAULT actually already appends to the
GRUB_CMDLINE_LINUX values.

Make DIB_BOOTLOADER_DEFAULT_CMDLINE only append itself to
GRUB_CMDLINE_LINUX_DEFAULT.  That seems to line up sensibly with the
name of the variable.

Documentation is enhanced around this, and a releasenote added.

[1] https://help.ubuntu.com/community/Grub2/Setup

Change-Id: I76b5442a9090c19a6540ed2d4ab324546f241ebf
Closes: #1791736
2018-09-13 09:51:50 +10:00
Zuul
b29aee3383 Merge "Install sudo on Gentoo images by deault" 2018-09-11 00:09:33 +00:00
Zuul
9f93b2ce3d Merge "Fix CentOS image build failure when dib runs on system where audit disabled" 2018-09-06 11:24:59 +00:00
Zuul
8473e2e373 Merge "Replace assertRaisesRegexp with assertRaisesRegex" 2018-09-06 11:22:20 +00:00
Olivier Bourdon
7566819139 Fix CentOS image build failure when dib runs on system where audit disabled
Without this fix, building a CentOS image on Ubuntu where audit=0 is passed
as a kernel boot parameter will lead to the following error:
disk-image-create centos7 dhcp-all-interfaces cloud-init-nocloud \
    devuser yum epel baremetal
... dib-run-parts Running tmpdir/hooks/cleanup.d/99-selinux-fixfiles-restore
... Error connecting to audit system.

Change-Id: I229d9b72f88bffddca42da57f01c27e902427071
2018-09-04 08:23:02 +02:00
Charalampos Kominos
c85141291e Fix bootloader packages for aarch64
Due to the arm naming convention, building centos images for arm64 and
aarch64 does not yield the same result. In order to locate grub2 on
aarch64 the correct mapping is added.

Change-Id: I1bb227b2523e420e394fec8c52c6c79fcdd31c53
Closes-Bug:#1789414
Signed-off-by: Charalampos Kominos <Charalampos.Kominos@enea.com>
2018-08-31 17:01:47 +02:00
Chuck Short
b67cfc6950 Replace assertRaisesRegexp with assertRaisesRegex
This replaces the deprecated (in python 3.2) unittest.TestCase method
assertRaisesRegexp() with assertRaisesRegex().

Change-Id: I0ac482741ad4adc1127dd9e9f64c1c8101f370e5
Signed-off-by: Chuck Short <chucks@redhat.com>
2018-08-27 10:40:36 -04:00
Zuul
b602c05551 Merge "Add netcat to redhat-common map-packages" 2018-08-23 11:26:12 +00:00
Zuul
f33526d20c Merge "better handle existing keywords files/directories" 2018-08-10 06:07:06 +00:00
Carlos Goncalves
138b14b010 Install ca-certificate with redhat-common
Change-Id: I36d44ca8b4f966476657ec2bb1ecc1458bb524fd
2018-08-04 00:30:49 +02:00
Carlos Goncalves
cf553ce0ac Add netcat to redhat-common map-packages
Change-Id: I273038611febe5e30f30dd8d3ba8990dbdc94647
2018-08-03 17:20:19 +02:00
Zuul
3be4b0c1fd Merge "Only detach device if all partitions have been cleaned" 2018-07-31 08:21:27 +00:00
Zuul
3197a7ef1b Merge "Move LVM cleanup phase into cleanup" 2018-07-31 00:30:47 +00:00
Zuul
53186f23a0 Merge "cache-url requires curl" 2018-07-31 00:20:56 +00:00
Zuul
d50bd1deb3 Merge "Don't quote names with sgdisk" 2018-07-30 06:26:25 +00:00
Yolanda Robla
64bb87f7b5 Only detach device if all partitions have been cleaned
Currently there is a bug, that tries to detach the device from a
partition at the first try, without considering that there may be
other partitions and volumes on it. Ensure that the detach is done
properly, and add a test to ensure that this happens correctly.

Change-Id: I35c5a473509f17a70270a2cbf5bf579faaeb123a
Fixes-Bug: #1777861
2018-07-30 16:24:57 +10:00
Ian Wienand
7302f38f97 Move LVM cleanup phase into cleanup
A recap -- we run umount phase then cleanup phase.

Currently we register a object to do the final LVM cleanup based on
the parent PV.  In light of I697bfbf042816c5ddf170bde9534cc4f0c7279ff,
I believe this should just be done in the cleanup phase.  Note there
was probably additional confusion because the partition removal was
done in the cleanup phase until
I7af3c5cf66afd81a481f454b5207af552ad52a32, where is was moved into the
umount phase.

Thus it is moved into the cleanup() function and this should now run,
per the comment, after everything is unmounted in umount phase.

This also exposes that we didn't have the cleanup phase in the unit
tests (because it wasn't doing anything I guess).  Add it.

Change-Id: I1c5f4ffc9619c774f78d21b918a81647b3dc28f5
2018-07-30 14:35:16 +10:00
Oliver Walsh
a514feba99 Add DIB element to blacklist nouveau
This ensures nouveau is not loaded at boot, which is required when installing
NVIDIA GPU drivers and to avoid issues with PCI passthrough of NVIDIA GPUs.

The option to disable kernel modesets ensures that it can be unloaded again if
it happens to be loaded after boot (e.g manually or implicitly by X).

bp tripleo-vgpu

Change-Id: I60815de86e7b22dfb39555af9d2d53564841e2ab
Related-bug: 1774674
2018-07-25 16:35:58 +01:00
Oliver Walsh
73e27a8504 modprobe DIB_MODPROBE_BLACKLIST should be optional
modprobe element currently fails when DIB_MODPROBE_BLACKLIST is not set.
As there are now two methods to control blacklisting this should be optional.

Change-Id: Ibf3c31a95177ba88c1b93228490c7f36f5b70b57
2018-07-25 16:33:47 +01:00
Zuul
48645abff6 Merge "Call kpartx remove in umount, not cleanup" 2018-07-24 23:05:16 +00:00
Zuul
0a40f45094 Merge "Move localloop to exec_sudo" 2018-07-24 23:05:15 +00:00
Zuul
961235854b Merge "block-device lvm: fix umount phase" 2018-07-24 11:26:11 +00:00
Zuul
9adf12fe4a Merge "Fix for proper LVM support" 2018-07-24 07:32:11 +00:00
Clint Byrum
d8907e78b5 cache-url requires curl
In some cases cache-url can get pulled in without curl, causing it to
fail.

Co-Authored-By: Adam Harwell <flux.adam@gmail.com>

Change-Id: Ibd66c2ca4f8cc423783555d8a99b1184f43adff2
2018-07-23 09:56:58 +00:00
Ian Wienand
1107326723 Update pylint to 1.7.6, uncap networkx
This review squashes:
    Iac9afc7766d3640815dc20cfd6de1245d36a09cc
    Ie894b5801bd7b3815432882cd626941e89d9f9a1

We need to do this as we can't fix pylint without networkx as that
failes requirements-chak due to us having a cap on networkx and we can't
uncap networkx as part of tripleo-buildimage installs without
constratints which gets us 2.1 and DIB desn't support 2.x

This is the commit message Iac9afc7766d3640815dc20cfd6de1245d36a09cc
---
One of the pylint dependencies has updated to be python3 only; this
version of pylint correctly caps things so it still works with
python2.

This also exposes that we need to uncap networkx due to
I34045f87ca19c2f184b040f4d89347374cce518b.  We should remain on
version 1 for now thanks to upper-constraints, but we need to maintain
the lower-constraint.
---

This is the commit message Ie894b5801bd7b3815432882cd626941e89d9f9a1
---
Support different versions of networkx

Since the entry of networkx 2.0 nodes has a different
behaviour. Checking if dg.nodes is iterable is enough to add
compatibility for new/older versions.
---

Change-Id: I82dc61fac6c156a4f0d574290c7632077aa53195
2018-07-18 09:27:01 +10:00
Zuul
b79952af2a Merge "Add new modprobe element" 2018-07-11 07:21:40 +00:00
Zuul
bbf69a90f3 Merge "Fix /etc/network/interfaces file contents" 2018-07-10 07:46:56 +00:00
Zuul
2343d4b577 Merge "Add keyring if supplied" 2018-07-10 07:46:55 +00:00
Olivier Bourdon
caf565673b Fix for proper LVM support
Without this fix, a LVM based ubuntu-minimal image will fail
booting due to the fact that the boot process will not be able
to retrieve the root filesystem using LABEL=(cloud)img-rootfs

Change-Id: If4ecf65868563f7b799160a58af6312bedf956bf
2018-07-09 14:15:57 +00:00
Hironori Shiina
7e4e6cfff4 Add expected semicolons for dhclient.conf
This patch adds an expected semicolon to an end of statement in
dhclient.conf for dhcp-all-interfaces element. Without this fix, an
error occurs when an image is booted with a message,
'semicolon expected.'.

Change-Id: I8311dbc67cc2815223111da01e7a7517c7d6f059
2018-07-06 13:42:25 +09:00
Sam Yaple
c144246cc9
Add keyring if supplied
When building with debootstrap, debootstrap will use the key to check
that everything is properly signed. It will not `apt-key add` the key
into the final environment, however.

Early adding the key after debootstrap before we need to read from the
private repo again prevents unsigned issues. This also maintains the
integrity of the packages in the environment throughout the build.

Change-Id: I5ca75ae4620c9fb26b512cb30f8cd79fa7a0373a
2018-07-02 14:33:35 -04:00
Ian Wienand
f94943344f Call kpartx remove in umount, not cleanup
Similar to I697bfbf042816c5ddf170bde9534cc4f0c7279ff, the order of
things called is "dib-block-device umount" *then* "dib-block-device
cleanup".

Because we're doing the "kpartx -d" here in cleanup, it means that the
loop-device is removed in umount phase from level0/localloop.py, then
afterwards we try and remove the partitions.

Change-Id: I7af3c5cf66afd81a481f454b5207af552ad52a32
TODO: a test case to ensure the ordering
2018-06-29 11:22:33 +10:00
Ian Wienand
a1a549548a Move localloop to exec_sudo
One call in localloop requires the output of the command, so modify
exec_sudo to buffer up output and return it.  This is modelled on the
same thing in package-installs-v2 which seems to work.  Rather than
return a subprocess exception, return a dib exception which everything
should have imported anyway.

The overall reason for this is to make our external calls more
consistent for mocking in unit testing.

Change-Id: I10d23b873dee9f775daef2a4c8be5671d02c386e
2018-06-29 11:22:24 +10:00
Zuul
927e8115f6 Merge "Fix bootloader for efi on rhel systems" 2018-06-28 15:02:30 +00:00
Yolanda Robla
31383970c7 Add new modprobe element
This element will replace modprobe-blacklist element. It wil
still have the blacklist functionality, but it also adds
the feature of passing a complete file with settings to the
modprobe.d directory. Adding this functionality, that will
allow elements that depends on this module, to just copy the
specified files to the final directory.

Change-Id: I9a44f7d11520b8b1e604956d3c1db2fc7e2bf457
2018-06-28 13:55:53 +02:00
Andreas Florath
f5736f3178 block-device lvm: fix umount phase
As described in blockdevice.py detachment and (most) resources
release must be done in the umount phase of a block device module.

Until now these jobs were done in the lvm cleanup() phase - which
is too late - especially when using nested LVMs.

This patch moves the functionality of the cleanup() phase to the
umount() phase for the lvm module.
It includes a test case that fails without applying the provided
source code changes.

Change-Id: I697bfbf042816c5ddf170bde9534cc4f0c7279ff
Signed-off-by: Andreas Florath <andreas@florath.net>
2018-06-28 15:21:59 +10:00
Zuul
e796b3bc18 Merge "Add iscsi-boot element for CentOS images" 2018-06-27 09:54:10 +00:00
Ian Wienand
b0da703f46 Don't quote names with sgdisk
Our sgdisk calls are putting extra double-quotes around the names of
partitions.  This confuses sfdisk, which confuses growpart, which
confuses growroot ... and you don't get your partition grown for EFI
boot.

Ensure we just bunch arguments into the list directly (for Popen)
rather than string split and have to worry about quoting.  Add a check
for this to our GPT unit test, extending it to include a space in the
name of the root partition.

Change-Id: I0a8cb69bb4c9c0865fbaa63ba0d7210028da552e
2018-06-27 18:10:08 +10:00
Matthew Thode
b9f1c7a22f
better handle existing keywords files/directories
The existing directories are needed for stage building (a part of the
Gentoo build process).  Normally these directories are empty, but there
are times where overrides need to be defined.  This commit handles
existing overrides for keywords.  For historical reasons the overrides
were able to be put in different files and directories, this
centralizes them.

This also updates the version of openssl/cryptography that works with
or without bindist.

Change-Id: I62c934ed305a711a4a9a3ef01fa55ad142aebb78
2018-06-25 09:20:52 -05:00
Hoang Trung Hieu
a4648872ba Add iscsi-boot element for CentOS images
This patch adds an element that handles the configuration for
creating a disk capable of being a remote root filesystem through
iSCSI on CentOS images.

Tested on Fujitsu Server and boot with BIOS and UEFI mode successfully.
- Tested Boot-From-Volume + EFI for centos7 with following elements:
  "centos7 vm devuser cloud-init-datasources dhcp-all-interfaces
   iscsi-boot dracut-regenerate block-device-efi"

Co-authored-By: Nguyen Van Trung <trungnv@vn.fujitsu.com>

Change-Id: Ia1f23d722dced6f254fd7aee86abe8066a72fa42
2018-06-25 12:01:52 +00:00
Zuul
e39adcd65f Merge "Remove redundant word" 2018-06-22 14:23:47 +00:00
chenxiangui
c809160906 Remove redundant word
Remove the redundant word 'the' in config.py

Change-Id: I3e9cb6390ce196f0a9022aef10f6c7b1ace36c48
2018-06-19 18:00:11 +08:00
Olivier Bourdon
11d91501d0 Fix /etc/network/interfaces file contents
According to http://bit.ly/2HA4oDO and
the official Ubuntu manual
http://manpages.ubuntu.com/manpages/xenial/man5/interfaces.5.html
source-dir support has been removed from Ubuntu >= 16.04/Xenial

Once an image is generated and booted, moving the dhcp interface(s)
declaration(s) from /etc/network/interfaces into specific subentries
of /etc/network/interfaces.d and calling 'service networking restart'
just make your instance unreachable and all interfaces are left
unconfigured.

This patchset fixes this issue

Change-Id: I6b6b99c81490c874c5db5405c2fbf3c180c87464
2018-06-19 11:26:21 +02:00
Yolanda Robla
9687a1efe1 Convert labels to upper case
When booting on UEFI, there was an issue mounting the vfat
filesystem. It was caused because the mount was defined in
/etc/fstab in lowercase, but the disk had it labeled in upper
case, and system could not find it. Conver the label to upper
case in case of fat/vfat.

Change-Id: Id3dee735e6f8fb221d199c4aba648f3e9a6e4206
2018-06-19 11:12:54 +02:00
Yolanda Robla
61e6566c48 Fix bootloader for efi on rhel systems
When building the image on a non-efi environment, it generates
linux16/initrd16 entries. But to boot from UEFI they need to have
linuxefi/initrdefi entries.
Use sed to replace those entries, in case we have an EFI image.

Change-Id: I47c96450e10f34b91bcc32888532bd7ab87cf316
2018-06-19 08:37:30 +02:00
Zuul
e210f79500 Merge "Don't run setfiles on /boot/efi" 2018-06-15 08:42:13 +00:00
Nguyen Van Trung
8d86ff1bec Don't run setfiles on /boot/efi
setfiles isn't supported on the vfat /boot/efi partition.  Add it to
the skip list.

Tested on Fujitsu Server successfully.

Change-Id: Iab262c4bdb0ecc25ca6b77ee4aff1ce442c0c578
2018-06-15 14:53:38 +10:00
Michael Turek
91e3b72a23 Add iscsi-boot element
This patch adds an element that handles the configuration for
creating a disk capable of being a remote root filesystem through
iSCSI on Ubuntu and Debian images.

Change-Id: Ibf9e39d2bdab530106015f156d23d28029d12b0d
Closes-bug: #1716794
2018-06-14 08:56:03 +07:00
Yolanda Robla
bfc60958bf Fix bootloader packages for rhel
When using uefi in rhel, the package mapping is incorrect.
We need to add specific grub-efi* mappings to use grub2-efi

Change-Id: I2db96ae85fd5e4638c794015b2f8164c018420e3
2018-06-08 17:14:19 +02:00
Zuul
fc3748f49b Merge "Fix encoding issue during processing output" 2018-06-06 07:58:14 +00:00
Zuul
53c2e35f1b Merge "Save and close stdout on exit" 2018-06-06 06:30:05 +00:00
Zuul
8bdba50459 Merge "Reduce path length in PS4 for debug" 2018-06-06 06:20:28 +00:00
Zuul
29230d74c5 Merge "Use surrogateescape with outfilter.py" 2018-06-06 06:20:27 +00:00
Zuul
225c09b245 Merge "elements: pip-and-virtualenv: Handle openSUSE Leap 15" 2018-05-31 21:54:59 +00:00
Zuul
c4c2fb746c Merge "Remove duplicate GRUB command line entry" 2018-05-31 16:00:29 +00:00
Markos Chandras
f37e85d547 elements: pip-and-virtualenv: Handle openSUSE Leap 15
We need to handle openSUSE Leap 15 when installing pip and virtualenv
packages. This fixes the following problem when the pip-and-virtualenv
elements is used:

2018-05-31 09:42:12.014 | + [[ opensuse = opensuse ]]
2018-05-31 09:42:12.014 | /tmp/in_target.d/install.d/04-install-pip: line 57: packages: unbound variable

Change-Id: Id7911b0a0836fa8dcc003e23fa515b78fba67126
2018-05-31 10:55:28 +01:00
Zuul
4c04b46db6 Merge "Allow to rebuild arbitrary images" 2018-05-29 17:06:48 +00:00
Zuul
df9402c81c Merge "elements: zypper-minimal: Add support for openSUSE Leap 15.X" 2018-05-25 08:55:53 +00:00
Ian Wienand
8fb2f5cb55 Save and close stdout on exit
Redirecting our output through outfilter.py is inherently a bit racy,
since the disk-image-create process will exit, and then you might get
outfilter.py flushing any remaining output as it closes.

On an interactive prompt this might lead to final output overwriting
the prompt, etc.  This can be a bit confusing when you start running
things in a loop.

If we save the original fd, then on the exit path close the redirected
fd's and wait a little bit for final output (as a result of the
close), we get a more consistent output.

Change-Id: I8efe57ab421c1941e99bdecab62c6e21a87e4584
2018-05-25 11:48:20 +10:00
Ian Wienand
7b98433c46 Reduce path length in PS4 for debug
Strip everything before "site-packages" in the output filename for the
PS4 prompt.  This makes the line in debug logs significantly shorter
as we don't have the full virtualenv path every single time.  The
important thing -- the file being called in the lib/ dir, is retained.

Change-Id: I00706b6f6c0425c7795f997c08ceda3374dc84b5
2018-05-25 11:48:20 +10:00
Ian Wienand
596062b0f9 Use surrogateescape with outfilter.py
When switching to using log-file capture, we're getting

 [gentoo/build-succeeds] outfile.write(ts_line.encode('utf-8'))
 [gentoo/build-succeeds] UnicodeEncodeError: 'utf-8' codec can't
   encode character 'udcc5' in position 59: surrogates not allowed

Use surrogateescape [1] on the output to avoid this

[1] https://www.python.org/dev/peps/pep-0383/

Change-Id: I2c2c537296edfa5a8fe661a41bd5bfb3bfcf57e3
2018-05-25 11:48:14 +10:00
Roman Gorshunov
84eea81efe Allow to rebuild arbitrary images
Patch allows to rebuild arbitrary images, which location, filename and
sha256sum are specified in variables, not only hardcoded $DIB_RELEASE/current.

Change-Id: I05418932a0c40d885fe00a49f1f49d7e86c67518
2018-05-24 10:19:59 +00:00
Zuul
668c93b118 Merge "Replace the ubuntu-minimal trusty test with a bionic one" 2018-05-21 06:01:53 +00:00
Ian Wienand
82eb1ca837 Replace the ubuntu-minimal trusty test with a bionic one
Add a bionic test in replacement of trusty.  We are already building
bionic images in the gate, so this seems like a good time to switch.

Change-Id: I20d4c25e9b79e7326c86767c36be8615ba0888a3
2018-05-21 12:51:31 +10:00
Roman Gorshunov
09af52a08c Remove non-maintained ubuntu-core element
Removing no longer working and no longer maintained ubuntu-core element, which
intent is unclear, and not documented.

Change-Id: Id847591d04fd7cd32c8903967da01ee0d303b267
Closes-Bug: 1771614
2018-05-18 09:28:02 -07:00
Markos Chandras
bb9ad09010 elements: zypper-minimal: Add support for openSUSE Leap 15.X
openSUSE Leap shares the same repo structure with 42.X

Change-Id: I23de11d81020c8aae641dfc01c1cbddf769f5c75
2018-05-18 11:55:56 +01:00
Stanislav Makar
2e6a19a018 Add Ubuntu 18.04 support
Use squashfs for more recent Ubuntu releases

Change-Id: I80df28be6e2a5e03ae1450e84fc05715f21a7750
Closes-bug: 1766850
2018-05-18 14:47:20 +10:00
Matthew Thode
e1f01d513a
IPA requires iptables
iptables -L is used in log collection

Change-Id: Id7e69a9e9f87db6621b928a2104df4b94f10044e
2018-05-17 11:14:44 -05:00
Doug Szumski
c1b1534c87 Remove duplicate GRUB command line entry
Without this change DIB appends a second command line entry to the GRUB
config. This causes the original command line entry to be ignored
when Linux is booted.

The expected behaviour is that DIB appends to the existing entry as
it does for Ubuntu and SUSE.

Following discussion on the review, this also removes the distro specific
switch statement, as update-grub just calls grub-mkconfig, meaning that
there was nothing distro specific in the first place.

Change-Id: I2298675dda1f699c572b3423e7274bc8bd7c1c9d
Closes-Bug: #1771366
2018-05-16 09:25:59 +01:00
Zuul
a8df61edbb Merge "rpm-distro: set the contentdir yum var" 2018-05-14 12:36:11 +00:00
Zuul
d79a7aaf01 Merge "Add pip cache cleanup to pip-and-virtualenv" 2018-05-14 09:03:49 +00:00
Zuul
1412e957ba Merge "Fixes add-apt-keys in dpkg element" 2018-05-14 08:52:44 +00:00
Tristan Cacqueray
25964c5c5b rpm-distro: set the contentdir yum var
Since CentOS-7.5, a new yum variable is needed for SIG repositories.
This change replicates the %post task of the centos-release package
to setup the contentdir yum var. This should fix issues when
repository url uses $contentdir and yum fail with:
http://mirror.centos.org/%24contentdir ... [Errno 14] HTTP Error 404 - Not Found

For more details see:

https://lists.centos.org/pipermail/centos-devel/2018-March/016542.html

This change also drops support for fedora without dnf.

Change-Id: I1819a48b94670577b0c5e29b24cebfb20ea07d28
2018-05-14 05:38:37 +00:00
XiaojueGuan
93b42f0f01 Trivial: update url to new url
Change-Id: I8d26fd0598626c8666a1852724f0620af9dc028d
2018-05-13 23:06:00 +08:00
Oded Le'Sage
10b2a5a4ee Fixes add-apt-keys in dpkg element
This commit addresses the issue described in bug 1768354 when using the
apt-sources element and adding a key to a custom repo, subsequent deb
package installs fail due no update of the repo before package install

Change-Id: I968b3422fab2fb2305426d49215391d8ba7499df
Closes-Bug: 1768354
2018-05-09 12:06:20 -05:00
Michael Johnson
7e79a933db Add pip cache cleanup to pip-and-virtualenv
The pip-and-virtualenv element provides pip inside the created images.
When this pip is used inside the chroot of the image it, by default, creates
a cache directory with the http packages and the resulting wheels.
In the case of the octavia ubuntu-minimal image this cache is using ~50MB of
cache that is not needed after the image finished building.

This package creates a finalise.d task that removes the cache from the
default location.

Change-Id: I4715437b068d04993ef755bd1e27963db1d22417
2018-05-09 09:31:17 -07:00
Matthew Thode
f7818e4a57
Install sudo on Gentoo images by deault
Change-Id: I30640017e3c652a467be421169029045a8b675d2
Close-Bug: #1766484
2018-04-24 11:44:39 -05:00
Tristan Cacqueray
abd63b01aa pip-and-virtualenv: fix install-pip when centos-release-openstack is enabled
When RDO projects repository is installed, the python-setuptools
package is obsoleted by python2-setuptools, this makes the install-pip
script failed:

  Package python-setuptools-0.9.8-7.el7.noarch is obsoleted by
  python2-setuptools-22.0.5-1.el7.noarch which is already installed

Then the "rpm -ql python-setuptools | xargs rm -rf" exit 1.  Check if
we have a record of the updated package obsoleting then old one; if
so, use it.

Change-Id: I2b0051bd9e81908c187098a7b82e120b999b111d
2018-04-23 08:18:04 +10:00
Paul Belanger
99eb9a2d7d
Fix epel element for centos-minimal
We no longer install wget / yum-utils for centos-minimal, this fixes
that.

Change-Id: I8d89026bd48cf7398cc1cbe41e3b7f00f682dbb8
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2018-04-18 20:15:07 -04:00
Paul Belanger
12a760f7b7 Revert "debootstrap: Call update-initramfs explicitly"
This currently breaks glean on rackspace, revert until we can figure
out why that is.

This reverts commit 43bc352c59.

Change-Id: Iae88a3b0457bab0b8f0fd1febf58732ca95e5dc9
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2018-04-17 21:47:46 +00:00
Ian Wienand
b423292cd0 Remove installed packages before pip install
The release of pip10 has shown up a few issues here

Firstly, pip10 now refuses to overwrite distutils installed packages,
which includes "python-virtualenv" on centos.  History has shown us
that we want the packages installed and overwritten, to avoid the
packages coming back and messing things up.

Pre-install all the packages, then list the files in the packages with
"rpm" directly and remove them.  This way pip is happy to install.

We need to take better account of the package names for this; on
Fedora things have switch to "python2-virtualenv" instead of
"python-virtualenv" and we can't use an alias to list the package
contents.

This also highlighted that python2-pip is in EPEL for centos, so
enable that when we install it.  Make the epel element a no-op for non
centos/rhe distros.

There is a related change in recent fedora that python3 now installs
binaries into /usr/local/bin.  There are commented swizzles in here to
ensure we retain the status quo of "pip" and "virtualenv" both being
python2 based, with the python3 versions being called explicitly
"pip3" and "virtualenv3" respectively.

Change-Id: I2ffdd9f615ae6b00428c17249e4f216774991b99
2018-04-17 16:09:04 +10:00
Matthew Thode
2dd6dd357a
allow building non-gentoo images on gentoo hosts
Can't use a variable that's not set.

Change-Id: I4a7cf2ac47c2254da2fec778437f67c1fe3707f7
2018-04-12 13:24:59 -05:00
Zuul
0769bfd4aa Merge "Set the dhclient timeout to match DIB_DHCP_TIMEOUT" 2018-04-11 10:12:45 +00:00
Ian Wienand
f52b385818 Don't only install python3-virtualenv
We added this sed in I422490ebe9a9c655552685bc2ff342d288335a9c to
avoid installing python2 packages on python3-only systems and thus
dragging in all of python2.

We made a similar change to python-pip in
I7d8ba9300039cce90965410a4e16ca9e711904c3; however we realised that
the gate (and other consumers) were relying on this element having
installed the python2 & 3 packages for consistency -- otherwise jobs
would install the python-pip packages and overwrite the
pip-from-source and mess everything up.  We reverted that in
I419dbdf4682394db68974944af1e5c432f3e0565 and added some clearer notes
that this element brings in python2 & 3, and if you want something
that doesn't do that then this element isn't for you.

However, we never fixed up the virtualenv package install -- currently
our Xenial images have a global virtualenv installed from source, but
the python-virtualenv packages aren't installed.  Thus if a job does
"apt-get install python-virtualenv" it overwrites the from-source
virtualenv with older parts and again messes everything up.

Probably most jobs just call "virtualenv" and assume it is there;
however in bringing up some rspec test for puppet I have hit this
issue as some modules specify dependencies on the virtualenv packages.

Thus install the python-virtualenv AND python3-virtualenv packages in
this element.

Change-Id: Ia84c38dc3c40a6080e144b563e10abca7dac2881
2018-04-10 12:34:03 +10:00
Clark Boylan
f3d58d9042 Don't use -e to test for what might be broken symlink
The behavior of test -e and [[ -e against broken symlinks is to fail
even if the symlink exists. However we want to test if the link exists
or if there is a file in that location. Therefore switch from test -e to
test -L and test -f to check if the file or link exists regardless of
link target validity.

Change-Id: I84a9b6731eccf950707be50aef464a2de1e33e8e
2018-04-06 15:04:56 -07:00
Derek Higgins
97399e9bb1 Set the dhclient timeout to match DIB_DHCP_TIMEOUT
On initial boot when networking is brought up by cloud-init this
is the timeout that dhclient adheres to. Centos configures
"timeout 300" (for an EC2 bug) in their cloud image, which results
in a 5 minutes delay to boot in cases where no dhcp available (e.g. IPv6
SLAAC). To reduce this boot delay and to provide consistency with
places where we have set other dhcp timeouts set this to DIB_DHCP_TIMEOUT.

Change-Id: I119a002070501c3dfe7c6730b07ee25f422b85b0
Related-Bug: #1758324
2018-04-05 14:29:22 +01:00
Ian Wienand
e443700b5c Formalise saving of /etc/resolv.conf
systemd-resolved has a new behaviour in bionic, in that if there is no
/etc/resolv.conf file when it installs, it assumes it is a fresh
system and makes /etc/resolf.conf a symlink into its compatability
files.

dib ends up saving & restoring whatever /etc/resolv.conf we have after
the inital chroot creation, which may not be what we want -- in the
above case it restores the system-resolved symlink.  For
openstack-infra, we use unbound and want simply "127.0.0.1" in a
/etc/resolv.conf file [1].

Formalise the ability to save specific contents into the final image.
Add documentation, and a note in the code that it's an external
interface.

I would have preferred to namespace the .ORIG file with DIB_ or
similar, but this unofficial interface has already escaped into the
wild.  Leave it as is for simplicity.

[1] Note that systemd-resolved will obey /etc/resolv.conf as you would
expect, if file exists.

Change-Id: Ie0e97d8072e2b21a54b053fa6fb07b62960c686d
2018-04-04 15:17:45 +10:00
Ian Wienand
855ab0d850 Restore tracing on exit points of block_device_create_config_file
We exit in several places and don't restore tracing.  Previously in
nodepool we relied on the default fallback, which did restore the
tracing.  Since we now use the MBR config file, we take the different
exit path without it and the debugging output is incomplete.

Change-Id: I586fc95517926025705ce376ec5c4aaf4122773f
2018-04-03 14:33:09 +10:00
Zuul
09e5b2d357 Merge "debootstrap: Call update-initramfs explicitly" 2018-03-29 06:02:58 +00:00
Zuul
c07e7349d5 Merge "Change the GENTOO_PORTAGE_CLEANUP variable default" 2018-03-29 05:33:55 +00:00
Zuul
e8744cf32b Merge "zypper-minimal: Set default locale env to C.UTF-8" 2018-03-29 05:33:52 +00:00
Zuul
e444de83e6 Merge "Clean up dib-python symlink" 2018-03-29 05:33:51 +00:00
Andreas Florath
43bc352c59 debootstrap: Call update-initramfs explicitly
Many elements install additional distribution packages.
In addition the user can provide a set of packages to be installed
via the '-p' switch.
Some of them influence the boot process and therefore the initramfs
needs to be updated. Because the package manager during the image
creation process is configured not to run package scripts, this needs
to be done explicitly.

This issue was found during development and debugging of the
block-device LVM plugin: Even when the e.g. the lvm2 package
was installed in the image, it was missing in the initramfs
because of the missing update.

Change-Id: I7c92033b3ca80cdd23d081002059d83ca3f53bdb
Signed-off-by: Andreas Florath <andreas@florath.net>
2018-03-29 04:14:52 +00:00
Matthew Thode
6a6d78e63c
Change the GENTOO_PORTAGE_CLEANUP variable default
Default the GENTOO_PORTAGE_CLEANUP to True.  By default we should not
ship package info, this bloats the image and is usually outdated by the
time it'd be consumed.

Change-Id: I14c2530d91807cbc6a3806e01c7e4f6f472b190d
2018-03-26 23:26:49 -05:00
Clark Boylan
301eac8e8b Fix element-provides in debian element
The debian element depends on debian-minimal now which provides
operating-system. This means that the debian element can no longer
provide operating-system and doing so results in an error when using the
debian element.

The fix is simple just rely on the fact that debian-minimal provides
operating-system and remove this element-provides from debian.

Fixes-Bug: 1758000
Change-Id: I524feeb82c19046ec987eb1186c7f4568309e559
2018-03-26 10:58:04 -07:00
Zuul
ffc06874ef Merge "install sudo in the devuser element" 2018-03-26 00:58:57 +00:00
Zuul
8ab38dc5cf Merge "Update Fedora defaults to 27" 2018-03-25 23:41:45 +00:00
Zuul
9d751463f1 Merge "enable systemd profile for Gentoo" 2018-03-23 19:01:35 +00:00