Commit Graph

467 Commits

Author SHA1 Message Date
Zuul
b79952af2a Merge "Add new modprobe element" 2018-07-11 07:21:40 +00:00
Zuul
bbf69a90f3 Merge "Fix /etc/network/interfaces file contents" 2018-07-10 07:46:56 +00:00
Zuul
2343d4b577 Merge "Add keyring if supplied" 2018-07-10 07:46:55 +00:00
Hironori Shiina
7e4e6cfff4 Add expected semicolons for dhclient.conf
This patch adds an expected semicolon to an end of statement in
dhclient.conf for dhcp-all-interfaces element. Without this fix, an
error occurs when an image is booted with a message,
'semicolon expected.'.

Change-Id: I8311dbc67cc2815223111da01e7a7517c7d6f059
2018-07-06 13:42:25 +09:00
Sam Yaple
c144246cc9
Add keyring if supplied
When building with debootstrap, debootstrap will use the key to check
that everything is properly signed. It will not `apt-key add` the key
into the final environment, however.

Early adding the key after debootstrap before we need to read from the
private repo again prevents unsigned issues. This also maintains the
integrity of the packages in the environment throughout the build.

Change-Id: I5ca75ae4620c9fb26b512cb30f8cd79fa7a0373a
2018-07-02 14:33:35 -04:00
Zuul
927e8115f6 Merge "Fix bootloader for efi on rhel systems" 2018-06-28 15:02:30 +00:00
Yolanda Robla
31383970c7 Add new modprobe element
This element will replace modprobe-blacklist element. It wil
still have the blacklist functionality, but it also adds
the feature of passing a complete file with settings to the
modprobe.d directory. Adding this functionality, that will
allow elements that depends on this module, to just copy the
specified files to the final directory.

Change-Id: I9a44f7d11520b8b1e604956d3c1db2fc7e2bf457
2018-06-28 13:55:53 +02:00
Zuul
e796b3bc18 Merge "Add iscsi-boot element for CentOS images" 2018-06-27 09:54:10 +00:00
Hoang Trung Hieu
a4648872ba Add iscsi-boot element for CentOS images
This patch adds an element that handles the configuration for
creating a disk capable of being a remote root filesystem through
iSCSI on CentOS images.

Tested on Fujitsu Server and boot with BIOS and UEFI mode successfully.
- Tested Boot-From-Volume + EFI for centos7 with following elements:
  "centos7 vm devuser cloud-init-datasources dhcp-all-interfaces
   iscsi-boot dracut-regenerate block-device-efi"

Co-authored-By: Nguyen Van Trung <trungnv@vn.fujitsu.com>

Change-Id: Ia1f23d722dced6f254fd7aee86abe8066a72fa42
2018-06-25 12:01:52 +00:00
Zuul
e39adcd65f Merge "Remove redundant word" 2018-06-22 14:23:47 +00:00
chenxiangui
c809160906 Remove redundant word
Remove the redundant word 'the' in config.py

Change-Id: I3e9cb6390ce196f0a9022aef10f6c7b1ace36c48
2018-06-19 18:00:11 +08:00
Olivier Bourdon
11d91501d0 Fix /etc/network/interfaces file contents
According to http://bit.ly/2HA4oDO and
the official Ubuntu manual
http://manpages.ubuntu.com/manpages/xenial/man5/interfaces.5.html
source-dir support has been removed from Ubuntu >= 16.04/Xenial

Once an image is generated and booted, moving the dhcp interface(s)
declaration(s) from /etc/network/interfaces into specific subentries
of /etc/network/interfaces.d and calling 'service networking restart'
just make your instance unreachable and all interfaces are left
unconfigured.

This patchset fixes this issue

Change-Id: I6b6b99c81490c874c5db5405c2fbf3c180c87464
2018-06-19 11:26:21 +02:00
Yolanda Robla
9687a1efe1 Convert labels to upper case
When booting on UEFI, there was an issue mounting the vfat
filesystem. It was caused because the mount was defined in
/etc/fstab in lowercase, but the disk had it labeled in upper
case, and system could not find it. Conver the label to upper
case in case of fat/vfat.

Change-Id: Id3dee735e6f8fb221d199c4aba648f3e9a6e4206
2018-06-19 11:12:54 +02:00
Yolanda Robla
61e6566c48 Fix bootloader for efi on rhel systems
When building the image on a non-efi environment, it generates
linux16/initrd16 entries. But to boot from UEFI they need to have
linuxefi/initrdefi entries.
Use sed to replace those entries, in case we have an EFI image.

Change-Id: I47c96450e10f34b91bcc32888532bd7ab87cf316
2018-06-19 08:37:30 +02:00
Zuul
e210f79500 Merge "Don't run setfiles on /boot/efi" 2018-06-15 08:42:13 +00:00
Nguyen Van Trung
8d86ff1bec Don't run setfiles on /boot/efi
setfiles isn't supported on the vfat /boot/efi partition.  Add it to
the skip list.

Tested on Fujitsu Server successfully.

Change-Id: Iab262c4bdb0ecc25ca6b77ee4aff1ce442c0c578
2018-06-15 14:53:38 +10:00
Michael Turek
91e3b72a23 Add iscsi-boot element
This patch adds an element that handles the configuration for
creating a disk capable of being a remote root filesystem through
iSCSI on Ubuntu and Debian images.

Change-Id: Ibf9e39d2bdab530106015f156d23d28029d12b0d
Closes-bug: #1716794
2018-06-14 08:56:03 +07:00
Yolanda Robla
bfc60958bf Fix bootloader packages for rhel
When using uefi in rhel, the package mapping is incorrect.
We need to add specific grub-efi* mappings to use grub2-efi

Change-Id: I2db96ae85fd5e4638c794015b2f8164c018420e3
2018-06-08 17:14:19 +02:00
Zuul
fc3748f49b Merge "Fix encoding issue during processing output" 2018-06-06 07:58:14 +00:00
Zuul
53c2e35f1b Merge "Save and close stdout on exit" 2018-06-06 06:30:05 +00:00
Zuul
8bdba50459 Merge "Reduce path length in PS4 for debug" 2018-06-06 06:20:28 +00:00
Zuul
29230d74c5 Merge "Use surrogateescape with outfilter.py" 2018-06-06 06:20:27 +00:00
Zuul
225c09b245 Merge "elements: pip-and-virtualenv: Handle openSUSE Leap 15" 2018-05-31 21:54:59 +00:00
Zuul
c4c2fb746c Merge "Remove duplicate GRUB command line entry" 2018-05-31 16:00:29 +00:00
Markos Chandras
f37e85d547 elements: pip-and-virtualenv: Handle openSUSE Leap 15
We need to handle openSUSE Leap 15 when installing pip and virtualenv
packages. This fixes the following problem when the pip-and-virtualenv
elements is used:

2018-05-31 09:42:12.014 | + [[ opensuse = opensuse ]]
2018-05-31 09:42:12.014 | /tmp/in_target.d/install.d/04-install-pip: line 57: packages: unbound variable

Change-Id: Id7911b0a0836fa8dcc003e23fa515b78fba67126
2018-05-31 10:55:28 +01:00
Zuul
4c04b46db6 Merge "Allow to rebuild arbitrary images" 2018-05-29 17:06:48 +00:00
Zuul
df9402c81c Merge "elements: zypper-minimal: Add support for openSUSE Leap 15.X" 2018-05-25 08:55:53 +00:00
Ian Wienand
8fb2f5cb55 Save and close stdout on exit
Redirecting our output through outfilter.py is inherently a bit racy,
since the disk-image-create process will exit, and then you might get
outfilter.py flushing any remaining output as it closes.

On an interactive prompt this might lead to final output overwriting
the prompt, etc.  This can be a bit confusing when you start running
things in a loop.

If we save the original fd, then on the exit path close the redirected
fd's and wait a little bit for final output (as a result of the
close), we get a more consistent output.

Change-Id: I8efe57ab421c1941e99bdecab62c6e21a87e4584
2018-05-25 11:48:20 +10:00
Ian Wienand
7b98433c46 Reduce path length in PS4 for debug
Strip everything before "site-packages" in the output filename for the
PS4 prompt.  This makes the line in debug logs significantly shorter
as we don't have the full virtualenv path every single time.  The
important thing -- the file being called in the lib/ dir, is retained.

Change-Id: I00706b6f6c0425c7795f997c08ceda3374dc84b5
2018-05-25 11:48:20 +10:00
Ian Wienand
596062b0f9 Use surrogateescape with outfilter.py
When switching to using log-file capture, we're getting

 [gentoo/build-succeeds] outfile.write(ts_line.encode('utf-8'))
 [gentoo/build-succeeds] UnicodeEncodeError: 'utf-8' codec can't
   encode character 'udcc5' in position 59: surrogates not allowed

Use surrogateescape [1] on the output to avoid this

[1] https://www.python.org/dev/peps/pep-0383/

Change-Id: I2c2c537296edfa5a8fe661a41bd5bfb3bfcf57e3
2018-05-25 11:48:14 +10:00
Roman Gorshunov
84eea81efe Allow to rebuild arbitrary images
Patch allows to rebuild arbitrary images, which location, filename and
sha256sum are specified in variables, not only hardcoded $DIB_RELEASE/current.

Change-Id: I05418932a0c40d885fe00a49f1f49d7e86c67518
2018-05-24 10:19:59 +00:00
Zuul
668c93b118 Merge "Replace the ubuntu-minimal trusty test with a bionic one" 2018-05-21 06:01:53 +00:00
Ian Wienand
82eb1ca837 Replace the ubuntu-minimal trusty test with a bionic one
Add a bionic test in replacement of trusty.  We are already building
bionic images in the gate, so this seems like a good time to switch.

Change-Id: I20d4c25e9b79e7326c86767c36be8615ba0888a3
2018-05-21 12:51:31 +10:00
Roman Gorshunov
09af52a08c Remove non-maintained ubuntu-core element
Removing no longer working and no longer maintained ubuntu-core element, which
intent is unclear, and not documented.

Change-Id: Id847591d04fd7cd32c8903967da01ee0d303b267
Closes-Bug: 1771614
2018-05-18 09:28:02 -07:00
Markos Chandras
bb9ad09010 elements: zypper-minimal: Add support for openSUSE Leap 15.X
openSUSE Leap shares the same repo structure with 42.X

Change-Id: I23de11d81020c8aae641dfc01c1cbddf769f5c75
2018-05-18 11:55:56 +01:00
Stanislav Makar
2e6a19a018 Add Ubuntu 18.04 support
Use squashfs for more recent Ubuntu releases

Change-Id: I80df28be6e2a5e03ae1450e84fc05715f21a7750
Closes-bug: 1766850
2018-05-18 14:47:20 +10:00
Doug Szumski
c1b1534c87 Remove duplicate GRUB command line entry
Without this change DIB appends a second command line entry to the GRUB
config. This causes the original command line entry to be ignored
when Linux is booted.

The expected behaviour is that DIB appends to the existing entry as
it does for Ubuntu and SUSE.

Following discussion on the review, this also removes the distro specific
switch statement, as update-grub just calls grub-mkconfig, meaning that
there was nothing distro specific in the first place.

Change-Id: I2298675dda1f699c572b3423e7274bc8bd7c1c9d
Closes-Bug: #1771366
2018-05-16 09:25:59 +01:00
Zuul
a8df61edbb Merge "rpm-distro: set the contentdir yum var" 2018-05-14 12:36:11 +00:00
Zuul
d79a7aaf01 Merge "Add pip cache cleanup to pip-and-virtualenv" 2018-05-14 09:03:49 +00:00
Zuul
1412e957ba Merge "Fixes add-apt-keys in dpkg element" 2018-05-14 08:52:44 +00:00
Tristan Cacqueray
25964c5c5b rpm-distro: set the contentdir yum var
Since CentOS-7.5, a new yum variable is needed for SIG repositories.
This change replicates the %post task of the centos-release package
to setup the contentdir yum var. This should fix issues when
repository url uses $contentdir and yum fail with:
http://mirror.centos.org/%24contentdir ... [Errno 14] HTTP Error 404 - Not Found

For more details see:

https://lists.centos.org/pipermail/centos-devel/2018-March/016542.html

This change also drops support for fedora without dnf.

Change-Id: I1819a48b94670577b0c5e29b24cebfb20ea07d28
2018-05-14 05:38:37 +00:00
XiaojueGuan
93b42f0f01 Trivial: update url to new url
Change-Id: I8d26fd0598626c8666a1852724f0620af9dc028d
2018-05-13 23:06:00 +08:00
Oded Le'Sage
10b2a5a4ee Fixes add-apt-keys in dpkg element
This commit addresses the issue described in bug 1768354 when using the
apt-sources element and adding a key to a custom repo, subsequent deb
package installs fail due no update of the repo before package install

Change-Id: I968b3422fab2fb2305426d49215391d8ba7499df
Closes-Bug: 1768354
2018-05-09 12:06:20 -05:00
Michael Johnson
7e79a933db Add pip cache cleanup to pip-and-virtualenv
The pip-and-virtualenv element provides pip inside the created images.
When this pip is used inside the chroot of the image it, by default, creates
a cache directory with the http packages and the resulting wheels.
In the case of the octavia ubuntu-minimal image this cache is using ~50MB of
cache that is not needed after the image finished building.

This package creates a finalise.d task that removes the cache from the
default location.

Change-Id: I4715437b068d04993ef755bd1e27963db1d22417
2018-05-09 09:31:17 -07:00
Tristan Cacqueray
abd63b01aa pip-and-virtualenv: fix install-pip when centos-release-openstack is enabled
When RDO projects repository is installed, the python-setuptools
package is obsoleted by python2-setuptools, this makes the install-pip
script failed:

  Package python-setuptools-0.9.8-7.el7.noarch is obsoleted by
  python2-setuptools-22.0.5-1.el7.noarch which is already installed

Then the "rpm -ql python-setuptools | xargs rm -rf" exit 1.  Check if
we have a record of the updated package obsoleting then old one; if
so, use it.

Change-Id: I2b0051bd9e81908c187098a7b82e120b999b111d
2018-04-23 08:18:04 +10:00
Paul Belanger
99eb9a2d7d
Fix epel element for centos-minimal
We no longer install wget / yum-utils for centos-minimal, this fixes
that.

Change-Id: I8d89026bd48cf7398cc1cbe41e3b7f00f682dbb8
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2018-04-18 20:15:07 -04:00
Paul Belanger
12a760f7b7 Revert "debootstrap: Call update-initramfs explicitly"
This currently breaks glean on rackspace, revert until we can figure
out why that is.

This reverts commit 43bc352c59.

Change-Id: Iae88a3b0457bab0b8f0fd1febf58732ca95e5dc9
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2018-04-17 21:47:46 +00:00
Ian Wienand
b423292cd0 Remove installed packages before pip install
The release of pip10 has shown up a few issues here

Firstly, pip10 now refuses to overwrite distutils installed packages,
which includes "python-virtualenv" on centos.  History has shown us
that we want the packages installed and overwritten, to avoid the
packages coming back and messing things up.

Pre-install all the packages, then list the files in the packages with
"rpm" directly and remove them.  This way pip is happy to install.

We need to take better account of the package names for this; on
Fedora things have switch to "python2-virtualenv" instead of
"python-virtualenv" and we can't use an alias to list the package
contents.

This also highlighted that python2-pip is in EPEL for centos, so
enable that when we install it.  Make the epel element a no-op for non
centos/rhe distros.

There is a related change in recent fedora that python3 now installs
binaries into /usr/local/bin.  There are commented swizzles in here to
ensure we retain the status quo of "pip" and "virtualenv" both being
python2 based, with the python3 versions being called explicitly
"pip3" and "virtualenv3" respectively.

Change-Id: I2ffdd9f615ae6b00428c17249e4f216774991b99
2018-04-17 16:09:04 +10:00
Zuul
0769bfd4aa Merge "Set the dhclient timeout to match DIB_DHCP_TIMEOUT" 2018-04-11 10:12:45 +00:00
Ian Wienand
f52b385818 Don't only install python3-virtualenv
We added this sed in I422490ebe9a9c655552685bc2ff342d288335a9c to
avoid installing python2 packages on python3-only systems and thus
dragging in all of python2.

We made a similar change to python-pip in
I7d8ba9300039cce90965410a4e16ca9e711904c3; however we realised that
the gate (and other consumers) were relying on this element having
installed the python2 & 3 packages for consistency -- otherwise jobs
would install the python-pip packages and overwrite the
pip-from-source and mess everything up.  We reverted that in
I419dbdf4682394db68974944af1e5c432f3e0565 and added some clearer notes
that this element brings in python2 & 3, and if you want something
that doesn't do that then this element isn't for you.

However, we never fixed up the virtualenv package install -- currently
our Xenial images have a global virtualenv installed from source, but
the python-virtualenv packages aren't installed.  Thus if a job does
"apt-get install python-virtualenv" it overwrites the from-source
virtualenv with older parts and again messes everything up.

Probably most jobs just call "virtualenv" and assume it is there;
however in bringing up some rspec test for puppet I have hit this
issue as some modules specify dependencies on the virtualenv packages.

Thus install the python-virtualenv AND python3-virtualenv packages in
this element.

Change-Id: Ia84c38dc3c40a6080e144b563e10abca7dac2881
2018-04-10 12:34:03 +10:00