Commit graph

1871 commits

Author SHA1 Message Date
Paul Belanger
18a664dd32 Don't use ssh-keygen -A for init scripts
We are running into race conditions with glean, which ssh-keygen -A is
not handling properly.  So, create a new script to first check if the
file exists, then use 'yes' to disable overwriting of existing files.

Change-Id: Ie82e1e3f832fcc8f32c7e1335c5f0ee16d36f9a8
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2016-10-18 14:25:28 -04:00
Markos Chandras
89c708ce61 elements: simple-init: Remove SUSE interfaces
Make sure SUSE interfaces are removed as well.

Change-Id: If993dc606217f2ec243392ac2fa588ebae1cce86
2016-10-18 19:02:41 +01:00
Jenkins
85354e5cd8 Merge "Add opensuse-minimal element" 2016-10-18 16:27:03 +00:00
Jenkins
e0f7b6c6d6 Merge "Add zypper-minimal element" 2016-10-18 16:24:07 +00:00
Jenkins
96ce53fa16 Merge "Fix grub installation for RHEL" 2016-10-12 14:22:49 +00:00
Jenkins
066b448d7c Merge "start cloud-init-local in the boot runlevel" 2016-10-12 02:37:04 +00:00
Gregory Haynes
fdffa56ff2 Add element for setting sysctl values
Theres a pretty standard workflow for setting a sysctl value which will
be applied on image boot which was written by tripleo. Lets move this in
tree as other folks (like Octavia) would like to depend on it.

Change-Id: I3c266870d417cdba3196f5fa65c4cd634ab13173
2016-10-10 15:33:48 +00:00
Jenkins
22bdaad684 Merge "Move the opensuse mkinitrd script to the zypper element" 2016-10-10 14:36:12 +00:00
Jenkins
a56daaeaf2 Merge "Add pkg-map for gentoo to runtime-ssh-host-keys" 2016-10-07 17:28:35 +00:00
Matthew Thode
82b299bbdf
start cloud-init-local in the boot runlevel
cloud-init-local needs to be run in the boot runlevel because it
modifies services in the default runlevel. When a runlevel is started
it is cached, so modifications that happen to the current runlevel while
you are in it are not acted upon.

Change-Id: Ifeae0071fc9e738ec223ec0df271559ad6e0196b
2016-10-07 11:16:28 -05:00
Jenkins
5897e4975d Merge "Disabling all previous repos registered in the system" 2016-10-07 11:05:46 +00:00
Gregory Haynes
c67e7ed15e Generate ssh-hostkeys on boot for ironic agent
Ironic agent images should have ssh hostkeys in image on boot.

Change-Id: If8d42bb8b8bff417d5b1d4d8bc371425697edde5
Closes-Bug: #1556145
2016-10-06 14:29:23 -07:00
Matthew Thode
3571647692
Add pkg-map for gentoo to runtime-ssh-host-keys
Openssh is provided by default so it is not needed to be installed here.

Change-Id: Id86f9a1d214c775570f0c9e2df4ea81367bb5b7e
2016-10-06 11:57:34 -05:00
Jenkins
12a73cf204 Merge "Default to http://ftp.us.debian.org/debian for debian-minimal" 2016-10-05 12:56:07 +00:00
Markos Chandras
90536dbab3 Add opensuse-minimal element
Add a new opensuse-minimal element to build small and highly
configurable openSUSE based images using the zypper-minimal element
as the main building mechanism

Change-Id: Iebfc4ad4aff763e511b093f1607b55851ccbddcb
2016-10-05 09:39:29 +01:00
Markos Chandras
c69c6c5a84 Add zypper-minimal element
Add a new zypper-minimal element to bootstrap SUSE-based distributions

Change-Id: Id63397e412a929d61247cfd3d9f8d4e758c1248a
2016-10-05 09:27:17 +01:00
Markos Chandras
8dde310cf3 Move the opensuse mkinitrd script to the zypper element
All SUSE-based elements can benefit from the mkinitrd phase to move it
to a more generic location.

Change-Id: Ife171d462a393b6ac0bf2c5eaa48ea25eaf4d1cc
2016-10-05 09:11:00 +01:00
Jenkins
80153520ac Merge "Move opensuse utils to zypper so they can be shared by SUSE-based distros" 2016-10-05 05:10:15 +00:00
Jenkins
04b05f261e Merge "Fix typo in extracting root partition" 2016-10-05 04:00:39 +00:00
Paul Belanger
7dc9465ed1
Default to http://ftp.us.debian.org/debian for debian-minimal
Since http://httpredir.debian.org is unreliable is selecting a mirror
to use, we'll now default to http://ftp.us.debian.org/debian.  In
fact, in openstack-infra we have been overriding httpredir.debian.org
for a while, now make this default in diskimage-builder.

Change-Id: I48658bc076e13a0913821197e4120c73618fef8f
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2016-10-01 21:10:08 -04:00
Markos Chandras
b9e9e2091e Move opensuse utils to zypper so they can be shared by SUSE-based distros
Move the opensuse utilities to the zypper element so they can be used by
SUSE or zypper based elements. This brings the zypper element somewhat
in line with the rest of the package manager elements.

Change-Id: I8aa2849231454216cdd47629a5e2d6e45769dbbe
2016-09-29 22:56:17 +01:00
Jenkins
b0d72a3161 Merge "Create runtime-ssh-host-keys element" 2016-09-27 23:10:05 +00:00
Yolanda Robla Mota
c6b4e639b0 Disabling all previous repos registered in the system
Depending on the pool id used, so many repos are brought,
including not valid ones that cause image to crash, or repos
that include conflicting packages.
Before enabling repos, disable all previous ones, so we
can be sure that we only bring the repos specified in the
parameters.

Change-Id: Ifd4d8d1d4fa954cd2593669e516e3201f2d6f6c1
2016-09-27 11:47:05 +02:00
Yolanda Robla Mota
f6e11c91c3 Fix typo in extracting root partition
Change-Id: Ie8dfd958d57ef92988647166f2031adb8406b0d7
2016-09-27 09:41:32 +02:00
Paul Belanger
45467e4229
Create runtime-ssh-host-keys element
Move managing of SSH host keys into a dedicated element.

Because glean doesn't generate SSH host keys anymore, we need to do it
with a systemd script. This is already handled by CentOS / Fedora so
we don't want to add it there.

This was done to address the upstream bug in debian:

  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=500192

Change-Id: I31ad667672e08350872db21a83445fe0aa7a4a39
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2016-09-26 12:32:38 -04:00
Jenkins
d6794ba673 Merge "Shorten DHCP timeout in dhcp-all-interfaces" 2016-09-23 14:20:09 +00:00
Jiri Stransky
93425d14ce Fix grub installation for RHEL
Grub is first removed and then installed during RHEL image building. The
grub2 package typically requires the same version of grub2-tools, so if
we just remove and install the grub2 package, the installation can
potentially fail on being out of sync with grub2-tools version. Removing
and reinstalling both packages fixes this issue. Those packages are
already in package map for RHEL as "grub-pc", so we can use this alias.

Change-Id: Iefd9c17fffd43de3fea260510ad218b1322eecb3
Closes-Bug: #1627000
2016-09-23 14:14:49 +02:00
Jenkins
3a0f46282f Merge "elements: opensuse: Add support for openSUSE Leap" 2016-09-23 08:48:46 +00:00
Jenkins
e84488cad1 Merge "Allow ramdisk-create to run without $USER set" 2016-09-23 06:46:31 +00:00
Ben Nemec
2747613ca2 Shorten DHCP timeout in dhcp-all-interfaces
We are currently wasting about 10 minutes per deploy waiting for
DHCP on interfaces that will never get it.  By default, the timeout
seems to be 5 minutes (the 10 minutes is because we boot both the
IPA ramdisk and the deployed image, and each waits for 5 minutes),
which is excessively long to get a DHCP response.  This change
shortens the time to 30 seconds.  If an interface hasn't gotten a
response in 30 seconds, chances are it's not going to.  A 30
second wait should reduce our wasted time to 1 minute, which is
more reasonable.

This is being done in the systemd unit file because the -timeout
option to dhclient doesn't seem to override what is configured in
dhclient.conf, and doing it in the systemd file means that this
change will be limited to only the interfaces configured by
dhcp-all-interfaces.

Change-Id: Ia8610e3def39c937eb0c861fdc9bc571ec39f9f4
Closes-Bug: 1626673
2016-09-22 17:01:06 -05:00
Jenkins
0559818e04 Merge "don't configure 'lo' for dhcp" 2016-09-20 23:37:42 +00:00
Jenkins
4d79e5f519 Merge "Document source glean installs in simple-init" 2016-09-20 23:05:53 +00:00
Jenkins
d8a800c638 Merge "Add libselinux-python to yum-minimal" 2016-09-20 06:23:34 +00:00
Ian Wienand
ce410de834 yum-minimal: Disable excludes when installing pkg manager
Because we are using the building platform's "yum" to do the initial
install into the chroot, it is affected by the base-system's
/etc/yum.conf.

pip-and-virtaulenv in I82acb865378a0fa5903a6267bfcee0e2962eced0 added
"exclude=python-pip..." in /etc/yum.conf to stop the package manager
overwriting the installed pip.  Now our CI images have built with
this, we are now picking up this exclude on centos.  Since on F24
dnf->python->python-pip we end up failing to build the the chroot
because python-pip can not be satisifed.  In a general sense, however,
this could be caused by any configuration put into /etc/yum.conf that
is incompatible with installing into the chroot.

yum has the option to disable all excludes which is used here.  This
seems to be the best way to isolate the chroot install from any
excludes that may have been done on the base system for various
reasons.  I did consider using a completely separate yum.conf we ship
with dib ... but let's start simple.

This should fix the current gate failures on centos

Change-Id: I4e4cc8ed09a29c4057ade34ea93025139e191bf5
2016-09-20 09:31:44 +10:00
Monty Taylor
fd18cb74b2
Add libselinux-python to yum-minimal
yum-minimal installs selinux but not libselinux-python, which makes
interacting with the node from ansible hard fail. Add it.

Change-Id: I403e7806ae10d5dd96d0727832f4da20e34b94c7
2016-09-17 01:25:31 +02:00
Markos Chandras
0ddea940ad elements: opensuse: Add support for openSUSE Leap
Add support for new openSUSE Leap releases. Moreover, document
common environment variables and remove old note.

Change-Id: I8cf0b215cb4d9231e5658d49e3fd598dfbb5fd37
2016-09-16 16:13:19 +01:00
Jenkins
63a27bd0c4 Merge "Fix mellanox element required kernel modules and user space packages" 2016-09-13 06:07:27 +00:00
John Trowbridge
49baaa4114 Remove EPEL as hardcoded dependency of centos elements
The previous commit removes dkms from the base element, which
means the centos elements should no longer have a dependency on
EPEL.  Therefore, we should not hardcode the epel dependency.  It
can still be included in image builds as desired by using the epel
element explicitly.

Co-Authored-By: Ben Nemec <bnemec@redhat.com>
Change-Id: Iceff0d5bedd9816adfd2990970e7c216b67b6bd0
2016-09-12 11:42:55 -05:00
Ben Nemec
01a0dbf7c0 Remove unnecessary dkms install from base
The use of dkms in base was actually removed long ago in
Ic2c345bf9f0738dadae611194e263d3a5d424a3e and it is creating an
unnecessary dependency on EPEL for the centos elements.

Change-Id: Iae3100471e50a9c39f40b450f087192918ae54b3
2016-09-12 11:42:51 -05:00
Noam Angel
07e34f90e7 Fix mellanox element required kernel modules and user space packages
This fix add need kernel module for Infiniband and ConnectX-4+ network
cards.
Also install by default required user space packages.
Change-Id: Ia2e7b1820f197778138a23fafaccb5a4fb44369a
2016-09-11 11:40:20 +00:00
Waldemar Znoinski
bc80572061 don't configure 'lo' for dhcp
On systemd-based operating systems that don't
use /etc/sysconfig/network-scripts
dhcp-all-interfaces configures 'lo' for dhcp.
This causes errors and fails networking.target
causing system-wide issues. This change excludes
'lo' at dhcp-all-interfaces udev rules level.

Closes-bug: #1621501
Change-Id: I7563b766827bedbea7ae1de35e5bdfcbf1fc0d1e
Co-Authored-By: Jeremy Stanley <fungi@yuggoth.org>
2016-09-09 09:01:59 +00:00
Jenkins
ca53af1184 Merge "fix systemd resource deadlock" 2016-09-06 23:33:03 +00:00
Jenkins
1041b5c9d5 Merge "Revert "Revert "Pre-install pip/virtualenv packages""" 2016-09-06 21:21:11 +00:00
Waldemar Znoinski
4b222b8263 fix systemd resource deadlock
Dependency to start network-pre (which
depends on network.target) before
dhcp-interface@.service collides with
Ubuntu's own network.target that suupose
to start after network-pre.

Change-Id: I9e59c970bfb1ebdaa15b4ec6b545761ede3ca056
Closes-bug: #1619816
2016-09-06 04:47:29 +00:00
Clark Boylan
07f22a3388 Document source glean installs in simple-init
It is possible and often desired to install glean from a source
repository when using the simple-init element. Document the process for
doing this.

Change-Id: Ie7c690406b14aae07d73261879b7ce8a2ed9dd8d
2016-09-01 15:03:38 -07:00
Jenkins
a2d3d340d3 Merge "Update portage only if needed" 2016-09-01 06:29:16 +00:00
Jenkins
c526220c6f Merge "Generate and use upper-constraints for ironic-agent" 2016-09-01 06:29:09 +00:00
Jenkins
02e3b4d5c0 Merge "Disabled IPv6 privacy extensions" 2016-09-01 06:11:08 +00:00
Jenkins
41c7dd2e56 Merge "Add tests for building *-minimal images" 2016-09-01 05:44:03 +00:00
Jenkins
b10752ca71 Merge "Update GRUB_MKCONFIG for detecting what's installed" 2016-09-01 05:14:53 +00:00
Jenkins
36c3d1ff50 Merge "add no_proxy when debootstrap trying to use proxy" 2016-09-01 05:14:47 +00:00
Kevin Carter
4b1a875e56 Disabled IPv6 privacy extensions
IPv6 privacy extensions can cause issues by preferring a temporary
network over a public one. This preference may limit connectivity
in certain situations. An example of a connectivity issue can be
seen where the command ``traceroute6`` fails or misses all hops
while other traffic to a given domain with a "AAAA" record may
succeed. To resolve this issue the IPv6 privacy extensions have
been disabled.

Change-Id: I62b9d6301b9e8b8e93b49cecbc96334ceea92fa5
Related-Bug: #1068756
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2016-09-01 14:41:26 +10:00
Jenkins
7e0e13106d Merge "Explain difference between two envvars" 2016-09-01 04:38:38 +00:00
Mathieu Mitchell
796dcd043b Generate and use upper-constraints for ironic-agent
Currently, ironic-python-agent is installed without using an
upper-constraints.txt file.

This commits ensures ironic-python-agent is installed using
upper-constraints.

Change-Id: I6be6cfc012941e2cc9996717cba39b5415b85e14
Closes-Bug: #1616554
2016-08-29 21:05:04 +00:00
Xiang Chen
ff234b65ea Explain difference between two envvars
Explain difference between 'DIB_OFFLINE' and
 'DIB_DEBIAN_USE_DEBOOTSTRAP_CACHE'

Those variables are not redundant,they hava different effective ranges.
However,some people may be confused about this and reported a bug.
So,this difference should be writen in the README file.

Closes-Bug: #1506275

Change-Id: Ie5316de41d129bf98781708954f09ef0b2592b53
2016-08-25 10:07:29 +08:00
Jenkins
0ca20753ce Merge "Add "audit"package to yum-minimal" 2016-08-22 17:00:27 +00:00
Gregory Haynes
a1bfca6022 Add tests for building *-minimal images
Lets make sure these images can be built.

Change-Id: Idbd07b98c0181738d002a53373425e056390beea
2016-08-22 16:53:32 +00:00
Matthew Thode
c3a036e54f
Update portage only if needed
Currently we update portage whenever we could need it.  Instead we
should update portage only if we actually need to.  This update adds a
check to do so.

Change-Id: Ifdb27fd844b0b3a169ced945ac7ee0ddc235e9ec
2016-08-22 07:14:25 -05:00
Matthew Thode
c1c2f7147b
Update GRUB_MKCONFIG for detecting what's installed
Gentoo has updated it's grub ebuild to default to the upstream
recommended installation parameter of grub-mkconfig instead of our
default multislot installation of grub2-mkconfig.  Update the command
line parameter so that it works with both.

Change-Id: I359b44338a4f76af7c026f5cad212e6dc3dbf2b3
2016-08-22 07:12:15 -05:00
Gregory Haynes
6180d82f14 Allow ramdisk-create to run without $USER set
It's possible this is run form an environment where $USER isn't set,
properly fallback to whoami in this case.

Change-Id: I1181f714c3c456ee264b34d282bac5c0adb67a0e
2016-08-19 16:04:32 +00:00
Jenkins
c20a42051a Merge "Change DIB_IPA_CERT resulting file name" 2016-08-18 13:35:12 +00:00
Juan Antonio Osorio Robles
1787754d57 Change DIB_IPA_CERT resulting file name
Even though this file ends up in the /tmp directory, for readability
it's good to point out that server.pem is not necessarily the
certificate for a server, but can be a CA certificate which is
trusted if this option is used.

Change-Id: Iea27a702a844456e4472957438f75ed3819d62ca
2016-08-15 15:19:04 +03:00
Yolanda Robla Mota
c1bac651cb Allow to skip kernel cleanup
For some use cases, it can be useful to keep all the kernels
and not just keep the latest one. Add a parameter that allows
it, and continue cleaning up kernels by default.

Change-Id: Ia6e6c1fa18e3724c1eb89226151d81e9e748b793
2016-08-12 12:49:19 +02:00
Jenkins
cf2030c2f6 Merge "secure_path in sudoers: deal with possible quotes" 2016-08-11 00:21:14 +00:00
Ian Wienand
944b4fea0f Add "audit"package to yum-minimal
Kernels are built with auditing support, and without the audit deamon
logs bubble up to spam the console and /var/log/messages.  This
package contains the audit daemon that catches these messages.

Change-Id: Ie3e216bab33b27f2d67a9379ddc3e89d66449251
2016-08-08 17:54:20 +10:00
Jenkins
46ce2ac527 Merge "Optionally remove portage files" 2016-08-04 05:20:54 +00:00
Gabriele Cerami
5e957f6350 secure_path in sudoers: deal with possible quotes
Sometimes the secure_path option value in /etc/sudoers is surrounded by
quotes, in this case the current command creates an invalid entry and
it's not possible to sudo anymore.
This fix adjust the sed command to deal with possible quotes

Change-Id: Ifd6f9e29b3c0d04d6f65d3f55524ad202fb3294e
2016-08-03 15:24:07 +02:00
Matthew Thode
ffb8efda74
Optionally remove portage files
Optionally remove portage files, so that we can cache package and
keep the portage directory around, specifically for nodepool.

This also adds a section to the Gentoo readme about the variable
and renames the 00-gentoo-distro-name environment file to a more
appropriate name of 00-gentoo-envars.

Also brought up was the location of the gentoo-releng.gpg file,
this has been moved and the refrencing paths updated.

Change-Id: I20c91b36082828faa1ca481585acc5f9933211e1
2016-08-02 16:42:29 -05:00
Ben Nemec
e49d347ba9 Generalize logic for skipping final image generation
Since the ironic-agent element builds the ramdisk and extracts the
kernel itself, there's no need to actually generate an image at the
end of the process.  Previously the unnecessary image was being
deleted, but this wastes a bunch of time compressing and converting
the image.  It's better to just not create the image at all.

This change adds a noop element called no-final-image that
disk-image-create looks for in the element list and, if found, will
cause it to skip the final image generation.  This is more flexible
than the previous ironic-agent-specific method that would have
required changes to disk-image-create for every element that wanted
to behave similarly.

Note that this cannot be done using an environment variable, because
element environments.d entries do not propagate out to
disk-image-create.  It also doesn't make sense as a user option
because it should be set by the element author, not the user.

Change-Id: I168feb18f0d578b3babbe4784d3ef75e755e1ebd
2016-07-28 13:14:36 -05:00
Jenkins
ed116d60a9 Merge "Fix proliant-tools dependencies" 2016-07-27 21:19:32 +00:00
Jenkins
e884309272 Merge "yum-minimal: set locale.conf and tz in chroot" 2016-07-27 08:52:39 +00:00
Xiang Chen
6ae60b2e8f add no_proxy when debootstrap trying to use proxy
Most of the time,no useing no_proxy is ok,but sometime this will cause problem.
Add no_proxy here will increase the robustness of the program .

Change-Id: I976e689760d2e6de9e2081fcdee4f71299e8470e
2016-07-22 14:56:22 +08:00
stephane
b7d391f25d Fix proliant-tools dependencies
The proliant-tools element was missing a few
dependencies which were stopping it from
building correctly.

Change-Id: Ib7159a0baa7932d1571272cefffaf01d60e9debc
Closes-Bug: #1590176
2016-07-20 14:50:21 -07:00
Jenkins
71c68bf18d Merge "Fix packaging problems for Debian" 2016-07-20 21:21:08 +00:00
Andreas Florath
a8c8c61711 Fix packaging problems for Debian
This patch solves three issues with Debian packaging / apt:

o When building 'testing' only default apt sources is
  included - backports, updates and security are skipped because they
  do not exists.

o The default release for Debian was `unstable`: this is now fixed to
  `stable`.

o Starting a Debian Stretch VM that was build with diskimage-builder
  does not work, because some mandatory packages are missing.
  This patch fixes this problem: it adds the mandatory packages and
  the test case.

Change-Id: If49b5b162c4da1e074e9b19324839bc59d87dc57
Signed-off-by: Andreas Florath <andreas@florath.net>
2016-07-19 07:11:57 +02:00
Jeffrey Zhang
cf667e5dc3 Remove the escape in the centos7 README file
No need to add escape character '\' in the README file

Change-Id: I475b3e3d12c093a396bdcc81acbceeba03aed6df
2016-07-19 11:32:49 +08:00
Jenkins
b8d10afd8f Merge "package-installs: add list to arch and "not-arch" list" 2016-07-15 06:57:59 +00:00
Ian Wienand
b745821353 yum-minimal: set locale.conf and tz in chroot
We should be doing more to ensure initial configuration during
configuration.  Taken from the steps done by [1], here we set
locale.conf and a general timezone.

The only reliable UTF8 locale is en_US.UTF-8; we don't want to use C
locale as it causes havoc with things like python3 and unicode.  We
set locale.conf to this.

For Fedora 24 ensure we install the en_* locales too (this is really a
bug separate to this -- when you log in, by default ssh tries to copy
over your locale env variables, so logging into a F24 system would
result in using invalid locales for the most common en_* cases).

While we are here, setup a timezone link.  It turns out infra puppet
overwrites this later, but at least we have a sane default.

[1] https://www.freedesktop.org/software/systemd/man/systemd-firstboot.html#

Change-Id: Ib8951a97f1772bc5228c682e88628ff53400a923
2016-07-15 13:35:10 +10:00
Jenkins
96ed39aa35 Merge "Make Fedora 24 the default" 2016-07-14 05:26:04 +00:00
Ian Wienand
995c539491 Revert "Revert "Pre-install pip/virtualenv packages""
This reverts commit a645fa4ffb.

It is really devstack causing problems here; it was removing the
python-virtualenv package & re-installing using pip (see depends-on).
This failed because the pip-install we did here removed the egg-file
that rpm expected to be there, so rpm bailed out on the removal.

But even if it worked, this just leads you back down the path of the
original problem; that the system packaged version can be re-installed
and overwrites the pip installed version.  Thus I still believe this
is the correct thing to do in the dib element.

Note it is not a common problem (devstack aside); most jobs don't
touch python-virtualenv & related packages (the one we did notice this
on was being brought over from travisci where it was required for some
reason).

Change-Id: I82acb865378a0fa5903a6267bfcee0e2962eced0
Depends-On: Ib0edf6c4ee8a510e9d671213de35d787f56acfed
2016-07-14 13:54:41 +10:00
stephane
09317fccca Make Fedora 24 the default
Fedora 24 is the current release, so use it in the
fedora and fedora-minimal elements.

Change-Id: I0ac0d0767d4de9d28ae86d1344fd7c9d8876e5a1
2016-07-08 09:15:08 -07:00
James Slagle
1d629ccd46 Check sudoers file after editing
"visudo -c" should be run after the sudoers file has been edited. This
will ensure that the file is still syntactically correct, and exit 1 if
it isn't.  Otherwise, obscure errors can occur later on, and it is
difficult to track them back to this script as the source of the error.

Change-Id: Id0e5114d72c0779952a0c2c2c06696929c6c8b17
2016-07-07 09:51:43 -04:00
Jenkins
8b00250547 Merge "Make ubuntu-core support releases" 2016-07-06 23:49:31 +00:00
Jenkins
15e6d51dfe Merge "Revert "Pre-install pip/virtualenv packages"" 2016-07-05 21:42:48 +00:00
Ricardo Carrillo Cruz
a645fa4ffb Revert "Pre-install pip/virtualenv packages"
This reverts commit a85ce75d6b.

The reason for this revert is because it breaks CentOS dsvm jobs:

http://logs.openstack.org/35/332435/6/check/gate-ansible-role-cloud-launcher-dsvm-ansible-func-centos-7/13b3b66/logs/devstacklog.txt.gz

It is due to DevStack removing python-virtualenv by default, causing issues
as CentOS images have virtualenv installed via system-package and pip
by DIB.

Closes-Bug: 1599277
Change-Id: Ib0c01a7b3be32159e7c864a465fe9455e8d651ef
2016-07-05 20:09:39 +00:00
Jenkins
1b1a6a91eb Merge "Add new posix element." 2016-07-05 19:13:06 +00:00
Ian Wienand
8a1c8370a1 package-installs: add list to arch and "not-arch" list
Icf8a075224833fcfbbe2128e8802ff41c39f3c09 looked rather ugly, and it's
easy for us to expand the processing done in the arch list.

Change "arch" to a comma-separated list of architectures that should
match for install.

Add a "not-arch" list which will exclude the package from installation
on those architectures.  (An aside -- I considered making it just he
one list with foo,!bar,moo but ! has special meaning in YAML, so it's
easier to have two lists).

$ ARCH=ppc64 package-installs-squash --elements ironic-agent --path=./elements/ /dev/stdout | grep dmidecode
$ ARCH=ppc64 package-installs-squash --elements ironic-agent --path=./elements/ /dev/stdout | grep lshw
    "lshw",
$ ARCH=amd64 package-installs-squash --elements ironic-agent --path=./elements/ /dev/stdout | grep lshw
$ ARCH=amd64 package-installs-squash --elements ironic-agent --path=./elements/ /dev/stdout | grep dmidecode
    "dmidecode",

Change-Id: Ic69dd02a09e6f3ba9078a2377d8df29871a20db2
2016-07-01 21:31:59 +02:00
Bernard Cafarelli
097240c226 Export YUM variable in centos bases
Other fedora/centos elements can use the YUM variable, already set in
some base elements (fedora, centos-minimal). This commit also exports it
for centos/centos7.

Set a fallback value in pip-and-virtualenv element.

Change-Id: I681d77b924be035c81043bb34c72ec5f859e7108
Closes-Bug: 1598087
2016-07-01 12:14:38 +02:00
Jenkins
908ca1d079 Merge "dmidecode does not exist for ppc64/ppc64el" 2016-06-30 05:00:02 +00:00
Mark Hamzy
76bf793a39 dmidecode does not exist for ppc64/ppc64el
Adding three separate lines because the arch tag does not support a list.

Change-Id: Icf8a075224833fcfbbe2128e8802ff41c39f3c09
2016-06-29 16:51:46 -05:00
Jenkins
14152b1657 Merge "Run RHEL system unregister element earlier" 2016-06-28 09:32:53 +00:00
Jenkins
4493aadbd1 Merge "Clean more from ironic-agent ramdisk image" 2016-06-27 15:13:55 +00:00
Gregory Haynes
fd5fbdd4b5 Make xenial the ubuntu default
This is the new LTS.

Change-Id: I42a6c0520dde8be21df396e7c2e6fb5ae55d2025
2016-06-25 04:20:47 +00:00
Ben Nemec
4e5dcdd9cf Clean more from ironic-agent ramdisk image
While we already clean a number of things off the ironic-agent
ramdisk, there are a few more significant ones that we should add
to the list.

First is the kernel source.  If you're rebuilding your kernel on
the agent ramdisk after the initial image build, then you need to
re-examine your life choices. ;-)

Second is /var/cache.  On yum-based distros, this contains a large
number of yum cache files that take up significant space.  We don't
really want to be copying around caches when booting a ramdisk
anyway, so cleaning this is the right thing to do regardless.

Third is all *.pyc or *.pyo files.  There are a lot of these, so
they eat up significant space and bloat the number of files in the
ramdisk, which makes it take longer to build.  the only purpose for
the files is to slightly speed up Python app startup, and we
probably lose more time transferring the files over the network
than we would gain in quicker start times.  Note that we were
already trying to remove these, but for some reason I was still
seeing them show up in my final images.  It makes more sense to
put them in the same pruning command as all the others anyway.

Fourth is /usr/include.  These are files only needed for
compilation.  See above for my thoughts on compiling in a ramdisk.

These changes have reduced the agent ramdisk from 391 MB to 333 MB
in my local centos 7 builds, and have reduced the number of files
in the ramdisk by over 18000.

Change-Id: I550f9904b9afd12d48da9ba24559acb23133d076
2016-06-24 13:21:32 -05:00
Ian Wienand
1b4e6eb5f7 Handle locales install on Fedora 24
Fedora 24 has split locales into separate packages.  Testing revealed
what is possibly a bug in the choosing of default packages, so add a
small work-around to ensure the minimal locale pack is installed.

This appears to be the only change required for fedora-minimal with
Fedora 24; at least to build with the project-config infra elements.

Change-Id: I64438c34c572ed96211384ae1bfb45b2949e4318
2016-06-24 10:28:06 +10:00
Bernard Cafarelli
87379da56a Run RHEL system unregister element earlier
This does not need to be the last finalise step, and some late finalise
steps can disable the network (for example, Octavia amphora DIB cleans
resolv.conf at 99) Moving it to 60 also aligns it with rhsm-unregister
rhel6 element, and still allows to run subscription-manager steps
before.

Also fix an unbound variable error that appeared when both
BASE_IMAGE_FILE and DIB_CLOUD_IMAGES are unset.

Change-Id: Icb0e20b01479fea345e01309fc4bf3f7f639900c
2016-06-22 17:10:04 +02:00
Jenkins
f9d3bf0b1c Merge "Pre-install pip/virtualenv packages" 2016-06-17 05:05:11 +00:00
Ian Wienand
a85ce75d6b Pre-install pip/virtualenv packages
If we're installing pip/virtualenv from source, we need to make sure
we pre-install the packaged versions before the upstream
versions. Otherwise, CI jobs later on that depend on packaged versions
of pip/virtualenv can bring them in and overwrite the upstream
versions we have installed, which leads to a heck of a mess and
usually very confusing failures.

I have also moved in a small hack from system-config:install_puppet.sh
that we found was necessary when using pip versions from upstream.

Note this is not as much of an issue on Debian/Ubuntu, as they keep
their pip packages in a separate place to the system packages, so you
don't have these overwite conflicts as much.

Change-Id: Ib40708c07b939b84661c44df88a5a308fd0c7216
2016-06-17 09:20:57 +10:00
stephane
7eb72c0874 Ironic agent kernel should be owned by user building image
The initramfs file created by the ironic-agent element is
owned by the user running disk-image-create; ensure that
the other files created by the element are also owned by
the user.

Change-Id: I829db5b8e8bf1fc68face9cd2bda52d2a5ccdd4f
Closes-Bug: 1593010
2016-06-16 14:28:23 -07:00
Jenkins
d9525cfee9 Merge "Add cinder-backup mappings" 2016-06-10 01:20:45 +00:00
Ian Wienand
b9fdc70e32 Move pkg-map to dib-python
The latest Fedora/Ubuntu images don't ship python2 by default, so we
need to use our dib-python wrapper for this so we work in python3 only
environments.

Closes-bug: 1577105
Change-Id: I1048ceef35f269960216066924986eec6117ca00
2016-06-08 21:16:50 +00:00
Erno Kuvaja
6fd6b3bb25 Add cinder-backup mappings
'cinder-backup' to 'openstack-cinder-backup' mappings where necessary

Change-Id: I5f57401295d655112f0ad1d6442268df149daa7e
2016-06-07 14:15:47 +02:00
Jenkins
fb92e61082 Merge "Add cloud-initramfs-growroot for Precise" 2016-06-07 01:14:10 +00:00
Jenkins
dc149ec9fd Merge "Add release to pkg-map" 2016-06-07 01:01:09 +00:00
Jenkins
fb6fa48f85 Merge "Export FS_TYPE and remove hardcoded ext4 values" 2016-06-07 01:01:04 +00:00
Jenkins
13323c9495 Merge "Export die() function" 2016-06-07 01:00:50 +00:00
Jenkins
536e8a2dd2 Merge "Cleanup source-repositories output" 2016-06-07 00:59:02 +00:00
Ian Wienand
74d0185dde Cleanup source-repositories output
This element takes up a *huge* part of the logs of openstack-infra
builds as we go and cache every git tree.

This silences most of the noise which will reduce the mess
in the logs considerably.

Note that we've had logging on this turned down since
I91c5e55814ba9840769357261d203f4850e2eba6 but it has been
ineffective in stopping the log-spew output, see the
dependency change.

Change-Id: I60f06f84d57087c82b3907575bff125015d35171
Depends-On: I1e39822f218dc0322e2490a770f3dc867a55802c
2016-06-06 15:57:39 +10:00
Jenkins
38dcce0fc4 Merge "Fix variable unbound error while REPOREF="*"" 2016-06-02 05:20:37 +00:00
Jenkins
419568884c Merge "Rework yum-minimal locale cleanup" 2016-05-31 06:42:20 +00:00
Ian Wienand
f5dff9c52a Rework yum-minimal locale cleanup
It turns out our manual locale cleanup is causing issues (see
I54490b17a7f8b2f977369044fcc6bb49cc13768e).  Upon further
investigation, I think this is a better approach than manually
deleting repos.

glibc on Fedora obeys the %_install_langs macro for reducing the
installed locales (as mentioned in the comments, F24 has moved to
having different packages, but worry about that later).

So our existing clear-out is really only required for CentOS, whose
glibc does not have any way to indicate to build less locales.
However, %_install_langs is still correct there, as it restricts some
of the translation files and other things installed with the %lang
macro in spec files.

This is complicated by us having to set this at glibc-common install
time, which happens with the "yum" from outside the chroot (i.e. on
trusty).  Since this is too old to have flags to pass this, we need to
fiddle with rpmmacros.

I've tested this with fedora-minimal builds and the locales file is
about 2MiB, which is what it was after the cleanups, and the listed
locales are only those we expect (i.e. it appears to be working).

Change-Id: I528a68beeb7b2ceec25ccbec1900670501608158
2016-05-31 15:14:24 +10:00
Andreas Florath
9fbc462e00 Remove Fedora 21 from test-build
Because Fedora 21 is EOL, the appropriate cloud images were
removed from the mirrors during the last days.
Because of the removal, currently all CI tests are failing.

This patch removes the Fedora 21 CI tests.

(Adding and supporting Fedora 23 is done with another patch -
because some additional changes are needed here.)

Change-Id: Ib85bb6fafd4f56ecc55dd420048f4d9e6e6969f3
Signed-off-by: Andreas Florath <andreas@florath.net>
2016-05-30 22:54:58 +02:00
Ian Wienand
7550d25db0 Export die() function
I realised I'd been using die() in a few places assuming it was
available, but it wasn't exported.  I guess it didn't matter because
whatever was wrong, we were failing anyway :)

This exports the function to make it available to sub-processes, which
should remove the need to source it as done in several places.

Change-Id: I7b9a5a6db406e160099b6ed9fde80455ae227327
2016-05-27 09:25:22 +10:00
Jenkins
5f4cac3303 Merge "Add a best-effort sudo safety check" 2016-05-26 17:31:00 +00:00
Liu Qing
d3255835d1 Fix variable unbound error while REPOREF="*"
If REPOREF="*", HAS_REF will be used without initialization. As -u is set
the script will terminate with error.

Change-Id: Ic1d88415adfef66dfc6c1d92610a45a9eb6359f3
Signed-off-by: Liu Qing <liuqing@chinac.com>
2016-05-26 09:24:59 +08:00
Jiri Stransky
1e9cf3a1c8 Remove deprecated overriding of cloud-init defaults
The "set to localhost by default" behavior for manage_etc_hosts has been
deprecated for more than a year now by change
Ia8582883f737548e2911d3f36a1943e5b236281b.

Setting that value to "localhost" is still possible, but it won't be the
default anymore. If the previous behavior of assigning the hostname and
FQDN to 127.0.0.1 is still desired in some environments, it can be
achieved by setting the DIB_CLOUD_INIT_ETC_HOSTS environment variable,
as the deprecation warning message suggested.

Change-Id: I5a19d46e2f305769a0c89c9d25d2e6be02910221
2016-05-23 17:44:32 +02:00
Jenkins
f0b57d5efd Merge "Fix apt-sources configuration for debian-minimal" 2016-05-21 20:10:18 +00:00
Ian Wienand
244f0a0001 Add cloud-initramfs-growroot for Precise
Add a pkg-map to install cloud-initramfs-growroot on Precise, as it is
required there because the kernel is too old to use growroot on a live
file-system.  This was generically removed in
e23b087505.

Change-Id: I5e6cbc4b74dc72a6f23f73a018c028f3ff7c0157
2016-05-20 14:32:41 +10:00
Ian Wienand
716b8e14fe Add release to pkg-map
Add ability to filter in pkg-map on release, which defaults to
DIB_RELEASE.  As per the examples, release is a more specific target
than distro, but distro still has to match.

Added a debug flag and ability to use a pkg-map directly for
debugging/development.

Change-Id: Ie282f96966e46236b06bc276de0168fc7a66c5da
2016-05-20 14:26:25 +10:00
Paul Belanger
be521bdec6 Export FS_TYPE and remove hardcoded ext4 values
Export FS_TYPE from img-defaults and use it to remove hard-coded
defaults in the debootstrap mounting.  Also, cleanup the suse element
as it should have access to the exported variable.

Change-Id: Ie9b671ca9336060a5ad294be48aa7eff442bf066
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2016-05-19 17:33:36 +10:00
Jenkins
95c874abb9 Merge "Add dhcp-all-interfaces.target for syncing units" 2016-05-18 19:39:10 +00:00
Jenkins
2fffc9f892 Merge "Fix path issue for locale-archive.tmpl" 2016-05-16 22:01:47 +00:00
Jenkins
02822dc6b9 Merge "Add centos7 test" 2016-05-16 21:43:28 +00:00
Paul Belanger
0478fb15db
Fix path issue for locale-archive.tmpl
Change-Id: Id589c16aab46d447b3c21f00f3acfd06890e43d2
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2016-05-16 09:23:58 -04:00
Sam Betts
eb99fe7144 Add dhcp-all-interfaces.target for syncing units
Currently there is no way for a service to become aware that
dhcp-all-interfaces is finished configuring all the interfaces at
boot time. This causes problems for applications like the
ironic-python-agent which scans the interfaces when it first starts as
part of the inspection stage and can race against dhcp-all-interfaces
bringing up the interfaces, leading to inconsistent results.

This patch ensures that the dhcp-all-interfaces script runs before any
network interface is configured and brought up by the rest of the
system, and also ensures that the ironic-agent element also waits for
the network to be online before starting. This is done by using the
network targets provided by systemd.

Change-Id: Id9583b7f54361aa603a6229da598ad6a0f0f7938
2016-05-16 10:15:53 +01:00
Ian Wienand
5d23abb338 centos-minimal: can be used with base
I'm not sure why this comment is here.  base works with centos-minimal
(we changed it in I2956aaa49ba3137a799f97e0983ab4b7c93a0a0c) and we've
got images deployed with such a configuration.

Update the comment

Change-Id: I3207f87dc29280183c0960ea863533a4d441081c
2016-05-16 13:36:55 +10:00
Jenkins
d253a94187 Merge "Do not remove sudo in ironic-agent" 2016-05-16 01:41:08 +00:00
vmud213
45628993e8 Do not remove sudo in ironic-agent
"ironic-agent" element is currently removing sudo, which breaks other
elements such as devuser.  There appears to be no security or other
reason to do this, it's just the way it has always been.  Leave sudo
in as it is considered part of the base cloud images.

Change-Id: Ida9b1885f745146071e4b2d85ae59341ac85d5c8
Closes-Bug: #1572486
2016-05-16 10:39:04 +10:00
Andreas Florath
92355004d9 Fix apt-sources configuration for debian-minimal
The apt-sources element did not work with debian-minimal, because
the later one overwrote the /etc/apt/sources.list file created by
the apt-sources element.

Two changes were made:
o the debian-minimal uses now files inside the /etc/apt/sources.list.d
  directory.  Therefore there is no possibilty for clashes between those
  two elements any more.
o instead of only adding backports, also the updates and the security
  repository is added by default which gives perfect initial
  configuration for a stable system.
  If you want to use local mirrors with other naming schemas or an
  unstable tree, there is the possibility to fully specify the
  repositories.

Change-Id: I69dbaa34be3db3d667e6bd8450ef4ce04a751c70
Signed-off-by: Andreas Florath <andreas@florath.net>
2016-05-13 09:26:49 +02:00
Jenkins
77d03556df Merge "Add EPEL as requirement of centos-minimal" 2016-05-13 03:18:58 +00:00
Jenkins
0a3d9999d6 Merge "Add install-bin element" 2016-05-13 03:00:25 +00:00
Jenkins
36a86afdad Merge "Install proliantutils in IPA's virtualenv" 2016-05-13 01:54:06 +00:00
Gregory Haynes
642f906476 Add install-bin element
The various distro elements include a copy of this script which installs
all files in the bin/ dir of the copied element hooks in to
/usr/local/bin. Lets share code rather than repeating ourselves.

Change-Id: I354382f8c42ede2e9b8c548b7df8367c03e6836e
2016-05-13 11:52:23 +10:00
Jenkins
8cb8806227 Merge "yum-minimal : better cleanup of initial yum failure" 2016-05-13 01:04:33 +00:00
Jenkins
88a9365816 Merge "yum-minimal: strip locale archive" 2016-05-13 01:04:27 +00:00
Jenkins
5657852b22 Merge "Fix up EPEL element" 2016-05-13 01:04:24 +00:00
Ian Wienand
672705831f Add a best-effort sudo safety check
As motivation for this; we have had two breakouts of dib in recent
memory.  One was a failure to unmount through symlinks in the core
code (I335316019ef948758392b03e91f9869102a472b9) and the other was
removing host keys on the build-system
(Ib01d71ff9415a0ae04d963f6e380aab9ac2260ce).

For the most part, dib runs unprivileged.  Bits of the core code are
hopefully well tested (modulo bugs like the first one!).  We give free
reign inside the chroot (although there is still some potential there
for adverse external affects via bind mounts).  Where we could be a
bit safer (and could have prevented at least the second of these
breakouts) is with some better checking that the "sudo" calls
*outside* the chroot at least looked sane.

This adds a basic check that we're using chroot or image paths when
calling sudo in those parts of elements that run *outside* the chroot.
Various files are updated to accomodate this check; mostly by just
ignoring it for existing code (I have not audited these calls).

Nobody is pretending this type of checking makes dib magically safe,
or removes the issues with it needing to do things as root during the
build.  But this can help find egregious errors like the key removal.

Change-Id: I161a5aea1d29dcdc7236f70d372c53246ec73749
2016-05-09 15:41:38 +10:00
Colleen Murphy
b5f51322a3 Fix OpenSUSE support
The dhcp-all-interfaces and simple-init elements did not have the ISC
DHCP Client package mapped for OpenSUSE, which caused DIB to fail with
"'isc-dhcp-client' not found in package names. Trying capabilities."

Similarly, the bootloader element did not have the grub-pc package
properly mapped for OpenSuse, which caused DIB to fail with "Package
'grub-pc' not found.".

This patch adds the package mappings for these elements so that the
opensuse element can be created and booted successfully.

Change-Id: Ife478158fec3a95de73a9206b38dcc6511d56cc8
2016-05-03 22:23:51 -07:00
Jenkins
83b607557e Merge "Remove cloud-initramfs-growroot package" 2016-05-03 13:24:53 +00:00
Nisha Agarwal
9d397d2568 Install proliantutils in IPA's virtualenv
The proliant-tools element helps to do RAID
configuration in ironic for HPE servers.
This fix proposes to install the proliantutils
in ironic-python-agent's virtualenv created
using ironic-agent element.

Closes-Bug: 1563648
Change-Id: If63c725a42740ab244a2b4004797cba09d0f154e
2016-05-02 01:56:26 -07:00
Ian Wienand
11128b0673 Use generic "dhcp-client" name
Every platform has a different name for their DHCP client, so use a
generic name "dhcp-client" in the package name and let everyone choose
their sub-name.  This also brings some consistency across simple-init
& dhcp-all-interfaces

Change-Id: I797aa7aacb13dfb7f35700463dc11d55552eb108
2016-04-22 11:31:54 +10:00
Gregory Haynes
a078e780ca dhcp-all-interfaces depends on dib-init-system
This element uses the dib-init-system command and therefore depends on
the element.

Change-Id: I1374500fb5b79e0f0c9c41346b5b7baf3f7755aa
2016-04-22 09:23:11 +10:00
Gregory Haynes
e096337a21 dhcp-all-interfaces depends on dhcp
Add package dependency for dhcp client

Change-Id: I63683485a5c5dbe65bfc38c8d64a88ee5549fda8
2016-04-22 09:23:09 +10:00
Jenkins
33d7e8b25e Merge "Add Gentoo to the dhcp-all-interfaces element" 2016-04-21 23:03:24 +00:00
Matthew Thode
de0cddc390
Add Gentoo to the dhcp-all-interfaces element
This makes use of the dhcpcd package and it's ability to run on all
interfaces by default.  We disable the privacy extensions and dhcp
overriding the hostname (both are enabled by default).  Other than
that it 'just works' and was the method used to bring up interfaces
on Gentoo Openstack images before we switched to building with DIB.

Change-Id: I02c14927d70b22f560c6fc149fefca0f93933f56
2016-04-21 16:40:06 -05:00
Jenkins
45afd99012 Merge "Handle unconfigured interfaces for dhcp-all-ifaces" 2016-04-21 05:23:37 +00:00
Jenkins
874fef9fe9 Merge "Really remove all interfaces in dhcp-all-ifaces" 2016-04-21 05:23:31 +00:00
Ian Wienand
7aa9157c33 yum-minimal: strip locale archive
Rather than removing all locale related stuff in cleanup, strip the
locale archive and rebuild it.

Building just en_US (along with POSIX/C) brings things inline with
debootstrap.  As discussed in the bug referenced, this is about the
best we can do for Centos7.

Fedora 24 has split languages out into packages so we don't have to do
this, but I have not dealt with that yet.  A guard is put in place so
we make sure we revisit this when we try to build F24.

Change-Id: I3f384d23e52effd6a09f47134746caa4a5c586be
2016-04-21 15:00:13 +10:00
Jenkins
a6754a5c3a Merge "Move selinux restore to end of finalise" 2016-04-21 04:30:08 +00:00
Abel Lopez
b2a2368844 Change to latest CentOS-6 image
cloud.centos.org appears to have changed their naming for images.
This latest iteration drops the YYYYMMDD in favor for YYMM, but
also has a 'latest' available without the date stamp.

This change will mean we no longer have to submit new code reviews
whenever centos changes.

Change-Id: I5a6a0de822561c1d0681abb9487993acf55918f1
2016-04-20 10:44:09 -07:00