Commit graph

3050 commits

Author SHA1 Message Date
Ian Wienand
ff79dc9793 Catch errors in DIB_INIT_SYSTEM export
When you source a file that just does

 export FOO=$(bar)

you miss any invalid return codes from "bar" (even under -e) because
bash returns the value of the "export", which is 0

On centos-minimal, we stopped bringing in systemd early and this was
causing dib-init-system to not know what init was available.  Since it
did not fail correctly, it lead to confusing errors much later in the
build when service files were not copied correctly.  See also
I24ce648485c3d6f3c27ab8f87a638516b3727017

A dib-lint check is added.  One minor fixup is in 00-set-apt-sources
(this one is less likely to cause problems).  I have run dib-lint over
project-config elements and none use this pattern.

Change-Id: I076c08190d40c315ad6a6d96a3823e9fc52630be
2016-11-23 23:03:50 +00:00
Luong Anh Tuan
29d50bc69b Replace six.iteritems() with .items()
We should avoid using six.iteritems/keys achieve iterators. We can use
dict.items/keys instead, as it will return iterators in PY3 as well.
And dict.items/keys will more readable.

In py2, the performance about list should be negligible, see
https://wiki.openstack.org/wiki/Python3 and
http://lists.openstack.org/pipermail/openstack-dev/2015-June/066391.html

Change-Id: I6353f0a1b423c6dbd0c71343f7919fd0de440e34
2016-11-23 12:03:05 +00:00
Ian Wienand
bc6be85424 Trace package install in package-installs-v2
When running the package install, trace the output so we can see what
packages were installed.

Change-Id: I5442f544ff0ef3ddffdbe6b898d178548d699a41
2016-11-23 19:58:45 +11:00
Ian Wienand
f15550f9fe Special case dib-python in dib-lint
It seems that on Xenial, it does not take much to confuse "file" and
it's mime guessing such that it thinks some files are not python.

"package-installs-v2" is a good example, since it has an interpreter
"dib-python" that "file" doesn't know about, and no extension.  While
looking at this, I've added emacs vars here so it opens in python
mode.

Change-Id: I01994b08c5ad8987925f1eec4062f5b6ee72eb8f
2016-11-23 19:58:43 +11:00
Markos Chandras
e4868d9ad6 elements: Drop unneeded DIB_INIT_SYSTEM usage
DIB_INIT_SYSTEM is exported by the dib-init-system element and contains
the output of the dib-init-system script so there is no need to
re-initialize it during various phases.

Change-Id: I09d6d10742689efe3d8eb9d64b539d6599b46227
2016-11-22 10:47:43 +00:00
Markos Chandras
bbcc22751f elements: Add new openssh-server element
Add new 'openssh-server' element to ensure that openssh server
is installed and enabled during boot. This is mostly useful for
*-minimal images which do not come with openssh installed and/or
enabled in order to keep a small dependency footprint.

Change-Id: Ide15ee04f5de123dbc8ce4bb56d638d8a167c341
2016-11-22 10:07:14 +00:00
Noam Angel
55b6101e1b add option to configure cloud-init to allow password authentication
This patch will configure cloud-init to allow password authentication.
This is usefull in case you use "devuser" element and want to ssh guest
image.

Change-Id: I00e38aa2753f26b4cdd34d0fd85fc8e0de78171f
2016-11-22 09:33:58 +00:00
Markos Chandras
d84df60345 elements: pip-and-virtualenv: Add python-xml dependency
SUSE packages the 'xml' python module as a separate package so make
sure it's pulled in before we attempt to install the pip module
since the latter depends on it. Fixes the following problem when
building with the opensuse-minimal and pip-and-virtualenv elements:

Traceback (most recent call last):
  File "/tmp/get-pip.py", line 19177, in <module>
    main()
  File "/tmp/get-pip.py", line 194, in main
    bootstrap(tmpdir=tmpdir)
  File "/tmp/get-pip.py", line 82, in bootstrap
    import pip
  File "/tmp/tmpOiESjX/pip.zip/pip/__init__.py", line 16, in <module>
  File "/tmp/tmpOiESjX/pip.zip/pip/vcs/subversion.py", line 9, in <module>
  File "/tmp/tmpOiESjX/pip.zip/pip/index.py", line 32, in <module>
  File "/tmp/tmpOiESjX/pip.zip/pip/_vendor/html5lib/__init__.py", line 16, in <module>
  File "/tmp/tmpOiESjX/pip.zip/pip/_vendor/html5lib/html5parser.py", line 6, in <module>
  File "/tmp/tmpOiESjX/pip.zip/pip/_vendor/html5lib/inputstream.py", line 10, in <module>
  File "/tmp/tmpOiESjX/pip.zip/pip/_vendor/html5lib/utils.py", line 10, in <module>
ImportError: No module named xml.etree.ElementTree

Change-Id: I1bec12dfcde05fb07f41bcec994148c3eacbb287
2016-11-21 15:54:18 +00:00
Jenkins
09cab0fa65 Merge "Turn off tracing around pid/chroot check" 2016-11-21 00:31:49 +00:00
Clark Boylan
c5ec1348c3 Fix runtime ssh host keys script
The script is set -e and set -o pipefail, unfortauntely this intersects
with `yes n`'s non zero exit code behavior when it receives an interrupt
like sigpipe. As a result stop setting pipefail so that we treat those
errors as "normal" and only fail if ssh-keygen fails.

Change-Id: I5447df97c9888cae3007e235e2fea44df61af28e
2016-11-19 18:02:33 -05:00
Ian Wienand
e0c346d479 Turn off tracing around pid/chroot check
In the error case, we get a spew of output as this check goes though
every pid checking if its in the chroot.  Disable tracing around the
call.

Change-Id: Ie84f12974755c0c2c51d7e7697337ed9b32a4a1c
2016-11-18 14:51:20 +11:00
Ian Wienand
7d5afecfd9 Merge remote-tracking branch 'origin/master' into merge-branch
Change-Id: Ibab1bb95521292ae818bd91f7073c3749a2cc0cb
2016-11-18 13:53:56 +11:00
gecong1973
f9244a8fce Fix a typo
TrivialFix

Change-Id: Iebe82e616eed2d9b9a99a9714230d480adbd055b
2016-11-18 10:48:05 +08:00
Jenkins
eeb9e0e4a6 Merge "Disable all repos in os-refresh-config too" 2016-11-18 00:23:04 +00:00
Jenkins
0fbf131550 Merge "Cleanup yumdownloader repos" 2016-11-18 00:23:00 +00:00
Jenkins
a25c922dc2 Merge "lib: common-functions: Fix tmpfs umounting" 2016-11-18 00:22:54 +00:00
Jenkins
27bd4741bf Merge "debian: install dialog package" 2016-11-18 00:21:23 +00:00
Jenkins
f5d23c975a Merge "Change path for dnf arch override so basearch is not overwritten." 2016-11-18 00:20:50 +00:00
Jenkins
66873240fb Merge "In disk-image-create, append to INSTALL_PACKAGES instead of clobbering." 2016-11-17 20:07:56 +00:00
Jenkins
0ea4ea5bed Merge "Don't use ssh-keygen -A for init scripts" 2016-11-17 19:54:25 +00:00
d.marlin
8d7362aa9b Change path for dnf arch override so basearch is not overwritten.
After writing the basearch value to /etc/dnf/vars/basearch the
arch value was overwriting the same file.  This appears to be
incorrect, so changing it to write /etc/dnf/vars/arch, which
matches the subsequent 'yum' code paths.

Change-Id: I5da54f03224c11f9e286f16b68533936c4174c2a
2016-11-17 03:36:54 -05:00
d.marlin
9b4d2a22e4 Fedora AArch64 (64-bit ARM) support in diskimage-builder
Add some checks for AArch64 to avoid the "Unknown architecture" or
"architecture not supported" messages, and allow builds to complete.

Change-Id: I89ba609abaeeb7019eb317cf13473929b2065230
2016-11-16 21:47:26 -05:00
Ben Nemec
fb8cf95b6f Disable all repos in os-refresh-config too
This change was made for pre-install so it applies during the
image build, but wasn't applied to the os-refresh-config script
that would run after deployment.  The same problems apply there,
so we should do the same thing.

Change-Id: I4b8534cc9586eeb588b5c358550e76e27d40556a
Closes-Bug: 1629922
2016-11-16 16:27:24 -06:00
Markos Chandras
a1f57b8cad lib: common-functions: Fix tmpfs umounting
It has been observed that some chroot operations spawn additional
processes which rely on chroot files. More specifically, zypper, uses
gpg-agent to import and validate gpg keys for its repositories. This
gpg-agent process may stay alive for longer which prevents unmounting of
the tmpfs directory since the gpg-agent process still uses libraries etc
which were present in the chroot. We try to solve this by using walking
all the pids in /proc to find out the running processes in the chroot and
kill them gracefully. If that fails for whatever reason, then we simply
keep trying to umount the tmpfs directory before we give up.

The gpg-agent process usually terminates soon after its home directory
disappears but on fast systems we can reach the 'umount tmpfs' point
before gpg-agent terminates by itself. The solution is generic enough so
other 'chroot processes' can also be handled appropriately.

Change-Id: Iccf332678c79266113e76f062884fc5ee79e515d
2016-11-16 15:44:01 +00:00
Jenkins
be1e563524 Merge "simplify ARCH param for rhel/centos param can be x86_64 and amd64" 2016-11-15 00:44:00 +00:00
Jenkins
cc3a28755d Merge "Move dib-run-parts into diskimage-builder" into feature/v2 2016-11-15 00:43:49 +00:00
Noam Angel
e88d6b37df add support for SUSE in dhcp-all-interfaces
This patch will add support for SUSE network scripts,
network script in SUSE saved under "/etc/sysconfig/network/ifcfg-*"
see: https://www.suse.com/documentation/sled11/book_sle_admin/data/sec_basicnet_manconf.html

Change-Id: I87ac2e327cee4945c15da9f2e4adc0a8b7650712
2016-11-15 10:20:09 +11:00
Noam Angel
d07d7ed15d simplify ARCH param for rhel/centos param can be x86_64 and amd64
for fedora/rhel/centos the main supported ARCH is x86_64. This patch allow
to call diskimage-builder with the above distro's with param ARCH=x86_64,
And also retain same behaiver when call with ARCH=amd64 as it translate
anyway to x86_64. Doing so wil simplify user expirience.

Change-Id: I229e0912434109b1b48a030bd35ad8dc1096a629
2016-11-15 10:18:14 +11:00
Jenkins
80976d9d44 Merge "Add element for setting sysctl values" 2016-11-14 17:19:04 +00:00
Saverio Proto
5d9d3d5cf0 debian: install dialog package
Without the dialog package is not possible
to properly use an interactive frontend.

debconf will print the following errors:

debconf: unable to initialize frontend: Dialog
debconf: (No usable dialog-like program is installed,
so the dialog based frontend cannot be used. at
/usr/share/perl5/Debconf/FrontEnd/Dialog.pm line 76, <> line 1.)

Change-Id: I0c7142f717cacf7437dbac1e1696f39b00cb4c49
2016-11-14 15:37:39 +01:00
Jenkins
0cf96d429c Merge "Don't include openstack/common in flake8 exclude list" 2016-11-14 10:08:07 +00:00
Jenkins
51aabd9248 Merge "Drop MANIFEST.in - it's not needed by pbr" 2016-11-14 10:08:01 +00:00
Jenkins
77e14e9313 Merge "elements: runtime-ssh-host-keys: Add openssh-client mapping for SUSE" 2016-11-14 10:01:11 +00:00
Jenkins
a0a19cbbfa Merge "elements: opensuse-minimal: Add support for building Tumbleweed images" 2016-11-14 10:01:05 +00:00
Jenkins
6224824bde Merge "Fix ironic-python-agent image not loading vfat mod" 2016-11-14 09:41:35 +00:00
Jenkins
cc8ab5f4b6 Merge "Avoid disabling rhel-7-server-rh-common-rpms" 2016-11-14 09:32:28 +00:00
Jenkins
c9403182f2 Merge "Updated from global requirements" 2016-11-14 07:31:48 +00:00
Jenkins
5c546ed565 Merge "Install lsb package by map name instead of package name" 2016-11-14 07:13:52 +00:00
Jenkins
3bfff8f905 Merge "elements: zypper-minimal: Mount common pseudo filesystems" 2016-11-14 07:08:23 +00:00
Jenkins
aec2128b38 Merge "elements: zypper-minimal: Refresh repositories after adding the cache" 2016-11-14 07:01:13 +00:00
Ben Nemec
79c27199f2 Install lsb package by map name instead of package name
We have a pkg-map entry for lsb_release, but in package-installs.yaml
we refer to the actual package name instead.  This will happen to
work on Red Hat platforms, but it's actually wrong.

Change-Id: Idb248f96e75fa1090422fa08e5fbb2385cc1f517
2016-11-11 14:30:21 -06:00
Oliver Walsh
ae66b64c34 In disk-image-create, append to INSTALL_PACKAGES instead of clobbering.
This allows -p <packagelist> to be used multiple times.

Change-Id: Iabe43982e1606c7ca963a1dd3b23ba47d148ae38
Closes-Bug: #1641157
2016-11-11 17:11:38 +00:00
Ian Wienand
7e60540c0d Cleanup yumdownloader repos
yumdownloader has to have all the repo XML files, etc, which adds up
to a not totally insignificant 150MiB or so.  Currently we're leaking
this directory for every build, which adds up on regualar builders
like nodepool.

Isolate the call with a separate TMPDIR so we can clean it up after
the initial download.

Change-Id: Ic65e8ca837cc76b7a1bb9f83027b4a5bdd270f75
2016-11-10 17:02:53 +11:00
OpenStack Proposal Bot
0f742ce24a Updated from global requirements
Change-Id: I4b8e6f6fdf78706dbb735e25195a1f83413ab3fd
2016-11-09 04:14:55 +00:00
Jenkins
587d14feed Merge "dhcp-all-interfaces: support Centos/RHEL 6" 2016-11-08 11:21:51 +00:00
Ian Wienand
3c6e589226 Remove execute perm from disk-image-create
Remove the x bit from lib/disk-image-create; because it's called
directly by the entry-point, it doesn't need to be exectuable.
This should also be clearer that you're not supposed to run it
by hand.

Remove some boilerplate from old file

Change-Id: Ibb6cdae613e6c9cf21dd6aecc8e1f739bc3a2643
2016-11-04 17:11:27 +11:00
Ian Wienand
d65678678e Move dib-run-parts into diskimage-builder
Move dib-run-parts from dib-utils into diskimage-builder directly.

For calling outside the chroot, we provide a standard entry-point
script.  However, as noted in the warning comment, the underlying
script is still copied directly into the chroot by the dib-run-parts
element.  I believe this to be the KISS approach.

This removes the dependency on dib-utils.  We have discussed this
previously and nobody seemed to think retiring dib-utils was going to
be an issue.

This also updates the documentation to not mention dib-utils, or using
disk-image-create via $PATH setup, but rather gives instructions on
installing from pip with a virtualenv.

Change-Id: Ic1e22ba498d2c368da7d72e2e2b70ff34324feb8
2016-11-04 17:07:37 +11:00
Luca Lorenzetto
c7219a5a60 Avoid disabling rhel-7-server-rh-common-rpms
while using disk-image-builder for building overcloud images for TripleO
using RDO, this repository is (in my opinion) wrongly disabled because
contains certain dependencies needed by RDO packages.
Example: python-cheetah is required for python-nova, but is not
available through RDO repository but only from
rhel-7-server-rh-common-rpms

Closes-Bug: #1638938
Change-Id: I76824c8ec02590397f1ff1d4f177ad061c7bf441
Signed-off-by: Luca Lorenzetto <lorenzetto.luca@gmail.com>
2016-11-03 15:15:42 +01:00
Markos Chandras
435f52a7d1 elements: zypper-minimal: Mount common pseudo filesystems
Mount all the usual /dev /sys /proc pseudo filesystems during the
root.d phase in order to make sure they are available for the rpm
post-installation phases.

Change-Id: I28221debf1036d9eb5137161757eb30811eafab1
2016-11-02 22:08:35 +00:00
Jenkins
c0219d0528 Merge "Updated from global requirements" 2016-11-02 21:06:16 +00:00