Commit graph

1673 commits

Author SHA1 Message Date
Jenkins
4216b23654 Merge "Adding InfiniBand Support" 2016-03-12 19:21:19 +00:00
Jenkins
1308ec4bc2 Merge "debian-minimal: configurable debootstrap components" 2016-03-12 19:13:03 +00:00
Jenkins
63336d1f6f Merge "Add force-confdef in debian package install" 2016-03-12 19:05:00 +00:00
Jenkins
6d1bc6f1c3 Merge "Refactor growroot for debuggabilty" 2016-03-12 19:04:22 +00:00
Jenkins
48587b3e93 Merge "Depend on ifupdown in simple-init" 2016-03-12 19:04:04 +00:00
Ian Wienand
204da89128 centos-minimal does not provide base
The "base" element is included by default, and if centos-minimal tries
to provide it then we die because two elements are providing the same
thing.

Change-Id: I2956aaa49ba3137a799f97e0983ab4b7c93a0a0c
2016-03-10 13:51:08 +11:00
Moshe Levi
9074f5f1fc Add lshw package to ironic-agent
This patch add the lshw utils to allow the agent
to retrive the vendor name. It also keeps the
/usr/share/misc/pci.ids file for Ubuntu distro.

Change-Id: I2828a7e585449f63887ad5a2e338b4e043d68e63
2016-03-09 12:54:57 +02:00
Ian Wienand
6f622fb7c9 yum-minimal: clear our rpm/dnf/yum data in chroot
I have seem some occasional odd failures coming from the "dnf -y
update" done by elements/base/install.d/00-up-to-date.

dnf seems to sometimes think a package is not installed when it really
is.  It then seems to try and re-install them, but notices they are
installed, and then bails with a failure exit [1].  The packages that
seem to cause this vary, but the common thread is that they seem to
have all been installed during the initial phase of installing the
package manager in the chroot.

I suspect that when we are building the chroot, we do our initial
install with the "external" yum & rpm.  Then we start using the
dnf/yum in the chroot, but we're actually using meta-data created by
the *external* tools -- which could be vastly different versions or
who-knows-what.  While I honestly I don't have an exact root cause,
empirically I've found rebuilding the rpm db always seems to fix
things up.

So this change takes care to rebuild the rpm db with the chroot
version of rpm, and clear out the package metadata for a refresh with
"update".  This should hopefully put us in a consistent state.

[1] http://paste.openstack.org/show/487356/

Change-Id: I565df23897ae511356c4861fdbe63823fa6b6ff9
2016-03-08 14:53:52 +11:00
Yolanda Robla
e083686e00 Create new partitioning-sfdisk element.
This element is similar to vm, but allows more flexibility
on disk partitioning. It receives a DIB_PARTITIONING_SFDISK_SCHEMA
setting, allowing to partition the disk according to project needs.

Change-Id: I01dca5d5fd670d317f7761911a1549507de6e97d
2016-03-07 19:50:55 +01:00
Jenkins
8b0d124a0e Merge "Fix spurious = in dib-python readme" 2016-03-07 04:59:22 +00:00
Jenkins
700445b3db Merge "Fix cloud-init-disable-resizefs README title" 2016-03-07 04:49:53 +00:00
Gregory Haynes
fbabe0b663 Fix spurious = in dib-python readme
Change-Id: I51e7961fab7f3eeaf3ce4eeb1dd5e6981eefb154
2016-03-07 04:47:35 +00:00
Jenkins
6e29b45d90 Merge "Replace sfdisk partitioning with parted" 2016-03-03 12:35:50 +00:00
Gregory Haynes
6ee7da704d Fix cloud-init-disable-resizefs README title
Element readme's need to start with a title that matches the element
name.

Change-Id: Idf48f70e112af1f17d398155a317872b1d569f21
2016-03-01 21:49:10 -08:00
Matthew Thode
175e292ec1 Add Gentoo to the list of supported distributions
We also document any additional package needed.

Change-Id: I7b9193fa5711138287f587935fc864fee4f95856
2016-02-29 12:20:50 +00:00
Gregory Haynes
cb15e7ef33 Prioritize venv python on host
If we are being run from a python 3 virtualenv where python2 is also
available on the host we will use the hosts' python2 over the virtualenv
python3 (even if the virtualenv is higher precedence in PATH).

Change-Id: I9a25b9e45845a4121aab8250fd53c6f006825742
2016-02-28 20:50:02 -08:00
Jenkins
103b7dea6d Merge "Fix building on gentoo hosts" 2016-02-27 14:44:46 +00:00
Jenkins
de4d0392d5 Merge "Install IPA in a virtual environment" 2016-02-26 16:23:37 +00:00
Jenkins
cebdba1f9c Merge "Revert "Correct rhel-common for rhel6"" 2016-02-22 16:57:40 +00:00
Matthew Thode
dacfc23c36
Fix building on gentoo hosts
This patch fixes the profile autodetect.  If the profile was passed at install
time, it would overridden by the environment.d script.  This was fixed by
accepting the value passed by the user, if one was passed.

The enviroment.d file is run outside of chroot, because of this eselect will
not be available on non-gentoo systems.  We change the behavior to work
elsewhere, it was not working on non-gentoo systems and giving bad results on
gentoo systems (returing values from the host, not the guest being built).

This patch also fixes shm detection by checking if it is mounted already.

Change-Id: Ie58d8e19529a731bfbc9eeb4bb246988d1aaa772
2016-02-21 22:46:13 -06:00
Monty Taylor
74487a383c Depend on ifupdown in simple-init
Newer distros, such as debian jessie and ubuntu xenial, do not provide
ifupdown by default, but simple-init depends on it. Add it to the pile.

Change-Id: I6f4876863c67c65a82464d4e0593015cdc839c5c
2016-02-20 08:19:21 -06:00
Jenkins
05edfb6d20 Merge "Don't remove python3 & grubby in 99-remove-extra-packages" 2016-02-19 01:46:38 +00:00
Ben Nemec
a342036e75 Revert "Correct rhel-common for rhel6"
This is breaking deployments that use registration.  I
believe os-refresh-config scripts don't source environment.d
files, so the variable is unset there.

Given that this a blocking issue for RHEL deployments, I'm
pushing a quick revert and we can come up with a proper fix
when this isn't stopping other work.

This reverts commit 71bd8b3a33.

Change-Id: I87504660352220d45f5445bb933edc1c28885fcb
2016-02-17 22:17:11 +00:00
Moshe Levi
14b900d3c9 Adding InfiniBand Support
This patch update the find_interface to lookup
for InfiniBand interface according to it's BOOTIF

Closes-Bug: #1532534

Change-Id: I21b91cfd10888ac036f6347a0a44cdca422830a6
2016-02-17 04:43:29 +00:00
Jenkins
352e58ceb7 Merge "yum-minimal: pre-install base packages" 2016-02-17 00:25:08 +00:00
Yolanda Robla
d5a6a7cb44 Add force-confdef in debian package install
In certain cases, with packages cached that need an upgrade,
when performing that action the system hangs waiting for
a prompt.
Add force-confdef option, that will let dpkg overwrite
configuration packages that were not modified. In combination
with already existing force-confold flag, it will allow
to avoid any conffile prompt.

Change-Id: Ifb177f9ac2c9ad29f8b92309c5b8cfe8e60a4e14
2016-02-16 14:10:19 +01:00
Lucas Alvares Gomes
006b3e6228 Install IPA in a virtual environment
In order to avoid conflict with installing dependencies for IPA from
pypi and distribution packages this patch propose installing IPA in a
virtual environment.

Closes-Bug: #1506792
Change-Id: I6a4c6403813d86f4110e98449ddd874109101b9e
2016-02-16 11:17:12 +00:00
Gregory Haynes
2678209e94 Don't cache debootstrap rootfs by default
By default we create a tarball of any debootstrap rootfs we create. For
the majority of use cases this is a large performance hit for no
benefit. Lets make this an opt-in feature.

Change-Id: I58fc485aacacaa17243bf9ce760ed91256d1f182
2016-02-16 07:55:19 +00:00
Ian Wienand
b960614c9c Don't remove python3 & grubby in 99-remove-extra-packages
python3 is a hard requirement of dnf so can't be removed [1]

grubby is also required for kernel installs on Fedora.  For too much
detail see I1a6e45d04755515286b3d49f8280c16b527e2f48; but the kernel,
via dracut, now has this as a "recommends" due to people removing it
and making unbootable systems.

[1] http://logs.openstack.org/76/248976/2/check/gate-dib-dsvm-functests-devstack-f21/734c8bd/console.html

Change-Id: I5867ecd57834eece9477aa9ea4b8bdd70e238084
2016-02-16 13:40:01 +11:00
Jenkins
9e16c8c5f1 Merge "Use dnf to cleanup old kernels" 2016-02-15 22:56:12 +00:00
Jenkins
99d9373eca Merge "Fix dpkg element for Ubuntu Xenial" 2016-02-15 20:33:35 +00:00
Antoine Musso
ee58c5b1cc debian-minimal: configurable debootstrap components
The debian-minimal element creates /etc/apt/sources.list solely with
the 'main' component. I need to add 'non-free' and 'contribs'. I tried
to pass them via DIB_DEBIAN_COMPONENTS but it is not recognized.

Make debian-minimal to honor DIB_DEBIAN_COMPONENTS. Note that it is
comma separated for 'debootstrap', so replace commas with spaces to fit
the needs of sources.list.

Example usage:

 DIB_DEBIAN_COMPONENTS='main,non-free,contrib'

Will debootstrap with the three components then when debian-minimal is
realized pre install a sources.list that has:

  deb http://example.org/debian jessie main contrib non-free

Change-Id: I1dca2e8ffd31044a6b441ccb277298601e62f67c
2016-02-12 22:57:58 +01:00
Jenkins
afd2eec60e Merge "Remove eclean-dist as it's not available by default" 2016-02-12 16:31:26 +00:00
Jenkins
e88792a3ed Merge "Increase interface has link retries to 20" 2016-02-12 08:26:09 +00:00
Jenkins
dcf2c4f261 Merge "Reorder the package-uninstall action" 2016-02-12 04:04:14 +00:00
Jenkins
5036ad6baf Merge "Do not remove python-dev from ironic-agent image" 2016-02-12 04:04:08 +00:00
Jenkins
4136479bd0 Merge "Fix startup race with growroot/systemd" 2016-02-12 04:03:55 +00:00
Matthew Thode
0ab0f69ce3 Remove eclean-dist as it's not available by default
Gentoo provides eclean-dist via the gentoolkit package
eclean-dist isn't needed anyway was /usr/portage is removed anyway

Removed redundant package update in cleanup.

Change-Id: Icf4f9ed549b9a6d923448d927d7c42bcf8d6091b
2016-02-12 00:31:27 +00:00
Matthew Thode
22ef95c1a1
Reorder the package-uninstall action
This action has been reordered so cleanup can occur before copy to blockdev
occurs.  Documentation has been added about the ordering of this element in
relation to cleanup actions as well.

Change-Id: I3f9334a3669ee588d7fa7129202c97fa22fdb050
2016-02-11 18:30:05 -06:00
Dmitry Tantsur
1f9e669dbc Do not remove python-dev from ironic-agent image
While it does save 1MiB of space, it might also pull python packages
depending on it. E.g. it makes impossible to install python-hardware
on the IPA image for advanced introspection.

Change-Id: Iab80dde63e6de62a5e45dcf404b4f9f633e50ac3
2016-02-11 13:47:31 +01:00
Jenkins
1cd7ca70d8 Merge "Correct rhel-common for rhel6" 2016-02-11 04:49:35 +00:00
Ian Wienand
349e8db2f7 yum-minimal: pre-install base packages
We were getting some subtle issues in fedora-minimal builds that
turned out to be because /var/run was not a symlink to /run.

Upon further investigation, it turns out that yum is creating a
/var/run directory for it's pid file when it starts working in the
empty chroot (which I verified by stracing it)

---
5905  stat("/home/ubuntu/tmp/dib-tmp/image.Ac4VZZsl/mnt/var/run", 0x7ffddffa0330) = -1 ENOENT (No such file or directory)
5905  mkdir("/home/ubuntu/tmp/dib-tmp/image.Ac4VZZsl/mnt/var/run", 0755) = 0
5905  open("/home/ubuntu/tmp/dib-tmp/image.Ac4VZZsl/mnt/var/run/yum.pid", O_WRONLY|O_CREAT|O_EXCL, 0644) = 6
---

Because this happens *before* we install "filesystem" (the package),
we mess up it's symlinking.

To work-around this, pre-install the trio of base packages (setup,
basesystem, filesystem) with rpm from outside the chroot.

Change-Id: I411b6ec9d91d95d3a0f98e76853086af3b70abe8
2016-02-11 15:42:10 +11:00
Jenkins
536b86fe09 Merge "Mark ironic-discoverd-ramdisk as deprecated in favor of ironic-agent" 2016-02-10 22:48:42 +00:00
Jenkins
910a99c3ca Merge "Fix growroot for Gentoo's openrc" 2016-02-10 21:50:50 +00:00
Colleen Murphy
a9b38d0b60 Fix dpkg element for Ubuntu Xenial
The Ubuntu Xenial cloud server images set the mode of
/var/lib/apt/lists/partial to 700, so when mounted it's inaccessible to
an unprivileged user, resulting in an error:

  find: `/tmp/image.aDQKdkRi/mnt/var/lib/apt/lists/partial': Permission denied

There's no reason an image should come with anything already in
/var/lib/apt/lists/partial, so just avoid trying to descend into that
directory when fixing the apt translations packages.

Change-Id: Id27f0166bfb09d67200f337a5ffff2f2037b7c1c
2016-02-10 12:50:43 -08:00
Jenkins
b13512a7f4 Merge "Only match #!/bin/bash in scripts" 2016-02-10 08:35:09 +00:00
Ian Wienand
63182a02e4 Refactor growroot for debuggabilty
This is a slight refactor I found useful when debugging.  The
udevadmin info query will be visbile in the output of "set -x" which
helps, and is the logs/journal.

We can also reduce some calls by keeping the value and just grepping
with a herefile.

This also does some error checking and bails out if it does not see
what it needs to continue.

Change-Id: I39c4d262f9c5ce53f6b83d95b1363a74834cf2c8
2016-02-10 14:02:28 +11:00
Ian Wienand
9cd35c36e4 Fix startup race with growroot/systemd
Tucked away in systemd-udev-settle.service is the following comment

 # This service can dynamically be pulled-in by legacy services which
 # cannot reliably cope with dynamic device configurations, and
 # wrongfully expect a populated /dev during bootup.

The info that the growroot script is querying is populated via udev,
particularly the blkid bits of [1].  This creates a race-condition
where sometimes udev has been triggered and the rules have applied and
sometimes not.  Obviously in the first case, the root disk is not
grown correctly.

systemd-udev-settle is mostly disabled on distros because it can cause
an increase in boot-time for systems with lots of disks; this is not
our situation so it makes basically no difference.

That said, I will investigate if some systemd people know even better
ways to do this (possibly the service should depend on block .device
targets in systemd, and then filter out and only apply to the root
disk?)

[1] https://github.com/systemd/systemd/blob/master/rules/60-persistent-storage.rules#L66

Change-Id: I453e3afcd953dfc29ab6c42ddc81e940cfa70ee0
2016-02-10 13:47:18 +11:00
Jenkins
a69dd548a7 Merge "Make dkms element depend on dkms package" 2016-02-08 20:17:26 +00:00
Ben Kero
41f503fef3 Replace sfdisk partitioning with parted
A TODO was placed on the partitioning section of the vm element to
replace sfdisk with a saner (and less arcane) way of partitioning. It
suggested parted for replacement. This changeset should reproduce the
same disk label and partition layout as sfdisk, but with less ioctl
errors and version dependency. It will also ensure partition alignment.

Change-Id: I5d8d75131458b73bfb05f80f1bfa7e2970e004b3
2016-02-08 12:15:50 -08:00
Gregory Haynes
6ab5078d0f Switch simple-init to pip-and-virtualenv element
We currently install pip from package in the simple-init element.
We should really allow users to select whether to install pip from
git or package.

Change-Id: Ia5e62b9635af90d81227274a1dd8f20474cdbf73
2016-02-08 16:58:14 +11:00
Ian Wienand
d5155254cb Revert "Skip centos functional testing"
This reverts commit d10871534f.

This is a place-holder so I don't forget to revert this when [1] is
resolved

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1303660

Change-Id: I18c8aa6e7233f74c680d23b573fe9c3817a5779a
2016-02-08 15:55:29 +11:00
Ian Wienand
cb0e0e903d Use dnf to cleanup old kernels
As described in the comment, there is a dnf equivalent of this command
that doesn't require us installing yum-utils (which drags in yum on
dnf-only systems such as f23)

This is a small consequence to this -- due to us not installing
yum-utils some installs will now be completely yum free.  This causes
a breakage in ironic-agent 99-remove-extra-packages where we remove
the yum package.  There is a long-standing bug/feature where missing
packages in a group of packages do not cause yum/dnf to exit with
failure, but uninstalling a single package will.  Because we have made
the systems yum-free, the uninstall of yum can fail in this corner
case.

It has always been like this, so I'm in favour of the "ain't broke"
approach.  To work-around this, I have just put yum into the existing
list of packages to be cleaned up.  I have added a note to the yum
installer taking note of this behaviour for future reference.

Change-Id: I8bbdc07ccdb89a105b4fc70d5a215077c42fcd03
2016-02-08 14:20:56 +11:00
Moshe Levi
a1d32c6d51 Increase interface has link retries to 20
InfiniBand interface takes more time to bring up then
Ethernet interface. This patch just increase the retries
to 20 times, to make it work for InfiniBand as well.

Change-Id: I5c4842696207885552413ea2d053f2e90bd6803c
2016-02-07 16:28:52 +02:00
Matthew Thode
9878469a53
Fix growroot for Gentoo's openrc
Let dib-init-system's postinstall handle enabling init scripts.

Change-Id: I7b0d235327021b7f478cdb9715d533b444843699
2016-02-04 23:41:45 -06:00
Jenkins
5ee1cbac4a Merge "Add support for OpenRC to dib-init-system" 2016-02-05 04:15:53 +00:00
Jenkins
50f7b00a54 Merge "Add support for gentoo to simple-init" 2016-02-04 08:00:47 +00:00
Jenkins
1dcd65fb9e Merge "Print unparsable file in pkg-map" 2016-02-04 04:47:20 +00:00
Matthew Thode
265b31e6b5
Add support for OpenRC to dib-init-system
Adds a post-install function that enables installed initscripts,
as that is not done by default in gentoo.

Change-Id: I04e8d506ddcbefa8a983dd31ad16df5e13cb26e7
Closes-Bug: 1539276
2016-02-02 17:23:35 -06:00
Jenkins
4526cf23ec Merge "Fix Gentoo hardened support" 2016-02-02 20:49:30 +00:00
Jenkins
66184b7b40 Merge "Add new cloud-init element" 2016-02-02 20:49:03 +00:00
Jenkins
cd3c704fbf Merge "Fix package-installs for python3" 2016-02-02 03:16:12 +00:00
Jenkins
893b452b11 Merge "Add Gentoo support to growroot" 2016-02-02 03:15:22 +00:00
Jenkins
4cb6bed971 Merge "Add support for Gentoo to source-repositories" 2016-02-02 03:15:16 +00:00
Matthew Thode
a2d4af9541
Add new cloud-init element
Allows for composable builds from a minimal source.

Change-Id: I16f560ca4b1737c0928f3f3b15ea2580df820e97
Closes-Bug: 1539272
2016-02-01 21:03:03 -06:00
Matthew Thode
01fce7b70c
Fix Gentoo hardened support
This checks the profile, if it has hardened in it's name it needs xattr support
unfortunately xattr support cannot yet be relied on everywhere, so it needs to
be disabled for hardened profile builds to correctly pax-mark.

Change-Id: I7fb855249a9e6c9b6497ab5061b4ea3c014f5081
Closes-Bug: 1537177
2016-02-01 20:56:37 -06:00
Ian Wienand
d10871534f Skip centos functional testing
Due to upstream bug [1] there are uninstallable packages which mean
our functional tests don't work.  We will revert this when things are
working upstream.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1303660

Change-Id: I93c2990472e88ab3e5ff14db56b4ff1b4dd965ef
2016-02-02 10:34:18 +11:00
Matthew Thode
c6ef183975 Fix package-installs for python3
subprocess.check_call() returns a byte-string which needs to be turned
into a unicode string for python3 compatability.

Also some minor refactoring while we're here.

Closes-Bug: 1536462
Change-Id: Icd957bc4d93ccad94b1246ad62e6e02ee14d9ca5
2016-02-01 22:22:15 +00:00
Matthew Thode
4480b5edef Add support for gentoo to simple-init
Needed to remove interface config

Depends-On: Id1487bc82ee4b03aa57258f71f48ca7d377afc47
Change-Id: Iaa1a16b0f65c088f03c9b440ef16427e93ec9b03
Closes-Bug: 1537453
2016-02-01 22:14:48 +00:00
Dmitry Tantsur
3d133a3eaf Mark ironic-discoverd-ramdisk as deprecated in favor of ironic-agent
Also fix no longer correct parts of its README.

Change-Id: I00cbe5b4f65f9909ae63b7eb4bbac0ba82c145d1
2016-02-01 16:47:37 +01:00
Jenkins
48ae802c5a Merge "Remove zero length files" 2016-02-01 05:46:44 +00:00
Matthew Thode
92de91476e Add Gentoo support to growroot
Add missing growroot initscript and pkg-map entries for Gentoo.

growpart was added to Gentoo with [1]

Update the readme to reflect reality too (fedora added with
I5630dc638f85b1e80795826ef36a306632075460)

[1] https://packages.gentoo.org/packages/sys-fs/growpart

Closes-Bug: #1539273
Change-Id: I29056c7297489ec04f37757dbe33976901eceb49
2016-02-01 10:55:03 +11:00
Matthew Thode
2afb99db10 Add support for Gentoo to source-repositories
As mentioned in package-installs.yaml, git is a transitive dependency
for pbr in this element.  Add pkg-map for the Gentoo package.

Change-Id: I7f2fe1663152ea66b941594e86f1da93ddd21677
Closes-Bug: 1539278
2016-02-01 10:02:32 +11:00
Ian Wienand
d8abe72537 Only match #!/bin/bash in scripts
Our dib-lint checking is only considering scripts with #!/bin/bash.
While there's nothing really wrong with some other shebang line like
"#!/usr/bin/env bash" let's keep things consistent.

We can use the same regex match to reduce a few forks in the main
checking.

Also a minor cleanup to the file matching

Change-Id: I609721b2671e704ea26075dad7e5b39a8b858f6b
2016-01-29 15:49:10 +11:00
Jenkins
4b1b813a68 Merge "Fix debian-minimal image building" 2016-01-29 04:32:24 +00:00
Jenkins
c31a59a2c9 Merge "Revert "Use pip 7 for ironic"" 2016-01-28 23:54:36 +00:00
Igor Belikov
78278405ce Fix debian-minimal image building
'locales' package gets installed before '12-debian-locale-gen' is executed
and generates effectively empty /etc/locales.gen in debian, which makes
dpkg-reconfigure call to locales ignore the values set by
debconf-set-selections.
* Remove /etc/locale.gen generated by 'locales' installation to ensure
  proper locales generation on debian images
* Remove 'locales-all' package installation from debian element since
  it's not needed anymore to build the image and cosnumes additional
  ~120MB of space
* Remove unused 'package-installs' dependency from debian-minimal
  element

Change-Id: Ic39ba2b5ceb5018efb75742547b2babf80827e56
Closes-Bug: #1452400
2016-01-28 16:41:35 +03:00
Ian Wienand
9305ea4b6d Add systemd/fedora support to growroot
Add systemd/fedora support to growroot element.  This involves
installing the correct packages, shipping the systemd service file and
ensuring it is enabled.

Note the required growfs/resize packages for Ubuntu/Debian are
installed in other places.  This is probably a bug in that path, but I
have not addressed that here.

I have tested this with a F23 build with all openstack-infra elements,
uploaded to RAX, and it boots and resizes the main file-system.

Change-Id: I5630dc638f85b1e80795826ef36a306632075460
2016-01-25 17:40:52 +11:00
Ben Nemec
46ef8e3f93 Revert "Use pip 7 for ironic"
Supposedly pip 8.0.1 fixes this.  Trying a revert to verify.

This reverts commit f068230a5f.

Change-Id: I35b88668bf16ad6205ec678f87e3981aebab32e9
2016-01-22 15:34:46 +00:00
Jenkins
d8b3acf816 Merge "Force dib-python symlink creation" 2016-01-22 01:04:47 +00:00
Jenkins
d79ecfd076 Merge "Use pip 7 for ironic" 2016-01-21 23:05:32 +00:00
Ben Nemec
1b0ca07d86 Force dib-python symlink creation
The undercloud actually has dib run twice on it - once to create
the instack image, and again when we run instack itself.  The
first run creates the dib-python symlink, and the second blows up
because the link already exists.  Force the link creation so the
script is idempotent.

Change-Id: I78f9e6f5afcf8ebe6d7911a7a434525ba7c737cf
2016-01-21 13:25:33 -06:00
Paul Belanger
b7f6527a0e Remove zero length files
This is to aid with Fedora packaging, since rpmlint complains about
including empty files.

Change-Id: I4ad867cd21304880a571e46805ab56044542400c
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2016-01-20 19:08:54 -05:00
Gregory Haynes
f068230a5f Use pip 7 for ironic
pip8 cannot install the ironic requirements due to
https://github.com/pypa/pip/issues/3384.

Change-Id: Ic0c70baca83b91cb55ddf4f787c4ff25aaefc062
2016-01-20 18:02:39 +00:00
Jenkins
9ad78ced3c Merge "Add pkg-map for redhat" 2016-01-19 21:37:06 +00:00
Jenkins
af85ea066a Merge "Run package-intalls with py3k if we must" 2016-01-19 05:58:58 +00:00
Jenkins
e52fbdc598 Merge "Revert "Fix discoverd bug when dmidecode reports GB"" 2016-01-17 14:26:58 +00:00
Abel Lopez
fe6e78147b Add pkg-map for redhat
The bootloader element lacks the mapping for grub-pc and extlinux
This adds it.

Change-Id: Ic7b82903f02faaab143d2bd682876bf4853fd90d
Closes-Bug: 1534387
2016-01-15 12:06:16 -08:00
Dmitry Tantsur
994e78209c Revert "Fix discoverd bug when dmidecode reports GB"
While the patch looks sane and the change worked locally,
it has broke the ironic-inspector gate. As we're close
to deprecating the DIB ramdisk in favor of IPA, I suggest
reverting it.

This reverts commit 802f14862c.

Change-Id: I0525e545cb2fe8ce184312a2f9bbe3763904f61a
Closes-Bug: #1534648
2016-01-15 16:07:58 +00:00
Ian Wienand
c110a56ea9 Print unparsable file in pkg-map
As you can see in the CI failures for
Ib11b9df84b593ab25232729a570c812f1b4b8774, you can not see what file
is causing the problems when the parser fails.  Add a print, but raise
the error as it contains the cause.

Change-Id: I127ff7d57d2d898969195464c6e774d496e872e6
2016-01-14 16:21:25 +11:00
Gregory Haynes
cd9fdf05e9 Fill out bootloader pkg-map
Our bootloader install fails on non-gentoo builds due to missing pkg-map
for grub-pc. This map should really live in the bootloader element, so
move it there and fill it out.

Change-Id: Ib11b9df84b593ab25232729a570c812f1b4b8774
2016-01-14 15:31:50 +11:00
Jenkins
d5da803ff8 Merge "Prune old branches when updating cache" 2016-01-14 00:03:28 +00:00
Matthew Thode
176ae7bff6
Initial add of gentoo support for diskimage-builder
uses upstream's stage4 images, includes all the needed bells and
whistles for openstack on kvm.

Change-Id: Ibca43173c30c2a74a73a2e2d9dd6d6d832c62694
Closes-Bug: 1530911
2016-01-12 19:15:00 -06:00
Jenkins
023cfc3541 Merge "Add pip-and-virtualenv element" 2016-01-12 10:37:11 +00:00
Jenkins
8903a7bc65 Merge "deploy-ironic: Fix syntax error when checking for root device hints" 2016-01-12 00:55:49 +00:00
Ian Wienand
4573d0d930 Prune old branches when updating cache
dib builds are failing in nodepool at the moment trying to download a
really old Fedora image.  What happens is that the devstack cache
script [1] goes through all the branches of the devstack checkout and
uses the inbuilt image-list tool [2] to get its list of images to
pre-download.  One of the old Fedora images required on the retired
branches has now gone, so the build halts with a 404 trying to get it.

Thus make sure we do a --prune when we fetch changes so that old
branches are removed from the cache.

[1] https://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/elements/cache-devstack/extra-data.d/55-cache-devstack-repos
[2] https://git.openstack.org/cgit/openstack-dev/devstack/tree/tools/image_list.sh

Change-Id: Ieb6a6e9f55bd93f63c3d0a71828c276c2d02e1b9
2016-01-12 11:25:43 +11:00
Abel Lopez
71bd8b3a33 Correct rhel-common for rhel6
Hardcoding subscription-manager to use rhel-7-server-rpms causes
users building rhel6 with Satellite6 to fail. This setting cannot
be overridden with environment variables, therefore needs to be
smarter. Setting RHEL_MAJ_VER in the rhel/rhel7 environment fixes.

Change-Id: Ifbd88bc76ef8b38a739272ba6e045a12849d68df
Closes-Bug: 1404364
2016-01-11 12:03:45 -08:00
Jenkins
14560600a6 Merge "Fix discoverd bug when dmidecode reports GB" 2016-01-08 01:22:54 +00:00
Lucas Alvares Gomes
5064026b73 deploy-ironic: Fix syntax error when checking for root device hints
This patch is fixing a syntax error in the 70-ironic-root-device init
script for the deploy-ironic element.

Change-Id: I767486ca5893605720fba41bee3af72725a26377
Closes-Bug: #1531835
2016-01-07 12:06:19 +00:00
Gregory Haynes
db3ee03672 Add pip-and-virtualenv element
This element allows installation of pip and virtualenv from either
distro packages or git.

Change-Id: Id294f0936c8fef8a3b27a415bfcc93b3f327e104
Depends-On: I731cc8a0f5bfeda8f17a78c33b9f44062323a361
2016-01-06 18:39:19 +00:00
Gregory Haynes
d1e32f80a6 Run package-intalls with py3k if we must
Use dib-python to run package-installs using the provided python
version. Automatically detect the python version for our
package-installs-squash since that runs outside the chroot.

Change-Id: I926022bcf8cbcd81b051026ffd5d6477650045ad
2016-01-04 23:37:30 +00:00
Ryan Hallisey
2b28993fb8 The mirror for installing epel is timing out
Fedora has changed the location of epel, shorting the link
from 'download.fedoraproject.org' to 'dl.fedoraproject.org'.
This change updates the epel mirror to prevent it from timing
out.

Change-Id: I87090282a2f5f757495daec6ad14123b436b1aa0
2016-01-04 07:45:34 -05:00
Gregory Haynes
09888ebd5b Make dkms element depend on dkms package
It currently does not and it is, obviously, a needed dependency.

Change-Id: I0c94573bd0878a84fccb3c2675c0bcaeeb5088fb
2015-12-26 22:07:19 +00:00
Joey D
802f14862c Fix discoverd bug when dmidecode reports GB
This fix uses dmidecode and awk to simply multiply by 1024 when
the value is represented in GB, otherwise it returns the given
value.  I should note that I've only observered this occurence
on "some" SuperMicro Hardware

Closes-Bug: #1486689

Change-Id: I352b1891326f72af3a56c7bbe8b7f3c422169404
2015-12-22 16:34:41 -06:00
Jenkins
59456684e7 Merge "Extend root device hints for different types of WWN" 2015-12-22 22:22:42 +00:00
Jenkins
7b20224cd6 Merge "Don't print trace unless trace is on in pkg-map" 2015-12-22 22:02:09 +00:00
Jenkins
3027093644 Merge "Package installs defaults to tracing off" 2015-12-22 22:02:03 +00:00
Jenkins
96d56efccc Merge "Fix growroot device detection" 2015-12-22 22:01:57 +00:00
Jenkins
b0eff0df07 Merge "yum-minimal : install selinux policy packages" 2015-12-22 20:16:11 +00:00
Jenkins
d450751617 Merge "yum-minimal: leave behind dummy /etc/resolv.conf" 2015-12-22 20:16:05 +00:00
Ian Wienand
fd2f55ee41 yum-minimal : install selinux policy packages
Install selinux policy packages as part of the base-installs.  selinux
is part of the base-system and the kernel boots by default in selinux
mode.

Without both of these, we can get in a situation where later scripts
(particuarly, some of the infra scripts) might install systemd-policy
without a base policy (targeted), leading to a messed up situation
where systemd will halt during boot due to missing policy files.

Change-Id: I6bf156304d1134fb328fba9b12dc364701b13696
2015-12-22 08:45:20 +11:00
Ian Wienand
1f499360fc yum-minimal: do not configure eth0 & eth1 for DHCP automatically
Add an environment variable to control the creation of eth0/1
interface enablement scripts.

With a tool such as glean, the presence of these scripts will indicate
the interface is configured and configuration-drive settings will not
be applied.  This means in a non-dhcp situation like on Rackspace,
network is broken.

On Fedora, where later systemd provides "predictable network interface
names" [1] eth0 & eth1 ironically aren't predictable so this just
confuses things.  You really need cloud-init or glean or something to
bring up your interfaces in a sane fashion.

This maintains the status-quo on centos-minimal, but disables creation
for fedora-minimal.

[1] http://www.freedesktop.org/wiki/Software/systemd/PredictableNetworkInterfaceNames/

Change-Id: I3f1ffeb6de3b1f952292a144efab9554f7f99a5f
2015-12-22 08:39:04 +11:00
Ian Wienand
5f3855f6f5 yum-minimal: leave behind dummy /etc/resolv.conf
As described in the comment, systemd will create a broken
/etc/resolv.conf link if there is no file in the base-image (as you
can read in the bug, it is debated if this is a bug or a feature).

The solution is to leave a dummy /etc/resolv.conf file in the image.
Whatever network manager you choose (NetworkManager, glean,
cloud-config, etc) will overwrite this anyway.

It's just that some tools, such as dhclient, get confused with the
broken symlink.  This affects you if you're using glean to configure
the network in a DHCP situation, for example -- dhclient won't
configure nameservers and everything goes to heck.

Change-Id: I734834d03e7fdb13f9ab2e86f877b07bf4a84ff9
2015-12-21 15:28:52 +11:00
Gregory Haynes
858fc3db30 Fix growroot device detection
We are incorrectly detecting major/minor device numbers for the growroot
rootfs. This can also be simplified by querying udev for partition
information.

Change-Id: I68059bf11f2563872f6b4d0e23fa09a15de980a8
2015-12-20 08:23:54 +00:00
Gregory Haynes
40799839e0 Package installs defaults to tracing off
There isn't a good reason we should be defaulting to trace on for these
scripts.

Change-Id: I1b9ac9388f51a49e6912d800914a0e0f2faec604
2015-12-20 00:03:51 +00:00
Gregory Haynes
6def86562a Don't print trace unless trace is on in pkg-map
The detection logic in pkg-map for DIB_DEBUG_TRACE assumes that this
variable being unset means tracing is on, when in fact this means
tracing is off.

Change-Id: I584a634c57bbe03e26a6ee94cef473e634616885
2015-12-19 23:50:15 +00:00
Jenkins
3a110bf96f Merge "Split vm and bootloader elements" 2015-12-16 16:22:56 +00:00
Jenkins
a72d59c862 Merge "Fix fedora-minimal on CentOS builds" 2015-12-16 01:25:13 +00:00
Jenkins
134d38f74c Merge "Allow grub2 to build with opensuse" 2015-12-16 01:25:10 +00:00
Jenkins
d716893e30 Merge "Add dib-python element" 2015-12-15 23:24:25 +00:00
Yolanda Robla
c54b335fcc Split vm and bootloader elements
In order to add more flexibility to the vm and bootloader
elements, split the functionality in two different ones, and
make vm depend on bootloader element.
This will allow to construct more elements that depend on
bootloader, and develop both elements independently.

Change-Id: Iad2503b7b8fe53b768a3bc79e4cb839700fbd747
2015-12-15 20:49:03 +01:00
Jenkins
be5b669af1 Merge "Add a new element hpdsa" 2015-12-10 21:46:06 +00:00
Jenkins
3ebd0ba902 Merge "Follow up patch for 25d3ee547176528e86d42eb026c99a134dff9452" 2015-12-10 21:25:03 +00:00
Jenkins
aabf5b6ad7 Merge "Add proliant-tools element" 2015-12-10 21:24:27 +00:00
Nisha Agarwal
4509e6f826 Add a new element hpdsa
This element enables creation of Ubuntu deploy ramdisk and
user images which could be used to deploy the HP Proliant
Servers with Dynamic Smart Array Controllers. Without this driver
the disk with the Dynamic Smart Array Controller is
not visible to the ramdisk.

Closes bug: #1492803

Change-Id: Ibb3b298cd379cd7333279484df6ae30e9d7f6aaa
2015-12-10 20:12:14 +00:00
Gregory Haynes
3afbeeaf4e Add dib-python element
Creating an element which we can use in #! lines to refer to either
python2 or python3 depending on what it available.

Change-Id: Ic47e18ad21c33ab9f0d11c04260a33725aeee814
2015-12-10 19:39:38 +00:00
Robert Li
1b8a53ff3a Allow grub2 to build with opensuse
Specify grub2 in grub2 element pkg-map

Change-Id: Ia8bc1cbe11b35802f5c979ab4bb03e4e75077e83
Closes-Bug: #1519209
2015-12-10 17:29:52 +00:00
Jenkins
a95b96f57f Merge "Use ironic-agent for source-repositories" 2015-12-09 23:18:58 +00:00
Jenkins
da7b125fea Merge "Add kmod to package-installs of ironic-agent" 2015-12-09 16:22:23 +00:00
Dmitry Tantsur
901344ef7c Add kmod to package-installs of ironic-agent
The modprobe utility is required by the rtslib package (iSCSI Linux-IO).
It will also be required for inspection.

Change-Id: I6760c86160d1ceba45aedde62597a711bcb4543d
2015-12-09 14:27:21 +00:00
Jenkins
a9c8e6d55d Merge "Load the 8021q kernel module in simple-init" 2015-12-09 01:31:40 +00:00
Jenkins
93a54e32b9 Merge "Fix grub-efi-amd64-signed install failure" 2015-12-09 01:07:35 +00:00
Colleen Murphy
808e8f8bdc Load the 8021q kernel module in simple-init
Vlan support was recently added to glean. However, if the 8021q module
is not loaded, glean will fail to bring up a tagged interfaced defined
in /etc/network/interfaces.d/. Manually attempting to bring up the
interface results in an error[1]. This patch ensures that the 8021q
module is loaded so that tagged interfaces can be brought up at boot.

[1] http://paste.openstack.org/show/480027/

Change-Id: I15d805c07d4b5e1161d831f0393d027e4325137f
2015-12-08 12:59:06 -08:00
Paul Belanger
2be1bdc701 Add openssh-server package-install to local-config
Since we are modifing SSH keys, it should be safe to assume
openssh-server should be installed too.

Change-Id: I17ff05642bb2f0868d4c17819cd91b179068399a
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2015-12-07 15:23:00 -05:00
Jane.zhang
7eb70becd3 Fix grub-efi-amd64-signed install failure
When build ubuntu iso image, it will install grub-efi-amd64-signed
and grub-efi-amd64 packages. Both of the postinst script will try
to find root device and install grub which will definitely fail in
such a chroot environment.
So the workaround is to skip error and remove postinst script.
And confirm the package be installed successfully at last.

Change-Id: Ie0aecb212b22362046db55b5ad8c64c3211c28e5
Closes-Bug: #1491280
Co-Authored-By: Jane.zhang <jian.zhang8@hpe.com>
2015-12-01 23:53:35 -08:00
Ian Wienand
ce781fbbc4 Fix fedora-minimal on CentOS builds
As described in the comments, CentOS overrides the "distroverpkg"
variable in yum.conf.  This is the package that yum queries to
establish the value of the $releasever variable.  On other platforms,
this defaults to "redhat-release" (which "fedora-release" provides) so
everything works.  It is only when the base-system "distroverpkg"
refers to a package not in the chroot we hit the issue.

We can avoid this by setting the releasever variable via the
commandline.

Change-Id: I231c3277960992cd479b8aff7838f246397936f2
2015-12-02 12:16:43 +11:00
Jenkins
3a3e4e4787 Merge "Add dynamic-login element" 2015-12-01 14:27:01 +00:00
Lucas Alvares Gomes
40197fa7f1 Follow up patch for 25d3ee5471
This patch is a follow up patch fixing some nits left by the review
25d3ee5471.

It does:

* Fix the README file to say that the password *must* be encrypted and
  the option values *must* be quoted

* Adds Type=oneshot in the upstart service config file so that upstart
  will not try to restart the service over and over.

* Enable setu, sete and setpipefail in the dynamic-login script

Change-Id: Iee5d75daef24469ccf47ca12de6ead37bf9d8d6f
2015-12-01 14:11:40 +00:00
Gregory Haynes
6857452a62 Add centos7 test
We should test that centos7 builds work

Change-Id: I2c7a69c09f6bfa83eed7eabb1193acbe65f8b420
2015-11-25 17:35:29 +00:00
Paul Belanger
35b363698b Add DIB_LOCAL_CONFIG_USERNAME to local-config
Allow a user to override the username on where .ssh/authorized_keys is
installed.

Change-Id: I030d5a89260aed8b23a35c4cdc2d67629934b076
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2015-11-24 10:42:14 -05:00
Dan Prince
d9dc8f1334 Use ironic-agent for source-repositories
I recently built a ramdisk for IPA and was confused by
the fact that the source-repositories name did not
match the element name. (this is a convention,
confusing when they don't match but certainly not
required).

This patch makes it so you can use DIB_REPOREF_ironic_agent to
customize the IPA ramdisk sources when building ramdisks.

For backwards compat if DIB_REPOREF_agent is set it automatically
sets the new DIB_REPOREF_ironic_agent to that value as well.

Change-Id: I082d989d0d85601f5984dc7c3767b8d66a3d5438
2015-11-22 15:03:31 -05:00
Lucas Alvares Gomes
25d3ee5471 Add dynamic-login element
Troubleshooting an image can be quite hard, specially if you can not get
a prompt you can enter commands to find out what went wrong. By default,
the images (specially ramdisks) doesn't have any SSH key or password for
any user. Of course one could use the ``devuser`` element to generate
an image with SSH keys and user/password in the image but that would be
a massive security hole and very it's discouraged to run in production
with a ramdisk like that.

This commit is adding a new element called dynamic-login, which inserts
a helper script into the image to allow operators to inject a SSH key
and/or change the root password dynamically when it boots via parameters
in the kernel command line.

Those parameters are:

sshkey = If the operator append sshkey="$PUBLIC_SSH_KEY" to the kernel
command line on boot, the helper script will append this key to the root
user authorized_keys.

rootpwd = If the operator append rootpwd="$ENCRYPTED_PASSWORD" to the
kernel command line on boot, the helper script will set the root password
to the one specified by this option. Note that this password should be
an encrypted password.

Change-Id: I6b87a1b90163d79745f30dfacd37516051fa0aea
2015-11-20 17:02:54 +00:00
Ian Wienand
1d476dd994 Remove fedora-minimal/install.d/99-ramdisk
When the kernel gets installed on Fedora, the rpm post scripts call
"/bin/kernel-install" [1] to install it.  This is a script provided by
systemd.

However, in [2], Fedora ships a patch to kernel-install that makes a
call-out to /sbin/new-kernel-pkg -- the install script provided by
grubby [3]

Without grubby installed, systemd's kernel-install script goes off and
runs dracut plugins directly [4], which eventually creates the initrd.
For reasons that are not clearly explained, the initrd will end up in
a a "machine-id" sub-directory of /boot (possibly, so you can symlink
it?).  It is also called "initrd", even though it's an initramfs, for
historical reasons in dracut I think.

It is at this point that I think 99-ramdisk has been written to move
the generated initrd file back into /boot.  Later on, when we build
the image, we run grub-install and it picks up the kernel and the
initrd and installs everything.

grubby's new-kernel-pkg [6] it's very similar -- it uses dracut to
make the initramfs ... but in this case it is put in /boot and is
actually called initramfs.

The subtle change that led me down this path is that dracut has been
modified to have a "Recommends" for grubby for >F22 [7].  After
discussing this change with the author, it turns out it was *always*
intended to use the grubby-based kernel install scripts for Fedora --
our builds have been incorrect in not including the package.  The
author got sick of people removing the package and making unbootable
systems, hence the change.

Thus this removes the workarounds in 99-ramdisk and replace it with an
install of the grubby package.  grubby's kernel install script will
put the kernel & generated initramfs in /boot, and it will be
installed correctly via the usual grub install later when we build the
disk image.

I have built F22 & F23 fedora-minimal images with this and they boot.

[1] http://pkgs.fedoraproject.org/cgit/kernel.git/tree/kernel.spec#n1832
[2] http://pkgs.fedoraproject.org/cgit/systemd.git/tree/kernel-install-grubby.patch
[3] http://linux.die.net/man/8/new-kernel-pkg
[4] https://github.com/haraldh/dracut/blob/master/50-dracut.install
[5] 81516adcb7
[6] https://github.com/rhinstaller/grubby/blob/master/new-kernel-pkg
[7] 47ff68e78b

Change-Id: I1a6e45d04755515286b3d49f8280c16b527e2f48
2015-11-19 21:03:45 +11:00
Gregory Haynes
e23b087505 Remove cloud-initramfs-growroot package
This package seems to be broken and isnt useful for recent releases
(where the rootfs can be resized online). Therefore this should be
optional so people can use things like the growroot element.

Change-Id: I6e3c8d095d9fc188094f3b8811f06be0847ef08c
2015-11-17 23:00:17 +00:00
Lucas Alvares Gomes
ff988ac45c Extend root device hints for different types of WWN
This patch is extending the root device hints to also look at
ID_WWN_WITH_EXTENSION and ID_WWN_VENDOR_EXTENSION from udev.

Prior to this patch the bash ramdisk only cared about ID_WWN but in some
systems in some platforms with a RAID controller, this ID can be same
even if they are different disks (see bug 1516641).

Related-Bug: #1516641
Change-Id: I45b3910d03d164d880b32169b91e94e88812e183
2015-11-17 22:42:44 +00:00
Jenkins
2ff566b80a Merge "Selectively prune /root for ironic-agent ramdisk" 2015-11-13 03:28:13 +00:00
Jenkins
5e571d9f44 Merge "Fixup RPM db path when building Fedora on Ubuntu" 2015-11-10 11:03:22 +00:00
Ian Wienand
3f3850aa0f Fixup RPM db path when building Fedora on Ubuntu
On Debian/Ubuntu installs of RPM, /usr/lib/rpm/macros sets

  %_dbpath  %(echo $HOME/.rpmdb)

which makes quite a bit of sense, because RPM is not the system
packager and thus RPM is setup to install things into a hierarchy in
the users homedir.

However, this messes things up when building a Fedora chroot on an
Ubuntu platform.

We use RPM & yum from the base-system to bootstrap the Fedora chroot.
While both obey --root flags, they still pick up the %_dbpath macro
and so end up creating the RPM database in <chroot>/home/user/.rpmdb

After we have bootstrapped yum/dnf, we execute further installation
commands from inside the chroot -- where we now have the Fedora
version of /usr/lib/rpm/macros and hence have _dbpath set to
/var/lib/rpm -- except there is no rpm database there.

Should anyone be finding this in the future, the actual issue that
appears is

  $ sudo chroot /opt/dib_tmp/image.b6B5S3f6/mnt dnf makecache
   Error: Failed to synchronize cache for repo 'fedora' from \
    'https://mirrors.fedoraproject.org/metalink?repo=fedora-$releasever&arch=x86_64': \
    Cannot prepare internal mirrorlist: file "repomd.xml" was not found in metalink

Note the issue there is that $releasever is not expanded, because the
rpmdb where this info is kept is not populated.

The trick is to make sure we override this value when using the host
rpm/yum to setup the chroot.  The bare rpm calls, which we use to
install the repos, have a --dbpath argument where we can override
this.  yum does not however, so we override this in the global
~/.rpmmacros while we are installing the packaging tools and
dependencies into the chroot.

Copious comments are included, because this is super-confusing.

Change-Id: I20801150ea02d1c64f118eb969fb2aec473476f7
2015-11-10 08:54:44 +00:00
Ian Wienand
8ee21cb9fd Remove unused RELEASE_RPMS variable
It was noticed during reviews of
Ic7aa8cbe13e4347b447e84bb9c12483a4e125228 these are unused

Change-Id: I9e0fa9d3e4864e54c6fe23f8b6e781e8d5d24bda
2015-11-10 07:17:52 +00:00
Jenkins
e90be5a595 Merge "Fix fedora-minimal kernel-install on older platforms" 2015-11-10 05:14:28 +00:00