Commit Graph

1746 Commits

Author SHA1 Message Date
Ian Wienand
5d23abb338 centos-minimal: can be used with base
I'm not sure why this comment is here.  base works with centos-minimal
(we changed it in I2956aaa49ba3137a799f97e0983ab4b7c93a0a0c) and we've
got images deployed with such a configuration.

Update the comment

Change-Id: I3207f87dc29280183c0960ea863533a4d441081c
2016-05-16 13:36:55 +10:00
Jenkins
d253a94187 Merge "Do not remove sudo in ironic-agent" 2016-05-16 01:41:08 +00:00
vmud213
45628993e8 Do not remove sudo in ironic-agent
"ironic-agent" element is currently removing sudo, which breaks other
elements such as devuser.  There appears to be no security or other
reason to do this, it's just the way it has always been.  Leave sudo
in as it is considered part of the base cloud images.

Change-Id: Ida9b1885f745146071e4b2d85ae59341ac85d5c8
Closes-Bug: #1572486
2016-05-16 10:39:04 +10:00
Andreas Florath
92355004d9 Fix apt-sources configuration for debian-minimal
The apt-sources element did not work with debian-minimal, because
the later one overwrote the /etc/apt/sources.list file created by
the apt-sources element.

Two changes were made:
o the debian-minimal uses now files inside the /etc/apt/sources.list.d
  directory.  Therefore there is no possibilty for clashes between those
  two elements any more.
o instead of only adding backports, also the updates and the security
  repository is added by default which gives perfect initial
  configuration for a stable system.
  If you want to use local mirrors with other naming schemas or an
  unstable tree, there is the possibility to fully specify the
  repositories.

Change-Id: I69dbaa34be3db3d667e6bd8450ef4ce04a751c70
Signed-off-by: Andreas Florath <andreas@florath.net>
2016-05-13 09:26:49 +02:00
Jenkins
77d03556df Merge "Add EPEL as requirement of centos-minimal" 2016-05-13 03:18:58 +00:00
Jenkins
0a3d9999d6 Merge "Add install-bin element" 2016-05-13 03:00:25 +00:00
Jenkins
36a86afdad Merge "Install proliantutils in IPA's virtualenv" 2016-05-13 01:54:06 +00:00
Gregory Haynes
642f906476 Add install-bin element
The various distro elements include a copy of this script which installs
all files in the bin/ dir of the copied element hooks in to
/usr/local/bin. Lets share code rather than repeating ourselves.

Change-Id: I354382f8c42ede2e9b8c548b7df8367c03e6836e
2016-05-13 11:52:23 +10:00
Jenkins
8cb8806227 Merge "yum-minimal : better cleanup of initial yum failure" 2016-05-13 01:04:33 +00:00
Jenkins
88a9365816 Merge "yum-minimal: strip locale archive" 2016-05-13 01:04:27 +00:00
Jenkins
5657852b22 Merge "Fix up EPEL element" 2016-05-13 01:04:24 +00:00
Ian Wienand
672705831f Add a best-effort sudo safety check
As motivation for this; we have had two breakouts of dib in recent
memory.  One was a failure to unmount through symlinks in the core
code (I335316019ef948758392b03e91f9869102a472b9) and the other was
removing host keys on the build-system
(Ib01d71ff9415a0ae04d963f6e380aab9ac2260ce).

For the most part, dib runs unprivileged.  Bits of the core code are
hopefully well tested (modulo bugs like the first one!).  We give free
reign inside the chroot (although there is still some potential there
for adverse external affects via bind mounts).  Where we could be a
bit safer (and could have prevented at least the second of these
breakouts) is with some better checking that the "sudo" calls
*outside* the chroot at least looked sane.

This adds a basic check that we're using chroot or image paths when
calling sudo in those parts of elements that run *outside* the chroot.
Various files are updated to accomodate this check; mostly by just
ignoring it for existing code (I have not audited these calls).

Nobody is pretending this type of checking makes dib magically safe,
or removes the issues with it needing to do things as root during the
build.  But this can help find egregious errors like the key removal.

Change-Id: I161a5aea1d29dcdc7236f70d372c53246ec73749
2016-05-09 15:41:38 +10:00
Colleen Murphy
b5f51322a3 Fix OpenSUSE support
The dhcp-all-interfaces and simple-init elements did not have the ISC
DHCP Client package mapped for OpenSUSE, which caused DIB to fail with
"'isc-dhcp-client' not found in package names. Trying capabilities."

Similarly, the bootloader element did not have the grub-pc package
properly mapped for OpenSuse, which caused DIB to fail with "Package
'grub-pc' not found.".

This patch adds the package mappings for these elements so that the
opensuse element can be created and booted successfully.

Change-Id: Ife478158fec3a95de73a9206b38dcc6511d56cc8
2016-05-03 22:23:51 -07:00
Jenkins
83b607557e Merge "Remove cloud-initramfs-growroot package" 2016-05-03 13:24:53 +00:00
Nisha Agarwal
9d397d2568 Install proliantutils in IPA's virtualenv
The proliant-tools element helps to do RAID
configuration in ironic for HPE servers.
This fix proposes to install the proliantutils
in ironic-python-agent's virtualenv created
using ironic-agent element.

Closes-Bug: 1563648
Change-Id: If63c725a42740ab244a2b4004797cba09d0f154e
2016-05-02 01:56:26 -07:00
Ian Wienand
11128b0673 Use generic "dhcp-client" name
Every platform has a different name for their DHCP client, so use a
generic name "dhcp-client" in the package name and let everyone choose
their sub-name.  This also brings some consistency across simple-init
& dhcp-all-interfaces

Change-Id: I797aa7aacb13dfb7f35700463dc11d55552eb108
2016-04-22 11:31:54 +10:00
Gregory Haynes
a078e780ca dhcp-all-interfaces depends on dib-init-system
This element uses the dib-init-system command and therefore depends on
the element.

Change-Id: I1374500fb5b79e0f0c9c41346b5b7baf3f7755aa
2016-04-22 09:23:11 +10:00
Gregory Haynes
e096337a21 dhcp-all-interfaces depends on dhcp
Add package dependency for dhcp client

Change-Id: I63683485a5c5dbe65bfc38c8d64a88ee5549fda8
2016-04-22 09:23:09 +10:00
Jenkins
33d7e8b25e Merge "Add Gentoo to the dhcp-all-interfaces element" 2016-04-21 23:03:24 +00:00
Matthew Thode
de0cddc390
Add Gentoo to the dhcp-all-interfaces element
This makes use of the dhcpcd package and it's ability to run on all
interfaces by default.  We disable the privacy extensions and dhcp
overriding the hostname (both are enabled by default).  Other than
that it 'just works' and was the method used to bring up interfaces
on Gentoo Openstack images before we switched to building with DIB.

Change-Id: I02c14927d70b22f560c6fc149fefca0f93933f56
2016-04-21 16:40:06 -05:00
Jenkins
45afd99012 Merge "Handle unconfigured interfaces for dhcp-all-ifaces" 2016-04-21 05:23:37 +00:00
Jenkins
874fef9fe9 Merge "Really remove all interfaces in dhcp-all-ifaces" 2016-04-21 05:23:31 +00:00
Ian Wienand
7aa9157c33 yum-minimal: strip locale archive
Rather than removing all locale related stuff in cleanup, strip the
locale archive and rebuild it.

Building just en_US (along with POSIX/C) brings things inline with
debootstrap.  As discussed in the bug referenced, this is about the
best we can do for Centos7.

Fedora 24 has split languages out into packages so we don't have to do
this, but I have not dealt with that yet.  A guard is put in place so
we make sure we revisit this when we try to build F24.

Change-Id: I3f384d23e52effd6a09f47134746caa4a5c586be
2016-04-21 15:00:13 +10:00
Jenkins
a6754a5c3a Merge "Move selinux restore to end of finalise" 2016-04-21 04:30:08 +00:00
Abel Lopez
b2a2368844 Change to latest CentOS-6 image
cloud.centos.org appears to have changed their naming for images.
This latest iteration drops the YYYYMMDD in favor for YYMM, but
also has a 'latest' available without the date stamp.

This change will mean we no longer have to submit new code reviews
whenever centos changes.

Change-Id: I5a6a0de822561c1d0681abb9487993acf55918f1
2016-04-20 10:44:09 -07:00
Ian Wienand
6a1eb2457c Move selinux restore to end of finalise
After a bit of spelunking, I90d0c96d5659326ba67d6119b96d9a4113adf7fe
was the original change that introduced the setfiles here rather than
autorelabel at boot time.

Touching the autorelabel file probably makes sense somewhere low, but
when we start relabling the file system we really should be doing that
as late as possible so we fix up everything that has come before.
Move this to 90 to capture this.

Change-Id: Iae0afe850f52ec3b59c49507fa9bbcc1c8f8cfa1
2016-04-20 13:52:37 +10:00
Ian Wienand
e2c0d16f84 yum-minimal : better cleanup of initial yum failure
If the initial yum install into the chroot fails, we can leave behind
a lockfile and an incorrectly modified rpmmacros.

Change this so we run the cleanup unconditionally.

Change-Id: Ia9f9c4c845e5f34d33ff9a4ab7226c9175283757
2016-04-20 09:42:42 +10:00
Jenkins
1fabb01a4f Merge "Prioritize venv python on host" 2016-04-18 23:17:28 +00:00
Jenkins
146be596f6 Merge "simple-init: Fix path for /etc/ssh test" 2016-04-18 19:37:35 +00:00
Jenkins
20def6a0cb Merge "dib-run-parts: make cp to target root more robust" 2016-04-18 19:37:29 +00:00
Jenkins
4dced6e90d Merge "Fix add-apt-repository package for precise" 2016-04-18 19:17:09 +00:00
Ian Wienand
a8d8724e3c Add EPEL as requirement of centos-minimal
I guess I hadn't tried centos-minimal without the puppet elements that
install this for us.  But the "base" element wants dkms, which is only
in EPEL for centos.  But it's a helpful convenience so is globally
useful.

Change-Id: Ia9af97efdbd855fb8202353196ad649093788cb8
2016-04-16 07:03:39 +10:00
Ian Wienand
2dc4154724 Fix up EPEL element
For whatever reason, RHEL identifies itself with DISTRO "rhel" for 6
and "rhel7" for 7, but centos just uses "centos" and DIB_RELEASE.  So
this was wrong and installing EPEL6 on centos7.

But we can simplify it completely for centos because that comes with
the epel-release package already included.

Change-Id: I2b8f5d30b850fef46b4a5ba32a917abcbf25932c
2016-04-15 12:37:22 +10:00
Jenkins
4f6ce09385 Merge "Support to add certificate in ironic-agent" 2016-04-14 18:30:04 +00:00
Aparna
cd66aebf40 Support to add certificate in ironic-agent
This commits provides support to add certificate while
building the image using ironic-agent element. The
certificate can be CA certificate or self-signed certificate.

The certificate is set to the environment variable
'DIB_IPA_CERT' which in turn is used by the ironic-agent
element while building the image.

Change-Id: I648f7934d4787dcc3030885cfca771b642a9595e
2016-04-14 13:42:36 +00:00
Clint Byrum
4ceb40e13d simple-init: Fix path for /etc/ssh test
The cleanup path was fixed, but not the actual test.

Change-Id: If9ff4ee55604fa317a9a5bda0eee0b2783ef079a
2016-04-13 14:53:21 -07:00
Jenkins
67bef7ed16 Merge "Debian: dont set always the hostname to debian" 2016-04-11 08:31:55 +00:00
Jenkins
a6dd8d6b5a Merge "Turn down tracing for source-repo cache" 2016-04-11 06:12:49 +00:00
Jenkins
69c811870c Merge "Fix ssh key cleanup to run in chroot" 2016-04-06 19:12:56 +00:00
Ben Kero
e64a6f8c9a Fix add-apt-repository package for precise
Running 'disk-image-create -a amd64 -o precise vm precise' will fail to
build with the following error [1]. To fix this, we should split out the
switch-case to allow base to install the 'software-properties-common'
package.

[1]

dib-run-parts Fri Mar 18 18:43:16 UTC 2016 Running /tmp/in_target.d/pre-install.d/03-baseline-tools
Hit:1 http://mirrors.cat.pdx.edu/ubuntu precise InRelease
Reading package lists... Done
Reading package lists... Done
Building dependency tree
Reading state information... Done
Package python-software-properties is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source
However the following packages replace it:
  software-properties-common

  E: Package 'python-software-properties' has no installation candidate

Change-Id: I011f02fcf70df9d1d6fa30ed89907fbc2588937e
2016-04-06 11:08:07 -07:00
Jenkins
74822636c3 Merge "Revert "Skip centos functional testing"" 2016-04-06 17:21:21 +00:00
Gregory Haynes
00d9a9d396 Fix ssh key cleanup to run in chroot
Right now this removes the ssh keys on the build host. Obviously this is
not what we want.

Change-Id: Ib01d71ff9415a0ae04d963f6e380aab9ac2260ce
2016-04-06 17:11:59 +00:00
Saverio Proto
0d364d49cf Debian: dont set always the hostname to debian
If we dont force the hostname to be debian, the hostname will
be the one choosen when creating the instance with horizon

This bug was introduced when the debian element was split
into debian and debian-minimal and debootstrap in commit 23aa76df

Change-Id: I007a3531f6876890a2e113082211889ec5ead1a1
2016-04-06 11:05:53 +02:00
Clark Boylan
677619b1cc Remove ssh host keys when using simple init
In order for glean to operate properly and regenerate ssh host keys it
needs to start on a host without any ssh host keys. We hadn't been doing
this meaning that new host keys were not generated on images using
simple-init. Fix this by having simple-init delete any host keys found
in /etc/ssh late in the build process.

This should force glean/sshd to generate new host keys on first boot
depending on the distro.

Change-Id: Ic917387d92febf21c2cc37dfc1fa83fcf0c8e469
2016-04-06 05:47:37 +00:00
Antoine Musso
33cd0fb40f dib-run-parts: make cp to target root more robust
The dib-run-parts/root.d/90-base-dib-run-parts script would mysteriously
fail whenever the dib-run-parts is not in the path.  The install
commands complaining with:

install: missing destination file operand after
'/tmp/image.ILFTBYVy/mnt/usr/local/bin/dib-run-parts'

The reason is chaining a command in exec bypasses set -e. For example:

 exec /bin/ls $(which doesnotexist)

Will works and continue.

Handle the which lookup in a variable to have the script abort
immediately whenever dib-run-parts is not found.

Left an inline comment to make sure the fix is kept around in the
future.

Signed-off-by: Dan Duvall <dduvall@wikimedia.org>
Change-Id: Ibb5d7342b0d06483b84f9dd567e1cc0bf90f194e
2016-04-05 16:29:57 +02:00
Jenkins
792036ec62 Merge "ironic-agent postinst fails on systemd with no iptables" 2016-04-04 19:35:42 +00:00
Jenkins
fd806e9f42 Merge "Set tgtd not auto-start on OS boot time." 2016-04-02 16:28:43 +00:00
Jenkins
f80aff3645 Merge "ironic-python-agent should use console output" 2016-03-31 23:54:23 +00:00
zhangjian
fd7ccf97ee Set tgtd not auto-start on OS boot time.
Tgtd starts after OS boots up automatically, listens on the port 3260,
which cause lio fails to bind the same port later.
This patch removes the auto-start script of tgtd.

In IPA service, lio or tgtd would be used to create iscsi target.
If tgtd be chosen, IPA will start tgtd before iscsi setting.
IF lio be chosen, tgtd would keep not running.

Change-Id: Ic62055ac2f9252df832739631bfcbabfac0f9c4e
Closes-Bug: #1554987
2016-03-29 06:29:38 -04:00
Jenkins
81633f69c8 Merge "add pkg-map to pip-and-virtualenv element" 2016-03-24 05:12:13 +00:00
Jenkins
caf0c404cc Merge "Set default locale to image in debootstrap element" 2016-03-23 06:52:49 +00:00
Gregory Haynes
6bdb810f38 Handle unconfigured interfaces for dhcp-all-ifaces
On ubuntu, if an interface exists but is unconfigured then ifquery
$iface will return 0 but no output. This will cause dhcp-all-interfaces
to not attempt to dhcp on that interface.

Change-Id: Ic1532728ae9ca6d759714392f727d25b814ef336
2016-03-23 06:00:49 +00:00
Gregory Haynes
4f4fab979e Really remove all interfaces in dhcp-all-ifaces
As the doc says, we want to remove all interfaces that come by default
in cloud images. Cloud images also come with an eth1 though, so lets
just remove all interfaces.

Change-Id: I49c74a3285e6a610ac723bdf976f3727d2736749
2016-03-23 05:57:56 +00:00
Jenkins
683757e572 Merge "Remove all interface configs for simple-init" 2016-03-22 16:08:29 +00:00
Jenkins
3db1f42220 Merge "Note requirement for parted on gentoo hosts" 2016-03-22 07:41:00 +00:00
stephane
dd7648277f ironic-agent postinst fails on systemd with no iptables
Where the base install for the image uses systemd but
the iptables service is not present, the postinstall
script for the ironic-agent element can fail. This is
due to 'systemctl disable iptables.service' returning
an error status when the service isn't present. Check
whether the service is enabled using 'systemctl
list-unit-files'.

Change-Id: I8069b062a27caead4097747d7aeae24a5262e117
2016-03-21 19:27:57 -07:00
Matthew Thode
66c90cac9b
add pkg-map to pip-and-virtualenv element
Add a pkg-map file so that Gentoo can avoid polluting the FS with
files not tracked in the package manager.

Changed the directory name so that the source-install method would
actually be disabled when DIB_INSTALLTYPE_pip_and_virtualenv=package
is set.  Also changed source-repository line to be consistent with
the element name.

Change-Id: I5225bd7113ff2588c755ad4cd4ae9a2d78fff196
2016-03-21 20:12:37 -05:00
Ian Wienand
8c837409b8 Turn down tracing for source-repo cache
If you check logs like [1] it's literally thousands of lines of the
same thing over-and-over as the git caching happens.  It is basically
all just noise unless you're debugging it specifically.  Up this to
tracing level 2 ("-x -x") to see it.  Add a note in the help about
multiple flags, which has always been intended but not documented.

Image builds should continue to run with single "-x", but we could
probably greatly increase signal:noise ratio in the logs with a little
more judicial use of this to turn down some of the very noisy &
repetitive parts.

[1] anything in http://nodepool.openstack.org/

Change-Id: I91c5e55814ba9840769357261d203f4850e2eba6
2016-03-22 09:57:30 +11:00
Gregory Haynes
6f542c7c32 ironic-python-agent should use console output
It is standard to IPA to log to the console for easy debugging by
watching the server console.

Change-Id: I2beb75a99824f0cf1ec88d0ecb54baa4e73f6f15
2016-03-21 20:31:31 +00:00
Jenkins
29d9eeaa7f Merge "Add psmisc to the packages for ironic-agent" 2016-03-21 18:42:07 +00:00
Jenkins
ac94cf2a4e Merge "Add testing for the Gentoo element" 2016-03-21 07:59:55 +00:00
Nisha Agarwal
55ea2b5633 Add psmisc to the packages for ironic-agent
This commit adds 'psmisc' and 'dosfstools' to the required
packages for ironic-agent element which in turn
installs ironic-python-agent and ironic-lib.
The ironic-lib requires the fuser command to be executed,
which needs the package 'psmisc'. Similarly mkfs with
'vfat' requires 'dosfstools' package.

Closes-Bug: 1554075
Change-Id: If421f2394c34bb938813d0e15e5085d093215921
2016-03-21 00:01:57 -07:00
Ben Kero
71cf0662e2 Add new posix element.
This element installs package to ensure that the resulting image
provides the binaries outlined in posix specification [1]. This solves
some problems such as cron missing from the ubuntu-minimal images, and
hopefully minimizing the problems from similar problems in the future.

This is the image size difference:

Resulting qcow image sizes:

                Ubuntu      Ubuntu-Minimal    Debian    Centos7
        None    334MB       330MB             352MB     470MB
        Posix   353MB       375MB
        Posix+  354MB       374MB             406MB     471MB

        Posix+ = Posix + gcc + cssc

[1] http://pubs.opengroup.org/onlinepubs/9699919799/idx/utilities.html

Change-Id: I2a8f77bf8a0fdb04b328e56425d43a60eb3f7eed
2016-03-20 10:42:34 -07:00
Jenkins
9c5cd224a5 Merge "Add lshw package to ironic-agent" 2016-03-20 15:08:03 +00:00
Matthew Thode
8fc5a17f3a
Note requirement for parted on gentoo hosts
The vm element makes use of parted.  In order to create partitions
parted needs to be installed, growroot does not work without partitions.

Change-Id: Ib46cb818e0116f050de01f9a19e6041328d6d478
2016-03-19 22:04:05 -05:00
Gregory Haynes
3cb4273590 Remove all interface configs for simple-init
We are currently removing only eth0 network config in the simple-init
element (since simple-init is responsible for networking if you are
using it). We install an eth1 as well in elements such as debootstrap.
Lets just rm eth*.

Change-Id: Id9ea8c5e402f3a06b9064a939ad4b2e168cd6223
2016-03-19 04:17:34 +00:00
Yolanda Robla
af84759e12 Set default locale to image in debootstrap element
Set is to C.UTF-8 as default, to reuse what is
available on this base image.

Change-Id: I3d2517582b836b2c6ec05538e6f9c58cada66965
2016-03-18 10:04:50 +01:00
Gregory Haynes
71eab09539 Make ubuntu-core support releases
Right now we hard code in a path for an alpha release image. There are
release of ubuntu-core now so lets support them.

Change-Id: Ice989d32e0910cbbfb711e06adb33b07682856c9
2016-03-17 16:40:10 +00:00
Matthew Thode
9707cf5282
Add testing for the Gentoo element
This commit adds testing for the gentoo element based off of the debian
element.  In adding testing support a couple of additional errors were
found and corrected, namely the following.

* shm is not in /proc/mounts though it is valid for use based off of
  the sticky flag.
* The path in ELEMENTS_PATH needs to be pruned to be usable.
* Added uuidgen-runtime to the list of packages installed for Ubuntu
  as the new ubuntu-trusty image does not ship with it.

Change-Id: I0d2768a912b350e1b2cf40d2fd227fdc767b1bcb
2016-03-17 00:13:32 -05:00
Jenkins
3fca968164 Merge "Don't cache debootstrap rootfs by default" 2016-03-16 00:06:18 +00:00
Jenkins
b7957e29fe Merge "centos-minimal does not provide base" 2016-03-14 03:27:56 +00:00
Jenkins
9b3d7504c5 Merge "yum-minimal: clear our rpm/dnf/yum data in chroot" 2016-03-13 23:44:49 +00:00
Jenkins
2a10525efe Merge "Create new partitioning-sfdisk element." 2016-03-12 20:00:12 +00:00
Jenkins
4216b23654 Merge "Adding InfiniBand Support" 2016-03-12 19:21:19 +00:00
Jenkins
1308ec4bc2 Merge "debian-minimal: configurable debootstrap components" 2016-03-12 19:13:03 +00:00
Jenkins
63336d1f6f Merge "Add force-confdef in debian package install" 2016-03-12 19:05:00 +00:00
Jenkins
6d1bc6f1c3 Merge "Refactor growroot for debuggabilty" 2016-03-12 19:04:22 +00:00
Jenkins
48587b3e93 Merge "Depend on ifupdown in simple-init" 2016-03-12 19:04:04 +00:00
Ian Wienand
204da89128 centos-minimal does not provide base
The "base" element is included by default, and if centos-minimal tries
to provide it then we die because two elements are providing the same
thing.

Change-Id: I2956aaa49ba3137a799f97e0983ab4b7c93a0a0c
2016-03-10 13:51:08 +11:00
Moshe Levi
9074f5f1fc Add lshw package to ironic-agent
This patch add the lshw utils to allow the agent
to retrive the vendor name. It also keeps the
/usr/share/misc/pci.ids file for Ubuntu distro.

Change-Id: I2828a7e585449f63887ad5a2e338b4e043d68e63
2016-03-09 12:54:57 +02:00
Ian Wienand
6f622fb7c9 yum-minimal: clear our rpm/dnf/yum data in chroot
I have seem some occasional odd failures coming from the "dnf -y
update" done by elements/base/install.d/00-up-to-date.

dnf seems to sometimes think a package is not installed when it really
is.  It then seems to try and re-install them, but notices they are
installed, and then bails with a failure exit [1].  The packages that
seem to cause this vary, but the common thread is that they seem to
have all been installed during the initial phase of installing the
package manager in the chroot.

I suspect that when we are building the chroot, we do our initial
install with the "external" yum & rpm.  Then we start using the
dnf/yum in the chroot, but we're actually using meta-data created by
the *external* tools -- which could be vastly different versions or
who-knows-what.  While I honestly I don't have an exact root cause,
empirically I've found rebuilding the rpm db always seems to fix
things up.

So this change takes care to rebuild the rpm db with the chroot
version of rpm, and clear out the package metadata for a refresh with
"update".  This should hopefully put us in a consistent state.

[1] http://paste.openstack.org/show/487356/

Change-Id: I565df23897ae511356c4861fdbe63823fa6b6ff9
2016-03-08 14:53:52 +11:00
Yolanda Robla
e083686e00 Create new partitioning-sfdisk element.
This element is similar to vm, but allows more flexibility
on disk partitioning. It receives a DIB_PARTITIONING_SFDISK_SCHEMA
setting, allowing to partition the disk according to project needs.

Change-Id: I01dca5d5fd670d317f7761911a1549507de6e97d
2016-03-07 19:50:55 +01:00
Jenkins
8b0d124a0e Merge "Fix spurious = in dib-python readme" 2016-03-07 04:59:22 +00:00
Jenkins
700445b3db Merge "Fix cloud-init-disable-resizefs README title" 2016-03-07 04:49:53 +00:00
Gregory Haynes
fbabe0b663 Fix spurious = in dib-python readme
Change-Id: I51e7961fab7f3eeaf3ce4eeb1dd5e6981eefb154
2016-03-07 04:47:35 +00:00
Jenkins
6e29b45d90 Merge "Replace sfdisk partitioning with parted" 2016-03-03 12:35:50 +00:00
Gregory Haynes
6ee7da704d Fix cloud-init-disable-resizefs README title
Element readme's need to start with a title that matches the element
name.

Change-Id: Idf48f70e112af1f17d398155a317872b1d569f21
2016-03-01 21:49:10 -08:00
Matthew Thode
175e292ec1 Add Gentoo to the list of supported distributions
We also document any additional package needed.

Change-Id: I7b9193fa5711138287f587935fc864fee4f95856
2016-02-29 12:20:50 +00:00
Gregory Haynes
cb15e7ef33 Prioritize venv python on host
If we are being run from a python 3 virtualenv where python2 is also
available on the host we will use the hosts' python2 over the virtualenv
python3 (even if the virtualenv is higher precedence in PATH).

Change-Id: I9a25b9e45845a4121aab8250fd53c6f006825742
2016-02-28 20:50:02 -08:00
Jenkins
103b7dea6d Merge "Fix building on gentoo hosts" 2016-02-27 14:44:46 +00:00
Jenkins
de4d0392d5 Merge "Install IPA in a virtual environment" 2016-02-26 16:23:37 +00:00
Jenkins
cebdba1f9c Merge "Revert "Correct rhel-common for rhel6"" 2016-02-22 16:57:40 +00:00
Matthew Thode
dacfc23c36
Fix building on gentoo hosts
This patch fixes the profile autodetect.  If the profile was passed at install
time, it would overridden by the environment.d script.  This was fixed by
accepting the value passed by the user, if one was passed.

The enviroment.d file is run outside of chroot, because of this eselect will
not be available on non-gentoo systems.  We change the behavior to work
elsewhere, it was not working on non-gentoo systems and giving bad results on
gentoo systems (returing values from the host, not the guest being built).

This patch also fixes shm detection by checking if it is mounted already.

Change-Id: Ie58d8e19529a731bfbc9eeb4bb246988d1aaa772
2016-02-21 22:46:13 -06:00
Monty Taylor
74487a383c Depend on ifupdown in simple-init
Newer distros, such as debian jessie and ubuntu xenial, do not provide
ifupdown by default, but simple-init depends on it. Add it to the pile.

Change-Id: I6f4876863c67c65a82464d4e0593015cdc839c5c
2016-02-20 08:19:21 -06:00
Jenkins
05edfb6d20 Merge "Don't remove python3 & grubby in 99-remove-extra-packages" 2016-02-19 01:46:38 +00:00
Ben Nemec
a342036e75 Revert "Correct rhel-common for rhel6"
This is breaking deployments that use registration.  I
believe os-refresh-config scripts don't source environment.d
files, so the variable is unset there.

Given that this a blocking issue for RHEL deployments, I'm
pushing a quick revert and we can come up with a proper fix
when this isn't stopping other work.

This reverts commit 71bd8b3a33.

Change-Id: I87504660352220d45f5445bb933edc1c28885fcb
2016-02-17 22:17:11 +00:00
Moshe Levi
14b900d3c9 Adding InfiniBand Support
This patch update the find_interface to lookup
for InfiniBand interface according to it's BOOTIF

Closes-Bug: #1532534

Change-Id: I21b91cfd10888ac036f6347a0a44cdca422830a6
2016-02-17 04:43:29 +00:00
Jenkins
352e58ceb7 Merge "yum-minimal: pre-install base packages" 2016-02-17 00:25:08 +00:00
Yolanda Robla
d5a6a7cb44 Add force-confdef in debian package install
In certain cases, with packages cached that need an upgrade,
when performing that action the system hangs waiting for
a prompt.
Add force-confdef option, that will let dpkg overwrite
configuration packages that were not modified. In combination
with already existing force-confold flag, it will allow
to avoid any conffile prompt.

Change-Id: Ifb177f9ac2c9ad29f8b92309c5b8cfe8e60a4e14
2016-02-16 14:10:19 +01:00
Lucas Alvares Gomes
006b3e6228 Install IPA in a virtual environment
In order to avoid conflict with installing dependencies for IPA from
pypi and distribution packages this patch propose installing IPA in a
virtual environment.

Closes-Bug: #1506792
Change-Id: I6a4c6403813d86f4110e98449ddd874109101b9e
2016-02-16 11:17:12 +00:00
Gregory Haynes
2678209e94 Don't cache debootstrap rootfs by default
By default we create a tarball of any debootstrap rootfs we create. For
the majority of use cases this is a large performance hit for no
benefit. Lets make this an opt-in feature.

Change-Id: I58fc485aacacaa17243bf9ce760ed91256d1f182
2016-02-16 07:55:19 +00:00
Ian Wienand
b960614c9c Don't remove python3 & grubby in 99-remove-extra-packages
python3 is a hard requirement of dnf so can't be removed [1]

grubby is also required for kernel installs on Fedora.  For too much
detail see I1a6e45d04755515286b3d49f8280c16b527e2f48; but the kernel,
via dracut, now has this as a "recommends" due to people removing it
and making unbootable systems.

[1] http://logs.openstack.org/76/248976/2/check/gate-dib-dsvm-functests-devstack-f21/734c8bd/console.html

Change-Id: I5867ecd57834eece9477aa9ea4b8bdd70e238084
2016-02-16 13:40:01 +11:00
Jenkins
9e16c8c5f1 Merge "Use dnf to cleanup old kernels" 2016-02-15 22:56:12 +00:00
Jenkins
99d9373eca Merge "Fix dpkg element for Ubuntu Xenial" 2016-02-15 20:33:35 +00:00
Antoine Musso
ee58c5b1cc debian-minimal: configurable debootstrap components
The debian-minimal element creates /etc/apt/sources.list solely with
the 'main' component. I need to add 'non-free' and 'contribs'. I tried
to pass them via DIB_DEBIAN_COMPONENTS but it is not recognized.

Make debian-minimal to honor DIB_DEBIAN_COMPONENTS. Note that it is
comma separated for 'debootstrap', so replace commas with spaces to fit
the needs of sources.list.

Example usage:

 DIB_DEBIAN_COMPONENTS='main,non-free,contrib'

Will debootstrap with the three components then when debian-minimal is
realized pre install a sources.list that has:

  deb http://example.org/debian jessie main contrib non-free

Change-Id: I1dca2e8ffd31044a6b441ccb277298601e62f67c
2016-02-12 22:57:58 +01:00
Jenkins
afd2eec60e Merge "Remove eclean-dist as it's not available by default" 2016-02-12 16:31:26 +00:00
Jenkins
e88792a3ed Merge "Increase interface has link retries to 20" 2016-02-12 08:26:09 +00:00
Jenkins
dcf2c4f261 Merge "Reorder the package-uninstall action" 2016-02-12 04:04:14 +00:00
Jenkins
5036ad6baf Merge "Do not remove python-dev from ironic-agent image" 2016-02-12 04:04:08 +00:00
Jenkins
4136479bd0 Merge "Fix startup race with growroot/systemd" 2016-02-12 04:03:55 +00:00
Matthew Thode
0ab0f69ce3 Remove eclean-dist as it's not available by default
Gentoo provides eclean-dist via the gentoolkit package
eclean-dist isn't needed anyway was /usr/portage is removed anyway

Removed redundant package update in cleanup.

Change-Id: Icf4f9ed549b9a6d923448d927d7c42bcf8d6091b
2016-02-12 00:31:27 +00:00
Matthew Thode
22ef95c1a1
Reorder the package-uninstall action
This action has been reordered so cleanup can occur before copy to blockdev
occurs.  Documentation has been added about the ordering of this element in
relation to cleanup actions as well.

Change-Id: I3f9334a3669ee588d7fa7129202c97fa22fdb050
2016-02-11 18:30:05 -06:00
Dmitry Tantsur
1f9e669dbc Do not remove python-dev from ironic-agent image
While it does save 1MiB of space, it might also pull python packages
depending on it. E.g. it makes impossible to install python-hardware
on the IPA image for advanced introspection.

Change-Id: Iab80dde63e6de62a5e45dcf404b4f9f633e50ac3
2016-02-11 13:47:31 +01:00
Jenkins
1cd7ca70d8 Merge "Correct rhel-common for rhel6" 2016-02-11 04:49:35 +00:00
Ian Wienand
349e8db2f7 yum-minimal: pre-install base packages
We were getting some subtle issues in fedora-minimal builds that
turned out to be because /var/run was not a symlink to /run.

Upon further investigation, it turns out that yum is creating a
/var/run directory for it's pid file when it starts working in the
empty chroot (which I verified by stracing it)

---
5905  stat("/home/ubuntu/tmp/dib-tmp/image.Ac4VZZsl/mnt/var/run", 0x7ffddffa0330) = -1 ENOENT (No such file or directory)
5905  mkdir("/home/ubuntu/tmp/dib-tmp/image.Ac4VZZsl/mnt/var/run", 0755) = 0
5905  open("/home/ubuntu/tmp/dib-tmp/image.Ac4VZZsl/mnt/var/run/yum.pid", O_WRONLY|O_CREAT|O_EXCL, 0644) = 6
---

Because this happens *before* we install "filesystem" (the package),
we mess up it's symlinking.

To work-around this, pre-install the trio of base packages (setup,
basesystem, filesystem) with rpm from outside the chroot.

Change-Id: I411b6ec9d91d95d3a0f98e76853086af3b70abe8
2016-02-11 15:42:10 +11:00
Jenkins
536b86fe09 Merge "Mark ironic-discoverd-ramdisk as deprecated in favor of ironic-agent" 2016-02-10 22:48:42 +00:00
Jenkins
910a99c3ca Merge "Fix growroot for Gentoo's openrc" 2016-02-10 21:50:50 +00:00
Colleen Murphy
a9b38d0b60 Fix dpkg element for Ubuntu Xenial
The Ubuntu Xenial cloud server images set the mode of
/var/lib/apt/lists/partial to 700, so when mounted it's inaccessible to
an unprivileged user, resulting in an error:

  find: `/tmp/image.aDQKdkRi/mnt/var/lib/apt/lists/partial': Permission denied

There's no reason an image should come with anything already in
/var/lib/apt/lists/partial, so just avoid trying to descend into that
directory when fixing the apt translations packages.

Change-Id: Id27f0166bfb09d67200f337a5ffff2f2037b7c1c
2016-02-10 12:50:43 -08:00
Jenkins
b13512a7f4 Merge "Only match #!/bin/bash in scripts" 2016-02-10 08:35:09 +00:00
Ian Wienand
63182a02e4 Refactor growroot for debuggabilty
This is a slight refactor I found useful when debugging.  The
udevadmin info query will be visbile in the output of "set -x" which
helps, and is the logs/journal.

We can also reduce some calls by keeping the value and just grepping
with a herefile.

This also does some error checking and bails out if it does not see
what it needs to continue.

Change-Id: I39c4d262f9c5ce53f6b83d95b1363a74834cf2c8
2016-02-10 14:02:28 +11:00
Ian Wienand
9cd35c36e4 Fix startup race with growroot/systemd
Tucked away in systemd-udev-settle.service is the following comment

 # This service can dynamically be pulled-in by legacy services which
 # cannot reliably cope with dynamic device configurations, and
 # wrongfully expect a populated /dev during bootup.

The info that the growroot script is querying is populated via udev,
particularly the blkid bits of [1].  This creates a race-condition
where sometimes udev has been triggered and the rules have applied and
sometimes not.  Obviously in the first case, the root disk is not
grown correctly.

systemd-udev-settle is mostly disabled on distros because it can cause
an increase in boot-time for systems with lots of disks; this is not
our situation so it makes basically no difference.

That said, I will investigate if some systemd people know even better
ways to do this (possibly the service should depend on block .device
targets in systemd, and then filter out and only apply to the root
disk?)

[1] https://github.com/systemd/systemd/blob/master/rules/60-persistent-storage.rules#L66

Change-Id: I453e3afcd953dfc29ab6c42ddc81e940cfa70ee0
2016-02-10 13:47:18 +11:00
Jenkins
a69dd548a7 Merge "Make dkms element depend on dkms package" 2016-02-08 20:17:26 +00:00
Ben Kero
41f503fef3 Replace sfdisk partitioning with parted
A TODO was placed on the partitioning section of the vm element to
replace sfdisk with a saner (and less arcane) way of partitioning. It
suggested parted for replacement. This changeset should reproduce the
same disk label and partition layout as sfdisk, but with less ioctl
errors and version dependency. It will also ensure partition alignment.

Change-Id: I5d8d75131458b73bfb05f80f1bfa7e2970e004b3
2016-02-08 12:15:50 -08:00
Gregory Haynes
6ab5078d0f Switch simple-init to pip-and-virtualenv element
We currently install pip from package in the simple-init element.
We should really allow users to select whether to install pip from
git or package.

Change-Id: Ia5e62b9635af90d81227274a1dd8f20474cdbf73
2016-02-08 16:58:14 +11:00
Ian Wienand
d5155254cb Revert "Skip centos functional testing"
This reverts commit d10871534f.

This is a place-holder so I don't forget to revert this when [1] is
resolved

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1303660

Change-Id: I18c8aa6e7233f74c680d23b573fe9c3817a5779a
2016-02-08 15:55:29 +11:00
Ian Wienand
cb0e0e903d Use dnf to cleanup old kernels
As described in the comment, there is a dnf equivalent of this command
that doesn't require us installing yum-utils (which drags in yum on
dnf-only systems such as f23)

This is a small consequence to this -- due to us not installing
yum-utils some installs will now be completely yum free.  This causes
a breakage in ironic-agent 99-remove-extra-packages where we remove
the yum package.  There is a long-standing bug/feature where missing
packages in a group of packages do not cause yum/dnf to exit with
failure, but uninstalling a single package will.  Because we have made
the systems yum-free, the uninstall of yum can fail in this corner
case.

It has always been like this, so I'm in favour of the "ain't broke"
approach.  To work-around this, I have just put yum into the existing
list of packages to be cleaned up.  I have added a note to the yum
installer taking note of this behaviour for future reference.

Change-Id: I8bbdc07ccdb89a105b4fc70d5a215077c42fcd03
2016-02-08 14:20:56 +11:00
Moshe Levi
a1d32c6d51 Increase interface has link retries to 20
InfiniBand interface takes more time to bring up then
Ethernet interface. This patch just increase the retries
to 20 times, to make it work for InfiniBand as well.

Change-Id: I5c4842696207885552413ea2d053f2e90bd6803c
2016-02-07 16:28:52 +02:00
Matthew Thode
9878469a53
Fix growroot for Gentoo's openrc
Let dib-init-system's postinstall handle enabling init scripts.

Change-Id: I7b0d235327021b7f478cdb9715d533b444843699
2016-02-04 23:41:45 -06:00
Jenkins
5ee1cbac4a Merge "Add support for OpenRC to dib-init-system" 2016-02-05 04:15:53 +00:00
Jenkins
50f7b00a54 Merge "Add support for gentoo to simple-init" 2016-02-04 08:00:47 +00:00
Jenkins
1dcd65fb9e Merge "Print unparsable file in pkg-map" 2016-02-04 04:47:20 +00:00
Matthew Thode
265b31e6b5
Add support for OpenRC to dib-init-system
Adds a post-install function that enables installed initscripts,
as that is not done by default in gentoo.

Change-Id: I04e8d506ddcbefa8a983dd31ad16df5e13cb26e7
Closes-Bug: 1539276
2016-02-02 17:23:35 -06:00
Jenkins
4526cf23ec Merge "Fix Gentoo hardened support" 2016-02-02 20:49:30 +00:00
Jenkins
66184b7b40 Merge "Add new cloud-init element" 2016-02-02 20:49:03 +00:00
Jenkins
cd3c704fbf Merge "Fix package-installs for python3" 2016-02-02 03:16:12 +00:00
Jenkins
893b452b11 Merge "Add Gentoo support to growroot" 2016-02-02 03:15:22 +00:00
Jenkins
4cb6bed971 Merge "Add support for Gentoo to source-repositories" 2016-02-02 03:15:16 +00:00
Matthew Thode
a2d4af9541
Add new cloud-init element
Allows for composable builds from a minimal source.

Change-Id: I16f560ca4b1737c0928f3f3b15ea2580df820e97
Closes-Bug: 1539272
2016-02-01 21:03:03 -06:00
Matthew Thode
01fce7b70c
Fix Gentoo hardened support
This checks the profile, if it has hardened in it's name it needs xattr support
unfortunately xattr support cannot yet be relied on everywhere, so it needs to
be disabled for hardened profile builds to correctly pax-mark.

Change-Id: I7fb855249a9e6c9b6497ab5061b4ea3c014f5081
Closes-Bug: 1537177
2016-02-01 20:56:37 -06:00
Ian Wienand
d10871534f Skip centos functional testing
Due to upstream bug [1] there are uninstallable packages which mean
our functional tests don't work.  We will revert this when things are
working upstream.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1303660

Change-Id: I93c2990472e88ab3e5ff14db56b4ff1b4dd965ef
2016-02-02 10:34:18 +11:00
Matthew Thode
c6ef183975 Fix package-installs for python3
subprocess.check_call() returns a byte-string which needs to be turned
into a unicode string for python3 compatability.

Also some minor refactoring while we're here.

Closes-Bug: 1536462
Change-Id: Icd957bc4d93ccad94b1246ad62e6e02ee14d9ca5
2016-02-01 22:22:15 +00:00
Matthew Thode
4480b5edef Add support for gentoo to simple-init
Needed to remove interface config

Depends-On: Id1487bc82ee4b03aa57258f71f48ca7d377afc47
Change-Id: Iaa1a16b0f65c088f03c9b440ef16427e93ec9b03
Closes-Bug: 1537453
2016-02-01 22:14:48 +00:00
Dmitry Tantsur
3d133a3eaf Mark ironic-discoverd-ramdisk as deprecated in favor of ironic-agent
Also fix no longer correct parts of its README.

Change-Id: I00cbe5b4f65f9909ae63b7eb4bbac0ba82c145d1
2016-02-01 16:47:37 +01:00
Jenkins
48ae802c5a Merge "Remove zero length files" 2016-02-01 05:46:44 +00:00
Matthew Thode
92de91476e Add Gentoo support to growroot
Add missing growroot initscript and pkg-map entries for Gentoo.

growpart was added to Gentoo with [1]

Update the readme to reflect reality too (fedora added with
I5630dc638f85b1e80795826ef36a306632075460)

[1] https://packages.gentoo.org/packages/sys-fs/growpart

Closes-Bug: #1539273
Change-Id: I29056c7297489ec04f37757dbe33976901eceb49
2016-02-01 10:55:03 +11:00
Matthew Thode
2afb99db10 Add support for Gentoo to source-repositories
As mentioned in package-installs.yaml, git is a transitive dependency
for pbr in this element.  Add pkg-map for the Gentoo package.

Change-Id: I7f2fe1663152ea66b941594e86f1da93ddd21677
Closes-Bug: 1539278
2016-02-01 10:02:32 +11:00
Ian Wienand
d8abe72537 Only match #!/bin/bash in scripts
Our dib-lint checking is only considering scripts with #!/bin/bash.
While there's nothing really wrong with some other shebang line like
"#!/usr/bin/env bash" let's keep things consistent.

We can use the same regex match to reduce a few forks in the main
checking.

Also a minor cleanup to the file matching

Change-Id: I609721b2671e704ea26075dad7e5b39a8b858f6b
2016-01-29 15:49:10 +11:00
Jenkins
4b1b813a68 Merge "Fix debian-minimal image building" 2016-01-29 04:32:24 +00:00
Jenkins
c31a59a2c9 Merge "Revert "Use pip 7 for ironic"" 2016-01-28 23:54:36 +00:00
Igor Belikov
78278405ce Fix debian-minimal image building
'locales' package gets installed before '12-debian-locale-gen' is executed
and generates effectively empty /etc/locales.gen in debian, which makes
dpkg-reconfigure call to locales ignore the values set by
debconf-set-selections.
* Remove /etc/locale.gen generated by 'locales' installation to ensure
  proper locales generation on debian images
* Remove 'locales-all' package installation from debian element since
  it's not needed anymore to build the image and cosnumes additional
  ~120MB of space
* Remove unused 'package-installs' dependency from debian-minimal
  element

Change-Id: Ic39ba2b5ceb5018efb75742547b2babf80827e56
Closes-Bug: #1452400
2016-01-28 16:41:35 +03:00