Commit Graph

2487 Commits

Author SHA1 Message Date
Gregory Haynes
71eab09539 Make ubuntu-core support releases
Right now we hard code in a path for an alpha release image. There are
release of ubuntu-core now so lets support them.

Change-Id: Ice989d32e0910cbbfb711e06adb33b07682856c9
2016-03-17 16:40:10 +00:00
Matthew Thode
9707cf5282
Add testing for the Gentoo element
This commit adds testing for the gentoo element based off of the debian
element.  In adding testing support a couple of additional errors were
found and corrected, namely the following.

* shm is not in /proc/mounts though it is valid for use based off of
  the sticky flag.
* The path in ELEMENTS_PATH needs to be pruned to be usable.
* Added uuidgen-runtime to the list of packages installed for Ubuntu
  as the new ubuntu-trusty image does not ship with it.

Change-Id: I0d2768a912b350e1b2cf40d2fd227fdc767b1bcb
2016-03-17 00:13:32 -05:00
OpenStack Proposal Bot
3bfc44cb99 Updated from global requirements
Change-Id: Id519f2c3129418821eae1692d4da8305729202e8
2016-03-16 13:26:57 +00:00
Jenkins
3fca968164 Merge "Don't cache debootstrap rootfs by default" 2016-03-16 00:06:18 +00:00
Jenkins
b7957e29fe Merge "centos-minimal does not provide base" 2016-03-14 03:27:56 +00:00
Jenkins
9b3d7504c5 Merge "yum-minimal: clear our rpm/dnf/yum data in chroot" 2016-03-13 23:44:49 +00:00
SamYaple
5b6716cee8 Use fstrim to prep the block device
This cuts the image size down alot, esspecially if there were lots of
small file deletes.

The fstrim utility is in the util-linux package and should be on
most all systems. fstrim also works with XFS, ext4, btrfs, etc
prodiving the kernel is new enough.

A reduction of 25% or more in size is common.

Change-Id: I269b4416be450369616f9b8e030f84c30e329804
2016-03-13 16:24:59 +00:00
Jenkins
2a10525efe Merge "Create new partitioning-sfdisk element." 2016-03-12 20:00:12 +00:00
Jenkins
4216b23654 Merge "Adding InfiniBand Support" 2016-03-12 19:21:19 +00:00
Jenkins
90318b71c8 Merge "Add --version option to disk-image-create" 2016-03-12 19:14:35 +00:00
Jenkins
1308ec4bc2 Merge "debian-minimal: configurable debootstrap components" 2016-03-12 19:13:03 +00:00
Jenkins
63336d1f6f Merge "Add force-confdef in debian package install" 2016-03-12 19:05:00 +00:00
Jenkins
6d1bc6f1c3 Merge "Refactor growroot for debuggabilty" 2016-03-12 19:04:22 +00:00
Jenkins
48587b3e93 Merge "Depend on ifupdown in simple-init" 2016-03-12 19:04:04 +00:00
SamYaple
0c32784663 Revert "Zerofree the image if possible"
This reverts commit 5184d02a7c.

The decision was made to go with fstrim because it is faster and more
universal that zerofree. The related-id has the patchset that implements
fstrim.

Related-Id: I269b4416be450369616f9b8e030f84c30e329804
Change-Id: If40cf2fc0ecd8686768cbfeac9ecee90907674e7
2016-03-10 20:38:01 +00:00
Ian Wienand
204da89128 centos-minimal does not provide base
The "base" element is included by default, and if centos-minimal tries
to provide it then we die because two elements are providing the same
thing.

Change-Id: I2956aaa49ba3137a799f97e0983ab4b7c93a0a0c
2016-03-10 13:51:08 +11:00
Moshe Levi
9074f5f1fc Add lshw package to ironic-agent
This patch add the lshw utils to allow the agent
to retrive the vendor name. It also keeps the
/usr/share/misc/pci.ids file for Ubuntu distro.

Change-Id: I2828a7e585449f63887ad5a2e338b4e043d68e63
2016-03-09 12:54:57 +02:00
Jenkins
5da32104b3 Merge "Zerofree the image if possible" 2016-03-08 04:53:52 +00:00
Ian Wienand
6f622fb7c9 yum-minimal: clear our rpm/dnf/yum data in chroot
I have seem some occasional odd failures coming from the "dnf -y
update" done by elements/base/install.d/00-up-to-date.

dnf seems to sometimes think a package is not installed when it really
is.  It then seems to try and re-install them, but notices they are
installed, and then bails with a failure exit [1].  The packages that
seem to cause this vary, but the common thread is that they seem to
have all been installed during the initial phase of installing the
package manager in the chroot.

I suspect that when we are building the chroot, we do our initial
install with the "external" yum & rpm.  Then we start using the
dnf/yum in the chroot, but we're actually using meta-data created by
the *external* tools -- which could be vastly different versions or
who-knows-what.  While I honestly I don't have an exact root cause,
empirically I've found rebuilding the rpm db always seems to fix
things up.

So this change takes care to rebuild the rpm db with the chroot
version of rpm, and clear out the package metadata for a refresh with
"update".  This should hopefully put us in a consistent state.

[1] http://paste.openstack.org/show/487356/

Change-Id: I565df23897ae511356c4861fdbe63823fa6b6ff9
2016-03-08 14:53:52 +11:00
Ian Wienand
fa3c5e3056 Add image size report
In the common case of not specifying a size, we are already running
"du" over the image to figure out how big it is.  Leverage that by
saving it's output and displaying a pruned list of big files when
requested.

We add a flag to show a summarised option (files >10MiB) and another
to show full output, should you wish that level of detail.

"Invocation" documentation is updated (and formatted a little better
while we're here).

Change-Id: I255800790a62fed1c82fcd311f1cc29c9867766d
2016-03-08 13:58:40 +11:00
Clark Boylan
5184d02a7c Zerofree the image if possible
If the image has an ext filesystem and the zerofree utility is present
on the build system then run zerofree. This should make images as
compressable as possible which is a nice feature when building
compressed qcow2 images.

Change-Id: Ia6062c291f7a3f58b85a4f408ecb3d0574c65d53
2016-03-07 17:45:14 -08:00
Yolanda Robla
e083686e00 Create new partitioning-sfdisk element.
This element is similar to vm, but allows more flexibility
on disk partitioning. It receives a DIB_PARTITIONING_SFDISK_SCHEMA
setting, allowing to partition the disk according to project needs.

Change-Id: I01dca5d5fd670d317f7761911a1549507de6e97d
2016-03-07 19:50:55 +01:00
Jenkins
8b0d124a0e Merge "Fix spurious = in dib-python readme" 2016-03-07 04:59:22 +00:00
Jenkins
700445b3db Merge "Fix cloud-init-disable-resizefs README title" 2016-03-07 04:49:53 +00:00
Gregory Haynes
fbabe0b663 Fix spurious = in dib-python readme
Change-Id: I51e7961fab7f3eeaf3ce4eeb1dd5e6981eefb154
2016-03-07 04:47:35 +00:00
Jenkins
6e29b45d90 Merge "Replace sfdisk partitioning with parted" 2016-03-03 12:35:50 +00:00
Gregory Haynes
6ee7da704d Fix cloud-init-disable-resizefs README title
Element readme's need to start with a title that matches the element
name.

Change-Id: Idf48f70e112af1f17d398155a317872b1d569f21
2016-03-01 21:49:10 -08:00
Gregory Haynes
3eeefefa85 Reorder developer quickstart docs
The quickstart should be the first bit of developer documentation, not
the last. Also add in a short blurb for the developer docs so we dont
have two doc titles back to back.

Change-Id: Icb5683b8eb22e759fefb1cb2252ed445dea5f7dd
2016-03-01 21:44:18 -08:00
Colleen Murphy
76a8d3704b Add --version option to disk-image-create
Being able to discover DIB's version from the command itself is
convenient. This patch adds a --version option to the disk-image-create
command, failing gracefully if diskimage-builder is not installed.

This adds an explicit dependency on pbr to the requirements since this
is required to run diskimage_builder/version.py outside of a test
environment.

This patch consciously chooses to only provide the long-form option
and no '-v' to allow for the future possibility that a '-v' might
indicate '--verbose' in the future.

Change-Id: I9fc084774d6c7a39a944b07680b3eb8be8e34f9c
2016-02-29 16:13:01 -08:00
Matthew Thode
175e292ec1 Add Gentoo to the list of supported distributions
We also document any additional package needed.

Change-Id: I7b9193fa5711138287f587935fc864fee4f95856
2016-02-29 12:20:50 +00:00
Gregory Haynes
cb15e7ef33 Prioritize venv python on host
If we are being run from a python 3 virtualenv where python2 is also
available on the host we will use the hosts' python2 over the virtualenv
python3 (even if the virtualenv is higher precedence in PATH).

Change-Id: I9a25b9e45845a4121aab8250fd53c6f006825742
2016-02-28 20:50:02 -08:00
Jenkins
103b7dea6d Merge "Fix building on gentoo hosts" 2016-02-27 14:44:46 +00:00
Jenkins
de4d0392d5 Merge "Install IPA in a virtual environment" 2016-02-26 16:23:37 +00:00
Jenkins
8e9c54d01a Merge "Rework functional test runner" 2016-02-23 00:28:08 +00:00
Jenkins
cebdba1f9c Merge "Revert "Correct rhel-common for rhel6"" 2016-02-22 16:57:40 +00:00
Matthew Thode
dacfc23c36
Fix building on gentoo hosts
This patch fixes the profile autodetect.  If the profile was passed at install
time, it would overridden by the environment.d script.  This was fixed by
accepting the value passed by the user, if one was passed.

The enviroment.d file is run outside of chroot, because of this eselect will
not be available on non-gentoo systems.  We change the behavior to work
elsewhere, it was not working on non-gentoo systems and giving bad results on
gentoo systems (returing values from the host, not the guest being built).

This patch also fixes shm detection by checking if it is mounted already.

Change-Id: Ie58d8e19529a731bfbc9eeb4bb246988d1aaa772
2016-02-21 22:46:13 -06:00
Monty Taylor
74487a383c Depend on ifupdown in simple-init
Newer distros, such as debian jessie and ubuntu xenial, do not provide
ifupdown by default, but simple-init depends on it. Add it to the pile.

Change-Id: I6f4876863c67c65a82464d4e0593015cdc839c5c
2016-02-20 08:19:21 -06:00
Ian Wienand
b18f71f781 Rework functional test runner
This simplifies and enhances the functional-test runner script for
much better interactive behaviour and to give us the ability to better
choose what is running in CI.

Firstly, I have split the image-output testing into a separate script.
This is not actually part of the functional testing of elements and is
both logically and functionally different.  It currently does not run
in upstream CI because we don't have docker in the images.  I have
nothing against it, but it can be it's own thing.

run_functests.sh is overhauled to have a useful interactive interface,
e.g.

---
 $ ./run_functests.sh -h
 run_functests.sh [-h] [-l] <test> <test> ...
   -h : show this help
   -l : list available tests
   <test> : functional test to run
            Special test 'all' will run all tests

 $ ./run_functests.sh -l
 The available functional tests are:

  apt-sources/test-sources
  debian/build-succeeds
  fedora/build-succeeds
  fedora/build-succeeds-f21
  ironic-agent/build-succeeds-fedora
---

As described there, you can run a single test, a number of tests, the
default tests (as CI will do) or all tests.  Running all tests is too
much for regular CI, but currently the only way to stop a low priority
test running, or temporarily pause is to remove it completely --
clearly sub-optimal (see I93c2990472e88ab3e5ff14db56b4ff1b4dd965ef).

There is nothing complicated about this, and to further simplify I
have merged the runner functions back into run_functests.sh which
remains a very modest ~150 lines, with most of that being argument
sanity.  With that and the image-format cleanup, we can remove the
indirection of the 3 small library files.

For consistency, I have renamed the "dib_functions_test" (that tests
things from the dib functions library) with a run_* prefix.

Because the default list is the same as the current functional tests
run, this does not modify the status-quo.  I plan to modify this,
however, to run fedora-minimal & centos-minimal tests in a future
change, as these are required to be stable for openstack ci.

Documentation is updated, and a README.rst is added in the tests
directory for discoverability.

Change-Id: I86d208bd34ff09a29fdb916a4e7ef740c7f65af8
2016-02-19 13:50:09 +11:00
Jenkins
05edfb6d20 Merge "Don't remove python3 & grubby in 99-remove-extra-packages" 2016-02-19 01:46:38 +00:00
Ben Nemec
a342036e75 Revert "Correct rhel-common for rhel6"
This is breaking deployments that use registration.  I
believe os-refresh-config scripts don't source environment.d
files, so the variable is unset there.

Given that this a blocking issue for RHEL deployments, I'm
pushing a quick revert and we can come up with a proper fix
when this isn't stopping other work.

This reverts commit 71bd8b3a33.

Change-Id: I87504660352220d45f5445bb933edc1c28885fcb
2016-02-17 22:17:11 +00:00
Moshe Levi
14b900d3c9 Adding InfiniBand Support
This patch update the find_interface to lookup
for InfiniBand interface according to it's BOOTIF

Closes-Bug: #1532534

Change-Id: I21b91cfd10888ac036f6347a0a44cdca422830a6
2016-02-17 04:43:29 +00:00
Jenkins
352e58ceb7 Merge "yum-minimal: pre-install base packages" 2016-02-17 00:25:08 +00:00
Jenkins
b52e8f3a3b Merge "Fix tar listing in functional tests" 2016-02-16 23:45:27 +00:00
Yolanda Robla
d5a6a7cb44 Add force-confdef in debian package install
In certain cases, with packages cached that need an upgrade,
when performing that action the system hangs waiting for
a prompt.
Add force-confdef option, that will let dpkg overwrite
configuration packages that were not modified. In combination
with already existing force-confold flag, it will allow
to avoid any conffile prompt.

Change-Id: Ifb177f9ac2c9ad29f8b92309c5b8cfe8e60a4e14
2016-02-16 14:10:19 +01:00
Lucas Alvares Gomes
006b3e6228 Install IPA in a virtual environment
In order to avoid conflict with installing dependencies for IPA from
pypi and distribution packages this patch propose installing IPA in a
virtual environment.

Closes-Bug: #1506792
Change-Id: I6a4c6403813d86f4110e98449ddd874109101b9e
2016-02-16 11:17:12 +00:00
Gregory Haynes
2678209e94 Don't cache debootstrap rootfs by default
By default we create a tarball of any debootstrap rootfs we create. For
the majority of use cases this is a large performance hit for no
benefit. Lets make this an opt-in feature.

Change-Id: I58fc485aacacaa17243bf9ce760ed91256d1f182
2016-02-16 07:55:19 +00:00
Ian Wienand
b960614c9c Don't remove python3 & grubby in 99-remove-extra-packages
python3 is a hard requirement of dnf so can't be removed [1]

grubby is also required for kernel installs on Fedora.  For too much
detail see I1a6e45d04755515286b3d49f8280c16b527e2f48; but the kernel,
via dracut, now has this as a "recommends" due to people removing it
and making unbootable systems.

[1] http://logs.openstack.org/76/248976/2/check/gate-dib-dsvm-functests-devstack-f21/734c8bd/console.html

Change-Id: I5867ecd57834eece9477aa9ea4b8bdd70e238084
2016-02-16 13:40:01 +11:00
Jenkins
9e16c8c5f1 Merge "Use dnf to cleanup old kernels" 2016-02-15 22:56:12 +00:00
Jenkins
99d9373eca Merge "Fix dpkg element for Ubuntu Xenial" 2016-02-15 20:33:35 +00:00
Antoine Musso
ee58c5b1cc debian-minimal: configurable debootstrap components
The debian-minimal element creates /etc/apt/sources.list solely with
the 'main' component. I need to add 'non-free' and 'contribs'. I tried
to pass them via DIB_DEBIAN_COMPONENTS but it is not recognized.

Make debian-minimal to honor DIB_DEBIAN_COMPONENTS. Note that it is
comma separated for 'debootstrap', so replace commas with spaces to fit
the needs of sources.list.

Example usage:

 DIB_DEBIAN_COMPONENTS='main,non-free,contrib'

Will debootstrap with the three components then when debian-minimal is
realized pre install a sources.list that has:

  deb http://example.org/debian jessie main contrib non-free

Change-Id: I1dca2e8ffd31044a6b441ccb277298601e62f67c
2016-02-12 22:57:58 +01:00