I have seem some occasional odd failures coming from the "dnf -y
update" done by elements/base/install.d/00-up-to-date.
dnf seems to sometimes think a package is not installed when it really
is. It then seems to try and re-install them, but notices they are
installed, and then bails with a failure exit [1]. The packages that
seem to cause this vary, but the common thread is that they seem to
have all been installed during the initial phase of installing the
package manager in the chroot.
I suspect that when we are building the chroot, we do our initial
install with the "external" yum & rpm. Then we start using the
dnf/yum in the chroot, but we're actually using meta-data created by
the *external* tools -- which could be vastly different versions or
who-knows-what. While I honestly I don't have an exact root cause,
empirically I've found rebuilding the rpm db always seems to fix
things up.
So this change takes care to rebuild the rpm db with the chroot
version of rpm, and clear out the package metadata for a refresh with
"update". This should hopefully put us in a consistent state.
[1] http://paste.openstack.org/show/487356/
Change-Id: I565df23897ae511356c4861fdbe63823fa6b6ff9
In the common case of not specifying a size, we are already running
"du" over the image to figure out how big it is. Leverage that by
saving it's output and displaying a pruned list of big files when
requested.
We add a flag to show a summarised option (files >10MiB) and another
to show full output, should you wish that level of detail.
"Invocation" documentation is updated (and formatted a little better
while we're here).
Change-Id: I255800790a62fed1c82fcd311f1cc29c9867766d
If the image has an ext filesystem and the zerofree utility is present
on the build system then run zerofree. This should make images as
compressable as possible which is a nice feature when building
compressed qcow2 images.
Change-Id: Ia6062c291f7a3f58b85a4f408ecb3d0574c65d53
This element is similar to vm, but allows more flexibility
on disk partitioning. It receives a DIB_PARTITIONING_SFDISK_SCHEMA
setting, allowing to partition the disk according to project needs.
Change-Id: I01dca5d5fd670d317f7761911a1549507de6e97d
The quickstart should be the first bit of developer documentation, not
the last. Also add in a short blurb for the developer docs so we dont
have two doc titles back to back.
Change-Id: Icb5683b8eb22e759fefb1cb2252ed445dea5f7dd
Being able to discover DIB's version from the command itself is
convenient. This patch adds a --version option to the disk-image-create
command, failing gracefully if diskimage-builder is not installed.
This adds an explicit dependency on pbr to the requirements since this
is required to run diskimage_builder/version.py outside of a test
environment.
This patch consciously chooses to only provide the long-form option
and no '-v' to allow for the future possibility that a '-v' might
indicate '--verbose' in the future.
Change-Id: I9fc084774d6c7a39a944b07680b3eb8be8e34f9c
If we are being run from a python 3 virtualenv where python2 is also
available on the host we will use the hosts' python2 over the virtualenv
python3 (even if the virtualenv is higher precedence in PATH).
Change-Id: I9a25b9e45845a4121aab8250fd53c6f006825742
This patch fixes the profile autodetect. If the profile was passed at install
time, it would overridden by the environment.d script. This was fixed by
accepting the value passed by the user, if one was passed.
The enviroment.d file is run outside of chroot, because of this eselect will
not be available on non-gentoo systems. We change the behavior to work
elsewhere, it was not working on non-gentoo systems and giving bad results on
gentoo systems (returing values from the host, not the guest being built).
This patch also fixes shm detection by checking if it is mounted already.
Change-Id: Ie58d8e19529a731bfbc9eeb4bb246988d1aaa772
Newer distros, such as debian jessie and ubuntu xenial, do not provide
ifupdown by default, but simple-init depends on it. Add it to the pile.
Change-Id: I6f4876863c67c65a82464d4e0593015cdc839c5c
This simplifies and enhances the functional-test runner script for
much better interactive behaviour and to give us the ability to better
choose what is running in CI.
Firstly, I have split the image-output testing into a separate script.
This is not actually part of the functional testing of elements and is
both logically and functionally different. It currently does not run
in upstream CI because we don't have docker in the images. I have
nothing against it, but it can be it's own thing.
run_functests.sh is overhauled to have a useful interactive interface,
e.g.
---
$ ./run_functests.sh -h
run_functests.sh [-h] [-l] <test> <test> ...
-h : show this help
-l : list available tests
<test> : functional test to run
Special test 'all' will run all tests
$ ./run_functests.sh -l
The available functional tests are:
apt-sources/test-sources
debian/build-succeeds
fedora/build-succeeds
fedora/build-succeeds-f21
ironic-agent/build-succeeds-fedora
---
As described there, you can run a single test, a number of tests, the
default tests (as CI will do) or all tests. Running all tests is too
much for regular CI, but currently the only way to stop a low priority
test running, or temporarily pause is to remove it completely --
clearly sub-optimal (see I93c2990472e88ab3e5ff14db56b4ff1b4dd965ef).
There is nothing complicated about this, and to further simplify I
have merged the runner functions back into run_functests.sh which
remains a very modest ~150 lines, with most of that being argument
sanity. With that and the image-format cleanup, we can remove the
indirection of the 3 small library files.
For consistency, I have renamed the "dib_functions_test" (that tests
things from the dib functions library) with a run_* prefix.
Because the default list is the same as the current functional tests
run, this does not modify the status-quo. I plan to modify this,
however, to run fedora-minimal & centos-minimal tests in a future
change, as these are required to be stable for openstack ci.
Documentation is updated, and a README.rst is added in the tests
directory for discoverability.
Change-Id: I86d208bd34ff09a29fdb916a4e7ef740c7f65af8
This is breaking deployments that use registration. I
believe os-refresh-config scripts don't source environment.d
files, so the variable is unset there.
Given that this a blocking issue for RHEL deployments, I'm
pushing a quick revert and we can come up with a proper fix
when this isn't stopping other work.
This reverts commit 71bd8b3a33.
Change-Id: I87504660352220d45f5445bb933edc1c28885fcb
This patch update the find_interface to lookup
for InfiniBand interface according to it's BOOTIF
Closes-Bug: #1532534
Change-Id: I21b91cfd10888ac036f6347a0a44cdca422830a6
In certain cases, with packages cached that need an upgrade,
when performing that action the system hangs waiting for
a prompt.
Add force-confdef option, that will let dpkg overwrite
configuration packages that were not modified. In combination
with already existing force-confold flag, it will allow
to avoid any conffile prompt.
Change-Id: Ifb177f9ac2c9ad29f8b92309c5b8cfe8e60a4e14
In order to avoid conflict with installing dependencies for IPA from
pypi and distribution packages this patch propose installing IPA in a
virtual environment.
Closes-Bug: #1506792
Change-Id: I6a4c6403813d86f4110e98449ddd874109101b9e
By default we create a tarball of any debootstrap rootfs we create. For
the majority of use cases this is a large performance hit for no
benefit. Lets make this an opt-in feature.
Change-Id: I58fc485aacacaa17243bf9ce760ed91256d1f182
python3 is a hard requirement of dnf so can't be removed [1]
grubby is also required for kernel installs on Fedora. For too much
detail see I1a6e45d04755515286b3d49f8280c16b527e2f48; but the kernel,
via dracut, now has this as a "recommends" due to people removing it
and making unbootable systems.
[1] http://logs.openstack.org/76/248976/2/check/gate-dib-dsvm-functests-devstack-f21/734c8bd/console.html
Change-Id: I5867ecd57834eece9477aa9ea4b8bdd70e238084
The debian-minimal element creates /etc/apt/sources.list solely with
the 'main' component. I need to add 'non-free' and 'contribs'. I tried
to pass them via DIB_DEBIAN_COMPONENTS but it is not recognized.
Make debian-minimal to honor DIB_DEBIAN_COMPONENTS. Note that it is
comma separated for 'debootstrap', so replace commas with spaces to fit
the needs of sources.list.
Example usage:
DIB_DEBIAN_COMPONENTS='main,non-free,contrib'
Will debootstrap with the three components then when debian-minimal is
realized pre install a sources.list that has:
deb http://example.org/debian jessie main contrib non-free
Change-Id: I1dca2e8ffd31044a6b441ccb277298601e62f67c
Gentoo provides eclean-dist via the gentoolkit package
eclean-dist isn't needed anyway was /usr/portage is removed anyway
Removed redundant package update in cleanup.
Change-Id: Icf4f9ed549b9a6d923448d927d7c42bcf8d6091b
This action has been reordered so cleanup can occur before copy to blockdev
occurs. Documentation has been added about the ordering of this element in
relation to cleanup actions as well.
Change-Id: I3f9334a3669ee588d7fa7129202c97fa22fdb050
While it does save 1MiB of space, it might also pull python packages
depending on it. E.g. it makes impossible to install python-hardware
on the IPA image for advanced introspection.
Change-Id: Iab80dde63e6de62a5e45dcf404b4f9f633e50ac3
We were getting some subtle issues in fedora-minimal builds that
turned out to be because /var/run was not a symlink to /run.
Upon further investigation, it turns out that yum is creating a
/var/run directory for it's pid file when it starts working in the
empty chroot (which I verified by stracing it)
---
5905 stat("/home/ubuntu/tmp/dib-tmp/image.Ac4VZZsl/mnt/var/run", 0x7ffddffa0330) = -1 ENOENT (No such file or directory)
5905 mkdir("/home/ubuntu/tmp/dib-tmp/image.Ac4VZZsl/mnt/var/run", 0755) = 0
5905 open("/home/ubuntu/tmp/dib-tmp/image.Ac4VZZsl/mnt/var/run/yum.pid", O_WRONLY|O_CREAT|O_EXCL, 0644) = 6
---
Because this happens *before* we install "filesystem" (the package),
we mess up it's symlinking.
To work-around this, pre-install the trio of base packages (setup,
basesystem, filesystem) with rpm from outside the chroot.
Change-Id: I411b6ec9d91d95d3a0f98e76853086af3b70abe8
The Ubuntu Xenial cloud server images set the mode of
/var/lib/apt/lists/partial to 700, so when mounted it's inaccessible to
an unprivileged user, resulting in an error:
find: `/tmp/image.aDQKdkRi/mnt/var/lib/apt/lists/partial': Permission denied
There's no reason an image should come with anything already in
/var/lib/apt/lists/partial, so just avoid trying to descend into that
directory when fixing the apt translations packages.
Change-Id: Id27f0166bfb09d67200f337a5ffff2f2037b7c1c
This is a slight refactor I found useful when debugging. The
udevadmin info query will be visbile in the output of "set -x" which
helps, and is the logs/journal.
We can also reduce some calls by keeping the value and just grepping
with a herefile.
This also does some error checking and bails out if it does not see
what it needs to continue.
Change-Id: I39c4d262f9c5ce53f6b83d95b1363a74834cf2c8
Tucked away in systemd-udev-settle.service is the following comment
# This service can dynamically be pulled-in by legacy services which
# cannot reliably cope with dynamic device configurations, and
# wrongfully expect a populated /dev during bootup.
The info that the growroot script is querying is populated via udev,
particularly the blkid bits of [1]. This creates a race-condition
where sometimes udev has been triggered and the rules have applied and
sometimes not. Obviously in the first case, the root disk is not
grown correctly.
systemd-udev-settle is mostly disabled on distros because it can cause
an increase in boot-time for systems with lots of disks; this is not
our situation so it makes basically no difference.
That said, I will investigate if some systemd people know even better
ways to do this (possibly the service should depend on block .device
targets in systemd, and then filter out and only apply to the root
disk?)
[1] https://github.com/systemd/systemd/blob/master/rules/60-persistent-storage.rules#L66
Change-Id: I453e3afcd953dfc29ab6c42ddc81e940cfa70ee0
