In shade, we use both md5 and sha256 checksums to help validate the
integrity of an image. Rather then having nodepool do this each time
for every time, have diskimage-builder create these files when we
build the image.
We've added a flag (disabled by default) to toggle this functionality.
Change-Id: I5815ba69b7d477f1e91dc8ec0c69c86168770964
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
Add a new opensuse-minimal element to build small and highly
configurable openSUSE based images using the zypper-minimal element
as the main building mechanism
Change-Id: Iebfc4ad4aff763e511b093f1607b55851ccbddcb
All SUSE-based elements can benefit from the mkinitrd phase to move it
to a more generic location.
Change-Id: Ife171d462a393b6ac0bf2c5eaa48ea25eaf4d1cc
Since http://httpredir.debian.org is unreliable is selecting a mirror
to use, we'll now default to http://ftp.us.debian.org/debian. In
fact, in openstack-infra we have been overriding httpredir.debian.org
for a while, now make this default in diskimage-builder.
Change-Id: I48658bc076e13a0913821197e4120c73618fef8f
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
Move the opensuse utilities to the zypper element so they can be used by
SUSE or zypper based elements. This brings the zypper element somewhat
in line with the rest of the package manager elements.
Change-Id: I8aa2849231454216cdd47629a5e2d6e45769dbbe
Depending on the pool id used, so many repos are brought,
including not valid ones that cause image to crash, or repos
that include conflicting packages.
Before enabling repos, disable all previous ones, so we
can be sure that we only bring the repos specified in the
parameters.
Change-Id: Ifd4d8d1d4fa954cd2593669e516e3201f2d6f6c1
Move managing of SSH host keys into a dedicated element.
Because glean doesn't generate SSH host keys anymore, we need to do it
with a systemd script. This is already handled by CentOS / Fedora so
we don't want to add it there.
This was done to address the upstream bug in debian:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=500192
Change-Id: I31ad667672e08350872db21a83445fe0aa7a4a39
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
Grub is first removed and then installed during RHEL image building. The
grub2 package typically requires the same version of grub2-tools, so if
we just remove and install the grub2 package, the installation can
potentially fail on being out of sync with grub2-tools version. Removing
and reinstalling both packages fixes this issue. Those packages are
already in package map for RHEL as "grub-pc", so we can use this alias.
Change-Id: Iefd9c17fffd43de3fea260510ad218b1322eecb3
Closes-Bug: #1627000
We are currently wasting about 10 minutes per deploy waiting for
DHCP on interfaces that will never get it. By default, the timeout
seems to be 5 minutes (the 10 minutes is because we boot both the
IPA ramdisk and the deployed image, and each waits for 5 minutes),
which is excessively long to get a DHCP response. This change
shortens the time to 30 seconds. If an interface hasn't gotten a
response in 30 seconds, chances are it's not going to. A 30
second wait should reduce our wasted time to 1 minute, which is
more reasonable.
This is being done in the systemd unit file because the -timeout
option to dhclient doesn't seem to override what is configured in
dhclient.conf, and doing it in the systemd file means that this
change will be limited to only the interfaces configured by
dhcp-all-interfaces.
Change-Id: Ia8610e3def39c937eb0c861fdc9bc571ec39f9f4
Closes-Bug: 1626673
Because we are using the building platform's "yum" to do the initial
install into the chroot, it is affected by the base-system's
/etc/yum.conf.
pip-and-virtaulenv in I82acb865378a0fa5903a6267bfcee0e2962eced0 added
"exclude=python-pip..." in /etc/yum.conf to stop the package manager
overwriting the installed pip. Now our CI images have built with
this, we are now picking up this exclude on centos. Since on F24
dnf->python->python-pip we end up failing to build the the chroot
because python-pip can not be satisifed. In a general sense, however,
this could be caused by any configuration put into /etc/yum.conf that
is incompatible with installing into the chroot.
yum has the option to disable all excludes which is used here. This
seems to be the best way to isolate the chroot install from any
excludes that may have been done on the base system for various
reasons. I did consider using a completely separate yum.conf we ship
with dib ... but let's start simple.
This should fix the current gate failures on centos
Change-Id: I4e4cc8ed09a29c4057ade34ea93025139e191bf5
yum-minimal installs selinux but not libselinux-python, which makes
interacting with the node from ansible hard fail. Add it.
Change-Id: I403e7806ae10d5dd96d0727832f4da20e34b94c7
Add support for new openSUSE Leap releases. Moreover, document
common environment variables and remove old note.
Change-Id: I8cf0b215cb4d9231e5658d49e3fd598dfbb5fd37
The previous commit removes dkms from the base element, which
means the centos elements should no longer have a dependency on
EPEL. Therefore, we should not hardcode the epel dependency. It
can still be included in image builds as desired by using the epel
element explicitly.
Co-Authored-By: Ben Nemec <bnemec@redhat.com>
Change-Id: Iceff0d5bedd9816adfd2990970e7c216b67b6bd0
The use of dkms in base was actually removed long ago in
Ic2c345bf9f0738dadae611194e263d3a5d424a3e and it is creating an
unnecessary dependency on EPEL for the centos elements.
Change-Id: Iae3100471e50a9c39f40b450f087192918ae54b3
This fix add need kernel module for Infiniband and ConnectX-4+ network
cards.
Also install by default required user space packages.
Change-Id: Ia2e7b1820f197778138a23fafaccb5a4fb44369a
On systemd-based operating systems that don't
use /etc/sysconfig/network-scripts
dhcp-all-interfaces configures 'lo' for dhcp.
This causes errors and fails networking.target
causing system-wide issues. This change excludes
'lo' at dhcp-all-interfaces udev rules level.
Closes-bug: #1621501
Change-Id: I7563b766827bedbea7ae1de35e5bdfcbf1fc0d1e
Co-Authored-By: Jeremy Stanley <fungi@yuggoth.org>
Move element-info from a wrapper script to a standard entry-point
console_script.
Update the documentation to explain how to run it for development. I
don't think we should support the idea that you can check-out the code
and run ./bin/disk-image-create -- it has dependencies (dib-utils,
etc) and needs to be run from a virtualenv (this is what CI in the
gate does). A follow-up can clean-up some of the path munging stuff
we have for this in disk-image-create.
Change-Id: Ic0c03995667f320a27ac30441279f3e6abb6bca8
Block device handling can be somewhat complex - especially
when taking things like md, lvm or encryption into account.
This patch factors out the creation and deletion of the local
loop image device handling into a python library.
The main propose of this patch is to implement the needed
infrastructure. Based on this, more advanced functions can be added.
Example: (advanced) partitioning, LVM, handling different boot
scenarios (BIOS, UEFI, ...), possibility of handling multiple images
(local loop image, iSCSI, physical hard disk, ...), handling of
different filesystems for different partitions / LVs.
Change-Id: Ib626b36a00f8a5dc3dbde8df3e2619a2438eaaf1
Signed-off-by: Andreas Florath <andreas@florath.net>
Using the explicit variables, these elements don't have to walk the
path to find the files in other elements; they can extract it directly
Change-Id: I0a64b45e9f2cfa28e84b2859d76b065a6c4590f0
These new variables are a list of elements chosen for the build along
with their full paths. For Python elements, IMAGE_ELEMENT_YAML is a
YAML formatted list that can be easily parsed. For bash elements,
"get_image_element_array" will produce an associative-array of the
same (working around lack of array export in Bash).
This list is intended for consumption of elements who need to copy
files from other elements, such as pkg-map and svc-map. As discussed
in I2a29861c67de2d25c595cb35d850e92807d26ac6, this list has already
been pruned and had overrides processed, so it is safe to simply walk
over this list with no further processing.
Since we're presenting the element list in a couple of different ways,
we combine it all into the element-info script. It will output an
eval-able string that declares the appropriate variables.
I've added some inline documentation so they still appear in grep.
The documentation is updated with examples, and moved to a more
appropriate location as a sub-section of the element sytle guide.
To test this out, use the associative-array in generate_hooks, where
we can now find the element's directory without searching.
Change-Id: Ibbd07d082ec827441def2d3f6240df3efdc6eae3
This is a re-factor of element_dependencies to achieve two things --
centralising override policy and storing path names.
Firstly we want to make the override policy for elements completely
explicit. Currently, elements that wish to copy parts of other
elements walk ELEMENTS_PATH themselves and look for elements in
IMAGE_ELEMENT. How they handle duplicate elements can differ, leading
to inconsistent behaviour.
We introduce logic in element-info to find elements in each of the
directories in ELEMENT_PATHS in *reverse* order -- that is to say,
earlier entries in the paths will overwrite later ones.
For example
ELEMENT_PATHS=foo:bar:baz
will mean that "foo/element" will override "baz/element", since "foo"
is first. This should be sane to anyone familiar with $PATH.
Documentation is clarified around this point and a test-case is added.
The second thing is that we want to keep the complete path of the
elements we have chosen. We want the aforementioned elements that
walk the element list to use these canonical paths to pickup files;
this way they don't need to make local decisions about element
overrides, but can simply iterate a list and copy/merge files if they
exist.
A follow-on change (I7092e1845942f249175933d67ab121188f3511fd) will
expose this data in a separate variable that can be parsed by elements
(a further follow-on I0a64b45e9f2cfa28e84b2859d76b065a6c4590f0
modifies the elements to use this information). Thus this does not
change the status-quo -- elements that are walking ELEMENTS_PATH
themselves and can/will continue doing that.
Change-Id: I2a29861c67de2d25c595cb35d850e92807d26ac6
Dependency to start network-pre (which
depends on network.target) before
dhcp-interface@.service collides with
Ubuntu's own network.target that suupose
to start after network-pre.
Change-Id: I9e59c970bfb1ebdaa15b4ec6b545761ede3ca056
Closes-bug: #1619816
Running the functional tests is time consuming. This patch adds the
option `-j <job count>` to the tests/run_functests.sh: when given the
test run in parallel up the <job count> jobs.
When using this, be sure to have enough resources (CPUs, RAM and HD
space) on the host.
In addition there was the need to change two things:
o Global /tmp/dib-test-should-fail was move to temporary build
directory of each execution.
o Because the logs might now interleave, each log line has now a
prefix of the name of the testcase.
[In my environment running functests sequential takes 15+ minutes,
running them parallel takes less than 6 minutes.]
Change-Id: Id9ea5131f0026c292ca6453ba2c80fe12c47f808
Signed-off-by: Andreas Florath <andreas@florath.net>
