Commit Graph

1787 Commits

Author SHA1 Message Date
Ian Wienand
97c01e48ed Move elements & lib relative to diskimage_builder package
Currently we have all our elements and library files in a top-level
directory and install them into
<root>/share/diskimage-builder/[elements|lib] (where root is either /
or the root of a virtualenv).

The problem with this is that editable/development installs (pip -e)
do *not* install data_files.  Thus we have no canonical location to
look for elements -- leading to the various odd things we do such as a
whole bunch of guessing at the top of disk-image-create and having a
special test-loader in tests/test_elements.py so we can run python
unit tests on those elements that have it.

data_files is really the wrong thing to use for what are essentially
assets of the program.  data_files install works well for things like
config-files, init.d files or dropping documentation files.

By moving the elements under the diskimage_builder package, we always
know where they are relative to where we import from.  In fact,
pkg_resources has an api for this which we wrap in the new
diskimage_builder/paths.py helper [1].

We use this helper to find the correct path in the couple of places we
need to find the base-elements dir, and for the paths to import the
library shell functions.

Elements such as svc-map and pkg-map include python unit-tests, which
we do not need tests/test_elements.py to special-case load any more.
They just get found automatically by the normal subunit loader.

I have a follow-on change (I69ca3d26fede0506a6353c077c69f735c8d84d28)
to move disk-image-create to a regular python entry-point.

Unfortunately, this has to move to work with setuptools.  You'd think
a symlink under diskimage_builder/[elements|lib] would work, but it
doesn't.

[1] this API handles stuff like getting files out of .zip archive
modules, which we don't do.  Essentially for us it's returning
__file__.

Change-Id: I5e3e3c97f385b1a4ff2031a161a55b231895df5b
2016-11-01 17:27:41 -07:00
Gregory Haynes
7cc65d3cbd Merge branch 'master' into feature/v2
Change-Id: I0747a27560161ef869868967ab140b5d0be9d4da
2016-10-31 11:21:38 -07:00
Jenkins
8abc90b22d Merge "Turn down yum install-packages" 2016-10-28 01:26:26 +00:00
Jenkins
5d2e7bade5 Merge "elements: source-repositories: Add git package mapping for SUSE" 2016-10-25 05:33:36 +00:00
Jenkins
c00777e761 Merge "elements: growroot: Add SUSE package mappings" 2016-10-25 05:27:38 +00:00
Jenkins
1d3d59470a Merge "Remove deprecated deploy-ironic element" into feature/v2 2016-10-24 03:55:50 +00:00
Jenkins
41dc4ab5f9 Merge "Remove deploy element" into feature/v2 2016-10-24 03:48:06 +00:00
Jenkins
dd93435b77 Merge "Remove deprecated ironic-discoverd-ramdisk" into feature/v2 2016-10-24 03:47:40 +00:00
Markos Chandras
7f1494a433 elements: zypper: Do not pull recommended packages
Recommended packages are usually useful but we normally don't need
them in order to have a working system. As a result, avoid pulling
them in when doing a regular package installation or a distribution
update. Extra packages can be pulled in using the usual '-p' parameter
or from within the elements that actually need them. The results of
this change are quite significant, resulting to gains from a few dozen
of MBs up to a few hundred depending on the selected elements.

Change-Id: I5838829c631990c7a1f3b67548accd9a603fe20c
2016-10-20 19:24:22 +01:00
Ian Wienand
a44b55ccaa Turn down yum install-packages
When debugging, this is very noisy for very little value.  If we need
to specifically debug this script we can turn up the level.

Change-Id: Ie15f16397c37e718aa919853697cbf2c5c08503c
2016-10-20 15:19:31 +11:00
Markos Chandras
cfcbd4ffbe elements: source-repositories: Add git package mapping for SUSE
The 'git' command line tool is in the git-core SUSE package

Change-Id: Ib2c5dc5ab9bbde2520f43682c654a9c3270bac09
2016-10-19 17:53:01 +01:00
Markos Chandras
1fe1e3f606 elements: growroot: Add SUSE package mappings
Add growpart and e2fsprogs package mappings for SUSE.

Change-Id: I4544c3b5bd561f7483cd10f65e2d6366b52d57cd
2016-10-19 16:16:35 +01:00
Jenkins
85354e5cd8 Merge "Add opensuse-minimal element" 2016-10-18 16:27:03 +00:00
Jenkins
e0f7b6c6d6 Merge "Add zypper-minimal element" 2016-10-18 16:24:07 +00:00
Jenkins
300d5d96a2 Merge "Remove deprecated serial-console element" into feature/v2 2016-10-14 17:13:38 +00:00
Jenkins
96ce53fa16 Merge "Fix grub installation for RHEL" 2016-10-12 14:22:49 +00:00
Jenkins
066b448d7c Merge "start cloud-init-local in the boot runlevel" 2016-10-12 02:37:04 +00:00
Jenkins
22bdaad684 Merge "Move the opensuse mkinitrd script to the zypper element" 2016-10-10 14:36:12 +00:00
Jenkins
a56daaeaf2 Merge "Add pkg-map for gentoo to runtime-ssh-host-keys" 2016-10-07 17:28:35 +00:00
Matthew Thode
82b299bbdf
start cloud-init-local in the boot runlevel
cloud-init-local needs to be run in the boot runlevel because it
modifies services in the default runlevel. When a runlevel is started
it is cached, so modifications that happen to the current runlevel while
you are in it are not acted upon.

Change-Id: Ifeae0071fc9e738ec223ec0df271559ad6e0196b
2016-10-07 11:16:28 -05:00
Jenkins
5897e4975d Merge "Disabling all previous repos registered in the system" 2016-10-07 11:05:46 +00:00
Matthew Thode
3571647692
Add pkg-map for gentoo to runtime-ssh-host-keys
Openssh is provided by default so it is not needed to be installed here.

Change-Id: Id86f9a1d214c775570f0c9e2df4ea81367bb5b7e
2016-10-06 11:57:34 -05:00
Gregory Haynes
e49c08192c Remove deploy element
This element only existed as a backwards-compat for deploy-baremetal.
Remove for v2.

Change-Id: Ic8cbd0809ccc15db749155e7af3aa0038cdb502a
2016-10-06 16:25:02 +00:00
Gregory Haynes
f73a3e3617 Remove deprecated deploy-ironic element
This element has been deprecated for some time. Remove for v2.

Change-Id: I51dcf6d8a099d551acd886349eda84bb49f48138
2016-10-06 16:20:08 +00:00
Gregory Haynes
0c55c52521 Remove deprecated ironic-discoverd-ramdisk
This element has been deprecated for some time. Remove for v2.

Change-Id: I7dd87acf1457b512fa07d0bca5a300ac58cdbb9d
2016-10-06 16:15:56 +00:00
Gregory Haynes
90bd87a686 Remove deprecated serial-console element
This element has been deprecated in favor of enable-serial-console.
Removing as part of v2.

Change-Id: I63e88b1e51db0bf21c314a741e8f61d9ee8d1566
2016-10-06 16:08:36 +00:00
Gregory Haynes
143f3414c3 Remove deprecated map-services
map-services has been deprecated for some time now, lets remove them as
part of v2.

Change-Id: I91400bf1715acc245c3a943fb5f4b46bd36e2446
2016-10-06 15:59:51 +00:00
Jenkins
12a73cf204 Merge "Default to http://ftp.us.debian.org/debian for debian-minimal" 2016-10-05 12:56:07 +00:00
Markos Chandras
90536dbab3 Add opensuse-minimal element
Add a new opensuse-minimal element to build small and highly
configurable openSUSE based images using the zypper-minimal element
as the main building mechanism

Change-Id: Iebfc4ad4aff763e511b093f1607b55851ccbddcb
2016-10-05 09:39:29 +01:00
Markos Chandras
c69c6c5a84 Add zypper-minimal element
Add a new zypper-minimal element to bootstrap SUSE-based distributions

Change-Id: Id63397e412a929d61247cfd3d9f8d4e758c1248a
2016-10-05 09:27:17 +01:00
Markos Chandras
8dde310cf3 Move the opensuse mkinitrd script to the zypper element
All SUSE-based elements can benefit from the mkinitrd phase to move it
to a more generic location.

Change-Id: Ife171d462a393b6ac0bf2c5eaa48ea25eaf4d1cc
2016-10-05 09:11:00 +01:00
Jenkins
80153520ac Merge "Move opensuse utils to zypper so they can be shared by SUSE-based distros" 2016-10-05 05:10:15 +00:00
Jenkins
04b05f261e Merge "Fix typo in extracting root partition" 2016-10-05 04:00:39 +00:00
Paul Belanger
7dc9465ed1
Default to http://ftp.us.debian.org/debian for debian-minimal
Since http://httpredir.debian.org is unreliable is selecting a mirror
to use, we'll now default to http://ftp.us.debian.org/debian.  In
fact, in openstack-infra we have been overriding httpredir.debian.org
for a while, now make this default in diskimage-builder.

Change-Id: I48658bc076e13a0913821197e4120c73618fef8f
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2016-10-01 21:10:08 -04:00
Markos Chandras
b9e9e2091e Move opensuse utils to zypper so they can be shared by SUSE-based distros
Move the opensuse utilities to the zypper element so they can be used by
SUSE or zypper based elements. This brings the zypper element somewhat
in line with the rest of the package manager elements.

Change-Id: I8aa2849231454216cdd47629a5e2d6e45769dbbe
2016-09-29 22:56:17 +01:00
Jenkins
b0d72a3161 Merge "Create runtime-ssh-host-keys element" 2016-09-27 23:10:05 +00:00
Yolanda Robla Mota
c6b4e639b0 Disabling all previous repos registered in the system
Depending on the pool id used, so many repos are brought,
including not valid ones that cause image to crash, or repos
that include conflicting packages.
Before enabling repos, disable all previous ones, so we
can be sure that we only bring the repos specified in the
parameters.

Change-Id: Ifd4d8d1d4fa954cd2593669e516e3201f2d6f6c1
2016-09-27 11:47:05 +02:00
Yolanda Robla Mota
f6e11c91c3 Fix typo in extracting root partition
Change-Id: Ie8dfd958d57ef92988647166f2031adb8406b0d7
2016-09-27 09:41:32 +02:00
Paul Belanger
45467e4229
Create runtime-ssh-host-keys element
Move managing of SSH host keys into a dedicated element.

Because glean doesn't generate SSH host keys anymore, we need to do it
with a systemd script. This is already handled by CentOS / Fedora so
we don't want to add it there.

This was done to address the upstream bug in debian:

  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=500192

Change-Id: I31ad667672e08350872db21a83445fe0aa7a4a39
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2016-09-26 12:32:38 -04:00
Jenkins
d6794ba673 Merge "Shorten DHCP timeout in dhcp-all-interfaces" 2016-09-23 14:20:09 +00:00
Jiri Stransky
93425d14ce Fix grub installation for RHEL
Grub is first removed and then installed during RHEL image building. The
grub2 package typically requires the same version of grub2-tools, so if
we just remove and install the grub2 package, the installation can
potentially fail on being out of sync with grub2-tools version. Removing
and reinstalling both packages fixes this issue. Those packages are
already in package map for RHEL as "grub-pc", so we can use this alias.

Change-Id: Iefd9c17fffd43de3fea260510ad218b1322eecb3
Closes-Bug: #1627000
2016-09-23 14:14:49 +02:00
Jenkins
3a0f46282f Merge "elements: opensuse: Add support for openSUSE Leap" 2016-09-23 08:48:46 +00:00
Jenkins
e84488cad1 Merge "Allow ramdisk-create to run without $USER set" 2016-09-23 06:46:31 +00:00
Ben Nemec
2747613ca2 Shorten DHCP timeout in dhcp-all-interfaces
We are currently wasting about 10 minutes per deploy waiting for
DHCP on interfaces that will never get it.  By default, the timeout
seems to be 5 minutes (the 10 minutes is because we boot both the
IPA ramdisk and the deployed image, and each waits for 5 minutes),
which is excessively long to get a DHCP response.  This change
shortens the time to 30 seconds.  If an interface hasn't gotten a
response in 30 seconds, chances are it's not going to.  A 30
second wait should reduce our wasted time to 1 minute, which is
more reasonable.

This is being done in the systemd unit file because the -timeout
option to dhclient doesn't seem to override what is configured in
dhclient.conf, and doing it in the systemd file means that this
change will be limited to only the interfaces configured by
dhcp-all-interfaces.

Change-Id: Ia8610e3def39c937eb0c861fdc9bc571ec39f9f4
Closes-Bug: 1626673
2016-09-22 17:01:06 -05:00
Jenkins
0559818e04 Merge "don't configure 'lo' for dhcp" 2016-09-20 23:37:42 +00:00
Jenkins
4d79e5f519 Merge "Document source glean installs in simple-init" 2016-09-20 23:05:53 +00:00
Ian Wienand
18a4109713 Merge remote-tracking branch 'origin/master' into feature/v2
Change-Id: I796546f3873a0c9072215e63703aaa64753c5afd
2016-09-20 20:23:11 +10:00
Jenkins
d8a800c638 Merge "Add libselinux-python to yum-minimal" 2016-09-20 06:23:34 +00:00
Ian Wienand
ce410de834 yum-minimal: Disable excludes when installing pkg manager
Because we are using the building platform's "yum" to do the initial
install into the chroot, it is affected by the base-system's
/etc/yum.conf.

pip-and-virtaulenv in I82acb865378a0fa5903a6267bfcee0e2962eced0 added
"exclude=python-pip..." in /etc/yum.conf to stop the package manager
overwriting the installed pip.  Now our CI images have built with
this, we are now picking up this exclude on centos.  Since on F24
dnf->python->python-pip we end up failing to build the the chroot
because python-pip can not be satisifed.  In a general sense, however,
this could be caused by any configuration put into /etc/yum.conf that
is incompatible with installing into the chroot.

yum has the option to disable all excludes which is used here.  This
seems to be the best way to isolate the chroot install from any
excludes that may have been done on the base system for various
reasons.  I did consider using a completely separate yum.conf we ship
with dib ... but let's start simple.

This should fix the current gate failures on centos

Change-Id: I4e4cc8ed09a29c4057ade34ea93025139e191bf5
2016-09-20 09:31:44 +10:00
Monty Taylor
fd18cb74b2
Add libselinux-python to yum-minimal
yum-minimal installs selinux but not libselinux-python, which makes
interacting with the node from ansible hard fail. Add it.

Change-Id: I403e7806ae10d5dd96d0727832f4da20e34b94c7
2016-09-17 01:25:31 +02:00