Commit Graph

592 Commits

Author SHA1 Message Date
Ian Wienand
a0f747932d Install systemd earlier for Stretch
Debian Stretch released as stable recently, and the init system is
less tightly specified in the base dependencies (for some info, see
[1]).  It seems, probably unintentionally, that in the previous
release systemd-sysv was brought in by debootstrap, but that is no
longer happening.

Add systemd as an early dependency of debian-minimal.

Remove the package-installs.yaml as that happens too late (other
things need to know the init system to write out service files, etc
and probe for systemd utils before package-installs).  As mentioned, I
do not believe the "only install systemd on testing" idea was actually
working here, because it was being brought in during the initial
debootstrap.

Update some documentation to explain what's going on

[1] https://lists.debian.org/debian-boot/2015/05/msg00156.html

Change-Id: Id67c0cf08728407d234976f9807d3bd71d12f758
2017-06-19 13:27:33 +10:00
Jenkins
1324f5b7db Merge "Remove use of 'which'." 2017-06-11 09:30:34 +00:00
Ian Wienand
5ac8a98e9a PPC bootloader; install to boot partition
Using the newly exposed variables from the prior change, install the
ppc bootloader to the boot partition, not the underlying loopback
device.

Change-Id: I0918e8df8797d6dbabf7af618989ab7f79ee9580
2017-06-08 17:14:22 +10:00
Ian Wienand
6c394f5746 Pass all blockdevices to bootloader
Currently we only export "image-block-device" which is the loopback
device (/dev/loopX) for the underlying image.  This is the device we
install grub to (from inside the chroot ...)

This is ok for x86, but is insufficient for some platforms like PPC
which have a separate boot partition.  They do not want to install to
the loop device, but do things like dd special ELF files into special
boot partitions.

The first problem seems to be that in level1/partitioning.py we have a
whole bunch of different paths that either call partprobe on the loop
device, or kpartx.  We have _all_part_devices_exist() that gates the
kpartx for unknown reasons.  We have detach_loopback() that does not
seem to remove losetup created devices.  I don't think this does
cleanup if it uses kpartx correctly.  It is extremley unclear what's
going to be mapped where.

This moves to us *only* using kpartx to map the partitions of the loop
device.  We will *not* call partprobe and create the /dev/loopXpN
devices and will only have the devicemapper nodes kpartx creates.
This seems to be best.  Cleanup happens inside partitioning.py.
practice.  Deeper thinking about this, and more cleanup of the
variables will be welcome.

This adds "image-block-devices" (note the extra "s") which exports all
the block devices with name and path.  This is in a string format that
can be eval'd to an array (you can't export arrays).

This is then used in a follow-on
(I0918e8df8797d6dbabf7af618989ab7f79ee9580) to pick the right
partition on PPC.

Change-Id: If8e33106b4104da2d56d7941ce96ffcb014907bc
2017-06-08 17:14:22 +10:00
Ian Wienand
90b56b3aab Move ppc block-device default to right $ARCH
The supported ppc ${ARCH} is "ppc64el" (at least in the gate testing
...) so move the file to that, so gets picked up by
block_device_create_config_file

Change-Id: I9273f35cdbfb0a62404461cbc1df9b2a92155fb0
2017-06-07 13:30:38 +10:00
Jenkins
ec70cb61f0 Merge "Trivial fix typos" 2017-06-05 05:54:50 +00:00
Jenkins
5a045e036d Merge "dhcp-all-interfaces.sh - Add support for InfiniBand interface DHCP" 2017-06-02 06:11:19 +00:00
Jenkins
80cc1d0ea4 Merge "Adjust package mapping for SUSE family" 2017-06-02 02:56:16 +00:00
Dirk Mueller
d0a398c167 Adjust package mapping for SUSE family
package-installs.yaml is installing python-dev, not python2-dev,
so we need to adjust the mapping accordingly.

In addition, zypper-minimal used an dpkg specific package name,
while there is a SUSE equivalent (and zypper-minimal is anyway
SUSE family specific)

Change-Id: Ia9dd061fa46a514781808d62e5e93b03f75c6745
2017-05-31 21:09:53 +02:00
Dirk Mueller
f58bf252de Drop support for Ubuntu precise
Ubuntu 12.04 LTS reached its regular End of Life on April 28, 2017.

Depends-On: I5e145095a10db112bb27516bfe652d2cdc052a61
Change-Id: I64af4c5183d77a75dcd062895d19b0a1330c8da8
2017-05-31 14:36:30 +02:00
Jenkins
b312c06dbb Merge "Decode string to bytes in dracut-regenerate" 2017-05-31 10:49:51 +00:00
Jenkins
d0e0714f71 Merge "Test openSUSE 42.2/42.3 image builds" 2017-05-31 04:37:57 +00:00
Vu Cong Tuan
6a72052108 Trivial fix typos
Change-Id: Ib86aa9938fd852610ec0a6d8d868181f87bd2f24
2017-05-31 11:17:05 +07:00
Jenkins
2bdc154df5 Merge "drop deprecated map-services/packages from zypper element" 2017-05-31 02:11:21 +00:00
Jenkins
05d64b99ce Merge "Remove ccache" 2017-05-31 01:48:01 +00:00
Jenkins
edaf577bad Merge "Remove dracut-network element" 2017-05-31 00:14:01 +00:00
Mark Goddard
54765fd2f4 Remove dracut-network element
This element has not been functioning correctly for some time due to
an incorrect path to select-boot-kernel-initrd (should be /usr/local/bin).

The dracut-regenerate element can be used to regenerate dracut ramdisks
and is more flexible than this element.

Change-Id: I33d555ffd4a92b2948b2ea4a66b151f0422ccb8c
Closes-Bug: #1688546
2017-05-31 08:36:56 +10:00
Andreas Florath
b107606a75 Remove ccache
This patch removes the ccache handling from the base element.  For
mostly all systems this was never used at all.

This is working towards the removal of the base element from DIB

Change-Id: Ieb16ef612ebd98470993dcd6f55b3a22d37084ba
Signed-off-by: Andreas Florath <andreas@florath.net>
2017-05-31 08:28:09 +10:00
Mark Goddard
aa6c1d01a9 Decode string to bytes in dracut-regenerate
In python3, the standard out data returned by
subprocess.Popen.communicate() will in most cases be bytes rather than a
string and must therefore be decoded.

Without this fix we hit the following error:

TypeError: a bytes-like object is required, not 'str'

Change-Id: I6d75f867ebfdb925970c3397175214b9050d7632
Closes-Bug: #1694463
2017-05-30 16:15:06 +01:00
Dirk Mueller
5d39f83f74 Test openSUSE 42.2/42.3 image builds
Currently openSUSE 42.3 has entered feature freeze mode
so it is a good point in time to verify that 42.3 builds
are working successfully. Also test opensuse-minimal for
platforms that support it (need working zypper package)

Change-Id: I4c613e1e68cb7375c29d544bbf70b5da9bf21414
2017-05-30 13:07:04 +02:00
Matthew Thode
ce7ea9d34c
allow uninstalls to fail on gentoo
The cleanup of packages should be more opertunistic, if it's not there
then fail quietly.

Change-Id: I207a1162abc9ca5e9636b8de192f21424db0f569
2017-05-29 23:46:42 -05:00
Dirk Mueller
b4edb7d0eb Disable recommended package installations for zypper-minimal
This is consistent with how dpkg based images are configured
and minimizes the nodepool images drastically (avoid installing
texlive for example)

Change-Id: I98fb31bc0e06869e9770fae3dbd62f0d86acb879
2017-05-26 09:47:07 +02:00
Dirk Mueller
f039a9b796 drop deprecated map-services/packages from zypper element
Change-Id: Ie3065dcc6aefccba93c02085e9977681d1b0535c
2017-05-25 23:43:21 +02:00
Jenkins
0208f83a97 Merge "Set manifest permissions in the image" 2017-05-24 06:58:10 +00:00
Noam Angel
f1369a1add Set manifest permissions in the image
This is a follow-on to 57ef187632.

There's two things going on here; DIB_MANIFEST_IMAGE_DIR is *outside*
the chroot on the build host.  We copy the files here for posterity, I
guess.  MANIFEST_IMAGE_PATH is *inside* the chroot and are the files
we want to ensure are locked to root.

The prior change modified the permissions on DIB_MANIFEST_IMAGE_DIR.
So the first time you build, it works -- then the second time,
assuming you're using the same output filename, it hits the root-owned
manifest directories and causes a build failure.

I have built with this and checked that the manifest files in the
image are locked to root:

 $ virt-ls -a ./test.qcow2 -l /etc/dib-manifests
 total 32
 drwxr-xr-x  2 0 0  4096 May 24 03:39 .
 drwxr-xr-x 53 0 0  4096 May 24 03:39 ..
 -rw-------  1 0 0 15236 May 24 03:39 dib-manifest-dpkg-test
 -rw-------  1 0 0    35 May 24 03:39 dib_arguments
 -rw-------  1 0 0   137 May 24 03:39 dib_environment

Related-Bug: #1671842
Change-Id: I08319d0b5fcc461d40fe0be8427dcf0e37ad21e6
2017-05-24 15:20:55 +10:00
Jenkins
57c40a2ac4 Merge "Add dracut-regenerate elements" 2017-05-23 07:35:51 +00:00
Angel Noam
ba4f72f4f0 dhcp-all-interfaces.sh - Add support for InfiniBand interface DHCP
Change-Id: Ic2a9e2909a8086903257d43fbda97694baa339b4
2017-05-22 07:48:32 +00:00
Ian Wienand
9eb71a1fe0 Switch debian to deb.debian.org
Per [1] this is the "official" CDN mirror, which I think is the most
appropriate for the default.  I think this addresses the concerns
httpredir service, which I don't think ever quite got out of beta.

[1] https://wiki.debian.org/DebianGeoMirror

Change-Id: I55f2a00b8bbb0f0a20d3be229e4c2c32a7b69057
2017-05-19 20:10:40 +00:00
Jesse Keating
84d10dce57 Remove use of 'which'.
Instead, either use the bash built-in of type to ensure it exists. Since
which is an external dep, things can fail oddly in a constrained
environment.

Also add a dib-lint test for this.

Change-Id: I645029f5b5bfe1198c89ce10fd3246be8636e8af
Signed-off-by: Jesse Keating <omgjlk@us.ibm.com>
2017-05-19 12:43:36 -07:00
Yolanda Robla
f23ea63341 Add dracut-regenerate elements
This new element will allow to regenerate dracut
on the produced images, to enable different modules. It
relies on a yaml blob to specify modules and packages
needed. It defaults to installing lvm and crypt.

Change-Id: I292fb70cde41ee6053b7b81a67931bcdaaa6d664
2017-05-19 09:43:11 +02:00
Gregory Haynes
57ef187632 Set manifests to mode 600 and owner root
Manifests files can release sensitive information and therefore should
have restrictive permissions.

Change-Id: I64d6c830217a7d8b0172df2dc774079dcd1e2a68
Related-Bug: #1671842
2017-05-18 10:10:15 +10:00
Yolanda Robla
6d0b9abc0f Apply setfiles on all mountpoints
With new block device definition, where content of the image
can be mounted on different partitions, is not enough with
executing setfiles on root directory. Instead of that, expose
all the mountpoints on the image, and apply setfiles on them.

Change-Id: I153f979722eaec49eab93d7cd398c5589b9bfc44
2017-05-16 07:51:48 +02:00
Andreas Florath
e4e23897a1 Refactor: block-device filesystem creation, mount and fstab
This patch finalizes the block device refactoring.  It moves the three
remaining levels (filesystem creation, mount and fstab handling) into
the new python module.

Now it is possible to use any number of disk images, any number of
partitions and used them mounted to different directories.

Notes:

 * unmount_dir : modified to only unmount the subdirs mounted by
   mount_proc_sys_dev().  dib-block-device unmounts
   $TMP_MOUNT_PATH/mnt (see I85e01f3898d3c043071de5fad82307cb091a64a9)

Change-Id: I592c0b1329409307197460cfa8fd69798013f1f8
Signed-off-by: Andreas Florath <andreas@florath.net>
Closes-Bug: #1664924
2017-05-12 13:52:02 +02:00
Yolanda Robla
fb70a49ba2 Add a test to validate we can build debian vms
This is a partial refactor from change
I592c0b1329409307197460cfa8fd69798013f1f8

Change-Id: I8822e68e41c4ebd47eea9ffed4557efc130a7bf7
Co-Authored-By: Andreas Florath <andreas@florath.net>
2017-05-05 19:17:39 +02:00
Jenkins
83ea4a17b8 Merge "Clean out apt index caches at end of image build" 2017-05-01 08:09:07 +00:00
Jenkins
ead4a50c2a Merge "Make Gentoo package updates work more often" 2017-05-01 08:05:04 +00:00
Jenkins
c36ac99458 Merge "Turn off strict_id mode for Ec2 datasource" 2017-05-01 08:04:52 +00:00
Matthew Thode
6f51fbe355
Make Gentoo package updates work more often
Some package updates are more complex and require things like --backtrack=99 to
be passed to emerge.  We also try harder to ensure the system is in a consistent
state as a last step.

Change-Id: Ia5d3514e8b2a6cb2d656ade997cebb798d9c0a47
2017-04-22 19:34:11 -05:00
Paul Belanger
1ce16a987b
Add yum-utils as EPEL dependency
With 8e822768f9 we added the ability to
disable the EPEL repository, however we need yum-utils to use
yum-config-manager.

Change-Id: Iea445f84494fd9a89fd93e9b35f920eb5e55211d
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2017-04-20 17:01:00 -04:00
Simon Leinen
0ff40cf63c Turn off strict_id mode for Ec2 datasource
Recent changes in the default configuration of cloud-init in Ubuntu
cause warnings when the Ec2 datasource is used on non-Amazon clouds,
see https://bugs.launchpad.net/cloud-init/+bug/1660385

We explicitly select the previous behavior when an Ec2 datasource is
desired.

Change-Id: Iebad8f6c0017fe08013dd5fe667c6132158b71cd
Closes-bug: 1683038
2017-04-19 09:47:56 +02:00
Paul Belanger
3e17ee3f48 Clean up dib-python symlink
We only need dib-python when we build the image, no need to leak it to
the final product.  Remove it in cleanup.d outside the chroot so
nothing can be using it.

Change-Id: I1e229caad7968fb3ab8e44ecdda427e174088d2d
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2017-04-12 14:33:57 +10:00
Ian Wienand
7a962e9d1c Skip python3-virtualenv on <= trusty
If DIB_PYTHON_VERSION is < 3 on the !redhat path, that means we're on
an older platform that may not have python3-virtualenv packages.  Skip
install.

Ensure the order of operations happens by forcing the installs

Also add a note about limited platform support (patches welcome :)

Change-Id: I18412767f0ebf946d557a0a126285369e96af159
2017-04-12 06:36:20 +10:00
Ian Wienand
79d4113cbe pip-and-virtualenv : install python2 & 3, and default to 2
Recent changes in project-config have shown that we leave the system
in an inconsistent state when installing from source.  On fedora, we
will have installed the python2 packages, but then used $DIB_PYTHON to
install python3 pip from source!

This tries to clarify the situation.  As described in the document,
with package installs, we just install the $DIB_PYTHON packaged
versions.

Source installs want to take over the global namespace.  This is the
price you pay for running the latest versions outside package managers
:) The only sane thing seems to be for us to normalise python2 &
python3 versions of pip, setuptools and virtualenv and then hacking
things such that "/usr/bin/pip" and "/usr/bin/virtalenv" remain
defaulted to python2 versions.

Documentation is added

Change-Id: Ibc6572b89e256d1f48b7fe7c672b8b9524dc704f
2017-04-11 18:59:11 +10:00
Ian Wienand
ffd4820d59 Install pip with python interpreter
Currently we install pip/virtualenv with "/usr/local/bin/dib-python".
This means that every time you create a virtualenv, the python
interpreter inside it is called "dib-python" which is confusing.

Add an env var DIB_PYTHON that points directly the to interpreter
available during build, for use when running scripts.

Change-Id: I88ad3c9eb958d58db4631d9b27bc2c592f970345
2017-04-11 18:59:09 +10:00
Jenkins
5c1579f0d6 Merge "Unify and fix doc of several Debian and Ubuntu elements" 2017-04-10 01:25:01 +00:00
Paul Belanger
1778fb57db Clean out apt index caches at end of image build
Apt gets confused if it talks to a mirror with an older index than the
index currently cached by apt. This can happen when image builds use a
newer index than the booted image. Avoid these problems entirely by
removing those index caches at the end of image building.

Change-Id: I245d516ee8a44831b2c29612b782bad555c48a3f
Co-Author: Clark Boylan <clark.boylan@gmail.com>
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2017-04-09 11:57:23 -04:00
Jenkins
3f5c558753 Merge "Fix package-installs-v2 output" 2017-04-07 13:24:17 +00:00
Jenkins
1b5045c563 Merge "Basic logging for package-installs-squash" 2017-04-07 13:24:10 +00:00
Jenkins
08ad06f86d Merge "Fix opensuse-minimal element on non-SUSE hosts" 2017-04-07 12:06:31 +00:00
Andreas Florath
ee8fa3c4bb Unify and fix doc of several Debian and Ubuntu elements
This patch removes three nearly-copies of debootstrap documentation
and fixes some documentation aspects.

Change-Id: Ief7794f5c1abad73788c063af6c862472cd34744
Signed-off-by: Andreas Florath <andreas@florath.net>
2017-04-07 05:56:50 +00:00
Ian Wienand
f068e6aa6e Fix package-installs-v2 output
The current output for package-installs-v2 is inscrutable [1]

The problem starts with process_output() which is not capturing
stderr.  This means that any stderr output is dislocated from any
stdout output around it.  This is *really* confusing as you get a
bunch of seemingly meaningless stderr output from any calls before you
see any stdout (e.g. in [1] you can see random yum error output that
should have been with the yum call)).  The simplest thing to do is to
redirect stderr to stdout which keeps everything in sync.

This causes a slight problem, however, because pkg-map outputs both
status information and errors on stderr.  To work around this but
maintain compatibility, we add a "--prefix" argument that prepends
mapped packages from pkg-map with a value we can match on.  The
existing status/debug output from pkg-map is low-value; modify the
call so that it will be traced only at higher debug levels (e.g. -x
-x).

The current loop is also calling pkg-map for every package in every
element (this is why in [1] the same message is repeated over and
over).  This is unnecessary; it only needs to pkg-map once for each
element, giving the package list as the arguments.  Create package
lists by element and pass those to pkg-map.

As a cleanup, there is no point in printing e.output if the
process_output fails for the install because we are already tracing
it; i.e. the output, even for failures, is already in the logs.
Printing it again just duplicates the output.

[2] is an extract showing what I feel is a much more understandable
log output for a fairly complex install.

[1] http://paste.openstack.org/show/595118/
[2] http://paste.openstack.org/show/595303/

Change-Id: Ia74602a5d2db032a476481caec0e45dab013d54f
2017-04-07 13:48:53 +10:00
Ian Wienand
facabe585c Basic logging for package-installs-squash
Some basic info logging and debugging for package-installs-squash.

Change-Id: If7c227cf0a53b71d334b5f5a6075120f2a7bf3a7
2017-04-07 13:48:52 +10:00
Corey O'Brien
0ea7e927de Fix typo in pip-and-virtualenv
Change-Id: I3058b45fff037106eba0267fd6629707a5ebb8b1
2017-04-06 10:19:40 -04:00
Jenkins
10baebe1e4 Merge "Run dib-run-parts out of /tmp" 2017-04-05 10:18:55 +00:00
Jenkins
0279258ac6 Merge "Don't provide dib-run-parts" 2017-04-05 07:01:29 +00:00
Jenkins
221c44f24c Merge "Adding aarch64 support for CentOS7" 2017-04-05 05:10:05 +00:00
Ian Wienand
6802cf7100 Run dib-run-parts out of /tmp
The dib-run-parts element was copying our internal version of
dib-run-parts into /usr/local/bin to be used running scripts inside
the target chroot.  However, it never cleaned up after itself.  This
means all images were left with an unmanaged local install of
dib-run-parts.

This copies dib-run-parts into the hooks directory of the chroot and
runs it from there.  It is cleaned up automatically on the exit path.

The dib-run-parts element is no longer required and it has been
removed from all dependencies.  It is left with a deprecation notice
in the README.  For compatability we convert it to simply install
dib-utils.

Codesearch shows no users depending on this unintentional implicit
install.  Note os-refresh-config depends on dib-utils and thus will
have an explicitly installed version.

Partial-Bug: #1673144
Change-Id: Ia2e96c00a4246c04beb96c17f83b8aefb69219ca
2017-04-05 13:11:22 +10:00
Ian Wienand
fd424757a6 Don't provide dib-run-parts
It was an oversight during v2 development for dib to start providing
dib-run-parts.  The intention was for dib to use a vendored
dib-run-parts directly from $_LIB and have no dependencies on
dib-utils at all.  By exporting dib-run-parts, we created an
unintentional conflict with the dib-utils package which provides the
same script.

Tools that depend on dib-utils are unaffected by this
(os-refresh-config).

The only tool that installs diskimage-builder and then assumes
dib-run-parts is available in the path is instack.  I have proposed
Ibfe972208df40fa092b11b5419043524c903f1b4 to modify that to use our
internal version.

Change-Id: I149c345d38d761a49b3a6ccc4833482f09f1cd05
2017-04-05 13:11:20 +10:00
Ian Wienand
8e822768f9 Add flag to disable EPEL
Add DIB_EPEL_DISABLED flag that allows installation of the EPEL repo,
but to have it disabled by default.  This will help when you have
unavoidable EPEL dependencies, but want to make sure you only pull
specific things in with "--enablerepo" calls when installing those
packages.

Change-Id: Iedf6167a7cd69418255ebbee095aea04c50d73fd
2017-04-05 05:59:40 +10:00
Hironori Shiina
eaa0569ac5 Fix code-block in README of rhel7 element
A code-block in README of rhel7 element is not rendered as expected.
This patch fixes it to be rendered correctly.

Change-Id: Ie8f4c05edd1dd93314290682e4b2734622894e15
2017-04-03 03:56:22 +00:00
Dirk Mueller
1bdeaadc95 Fix opensuse-minimal element on non-SUSE hosts
zypper on non-suse hosts is not parsing the pattern repodata because
those are marked as an inofficial extension to the repomd specification.
This is not a big issue as there is meanwhile in newer openSUSE
distributions a pattern *package* that depends on the same packages like
the pattern would do, so we can just replace it with that.

Change-Id: I0c8f713075bd7e5bf1d425f81933b4666654add7
Depends-On: I34e98f0f7693859ed05011b008334628adff612f
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2017-04-02 19:37:01 -04:00
Paul Belanger
87236af75c Have simple-init enable network.service
When a glean is running on centos with multiple NICs, it will try to
systemctl enable network.service multiple times for each interface.
Because of systemd magic, it is possible for the systemctl command to
fail in a race condition.

glean shouldn't be enabling network.service during boot in
pre-networking phases (Ib2b618dd975ca44e9c6b0a2c9027642ffc46b9b0).  I
have proposed I8319f1ed6498a9d447950c2b4b34bca59e7b97e4 to remove this
and document the behaviour.

This also bring across suse's version
(I20bffabd333ea290d8712ec2a467f2b2d5678f3a)

Change-Id: I89d9443cb61e287bd0d9da3f48315272218ee335
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2017-03-28 19:28:51 +11:00
Jenkins
f128d08cf0 Merge "Use correct Ubuntu distro url on non-x86 arches" 2017-03-24 04:42:59 +00:00
Jenkins
94e2973785 Merge "functests: skip qcow2 generically but add specific test" 2017-03-24 01:07:10 +00:00
Mikhail S Medvedev
4d4417c9a6 Use correct Ubuntu distro url on non-x86 arches
diskimage-builder usually provides defaults that work out of the box.
One default that does not work outside of x86 land is Ubuntu distro
mirror url. Considering there are only two valid default options, we can
automatically choose a better default.

This patch changes behavior only for architectures known to be using
http://ports.ubuntu.com/ubuntu-ports. All others still would use
http://archive.ubuntu.com/ubuntu as default. It provides some guarantee
that we do not introduce a regression.

Change-Id: If95a64bac0c88f30736da4bae7f1fdce126c0bf6
2017-03-23 09:27:25 -05:00
gecong1973
8ff6f51b81 Typo fix: curent => current
Change-Id: I5ce5a98c355310d6f5f6c09e19993b01ed025f0e
2017-03-23 15:02:20 +08:00
Ian Wienand
76ae374413 functests: skip qcow2 generically but add specific test
We somewhat discussed skipping qcow2 generation previously in
I9372e195913798a851c96e62eee89029e067baa1.  As recent issues with PPC
testing have shown, we are not actually testing the "vm" element and
hence the bootloader path in the functional tests.

I don't think we need to test this on every element; it overlaps
somewhat with the testing done by the nodepool jobs which build full
images and boot them.  I also didn't want to introduce a separate run
for this.  Thus it seems valuable to at least have one element
enhanced to do this installation and conversion in our default tests
for basic sanity.

This disables qcow generation by default, as per the other change, but
allows an element to drop a file that will override the output
formats.  The Xenial element is modified to produce a qcow2 using
this, and also introduces a dependency on the "vm" element so it tries
to install the bootloader.

We now exit if the .qcow2 fails to build as well.

Change-Id: I1a6acefe52f8c696c39b2d592fdc7ae32a87e6fe
2017-03-23 13:49:24 +11:00
Jenkins
2ea3a434f7 Merge "Add default PPC block-device layout" 2017-03-23 01:03:13 +00:00
Ian Wienand
19fcd263d0 Add default PPC block-device layout
Add a default PPC block-device layout.  I've extracted this into
separate yaml files for ease of editing and to facilitate things like
longer comments.

This is not sufficient to get PPC images working, but it is required.

Change-Id: I09e5d1ed92260bdb632333f5203dd7e70d512dc8
2017-03-23 09:44:01 +11:00
Dan Radez
06576a02f0 Adding aarch64 support for CentOS7
Using delorian and delorian-deps for aarch64
this will produce an overcloud-full image

Change-Id: Id0dc83d9c558b0c0bbfaed7727883541d140472c
2017-03-22 10:46:54 -04:00
Yolanda Robla
8a3a3ffe6e Fix typo in CMDLINE env var for bootloader
The correct spelling of the env var needs to be
DIB_BOOTLOADER_DEFAULT_CMDLINE

Change-Id: I4ae2aed928723588e736b63283a54eca5f4b4c3b
2017-03-22 12:02:37 +01:00
Jenkins
f1d53f2e31 Merge "Use sphinx warning-is-error" 2017-03-15 23:56:59 +00:00
Ian Wienand
fea6ab1624 Use sphinx warning-is-error
Sphix 1.5 (I9e7261c4124b71eeb6bddd9e21747b61bbdc16fa) includes
"warning-is-error" which supersedes pbr's warnerrors.  Enable this and
fix up the resulting failures

 - trailing lines for lists in element_deps directive
 - missing README's that are linked
 - syntax error and highlighting in building instructions

Change-Id: I6549551b4a9bf47076c9811a7a38a666cbea2a50
2017-03-14 14:49:49 +11:00
Michael Johnson
fae8004e4d squash-package-install to use the correct python
99-squash-package-install in the package-installs element does not
know which python environments the requirements were installed into.
This can cause it to select the wrong python to run the
package-installs-squash script.

Co-Authored-By: Adam Harwell <flux.adam@gmail.com>
Change-Id: I5fab0e192c3a2dad8f60e821c184479e24e33bcd
2017-03-13 20:24:39 +11:00
Jenkins
8b19f7f99d Merge "Fix dib-init-system for Debian Jessie and Debian Stretch" 2017-03-13 03:29:41 +00:00
Andreas Florath
20389d755f Fix dib-init-system for Debian Jessie and Debian Stretch
On Debian Jessie and Debian Stretch systemctl is in /bin.
If the package systemd-sysv is not installed the script
dib-init-system did not find the init system.
This patch fixes the problem: it also looks in /bin
for systemctl and if found decides for systemd.

Change-Id: I5a18052a070bad5e16b14672237a1e2b38513949
Signed-off-by: Andreas Florath <andreas@florath.net>
2017-03-11 14:04:13 +00:00
Andreas Florath
c99ea56726 Replace architecture-emulation-binaries with qemu-debootstrap
The architecture-emulation-binaries element does for a limited number
of architectures on Ubuntu exactly the same as `qemu-debootstrap`.
`qemu-debootstrap` comes in the qemu-user-static package that is
anyhow used when cross-building an image.

This patch replaces the complete architecture-emulation-binaries
element with a call to qemu-debootstrap.

Change-Id: Ib9667307bfd3ff7592444a2ec5b04aa5365a1872
Signed-off-by: Andreas Florath <andreas@florath.net>
2017-03-11 13:58:37 +00:00
Yolanda Robla
11275c212b Send custom parameters in bootloader to GRUB_CMDLINE
Depending on the types of deployment (security, nfv...) some extra
kernel flags are needed on the images. This change exposes the
DIB_BOOTLOADER_DEFAULT_CMDLINE parameter, defaulting to the
existing 'nofb nomodeset vga=normal', that will allow to modify
these boot params.

Change-Id: I67d191fa5ca44a57f776cb9739a02dd71212969c
Closes-Bug: #1668890
2017-03-08 15:43:46 +01:00
Ian Wienand
f1776d72aa Merge branch 'master' into merge-branch
Change-Id: Ib3c4b0030c94459a8c2eca836a16e42edf1accd4
2017-03-02 13:49:29 +11:00
Andreas Florath
003fc17208 Fix #1627402: fix grub2 element for ubuntu xenial
Looks that the special handling for Ubuntu is not needed any longer
(its a pity that there are no detailed comments...).

The grub2 element is a second implementation of the bootstrap element
- but because there are some features that come only here, e.g. efi
boot, it should be working as long as this is not implemented in the
bootloader element.

Change-Id: I74269116ea30b84f3259805720d5cd1616f960c5
Signed-off-by: Andreas Florath <andreas@florath.net>
Closes-Bug: #1627402
2017-02-11 18:44:18 +00:00
Ian Wienand
7a155e08bf Merge branch 'master' into merge-branch
Change-Id: I28e4c7837d84e8b66eff3d182666c5a87a9e3c9b
2017-02-09 13:35:53 +11:00
Andreas Florath
fdd2c4b236 Semi-automatic doc generation of element dependency
Currently there is no description of dependencies in the generated
documentation of the elements: therefore a user of an element does not
know which other elements are automatically included and e.g. which
configuration options are available. In addition there are some
copy&pastes of parts of the README.rst scattered thought different
Ubuntu and Debian specific elements.

This patch adds a semi-automatic generation of dependency information
of all elements.  Nevertheless these are not automatically included.
The author of the element's README.rst can decide if and where the
dependency information should appear and can use the descriptor

.. element_deps::

for this.

This patch adds the dependency information for some Debian and
Ubuntu patches - and creates the base for later removing the
duplicated parts.

A call is added to element_dependencies._find_all_elements() to
populate reverse dependencies for Element objects.

(This is a reworking of I31d2b6050b6c46fefe37378698e9a330025db430 for
the feature/v2 branch)

Change-Id: Iebb83916fed71565071246baa550849eef40560b
2017-02-09 09:50:30 +11:00
Andreas Florath
a8953dd277 block-device: change top level config from dict to list
With the old configuration structure it was only possible
to use one image and one partition layout.  The new
block-device configuration uses a list at top level;
therefore it is possible to use multiple instances
of each element type.

Change-Id: I9db4327486b676887d6ce09609994116dbebfc89
Signed-off-by: Andreas Florath <andreas@florath.net>
2017-02-03 20:15:39 +00:00
Ian Wienand
3f8800832a Release 1.27.0
-----BEGIN PGP SIGNATURE-----
 
 iQEcBAABAgAGBQJYk8+9AAoJEBty/58O8cX8LdIH+wU/VrEVs0XYohiL6DUgabzs
 112U3UUihH5xMc/ca9Tarx+XwEvfMZkwYN2Qr0JoRJjmSt2AL6AezUhGSV+98vaY
 iQEccaFDFYlyDHm4V2r7N1xwS0B3mx87FPqVQQSUKlc3JsQxCy4o9RtD9aM8Gvqy
 +gAxMxL3p3O131K0Rvb0U5lC1FLgft9SuljCV8i5nU4/HdoryD6hedz2/ss8a9KG
 KKEdBKvPBKn73+nb8peQD/VXpej9C31r87q5VEjUsZkJ7gduY/qYLlGGgoBQqAXN
 WQ/ef1RkQKW5ba2jsjnk7fdOrA0+wYENxorR2WecuZbe2ieXw6fP3lYiD6VeWsM=
 =IUuh
 -----END PGP SIGNATURE-----

Merge tag '1.27.0' into merge-branch

Release 1.27.0

Change-Id: I9f6948636cae6d375d1d8315976504021f5a3bbb
2017-02-03 11:49:45 +11:00
Ian Wienand
adf39c52cf Release 1.21.1
-----BEGIN PGP SIGNATURE-----
 
 iQEcBAABAgAGBQJYW2GoAAoJEBty/58O8cX8uSAH/15dJsglP6Zie7jSSJcR6k+e
 PJembHn9qrqrCjmJ5EwakojySaaLhwEJKvlP54OU9v7pmUXL9gJtK2OzW54LQ41g
 xBHIu0Pg4z7juyHm9+1P2Sr7Mzs1pVSEbsIYpDYUU19eghI1EAeIj3I1woKgajN7
 JlI61j3r67G6EAVtPOnmD1jvXS8CrtjiJ9wtWTH20pWfmksovg/GuXUCZrLkAAhO
 NcK35CdMii1Hkr7XOH424La/Ar+3qfUX18ZkbJY6yHzkq/ityTzzKOFjAaDl2Jg9
 WNc+SLCVYpPhPwgt7miTywamUNj3ZviA5/Hd8fuLXmtHSLQ23WOtBiaQMLtwXHs=
 =8dIl
 -----END PGP SIGNATURE-----

Merge tag '1.26.1' into merge-branch

Release 1.21.1

Change-Id: Ib9eb3dd1d384fc5b6a9846608216e056c57a173a
2017-02-02 20:36:23 +11:00
Ian Wienand
bfca36c772 Release 1.25.2
-----BEGIN PGP SIGNATURE-----
 
 iQEcBAABAgAGBQJYV1yqAAoJEBty/58O8cX8hLwIAKP66w6MdPN8PDgUOteui/Sx
 N0UFKJ9yR4GQOAP0NffPLjch5/g0iJLs3eFKOhtGC1LjbDjpVgjX8vW18ib8wBZK
 GemOZPF3uxg8FROrZF1vpoDy/cHgL1YV10hCnwdjN/r9rb8zOuSabqjW+Dennj2n
 fZ0SJfa8Owfudn3YxGuOymVb/wMtEloDmVGBEI1Y+h7osELCCDi3OXmwsA8qMsdl
 cTwbeugBs4PlOVbZUK/JKGuwIHKgPnDYzYu5KpXw77/MdjGT0fo5Tlq5AOBDI2sC
 9JOFEBDli4Ro05VwvI58ADMpvvOax+9EvOhLbB1dRPdZl21Iyb6gOdy2PUbFO0c=
 =aKxq
 -----END PGP SIGNATURE-----

Merge tag '1.25.2' into merge-branch

Release 1.25.2

Change-Id: I698bcf2e82117bd81649cd065a7af5cac85990c7
2017-02-02 11:20:00 +11:00
Ian Wienand
b6e631360f Release 1.24.0
-----BEGIN PGP SIGNATURE-----
 
 iQEcBAABAgAGBQJYP+r7AAoJEBty/58O8cX8ZcMIAI7Yy2BfGqY/jfyLYNryATSu
 g78McMxAeOiPYII+KKyVkeWsChKJUafLZr/ppO95pm7ZLXeHQeC4MHDI68RmghIX
 HVXn4hPYzwlZ+Yr4hqaEf11H+vw0TDxo2a5bxlQZw+FCx9+3lkR60QxWf8a6QCxv
 nLqKIhtpwGi9mIyvAya+MaOmHMHqbNYgIBHKP0yXhTdxoHZoJWZeXN1rkelwAF9N
 TS1+KwIk+uemn1ZlqzYDWj8euyFyAv8L0WlF0ztCKe7TAFvNeMo26fLykeKHVDEo
 GC5R8MshiOtihPC972rNIponIzd62AetTlDxfK+gcUZO21pNTrlgdLX5Jn04fo0=
 =BpQc
 -----END PGP SIGNATURE-----

Merge tag '1.24.0' into merge-branch

Release 1.24.0

Change-Id: I23cfa6f8dc66bb6a1d957a49a5b468cfd47c005b
2017-01-31 14:14:19 +11:00
Andreas Florath
ec7f56c1b2 Refactor: block-device handling (partitioning)
During the creation of a disk image (e.g. for a VM), there is the need
to create, setup, configure and afterwards detach some kind of storage
where the newly installed OS can be copied to or directly installed
in.

This patch implements partitioning handling.

Change-Id: I0ca6a4ae3a2684d473b44e5f332ee4225ee30f8c
Signed-off-by: Andreas Florath <andreas@florath.net>
2017-01-24 19:59:10 +00:00
Ian Wienand
448a2602fe Merge remote-tracking branch 'origin/master' into merge-branch
Change-Id: I05cc51c699008018d557ed3874d71af26fd60240
2016-11-29 07:43:46 +11:00
Ian Wienand
7d5afecfd9 Merge remote-tracking branch 'origin/master' into merge-branch
Change-Id: Ibab1bb95521292ae818bd91f7073c3749a2cc0cb
2016-11-18 13:53:56 +11:00
Ian Wienand
d65678678e Move dib-run-parts into diskimage-builder
Move dib-run-parts from dib-utils into diskimage-builder directly.

For calling outside the chroot, we provide a standard entry-point
script.  However, as noted in the warning comment, the underlying
script is still copied directly into the chroot by the dib-run-parts
element.  I believe this to be the KISS approach.

This removes the dependency on dib-utils.  We have discussed this
previously and nobody seemed to think retiring dib-utils was going to
be an issue.

This also updates the documentation to not mention dib-utils, or using
disk-image-create via $PATH setup, but rather gives instructions on
installing from pip with a virtualenv.

Change-Id: Ic1e22ba498d2c368da7d72e2e2b70ff34324feb8
2016-11-04 17:07:37 +11:00
Ian Wienand
97c01e48ed Move elements & lib relative to diskimage_builder package
Currently we have all our elements and library files in a top-level
directory and install them into
<root>/share/diskimage-builder/[elements|lib] (where root is either /
or the root of a virtualenv).

The problem with this is that editable/development installs (pip -e)
do *not* install data_files.  Thus we have no canonical location to
look for elements -- leading to the various odd things we do such as a
whole bunch of guessing at the top of disk-image-create and having a
special test-loader in tests/test_elements.py so we can run python
unit tests on those elements that have it.

data_files is really the wrong thing to use for what are essentially
assets of the program.  data_files install works well for things like
config-files, init.d files or dropping documentation files.

By moving the elements under the diskimage_builder package, we always
know where they are relative to where we import from.  In fact,
pkg_resources has an api for this which we wrap in the new
diskimage_builder/paths.py helper [1].

We use this helper to find the correct path in the couple of places we
need to find the base-elements dir, and for the paths to import the
library shell functions.

Elements such as svc-map and pkg-map include python unit-tests, which
we do not need tests/test_elements.py to special-case load any more.
They just get found automatically by the normal subunit loader.

I have a follow-on change (I69ca3d26fede0506a6353c077c69f735c8d84d28)
to move disk-image-create to a regular python entry-point.

Unfortunately, this has to move to work with setuptools.  You'd think
a symlink under diskimage_builder/[elements|lib] would work, but it
doesn't.

[1] this API handles stuff like getting files out of .zip archive
modules, which we don't do.  Essentially for us it's returning
__file__.

Change-Id: I5e3e3c97f385b1a4ff2031a161a55b231895df5b
2016-11-01 17:27:41 -07:00