Commit Graph

47 Commits

Author SHA1 Message Date
Ian Wienand
c48a40d479 Move several packages to bindep.txt
Move several parts of the "install_test_deps.sh" script into the more
standard bindep.txt.  This list is intentionally restricted as a first
step.

Developer documentation is updated to use bindep and clarified
slightly.

Change-Id: I7520902dc324d920a0c7c44a2d35fe49f9b05614
2018-10-29 09:52:13 +11:00
Paul Belanger
7173658df2 Add support for Fedora 28, remove EOL Fedora 26
This updates diskimage-builder to support current Fedora releases (27
and 28) and removes support for Fedora 26 which is EOL as of June
2018.

Change-Id: I602b22ed4d5397b39dc1eef67964f6fbdcd93060
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2018-10-18 19:43:12 +11:00
kaiokassiano
87c3d28206 Fix typo in installation.rst
Trivial fix.

Change-Id: I7d4fa9e4007954d5d847e659a17ac602f337e78f
2018-07-31 18:19:10 -03:00
Ian Wienand
e443700b5c Formalise saving of /etc/resolv.conf
systemd-resolved has a new behaviour in bionic, in that if there is no
/etc/resolv.conf file when it installs, it assumes it is a fresh
system and makes /etc/resolf.conf a symlink into its compatability
files.

dib ends up saving & restoring whatever /etc/resolv.conf we have after
the inital chroot creation, which may not be what we want -- in the
above case it restores the system-resolved symlink.  For
openstack-infra, we use unbound and want simply "127.0.0.1" in a
/etc/resolv.conf file [1].

Formalise the ability to save specific contents into the final image.
Add documentation, and a note in the code that it's an external
interface.

I would have preferred to namespace the .ORIG file with DIB_ or
similar, but this unofficial interface has already escaped into the
wild.  Leave it as is for simplicity.

[1] Note that systemd-resolved will obey /etc/resolv.conf as you would
expect, if file exists.

Change-Id: Ie0e97d8072e2b21a54b053fa6fb07b62960c686d
2018-04-04 15:17:45 +10:00
Ian Wienand
adb0341064 Add block-device defaults
This moves the block-device default out of the "vm" element and into a
selection of other elements.  There's "mbr" which retains the status
quo.  There's an EFI version that has the boot/grub partitions as
required.  In between there's the GPT only version, which is useful
for architectures like power without EFI, but still want possible
larger disks using GPT.

Change-Id: I4a566a97d073fc0dda0ab2494ac988fe015800a9
2018-02-23 10:04:40 +11:00
Ian Wienand
55b479b54f GPT partitioning support
This adds support for a GPT label type to the partitioning code.  This
is relatively straight-forward translation of the partition config
into a sgparted command-line and subsequent call.

A unit test is added based on a working GPT/EFI configuration and the
fedora-minimal functional test is updated to build a single-partition
GPT based using the new block-device-gpt override element.  See notes
in the sample configuration files about partition requirements and
types.

Documentation has been updated.

Co-Authored-By: Marcin Juszkiewicz <marcin.juszkiewicz@linaro.org>
Change-Id: I6b819a8071389e7e4eb4874ff7750bd192695ff2
2018-02-23 10:04:26 +11:00
Kevin TIBI
9c769ff9ba Fix typo doc
Change-Id: I93bf21d2fb1f96c2e366b3d41af32b914350d1ce
2018-02-12 14:26:29 +01:00
Tim Flink
490cf3aa49 Add support for Fedora 27, remove EOL Fedora 25
This updates diskimage-builder to support current Fedora releases (26
and 27) and removes support for Fedora 25 which is EOL as of December
12, 2017.

Change-Id: I227a607c6c468cc8b7bb154a189e9c8ce2021192
2018-01-23 11:31:21 +00:00
Zhiguo Deng
271dc36f33 Add zipl element as s390x architecture bootloader
s390x architecture uses zipl as bootloader. When used in combination
with the vm element it replaces the existing bootloader element.
It's mandatory for s390x vm images.

Use cases
---------

* Allow users to create s390x images that run on nova with s390x
  libvirt/kvm backend
* Building nodepool images for s390x third party CI

Supported Distros
-----------------
The following listing shows all Distros that officially support
s390x and how those Distros are supported in DIB with this patch.

* SLES - not supported (SLES is not supported in DIB)
* RHEL - not suppoprted (RHEL is not supported as KVM guest on s390x,
                         therefore there's no rhel7 qcow image for s390x available
                         like it is for other archictectures)
* Ubuntu - supported

Ubuntu images can for example be built using the following commands:

  $ disk-image-create ubuntu-minimal zipl vm
  $ disk-image-create ubuntu-minimal zipl
  $ disk-image-create ubuntu zipl vm

Testing
-------

Cross architecture building of s390x images is not supported so far.

The plan is to set up a ThirdParty CI that builds the image for s390x and
provides the logs.

Co-Authored-By: Andreas Scheuring <andreas.scheuring@de.ibm.com>
Co-Authored-By: Holger Smolinsky <holger@smolinski.name>
Co-Authored-By: Zhiguo Deng <bjzgdeng@linux.vnet.ibm.com>
Co-Authored-By: Arne Recknagel <arne.recknagel@hotmail.com>

Closes-Bug: #1730641

Change-Id: I576e7edda68da12e97c60af38f457915efe7b934
2017-11-07 17:19:27 +01:00
Yolanda Robla
b84997036d Mention the need of dracut-regenerate element
When using volumes, booting from them will need that the ramdisk
image used has support for that. In case the lvm module is not
included, mention the need of adding dracut-regenerate in the
elements that are needed.

Change-Id: I6e1f618dcfc5ef3be01c83904ffe6dd33db72bb7
Partially-Fixes-Bug: #1715686
2017-09-11 14:25:00 +02:00
Jenkins
fa3797848b Merge "LVM support for dib-block-device" 2017-08-24 09:18:26 +00:00
Yolanda Robla
c2dc3dc78e LVM support for dib-block-device
This provides a basic LVM support to dib-block-device.

Co-Authored-By: Ian Wienand <iwienand@redhat.com>

Change-Id: Ibd624d9f95ee68b20a15891f639ddd5b3188cdf9
2017-08-24 16:22:56 +10:00
Yolanda Robla
1492d1c55f Add kpartx as a requirement to build images
Testing on a ubuntu cloud image, the build is not possible
because it is missing the kpartx dependency. Adding that
as a requirement, when producing images with partitions.

Change-Id: I0c8f39b5233dd450d8130735ed801bbed6bca0e9
2017-08-11 15:16:13 +02:00
Jenkins
ced9b51f6e Merge "Allow users to specify partition type in the MBR PTE" 2017-08-04 05:19:02 +00:00
Amrith Kumar
52faa0e1d9 Allow users to specify partition type in the MBR PTE
The MBR Partition Table Entry (PTE) allows one to specify many
possible partition types and one of the benefits of this is being able
to specify the CHS variant or the LBA variant.

By default, LBA only creates partitions of type 0x83 (of course,
that's only because the documentation doesn't tell you how to make it
do anything else).

I will take up Ian's suggestion in patch set 2 for a more rigorous
test in an independent patch set.

Change-Id: If3068535980eac2e58d4025444c65147a8c7fedc
Closes-Bug:#1703352
2017-07-29 06:34:25 -04:00
Jenkins
f9700225b9 Merge "doc: supported_distros: Add openSUSE Leap 42.2/3 and Tumbleweed" 2017-07-20 11:13:32 +00:00
Markos Chandras
710c20196b doc: supported_distros: Add openSUSE Leap 42.2/3 and Tumbleweed
Depends-On: I08a663fd03c8545de09d650001ab250eaf40e427

Change-Id: I924987457e047beaf13de6ca47d8d4a5fedff513
2017-07-19 14:59:29 +02:00
Tony Breeds
7aa6a69983 [doc] Add some notes about PowerPC Architecture names
Change-Id: I58eb3e442344bbbcf7c5d71c13688baaacd70105
2017-06-27 10:27:10 +10:00
Ian Wienand
00da1982ce Add a more generic tree->graph parser
This moves to a more generic config parser that doesn't have plugins
parsing part of the tree.

I understand why it ended up that way; we have "partitions" key which
has special semantics compared to others keys and there was a desire
to keep it isolated from core tree->graph code.  But this isn't really
isolated; you have to reverse-engineer several module-crossing
boundaries, extras classes and repetitive recursive functions.

Ultimately, plugins should have access to the node graph, but not
participate in configuration parsing.  This way we ensure that plugins
can't invent new methods of configuration parsing.

Note: unit tests produce the same tree -> graph conversion as the old
method.  i.e. this is not intended to have a functional change.

Change-Id: I8a5d62a076a5a50597f2f1df3a8615afba6dadb2
2017-05-26 10:13:14 +10:00
Andreas Florath
e4e23897a1 Refactor: block-device filesystem creation, mount and fstab
This patch finalizes the block device refactoring.  It moves the three
remaining levels (filesystem creation, mount and fstab handling) into
the new python module.

Now it is possible to use any number of disk images, any number of
partitions and used them mounted to different directories.

Notes:

 * unmount_dir : modified to only unmount the subdirs mounted by
   mount_proc_sys_dev().  dib-block-device unmounts
   $TMP_MOUNT_PATH/mnt (see I85e01f3898d3c043071de5fad82307cb091a64a9)

Change-Id: I592c0b1329409307197460cfa8fd69798013f1f8
Signed-off-by: Andreas Florath <andreas@florath.net>
Closes-Bug: #1664924
2017-05-12 13:52:02 +02:00
Yolanda Robla
c3dda92b67 Refactor documentation on image creation
Add the bits of documentation that talk about image creation
at the scope of level 1.
This is a partial refactor of change
I592c0b1329409307197460cfa8fd69798013f1f8

Change-Id: I2619c9ebf3ecfeea67fe9063a169d8324d7ffdf2
Co-Authored-By: Andreas Florath <andreas@florath.net>
2017-05-05 17:29:42 +00:00
Yolanda Robla
340451ea00 Improve documentation for image creation
Add the tree-like vs complete digraph configuration
for images. This is a partial refactor from
I3600c6a3d663c697b59d91bd3fbb5e408af345e4

Change-Id: Ia7a8321e63d59771fe47d8e262b9aacffd60d8d9
Co-Authored-By: Andreas Florath <andreas@florath.net>
2017-05-01 15:22:35 +00:00
Ian Wienand
fea6ab1624 Use sphinx warning-is-error
Sphix 1.5 (I9e7261c4124b71eeb6bddd9e21747b61bbdc16fa) includes
"warning-is-error" which supersedes pbr's warnerrors.  Enable this and
fix up the resulting failures

 - trailing lines for lists in element_deps directive
 - missing README's that are linked
 - syntax error and highlighting in building instructions

Change-Id: I6549551b4a9bf47076c9811a7a38a666cbea2a50
2017-03-14 14:49:49 +11:00
Andreas Florath
a8953dd277 block-device: change top level config from dict to list
With the old configuration structure it was only possible
to use one image and one partition layout.  The new
block-device configuration uses a list at top level;
therefore it is possible to use multiple instances
of each element type.

Change-Id: I9db4327486b676887d6ce09609994116dbebfc89
Signed-off-by: Andreas Florath <andreas@florath.net>
2017-02-03 20:15:39 +00:00
Ian Wienand
adf39c52cf Release 1.21.1
-----BEGIN PGP SIGNATURE-----
 
 iQEcBAABAgAGBQJYW2GoAAoJEBty/58O8cX8uSAH/15dJsglP6Zie7jSSJcR6k+e
 PJembHn9qrqrCjmJ5EwakojySaaLhwEJKvlP54OU9v7pmUXL9gJtK2OzW54LQ41g
 xBHIu0Pg4z7juyHm9+1P2Sr7Mzs1pVSEbsIYpDYUU19eghI1EAeIj3I1woKgajN7
 JlI61j3r67G6EAVtPOnmD1jvXS8CrtjiJ9wtWTH20pWfmksovg/GuXUCZrLkAAhO
 NcK35CdMii1Hkr7XOH424La/Ar+3qfUX18ZkbJY6yHzkq/ityTzzKOFjAaDl2Jg9
 WNc+SLCVYpPhPwgt7miTywamUNj3ZviA5/Hd8fuLXmtHSLQ23WOtBiaQMLtwXHs=
 =8dIl
 -----END PGP SIGNATURE-----

Merge tag '1.26.1' into merge-branch

Release 1.21.1

Change-Id: Ib9eb3dd1d384fc5b6a9846608216e056c57a173a
2017-02-02 20:36:23 +11:00
Ian Wienand
bfca36c772 Release 1.25.2
-----BEGIN PGP SIGNATURE-----
 
 iQEcBAABAgAGBQJYV1yqAAoJEBty/58O8cX8hLwIAKP66w6MdPN8PDgUOteui/Sx
 N0UFKJ9yR4GQOAP0NffPLjch5/g0iJLs3eFKOhtGC1LjbDjpVgjX8vW18ib8wBZK
 GemOZPF3uxg8FROrZF1vpoDy/cHgL1YV10hCnwdjN/r9rb8zOuSabqjW+Dennj2n
 fZ0SJfa8Owfudn3YxGuOymVb/wMtEloDmVGBEI1Y+h7osELCCDi3OXmwsA8qMsdl
 cTwbeugBs4PlOVbZUK/JKGuwIHKgPnDYzYu5KpXw77/MdjGT0fo5Tlq5AOBDI2sC
 9JOFEBDli4Ro05VwvI58ADMpvvOax+9EvOhLbB1dRPdZl21Iyb6gOdy2PUbFO0c=
 =aKxq
 -----END PGP SIGNATURE-----

Merge tag '1.25.2' into merge-branch

Release 1.25.2

Change-Id: I698bcf2e82117bd81649cd065a7af5cac85990c7
2017-02-02 11:20:00 +11:00
Andreas Florath
ec7f56c1b2 Refactor: block-device handling (partitioning)
During the creation of a disk image (e.g. for a VM), there is the need
to create, setup, configure and afterwards detach some kind of storage
where the newly installed OS can be copied to or directly installed
in.

This patch implements partitioning handling.

Change-Id: I0ca6a4ae3a2684d473b44e5f332ee4225ee30f8c
Signed-off-by: Andreas Florath <andreas@florath.net>
2017-01-24 19:59:10 +00:00
Tristan Cacqueray
9d13084c41 Add squashfs output image format
The squashfs format brings a couple of advantages over the other
formats. Image is often an order of magnitude smaller and it can
be used natively, either as an initrd, either with loop mount.

Change-Id: If72940b0c4dafb2504c52dd0429a8eb3f8305751
2016-12-19 07:21:39 +00:00
Paul Belanger
da41ee6012 Add output image format tgz support
We now support tgz (tar.gz) as an output format.

Change-Id: Iadec92f2f96c3f904f28bd49f87ffc7d48ef7bd7
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2016-12-17 16:41:42 -05:00
Ian Wienand
5e2330d89c Put MKFS_OPTS after filesystem type
mkfs's arguments are

 mkfs [options] [-t type] [fs-options] device [size]

So it seems our MKFS_OPTS are really supposed to be fs-options, rather
than options to mkfs itself.

Why didn't we notice?  It's quite a trap -- mkfs.ext2 has a "-t"
option, so when we're calling

 $ mkfs -i 4096 ... -t ext4 ...

We actually just fall-back to the default from the mkfs wrapper which
is mkfs.ext2 which works!  But when you make that, say, xfs, we're not
calling the right wrapper at all.

Also update documentation

Closes-Bug: #1648287
Change-Id: I3ea5807088ab361bd9c235c07fb1553fbaf9178b
2016-12-09 07:22:38 +11:00
Ian Wienand
d65678678e Move dib-run-parts into diskimage-builder
Move dib-run-parts from dib-utils into diskimage-builder directly.

For calling outside the chroot, we provide a standard entry-point
script.  However, as noted in the warning comment, the underlying
script is still copied directly into the chroot by the dib-run-parts
element.  I believe this to be the KISS approach.

This removes the dependency on dib-utils.  We have discussed this
previously and nobody seemed to think retiring dib-utils was going to
be an issue.

This also updates the documentation to not mention dib-utils, or using
disk-image-create via $PATH setup, but rather gives instructions on
installing from pip with a virtualenv.

Change-Id: Ic1e22ba498d2c368da7d72e2e2b70ff34324feb8
2016-11-04 17:07:37 +11:00
Gregory Haynes
7cc65d3cbd Merge branch 'master' into feature/v2
Change-Id: I0747a27560161ef869868967ab140b5d0be9d4da
2016-10-31 11:21:38 -07:00
Matthew Thode
bb29082910
Document install of diskimage-builder on Gentoo
Change-Id: I27bf3499f998dcbdeb5cfebddde306e5200cc0b5
2016-10-11 20:17:49 -05:00
Gregory Haynes
7f9506962a Fix formatting for supported distros in docs
The supported distros table is not formatted consistently.

Change-Id: Ic9e32786419243c118088f933d86329480f3345b
2016-10-07 15:19:47 +11:00
Andreas Florath
3d48a528c1 Refactor: block-device handling (local loop)
Block device handling can be somewhat complex - especially
when taking things like md, lvm or encryption into account.

This patch factors out the creation and deletion of the local
loop image device handling into a python library.

The main propose of this patch is to implement the needed
infrastructure.  Based on this, more advanced functions can be added.
Example: (advanced) partitioning, LVM, handling different boot
scenarios (BIOS, UEFI, ...), possibility of handling multiple images
(local loop image, iSCSI, physical hard disk, ...), handling of
different filesystems for different partitions / LVs.

Change-Id: Ib626b36a00f8a5dc3dbde8df3e2619a2438eaaf1
Signed-off-by: Andreas Florath <andreas@florath.net>
2016-09-08 04:31:01 +00:00
SamYaple
0c32784663 Revert "Zerofree the image if possible"
This reverts commit 5184d02a7c.

The decision was made to go with fstrim because it is faster and more
universal that zerofree. The related-id has the patchset that implements
fstrim.

Related-Id: I269b4416be450369616f9b8e030f84c30e329804
Change-Id: If40cf2fc0ecd8686768cbfeac9ecee90907674e7
2016-03-10 20:38:01 +00:00
Clark Boylan
5184d02a7c Zerofree the image if possible
If the image has an ext filesystem and the zerofree utility is present
on the build system then run zerofree. This should make images as
compressable as possible which is a nice feature when building
compressed qcow2 images.

Change-Id: Ia6062c291f7a3f58b85a4f408ecb3d0574c65d53
2016-03-07 17:45:14 -08:00
Matthew Thode
175e292ec1 Add Gentoo to the list of supported distributions
We also document any additional package needed.

Change-Id: I7b9193fa5711138287f587935fc864fee4f95856
2016-02-29 12:20:50 +00:00
Isaac Beckman
24d1527111 Move speedup section to image building guide
speedup section explains  the user how to sppedup image build by using
tmpfs. The  correct user guide to have this section, is the user guide
about image building rather than the installation user guide.

Change-Id: I96b90bd79df53db4f926a928ae3c86b888315230
2016-01-18 07:02:00 +00:00
Abel Lopez
5e9c451d5f Document byte-to-inode ratio
End user docs would benefit from a section about the byte-to-inode
ratio, and why it's set the way it is. This update explains why
and how to manipulate the ratio depending on the intended use.

Change-Id: Iffb5ef6f4c7c74f4aa6e25912d4991d7a611c8fe
Closes-bug: 1512841
2015-12-16 13:58:17 -08:00
Gregory Haynes
930b597220 Move install-types doc to user guide
Install-types are a user facing feature, not just for developers. Lets
move the docs on them in to the user guide.

Change-Id: I6ee8f657c270cf90da9c0729494740bb23aa47c5
2015-11-13 18:45:28 +00:00
Jenkins
fe23b262fd Merge "Add documentation of output formats for users" 2015-09-11 13:06:42 +00:00
Gregory Haynes
486f3c01eb Add documentation of output formats for users
A blurb on output formats is useful for our users.

Change-Id: Iffa5036a84c1500ccb38cd3edb258ddbf5148a3e
2015-08-02 23:19:39 -07:00
Gregory Haynes
837bac1070 Fix link in installation doc
The link to building an image is not being properly parsed by sphinx in
the install user guide.

Change-Id: I1e73a4722b603ba11805031c522b107cae8c86b4
2015-07-24 23:47:30 +00:00
Gregory Haynes
89d0896b97 Document our supported distributions
Before we commit to supporting a set of distributions, we should
document what they are.

Change-Id: I076c326ec0eb4d1640a4229c0dfebd3d2cc880b4
2015-07-13 00:36:44 +00:00
Gregory Haynes
d98e6dcff9 Update install docs to be more user friendly
Our install docs are out of date and not very user friendly. Lets fix
that.

Change-Id: Idaff33096bf32865020b85ee776abd6691ac45ad
2015-04-12 16:08:47 +00:00
Gregory Haynes
360033027f Create a user guide
Our docs are very developer focused. Lets create a separate user guide
to help new users get started.

Change-Id: I8a03920e6d3306dd0405177875ea55ccb4b40fea
2015-04-01 19:51:08 +00:00