Commit Graph

115 Commits

Author SHA1 Message Date
Tony Breeds
7aa6a69983 [doc] Add some notes about PowerPC Architecture names
Change-Id: I58eb3e442344bbbcf7c5d71c13688baaacd70105
2017-06-27 10:27:10 +10:00
Andreas Florath
b107606a75 Remove ccache
This patch removes the ccache handling from the base element.  For
mostly all systems this was never used at all.

This is working towards the removal of the base element from DIB

Change-Id: Ieb16ef612ebd98470993dcd6f55b3a22d37084ba
Signed-off-by: Andreas Florath <andreas@florath.net>
2017-05-31 08:28:09 +10:00
Ian Wienand
00da1982ce Add a more generic tree->graph parser
This moves to a more generic config parser that doesn't have plugins
parsing part of the tree.

I understand why it ended up that way; we have "partitions" key which
has special semantics compared to others keys and there was a desire
to keep it isolated from core tree->graph code.  But this isn't really
isolated; you have to reverse-engineer several module-crossing
boundaries, extras classes and repetitive recursive functions.

Ultimately, plugins should have access to the node graph, but not
participate in configuration parsing.  This way we ensure that plugins
can't invent new methods of configuration parsing.

Note: unit tests produce the same tree -> graph conversion as the old
method.  i.e. this is not intended to have a functional change.

Change-Id: I8a5d62a076a5a50597f2f1df3a8615afba6dadb2
2017-05-26 10:13:14 +10:00
Ian Wienand
78c0766bec Produce API documentation
There's an increasing amount of pydoc based documentation.  Output the
module reference and link it into the developers main page.

One fixup to the rst needed

Change-Id: I1d43a1fe1c735eb4559e3d2b40834d1c8115c586
2017-05-25 14:26:31 +10:00
Andreas Florath
e4e23897a1 Refactor: block-device filesystem creation, mount and fstab
This patch finalizes the block device refactoring.  It moves the three
remaining levels (filesystem creation, mount and fstab handling) into
the new python module.

Now it is possible to use any number of disk images, any number of
partitions and used them mounted to different directories.

Notes:

 * unmount_dir : modified to only unmount the subdirs mounted by
   mount_proc_sys_dev().  dib-block-device unmounts
   $TMP_MOUNT_PATH/mnt (see I85e01f3898d3c043071de5fad82307cb091a64a9)

Change-Id: I592c0b1329409307197460cfa8fd69798013f1f8
Signed-off-by: Andreas Florath <andreas@florath.net>
Closes-Bug: #1664924
2017-05-12 13:52:02 +02:00
Yolanda Robla
c3dda92b67 Refactor documentation on image creation
Add the bits of documentation that talk about image creation
at the scope of level 1.
This is a partial refactor of change
I592c0b1329409307197460cfa8fd69798013f1f8

Change-Id: I2619c9ebf3ecfeea67fe9063a169d8324d7ffdf2
Co-Authored-By: Andreas Florath <andreas@florath.net>
2017-05-05 17:29:42 +00:00
Yolanda Robla
340451ea00 Improve documentation for image creation
Add the tree-like vs complete digraph configuration
for images. This is a partial refactor from
I3600c6a3d663c697b59d91bd3fbb5e408af345e4

Change-Id: Ia7a8321e63d59771fe47d8e262b9aacffd60d8d9
Co-Authored-By: Andreas Florath <andreas@florath.net>
2017-05-01 15:22:35 +00:00
Dan Radez
06576a02f0 Adding aarch64 support for CentOS7
Using delorian and delorian-deps for aarch64
this will produce an overcloud-full image

Change-Id: Id0dc83d9c558b0c0bbfaed7727883541d140472c
2017-03-22 10:46:54 -04:00
Ian Wienand
fea6ab1624 Use sphinx warning-is-error
Sphix 1.5 (I9e7261c4124b71eeb6bddd9e21747b61bbdc16fa) includes
"warning-is-error" which supersedes pbr's warnerrors.  Enable this and
fix up the resulting failures

 - trailing lines for lists in element_deps directive
 - missing README's that are linked
 - syntax error and highlighting in building instructions

Change-Id: I6549551b4a9bf47076c9811a7a38a666cbea2a50
2017-03-14 14:49:49 +11:00
Ian Wienand
7a155e08bf Merge branch 'master' into merge-branch
Change-Id: I28e4c7837d84e8b66eff3d182666c5a87a9e3c9b
2017-02-09 13:35:53 +11:00
Andreas Florath
fdd2c4b236 Semi-automatic doc generation of element dependency
Currently there is no description of dependencies in the generated
documentation of the elements: therefore a user of an element does not
know which other elements are automatically included and e.g. which
configuration options are available. In addition there are some
copy&pastes of parts of the README.rst scattered thought different
Ubuntu and Debian specific elements.

This patch adds a semi-automatic generation of dependency information
of all elements.  Nevertheless these are not automatically included.
The author of the element's README.rst can decide if and where the
dependency information should appear and can use the descriptor

.. element_deps::

for this.

This patch adds the dependency information for some Debian and
Ubuntu patches - and creates the base for later removing the
duplicated parts.

A call is added to element_dependencies._find_all_elements() to
populate reverse dependencies for Element objects.

(This is a reworking of I31d2b6050b6c46fefe37378698e9a330025db430 for
the feature/v2 branch)

Change-Id: Iebb83916fed71565071246baa550849eef40560b
2017-02-09 09:50:30 +11:00
Ian Wienand
f3121f0fb7 Fix up doc errors
It seems that pbr's "warnerrors" isn't actually doing anything at the
moment (I680b448471e687919d202e8f2abe57f8ba3b22ee) meaning we're able
to commit docs with RST parser errors.  Fix all these up in some
minimal way (I have not audited content, just made it pass build).

Link the specs in so they're referenced from the top level.

Change-Id: Id67b9ea7ba8f49b43969c58ca3fb7fa1243538a4
2017-02-08 16:07:01 +11:00
Ian Wienand
3654aa4205 Fix up element doc generation
Turns out that you really have to have the elements dir symlinked
under the source directory for the globbing to match anything.  AFAICT
there's no way to add external directories to the sphinx build.

Change-Id: Iaa3576ef250822b0d57bebce1ebbe03e5bafa042
2017-02-08 14:32:43 +11:00
Jenkins
740c0a85df Merge "Move generation of dib_[environment|args] to manifest element" 2017-02-07 20:23:06 +00:00
Andreas Florath
870374c8da Move generation of dib_[environment|args] to manifest element
dib_[environment|args] manifest files are currently generated by the
base element and then moved by the manifest element.

This creates too many corner cases -- if you don't include the base
element (we are trying to empty it ATM) you don't get the env/args
saved at all; if you include base but don't include the manifest
element they're saved to /etc, but if you do have the manifest element
they're moved to the manifest dir.

Move generation of these into the manifest element directly and update
the documentation to reflect this.  In practice this doesn't change
things, because the "manifests" element gets pulled in via deps for
most builds.

Change-Id: I3f23037058137d166b29f0b70fd1a02c22c07fc8
Signed-off-by: Andreas Florath <andreas@florath.net>
2017-02-07 11:33:16 +11:00
Andreas Florath
a8953dd277 block-device: change top level config from dict to list
With the old configuration structure it was only possible
to use one image and one partition layout.  The new
block-device configuration uses a list at top level;
therefore it is possible to use multiple instances
of each element type.

Change-Id: I9db4327486b676887d6ce09609994116dbebfc89
Signed-off-by: Andreas Florath <andreas@florath.net>
2017-02-03 20:15:39 +00:00
Ian Wienand
3f8800832a Release 1.27.0
-----BEGIN PGP SIGNATURE-----
 
 iQEcBAABAgAGBQJYk8+9AAoJEBty/58O8cX8LdIH+wU/VrEVs0XYohiL6DUgabzs
 112U3UUihH5xMc/ca9Tarx+XwEvfMZkwYN2Qr0JoRJjmSt2AL6AezUhGSV+98vaY
 iQEccaFDFYlyDHm4V2r7N1xwS0B3mx87FPqVQQSUKlc3JsQxCy4o9RtD9aM8Gvqy
 +gAxMxL3p3O131K0Rvb0U5lC1FLgft9SuljCV8i5nU4/HdoryD6hedz2/ss8a9KG
 KKEdBKvPBKn73+nb8peQD/VXpej9C31r87q5VEjUsZkJ7gduY/qYLlGGgoBQqAXN
 WQ/ef1RkQKW5ba2jsjnk7fdOrA0+wYENxorR2WecuZbe2ieXw6fP3lYiD6VeWsM=
 =IUuh
 -----END PGP SIGNATURE-----

Merge tag '1.27.0' into merge-branch

Release 1.27.0

Change-Id: I9f6948636cae6d375d1d8315976504021f5a3bbb
2017-02-03 11:49:45 +11:00
Ian Wienand
adf39c52cf Release 1.21.1
-----BEGIN PGP SIGNATURE-----
 
 iQEcBAABAgAGBQJYW2GoAAoJEBty/58O8cX8uSAH/15dJsglP6Zie7jSSJcR6k+e
 PJembHn9qrqrCjmJ5EwakojySaaLhwEJKvlP54OU9v7pmUXL9gJtK2OzW54LQ41g
 xBHIu0Pg4z7juyHm9+1P2Sr7Mzs1pVSEbsIYpDYUU19eghI1EAeIj3I1woKgajN7
 JlI61j3r67G6EAVtPOnmD1jvXS8CrtjiJ9wtWTH20pWfmksovg/GuXUCZrLkAAhO
 NcK35CdMii1Hkr7XOH424La/Ar+3qfUX18ZkbJY6yHzkq/ityTzzKOFjAaDl2Jg9
 WNc+SLCVYpPhPwgt7miTywamUNj3ZviA5/Hd8fuLXmtHSLQ23WOtBiaQMLtwXHs=
 =8dIl
 -----END PGP SIGNATURE-----

Merge tag '1.26.1' into merge-branch

Release 1.21.1

Change-Id: Ib9eb3dd1d384fc5b6a9846608216e056c57a173a
2017-02-02 20:36:23 +11:00
Ian Wienand
bfca36c772 Release 1.25.2
-----BEGIN PGP SIGNATURE-----
 
 iQEcBAABAgAGBQJYV1yqAAoJEBty/58O8cX8hLwIAKP66w6MdPN8PDgUOteui/Sx
 N0UFKJ9yR4GQOAP0NffPLjch5/g0iJLs3eFKOhtGC1LjbDjpVgjX8vW18ib8wBZK
 GemOZPF3uxg8FROrZF1vpoDy/cHgL1YV10hCnwdjN/r9rb8zOuSabqjW+Dennj2n
 fZ0SJfa8Owfudn3YxGuOymVb/wMtEloDmVGBEI1Y+h7osELCCDi3OXmwsA8qMsdl
 cTwbeugBs4PlOVbZUK/JKGuwIHKgPnDYzYu5KpXw77/MdjGT0fo5Tlq5AOBDI2sC
 9JOFEBDli4Ro05VwvI58ADMpvvOax+9EvOhLbB1dRPdZl21Iyb6gOdy2PUbFO0c=
 =aKxq
 -----END PGP SIGNATURE-----

Merge tag '1.25.2' into merge-branch

Release 1.25.2

Change-Id: I698bcf2e82117bd81649cd065a7af5cac85990c7
2017-02-02 11:20:00 +11:00
Jenkins
b4266001d0 Merge "Spec for changing the block device handling: partitioning" 2017-01-31 17:45:19 +00:00
Andreas Florath
ec7f56c1b2 Refactor: block-device handling (partitioning)
During the creation of a disk image (e.g. for a VM), there is the need
to create, setup, configure and afterwards detach some kind of storage
where the newly installed OS can be copied to or directly installed
in.

This patch implements partitioning handling.

Change-Id: I0ca6a4ae3a2684d473b44e5f332ee4225ee30f8c
Signed-off-by: Andreas Florath <andreas@florath.net>
2017-01-24 19:59:10 +00:00
Julia Kreger
197ab47226 Update component docs to refer to ironic-agent
The components documentation was previously referring to the
ramdisk image for deployment, which was previously deprecated.

Corrected to point to the ironic-agent element.

Change-Id: I770460041eb13523896aaadb7705bdc3db1a54ca
2017-01-03 23:34:13 +00:00
Tristan Cacqueray
9d13084c41 Add squashfs output image format
The squashfs format brings a couple of advantages over the other
formats. Image is often an order of magnitude smaller and it can
be used natively, either as an initrd, either with loop mount.

Change-Id: If72940b0c4dafb2504c52dd0429a8eb3f8305751
2016-12-19 07:21:39 +00:00
Paul Belanger
da41ee6012 Add output image format tgz support
We now support tgz (tar.gz) as an output format.

Change-Id: Iadec92f2f96c3f904f28bd49f87ffc7d48ef7bd7
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2016-12-17 16:41:42 -05:00
Ian Wienand
5e2330d89c Put MKFS_OPTS after filesystem type
mkfs's arguments are

 mkfs [options] [-t type] [fs-options] device [size]

So it seems our MKFS_OPTS are really supposed to be fs-options, rather
than options to mkfs itself.

Why didn't we notice?  It's quite a trap -- mkfs.ext2 has a "-t"
option, so when we're calling

 $ mkfs -i 4096 ... -t ext4 ...

We actually just fall-back to the default from the mkfs wrapper which
is mkfs.ext2 which works!  But when you make that, say, xfs, we're not
calling the right wrapper at all.

Also update documentation

Closes-Bug: #1648287
Change-Id: I3ea5807088ab361bd9c235c07fb1553fbaf9178b
2016-12-09 07:22:38 +11:00
Ian Wienand
7d5afecfd9 Merge remote-tracking branch 'origin/master' into merge-branch
Change-Id: Ibab1bb95521292ae818bd91f7073c3749a2cc0cb
2016-11-18 13:53:56 +11:00
Ian Wienand
d65678678e Move dib-run-parts into diskimage-builder
Move dib-run-parts from dib-utils into diskimage-builder directly.

For calling outside the chroot, we provide a standard entry-point
script.  However, as noted in the warning comment, the underlying
script is still copied directly into the chroot by the dib-run-parts
element.  I believe this to be the KISS approach.

This removes the dependency on dib-utils.  We have discussed this
previously and nobody seemed to think retiring dib-utils was going to
be an issue.

This also updates the documentation to not mention dib-utils, or using
disk-image-create via $PATH setup, but rather gives instructions on
installing from pip with a virtualenv.

Change-Id: Ic1e22ba498d2c368da7d72e2e2b70ff34324feb8
2016-11-04 17:07:37 +11:00
Ian Wienand
97c01e48ed Move elements & lib relative to diskimage_builder package
Currently we have all our elements and library files in a top-level
directory and install them into
<root>/share/diskimage-builder/[elements|lib] (where root is either /
or the root of a virtualenv).

The problem with this is that editable/development installs (pip -e)
do *not* install data_files.  Thus we have no canonical location to
look for elements -- leading to the various odd things we do such as a
whole bunch of guessing at the top of disk-image-create and having a
special test-loader in tests/test_elements.py so we can run python
unit tests on those elements that have it.

data_files is really the wrong thing to use for what are essentially
assets of the program.  data_files install works well for things like
config-files, init.d files or dropping documentation files.

By moving the elements under the diskimage_builder package, we always
know where they are relative to where we import from.  In fact,
pkg_resources has an api for this which we wrap in the new
diskimage_builder/paths.py helper [1].

We use this helper to find the correct path in the couple of places we
need to find the base-elements dir, and for the paths to import the
library shell functions.

Elements such as svc-map and pkg-map include python unit-tests, which
we do not need tests/test_elements.py to special-case load any more.
They just get found automatically by the normal subunit loader.

I have a follow-on change (I69ca3d26fede0506a6353c077c69f735c8d84d28)
to move disk-image-create to a regular python entry-point.

Unfortunately, this has to move to work with setuptools.  You'd think
a symlink under diskimage_builder/[elements|lib] would work, but it
doesn't.

[1] this API handles stuff like getting files out of .zip archive
modules, which we don't do.  Essentially for us it's returning
__file__.

Change-Id: I5e3e3c97f385b1a4ff2031a161a55b231895df5b
2016-11-01 17:27:41 -07:00
Jenkins
cc1b37e731 Merge "Don't set tracing in environment files" 2016-11-01 01:42:59 +00:00
Gregory Haynes
7cc65d3cbd Merge branch 'master' into feature/v2
Change-Id: I0747a27560161ef869868967ab140b5d0be9d4da
2016-10-31 11:21:38 -07:00
Ian Wienand
9e392f56b0 Don't set tracing in environment files
Because environment files are sourced into the current environment,
they shouldn't be setting global settings like tracing else they
affect every preceeding import.  This is quite confusing when only
half your imports are traced in the logs, because it was either turned
on, or off, by a preceeding environment import.

There is a corresponding dib-run-parts change in
I29f7df1514aeb988222d1094e8269eddb485c2a0 that will greatly increase
debugability for environment files by deliberately logging what files
are sourced and consistently turning on tracing around their import.

This isn't strictly necessary (since dib-run-parts with the prior
change will just turn tracing off after import anyway) but it's a
decent cleanup for consistency.  A bare-minimum dib-lint check is
added.  Documentation is updated.

Change-Id: I10f68be0642835a04af7e5a2bc101502f61e5357
2016-10-20 13:58:00 +11:00
Jenkins
2e0f812efa Merge "Add option to be able to run_functests.sh in parallel" 2016-10-18 21:03:05 +00:00
Matthew Thode
bb29082910
Document install of diskimage-builder on Gentoo
Change-Id: I27bf3499f998dcbdeb5cfebddde306e5200cc0b5
2016-10-11 20:17:49 -05:00
Jenkins
fd6e1475a1 Merge "Rename devloper documentation to developer guide" 2016-10-07 05:01:59 +00:00
Jenkins
1daae21508 Merge "Remove copyright from docs / toc" 2016-10-07 05:01:53 +00:00
Jenkins
a30a72a3b2 Merge "Use globaltoc in docs for sidebar" 2016-10-07 05:00:52 +00:00
Jenkins
ce461668a0 Merge "Fix developer guide TOC" 2016-10-07 05:00:50 +00:00
Gregory Haynes
7f9506962a Fix formatting for supported distros in docs
The supported distros table is not formatted consistently.

Change-Id: Ic9e32786419243c118088f933d86329480f3345b
2016-10-07 15:19:47 +11:00
Gregory Haynes
5758176a42 Rename devloper documentation to developer guide
We currently have 'user guide' and 'developer documentation'. Lets
rename to 'developer guide' for consistency.

Change-Id: I834ea313bc34275ef33e8c49a1689dff41892015
2016-10-07 15:15:30 +11:00
Gregory Haynes
9f3bb9652a Remove copyright from docs / toc
This is redundant (we have these copyright notices elsewhere) and
provide no value to the user.

Change-Id: I3699c01f0d4126cdffb3e3edf87ce62325f9f67a
2016-10-07 15:15:30 +11:00
Gregory Haynes
14b9d7ff3e Use globaltoc in docs for sidebar
By default sphinx uses localtoc which means 'show TOC for this page'.
Using a global table of contents on the sidebar is much more user
friendly wih our docs structure.

Change-Id: I215732d3848b4b75d9171bdbaaf2ff2e4dcc01f0
2016-10-07 15:15:30 +11:00
Gregory Haynes
c80cf9ec58 Fix developer guide TOC
The table of contents for our developer guide does not show due to the
fact that it is past the first sub-header.

Change-Id: I8459a4949e3e4822b0a3cd4f163475d2c60b0f2e
2016-10-07 15:15:28 +11:00
Jenkins
9a4f9e541e Merge "Add low-hanging-fruit bug tag to docs" 2016-10-07 04:07:36 +00:00
Gregory Haynes
3e4889cd6a Add low-hanging-fruit bug tag to docs
This is useful in the developer quickstart guide.

Change-Id: I54e7166bace055ae583f5dc84b648bba5f4f9cf4
2016-10-06 14:51:17 -07:00
Gregory Haynes
9a8d67e36f Add bugs link to docs
Change-Id: I09dcfe5abbc76e1383851282f063f4c338f6d93a
2016-10-06 14:49:30 -07:00
Hironori Shiina
30b3fc8dcc Fix a command in Developer Documentation
Fix a command for creating a new virtualenv.

Change-Id: Ia4981af390bf5218f22ea753db86a5edfbb602f2
2016-10-04 22:22:26 +09:00
Ian Wienand
18a4109713 Merge remote-tracking branch 'origin/master' into feature/v2
Change-Id: I796546f3873a0c9072215e63703aaa64753c5afd
2016-09-20 20:23:11 +10:00
Jenkins
61123b0286 Merge "Spec for changing the block device handling" 2016-09-12 21:44:20 +00:00
Jenkins
668561d156 Merge "Add specs dir" 2016-09-12 21:44:13 +00:00
Ian Wienand
91b431ce78 Move element-info to a standard entry-point
Move element-info from a wrapper script to a standard entry-point
console_script.

Update the documentation to explain how to run it for development.  I
don't think we should support the idea that you can check-out the code
and run ./bin/disk-image-create -- it has dependencies (dib-utils,
etc) and needs to be run from a virtualenv (this is what CI in the
gate does).  A follow-up can clean-up some of the path munging stuff
we have for this in disk-image-create.

Change-Id: Ic0c03995667f320a27ac30441279f3e6abb6bca8
2016-09-08 15:29:56 +10:00