Commit Graph

191 Commits

Author SHA1 Message Date
Ian Wienand
f60dd38482 Add timestamp output filter
This adds a devstack-inspired output filter to standardise
timestamping.

Currently, python tools timestamp always (timestamp setup in
logging_config.py) but all the surrounding bash does not.

We have extra timestamps added in run_functests.sh for our own
purposes to get the bash timestamps; but this ends up giving us
double-timestamps for the python bits.  Additionally, callers such as
nodepool capture our output and put their own timestamps on it, and
again have the double-timestamps.

This uses a lightly modified outfilter.py from devstack to standardise
this.

All output is run through this filter, which will timestamp it.  I
have removed the places where we double-timestamp -- logging_config.py
and the prefix in dib-run-parts.

An env option is added to turn timestamps off completely (does not
seem worth taking up a command-line option for).  For callers like
nodepool, they can set this and will just have their own timestamps as
they collect the lines.

Since all logging is going through outfilter, it's easy to add a
--logfile option.  I think this will be quite handy; personally I'm
always redirecting dib runs to files for debugging.

I've also added a "quiet" option.  I think this could be useful in
run_tests.sh if we were to start logging the output of each test to
individual files.  This would be much easier to deal with than the
very large log files we get (especially if we wanted to turn on
parallel running...)

Change-Id: I202e1cb200bde17f6d7770cf1e2710bbf4cca64c
2017-06-16 15:58:50 +10:00
Mark Goddard
54765fd2f4 Remove dracut-network element
This element has not been functioning correctly for some time due to
an incorrect path to select-boot-kernel-initrd (should be /usr/local/bin).

The dracut-regenerate element can be used to regenerate dracut ramdisks
and is more flexible than this element.

Change-Id: I33d555ffd4a92b2948b2ea4a66b151f0422ccb8c
Closes-Bug: #1688546
2017-05-31 08:36:56 +10:00
Andreas Florath
e4e23897a1 Refactor: block-device filesystem creation, mount and fstab
This patch finalizes the block device refactoring.  It moves the three
remaining levels (filesystem creation, mount and fstab handling) into
the new python module.

Now it is possible to use any number of disk images, any number of
partitions and used them mounted to different directories.

Notes:

 * unmount_dir : modified to only unmount the subdirs mounted by
   mount_proc_sys_dev().  dib-block-device unmounts
   $TMP_MOUNT_PATH/mnt (see I85e01f3898d3c043071de5fad82307cb091a64a9)

Change-Id: I592c0b1329409307197460cfa8fd69798013f1f8
Signed-off-by: Andreas Florath <andreas@florath.net>
Closes-Bug: #1664924
2017-05-12 13:52:02 +02:00
Jenkins
a076bf66d6 Merge "Move do_extra_package_install to run in install phase" 2017-04-10 01:24:55 +00:00
Noam Angel
95503b4201 Move do_extra_package_install to run in install phase
This change move "do_extra_package_install" from pre-install to install
phase.

Extra packages are added by user request using the flag "-p", This
package should not be something the elements depend on.
The reason behind this patch is to move the extra package install to
a proper phase, Also more reasonable if base element run package update
to be before we install extra packages.

Change-Id: I68cc773aba9aa01743f0dda9f4e635e4cac2a282
2017-04-10 08:32:47 +10:00
Ian Wienand
6802cf7100 Run dib-run-parts out of /tmp
The dib-run-parts element was copying our internal version of
dib-run-parts into /usr/local/bin to be used running scripts inside
the target chroot.  However, it never cleaned up after itself.  This
means all images were left with an unmanaged local install of
dib-run-parts.

This copies dib-run-parts into the hooks directory of the chroot and
runs it from there.  It is cleaned up automatically on the exit path.

The dib-run-parts element is no longer required and it has been
removed from all dependencies.  It is left with a deprecation notice
in the README.  For compatability we convert it to simply install
dib-utils.

Codesearch shows no users depending on this unintentional implicit
install.  Note os-refresh-config depends on dib-utils and thus will
have an explicitly installed version.

Partial-Bug: #1673144
Change-Id: Ia2e96c00a4246c04beb96c17f83b8aefb69219ca
2017-04-05 13:11:22 +10:00
Ian Wienand
fd424757a6 Don't provide dib-run-parts
It was an oversight during v2 development for dib to start providing
dib-run-parts.  The intention was for dib to use a vendored
dib-run-parts directly from $_LIB and have no dependencies on
dib-utils at all.  By exporting dib-run-parts, we created an
unintentional conflict with the dib-utils package which provides the
same script.

Tools that depend on dib-utils are unaffected by this
(os-refresh-config).

The only tool that installs diskimage-builder and then assumes
dib-run-parts is available in the path is instack.  I have proposed
Ibfe972208df40fa092b11b5419043524c903f1b4 to modify that to use our
internal version.

Change-Id: I149c345d38d761a49b3a6ccc4833482f09f1cd05
2017-04-05 13:11:20 +10:00
Jenkins
8b19f7f99d Merge "Fix dib-init-system for Debian Jessie and Debian Stretch" 2017-03-13 03:29:41 +00:00
Andreas Florath
20389d755f Fix dib-init-system for Debian Jessie and Debian Stretch
On Debian Jessie and Debian Stretch systemctl is in /bin.
If the package systemd-sysv is not installed the script
dib-init-system did not find the init system.
This patch fixes the problem: it also looks in /bin
for systemctl and if found decides for systemd.

Change-Id: I5a18052a070bad5e16b14672237a1e2b38513949
Signed-off-by: Andreas Florath <andreas@florath.net>
2017-03-11 14:04:13 +00:00
Ian Wienand
6887e796e1 Add 2.0.0 release notes
This is a distillation of a longer email I sent to openstack-dev [1]
as an omnibus update for 2.0.0 release notes.

[1] http://lists.openstack.org/pipermail/openstack-dev/2017-February/111833.html

Change-Id: Ic0b012626e0850c41d532e5dfdf82538cc6397d8
2017-03-10 14:29:30 +11:00
Andreas Florath
fdd2c4b236 Semi-automatic doc generation of element dependency
Currently there is no description of dependencies in the generated
documentation of the elements: therefore a user of an element does not
know which other elements are automatically included and e.g. which
configuration options are available. In addition there are some
copy&pastes of parts of the README.rst scattered thought different
Ubuntu and Debian specific elements.

This patch adds a semi-automatic generation of dependency information
of all elements.  Nevertheless these are not automatically included.
The author of the element's README.rst can decide if and where the
dependency information should appear and can use the descriptor

.. element_deps::

for this.

This patch adds the dependency information for some Debian and
Ubuntu patches - and creates the base for later removing the
duplicated parts.

A call is added to element_dependencies._find_all_elements() to
populate reverse dependencies for Element objects.

(This is a reworking of I31d2b6050b6c46fefe37378698e9a330025db430 for
the feature/v2 branch)

Change-Id: Iebb83916fed71565071246baa550849eef40560b
2017-02-09 09:50:30 +11:00
Ian Wienand
3f8800832a Release 1.27.0
-----BEGIN PGP SIGNATURE-----
 
 iQEcBAABAgAGBQJYk8+9AAoJEBty/58O8cX8LdIH+wU/VrEVs0XYohiL6DUgabzs
 112U3UUihH5xMc/ca9Tarx+XwEvfMZkwYN2Qr0JoRJjmSt2AL6AezUhGSV+98vaY
 iQEccaFDFYlyDHm4V2r7N1xwS0B3mx87FPqVQQSUKlc3JsQxCy4o9RtD9aM8Gvqy
 +gAxMxL3p3O131K0Rvb0U5lC1FLgft9SuljCV8i5nU4/HdoryD6hedz2/ss8a9KG
 KKEdBKvPBKn73+nb8peQD/VXpej9C31r87q5VEjUsZkJ7gduY/qYLlGGgoBQqAXN
 WQ/ef1RkQKW5ba2jsjnk7fdOrA0+wYENxorR2WecuZbe2ieXw6fP3lYiD6VeWsM=
 =IUuh
 -----END PGP SIGNATURE-----

Merge tag '1.27.0' into merge-branch

Release 1.27.0

Change-Id: I9f6948636cae6d375d1d8315976504021f5a3bbb
2017-02-03 11:49:45 +11:00
Jenkins
111cb51055 Merge "Unify tidy up logs in lib/img-functions" 2017-02-03 00:29:48 +00:00
Ian Wienand
adf39c52cf Release 1.21.1
-----BEGIN PGP SIGNATURE-----
 
 iQEcBAABAgAGBQJYW2GoAAoJEBty/58O8cX8uSAH/15dJsglP6Zie7jSSJcR6k+e
 PJembHn9qrqrCjmJ5EwakojySaaLhwEJKvlP54OU9v7pmUXL9gJtK2OzW54LQ41g
 xBHIu0Pg4z7juyHm9+1P2Sr7Mzs1pVSEbsIYpDYUU19eghI1EAeIj3I1woKgajN7
 JlI61j3r67G6EAVtPOnmD1jvXS8CrtjiJ9wtWTH20pWfmksovg/GuXUCZrLkAAhO
 NcK35CdMii1Hkr7XOH424La/Ar+3qfUX18ZkbJY6yHzkq/ityTzzKOFjAaDl2Jg9
 WNc+SLCVYpPhPwgt7miTywamUNj3ZviA5/Hd8fuLXmtHSLQ23WOtBiaQMLtwXHs=
 =8dIl
 -----END PGP SIGNATURE-----

Merge tag '1.26.1' into merge-branch

Release 1.21.1

Change-Id: Ib9eb3dd1d384fc5b6a9846608216e056c57a173a
2017-02-02 20:36:23 +11:00
Ian Wienand
b6e631360f Release 1.24.0
-----BEGIN PGP SIGNATURE-----
 
 iQEcBAABAgAGBQJYP+r7AAoJEBty/58O8cX8ZcMIAI7Yy2BfGqY/jfyLYNryATSu
 g78McMxAeOiPYII+KKyVkeWsChKJUafLZr/ppO95pm7ZLXeHQeC4MHDI68RmghIX
 HVXn4hPYzwlZ+Yr4hqaEf11H+vw0TDxo2a5bxlQZw+FCx9+3lkR60QxWf8a6QCxv
 nLqKIhtpwGi9mIyvAya+MaOmHMHqbNYgIBHKP0yXhTdxoHZoJWZeXN1rkelwAF9N
 TS1+KwIk+uemn1ZlqzYDWj8euyFyAv8L0WlF0ztCKe7TAFvNeMo26fLykeKHVDEo
 GC5R8MshiOtihPC972rNIponIzd62AetTlDxfK+gcUZO21pNTrlgdLX5Jn04fo0=
 =BpQc
 -----END PGP SIGNATURE-----

Merge tag '1.24.0' into merge-branch

Release 1.24.0

Change-Id: I23cfa6f8dc66bb6a1d957a49a5b468cfd47c005b
2017-01-31 14:14:19 +11:00
Andreas Florath
ec7f56c1b2 Refactor: block-device handling (partitioning)
During the creation of a disk image (e.g. for a VM), there is the need
to create, setup, configure and afterwards detach some kind of storage
where the newly installed OS can be copied to or directly installed
in.

This patch implements partitioning handling.

Change-Id: I0ca6a4ae3a2684d473b44e5f332ee4225ee30f8c
Signed-off-by: Andreas Florath <andreas@florath.net>
2017-01-24 19:59:10 +00:00
Jenkins
f3d0d6e8ec Merge "Remove yum chroot caching" 2017-01-17 01:10:41 +00:00
Andreas Florath
022d93ee82 Unify tidy up logs in lib/img-functions
Cleaning logs was split, some was done in the
img-functions.finalise_base, some was done in the base element.
The version unifies tidy up logs in the lib/img-functions.
Especially when building docker container images the base element
cannot be used. This patch removes about some hundreds KB of
useless logs in cases when the base element is not used.

Change-Id: I165bafb73daf9144c2f3a83930e85e8d8cf5fae3
Signed-off-by: Andreas Florath <andreas@florath.net>
2017-01-14 09:40:44 +00:00
Jenkins
c73e6b5ab1 Merge "Set grub timeout default" 2016-12-20 22:45:07 +00:00
Ian Wienand
4585955a8b Remove yum chroot caching
Every run we are doing a full tar.gz of the chroot environment that
never gets used.

It's not suitable for CI since we use fresh images each time there.

The cache in general isn't really isn't a very safe thing to have
around, because there's no invalidation procedure and no real way to
make one -- we've no guarantee that a new chroot build even moments
after a previous one wouldn't bring in or different packages, etc (of
course this is *unlikely*, but the longer you go between builds the
worse the problem becomes.  Also, tons of packages get installed after
this not from any cache, so potential speed-up is rather marginal.

Debian turned this off with I58fc485aacacaa17243bf9ce760ed91256d1f182.
However, given the reasons above and it's complete lack of testing, I
don't see this as useful.

If we really want this type of thing, I think we should come up with a
way to use a persistent external yum/dnf cache that yum/dnf keeps in
sync with it's usual invalidation rules.

Change-Id: I66789c35db75c41bc45ea1ad2e26f87456de4e4d
2016-12-20 13:56:46 +11:00
Ian Wienand
61087d33e9 Set grub timeout default
Set the grub timeout to 5 seconds by default, and add notes on how to
update this.  This will stop infra having to carry an element that
goes and rewrites the grub configuration.

Change-Id: I556b3f48eff1b67ee8c4b9b64f749af95100fb99
2016-12-20 11:46:22 +11:00
Tristan Cacqueray
9d13084c41 Add squashfs output image format
The squashfs format brings a couple of advantages over the other
formats. Image is often an order of magnitude smaller and it can
be used natively, either as an initrd, either with loop mount.

Change-Id: If72940b0c4dafb2504c52dd0429a8eb3f8305751
2016-12-19 07:21:39 +00:00
Jenkins
fffe15e763 Merge "Perform package install outside of debootstrap" 2016-11-30 05:32:09 +00:00
Gregory Haynes
45df304d48 Perform package install outside of debootstrap
Debootstrap only supports one apt repository to install packages from.
As a result, we do not consider the updates repo during debootstrap
causing us install a second kernel when we do an apt-get dist-upgrade
during build.

Lets use debootstrap to get us a minimal chroot, then add our repos and
install the correct packages from the start.

We also have to reorder the dpkg root.d scripts which configure apt so
they run before we perform our package installs.

Change-Id: I6a592db6f0a01d3b19d8e0786e63f1315a1ef647
Closes-Bug: #1637516
2016-11-30 15:16:46 +11:00
Ian Wienand
448a2602fe Merge remote-tracking branch 'origin/master' into merge-branch
Change-Id: I05cc51c699008018d557ed3874d71af26fd60240
2016-11-29 07:43:46 +11:00
Markos Chandras
bbcc22751f elements: Add new openssh-server element
Add new 'openssh-server' element to ensure that openssh server
is installed and enabled during boot. This is mostly useful for
*-minimal images which do not come with openssh installed and/or
enabled in order to keep a small dependency footprint.

Change-Id: Ide15ee04f5de123dbc8ce4bb56d638d8a167c341
2016-11-22 10:07:14 +00:00
Ian Wienand
d65678678e Move dib-run-parts into diskimage-builder
Move dib-run-parts from dib-utils into diskimage-builder directly.

For calling outside the chroot, we provide a standard entry-point
script.  However, as noted in the warning comment, the underlying
script is still copied directly into the chroot by the dib-run-parts
element.  I believe this to be the KISS approach.

This removes the dependency on dib-utils.  We have discussed this
previously and nobody seemed to think retiring dib-utils was going to
be an issue.

This also updates the documentation to not mention dib-utils, or using
disk-image-create via $PATH setup, but rather gives instructions on
installing from pip with a virtualenv.

Change-Id: Ic1e22ba498d2c368da7d72e2e2b70ff34324feb8
2016-11-04 17:07:37 +11:00
Gregory Haynes
7cc65d3cbd Merge branch 'master' into feature/v2
Change-Id: I0747a27560161ef869868967ab140b5d0be9d4da
2016-10-31 11:21:38 -07:00
Markos Chandras
90536dbab3 Add opensuse-minimal element
Add a new opensuse-minimal element to build small and highly
configurable openSUSE based images using the zypper-minimal element
as the main building mechanism

Change-Id: Iebfc4ad4aff763e511b093f1607b55851ccbddcb
2016-10-05 09:39:29 +01:00
Paul Belanger
45467e4229
Create runtime-ssh-host-keys element
Move managing of SSH host keys into a dedicated element.

Because glean doesn't generate SSH host keys anymore, we need to do it
with a systemd script. This is already handled by CentOS / Fedora so
we don't want to add it there.

This was done to address the upstream bug in debian:

  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=500192

Change-Id: I31ad667672e08350872db21a83445fe0aa7a4a39
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2016-09-26 12:32:38 -04:00
Andreas Florath
19efc60ce8 Add release notes for block device handling.
Change-Id: I646447ef6d067cdc4895cedefc3fcae37986d1c7
Signed-off-by: Andreas Florath <andreas@florath.net>
2016-09-23 09:36:19 +02:00
Ian Wienand
91b431ce78 Move element-info to a standard entry-point
Move element-info from a wrapper script to a standard entry-point
console_script.

Update the documentation to explain how to run it for development.  I
don't think we should support the idea that you can check-out the code
and run ./bin/disk-image-create -- it has dependencies (dib-utils,
etc) and needs to be run from a virtualenv (this is what CI in the
gate does).  A follow-up can clean-up some of the path munging stuff
we have for this in disk-image-create.

Change-Id: Ic0c03995667f320a27ac30441279f3e6abb6bca8
2016-09-08 15:29:56 +10:00
Ian Wienand
37a53354ec Add IMAGE_ELEMENT_YAML and get_image_element_array
These new variables are a list of elements chosen for the build along
with their full paths.  For Python elements, IMAGE_ELEMENT_YAML is a
YAML formatted list that can be easily parsed.  For bash elements,
"get_image_element_array" will produce an associative-array of the
same (working around lack of array export in Bash).

This list is intended for consumption of elements who need to copy
files from other elements, such as pkg-map and svc-map.  As discussed
in I2a29861c67de2d25c595cb35d850e92807d26ac6, this list has already
been pruned and had overrides processed, so it is safe to simply walk
over this list with no further processing.

Since we're presenting the element list in a couple of different ways,
we combine it all into the element-info script.  It will output an
eval-able string that declares the appropriate variables.

I've added some inline documentation so they still appear in grep.
The documentation is updated with examples, and moved to a more
appropriate location as a sub-section of the element sytle guide.

To test this out, use the associative-array in generate_hooks, where
we can now find the element's directory without searching.

Change-Id: Ibbd07d082ec827441def2d3f6240df3efdc6eae3
2016-09-08 11:08:07 +10:00
Ian Wienand
274be6de55 Making element overriding explicit
This is a re-factor of element_dependencies to achieve two things --
centralising override policy and storing path names.

Firstly we want to make the override policy for elements completely
explicit.  Currently, elements that wish to copy parts of other
elements walk ELEMENTS_PATH themselves and look for elements in
IMAGE_ELEMENT.  How they handle duplicate elements can differ, leading
to inconsistent behaviour.

We introduce logic in element-info to find elements in each of the
directories in ELEMENT_PATHS in *reverse* order -- that is to say,
earlier entries in the paths will overwrite later ones.

For example

 ELEMENT_PATHS=foo:bar:baz

will mean that "foo/element" will override "baz/element", since "foo"
is first.  This should be sane to anyone familiar with $PATH.
Documentation is clarified around this point and a test-case is added.

The second thing is that we want to keep the complete path of the
elements we have chosen.  We want the aforementioned elements that
walk the element list to use these canonical paths to pickup files;
this way they don't need to make local decisions about element
overrides, but can simply iterate a list and copy/merge files if they
exist.

A follow-on change (I7092e1845942f249175933d67ab121188f3511fd) will
expose this data in a separate variable that can be parsed by elements
(a further follow-on I0a64b45e9f2cfa28e84b2859d76b065a6c4590f0
modifies the elements to use this information).  Thus this does not
change the status-quo -- elements that are walking ELEMENTS_PATH
themselves and can/will continue doing that.

Change-Id: I2a29861c67de2d25c595cb35d850e92807d26ac6
2016-09-08 10:58:19 +10:00
Jenkins
b8d10afd8f Merge "package-installs: add list to arch and "not-arch" list" 2016-07-15 06:57:59 +00:00
Gregory Haynes
fcec06bfc0 Add 1.18.1 releasenotes
Change-Id: I5f820ea419fafa8f8d4c8b67522090003f95464e
2016-07-05 22:17:50 +00:00
Ian Wienand
8a1c8370a1 package-installs: add list to arch and "not-arch" list
Icf8a075224833fcfbbe2128e8802ff41c39f3c09 looked rather ugly, and it's
easy for us to expand the processing done in the arch list.

Change "arch" to a comma-separated list of architectures that should
match for install.

Add a "not-arch" list which will exclude the package from installation
on those architectures.  (An aside -- I considered making it just he
one list with foo,!bar,moo but ! has special meaning in YAML, so it's
easier to have two lists).

$ ARCH=ppc64 package-installs-squash --elements ironic-agent --path=./elements/ /dev/stdout | grep dmidecode
$ ARCH=ppc64 package-installs-squash --elements ironic-agent --path=./elements/ /dev/stdout | grep lshw
    "lshw",
$ ARCH=amd64 package-installs-squash --elements ironic-agent --path=./elements/ /dev/stdout | grep lshw
$ ARCH=amd64 package-installs-squash --elements ironic-agent --path=./elements/ /dev/stdout | grep dmidecode
    "dmidecode",

Change-Id: Ic69dd02a09e6f3ba9078a2377d8df29871a20db2
2016-07-01 21:31:59 +02:00
Ian Wienand
ff5fd7b264 Release notes for 1.18
Mostly small bug fixes that don't need to be called out.

Change-Id: I6dc92a1b7f003eb0e8b81e6f32c46871b627454f
2016-06-28 09:20:21 +10:00
Ian Wienand
1401a1b2c6 Release notes for 1.17.0
Release notes for 1.17.0

Change-Id: I8d410c73b9492d8c5c6e8c017fa7eb6df09b7778
2016-06-07 10:57:10 +10:00
Ian Wienand
8bda23a1d8 Add 1.16.0 releasenotes
Add some releasenotes for prior changes since 1.15.0

Change-Id: Ie2e6f7672df2076c2936ef4d0bbcc477abb0fa76
2016-05-16 11:09:39 +10:00
Ian Wienand
634391185c Add releasenotes
Use reno to start at keeping release notes.  Add an initial log.

Change-Id: Iba3ebd3b01c15030ac2585dda82e43657e511310
2016-04-21 13:19:53 +10:00