Commit Graph

3236 Commits

Author SHA1 Message Date
Andreas Florath
6ca6057981 Add python logger configuration
This adds a common usable logging configuration for DIB, that
can be used by different python tools or elements.

This change is a factored out patch from the block device
refactoring.

Change-Id: I34d44610f74030936e7c0f602340b183baec3f1b
Signed-off-by: Andreas Florath <andreas@florath.net>
2016-06-16 13:54:50 +10:00
Thomas Bechtold
677324cb44 Remove discover from test-requirements
It's only needed for python < 2.7 which is not supported

Change-Id: I208bef315e5fd309ad0e8d915928fdc7aa0b9c91
2016-06-10 10:31:45 +02:00
Jenkins
d9525cfee9 Merge "Add cinder-backup mappings" 2016-06-10 01:20:45 +00:00
Ian Wienand
b9fdc70e32 Move pkg-map to dib-python
The latest Fedora/Ubuntu images don't ship python2 by default, so we
need to use our dib-python wrapper for this so we work in python3 only
environments.

Closes-bug: 1577105
Change-Id: I1048ceef35f269960216066924986eec6117ca00
2016-06-08 21:16:50 +00:00
Erno Kuvaja
6fd6b3bb25 Add cinder-backup mappings
'cinder-backup' to 'openstack-cinder-backup' mappings where necessary

Change-Id: I5f57401295d655112f0ad1d6442268df149daa7e
2016-06-07 14:15:47 +02:00
Jenkins
c621c5cc82 Merge "Release notes for 1.17.0" 2016-06-07 02:36:52 +00:00
Jenkins
fb92e61082 Merge "Add cloud-initramfs-growroot for Precise" 2016-06-07 01:14:10 +00:00
Jenkins
a10a664a2a Merge "Add PS4 to show file/function/line in debug output" 2016-06-07 01:14:08 +00:00
Jenkins
59bb62d1ee Merge "Refactor: remove unused functions" 2016-06-07 01:01:13 +00:00
Jenkins
dc149ec9fd Merge "Add release to pkg-map" 2016-06-07 01:01:09 +00:00
Jenkins
fb6fa48f85 Merge "Export FS_TYPE and remove hardcoded ext4 values" 2016-06-07 01:01:04 +00:00
Jenkins
9dedca018f Merge "Handle file magic type varying order matching" 2016-06-07 01:00:57 +00:00
Jenkins
13323c9495 Merge "Export die() function" 2016-06-07 01:00:50 +00:00
Jenkins
84051eb0a5 Merge "Refactor: rename temporary directories" 2016-06-07 00:59:49 +00:00
Jenkins
536e8a2dd2 Merge "Cleanup source-repositories output" 2016-06-07 00:59:02 +00:00
Ian Wienand
1401a1b2c6 Release notes for 1.17.0
Release notes for 1.17.0

Change-Id: I8d410c73b9492d8c5c6e8c017fa7eb6df09b7778
2016-06-07 10:57:10 +10:00
Ian Wienand
74d0185dde Cleanup source-repositories output
This element takes up a *huge* part of the logs of openstack-infra
builds as we go and cache every git tree.

This silences most of the noise which will reduce the mess
in the logs considerably.

Note that we've had logging on this turned down since
I91c5e55814ba9840769357261d203f4850e2eba6 but it has been
ineffective in stopping the log-spew output, see the
dependency change.

Change-Id: I60f06f84d57087c82b3907575bff125015d35171
Depends-On: I1e39822f218dc0322e2490a770f3dc867a55802c
2016-06-06 15:57:39 +10:00
Andreas Florath
a6a5c385eb Refactor: rename temporary directories
The temporary directories for image creation and building the OS both
started with 'image' as their names followed by some random
characters.  During debugging this is annoying, because on first sight
it is not clear, where which files are stored.

This patch renames them to dib_build.XXXXXXXX and dib_image.XXXXXXXX.

This patch introduces no user-visible change:
the temporary directories are only used during the
run of disk-image-builder.

Change-Id: I249cdb7750fe9a746b375b462789cd9b82681a2e
Signed-off-by: Andreas Florath <andreas@florath.net>
2016-06-03 07:21:03 +02:00
Andreas Florath
b365d3c833 Refactor: remove unused functions
There were a couple of functions which were unused:
ensure_nbd, map_nbd, unmount_qcow_image, mount_qcow_image, ensure_sudo
Because some of them use 'trap ... EXIT' this hinders introducing a
separate exit phase - therefore they are removed.

(It would also be impossible to use them in the current setup, because
they overwrite the 'trap ... EXIT' of the disk-image-creates 'main'.)

Change-Id: If932a557dca9aea4864154ad6c4f286373d6dd7c
Signed-off-by: Andreas Florath <andreas@florath.net>
2016-06-02 21:06:12 +02:00
Jenkins
38dcce0fc4 Merge "Fix variable unbound error while REPOREF="*"" 2016-06-02 05:20:37 +00:00
Jenkins
419568884c Merge "Rework yum-minimal locale cleanup" 2016-05-31 06:42:20 +00:00
Ian Wienand
f5dff9c52a Rework yum-minimal locale cleanup
It turns out our manual locale cleanup is causing issues (see
I54490b17a7f8b2f977369044fcc6bb49cc13768e).  Upon further
investigation, I think this is a better approach than manually
deleting repos.

glibc on Fedora obeys the %_install_langs macro for reducing the
installed locales (as mentioned in the comments, F24 has moved to
having different packages, but worry about that later).

So our existing clear-out is really only required for CentOS, whose
glibc does not have any way to indicate to build less locales.
However, %_install_langs is still correct there, as it restricts some
of the translation files and other things installed with the %lang
macro in spec files.

This is complicated by us having to set this at glibc-common install
time, which happens with the "yum" from outside the chroot (i.e. on
trusty).  Since this is too old to have flags to pass this, we need to
fiddle with rpmmacros.

I've tested this with fedora-minimal builds and the locales file is
about 2MiB, which is what it was after the cleanups, and the listed
locales are only those we expect (i.e. it appears to be working).

Change-Id: I528a68beeb7b2ceec25ccbec1900670501608158
2016-05-31 15:14:24 +10:00
Andreas Florath
9fbc462e00 Remove Fedora 21 from test-build
Because Fedora 21 is EOL, the appropriate cloud images were
removed from the mirrors during the last days.
Because of the removal, currently all CI tests are failing.

This patch removes the Fedora 21 CI tests.

(Adding and supporting Fedora 23 is done with another patch -
because some additional changes are needed here.)

Change-Id: Ib85bb6fafd4f56ecc55dd420048f4d9e6e6969f3
Signed-off-by: Andreas Florath <andreas@florath.net>
2016-05-30 22:54:58 +02:00
Ian Wienand
7550d25db0 Export die() function
I realised I'd been using die() in a few places assuming it was
available, but it wasn't exported.  I guess it didn't matter because
whatever was wrong, we were failing anyway :)

This exports the function to make it available to sub-processes, which
should remove the need to source it as done in several places.

Change-Id: I7b9a5a6db406e160099b6ed9fde80455ae227327
2016-05-27 09:25:22 +10:00
Jenkins
5f4cac3303 Merge "Add a best-effort sudo safety check" 2016-05-26 17:31:00 +00:00
Liu Qing
d3255835d1 Fix variable unbound error while REPOREF="*"
If REPOREF="*", HAS_REF will be used without initialization. As -u is set
the script will terminate with error.

Change-Id: Ic1d88415adfef66dfc6c1d92610a45a9eb6359f3
Signed-off-by: Liu Qing <liuqing@chinac.com>
2016-05-26 09:24:59 +08:00
Darragh Bailey
716be1d6f0 Handle file magic type varying order matching
Allow file test to return all possible mime-types and ensure anything
that matches the python file type is parsed by flake8 instead of
relying on the first match returned.

Closes-Bug: #1585688
Change-Id: Iba31f1853537fe9234ab6f83d66f13dc1c578abb
2016-05-25 17:55:59 +01:00
Jiri Stransky
1e9cf3a1c8 Remove deprecated overriding of cloud-init defaults
The "set to localhost by default" behavior for manage_etc_hosts has been
deprecated for more than a year now by change
Ia8582883f737548e2911d3f36a1943e5b236281b.

Setting that value to "localhost" is still possible, but it won't be the
default anymore. If the previous behavior of assigning the hostname and
FQDN to 127.0.0.1 is still desired in some environments, it can be
achieved by setting the DIB_CLOUD_INIT_ETC_HOSTS environment variable,
as the deprecation warning message suggested.

Change-Id: I5a19d46e2f305769a0c89c9d25d2e6be02910221
2016-05-23 17:44:32 +02:00
Jenkins
f0b57d5efd Merge "Fix apt-sources configuration for debian-minimal" 2016-05-21 20:10:18 +00:00
Ian Wienand
244f0a0001 Add cloud-initramfs-growroot for Precise
Add a pkg-map to install cloud-initramfs-growroot on Precise, as it is
required there because the kernel is too old to use growroot on a live
file-system.  This was generically removed in
e23b087505.

Change-Id: I5e6cbc4b74dc72a6f23f73a018c028f3ff7c0157
2016-05-20 14:32:41 +10:00
Ian Wienand
716b8e14fe Add release to pkg-map
Add ability to filter in pkg-map on release, which defaults to
DIB_RELEASE.  As per the examples, release is a more specific target
than distro, but distro still has to match.

Added a debug flag and ability to use a pkg-map directly for
debugging/development.

Change-Id: Ie282f96966e46236b06bc276de0168fc7a66c5da
2016-05-20 14:26:25 +10:00
Paul Belanger
be521bdec6 Export FS_TYPE and remove hardcoded ext4 values
Export FS_TYPE from img-defaults and use it to remove hard-coded
defaults in the debootstrap mounting.  Also, cleanup the suse element
as it should have access to the exported variable.

Change-Id: Ie9b671ca9336060a5ad294be48aa7eff442bf066
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2016-05-19 17:33:36 +10:00
Jenkins
95c874abb9 Merge "Add dhcp-all-interfaces.target for syncing units" 2016-05-18 19:39:10 +00:00
Ian Wienand
34efed027a Add PS4 to show file/function/line in debug output
For something fairly simple, I went back-and-forward with this a bit.

Firstly, I realise calling readlink constantly sucks.  Due to the way
we call dib and source various files, you end up with the source-file
from "caller" being usually a very ugly path including levels of "../"
indirection.  Cleaning this up to something canonical is the only sane
way to present it.

Because we evaluate _ps4() from a sub-shell in the PS4 string, there's
no way for it to do something like build a global in-memory cache in
an associative array or similar.  It could write out a temp file or
some other side-band method, but the overheads of managing this don't
seem any different to just calling readlink.  If anyone can think of a
bash-hack around this that doesn't involve a fork() I'm interested.

We could potentially strip some of the leading paths in the assumption
you know what they are; but it gets complex when things are split
across /usr/bin & /usr/lib and external elements, etc.  I thought
about arbitrarily shortening it (e.g. just take last 20 characters)
which gives you enough of an idea of the file, but looks a bit ugly.
Or we could just leave the file-name out all together and assume the
function name is unique enough; this also seemed a bit ugly.

Obviously it's a matter of taste in the output.  It is certainly
wider, but it also adds a lot of information.  It also makes it fairly
clear where there are things we can make less verbose,
e.g. I1e39822f218dc0322e2490a770f3dc867a55802c disables tracing in
run-parts which is just noise.  There's a few other frequently used
loops that we could disable tracing for by default to benefit
signal:noise.

tl;dr : take a look at the logs.  I think it is a step in the right
direction of making the logs more usable for debugging.

Change-Id: I8054a3050415fcb527baeb7012bf133e5c864bf3
2016-05-18 09:23:15 +10:00
Ben Kero
9a00bf144f Revert "Revert "Properly fail/trap in eval_run_d"" and fix PIPESTATUS
This reverts commit f07e33a2e9.

This change reverts the revert while fixing the underlying issue --
$PIPESTATUS needed to be encapsulated in ${}s

Change-Id: I1df06ffa7aecf4ea4b8e187dc756e9fc779786bc
2016-05-16 16:25:45 -07:00
Jenkins
99ddd58a15 Merge "Revert "Properly fail/trap in eval_run_d"" 2016-05-16 22:46:31 +00:00
Jenkins
2fffc9f892 Merge "Fix path issue for locale-archive.tmpl" 2016-05-16 22:01:47 +00:00
Jenkins
02822dc6b9 Merge "Add centos7 test" 2016-05-16 21:43:28 +00:00
Paul Belanger
0478fb15db
Fix path issue for locale-archive.tmpl
Change-Id: Id589c16aab46d447b3c21f00f3acfd06890e43d2
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2016-05-16 09:23:58 -04:00
Derek Higgins
f07e33a2e9 Revert "Properly fail/trap in eval_run_d"
This reverts commit 0d1d6bec7c.
This patch breaks tripleo-ci (the instack.qcow2 images is failing
to build) and was merged without passing CI.

Closes-Bug: #1582115
Change-Id: Ic4725ad0689c937fb4c8c792e1eaff5f4ea9ada9
2016-05-16 13:16:12 +01:00
Sam Betts
eb99fe7144 Add dhcp-all-interfaces.target for syncing units
Currently there is no way for a service to become aware that
dhcp-all-interfaces is finished configuring all the interfaces at
boot time. This causes problems for applications like the
ironic-python-agent which scans the interfaces when it first starts as
part of the inspection stage and can race against dhcp-all-interfaces
bringing up the interfaces, leading to inconsistent results.

This patch ensures that the dhcp-all-interfaces script runs before any
network interface is configured and brought up by the rest of the
system, and also ensures that the ironic-agent element also waits for
the network to be online before starting. This is done by using the
network targets provided by systemd.

Change-Id: Id9583b7f54361aa603a6229da598ad6a0f0f7938
2016-05-16 10:15:53 +01:00
Ian Wienand
e142014dbd Small doc update, add link to relnotes
I noticed we have no way easy way to get to the release notes.  I
updated the front-page text to be a simpler introduction/overview and
provided links to the relnotes and source directly.

Change-Id: I5e339baf2921752ca3d409d82e0cbfc856ead1f8
2016-05-16 15:13:39 +10:00
Ian Wienand
5d23abb338 centos-minimal: can be used with base
I'm not sure why this comment is here.  base works with centos-minimal
(we changed it in I2956aaa49ba3137a799f97e0983ab4b7c93a0a0c) and we've
got images deployed with such a configuration.

Update the comment

Change-Id: I3207f87dc29280183c0960ea863533a4d441081c
2016-05-16 13:36:55 +10:00
Jenkins
ae10241f3e Merge "Add 1.16.0 releasenotes" 2016-05-16 03:05:03 +00:00
Jenkins
e7625214f6 Merge "Properly fail/trap in eval_run_d" 2016-05-16 03:04:56 +00:00
Jenkins
d253a94187 Merge "Do not remove sudo in ironic-agent" 2016-05-16 01:41:08 +00:00
Gregory Haynes
0d1d6bec7c Properly fail/trap in eval_run_d
In phases which are called from eval_run_d (block-device.d) we do not
listen to exit 1's nor do we allow break=after-error. This is because
the run_d function is called in a subshell in order to grab its output.

This also turns on pipefail in the main disk-image-create script.

Change-Id: I88ab2e7104148437eabfe6880e3a1e5ebbb2c15d
2016-05-16 01:37:02 +00:00
Ian Wienand
8bda23a1d8 Add 1.16.0 releasenotes
Add some releasenotes for prior changes since 1.15.0

Change-Id: Ie2e6f7672df2076c2936ef4d0bbcc477abb0fa76
2016-05-16 11:09:39 +10:00
vmud213
45628993e8 Do not remove sudo in ironic-agent
"ironic-agent" element is currently removing sudo, which breaks other
elements such as devuser.  There appears to be no security or other
reason to do this, it's just the way it has always been.  Leave sudo
in as it is considered part of the base cloud images.

Change-Id: Ida9b1885f745146071e4b2d85ae59341ac85d5c8
Closes-Bug: #1572486
2016-05-16 10:39:04 +10:00
Gregory Haynes
feb4eda17a Fail functests if refusing to run tests
We made this non-failing so tests could pass before
I92299b2f59aa8e7de8995a6294bf6c88bbd4cdc5 merges. We should make this a
test failure.

Change-Id: I4dd38c835d6fb397bbd0475120a4b0f2ab0290a8
Depends-On: I92299b2f59aa8e7de8995a6294bf6c88bbd4cdc5
2016-05-13 17:07:32 +00:00