Commit Graph

2341 Commits

Author SHA1 Message Date
Ian Wienand
8a1c8370a1 package-installs: add list to arch and "not-arch" list
Icf8a075224833fcfbbe2128e8802ff41c39f3c09 looked rather ugly, and it's
easy for us to expand the processing done in the arch list.

Change "arch" to a comma-separated list of architectures that should
match for install.

Add a "not-arch" list which will exclude the package from installation
on those architectures.  (An aside -- I considered making it just he
one list with foo,!bar,moo but ! has special meaning in YAML, so it's
easier to have two lists).

$ ARCH=ppc64 package-installs-squash --elements ironic-agent --path=./elements/ /dev/stdout | grep dmidecode
$ ARCH=ppc64 package-installs-squash --elements ironic-agent --path=./elements/ /dev/stdout | grep lshw
    "lshw",
$ ARCH=amd64 package-installs-squash --elements ironic-agent --path=./elements/ /dev/stdout | grep lshw
$ ARCH=amd64 package-installs-squash --elements ironic-agent --path=./elements/ /dev/stdout | grep dmidecode
    "dmidecode",

Change-Id: Ic69dd02a09e6f3ba9078a2377d8df29871a20db2
2016-07-01 21:31:59 +02:00
Mark Hamzy
76bf793a39 dmidecode does not exist for ppc64/ppc64el
Adding three separate lines because the arch tag does not support a list.

Change-Id: Icf8a075224833fcfbbe2128e8802ff41c39f3c09
2016-06-29 16:51:46 -05:00
OpenStack Proposal Bot
7356941996 Updated from global requirements
Change-Id: I2b3f5b34463b14028213f95933c5b53c2db9eb5d
2016-06-21 17:58:29 +00:00
Jenkins
abe666f17e Merge "Remove discover from test-requirements" 2016-06-17 05:19:11 +00:00
Jenkins
f9d3bf0b1c Merge "Pre-install pip/virtualenv packages" 2016-06-17 05:05:11 +00:00
Jenkins
d86ad43060 Merge "Ironic agent kernel should be owned by user building image" 2016-06-17 01:09:20 +00:00
Ian Wienand
a85ce75d6b Pre-install pip/virtualenv packages
If we're installing pip/virtualenv from source, we need to make sure
we pre-install the packaged versions before the upstream
versions. Otherwise, CI jobs later on that depend on packaged versions
of pip/virtualenv can bring them in and overwrite the upstream
versions we have installed, which leads to a heck of a mess and
usually very confusing failures.

I have also moved in a small hack from system-config:install_puppet.sh
that we found was necessary when using pip versions from upstream.

Note this is not as much of an issue on Debian/Ubuntu, as they keep
their pip packages in a separate place to the system packages, so you
don't have these overwite conflicts as much.

Change-Id: Ib40708c07b939b84661c44df88a5a308fd0c7216
2016-06-17 09:20:57 +10:00
stephane
7eb72c0874 Ironic agent kernel should be owned by user building image
The initramfs file created by the ironic-agent element is
owned by the user running disk-image-create; ensure that
the other files created by the element are also owned by
the user.

Change-Id: I829db5b8e8bf1fc68face9cd2bda52d2a5ccdd4f
Closes-Bug: 1593010
2016-06-16 14:28:23 -07:00
Ian Wienand
8661dd1a31 Introspect logging testing more
This adds some fixtures to ensure the log output is what we expect,
along with ensuring we test the operation of the debug environment
variables.

Change-Id: I01c36299539b15a633b9307da5a348a5ae4e2563
2016-06-16 13:54:52 +10:00
Andreas Florath
6ca6057981 Add python logger configuration
This adds a common usable logging configuration for DIB, that
can be used by different python tools or elements.

This change is a factored out patch from the block device
refactoring.

Change-Id: I34d44610f74030936e7c0f602340b183baec3f1b
Signed-off-by: Andreas Florath <andreas@florath.net>
2016-06-16 13:54:50 +10:00
Thomas Bechtold
677324cb44 Remove discover from test-requirements
It's only needed for python < 2.7 which is not supported

Change-Id: I208bef315e5fd309ad0e8d915928fdc7aa0b9c91
2016-06-10 10:31:45 +02:00
Jenkins
d9525cfee9 Merge "Add cinder-backup mappings" 2016-06-10 01:20:45 +00:00
Ian Wienand
b9fdc70e32 Move pkg-map to dib-python
The latest Fedora/Ubuntu images don't ship python2 by default, so we
need to use our dib-python wrapper for this so we work in python3 only
environments.

Closes-bug: 1577105
Change-Id: I1048ceef35f269960216066924986eec6117ca00
2016-06-08 21:16:50 +00:00
Erno Kuvaja
6fd6b3bb25 Add cinder-backup mappings
'cinder-backup' to 'openstack-cinder-backup' mappings where necessary

Change-Id: I5f57401295d655112f0ad1d6442268df149daa7e
2016-06-07 14:15:47 +02:00
Jenkins
c621c5cc82 Merge "Release notes for 1.17.0" 2016-06-07 02:36:52 +00:00
Jenkins
fb92e61082 Merge "Add cloud-initramfs-growroot for Precise" 2016-06-07 01:14:10 +00:00
Jenkins
a10a664a2a Merge "Add PS4 to show file/function/line in debug output" 2016-06-07 01:14:08 +00:00
Jenkins
59bb62d1ee Merge "Refactor: remove unused functions" 2016-06-07 01:01:13 +00:00
Jenkins
dc149ec9fd Merge "Add release to pkg-map" 2016-06-07 01:01:09 +00:00
Jenkins
fb6fa48f85 Merge "Export FS_TYPE and remove hardcoded ext4 values" 2016-06-07 01:01:04 +00:00
Jenkins
9dedca018f Merge "Handle file magic type varying order matching" 2016-06-07 01:00:57 +00:00
Jenkins
13323c9495 Merge "Export die() function" 2016-06-07 01:00:50 +00:00
Jenkins
84051eb0a5 Merge "Refactor: rename temporary directories" 2016-06-07 00:59:49 +00:00
Jenkins
536e8a2dd2 Merge "Cleanup source-repositories output" 2016-06-07 00:59:02 +00:00
Ian Wienand
1401a1b2c6 Release notes for 1.17.0
Release notes for 1.17.0

Change-Id: I8d410c73b9492d8c5c6e8c017fa7eb6df09b7778
2016-06-07 10:57:10 +10:00
Ian Wienand
74d0185dde Cleanup source-repositories output
This element takes up a *huge* part of the logs of openstack-infra
builds as we go and cache every git tree.

This silences most of the noise which will reduce the mess
in the logs considerably.

Note that we've had logging on this turned down since
I91c5e55814ba9840769357261d203f4850e2eba6 but it has been
ineffective in stopping the log-spew output, see the
dependency change.

Change-Id: I60f06f84d57087c82b3907575bff125015d35171
Depends-On: I1e39822f218dc0322e2490a770f3dc867a55802c
2016-06-06 15:57:39 +10:00
Andreas Florath
a6a5c385eb Refactor: rename temporary directories
The temporary directories for image creation and building the OS both
started with 'image' as their names followed by some random
characters.  During debugging this is annoying, because on first sight
it is not clear, where which files are stored.

This patch renames them to dib_build.XXXXXXXX and dib_image.XXXXXXXX.

This patch introduces no user-visible change:
the temporary directories are only used during the
run of disk-image-builder.

Change-Id: I249cdb7750fe9a746b375b462789cd9b82681a2e
Signed-off-by: Andreas Florath <andreas@florath.net>
2016-06-03 07:21:03 +02:00
Andreas Florath
b365d3c833 Refactor: remove unused functions
There were a couple of functions which were unused:
ensure_nbd, map_nbd, unmount_qcow_image, mount_qcow_image, ensure_sudo
Because some of them use 'trap ... EXIT' this hinders introducing a
separate exit phase - therefore they are removed.

(It would also be impossible to use them in the current setup, because
they overwrite the 'trap ... EXIT' of the disk-image-creates 'main'.)

Change-Id: If932a557dca9aea4864154ad6c4f286373d6dd7c
Signed-off-by: Andreas Florath <andreas@florath.net>
2016-06-02 21:06:12 +02:00
Jenkins
38dcce0fc4 Merge "Fix variable unbound error while REPOREF="*"" 2016-06-02 05:20:37 +00:00
Jenkins
419568884c Merge "Rework yum-minimal locale cleanup" 2016-05-31 06:42:20 +00:00
Ian Wienand
f5dff9c52a Rework yum-minimal locale cleanup
It turns out our manual locale cleanup is causing issues (see
I54490b17a7f8b2f977369044fcc6bb49cc13768e).  Upon further
investigation, I think this is a better approach than manually
deleting repos.

glibc on Fedora obeys the %_install_langs macro for reducing the
installed locales (as mentioned in the comments, F24 has moved to
having different packages, but worry about that later).

So our existing clear-out is really only required for CentOS, whose
glibc does not have any way to indicate to build less locales.
However, %_install_langs is still correct there, as it restricts some
of the translation files and other things installed with the %lang
macro in spec files.

This is complicated by us having to set this at glibc-common install
time, which happens with the "yum" from outside the chroot (i.e. on
trusty).  Since this is too old to have flags to pass this, we need to
fiddle with rpmmacros.

I've tested this with fedora-minimal builds and the locales file is
about 2MiB, which is what it was after the cleanups, and the listed
locales are only those we expect (i.e. it appears to be working).

Change-Id: I528a68beeb7b2ceec25ccbec1900670501608158
2016-05-31 15:14:24 +10:00
Andreas Florath
9fbc462e00 Remove Fedora 21 from test-build
Because Fedora 21 is EOL, the appropriate cloud images were
removed from the mirrors during the last days.
Because of the removal, currently all CI tests are failing.

This patch removes the Fedora 21 CI tests.

(Adding and supporting Fedora 23 is done with another patch -
because some additional changes are needed here.)

Change-Id: Ib85bb6fafd4f56ecc55dd420048f4d9e6e6969f3
Signed-off-by: Andreas Florath <andreas@florath.net>
2016-05-30 22:54:58 +02:00
Ian Wienand
7550d25db0 Export die() function
I realised I'd been using die() in a few places assuming it was
available, but it wasn't exported.  I guess it didn't matter because
whatever was wrong, we were failing anyway :)

This exports the function to make it available to sub-processes, which
should remove the need to source it as done in several places.

Change-Id: I7b9a5a6db406e160099b6ed9fde80455ae227327
2016-05-27 09:25:22 +10:00
Jenkins
5f4cac3303 Merge "Add a best-effort sudo safety check" 2016-05-26 17:31:00 +00:00
Liu Qing
d3255835d1 Fix variable unbound error while REPOREF="*"
If REPOREF="*", HAS_REF will be used without initialization. As -u is set
the script will terminate with error.

Change-Id: Ic1d88415adfef66dfc6c1d92610a45a9eb6359f3
Signed-off-by: Liu Qing <liuqing@chinac.com>
2016-05-26 09:24:59 +08:00
Darragh Bailey
716be1d6f0 Handle file magic type varying order matching
Allow file test to return all possible mime-types and ensure anything
that matches the python file type is parsed by flake8 instead of
relying on the first match returned.

Closes-Bug: #1585688
Change-Id: Iba31f1853537fe9234ab6f83d66f13dc1c578abb
2016-05-25 17:55:59 +01:00
Jiri Stransky
1e9cf3a1c8 Remove deprecated overriding of cloud-init defaults
The "set to localhost by default" behavior for manage_etc_hosts has been
deprecated for more than a year now by change
Ia8582883f737548e2911d3f36a1943e5b236281b.

Setting that value to "localhost" is still possible, but it won't be the
default anymore. If the previous behavior of assigning the hostname and
FQDN to 127.0.0.1 is still desired in some environments, it can be
achieved by setting the DIB_CLOUD_INIT_ETC_HOSTS environment variable,
as the deprecation warning message suggested.

Change-Id: I5a19d46e2f305769a0c89c9d25d2e6be02910221
2016-05-23 17:44:32 +02:00
Jenkins
f0b57d5efd Merge "Fix apt-sources configuration for debian-minimal" 2016-05-21 20:10:18 +00:00
Ian Wienand
244f0a0001 Add cloud-initramfs-growroot for Precise
Add a pkg-map to install cloud-initramfs-growroot on Precise, as it is
required there because the kernel is too old to use growroot on a live
file-system.  This was generically removed in
e23b087505.

Change-Id: I5e6cbc4b74dc72a6f23f73a018c028f3ff7c0157
2016-05-20 14:32:41 +10:00
Ian Wienand
716b8e14fe Add release to pkg-map
Add ability to filter in pkg-map on release, which defaults to
DIB_RELEASE.  As per the examples, release is a more specific target
than distro, but distro still has to match.

Added a debug flag and ability to use a pkg-map directly for
debugging/development.

Change-Id: Ie282f96966e46236b06bc276de0168fc7a66c5da
2016-05-20 14:26:25 +10:00
Paul Belanger
be521bdec6 Export FS_TYPE and remove hardcoded ext4 values
Export FS_TYPE from img-defaults and use it to remove hard-coded
defaults in the debootstrap mounting.  Also, cleanup the suse element
as it should have access to the exported variable.

Change-Id: Ie9b671ca9336060a5ad294be48aa7eff442bf066
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2016-05-19 17:33:36 +10:00
Jenkins
95c874abb9 Merge "Add dhcp-all-interfaces.target for syncing units" 2016-05-18 19:39:10 +00:00
Ian Wienand
34efed027a Add PS4 to show file/function/line in debug output
For something fairly simple, I went back-and-forward with this a bit.

Firstly, I realise calling readlink constantly sucks.  Due to the way
we call dib and source various files, you end up with the source-file
from "caller" being usually a very ugly path including levels of "../"
indirection.  Cleaning this up to something canonical is the only sane
way to present it.

Because we evaluate _ps4() from a sub-shell in the PS4 string, there's
no way for it to do something like build a global in-memory cache in
an associative array or similar.  It could write out a temp file or
some other side-band method, but the overheads of managing this don't
seem any different to just calling readlink.  If anyone can think of a
bash-hack around this that doesn't involve a fork() I'm interested.

We could potentially strip some of the leading paths in the assumption
you know what they are; but it gets complex when things are split
across /usr/bin & /usr/lib and external elements, etc.  I thought
about arbitrarily shortening it (e.g. just take last 20 characters)
which gives you enough of an idea of the file, but looks a bit ugly.
Or we could just leave the file-name out all together and assume the
function name is unique enough; this also seemed a bit ugly.

Obviously it's a matter of taste in the output.  It is certainly
wider, but it also adds a lot of information.  It also makes it fairly
clear where there are things we can make less verbose,
e.g. I1e39822f218dc0322e2490a770f3dc867a55802c disables tracing in
run-parts which is just noise.  There's a few other frequently used
loops that we could disable tracing for by default to benefit
signal:noise.

tl;dr : take a look at the logs.  I think it is a step in the right
direction of making the logs more usable for debugging.

Change-Id: I8054a3050415fcb527baeb7012bf133e5c864bf3
2016-05-18 09:23:15 +10:00
Ben Kero
9a00bf144f Revert "Revert "Properly fail/trap in eval_run_d"" and fix PIPESTATUS
This reverts commit f07e33a2e9.

This change reverts the revert while fixing the underlying issue --
$PIPESTATUS needed to be encapsulated in ${}s

Change-Id: I1df06ffa7aecf4ea4b8e187dc756e9fc779786bc
2016-05-16 16:25:45 -07:00
Jenkins
99ddd58a15 Merge "Revert "Properly fail/trap in eval_run_d"" 2016-05-16 22:46:31 +00:00
Jenkins
2fffc9f892 Merge "Fix path issue for locale-archive.tmpl" 2016-05-16 22:01:47 +00:00
Jenkins
02822dc6b9 Merge "Add centos7 test" 2016-05-16 21:43:28 +00:00
Paul Belanger
0478fb15db
Fix path issue for locale-archive.tmpl
Change-Id: Id589c16aab46d447b3c21f00f3acfd06890e43d2
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2016-05-16 09:23:58 -04:00
Derek Higgins
f07e33a2e9 Revert "Properly fail/trap in eval_run_d"
This reverts commit 0d1d6bec7c.
This patch breaks tripleo-ci (the instack.qcow2 images is failing
to build) and was merged without passing CI.

Closes-Bug: #1582115
Change-Id: Ic4725ad0689c937fb4c8c792e1eaff5f4ea9ada9
2016-05-16 13:16:12 +01:00
Sam Betts
eb99fe7144 Add dhcp-all-interfaces.target for syncing units
Currently there is no way for a service to become aware that
dhcp-all-interfaces is finished configuring all the interfaces at
boot time. This causes problems for applications like the
ironic-python-agent which scans the interfaces when it first starts as
part of the inspection stage and can race against dhcp-all-interfaces
bringing up the interfaces, leading to inconsistent results.

This patch ensures that the dhcp-all-interfaces script runs before any
network interface is configured and brought up by the rest of the
system, and also ensures that the ironic-agent element also waits for
the network to be online before starting. This is done by using the
network targets provided by systemd.

Change-Id: Id9583b7f54361aa603a6229da598ad6a0f0f7938
2016-05-16 10:15:53 +01:00