Commit Graph

2503 Commits

Author SHA1 Message Date
Dan Prince
5254aedfe5 Yum: support pkg-map in bin/install-packages
Updates the yum element's bin/install-packages script
so that if supports using pkg-map instead of map-packages
for package mapping. To make use of the new change
simply add the -m <element name> to allow install-packages
to know which element namespace to use when installing packages.

Use of the new -m option is off by default.

As part of this change we also updated install-packages
to use getopt for in script argument parsing.

Change-Id: I19240e74b69859f7faf9cd80440633234a677d63
2014-06-11 08:09:27 -04:00
Derek Higgins
e7d105a751 Add tar as an output type
Add support for d-i-b to create tarballs, the primary consumer for
tarball images would be linux containers.

Change-Id: I27d67401f3e4415226a4a51e1dde46f739c0220a
2014-06-11 10:48:50 +01:00
Robert Collins
d5e0c2d778 Correct source-repository comments
git revisions are not references.

Change-Id: If450e70e46cd30a2662b7246a2555c118ed33ad2
2014-06-11 19:32:27 +12:00
Jenkins
6ff1681915 Merge "Reinstate Trusty as default for Ubuntu" 2014-06-11 04:28:56 +00:00
Jenkins
028bb34a31 Merge "Check return code of element-info" 2014-06-11 04:22:35 +00:00
Jenkins
336398d57a Merge "Only use Ec2 cloud-init data source for Ubuntu" 2014-06-10 23:06:27 +00:00
Gonéri Le Bouder
eca59b2e97 dpkg: local cache for .deb files
With this patch, /var/cache/apt/archives directory content is preserved.
The directory is actually a bind mount of the
~/.cache/image-create/apt/$DISTRO_NAME directory, much like what we do
for ccache.
You can use DIB_APT_LOCAL_CACHE=0 to disable this behavior.

This trivial change improve performance A LOT (>30%), even if a local
HTTP proxy because:
 - there is no need to copy again and again the same files
 - we avoid network latency

The patch has been tested with Debian and Ubuntu with every elements
from the tripleo-image-elements repository, the final size of the cache
directory is about 700MB per distribution subdirectory.

Change-Id: I4fab499493f734c7c546d4d23b1a98f0e7523a39
2014-06-10 17:20:37 +02:00
Jenkins
343f70cd23 Merge "Add new cloud-init-datasources element" 2014-06-10 06:43:47 +00:00
Gonéri Le Bouder
1de9612dad map-services: add openvswitch
OVS service is called:
 - openvswitch-switch on Debian/Ubuntu
 - openvswitch on RHEL/Fedora

This changes is needed to get the neutron element to work
on Debian/Ubuntu with systemd:
  I6e3df30dc3a6918f3a949a7dac47289ede5c3d1f

Change-Id: I2748a9db7f8ee456698187a9646bc7cda2256fe7
2014-06-09 09:18:02 +00:00
Giulio Fidente
3262e76248 Ensure dib-run-parts profiling works with py3 and py2
This change adjusts the "inline" python function used to profile
run-parts duration to make it run successfully on both py2 and py3

Change-Id: Ib5287a56b8e2b3e882dca489192f01d9f01cd8ba
2014-06-06 18:37:39 +02:00
iberezovskiy
b740c50c76 Rename old image file instead of rewrite it
If we have the old image file with same name as new image file
we shouldn't rewrite it, just need to rename old.
Uses date number in old image name.

Fixes TODO in lib/common-functions

Change-Id: Ica58a0f4435217912bddf950322299d2ed2889f3
2014-06-06 12:59:28 +04:00
Tomas Hancock
6b8a0408d8 Add missing dollar
Check is missing a dollar leading to an error on deploy ramdisk
consoles when booting .

Change-Id: Id61af0f7abbc353e0750580b80e448820cc17b10
2014-06-04 14:17:22 +01:00
Om Kumar
538b978701 Replacing deploy to deploy-baremetal in README.md
Now that deploy has been renamed to deploy-baremetal, README.md needs
to be updated to refer to the new elements.

Change-Id: I7c0d5bfe158628ea1faf2c2914ee7f532d172a28
2014-06-04 03:48:24 +00:00
Adam Gandelman
40914bd121 Reinstate Trusty as default for Ubuntu
Now that issues associated with this bump have been addressed
elsewhere, reinstate trusty as the default DIB_RELEASE for
the ubuntu element.  Also adds a note about memory requirements
to the ramdisk element.

Change-Id: Ia89c83ba0ee4ad3bd89b833ba0db08135eac63ed
2014-06-03 16:28:25 -07:00
Adam Gandelman
f61c1acf81 Only use Ec2 cloud-init data source for Ubuntu
Default to only having cloud-init query Ec2 on first boot for Ubuntu,
until cloud-init has been SRU'd to fix the CloudSigma data source issue
that causes Trusty boots to hang.

Change-Id: Icb3734d5ae78f4a0a6c0fae1af4a2ce3c809308c
Partial-bug: #1316475
2014-06-03 16:27:41 -07:00
Ryan Brady
dcba738541 Remove uneeded code from pkg-map
This patch removes a couple of lines of redundant code.  In Python,
the get method on a dictionary defaults the return value to None,
so the method never raises a key error.

https://docs.python.org/2/library/stdtypes.html#typesmapping

Change-Id: I180877f9be23d373cd63771bb3682d6587ec6fac
2014-06-03 11:37:15 -04:00
tanlin
ebd54b1010 Correct the wrong rename in rhel element
There is a substitution of suffix in rhel element, like
rhel-guest-image-6.5-20140121.0.x86_64.qcow2 to
rhel-guest-image-6.5-20140121.0.x86_64.raw

The convert is correct but the substitution of the suffix is wrong.

${PARAMETER#PATTERN} will match the string from the beginning and generate rhel-guest-image-6.5-20140121.0.x86_64.qcow2.raw

We should use ${PARAMETER%PATTERN} instead, which is from the end.

Change-Id: Ic45a3f800058ea9e5d746ff543c068a54632407d
2014-06-03 06:42:35 +00:00
James Slagle
6fd78ffcfd Add manifests to .gitignore
A common use case is to run diskimage-builder directly from the git
checkout.  This leaves manifests and .d directories behind for each
image build in the current directory, so add these to .gitignore.

Change-Id: Id4b432d30dead830b0886bccb762c4e3922b52e1
2014-06-02 08:24:30 -04:00
James Slagle
813b4bc242 Check return code of element-info
Ensure the return code of element-info is actually checked by moving the
export IMAGE_ELEMENT to the subsequent line.

A recent change (Ic0a061995e2ae708c95a535cba6237bff58f7d93) exported the
IMAGE_ELEMENT variable. This caused the return code of element-info to not
actually be checked, since the export command is now checked instead and
almost always will return 0.

This caused a misleading error message when using diskimage-builder with an
unknown element since the script didn't exit when it should have:
$ bin/disk-image-create -a amd64 foobar vm
Building elements: base  foobar vm
ERROR: Element 'foobar' not found in 'bin/../elements'
Expanded element dependencies to:
Building in /var/tmp/image.DuIDY1lW
Please include at least one distribution root element.

Change-Id: I3d9c02bf0b1a4f320280185dd3fab0697d2c92f2
2014-05-30 15:41:05 -04:00
Adam Gandelman
f645287ec4 Add new cloud-init-datasources element
This moves cloud-init data source configuration to a general purpose
cloud-init-datasources element that can be used to explicitly configure
the list of cloud-init sources that will be queried on first boot.

cloud-init-nocloud now depends on this new element to configure the
datasource_list while continuing to prep the image for a nocloud first boot.

Change-Id: Ibcc3b86d6ca567a23f89b7a1a36bc713e444ef68
Closes-bug: #1316475
2014-05-29 21:54:59 -07:00
Dan Prince
50cb019a25 Set DISTRO_NAME in OS environment.d
Updates the various operating system elements so that we
set the DISTRO_NAME variable for each distro.

This env var is used by bin/pkg-map to set the default
distribution name for package name mappings.

Change-Id: Ib4b05eb7191dd50d25799a0bac51fd15c01b74cb
2014-05-28 12:22:02 -04:00
Dan Prince
c55539b142 Add pkg-map element.
Adds a new pkg-map element which can be used to install
packages based on an in element 'pkg-map' JSON config
file format.

As part of this change we expose DIB's IMAGE_ELEMENT variable
so that we can have access to the element names which are being
installed in a clean manner.

This change is intended to decouple elements from DIB
and allow new elements to support multiple distributions
(with various package naming schemes) without having to
constantly maintain DIB's various bin/map-packages files.

Change-Id: Ic0a061995e2ae708c95a535cba6237bff58f7d93
2014-05-28 12:21:44 -04:00
Ben Nemec
e824b43cbd Factor out error behavior in dib-lint
Currently when dib-lint finds a problem it does something like:

 echo "ERROR: Problem found"
 rc=1

This is repetitive and error-prone since it's easy to forget to set
rc to actually fail the check.  This change makes those two steps
a single function call.

Change-Id: I40b5bf39348a69add1f955c49f310e3bda21be0e
2014-05-28 11:04:27 -05:00
Jenkins
43827916d5 Merge "dib-lint: ensure file finish with a new line" 2014-05-28 00:41:03 +00:00
Jenkins
ab8a0f4bdd Merge "add some missing \n at end of file" 2014-05-28 00:40:57 +00:00
Jenkins
b759cfed28 Merge "Check for set -o pipefail" 2014-05-27 18:55:03 +00:00
Jenkins
e71e8efb51 Merge "Set -o pipefail new scripts" 2014-05-27 18:54:57 +00:00
Juerg Haefliger
562546ecb6 Fix package removal
Shift first to drop the first commandline argument (-e).

Change-Id: I73a0100484b6443ad5bdf500dd032af20eac7d3b
2014-05-27 07:50:42 +02:00
Jon-Paul Sullivan
ec234d356e Create an element to allow using a dpkg manifest
Enable reusing of a dpkg manifest to pin package versions at build
time

Change-Id: I23e83268d282c972b335a8cf0dfc45030a0ab4ff
2014-05-26 11:23:48 +01:00
Ben Nemec
2830ef4854 Check for set -o pipefail
Like the other set lint checks, this will hopefully catch errors
in element scripts sooner.

Change-Id: Ib1600938f6ffed657ff2950ef54bbcbdb5a0db08
2014-05-23 15:57:27 -05:00
Ben Nemec
79464cf00f Set -o pipefail new scripts
A few scripts were added without set -o pipefail since the original
change to add that everywhere.  This will get the dib-lint check
passing again.

Change-Id: I96bef45cc10ff9bbcf2c4f1b796b8cd188e10485
2014-05-23 15:57:27 -05:00
Jon-Paul Sullivan
76c59bfbde Add support for source-repos gerrit refs
Add the ability to supply a gerrit ref in DIB_REPOREF_* and
have that reference fetched from the remote and checked out for
building within the image.

Closes-Bug: #1314021
Change-Id: I5e5742c4cbff98e14121c50dde5e512f192b5415
Co-Authored-By: Robert Collins <rbtcollins@hp.com>
2014-05-23 16:00:22 +01:00
Jenkins
ce85042b9c Merge "Stop using bash arrays for whitelisting in yum" 2014-05-22 23:42:36 +00:00
Jenkins
0f6f533993 Merge "Export unbound variable DIB_RHSM_USER" 2014-05-22 17:02:19 +00:00
Jenkins
8737df75cf Merge "Remove the temporary deploy element after rename" 2014-05-22 10:48:50 +00:00
Steve Kowalik
e02dbd717b Stop using bash arrays for whitelisting in yum
Using -n for bash arrays was a little bit bad, and also resulted in
errors when running under set -u. Wean the yum element off the array
and switch to a straight string.

Change-Id: Ifd61fce7dadd5820022e9b2ab42020ba0cb71ce6
2014-05-22 19:05:28 +10:00
Jenkins
2afe36eca8 Merge "Add switch to turn on caching for debian element." 2014-05-22 09:05:17 +00:00
Jenkins
273d364fc4 Merge "Ensure scripts are set -u" 2014-05-22 03:03:28 +00:00
Jenkins
9941326be6 Merge "set -u doesn't permit bash arrays to test -n" 2014-05-21 20:21:11 +00:00
Jan Grant
b7aa04ed9d Tidy up SuSE kernel selection
This is just removing a C&P line that only ever returns a single result.

Change-Id: Iad37e982412a9199d86d9b4481cc14a85eef704e
2014-05-21 10:08:24 +01:00
Jan Grant
6df7039ba1 Parameterise PXE kernel and initrd selection
When using custom-built kernels, their filenames may not necessarily
follow the standard naming conventions for the stock systems in question.

This patch permits the specification of an alternative pattern to use
when looking for the correct kernel to construct the PXE images from.

Example:
  export DIB_BAREMETAL_KERNEL_PATTERN="kernel*jang"
  export DIB_BAREMETAL_INITRD_PATTERN="initrd*jang"

Change-Id: I86e5254051addd5e89dd4e363e9bba117e65353d
2014-05-21 10:08:23 +01:00
Jenkins
44922b8247 Merge "Ensure we can read the kernel out." 2014-05-20 03:18:42 +00:00
Jenkins
f70a9fa4e2 Merge "Add map-services for debian distros." 2014-05-19 21:23:00 +00:00
Jenkins
d69c0fce86 Merge "Fedora: Add support for configuring Yum mirrors" 2014-05-19 15:52:13 +00:00
Kiall Mac Innes
1757214196 Debian: Support additional debootstrap arguments
Add support for supplying a custom keyring and debootstrap script
using two new environment variables:

* DIB_DEBIAN_KEYRING - Path to a gpg keyring.
* DIB_DEBIAN_DEBOOTSTRAP_SCRIPT - Path to a alternative debootstrap
  script.

Change-Id: I42e8832b435b8a77b0681b5bc9d9332bbe860cd5
2014-05-18 19:40:38 +00:00
Dave Thomas
1a70916bca set -u doesn't permit bash arrays to test -n
Although set -u allows empty WHITELIST=() list to be set, you can't test
-n "$WHITELIST".  Looks like the intention of the commit that caused
this issue (Change-ID Iad2d490c86dceab148ea9ab08f457c49a5d5352e) is to
set a policy to prevent future use of bugs due to unbound variables.

Thanks StevenK for pairing with me and helping me hone my bash skills.

Change-Id: I2349c990ba53b0aaecd5119812582c7c848f1e4e
Closes-Bug: #1319987
2014-05-16 13:16:06 -07:00
Jenkins
5b16eb0c33 Merge "indent: search for !=4 spaces indentation" 2014-05-14 08:33:22 +00:00
Jenkins
a719b9fa5f Merge "4 spaces indent" 2014-05-14 08:26:08 +00:00
Robert Collins
072fee1a74 Ensure we can read the kernel out.
If the kernel is restricted read (as a security measure) we may not be
able to read it without chmodding it.

Change-Id: Ib577705c27d8c351288334de03a7848cf49f2de9
2014-05-14 04:25:05 +12:00
iberezovskiy
8995d6402a Export unbound variable DIB_RHSM_USER
If 'DIB_RHSM_USER' isn't set, the image build with rhel
element fails because of check 'set -u'. So, we should export
'DIB_RHSM_USER' in script to avoid error during build

Change-Id: Ic63a20e3d91a25bbe804440a5919b035489c60a3
Closes-bug: #1318717
2014-05-13 12:03:27 +04:00