Problems were seen when building images in parallel with
git clones and fetches interfering with each other.
Stop it.
Co-Authored-By: Michael Kerrin <michael.kerrin@hp.com>
Change-Id: Iaa48f259755d8b22327459a5c6d224a342909aee
The last patch to do this only handled the case of SHA256SUM being
stale, but the image is more likely to be stale in the default
configuration as the hash file is downloaded over HTTPS.
Change-Id: I6531fa684e560cad48c3696394d48a600680c875
Closes-Bug: #1336067
When the call to pkg-map fails, it prints the error to stdout. However,
this output is lost when pkg-map is executed in a subshell, so the
actual error is never seen. This change adds an explicit echo so the
error is shown.
Change-Id: Id6f4fae6acadfe2839b408fb2dd11fb65d65df6e
Optimized the creation of directories by using the Brace expansion mechanism.
And corrected the dirname resolution to follow correct convention.
Change-Id: I048c04385d2706c4e38f8548555a59b0437e1fcd
This reverts commit 32eda4b92f.
I noticed today that environment.d files no longer seem to get sourced
by DIB. Reverting this commit fixes it.
Change-Id: I08a8feab541901c8fd0a803628f5aeb6c0eec598
Closes-bug: #1339116
Some users on corp networks are finding that cloud images appear in
their proxies before SHA256SUMS files.
We now try to force any to upstream proxies to update their copy and
retry the checksum, rather than immediately failing.
We also now add proper argument parsing to cache-url, and associated
help functionality, since we have now grown our first argument.
Change-Id: I44d44b62db8d1df990606fdf087ec1b837f491f6
Closes-Bug: #1336067
There are some situations where the list of packages to install
in a image is blank (packages blacklisted) on opensuse based
distributions.
Change-Id: Ib720473fa63671ed974d41a86fe10a9b10720f5d
The current glob match for environment files can source editor backup
files (foo.bash~) which will override the real changes you have made.
Other parts matche use the regex to avoid matching such files, so do
the same for environment file matching. Note this has to match "."
unlike the other regex, as most env files are "foo.bash"
Change-Id: I934486b3ff5884063d29c6d9b66fd9b11140464c
Rename rhel 10-yum-blacklist to 10-rhel-blacklist to
avoid future name conflict with a yum element blacklist.
Change-Id: Ic2136a06e1ec8f19908ada978782733d5fa7cacc
Force LC_ALL to C to allow printf to work independently
from the locale. Gives invalid number error if run in non-english
locale.
Change-Id: Ia68853505485fefbf0890313456e7edb6097666b
Closes-Bug: #1335932
This patch updates the openSuse element to work with the new
base image file format. It should be noted that the base image
file name has changed several times while updating this element,
and and may need to be set manually via the "BASE_IMAGE_NAME"
env var.
Change-Id: I4dac8bf9a4bf76a00d4a04cbf063fd245b11f3d6
The $DISTRO_NAME var is now set in all of the OS elements
in environment.d. This patch removes the call to
lsb_release and instead uses the $DISTRO_NAME variable.
Change-Id: I7088eb88b6a3611fef5a21a6b62975876549465f
The current "redhat" os family omits RHEL7. The rhel7 element has the
corresponding $DISTRO_NAME set in environment.d. This patch adds rhel7
so a pkg-map json file using "redhat" for the os family will work the
same as rhel, centos and fedora.
Change-Id: If6b4cbba618167669a499e6121f234942ca9367a
Several elements were using almost similar code to
get the current kernel and initrd inside an image.
After factoring out this code into a common element,
modify the elements to make use of it.
Change-Id: Ib547cd96b2b354b1c97342cacf5f0d0a3f70a3da
It may be desirable to specify file://tmp/pypi as one of the list
of mirrors specified by PYPI_MIRROR_URL, so the local pypi mirror
should be mounted as long as the directory exists.
Even though the path is mounted, it is only used by pip if there is
no specified PYPI_MIRROR_URL, or if one PYPI_MIRROR_URL entry
specifies file://tmp/pypi.
The pypi element previously assumed that a local mirror would be
a complete offline mirror. This change allows for a mixture of
local mirror plus remote mirrors. This means that the local
mirror could (for example) contain a few locally built pip packages
from arbitrary git checkouts for gate testing of those projects.
In this scenario, most packages would be downloaded from the
configured remote mirrors.
This change is required for synchronous gating on the os-*-config
tools.
Change-Id: I0f6d153c06993ffd0d90bd8ca1717462e8b6b541
Encapsulate the new select-boot-kernel-initrd common function
into an element so other elements can make use of it.
Change-Id: Ie587bd5dae1afebfd5646190dfa951a7949938cd
Debian based distros do not enable the serial console by default
in grub.cfg, instead sending all boot messages to the display
Change-Id: I0e80859198e77862ab0301ab27ebe11702fba09d
Ensure cache mount directory exists before its used. The zypper element
would error when attempting to mount the cache directory because the
mount point did not exist. This patch corrects that.
Change-Id: Iaa3c2a0254b12bd847643a61a99f5a234097fd21
Scripts in pre-install.d run as root so there is no need to sudo commands.
This patch also helps avoid an error when sudo is not installed in the
default image.
Change-Id: Iab20edfb161189f8c4e0cfa7d6946f63163d6413
The DIB_CLOUD_IMAGES variable in ubuntu's root.d is already used
everywhere with a /, so we can safely drop it from the default
value.
Change-Id: I39dbd3f4004052fcccb4131dc838759f4c82312a
With this change package name to distro mappings used
within the base element are now stored in the top
level pkg-map file.
Additionally, all bin/install-packages commands are updated
to use the -m option so that the base element pkg-map
namespace is used by default.
Change-Id: I3bdbf987c8525f9ef67ca0c263651d6613f29ef0
Updates the opensuse element's bin/install-packages script
so that if supports using pkg-map instead of map-packages
for package mapping. To make use of the new change
simply add the -m <element name> to allow install-packages
to know which element namespace to use when installing packages.
Use of the new -m option is off by default.
As part of this change we also updated install-packages
to use getopt for in script argument parsing.
Change-Id: I56da8adf6c445c50c7642e221c6c11ba59330404
Updates the dpkg element's bin/install-packages script
so that if supports using pkg-map for package mapping.
To make use of the new change simply add the -m <element name>
to allow install-packages to know which element namespace to use
when installing packages.
Use of the new -m option is off by default.
As part of this change we also updated install-packages
to use getopt for in script argument parsing.
Change-Id: Idfc40f2d75828a0f09d227f0332ccef8f0183efc
export $SYSTEMD_SYSTEM_UNIT_DIR to know where .service files should be
moved.
So far those files are moved directly in /usr/lib/systemd/system whereas
Debian use /lib/systemd/system.
We load the value from the systemd pkg-config file.
The patch don't use pkg-config directly for three reasons:
- environment.d files are run before install.d and the other similar
targets, making hard to install an additional dependency
- the .pc file can easily be parsed using awk
- pkg-config package name is not standard across the different
supported distributions
$SYSTEMD_SYSTEM_UNIT_DIR default value is /lib because, as pointed by James
Slagle, /lib is actually a symlink to /usr/lib/ on Fedora and will work
for Debian and Fedora.
See: https://bugs.debian.org/719634
Change-Id: I413a92284e6a79e7fcdde99c5138bc3ce8e85e80
The 'None' Datasource does not exist in precise 12.04 (which uses
cloud-init 0.6.3), so conditionally ensure that we are not including
this in the datasource_list in case we're building a precise image.
Partial-bug: 1329068
Change-Id: I6efb91e17bf896bbf7d67c31c4eb5098d5d61a71
We use an element remove-serial-console in image elements.
Renaming this element to match. Also renaming allows us to use
the serial console in devtest runs.
See - https://review.openstack.org/#/c/86272/
Change-Id: I8f1138f58d0046ccdeefa3decc45c36d95c6bf4b
diskimage-builder/elements/base/cleanup.d/99-tidy-logs added
to tidy up (remove) initial logs files on disk images.
Makes for slightly smaller images, and easier times tracing post
boot issues.
Change-Id: Iee98b5d818e080bcd9635482bc29349a71938087
source-directory ifupdown directive has been introduced in version
0.7.44. Wheezy comes with version 0.7.8.
Without this change, the eth0 interface will never get an IP address on
Wheezy.
Change-Id: If40ed3b82ca78280190b66c06a0a77bb71270d7d
The script to find best kernel from image (used by vm
and baremetal element) is duplicated and is not in sync
with each other. Moving the code to img-functions as a
function will reduce duplicated code and make it reusable
in future.
Since img-functions is not accessible in chroot env,
kernel selection is being moved from finalise.d to
cleanup.d in the vm element.
Change-Id: I8fbccc13a2c61a5191ef9ea5d2a8302a3e43b000
Updates the 51-install-tgt and 52-install-busybox files
in both the deploy-ironic and deploy-baremetal elements
so that they have unique names.
While an end user would never use these elements at the same time we
should always use unique names for things per DIB conventions.
Change-Id: I8926870e45849341c10b2d479ddee45f9b625e2b
While an end user would never use these elements at the same time we
should always use unique names for things per DIB conventions.
Change-Id: I53b10522183e8566c62922b91878e5cf767fe2c6
Both the rpm-distro and redhat-common elements had the
same exact 00-usr-local-bin-secure-path. This commit
deletes the one in redhat-common and leaves the one
in rpm-distro (which is also required by the same
elements).
Change-Id: Ie9ef29d268b3737a6741fce016ef335db6651b59
While an end user would never use both redhat-common and ubuntu
elements at the same time we should always use unique names
for things per DIB conventions.
Change-Id: I2cabc583a1fd41d4fe7fbf8cf93127d49e26aff4
Updates the yum element's bin/install-packages script
so that if supports using pkg-map instead of map-packages
for package mapping. To make use of the new change
simply add the -m <element name> to allow install-packages
to know which element namespace to use when installing packages.
Use of the new -m option is off by default.
As part of this change we also updated install-packages
to use getopt for in script argument parsing.
Change-Id: I19240e74b69859f7faf9cd80440633234a677d63
With this patch, /var/cache/apt/archives directory content is preserved.
The directory is actually a bind mount of the
~/.cache/image-create/apt/$DISTRO_NAME directory, much like what we do
for ccache.
You can use DIB_APT_LOCAL_CACHE=0 to disable this behavior.
This trivial change improve performance A LOT (>30%), even if a local
HTTP proxy because:
- there is no need to copy again and again the same files
- we avoid network latency
The patch has been tested with Debian and Ubuntu with every elements
from the tripleo-image-elements repository, the final size of the cache
directory is about 700MB per distribution subdirectory.
Change-Id: I4fab499493f734c7c546d4d23b1a98f0e7523a39
OVS service is called:
- openvswitch-switch on Debian/Ubuntu
- openvswitch on RHEL/Fedora
This changes is needed to get the neutron element to work
on Debian/Ubuntu with systemd:
I6e3df30dc3a6918f3a949a7dac47289ede5c3d1f
Change-Id: I2748a9db7f8ee456698187a9646bc7cda2256fe7
This change adjusts the "inline" python function used to profile
run-parts duration to make it run successfully on both py2 and py3
Change-Id: Ib5287a56b8e2b3e882dca489192f01d9f01cd8ba
Now that issues associated with this bump have been addressed
elsewhere, reinstate trusty as the default DIB_RELEASE for
the ubuntu element. Also adds a note about memory requirements
to the ramdisk element.
Change-Id: Ia89c83ba0ee4ad3bd89b833ba0db08135eac63ed
Default to only having cloud-init query Ec2 on first boot for Ubuntu,
until cloud-init has been SRU'd to fix the CloudSigma data source issue
that causes Trusty boots to hang.
Change-Id: Icb3734d5ae78f4a0a6c0fae1af4a2ce3c809308c
Partial-bug: #1316475
This patch removes a couple of lines of redundant code. In Python,
the get method on a dictionary defaults the return value to None,
so the method never raises a key error.
https://docs.python.org/2/library/stdtypes.html#typesmapping
Change-Id: I180877f9be23d373cd63771bb3682d6587ec6fac
There is a substitution of suffix in rhel element, like
rhel-guest-image-6.5-20140121.0.x86_64.qcow2 to
rhel-guest-image-6.5-20140121.0.x86_64.raw
The convert is correct but the substitution of the suffix is wrong.
${PARAMETER#PATTERN} will match the string from the beginning and generate rhel-guest-image-6.5-20140121.0.x86_64.qcow2.raw
We should use ${PARAMETER%PATTERN} instead, which is from the end.
Change-Id: Ic45a3f800058ea9e5d746ff543c068a54632407d
This moves cloud-init data source configuration to a general purpose
cloud-init-datasources element that can be used to explicitly configure
the list of cloud-init sources that will be queried on first boot.
cloud-init-nocloud now depends on this new element to configure the
datasource_list while continuing to prep the image for a nocloud first boot.
Change-Id: Ibcc3b86d6ca567a23f89b7a1a36bc713e444ef68
Closes-bug: #1316475
Updates the various operating system elements so that we
set the DISTRO_NAME variable for each distro.
This env var is used by bin/pkg-map to set the default
distribution name for package name mappings.
Change-Id: Ib4b05eb7191dd50d25799a0bac51fd15c01b74cb
Adds a new pkg-map element which can be used to install
packages based on an in element 'pkg-map' JSON config
file format.
As part of this change we expose DIB's IMAGE_ELEMENT variable
so that we can have access to the element names which are being
installed in a clean manner.
This change is intended to decouple elements from DIB
and allow new elements to support multiple distributions
(with various package naming schemes) without having to
constantly maintain DIB's various bin/map-packages files.
Change-Id: Ic0a061995e2ae708c95a535cba6237bff58f7d93
A few scripts were added without set -o pipefail since the original
change to add that everywhere. This will get the dib-lint check
passing again.
Change-Id: I96bef45cc10ff9bbcf2c4f1b796b8cd188e10485
Add the ability to supply a gerrit ref in DIB_REPOREF_* and
have that reference fetched from the remote and checked out for
building within the image.
Closes-Bug: #1314021
Change-Id: I5e5742c4cbff98e14121c50dde5e512f192b5415
Co-Authored-By: Robert Collins <rbtcollins@hp.com>
Using -n for bash arrays was a little bit bad, and also resulted in
errors when running under set -u. Wean the yum element off the array
and switch to a straight string.
Change-Id: Ifd61fce7dadd5820022e9b2ab42020ba0cb71ce6
When using custom-built kernels, their filenames may not necessarily
follow the standard naming conventions for the stock systems in question.
This patch permits the specification of an alternative pattern to use
when looking for the correct kernel to construct the PXE images from.
Example:
export DIB_BAREMETAL_KERNEL_PATTERN="kernel*jang"
export DIB_BAREMETAL_INITRD_PATTERN="initrd*jang"
Change-Id: I86e5254051addd5e89dd4e363e9bba117e65353d
Add support for supplying a custom keyring and debootstrap script
using two new environment variables:
* DIB_DEBIAN_KEYRING - Path to a gpg keyring.
* DIB_DEBIAN_DEBOOTSTRAP_SCRIPT - Path to a alternative debootstrap
script.
Change-Id: I42e8832b435b8a77b0681b5bc9d9332bbe860cd5
Although set -u allows empty WHITELIST=() list to be set, you can't test
-n "$WHITELIST". Looks like the intention of the commit that caused
this issue (Change-ID Iad2d490c86dceab148ea9ab08f457c49a5d5352e) is to
set a policy to prevent future use of bugs due to unbound variables.
Thanks StevenK for pairing with me and helping me hone my bash skills.
Change-Id: I2349c990ba53b0aaecd5119812582c7c848f1e4e
Closes-Bug: #1319987
If the kernel is restricted read (as a security measure) we may not be
able to read it without chmodding it.
Change-Id: Ib577705c27d8c351288334de03a7848cf49f2de9
If 'DIB_RHSM_USER' isn't set, the image build with rhel
element fails because of check 'set -u'. So, we should export
'DIB_RHSM_USER' in script to avoid error during build
Change-Id: Ic63a20e3d91a25bbe804440a5919b035489c60a3
Closes-bug: #1318717
When the cache tarball gets fail the output doesn't indicate
what url is failing. Adding '-x' displays the urls attempted.
Change-Id: I71b2dbfd712e72c591410ca9b68edc76d493ad6e
As with the previous similar changes, this is intended to catch
problems as they happen instead of ignoring them and continuing on
to potentially fail later. Setting this on all existing scripts
will allow us to enforce use via Jenkins.
Change-Id: Iad2d490c86dceab148ea9ab08f457c49a5d5352e
Updates the fedora element so that DIB_DISTRIBUTION_MIRROR can
be used to configure a custom Yum mirror on Fedora. This
implementation replaces the baseurl in each of the default
Fedora Yum .repo files so that a custom mirror URL/path
can be supplied. The URL should point to the directory
where the updates and release directories exist. Example:
http://download.fedoraproject.org/pub/fedora/linux
As part of this change we also remove any existing
metalink settings in the .repo files to cover the case
where an end user runs their own createrepo command
to create the mirror... which means your metadata won't match that
of the normal Fedora mirrors and your baseurl won't get used.
Change-Id: I1a49a2941b1cc489e91a72b87430393cd32e865d
DIB's map-services is currently used in TIE's os-svc-config
binaries to support service name lookups. Many of the systemd
specific functions require bin/map-services to exist.
This change adds bin/map-services files that are noops
to both the debian and ubuntu elements so that as those
distros move towards using systemd they can make use
of the functions we already have in place.
Change-Id: If8d61f3858b11de86c3292e840d033e5e3cecedb
If scripts are not set -e then errors can be ignored, causing more
confusing failures later.
Also adds an exclusion comment to the ramdisk init script since we
don't want that to exit on failure.
Change-Id: Idf43993bd10b1ef16c1d3b0d9df8d0ad94c46458
Adds a new element, package-installs, that provides an interface for
declarative package installs and uninstalls. Packages to install can be
added to an install.d/package-installs-<element-name> file. The set of
packages listed across such files are installed in a single transaction
at the beginning of install.d.
Prefacing the package name with a "-" indicates that the package should
be uninstalled at the end of the install.d phase. Again, the full set of
uninstalls are done in a single transaction.
An element providing a package-installs file should add package-installs
to its element-deps file.
Change-Id: I5b540388eff1079c8dee933b869463371481152b
This commit fixes the if/else block around Debian stable/unstable which
was badly merged during the last rebase.
Change-Id: Ia2eb806da3d7c8d28afffb026f35600c7125d8d7
This makes it easier to work with temporary files - less traps
scattered around higher layer code.
Change-Id: I2fdd93115a7b0d289c2e28f8c133d4059de75b87
This patch adds a debian-systemd element to get systemd support with
Debian.
A backport of systemd is used on Debian Wheezy.
Change-Id: I095c81b2c32844fccf940c97b79054923c25b8c2
Upstart cannot be installed by debootstrap on Wheezy because of a
conflict with sysvinit. sysvinit is flagged a being “essential” and
apt-get will refuse to remove it.
See: https://bugs.debian.org/668001
This patch uses another strategy. We install a standard sysvinit
Debian chroot with debootstrap and during a second step, we
replace sysvinit with the wanted init system.
Change-Id: I081c81b2c32874fccf940c97b79054923c25b8c2
Cloud images expect the primary interface to run dhclient in order
to come up and acquire an IP address. Although we have technology
to manage without this for the undercloud and so on, the seed
still relies on this machinery in order to bring up its first
interface.
Change-Id: I079a679779de760ee9ca6f93cbb22e69a65e03af
Using set -e in all of our scripts will prevent some subtle bugs
from slipping in, and will allow us to enforce use of set -e with
tooling.
This change also adds -u and set -o pipefail in the less complex
scripts where it is unlikely to cause problems. A follow-up change
will enable those options in the complex scripts so that if it
breaks something it can be reverted easily.
Change-Id: I0ad358ccb98da7277a0ee2e9ce8fda98438675eb
In the mailing list discussion, everyone seemed to agree that we
should just be explicit that these are bash scripts since that's
where we're testing them and there's no good way to ensure they
work on any other shells right now.
Change-Id: I86c41d2cddf6e91e7deeb4c2ab51635c80a1ce0c
The phrase is no longer needed as of August 23, 2000 with Nicaragua's
joining of the Berne Convention.
Additionally, in at least one instance,
elements/cache-url/bin/cache-url, its existence in the file between
Copyright lines is just weird and feels misleading, even though it is
not.
Remove all of the lines, because sanity.
Change-Id: I24fd76c2b4f66b8036010b5079db39ead729abee
