While we already clean a number of things off the ironic-agent
ramdisk, there are a few more significant ones that we should add
to the list.
First is the kernel source. If you're rebuilding your kernel on
the agent ramdisk after the initial image build, then you need to
re-examine your life choices. ;-)
Second is /var/cache. On yum-based distros, this contains a large
number of yum cache files that take up significant space. We don't
really want to be copying around caches when booting a ramdisk
anyway, so cleaning this is the right thing to do regardless.
Third is all *.pyc or *.pyo files. There are a lot of these, so
they eat up significant space and bloat the number of files in the
ramdisk, which makes it take longer to build. the only purpose for
the files is to slightly speed up Python app startup, and we
probably lose more time transferring the files over the network
than we would gain in quicker start times. Note that we were
already trying to remove these, but for some reason I was still
seeing them show up in my final images. It makes more sense to
put them in the same pruning command as all the others anyway.
Fourth is /usr/include. These are files only needed for
compilation. See above for my thoughts on compiling in a ramdisk.
These changes have reduced the agent ramdisk from 391 MB to 333 MB
in my local centos 7 builds, and have reduced the number of files
in the ramdisk by over 18000.
Change-Id: I550f9904b9afd12d48da9ba24559acb23133d076
Fedora 24 has split locales into separate packages. Testing revealed
what is possibly a bug in the choosing of default packages, so add a
small work-around to ensure the minimal locale pack is installed.
This appears to be the only change required for fedora-minimal with
Fedora 24; at least to build with the project-config infra elements.
Change-Id: I64438c34c572ed96211384ae1bfb45b2949e4318
This does not need to be the last finalise step, and some late finalise
steps can disable the network (for example, Octavia amphora DIB cleans
resolv.conf at 99) Moving it to 60 also aligns it with rhsm-unregister
rhel6 element, and still allows to run subscription-manager steps
before.
Also fix an unbound variable error that appeared when both
BASE_IMAGE_FILE and DIB_CLOUD_IMAGES are unset.
Change-Id: Icb0e20b01479fea345e01309fc4bf3f7f639900c
If we're installing pip/virtualenv from source, we need to make sure
we pre-install the packaged versions before the upstream
versions. Otherwise, CI jobs later on that depend on packaged versions
of pip/virtualenv can bring them in and overwrite the upstream
versions we have installed, which leads to a heck of a mess and
usually very confusing failures.
I have also moved in a small hack from system-config:install_puppet.sh
that we found was necessary when using pip versions from upstream.
Note this is not as much of an issue on Debian/Ubuntu, as they keep
their pip packages in a separate place to the system packages, so you
don't have these overwite conflicts as much.
Change-Id: Ib40708c07b939b84661c44df88a5a308fd0c7216
The initramfs file created by the ironic-agent element is
owned by the user running disk-image-create; ensure that
the other files created by the element are also owned by
the user.
Change-Id: I829db5b8e8bf1fc68face9cd2bda52d2a5ccdd4f
Closes-Bug: 1593010
The latest Fedora/Ubuntu images don't ship python2 by default, so we
need to use our dib-python wrapper for this so we work in python3 only
environments.
Closes-bug: 1577105
Change-Id: I1048ceef35f269960216066924986eec6117ca00
This element takes up a *huge* part of the logs of openstack-infra
builds as we go and cache every git tree.
This silences most of the noise which will reduce the mess
in the logs considerably.
Note that we've had logging on this turned down since
I91c5e55814ba9840769357261d203f4850e2eba6 but it has been
ineffective in stopping the log-spew output, see the
dependency change.
Change-Id: I60f06f84d57087c82b3907575bff125015d35171
Depends-On: I1e39822f218dc0322e2490a770f3dc867a55802c
It turns out our manual locale cleanup is causing issues (see
I54490b17a7f8b2f977369044fcc6bb49cc13768e). Upon further
investigation, I think this is a better approach than manually
deleting repos.
glibc on Fedora obeys the %_install_langs macro for reducing the
installed locales (as mentioned in the comments, F24 has moved to
having different packages, but worry about that later).
So our existing clear-out is really only required for CentOS, whose
glibc does not have any way to indicate to build less locales.
However, %_install_langs is still correct there, as it restricts some
of the translation files and other things installed with the %lang
macro in spec files.
This is complicated by us having to set this at glibc-common install
time, which happens with the "yum" from outside the chroot (i.e. on
trusty). Since this is too old to have flags to pass this, we need to
fiddle with rpmmacros.
I've tested this with fedora-minimal builds and the locales file is
about 2MiB, which is what it was after the cleanups, and the listed
locales are only those we expect (i.e. it appears to be working).
Change-Id: I528a68beeb7b2ceec25ccbec1900670501608158
Because Fedora 21 is EOL, the appropriate cloud images were
removed from the mirrors during the last days.
Because of the removal, currently all CI tests are failing.
This patch removes the Fedora 21 CI tests.
(Adding and supporting Fedora 23 is done with another patch -
because some additional changes are needed here.)
Change-Id: Ib85bb6fafd4f56ecc55dd420048f4d9e6e6969f3
Signed-off-by: Andreas Florath <andreas@florath.net>
I realised I'd been using die() in a few places assuming it was
available, but it wasn't exported. I guess it didn't matter because
whatever was wrong, we were failing anyway :)
This exports the function to make it available to sub-processes, which
should remove the need to source it as done in several places.
Change-Id: I7b9a5a6db406e160099b6ed9fde80455ae227327
If REPOREF="*", HAS_REF will be used without initialization. As -u is set
the script will terminate with error.
Change-Id: Ic1d88415adfef66dfc6c1d92610a45a9eb6359f3
Signed-off-by: Liu Qing <liuqing@chinac.com>
The "set to localhost by default" behavior for manage_etc_hosts has been
deprecated for more than a year now by change
Ia8582883f737548e2911d3f36a1943e5b236281b.
Setting that value to "localhost" is still possible, but it won't be the
default anymore. If the previous behavior of assigning the hostname and
FQDN to 127.0.0.1 is still desired in some environments, it can be
achieved by setting the DIB_CLOUD_INIT_ETC_HOSTS environment variable,
as the deprecation warning message suggested.
Change-Id: I5a19d46e2f305769a0c89c9d25d2e6be02910221
Add a pkg-map to install cloud-initramfs-growroot on Precise, as it is
required there because the kernel is too old to use growroot on a live
file-system. This was generically removed in
e23b087505.
Change-Id: I5e6cbc4b74dc72a6f23f73a018c028f3ff7c0157
Add ability to filter in pkg-map on release, which defaults to
DIB_RELEASE. As per the examples, release is a more specific target
than distro, but distro still has to match.
Added a debug flag and ability to use a pkg-map directly for
debugging/development.
Change-Id: Ie282f96966e46236b06bc276de0168fc7a66c5da
Export FS_TYPE from img-defaults and use it to remove hard-coded
defaults in the debootstrap mounting. Also, cleanup the suse element
as it should have access to the exported variable.
Change-Id: Ie9b671ca9336060a5ad294be48aa7eff442bf066
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
Currently there is no way for a service to become aware that
dhcp-all-interfaces is finished configuring all the interfaces at
boot time. This causes problems for applications like the
ironic-python-agent which scans the interfaces when it first starts as
part of the inspection stage and can race against dhcp-all-interfaces
bringing up the interfaces, leading to inconsistent results.
This patch ensures that the dhcp-all-interfaces script runs before any
network interface is configured and brought up by the rest of the
system, and also ensures that the ironic-agent element also waits for
the network to be online before starting. This is done by using the
network targets provided by systemd.
Change-Id: Id9583b7f54361aa603a6229da598ad6a0f0f7938
I'm not sure why this comment is here. base works with centos-minimal
(we changed it in I2956aaa49ba3137a799f97e0983ab4b7c93a0a0c) and we've
got images deployed with such a configuration.
Update the comment
Change-Id: I3207f87dc29280183c0960ea863533a4d441081c
"ironic-agent" element is currently removing sudo, which breaks other
elements such as devuser. There appears to be no security or other
reason to do this, it's just the way it has always been. Leave sudo
in as it is considered part of the base cloud images.
Change-Id: Ida9b1885f745146071e4b2d85ae59341ac85d5c8
Closes-Bug: #1572486
The apt-sources element did not work with debian-minimal, because
the later one overwrote the /etc/apt/sources.list file created by
the apt-sources element.
Two changes were made:
o the debian-minimal uses now files inside the /etc/apt/sources.list.d
directory. Therefore there is no possibilty for clashes between those
two elements any more.
o instead of only adding backports, also the updates and the security
repository is added by default which gives perfect initial
configuration for a stable system.
If you want to use local mirrors with other naming schemas or an
unstable tree, there is the possibility to fully specify the
repositories.
Change-Id: I69dbaa34be3db3d667e6bd8450ef4ce04a751c70
Signed-off-by: Andreas Florath <andreas@florath.net>
The various distro elements include a copy of this script which installs
all files in the bin/ dir of the copied element hooks in to
/usr/local/bin. Lets share code rather than repeating ourselves.
Change-Id: I354382f8c42ede2e9b8c548b7df8367c03e6836e
As motivation for this; we have had two breakouts of dib in recent
memory. One was a failure to unmount through symlinks in the core
code (I335316019ef948758392b03e91f9869102a472b9) and the other was
removing host keys on the build-system
(Ib01d71ff9415a0ae04d963f6e380aab9ac2260ce).
For the most part, dib runs unprivileged. Bits of the core code are
hopefully well tested (modulo bugs like the first one!). We give free
reign inside the chroot (although there is still some potential there
for adverse external affects via bind mounts). Where we could be a
bit safer (and could have prevented at least the second of these
breakouts) is with some better checking that the "sudo" calls
*outside* the chroot at least looked sane.
This adds a basic check that we're using chroot or image paths when
calling sudo in those parts of elements that run *outside* the chroot.
Various files are updated to accomodate this check; mostly by just
ignoring it for existing code (I have not audited these calls).
Nobody is pretending this type of checking makes dib magically safe,
or removes the issues with it needing to do things as root during the
build. But this can help find egregious errors like the key removal.
Change-Id: I161a5aea1d29dcdc7236f70d372c53246ec73749
The dhcp-all-interfaces and simple-init elements did not have the ISC
DHCP Client package mapped for OpenSUSE, which caused DIB to fail with
"'isc-dhcp-client' not found in package names. Trying capabilities."
Similarly, the bootloader element did not have the grub-pc package
properly mapped for OpenSuse, which caused DIB to fail with "Package
'grub-pc' not found.".
This patch adds the package mappings for these elements so that the
opensuse element can be created and booted successfully.
Change-Id: Ife478158fec3a95de73a9206b38dcc6511d56cc8
The proliant-tools element helps to do RAID
configuration in ironic for HPE servers.
This fix proposes to install the proliantutils
in ironic-python-agent's virtualenv created
using ironic-agent element.
Closes-Bug: 1563648
Change-Id: If63c725a42740ab244a2b4004797cba09d0f154e
Every platform has a different name for their DHCP client, so use a
generic name "dhcp-client" in the package name and let everyone choose
their sub-name. This also brings some consistency across simple-init
& dhcp-all-interfaces
Change-Id: I797aa7aacb13dfb7f35700463dc11d55552eb108
This makes use of the dhcpcd package and it's ability to run on all
interfaces by default. We disable the privacy extensions and dhcp
overriding the hostname (both are enabled by default). Other than
that it 'just works' and was the method used to bring up interfaces
on Gentoo Openstack images before we switched to building with DIB.
Change-Id: I02c14927d70b22f560c6fc149fefca0f93933f56
Rather than removing all locale related stuff in cleanup, strip the
locale archive and rebuild it.
Building just en_US (along with POSIX/C) brings things inline with
debootstrap. As discussed in the bug referenced, this is about the
best we can do for Centos7.
Fedora 24 has split languages out into packages so we don't have to do
this, but I have not dealt with that yet. A guard is put in place so
we make sure we revisit this when we try to build F24.
Change-Id: I3f384d23e52effd6a09f47134746caa4a5c586be
cloud.centos.org appears to have changed their naming for images.
This latest iteration drops the YYYYMMDD in favor for YYMM, but
also has a 'latest' available without the date stamp.
This change will mean we no longer have to submit new code reviews
whenever centos changes.
Change-Id: I5a6a0de822561c1d0681abb9487993acf55918f1
After a bit of spelunking, I90d0c96d5659326ba67d6119b96d9a4113adf7fe
was the original change that introduced the setfiles here rather than
autorelabel at boot time.
Touching the autorelabel file probably makes sense somewhere low, but
when we start relabling the file system we really should be doing that
as late as possible so we fix up everything that has come before.
Move this to 90 to capture this.
Change-Id: Iae0afe850f52ec3b59c49507fa9bbcc1c8f8cfa1
If the initial yum install into the chroot fails, we can leave behind
a lockfile and an incorrectly modified rpmmacros.
Change this so we run the cleanup unconditionally.
Change-Id: Ia9f9c4c845e5f34d33ff9a4ab7226c9175283757
I guess I hadn't tried centos-minimal without the puppet elements that
install this for us. But the "base" element wants dkms, which is only
in EPEL for centos. But it's a helpful convenience so is globally
useful.
Change-Id: Ia9af97efdbd855fb8202353196ad649093788cb8
For whatever reason, RHEL identifies itself with DISTRO "rhel" for 6
and "rhel7" for 7, but centos just uses "centos" and DIB_RELEASE. So
this was wrong and installing EPEL6 on centos7.
But we can simplify it completely for centos because that comes with
the epel-release package already included.
Change-Id: I2b8f5d30b850fef46b4a5ba32a917abcbf25932c
This commits provides support to add certificate while
building the image using ironic-agent element. The
certificate can be CA certificate or self-signed certificate.
The certificate is set to the environment variable
'DIB_IPA_CERT' which in turn is used by the ironic-agent
element while building the image.
Change-Id: I648f7934d4787dcc3030885cfca771b642a9595e
Running 'disk-image-create -a amd64 -o precise vm precise' will fail to
build with the following error [1]. To fix this, we should split out the
switch-case to allow base to install the 'software-properties-common'
package.
[1]
dib-run-parts Fri Mar 18 18:43:16 UTC 2016 Running /tmp/in_target.d/pre-install.d/03-baseline-tools
Hit:1 http://mirrors.cat.pdx.edu/ubuntu precise InRelease
Reading package lists... Done
Reading package lists... Done
Building dependency tree
Reading state information... Done
Package python-software-properties is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source
However the following packages replace it:
software-properties-common
E: Package 'python-software-properties' has no installation candidate
Change-Id: I011f02fcf70df9d1d6fa30ed89907fbc2588937e
If we dont force the hostname to be debian, the hostname will
be the one choosen when creating the instance with horizon
This bug was introduced when the debian element was split
into debian and debian-minimal and debootstrap in commit 23aa76df
Change-Id: I007a3531f6876890a2e113082211889ec5ead1a1
In order for glean to operate properly and regenerate ssh host keys it
needs to start on a host without any ssh host keys. We hadn't been doing
this meaning that new host keys were not generated on images using
simple-init. Fix this by having simple-init delete any host keys found
in /etc/ssh late in the build process.
This should force glean/sshd to generate new host keys on first boot
depending on the distro.
Change-Id: Ic917387d92febf21c2cc37dfc1fa83fcf0c8e469
The dib-run-parts/root.d/90-base-dib-run-parts script would mysteriously
fail whenever the dib-run-parts is not in the path. The install
commands complaining with:
install: missing destination file operand after
'/tmp/image.ILFTBYVy/mnt/usr/local/bin/dib-run-parts'
The reason is chaining a command in exec bypasses set -e. For example:
exec /bin/ls $(which doesnotexist)
Will works and continue.
Handle the which lookup in a variable to have the script abort
immediately whenever dib-run-parts is not found.
Left an inline comment to make sure the fix is kept around in the
future.
Signed-off-by: Dan Duvall <dduvall@wikimedia.org>
Change-Id: Ibb5d7342b0d06483b84f9dd567e1cc0bf90f194e
Tgtd starts after OS boots up automatically, listens on the port 3260,
which cause lio fails to bind the same port later.
This patch removes the auto-start script of tgtd.
In IPA service, lio or tgtd would be used to create iscsi target.
If tgtd be chosen, IPA will start tgtd before iscsi setting.
IF lio be chosen, tgtd would keep not running.
Change-Id: Ic62055ac2f9252df832739631bfcbabfac0f9c4e
Closes-Bug: #1554987
On ubuntu, if an interface exists but is unconfigured then ifquery
$iface will return 0 but no output. This will cause dhcp-all-interfaces
to not attempt to dhcp on that interface.
Change-Id: Ic1532728ae9ca6d759714392f727d25b814ef336
As the doc says, we want to remove all interfaces that come by default
in cloud images. Cloud images also come with an eth1 though, so lets
just remove all interfaces.
Change-Id: I49c74a3285e6a610ac723bdf976f3727d2736749
Where the base install for the image uses systemd but
the iptables service is not present, the postinstall
script for the ironic-agent element can fail. This is
due to 'systemctl disable iptables.service' returning
an error status when the service isn't present. Check
whether the service is enabled using 'systemctl
list-unit-files'.
Change-Id: I8069b062a27caead4097747d7aeae24a5262e117
Add a pkg-map file so that Gentoo can avoid polluting the FS with
files not tracked in the package manager.
Changed the directory name so that the source-install method would
actually be disabled when DIB_INSTALLTYPE_pip_and_virtualenv=package
is set. Also changed source-repository line to be consistent with
the element name.
Change-Id: I5225bd7113ff2588c755ad4cd4ae9a2d78fff196
If you check logs like [1] it's literally thousands of lines of the
same thing over-and-over as the git caching happens. It is basically
all just noise unless you're debugging it specifically. Up this to
tracing level 2 ("-x -x") to see it. Add a note in the help about
multiple flags, which has always been intended but not documented.
Image builds should continue to run with single "-x", but we could
probably greatly increase signal:noise ratio in the logs with a little
more judicial use of this to turn down some of the very noisy &
repetitive parts.
[1] anything in http://nodepool.openstack.org/
Change-Id: I91c5e55814ba9840769357261d203f4850e2eba6
This commit adds 'psmisc' and 'dosfstools' to the required
packages for ironic-agent element which in turn
installs ironic-python-agent and ironic-lib.
The ironic-lib requires the fuser command to be executed,
which needs the package 'psmisc'. Similarly mkfs with
'vfat' requires 'dosfstools' package.
Closes-Bug: 1554075
Change-Id: If421f2394c34bb938813d0e15e5085d093215921
This element installs package to ensure that the resulting image
provides the binaries outlined in posix specification [1]. This solves
some problems such as cron missing from the ubuntu-minimal images, and
hopefully minimizing the problems from similar problems in the future.
This is the image size difference:
Resulting qcow image sizes:
Ubuntu Ubuntu-Minimal Debian Centos7
None 334MB 330MB 352MB 470MB
Posix 353MB 375MB
Posix+ 354MB 374MB 406MB 471MB
Posix+ = Posix + gcc + cssc
[1] http://pubs.opengroup.org/onlinepubs/9699919799/idx/utilities.html
Change-Id: I2a8f77bf8a0fdb04b328e56425d43a60eb3f7eed
The vm element makes use of parted. In order to create partitions
parted needs to be installed, growroot does not work without partitions.
Change-Id: Ib46cb818e0116f050de01f9a19e6041328d6d478
We are currently removing only eth0 network config in the simple-init
element (since simple-init is responsible for networking if you are
using it). We install an eth1 as well in elements such as debootstrap.
Lets just rm eth*.
Change-Id: Id9ea8c5e402f3a06b9064a939ad4b2e168cd6223
