This patch fixes the profile autodetect. If the profile was passed at install
time, it would overridden by the environment.d script. This was fixed by
accepting the value passed by the user, if one was passed.
The enviroment.d file is run outside of chroot, because of this eselect will
not be available on non-gentoo systems. We change the behavior to work
elsewhere, it was not working on non-gentoo systems and giving bad results on
gentoo systems (returing values from the host, not the guest being built).
This patch also fixes shm detection by checking if it is mounted already.
Change-Id: Ie58d8e19529a731bfbc9eeb4bb246988d1aaa772
Newer distros, such as debian jessie and ubuntu xenial, do not provide
ifupdown by default, but simple-init depends on it. Add it to the pile.
Change-Id: I6f4876863c67c65a82464d4e0593015cdc839c5c
This is breaking deployments that use registration. I
believe os-refresh-config scripts don't source environment.d
files, so the variable is unset there.
Given that this a blocking issue for RHEL deployments, I'm
pushing a quick revert and we can come up with a proper fix
when this isn't stopping other work.
This reverts commit 71bd8b3a33.
Change-Id: I87504660352220d45f5445bb933edc1c28885fcb
This patch update the find_interface to lookup
for InfiniBand interface according to it's BOOTIF
Closes-Bug: #1532534
Change-Id: I21b91cfd10888ac036f6347a0a44cdca422830a6
In certain cases, with packages cached that need an upgrade,
when performing that action the system hangs waiting for
a prompt.
Add force-confdef option, that will let dpkg overwrite
configuration packages that were not modified. In combination
with already existing force-confold flag, it will allow
to avoid any conffile prompt.
Change-Id: Ifb177f9ac2c9ad29f8b92309c5b8cfe8e60a4e14
In order to avoid conflict with installing dependencies for IPA from
pypi and distribution packages this patch propose installing IPA in a
virtual environment.
Closes-Bug: #1506792
Change-Id: I6a4c6403813d86f4110e98449ddd874109101b9e
By default we create a tarball of any debootstrap rootfs we create. For
the majority of use cases this is a large performance hit for no
benefit. Lets make this an opt-in feature.
Change-Id: I58fc485aacacaa17243bf9ce760ed91256d1f182
python3 is a hard requirement of dnf so can't be removed [1]
grubby is also required for kernel installs on Fedora. For too much
detail see I1a6e45d04755515286b3d49f8280c16b527e2f48; but the kernel,
via dracut, now has this as a "recommends" due to people removing it
and making unbootable systems.
[1] http://logs.openstack.org/76/248976/2/check/gate-dib-dsvm-functests-devstack-f21/734c8bd/console.html
Change-Id: I5867ecd57834eece9477aa9ea4b8bdd70e238084
The debian-minimal element creates /etc/apt/sources.list solely with
the 'main' component. I need to add 'non-free' and 'contribs'. I tried
to pass them via DIB_DEBIAN_COMPONENTS but it is not recognized.
Make debian-minimal to honor DIB_DEBIAN_COMPONENTS. Note that it is
comma separated for 'debootstrap', so replace commas with spaces to fit
the needs of sources.list.
Example usage:
DIB_DEBIAN_COMPONENTS='main,non-free,contrib'
Will debootstrap with the three components then when debian-minimal is
realized pre install a sources.list that has:
deb http://example.org/debian jessie main contrib non-free
Change-Id: I1dca2e8ffd31044a6b441ccb277298601e62f67c
Gentoo provides eclean-dist via the gentoolkit package
eclean-dist isn't needed anyway was /usr/portage is removed anyway
Removed redundant package update in cleanup.
Change-Id: Icf4f9ed549b9a6d923448d927d7c42bcf8d6091b
This action has been reordered so cleanup can occur before copy to blockdev
occurs. Documentation has been added about the ordering of this element in
relation to cleanup actions as well.
Change-Id: I3f9334a3669ee588d7fa7129202c97fa22fdb050
While it does save 1MiB of space, it might also pull python packages
depending on it. E.g. it makes impossible to install python-hardware
on the IPA image for advanced introspection.
Change-Id: Iab80dde63e6de62a5e45dcf404b4f9f633e50ac3
We were getting some subtle issues in fedora-minimal builds that
turned out to be because /var/run was not a symlink to /run.
Upon further investigation, it turns out that yum is creating a
/var/run directory for it's pid file when it starts working in the
empty chroot (which I verified by stracing it)
---
5905 stat("/home/ubuntu/tmp/dib-tmp/image.Ac4VZZsl/mnt/var/run", 0x7ffddffa0330) = -1 ENOENT (No such file or directory)
5905 mkdir("/home/ubuntu/tmp/dib-tmp/image.Ac4VZZsl/mnt/var/run", 0755) = 0
5905 open("/home/ubuntu/tmp/dib-tmp/image.Ac4VZZsl/mnt/var/run/yum.pid", O_WRONLY|O_CREAT|O_EXCL, 0644) = 6
---
Because this happens *before* we install "filesystem" (the package),
we mess up it's symlinking.
To work-around this, pre-install the trio of base packages (setup,
basesystem, filesystem) with rpm from outside the chroot.
Change-Id: I411b6ec9d91d95d3a0f98e76853086af3b70abe8
The Ubuntu Xenial cloud server images set the mode of
/var/lib/apt/lists/partial to 700, so when mounted it's inaccessible to
an unprivileged user, resulting in an error:
find: `/tmp/image.aDQKdkRi/mnt/var/lib/apt/lists/partial': Permission denied
There's no reason an image should come with anything already in
/var/lib/apt/lists/partial, so just avoid trying to descend into that
directory when fixing the apt translations packages.
Change-Id: Id27f0166bfb09d67200f337a5ffff2f2037b7c1c
This is a slight refactor I found useful when debugging. The
udevadmin info query will be visbile in the output of "set -x" which
helps, and is the logs/journal.
We can also reduce some calls by keeping the value and just grepping
with a herefile.
This also does some error checking and bails out if it does not see
what it needs to continue.
Change-Id: I39c4d262f9c5ce53f6b83d95b1363a74834cf2c8
Tucked away in systemd-udev-settle.service is the following comment
# This service can dynamically be pulled-in by legacy services which
# cannot reliably cope with dynamic device configurations, and
# wrongfully expect a populated /dev during bootup.
The info that the growroot script is querying is populated via udev,
particularly the blkid bits of [1]. This creates a race-condition
where sometimes udev has been triggered and the rules have applied and
sometimes not. Obviously in the first case, the root disk is not
grown correctly.
systemd-udev-settle is mostly disabled on distros because it can cause
an increase in boot-time for systems with lots of disks; this is not
our situation so it makes basically no difference.
That said, I will investigate if some systemd people know even better
ways to do this (possibly the service should depend on block .device
targets in systemd, and then filter out and only apply to the root
disk?)
[1] https://github.com/systemd/systemd/blob/master/rules/60-persistent-storage.rules#L66
Change-Id: I453e3afcd953dfc29ab6c42ddc81e940cfa70ee0
A TODO was placed on the partitioning section of the vm element to
replace sfdisk with a saner (and less arcane) way of partitioning. It
suggested parted for replacement. This changeset should reproduce the
same disk label and partition layout as sfdisk, but with less ioctl
errors and version dependency. It will also ensure partition alignment.
Change-Id: I5d8d75131458b73bfb05f80f1bfa7e2970e004b3
We currently install pip from package in the simple-init element.
We should really allow users to select whether to install pip from
git or package.
Change-Id: Ia5e62b9635af90d81227274a1dd8f20474cdbf73
As described in the comment, there is a dnf equivalent of this command
that doesn't require us installing yum-utils (which drags in yum on
dnf-only systems such as f23)
This is a small consequence to this -- due to us not installing
yum-utils some installs will now be completely yum free. This causes
a breakage in ironic-agent 99-remove-extra-packages where we remove
the yum package. There is a long-standing bug/feature where missing
packages in a group of packages do not cause yum/dnf to exit with
failure, but uninstalling a single package will. Because we have made
the systems yum-free, the uninstall of yum can fail in this corner
case.
It has always been like this, so I'm in favour of the "ain't broke"
approach. To work-around this, I have just put yum into the existing
list of packages to be cleaned up. I have added a note to the yum
installer taking note of this behaviour for future reference.
Change-Id: I8bbdc07ccdb89a105b4fc70d5a215077c42fcd03
InfiniBand interface takes more time to bring up then
Ethernet interface. This patch just increase the retries
to 20 times, to make it work for InfiniBand as well.
Change-Id: I5c4842696207885552413ea2d053f2e90bd6803c
Adds a post-install function that enables installed initscripts,
as that is not done by default in gentoo.
Change-Id: I04e8d506ddcbefa8a983dd31ad16df5e13cb26e7
Closes-Bug: 1539276
This checks the profile, if it has hardened in it's name it needs xattr support
unfortunately xattr support cannot yet be relied on everywhere, so it needs to
be disabled for hardened profile builds to correctly pax-mark.
Change-Id: I7fb855249a9e6c9b6497ab5061b4ea3c014f5081
Closes-Bug: 1537177
Due to upstream bug [1] there are uninstallable packages which mean
our functional tests don't work. We will revert this when things are
working upstream.
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1303660
Change-Id: I93c2990472e88ab3e5ff14db56b4ff1b4dd965ef
subprocess.check_call() returns a byte-string which needs to be turned
into a unicode string for python3 compatability.
Also some minor refactoring while we're here.
Closes-Bug: 1536462
Change-Id: Icd957bc4d93ccad94b1246ad62e6e02ee14d9ca5
Add missing growroot initscript and pkg-map entries for Gentoo.
growpart was added to Gentoo with [1]
Update the readme to reflect reality too (fedora added with
I5630dc638f85b1e80795826ef36a306632075460)
[1] https://packages.gentoo.org/packages/sys-fs/growpart
Closes-Bug: #1539273
Change-Id: I29056c7297489ec04f37757dbe33976901eceb49
As mentioned in package-installs.yaml, git is a transitive dependency
for pbr in this element. Add pkg-map for the Gentoo package.
Change-Id: I7f2fe1663152ea66b941594e86f1da93ddd21677
Closes-Bug: 1539278
Our dib-lint checking is only considering scripts with #!/bin/bash.
While there's nothing really wrong with some other shebang line like
"#!/usr/bin/env bash" let's keep things consistent.
We can use the same regex match to reduce a few forks in the main
checking.
Also a minor cleanup to the file matching
Change-Id: I609721b2671e704ea26075dad7e5b39a8b858f6b
'locales' package gets installed before '12-debian-locale-gen' is executed
and generates effectively empty /etc/locales.gen in debian, which makes
dpkg-reconfigure call to locales ignore the values set by
debconf-set-selections.
* Remove /etc/locale.gen generated by 'locales' installation to ensure
proper locales generation on debian images
* Remove 'locales-all' package installation from debian element since
it's not needed anymore to build the image and cosnumes additional
~120MB of space
* Remove unused 'package-installs' dependency from debian-minimal
element
Change-Id: Ic39ba2b5ceb5018efb75742547b2babf80827e56
Closes-Bug: #1452400
Add systemd/fedora support to growroot element. This involves
installing the correct packages, shipping the systemd service file and
ensuring it is enabled.
Note the required growfs/resize packages for Ubuntu/Debian are
installed in other places. This is probably a bug in that path, but I
have not addressed that here.
I have tested this with a F23 build with all openstack-infra elements,
uploaded to RAX, and it boots and resizes the main file-system.
Change-Id: I5630dc638f85b1e80795826ef36a306632075460
The undercloud actually has dib run twice on it - once to create
the instack image, and again when we run instack itself. The
first run creates the dib-python symlink, and the second blows up
because the link already exists. Force the link creation so the
script is idempotent.
Change-Id: I78f9e6f5afcf8ebe6d7911a7a434525ba7c737cf
This is to aid with Fedora packaging, since rpmlint complains about
including empty files.
Change-Id: I4ad867cd21304880a571e46805ab56044542400c
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
The bootloader element lacks the mapping for grub-pc and extlinux
This adds it.
Change-Id: Ic7b82903f02faaab143d2bd682876bf4853fd90d
Closes-Bug: 1534387
While the patch looks sane and the change worked locally,
it has broke the ironic-inspector gate. As we're close
to deprecating the DIB ramdisk in favor of IPA, I suggest
reverting it.
This reverts commit 802f14862c.
Change-Id: I0525e545cb2fe8ce184312a2f9bbe3763904f61a
Closes-Bug: #1534648
As you can see in the CI failures for
Ib11b9df84b593ab25232729a570c812f1b4b8774, you can not see what file
is causing the problems when the parser fails. Add a print, but raise
the error as it contains the cause.
Change-Id: I127ff7d57d2d898969195464c6e774d496e872e6
Our bootloader install fails on non-gentoo builds due to missing pkg-map
for grub-pc. This map should really live in the bootloader element, so
move it there and fill it out.
Change-Id: Ib11b9df84b593ab25232729a570c812f1b4b8774
uses upstream's stage4 images, includes all the needed bells and
whistles for openstack on kvm.
Change-Id: Ibca43173c30c2a74a73a2e2d9dd6d6d832c62694
Closes-Bug: 1530911
Hardcoding subscription-manager to use rhel-7-server-rpms causes
users building rhel6 with Satellite6 to fail. This setting cannot
be overridden with environment variables, therefore needs to be
smarter. Setting RHEL_MAJ_VER in the rhel/rhel7 environment fixes.
Change-Id: Ifbd88bc76ef8b38a739272ba6e045a12849d68df
Closes-Bug: 1404364
This patch is fixing a syntax error in the 70-ironic-root-device init
script for the deploy-ironic element.
Change-Id: I767486ca5893605720fba41bee3af72725a26377
Closes-Bug: #1531835
This element allows installation of pip and virtualenv from either
distro packages or git.
Change-Id: Id294f0936c8fef8a3b27a415bfcc93b3f327e104
Depends-On: I731cc8a0f5bfeda8f17a78c33b9f44062323a361
Use dib-python to run package-installs using the provided python
version. Automatically detect the python version for our
package-installs-squash since that runs outside the chroot.
Change-Id: I926022bcf8cbcd81b051026ffd5d6477650045ad
Fedora has changed the location of epel, shorting the link
from 'download.fedoraproject.org' to 'dl.fedoraproject.org'.
This change updates the epel mirror to prevent it from timing
out.
Change-Id: I87090282a2f5f757495daec6ad14123b436b1aa0
This fix uses dmidecode and awk to simply multiply by 1024 when
the value is represented in GB, otherwise it returns the given
value. I should note that I've only observered this occurence
on "some" SuperMicro Hardware
Closes-Bug: #1486689
Change-Id: I352b1891326f72af3a56c7bbe8b7f3c422169404
Install selinux policy packages as part of the base-installs. selinux
is part of the base-system and the kernel boots by default in selinux
mode.
Without both of these, we can get in a situation where later scripts
(particuarly, some of the infra scripts) might install systemd-policy
without a base policy (targeted), leading to a messed up situation
where systemd will halt during boot due to missing policy files.
Change-Id: I6bf156304d1134fb328fba9b12dc364701b13696
Add an environment variable to control the creation of eth0/1
interface enablement scripts.
With a tool such as glean, the presence of these scripts will indicate
the interface is configured and configuration-drive settings will not
be applied. This means in a non-dhcp situation like on Rackspace,
network is broken.
On Fedora, where later systemd provides "predictable network interface
names" [1] eth0 & eth1 ironically aren't predictable so this just
confuses things. You really need cloud-init or glean or something to
bring up your interfaces in a sane fashion.
This maintains the status-quo on centos-minimal, but disables creation
for fedora-minimal.
[1] http://www.freedesktop.org/wiki/Software/systemd/PredictableNetworkInterfaceNames/
Change-Id: I3f1ffeb6de3b1f952292a144efab9554f7f99a5f
As described in the comment, systemd will create a broken
/etc/resolv.conf link if there is no file in the base-image (as you
can read in the bug, it is debated if this is a bug or a feature).
The solution is to leave a dummy /etc/resolv.conf file in the image.
Whatever network manager you choose (NetworkManager, glean,
cloud-config, etc) will overwrite this anyway.
It's just that some tools, such as dhclient, get confused with the
broken symlink. This affects you if you're using glean to configure
the network in a DHCP situation, for example -- dhclient won't
configure nameservers and everything goes to heck.
Change-Id: I734834d03e7fdb13f9ab2e86f877b07bf4a84ff9
We are incorrectly detecting major/minor device numbers for the growroot
rootfs. This can also be simplified by querying udev for partition
information.
Change-Id: I68059bf11f2563872f6b4d0e23fa09a15de980a8
The detection logic in pkg-map for DIB_DEBUG_TRACE assumes that this
variable being unset means tracing is on, when in fact this means
tracing is off.
Change-Id: I584a634c57bbe03e26a6ee94cef473e634616885
In order to add more flexibility to the vm and bootloader
elements, split the functionality in two different ones, and
make vm depend on bootloader element.
This will allow to construct more elements that depend on
bootloader, and develop both elements independently.
Change-Id: Iad2503b7b8fe53b768a3bc79e4cb839700fbd747
This element enables creation of Ubuntu deploy ramdisk and
user images which could be used to deploy the HP Proliant
Servers with Dynamic Smart Array Controllers. Without this driver
the disk with the Dynamic Smart Array Controller is
not visible to the ramdisk.
Closes bug: #1492803
Change-Id: Ibb3b298cd379cd7333279484df6ae30e9d7f6aaa
Creating an element which we can use in #! lines to refer to either
python2 or python3 depending on what it available.
Change-Id: Ic47e18ad21c33ab9f0d11c04260a33725aeee814
The modprobe utility is required by the rtslib package (iSCSI Linux-IO).
It will also be required for inspection.
Change-Id: I6760c86160d1ceba45aedde62597a711bcb4543d
Vlan support was recently added to glean. However, if the 8021q module
is not loaded, glean will fail to bring up a tagged interfaced defined
in /etc/network/interfaces.d/. Manually attempting to bring up the
interface results in an error[1]. This patch ensures that the 8021q
module is loaded so that tagged interfaces can be brought up at boot.
[1] http://paste.openstack.org/show/480027/
Change-Id: I15d805c07d4b5e1161d831f0393d027e4325137f
Since we are modifing SSH keys, it should be safe to assume
openssh-server should be installed too.
Change-Id: I17ff05642bb2f0868d4c17819cd91b179068399a
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
When build ubuntu iso image, it will install grub-efi-amd64-signed
and grub-efi-amd64 packages. Both of the postinst script will try
to find root device and install grub which will definitely fail in
such a chroot environment.
So the workaround is to skip error and remove postinst script.
And confirm the package be installed successfully at last.
Change-Id: Ie0aecb212b22362046db55b5ad8c64c3211c28e5
Closes-Bug: #1491280
Co-Authored-By: Jane.zhang <jian.zhang8@hpe.com>
As described in the comments, CentOS overrides the "distroverpkg"
variable in yum.conf. This is the package that yum queries to
establish the value of the $releasever variable. On other platforms,
this defaults to "redhat-release" (which "fedora-release" provides) so
everything works. It is only when the base-system "distroverpkg"
refers to a package not in the chroot we hit the issue.
We can avoid this by setting the releasever variable via the
commandline.
Change-Id: I231c3277960992cd479b8aff7838f246397936f2
This patch is a follow up patch fixing some nits left by the review
25d3ee5471.
It does:
* Fix the README file to say that the password *must* be encrypted and
the option values *must* be quoted
* Adds Type=oneshot in the upstart service config file so that upstart
will not try to restart the service over and over.
* Enable setu, sete and setpipefail in the dynamic-login script
Change-Id: Iee5d75daef24469ccf47ca12de6ead37bf9d8d6f
Allow a user to override the username on where .ssh/authorized_keys is
installed.
Change-Id: I030d5a89260aed8b23a35c4cdc2d67629934b076
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
I recently built a ramdisk for IPA and was confused by
the fact that the source-repositories name did not
match the element name. (this is a convention,
confusing when they don't match but certainly not
required).
This patch makes it so you can use DIB_REPOREF_ironic_agent to
customize the IPA ramdisk sources when building ramdisks.
For backwards compat if DIB_REPOREF_agent is set it automatically
sets the new DIB_REPOREF_ironic_agent to that value as well.
Change-Id: I082d989d0d85601f5984dc7c3767b8d66a3d5438
Troubleshooting an image can be quite hard, specially if you can not get
a prompt you can enter commands to find out what went wrong. By default,
the images (specially ramdisks) doesn't have any SSH key or password for
any user. Of course one could use the ``devuser`` element to generate
an image with SSH keys and user/password in the image but that would be
a massive security hole and very it's discouraged to run in production
with a ramdisk like that.
This commit is adding a new element called dynamic-login, which inserts
a helper script into the image to allow operators to inject a SSH key
and/or change the root password dynamically when it boots via parameters
in the kernel command line.
Those parameters are:
sshkey = If the operator append sshkey="$PUBLIC_SSH_KEY" to the kernel
command line on boot, the helper script will append this key to the root
user authorized_keys.
rootpwd = If the operator append rootpwd="$ENCRYPTED_PASSWORD" to the
kernel command line on boot, the helper script will set the root password
to the one specified by this option. Note that this password should be
an encrypted password.
Change-Id: I6b87a1b90163d79745f30dfacd37516051fa0aea
When the kernel gets installed on Fedora, the rpm post scripts call
"/bin/kernel-install" [1] to install it. This is a script provided by
systemd.
However, in [2], Fedora ships a patch to kernel-install that makes a
call-out to /sbin/new-kernel-pkg -- the install script provided by
grubby [3]
Without grubby installed, systemd's kernel-install script goes off and
runs dracut plugins directly [4], which eventually creates the initrd.
For reasons that are not clearly explained, the initrd will end up in
a a "machine-id" sub-directory of /boot (possibly, so you can symlink
it?). It is also called "initrd", even though it's an initramfs, for
historical reasons in dracut I think.
It is at this point that I think 99-ramdisk has been written to move
the generated initrd file back into /boot. Later on, when we build
the image, we run grub-install and it picks up the kernel and the
initrd and installs everything.
grubby's new-kernel-pkg [6] it's very similar -- it uses dracut to
make the initramfs ... but in this case it is put in /boot and is
actually called initramfs.
The subtle change that led me down this path is that dracut has been
modified to have a "Recommends" for grubby for >F22 [7]. After
discussing this change with the author, it turns out it was *always*
intended to use the grubby-based kernel install scripts for Fedora --
our builds have been incorrect in not including the package. The
author got sick of people removing the package and making unbootable
systems, hence the change.
Thus this removes the workarounds in 99-ramdisk and replace it with an
install of the grubby package. grubby's kernel install script will
put the kernel & generated initramfs in /boot, and it will be
installed correctly via the usual grub install later when we build the
disk image.
I have built F22 & F23 fedora-minimal images with this and they boot.
[1] http://pkgs.fedoraproject.org/cgit/kernel.git/tree/kernel.spec#n1832
[2] http://pkgs.fedoraproject.org/cgit/systemd.git/tree/kernel-install-grubby.patch
[3] http://linux.die.net/man/8/new-kernel-pkg
[4] https://github.com/haraldh/dracut/blob/master/50-dracut.install
[5] 81516adcb7
[6] https://github.com/rhinstaller/grubby/blob/master/new-kernel-pkg
[7] 47ff68e78b
Change-Id: I1a6e45d04755515286b3d49f8280c16b527e2f48
This package seems to be broken and isnt useful for recent releases
(where the rootfs can be resized online). Therefore this should be
optional so people can use things like the growroot element.
Change-Id: I6e3c8d095d9fc188094f3b8811f06be0847ef08c
This patch is extending the root device hints to also look at
ID_WWN_WITH_EXTENSION and ID_WWN_VENDOR_EXTENSION from udev.
Prior to this patch the bash ramdisk only cared about ID_WWN but in some
systems in some platforms with a RAID controller, this ID can be same
even if they are different disks (see bug 1516641).
Related-Bug: #1516641
Change-Id: I45b3910d03d164d880b32169b91e94e88812e183
On Debian/Ubuntu installs of RPM, /usr/lib/rpm/macros sets
%_dbpath %(echo $HOME/.rpmdb)
which makes quite a bit of sense, because RPM is not the system
packager and thus RPM is setup to install things into a hierarchy in
the users homedir.
However, this messes things up when building a Fedora chroot on an
Ubuntu platform.
We use RPM & yum from the base-system to bootstrap the Fedora chroot.
While both obey --root flags, they still pick up the %_dbpath macro
and so end up creating the RPM database in <chroot>/home/user/.rpmdb
After we have bootstrapped yum/dnf, we execute further installation
commands from inside the chroot -- where we now have the Fedora
version of /usr/lib/rpm/macros and hence have _dbpath set to
/var/lib/rpm -- except there is no rpm database there.
Should anyone be finding this in the future, the actual issue that
appears is
$ sudo chroot /opt/dib_tmp/image.b6B5S3f6/mnt dnf makecache
Error: Failed to synchronize cache for repo 'fedora' from \
'https://mirrors.fedoraproject.org/metalink?repo=fedora-$releasever&arch=x86_64': \
Cannot prepare internal mirrorlist: file "repomd.xml" was not found in metalink
Note the issue there is that $releasever is not expanded, because the
rpmdb where this info is kept is not populated.
The trick is to make sure we override this value when using the host
rpm/yum to setup the chroot. The bare rpm calls, which we use to
install the repos, have a --dbpath argument where we can override
this. yum does not however, so we override this in the global
~/.rpmmacros while we are installing the packaging tools and
dependencies into the chroot.
Copious comments are included, because this is super-confusing.
Change-Id: I20801150ea02d1c64f118eb969fb2aec473476f7
This new element installs hpssacli utility (for configuring
RAID) and installs proliantutils python module (which has
ironic-python-agent hardware manager for HP ProLiant hardware).
This module also exposes a new environment variable DIB_HPSSACLI_URL
which allows operator to pass a custom HTTP(S) URL for RPM of hpssacli
utility.
NOTE: This module currently supports only installing from source.
Change-Id: I0494e3db623fdd7ea9182ffba21c0652aaad113c
fedora-minimal fails to build on Ubuntu Trusty due do being unable to
find the initrd (see Id4c04d7ae20068643df34d2fa31068e8a917a52d).
This is a rather obscure problem that comes from the intersection of
several things.
The first thing to note is that the post-install scripts of the
kernel-core package use kernel-install [1]. For whatever reason, this
installs the kernel to /boot/MACHINE-ID/KERNEL-VERSION
MACHINE-ID comes from /etc/machine-id; a UUID that should have been
created by the systemd post-inst scripts with systemd-machine-id-setup
[2].
The chroot environment provided for root.d elements has no kernel
file-systems like /proc or /dev mounted. This is where differences in
the base-system come into play -- on more recent systems that
implement getrandom() systemd does not need /dev/urandom to generate
the machine-id [3]; we get a value and /etc/machine-id is populated.
On older platforms (Trusty), systemd-machine-id-setup fails (unable to
access /dev/urandom) and we end up with a blank /etc/machine-id. This
ends up making kernel-install (the script) fail during yum's
installation of kernel-core, which means the initrd is not installed
correctly.
We end up bailing out in fedora-minimal/install.d/99-ramdisk, where we
try to put the installed ramdisk in /boot for the later grub install
scripts to find.
The solution here is to mount the standard kernel file-systems within
the chroot before we try installing.
[1] http://www.freedesktop.org/software/systemd/man/kernel-install.html
[2] http://www.freedesktop.org/software/systemd/man/systemd-machine-id-setup.html
[3] https://github.com/systemd/systemd/blob/master/src/basic/random-util.c
Change-Id: Ibcce35da928f64e6a719b070bcc833346ee7ee92
Clarify what this script is doing. It currently fails on some
platforms due to earlier errors, see
Ibcce35da928f64e6a719b070bcc833346ee7ee92
Change-Id: Id4c04d7ae20068643df34d2fa31068e8a917a52d
The check suffered from various flaws.
First, due to missing quotes around $initrd, 'wc -l' would always see
1 line no matter how many results the find returned.
Second, echo adds a line break making 'wc -l' count 1 even for empty
string. We need to add a check for empty string.
Change-Id: Ib2c67960f566dbdc471d9585a4cef1beb1cc38ab
Closes-Bug: #1506692
5af25b5f fixed the hostname of Debian images to "debian" since a lack of
hostname definition set the hostname to "(None)".
It has been done by introducing /etc/cloud/cloud.cfg.d/01_hostname.cfg
with content:
hostname: debian
Review supposed the hostname would be overriden by cloud meta-data. That
might have stand true for Wheezy but it is not the case for Jessie.
cloud-init 0.7.6 ignores cloud metadata whenever "hostname" or "fqdn"
are set in a config file. Roughly:
# no fqdn set, get fqdn from cloud
# get hostname from cfg if available otherwise cloud
fqdn = cloud.get_hostname(fqdn=True)
if "hostname" in cfg:
# hashar: set from config file NOT cloud
hostname = cfg['hostname']
else:
# fallback to cloud
hostname = cloud.get_hostname()
Relevant code is
https://github.com/number5/cloud-init/blob/0.7.6/cloudinit/util.py#L839-L860
Only inject "hostname: debian" for the Wheezy release.
Bug: https://phabricator.wikimedia.org/T117283
Change-Id: I6e2522bd725cbf9651f11c76ecdc72ecbc92f402
Previously all files in /root were ignored when building the
ironic-agent ramdisk. This prevented for example to use the
local-config element to connect to the ramdisk via ssh as root user.
This commit change the exclude rule on /root to only ignore the
/root/.cache directory.
Change-Id: I18d839e8d97636f5f2164ba407f252407d9bc956
Closes-Bug: #1451668
yum-minimal/root.d/08-yum-chroot runs before yum/root.d/50-yum-cache,
and thus if run on a completely fresh system will fail in
08-yum-chroot as the YUM_CACHE directory isn't made.
This is probably hidden by testing & nodepool builds, because it sets
DIB_IMAGE_CACHE. It was hidden from me because locally I have done
builds using the "yum" element previously, which had created the
cache.
Change-Id: I333f5f7e67d198f75a522cc296c118c2e94a5ecb
download.fedoraproject.org uses dns round robin and occasionally
hits a bad server. Using DIB_EPEL_MIRROR when finding the
epel-release package will allow us to avoid it e.g. in ci.
Change-Id: I756223b3e669532476663c05e79c238449b8a0db
Xen paravirtualised disks (supported by most modern kernels) have the
"xvd" prefix (e.g. xvda0). The functions to strip partitions need to
match on Xen PV disks otherwise the device name is discarded.
Change-Id: I5539d2afba3fae30d1ddb49dcbf077113d38bbf7
Closes-Bug: #1498576
Grub fails to install the bootloader due to it being on the root
partition of a block device. This is not actually a problem for us, so
we need to force it to succeed.
Change-Id: I335ef04ca8a8a8a5c242d3444b09bcce0a9f51e7
Without this patch, the devuser element attempts to find public keys by
iterating over the string "rsa dsa". When two keys are grouped together
in quotes, a bash for loop treats it as a single key. You can see the
issue this causes when debug output is turned on:
+ for fmt in '"rsa dsa"'
+ '[' -f '/home/krinkle/.ssh/id_rsa dsa.pub' ']'
This is not a reasonably named key to look for, so this patch removes
the quotes so that the loop will look for id_rsa.pub and id_dsa.pub
separately.
Change-Id: I0b5b1abd14013de85d90e76a95918a8071a5e013
Make sure we reset the yum/dnf cache to /var/cache/${YUM}, not just
/var/cache/yum
This was resulting in the F22 fedora-minimal image being larger than
the base-image. Because F22 fedora-minimal does some installs with
dnf when bootstrapping the chroot before we set "cachedir=" to the
bind-mounted external cache, we have "/var/cache/dnf" created and and
populated with the package meta-data, etc.
When we globally point dnf to /var/cache/yum here, we effectively
orphan the /var/cache/dnf created in those first steps. dnf doesn't
care, but we end up with two copies of all the package metadata, etc
in "/var/cache/dnf" & "/var/cache/yum".
This also cleans up the sed a bit, by just replacing the lines.
Change-Id: Icc98fe30c34cb941aed4b987647ab67ac34af15a
I'm not sure why we try to do an extra install of these, it is done
inside the chroot in _install_repos. Currently it just gets skipped
saying the packages are already installed.
Change-Id: Ic7aa8cbe13e4347b447e84bb9c12483a4e125228
Add basic F22/dnf support to yum-minimal path. We extract common
code, add some comments and reduce duplication.
Change-Id: If4bd5f88e26bd6f2168958f1ec1efff1072de7ba
Evidently the readme file hasn't been updated since rhel7 finished
beta, so this is long overdue.
In addition, since it's not possible to download the base image
file directly, let's stop pretending we can and bail out if the user
didn't set the necessary env vars.
Also updated the README to use the new table format instead of free text
Co-Authored-By: Augustina Ragwitz <aragwitz+lp@pobox.com>
Change-Id: Ie8343ee2ce1715583c28de7f59daed7e58c8ca0f
Move yum-based install into a function, to make way for a second
related function where use dnf later
Change-Id: Iad09f3753ecdfa0c10cb8a0970a3c8e5a2dccab1
Find doesn't like listings disappearing while its trying to find them,
in this case if a PID directory disappears while find is running. Using
-xdev prevents find from going into ./proc and as a side effect /dev
will also be avoided which is mounted on boot so not needed either.
Change-Id: Iaa282e58d81d533ad4445da0a44200dd14bf0850
Closes-bug: #1502142
Reorder the script number of 'elements/dkms/post-install.d/99-dkms'
to 'elements/dkms/post-install.d/97-dkms' to ensure that
it will always get executed before the
'elements/ramdisk/post-install.d/99-build-ramdisk'. This
would make sure that the DKMS module is there in the ramdisk.
Closes bug: #1492904
Change-Id: I2145d0ac29646335f76745a7678d169a62f13d44
Traversing the /proc filesystem causes find to error if it changes
while its being searched.
We have had a lot of ci failures on this find command since it was
added in Ibe40e6b8b884f37e3b5aeab6e7654593bcd63123
Change-Id: Ia8cfc923cce749a69d5108e588db2360238d866c
Closes-Bug: #1501949
Otherwise on rebuilding the agent the following error is produced:
failed to create hard link /home/stack/ironic-agent.vmlinuz
and the vmlinuz file is not updated.
Change-Id: I2015da889c932a854727235b1e34256a28e9eac6
Always no quotes for $() statement.
We don't need quotes to hold blanks in result:
# i=$(echo 1 2 3)
# echo $i
1 2 3
#
These quotes can make something wrong in some case:
# i=$(echo '!')
#
# i="$(echo '!')"
-bash: !: event not found
#
No real problem for current code, only to use a better code style.
Change-Id: I5909636bdc8de3d44a305d033c8c892af446acf3
Signed-off-by: Zhao Lei <zhaolei@cn.fujitsu.com>
When a ubuntu/IPA ramdisk is used to boot a baremetal machine with
ironic agent-ilo driver, it fails at the point of mounting /proc
and /sys. After the vmlinuz(kernel) is started and it tries to
load the partitions on ramdisk. It need the directory of "/sys"
and "/proc" to mount the corresponding filesystems.
In order to fix this issue, the directories of "sys" and "proc"
are retained but the subdirectories or files under them are empty.
With this change, the directories of "/sys" and "/proc" shows
up in the ramdisk and kernel will mount sys and proc filesystems
on them respectively.
Closes-Bug: #1488445
Change-Id: Iad5d62f373b73789118f23db4c932ea6e9a784c3
Signed-off-by: Gary Duan <duanlg@live.cn>
ironic-agent requires expect to be installed for config drive
creation and hence this commit adds it.
Change-Id: Ie1c0f488f416b4c373aa7f38dfd8df1917cd6be2
Depends-On: Ib4dd8c082a50e1dbaf0df91477b062716cb780ff
Closes-Bug: #1486967
fedora-release >= 22 has acquired a dependency on /bin/sh. This comes
from a %posttrans section of the spec file, which is symlinking the
os-release file.
As discussed in [1], the links are setup correctly in the rpm, so the
post-install script isn't doing anything. Thus we can safely ignore
the dependency with --nodeps
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1265873
Change-Id: Icf17c84580a75d42d8e90d5d6e81ae7f5f576c32
Adds support for debian to iso.
Updated README to further clarify what the element does.
Change-Id: I17ac89cfbc84365860c591fab0e4c78123035983
Co-Authored-By: zhangjian <jian.zhang8@hpe.com>
The ironic-agent element is created using the disk-image-create utility
(even being a ramdisk) and outputs a .vmlinuz file for the kernel
(different than the ramdisk-image-create which outputs a .kernel file
for the kernel). This is inconsistent and make scripting against the
diskimage-builder more complicated if one wants to support different
types of ramdisk.
This patch creates a hard link for the .vmlinuz file to a .kernel file
at the end of the process (to keep backward compatibility) and print a
deprecation message.
Depends-On: I81400305f166d62aa4612aab54602abb8178b64c
Change-Id: I476f9ec9ec4206ece0261eaaf2b4182c6bcbd802
Closes-Bug: #1482606
Added support for ramdisk-type elements in tests/test_functions.bash
Elements are distinguished by element-type file in a test element.
Note that ironic-agent ramdisk is built with disk-image-create.
Change-Id: I4759859e7f3c004c2d00e7318729602e6c3c4d95
Largely enhance the documentation so it renders nicely when generated
with Sphinx.
Culpirt: the 'package' type is documented but unhandled in the shell
script.
Change-Id: I9f4f46e770077c147c0a5b1245b779bc3afa4e98
The centos cloud images are both arround a GB in size, and
downloading them causes lot of CI timeouts, downloading the
compressed version saves 700MB of bandwith and should save
a lot of time.
Change-Id: I8dcd1db81fe5c4661945638ef3e6344fdf651243
The ability to specify a distro mirror is part of other
distro elements, centos should have one too.
Change-Id: I4cc9062ff92fbe301f414820798e08c66e9793f7
This exact repository along with others are part of the
centos cloud image. From the looks of it this was required for
a very early cloud image.
Change-Id: Ib928e4ea739bc48f196f81c96ed4fba3177471f0
Completed in Kilo the blueprint ipa-as-default-ramdisk [1] ported all
the Ironic drivers in tree to be able to use the IPA ramdisk for the
deployment.
Now in Liberty the blueprint deprecate-bash-ramdisk is deprecating the
bash ramdisk created using the "deploy-ironic" element in DIB.
This patch is printing a deprecation message when the user uses the
"deploy-ironic" element and as well updating the README file to indicate
that it has been deprecated.
[1] https://blueprints.launchpad.net/ironic/+spec/ipa-as-default-ramdisk
[2] https://blueprints.launchpad.net/ironic/+spec/deprecate-bash-ramdisk
Related-Blueprint: deprecate-bash-ramdisk
Change-Id: I8057f52104225326f45eb3ae6065cd02a27f5ef2
In Id1e430e7d050a0b99ac449e2ea435e06cda1c4e6 I made the mistake of not
actually removing grub in 15-remove-grub.
This restores the removal phase and adds a bunch of comments. It
seems the centos7 and centos (6) images have grub2 installed, but F22
does not; hence the check.
For anyone interested in the history; it seems the whole idea of
removing grub and re-installing it in the finalise stage is to do with
Ubuntu grub scripts failing in the chroot. It is not clear this does,
or has ever, affected rpm based systems; but that's how it is, so
leave well enough alone.
The whole reasoning behind the rpm download & re-install is actually
explained in If095adc4abb52a19a3aa0b1caebfb3e4d8f605ef, but over time
the comments got lost as code moved around. I've restored in here
some detailed explaination of why we don't just re-install the package
"normally". I've also added a note to the pre-install of various
things that are related to this step. Again I think there are some
questions around this that we can investigate in another change.
Change-Id: I1acd19da8567ab93b5003caf67673cc70efea5fa
Currently they are used for inspection, but may be also used for
other purposes, as they're accessed from IPA generic hardware layer.
Change-Id: I32c6a711d466131b9445023812a2a260ed2e01f3
Switch to using svc-map element for systemd based agent.
This allows both .deb and .rpm installs to share the
element for systemd based installs. There are not any
plans to package a .rpm package for upstart or sysv, so
these are left as is.
Change-Id: Idca7ad97355cae785162989774a7e6dea6fdc5b5
Closes-Bug: #1490584
Fixing the ironic-agent pkg-map by adding missing commas. Validated
updated form passes json linting. Also includes a listing for curl.
Change-Id: I1983f7a581be3a5aaa771b19c6609cf12b61a7bb
Closes-Bug: #1488969
Appears that growroot was running before /dev is mounted so the script
is unable to introspect the filesystem partition info. Run this after
all local filesystems are mounted to fix this issue.
Change-Id: Ia7c41ba6ef79788fdbf198998622eeaa20dd4245
We can resize the rootfs without the initrd based approach. Create a
growroot element which performs rootfs resizing as part of system init.
Change-Id: Ibeb846b0170d141fb72323a441d14b65b93ae0a1
There is a bug where the init scripts element incorrectly munges the
install path making it useless. Also removing the dep on rsync since
this occurs from inside the chroot.
Change-Id: I8f2717d36d7d2ff4b195ec21e91afeaf30a1d803
This patch is reducing the size of the ramdisk image generated by the
ironic-agent element. It does remove extra packages (graphical stuff,
dev stuff, miscs, docs, etc...) and purges directories that are not
needed for a ramdisk (like /boot since it boots using an external
kernel)
Currently it was tested generating a Fedora 22 image and reduced the
size of the final image from 464 MB to 211MB compacted (54% decrease).
I was able to boot a VM with 1.3 GiB of ram instead of the previous 3 GiB
needed.
Change-Id: Id6333ca5d99716ccad75ea1964896acf371fa72a
The default value was set in the centos7 element, but not
exported, which caused issues in rpm-distro. Also changed
a test in rpm-distro to only check for DIB_RELEASE > 22
if it's fedora.
Closes-Bug: #1477172
Change-Id: Ib6f4227411c2e8f1965c3b78bc318512c59a7876
The script for ironic-agent utilizes curl, however an extremely
minimal system may not have it, and as such we should list it as
a package that must be installed to support the element.
Change-Id: Id118f84e2d5e6adf0ae3d653864565368b0d76bf
As described in the comments, sfdisk was rewritten for util-linux 2.26
(as shipped in F22) and now interprets arguments a sectors, rather
than cylinders.
The current partitioning line is "1 - - *" (start/size/type/bootable)
which means you start getting:
---
/usr/sbin/grub2-install: warning: this msdos-style partition label has
no post-MBR gap; embedding won't be possible.
/usr/sbin/grub2-install: warning: Embedding is not possible. GRUB can
only be installed in this setup by using blocklists. However,
blocklists are UNRELIABLE and their use is discoura ged..
/usr/sbin/grub2-install: error: will not proceed with blocklists.
---
when building images, because the start is interpreted by the new
sfdisk as sector 1 and it crams the partition right next to the MBR.
Specifying "-" for the size is undefined in the man page; even reading
the source it's not totally clear what "-" for the size does [2]. In
any case, the alignment is wrong in sectors or cylinders; we want to
be a multiple of 4KiB for best performance.
The intent here is to create one single, Linux, bootable, partition
taking up the whole disk starting at 1MiB, so "2048 + L *" makes this
clear.
We use the -uS argument to ensure both versions treat this start-value
as a sector offset (newer sfdisk essentially ignores the argument).
As described in the comments, bugs in the older sfdisk necessitate
usage of "--force".
Although we could choose more or less, it seems most common to align
to a 1MiB boundary (i.e. starting at sector 2048). libguestfs has
some disucssion around --alignment and where it sets it's default to
this [3]. The 2.26-era sfdisk also defaults to putting partitions
here. 1MiB should be enough for GPT schemes in the future as well.
[1] https://github.com/karelzak/util-linux/blob/master/libfdisk/src/script.c#L1050
[2] https://bugzilla.redhat.com/show_bug.cgi?id=1249893
[3] http://libguestfs.org/virt-resize.1.html
Change-Id: I2c2966f98d1d5ad4ebb433ea148b3b26c65dc1b5
Using the default release for the Debian test-element leaves
us at the mercy of packaging problems in Debian unstable. Since
we only care if the image itself builds, use the safe choice of
Debian stable.
Change-Id: I4dd58171489aa07d60db7e4e353ccde6aa534da7
Splits the install.d into source-install and package-install in
order to allow building from a distro packaged version of
ironic-python-agent.
Change-Id: I17513c29efd8c199e07ac1ef20ea5d7456585413
