Currently we have all our elements and library files in a top-level
directory and install them into
<root>/share/diskimage-builder/[elements|lib] (where root is either /
or the root of a virtualenv).
The problem with this is that editable/development installs (pip -e)
do *not* install data_files. Thus we have no canonical location to
look for elements -- leading to the various odd things we do such as a
whole bunch of guessing at the top of disk-image-create and having a
special test-loader in tests/test_elements.py so we can run python
unit tests on those elements that have it.
data_files is really the wrong thing to use for what are essentially
assets of the program. data_files install works well for things like
config-files, init.d files or dropping documentation files.
By moving the elements under the diskimage_builder package, we always
know where they are relative to where we import from. In fact,
pkg_resources has an api for this which we wrap in the new
diskimage_builder/paths.py helper [1].
We use this helper to find the correct path in the couple of places we
need to find the base-elements dir, and for the paths to import the
library shell functions.
Elements such as svc-map and pkg-map include python unit-tests, which
we do not need tests/test_elements.py to special-case load any more.
They just get found automatically by the normal subunit loader.
I have a follow-on change (I69ca3d26fede0506a6353c077c69f735c8d84d28)
to move disk-image-create to a regular python entry-point.
Unfortunately, this has to move to work with setuptools. You'd think
a symlink under diskimage_builder/[elements|lib] would work, but it
doesn't.
[1] this API handles stuff like getting files out of .zip archive
modules, which we don't do. Essentially for us it's returning
__file__.
Change-Id: I5e3e3c97f385b1a4ff2031a161a55b231895df5b
It's possible this is run form an environment where $USER isn't set,
properly fallback to whoami in this case.
Change-Id: I1181f714c3c456ee264b34d282bac5c0adb67a0e
While we already clean a number of things off the ironic-agent
ramdisk, there are a few more significant ones that we should add
to the list.
First is the kernel source. If you're rebuilding your kernel on
the agent ramdisk after the initial image build, then you need to
re-examine your life choices. ;-)
Second is /var/cache. On yum-based distros, this contains a large
number of yum cache files that take up significant space. We don't
really want to be copying around caches when booting a ramdisk
anyway, so cleaning this is the right thing to do regardless.
Third is all *.pyc or *.pyo files. There are a lot of these, so
they eat up significant space and bloat the number of files in the
ramdisk, which makes it take longer to build. the only purpose for
the files is to slightly speed up Python app startup, and we
probably lose more time transferring the files over the network
than we would gain in quicker start times. Note that we were
already trying to remove these, but for some reason I was still
seeing them show up in my final images. It makes more sense to
put them in the same pruning command as all the others anyway.
Fourth is /usr/include. These are files only needed for
compilation. See above for my thoughts on compiling in a ramdisk.
These changes have reduced the agent ramdisk from 391 MB to 333 MB
in my local centos 7 builds, and have reduced the number of files
in the ramdisk by over 18000.
Change-Id: I550f9904b9afd12d48da9ba24559acb23133d076
The initramfs file created by the ironic-agent element is
owned by the user running disk-image-create; ensure that
the other files created by the element are also owned by
the user.
Change-Id: I829db5b8e8bf1fc68face9cd2bda52d2a5ccdd4f
Closes-Bug: 1593010
As motivation for this; we have had two breakouts of dib in recent
memory. One was a failure to unmount through symlinks in the core
code (I335316019ef948758392b03e91f9869102a472b9) and the other was
removing host keys on the build-system
(Ib01d71ff9415a0ae04d963f6e380aab9ac2260ce).
For the most part, dib runs unprivileged. Bits of the core code are
hopefully well tested (modulo bugs like the first one!). We give free
reign inside the chroot (although there is still some potential there
for adverse external affects via bind mounts). Where we could be a
bit safer (and could have prevented at least the second of these
breakouts) is with some better checking that the "sudo" calls
*outside* the chroot at least looked sane.
This adds a basic check that we're using chroot or image paths when
calling sudo in those parts of elements that run *outside* the chroot.
Various files are updated to accomodate this check; mostly by just
ignoring it for existing code (I have not audited these calls).
Nobody is pretending this type of checking makes dib magically safe,
or removes the issues with it needing to do things as root during the
build. But this can help find egregious errors like the key removal.
Change-Id: I161a5aea1d29dcdc7236f70d372c53246ec73749
This patch add the lshw utils to allow the agent
to retrive the vendor name. It also keeps the
/usr/share/misc/pci.ids file for Ubuntu distro.
Change-Id: I2828a7e585449f63887ad5a2e338b4e043d68e63
Previously all files in /root were ignored when building the
ironic-agent ramdisk. This prevented for example to use the
local-config element to connect to the ramdisk via ssh as root user.
This commit change the exclude rule on /root to only ignore the
/root/.cache directory.
Change-Id: I18d839e8d97636f5f2164ba407f252407d9bc956
Closes-Bug: #1451668
Find doesn't like listings disappearing while its trying to find them,
in this case if a PID directory disappears while find is running. Using
-xdev prevents find from going into ./proc and as a side effect /dev
will also be avoided which is mounted on boot so not needed either.
Change-Id: Iaa282e58d81d533ad4445da0a44200dd14bf0850
Closes-bug: #1502142
Traversing the /proc filesystem causes find to error if it changes
while its being searched.
We have had a lot of ci failures on this find command since it was
added in Ibe40e6b8b884f37e3b5aeab6e7654593bcd63123
Change-Id: Ia8cfc923cce749a69d5108e588db2360238d866c
Closes-Bug: #1501949
Otherwise on rebuilding the agent the following error is produced:
failed to create hard link /home/stack/ironic-agent.vmlinuz
and the vmlinuz file is not updated.
Change-Id: I2015da889c932a854727235b1e34256a28e9eac6
When a ubuntu/IPA ramdisk is used to boot a baremetal machine with
ironic agent-ilo driver, it fails at the point of mounting /proc
and /sys. After the vmlinuz(kernel) is started and it tries to
load the partitions on ramdisk. It need the directory of "/sys"
and "/proc" to mount the corresponding filesystems.
In order to fix this issue, the directories of "sys" and "proc"
are retained but the subdirectories or files under them are empty.
With this change, the directories of "/sys" and "/proc" shows
up in the ramdisk and kernel will mount sys and proc filesystems
on them respectively.
Closes-Bug: #1488445
Change-Id: Iad5d62f373b73789118f23db4c932ea6e9a784c3
Signed-off-by: Gary Duan <duanlg@live.cn>
The ironic-agent element is created using the disk-image-create utility
(even being a ramdisk) and outputs a .vmlinuz file for the kernel
(different than the ramdisk-image-create which outputs a .kernel file
for the kernel). This is inconsistent and make scripting against the
diskimage-builder more complicated if one wants to support different
types of ramdisk.
This patch creates a hard link for the .vmlinuz file to a .kernel file
at the end of the process (to keep backward compatibility) and print a
deprecation message.
Depends-On: I81400305f166d62aa4612aab54602abb8178b64c
Change-Id: I476f9ec9ec4206ece0261eaaf2b4182c6bcbd802
Closes-Bug: #1482606
This patch is reducing the size of the ramdisk image generated by the
ironic-agent element. It does remove extra packages (graphical stuff,
dev stuff, miscs, docs, etc...) and purges directories that are not
needed for a ramdisk (like /boot since it boots using an external
kernel)
Currently it was tested generating a Fedora 22 image and reduced the
size of the final image from 464 MB to 211MB compacted (54% decrease).
I was able to boot a VM with 1.3 GiB of ram instead of the previous 3 GiB
needed.
Change-Id: Id6333ca5d99716ccad75ea1964896acf371fa72a
/tmp does not contain anything useful anyway, and excluding its content
makes the initramfs smaller too.
Change-Id: Ia72867e0cdebacf668ac1a1f551a965da0d69694
There is a wide variety of tracing options through the various shell
scripts. Some use "set -eux", others explicity set xtrace and others
do nothing. There is a "-x" option to bin/disk-image-create but it
doesn't flow down to the many scripts it calls.
This adds a global integer variable set by disk-image-create
DIB_DEBUG_TRACE. All scripts have a stanza added to detect this and
turn on tracing. Any other tracing methods are rolled into this. So
the standard header is
---
if [ "${DIB_DEBUG_TRACE:-0}" -gt 0 ]; then
set -x
fi
set -eu
set -o pipefail
---
Multiple -x options can be specified to dib-create-image, which
increases the value of DIB_DEBUG_TRACE. If script authors feel their
script should only trace at higher levels, they should modify the
"-gt" value. If they feel it should trace by default, they can modify
the default value also.
Changes to pachset 16 : scripts which currently trace themselves by
default have retained this behaviour with DIB_DEBUG_TRACE defaulting
to "1". This was done by running [1] on patch set 15. See the thread
beginning at [2]
dib-lint is also updated to look for the variable being matched.
[1] https://gist.github.com/ianw/71bbda9e6acc74ccd0fd
[2] http://lists.openstack.org/pipermail/openstack-dev/2014-November/051575.html
Change-Id: I6c5a962260741dcf6f89da9a33b96372a719b7b0
The element was prepending the work directory forcibly, which will have
problems when the image name has an absolute path, which is allowed.
Change-Id: I0cb7b96e24daab8ee73611936af72074c70ac1aa
Closes-Bug: #1400405
This element creates kernel and ramdisk files based on Fedora,
example:
disk-image-create -a i386 -o test fedora ironic-agent disable-selinux
Change-Id: Ifa133d1680b81cb87d32a405aa7d7b40fe91f835
