As described in pep282 [1], the variable part of a log message
should be passed in via parameter. In this case the parameters
are evaluated only when they need to be.
This patch fixes (unifies) this for DIB.
A check using pylint was added that this kind of passing parameters to
the logging subsystem is enforced in future. As a blueprint a similar
(stripped-down) approach from cinder [2] was used.
[1] https://www.python.org/dev/peps/pep-0282/
[2] https://github.com/openstack/cinder/blob/master/tox.ini
Change-Id: I2d7bcc863e4e9583d82d204438b3c781ac99824e
Signed-off-by: Andreas Florath <andreas@florath.net>
This is an initial creation of pylint with a basic indent checker.
Small issues corrected. Job added to gate with
Ib554a284e92583cc1d6a5c2219b3922852ca4c73
Change-Id: I7e24d8348db3aef79e1395d12692199a1f80161a
Co-Authored-By: Andreas Florath <andreas@florath.net>
This is consistent with how dpkg based images are configured
and minimizes the nodepool images drastically (avoid installing
texlive for example)
Change-Id: I98fb31bc0e06869e9770fae3dbd62f0d86acb879
This was suggested in a review comment in
I8a5d62a076a5a50597f2f1df3a8615afba6dadb2. It works out quite nicely
because the BlockDevice() driver now doesn't need to know anything
about stevedore or plugins, and just works on the node list. It also
simplifies the unit testing by not having to call create_graph through
a BlockDevice object.
Change-Id: I98512f6cf42e256d2ea8225a0b496d303bf357b8
This completes the transitions started in
Ic5a61365ef0132476b11bdbf1dd96885e91c3cb6
The new file plugin.py is the place to start with this change. The
abstract base classes PluginBase and NodeBase are heavily documented.
NodeBase essentially replaces Digraph.Node
The changes in level?/*.py make no functional changes, but are just
refactoring to implement the plugin and node classes consistently.
Additionally we have added asserts during parsing & generation to
ensure plugins are implemented PluginBase, and get_nodes() is always
returning NodeBase objects for the graph.
Change-Id: Ie648e9224749491260dea65d7e8b8151a6824b9c
This switches the code to use networkx for the digraph implementation.
Note that the old implementation specifically isn't removed in this
change -- for review clarity. It will be replaced by a base class
that defines things properly to the API described below.
Plugins return a node object with three functions
get_name() : return the unique name of this node
get_nodes() : return a list of nodes for insertion into the graph.
Usually this is just "self". Some special things like partitioning
add extra nodes at this point, however.
get_edges() : return a tuple of two lists; edges_from and edges_to
As you would expect the first is a list of node names that points to
us, and the second is a list of node names we point to. Usually
this is only populated as ([self.base],[]) -- i.e. our "base" node
points to us. Some plugins, such as mounting, create links both to
and from themselves, however.
Plugins have been updated, some test cases added (error cases
specifically)
Change-Id: Ic5a61365ef0132476b11bdbf1dd96885e91c3cb6
This moves to a more generic config parser that doesn't have plugins
parsing part of the tree.
I understand why it ended up that way; we have "partitions" key which
has special semantics compared to others keys and there was a desire
to keep it isolated from core tree->graph code. But this isn't really
isolated; you have to reverse-engineer several module-crossing
boundaries, extras classes and repetitive recursive functions.
Ultimately, plugins should have access to the node graph, but not
participate in configuration parsing. This way we ensure that plugins
can't invent new methods of configuration parsing.
Note: unit tests produce the same tree -> graph conversion as the old
method. i.e. this is not intended to have a functional change.
Change-Id: I8a5d62a076a5a50597f2f1df3a8615afba6dadb2
Add a range of unit-testing for configuration parsing, graph
generation and mount-point generation. Unfortunately there's some
global variable hacks, and some stubs, but it's a start.
Change-Id: I9e4f950c2c2ea656fc0c1a14594059fb4c62fa35
There's an increasing amount of pydoc based documentation. Output the
module reference and link it into the developers main page.
One fixup to the rst needed
Change-Id: I1d43a1fe1c735eb4559e3d2b40834d1c8115c586
This is a follow-on to 57ef187632.
There's two things going on here; DIB_MANIFEST_IMAGE_DIR is *outside*
the chroot on the build host. We copy the files here for posterity, I
guess. MANIFEST_IMAGE_PATH is *inside* the chroot and are the files
we want to ensure are locked to root.
The prior change modified the permissions on DIB_MANIFEST_IMAGE_DIR.
So the first time you build, it works -- then the second time,
assuming you're using the same output filename, it hits the root-owned
manifest directories and causes a build failure.
I have built with this and checked that the manifest files in the
image are locked to root:
$ virt-ls -a ./test.qcow2 -l /etc/dib-manifests
total 32
drwxr-xr-x 2 0 0 4096 May 24 03:39 .
drwxr-xr-x 53 0 0 4096 May 24 03:39 ..
-rw------- 1 0 0 15236 May 24 03:39 dib-manifest-dpkg-test
-rw------- 1 0 0 35 May 24 03:39 dib_arguments
-rw------- 1 0 0 137 May 24 03:39 dib_environment
Related-Bug: #1671842
Change-Id: I08319d0b5fcc461d40fe0be8427dcf0e37ad21e6
Move Partition() object creation into the actual Partition object,
rather than having the logic within the Partitioning() object
Change-Id: I833ed419a0fca38181a9e2db28e5af87500d8ba4
Split Partition() into it's own file for clarity. This will be
followed-on by less dependence between Partitions and Partition
Change-Id: I860f6a1787c0e4fe99f93919ac37cf7d80bfaae9
Moving the exception didn't cause problems in
I925ed62bdc808f0e07862f6e0905e80b50fbe942, but in later changes where
we split blockdevice.py up a bit more, we can get a bit tangled with
circular imports.
Change-Id: I8297483f64c4e1deecd5ec88ee40e9198bb83589
Because in some cases (e.g. partitioning) the order is needed,
add weights to the digraph to get an (somewhat) stable
topological sort.
Change-Id: I5ef1acc6338ac93c593faa0eafe26cbed42ed887
Signed-off-by: Andreas Florath <andreas@florath.net>
Per [1] this is the "official" CDN mirror, which I think is the most
appropriate for the default. I think this addresses the concerns
httpredir service, which I don't think ever quite got out of beta.
[1] https://wiki.debian.org/DebianGeoMirror
Change-Id: I55f2a00b8bbb0f0a20d3be229e4c2c32a7b69057
Instead, either use the bash built-in of type to ensure it exists. Since
which is an external dep, things can fail oddly in a constrained
environment.
Also add a dib-lint test for this.
Change-Id: I645029f5b5bfe1198c89ce10fd3246be8636e8af
Signed-off-by: Jesse Keating <omgjlk@us.ibm.com>
This new element will allow to regenerate dracut
on the produced images, to enable different modules. It
relies on a yaml blob to specify modules and packages
needed. It defaults to installing lvm and crypt.
Change-Id: I292fb70cde41ee6053b7b81a67931bcdaaa6d664
"log and throw" is arguably an anti-pattern; the error message either
bubles-up into the exception, or the handler figures it out. We have
an example where this logs, and then the handler in blockdevice.py
catches it and logs it again.
Less layers is better; just raise the exception, and use log.exception
to get tracebacks where handled.
Change-Id: I8efd94fbe52a3911253753f447afdb7565849185
Manifests files can release sensitive information and therefore should
have restrictive permissions.
Change-Id: I64d6c830217a7d8b0172df2dc774079dcd1e2a68
Related-Bug: #1671842
A majority of the "plugins" aren't implementing the plugin class.
Clearly we need some refactoring of the ideas here. Remove for
simplicity.
Change-Id: If399a371b171f4fd17cfa5856fe55daca4c86e60
To avoid failures with double unmount, skip unmounting
the mountpoints that are managed by block device.
Change-Id: I228779eb9bf544a27a53e5017c87573023fd375a
With new block device definition, where content of the image
can be mounted on different partitions, is not enough with
executing setfiles on root directory. Instead of that, expose
all the mountpoints on the image, and apply setfiles on them.
Change-Id: I153f979722eaec49eab93d7cd398c5589b9bfc44
This patch finalizes the block device refactoring. It moves the three
remaining levels (filesystem creation, mount and fstab handling) into
the new python module.
Now it is possible to use any number of disk images, any number of
partitions and used them mounted to different directories.
Notes:
* unmount_dir : modified to only unmount the subdirs mounted by
mount_proc_sys_dev(). dib-block-device unmounts
$TMP_MOUNT_PATH/mnt (see I85e01f3898d3c043071de5fad82307cb091a64a9)
Change-Id: I592c0b1329409307197460cfa8fd69798013f1f8
Signed-off-by: Andreas Florath <andreas@florath.net>
Closes-Bug: #1664924
The args agument was only used to find the symbol for the getval
command. Have the command pass the symbol to find in directly. We
can therefore remove the args paramater to the BlockDevice() creation.
Change-Id: I8e357131b70a00e4a2c4792c009f6058d1d5ae9e
Move argument parsing to subparsers, rather than positional arguments.
This better reflects the tool's role as a driver and allows
sub-commands to deal with arguments in a natural way.
Change-Id: Iae8c368e0f3fe47abfddb9e0a1558bd5b3423aee
I accidentally dropped the clearing of this file when it moved to
cmd.py during rebase of I1919f6e865acae14ee95cd025c9c7b75ca266a9c
Change-Id: Ibe9fcde594770cb51c732cc253987308dc038083
DIB_BLOCK_DEVICE_PARAMS_YAML should be exported, and the
dib-block-device will take this as the value of --params. Remove this
to simplify the command-line
Change-Id: I6764ed223ecd36f9d24e19f164b6a927380b410f
This creates a BlockDeviceCmd object to hold the main() function.
This doesn't really do anything different right now, but sets a base
for using argparse subparsers to handle the command-line
Change-Id: I4acf95ff4d554a3b4e7e2244ab1706631b98458f
This moves the YAML parameter parsing into the command-line driver.
It makes the argument optional so it can be taken from the environment
variable directly. The parsed YAML is passed to the BlockDevice
object.
Change-Id: I6fa5e5b7d1fccfc7cf47d6e4a1fa6e560734680d
To avoid any confusion, commands passed to exec_sudo() should be a
list of "str"s. Log a message if we see unicode issues.
This also adds a debug trace of all output. stderr is captured.
This is modified to raise CalledProcessError on failure, like
check_call(). Calls that are ok to fail will need to explicitly catch
and ignore this.
The two calls that we expect to fail are wrapped
We wish to try rolling back if one of these command raises an
exception. Modify the create handler to initiate rollback on all
exceptions.
Change-Id: Iee4fa41ffaf243e4728bf3a5eeec5c8fa8d2dadc
The await function is essentially a non-standard check_output call.
Let's use standard calls to increase maintainability.
Change-Id: I2c25e1cd7122791fcaa86b46bd801e661471bc9e
As this method can be introduced without any dependency,
provide it on an independent change to simplify reviews.
This is a partial refactor based on
I592c0b1329409307197460cfa8fd69798013f1f8
Change-Id: Idaf3d2b3b3e23d0b9d6bc071d67b961a829ae422
Co-Authored-By: Andreas Florath <andreas@florath.net>
This is a partial refactor from change
I592c0b1329409307197460cfa8fd69798013f1f8
Change-Id: I8822e68e41c4ebd47eea9ffed4557efc130a7bf7
Co-Authored-By: Andreas Florath <andreas@florath.net>
Add a new method in the block device library called
exec_sudo, so it can be reused.
This is a partial refactor of change
I592c0b1329409307197460cfa8fd69798013f1f8
Change-Id: Id621f6d029e1275a35c4fd3f19b57c8518076134
Co-Authored-By: Andreas Florath <andreas@florath.net>
As part of the final steps, refactor the bits belonging
to block device and functions. This is a partial refactor
from I3600c6a3d663c697b59d91bd3fbb5e408af345e4
Change-Id: I7aa4fe0466e44846d8fa3194575d446fe4b5b2e6
Co-Authored-By: Andreas Florath <andreas@florath.net>
Introducing the refactors of the block device to allow a tree-like
configuration, and start using it for the partitions level.
Based on patch I3600c6a3d663c697b59d91bd3fbb5e408af345e4
Change-Id: I58bb3c256a1dfd100d29266571c333c2d43334f7
Co-Authored-By: Andreas Florath <andreas@florath.net>
It seems that the redhat nodepool job is quite reliably geting a
"floating point" error during centos image build. This happens after
03-yum-cleanup which is pruning the locales. This might be a
red-herring, since the logs are full of
/bin/bash: warning: setlocale: LC_ALL: cannot change locale (C.UTF-8)
I think in our recent de-puppetisation of hosts, something might have
changed that is setting LC_ALL=C.UTF-8 for the jenkins user, at least
on Ubuntu. This is a problem for centos, as it doesn't have C.UTF-8
locale. I then think using the invalid locale is what leads the the
floating-point error when doing some maths in dib-run-parts to
calculate runtimes.
We are currently overriding LANG, but we really want LC_ALL to ensure
this applies globally.
Change-Id: I8e7cae093c4b32e0d20b73ae0086f14c7cc6a9cb
Add a new getval call that allows to retrieve values
from the block device. Also isolating the block device
information into a 'blockdev' dictionary entry, to better
return it with the getval command.
This is a refactor from the original code at
I3600c6a3d663c697b59d91bd3fbb5e408af345e4.
Change-Id: I93d33669a3a0ae644eab9f9b955bb5a9470cadeb
Co-Authored-By: Andreas Florath <andreas@florath.net>
The original approach was to pass each and every command
line parameter to the block device. While the block device
functionality gets extended, this is not any longer practical.
Instead of passing in all the parameters separately this patch
collects these in a YAML file that is passed in to the block device
layer.
Change-Id: I9d07593a01441b62632234468ac25a982cf1a9f0
Signed-off-by: Andreas Florath <andreas@florath.net>
Some package updates are more complex and require things like --backtrack=99 to
be passed to emerge. We also try harder to ensure the system is in a consistent
state as a last step.
Change-Id: Ia5d3514e8b2a6cb2d656ade997cebb798d9c0a47
With 8e822768f9 we added the ability to
disable the EPEL repository, however we need yum-utils to use
yum-config-manager.
Change-Id: Iea445f84494fd9a89fd93e9b35f920eb5e55211d
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
Recent changes in the default configuration of cloud-init in Ubuntu
cause warnings when the Ec2 datasource is used on non-Amazon clouds,
see https://bugs.launchpad.net/cloud-init/+bug/1660385
We explicitly select the previous behavior when an Ec2 datasource is
desired.
Change-Id: Iebad8f6c0017fe08013dd5fe667c6132158b71cd
Closes-bug: 1683038
If DIB_PYTHON_VERSION is < 3 on the !redhat path, that means we're on
an older platform that may not have python3-virtualenv packages. Skip
install.
Ensure the order of operations happens by forcing the installs
Also add a note about limited platform support (patches welcome :)
Change-Id: I18412767f0ebf946d557a0a126285369e96af159
Recent changes in project-config have shown that we leave the system
in an inconsistent state when installing from source. On fedora, we
will have installed the python2 packages, but then used $DIB_PYTHON to
install python3 pip from source!
This tries to clarify the situation. As described in the document,
with package installs, we just install the $DIB_PYTHON packaged
versions.
Source installs want to take over the global namespace. This is the
price you pay for running the latest versions outside package managers
:) The only sane thing seems to be for us to normalise python2 &
python3 versions of pip, setuptools and virtualenv and then hacking
things such that "/usr/bin/pip" and "/usr/bin/virtalenv" remain
defaulted to python2 versions.
Documentation is added
Change-Id: Ibc6572b89e256d1f48b7fe7c672b8b9524dc704f
Currently we install pip/virtualenv with "/usr/local/bin/dib-python".
This means that every time you create a virtualenv, the python
interpreter inside it is called "dib-python" which is confusing.
Add an env var DIB_PYTHON that points directly the to interpreter
available during build, for use when running scripts.
Change-Id: I88ad3c9eb958d58db4631d9b27bc2c592f970345
This change move "do_extra_package_install" from pre-install to install
phase.
Extra packages are added by user request using the flag "-p", This
package should not be something the elements depend on.
The reason behind this patch is to move the extra package install to
a proper phase, Also more reasonable if base element run package update
to be before we install extra packages.
Change-Id: I68cc773aba9aa01743f0dda9f4e635e4cac2a282
Apt gets confused if it talks to a mirror with an older index than the
index currently cached by apt. This can happen when image builds use a
newer index than the booted image. Avoid these problems entirely by
removing those index caches at the end of image building.
Change-Id: I245d516ee8a44831b2c29612b782bad555c48a3f
Co-Author: Clark Boylan <clark.boylan@gmail.com>
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
This patch removes three nearly-copies of debootstrap documentation
and fixes some documentation aspects.
Change-Id: Ief7794f5c1abad73788c063af6c862472cd34744
Signed-off-by: Andreas Florath <andreas@florath.net>
The current output for package-installs-v2 is inscrutable [1]
The problem starts with process_output() which is not capturing
stderr. This means that any stderr output is dislocated from any
stdout output around it. This is *really* confusing as you get a
bunch of seemingly meaningless stderr output from any calls before you
see any stdout (e.g. in [1] you can see random yum error output that
should have been with the yum call)). The simplest thing to do is to
redirect stderr to stdout which keeps everything in sync.
This causes a slight problem, however, because pkg-map outputs both
status information and errors on stderr. To work around this but
maintain compatibility, we add a "--prefix" argument that prepends
mapped packages from pkg-map with a value we can match on. The
existing status/debug output from pkg-map is low-value; modify the
call so that it will be traced only at higher debug levels (e.g. -x
-x).
The current loop is also calling pkg-map for every package in every
element (this is why in [1] the same message is repeated over and
over). This is unnecessary; it only needs to pkg-map once for each
element, giving the package list as the arguments. Create package
lists by element and pass those to pkg-map.
As a cleanup, there is no point in printing e.output if the
process_output fails for the install because we are already tracing
it; i.e. the output, even for failures, is already in the logs.
Printing it again just duplicates the output.
[2] is an extract showing what I feel is a much more understandable
log output for a fairly complex install.
[1] http://paste.openstack.org/show/595118/
[2] http://paste.openstack.org/show/595303/
Change-Id: Ia74602a5d2db032a476481caec0e45dab013d54f
DIB_PYTHON_EXEC was added in I5fab0e192c3a2dad8f60e821c184479e24e33bcd
to export the python that disk-image-create is running under. If dib
is installed with python3 then just calls "python" (python2) to run
sub-scripts, it fails to find itself.
This has been tested in devstack gates that use py3x
Change-Id: Ia1028972bfc0517b468b279aab9decdbcd7424ca
If the path is missing, unmount_dir currently exits with an error which
unintentionally aborts cleanup efforts early. This change makes
unmount_dir idempotent by exiting successfully if a directory doesn't
exist.
Change-Id: I1491b4344e8569ecb2833f44baee445a89a39d61
The dib-run-parts element was copying our internal version of
dib-run-parts into /usr/local/bin to be used running scripts inside
the target chroot. However, it never cleaned up after itself. This
means all images were left with an unmanaged local install of
dib-run-parts.
This copies dib-run-parts into the hooks directory of the chroot and
runs it from there. It is cleaned up automatically on the exit path.
The dib-run-parts element is no longer required and it has been
removed from all dependencies. It is left with a deprecation notice
in the README. For compatability we convert it to simply install
dib-utils.
Codesearch shows no users depending on this unintentional implicit
install. Note os-refresh-config depends on dib-utils and thus will
have an explicitly installed version.
Partial-Bug: #1673144
Change-Id: Ia2e96c00a4246c04beb96c17f83b8aefb69219ca
It was an oversight during v2 development for dib to start providing
dib-run-parts. The intention was for dib to use a vendored
dib-run-parts directly from $_LIB and have no dependencies on
dib-utils at all. By exporting dib-run-parts, we created an
unintentional conflict with the dib-utils package which provides the
same script.
Tools that depend on dib-utils are unaffected by this
(os-refresh-config).
The only tool that installs diskimage-builder and then assumes
dib-run-parts is available in the path is instack. I have proposed
Ibfe972208df40fa092b11b5419043524c903f1b4 to modify that to use our
internal version.
Change-Id: I149c345d38d761a49b3a6ccc4833482f09f1cd05
Add DIB_EPEL_DISABLED flag that allows installation of the EPEL repo,
but to have it disabled by default. This will help when you have
unavoidable EPEL dependencies, but want to make sure you only pull
specific things in with "--enablerepo" calls when installing those
packages.
Change-Id: Iedf6167a7cd69418255ebbee095aea04c50d73fd
A code-block in README of rhel7 element is not rendered as expected.
This patch fixes it to be rendered correctly.
Change-Id: Ie8f4c05edd1dd93314290682e4b2734622894e15
zypper on non-suse hosts is not parsing the pattern repodata because
those are marked as an inofficial extension to the repomd specification.
This is not a big issue as there is meanwhile in newer openSUSE
distributions a pattern *package* that depends on the same packages like
the pattern would do, so we can just replace it with that.
Change-Id: I0c8f713075bd7e5bf1d425f81933b4666654add7
Depends-On: I34e98f0f7693859ed05011b008334628adff612f
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
When a glean is running on centos with multiple NICs, it will try to
systemctl enable network.service multiple times for each interface.
Because of systemd magic, it is possible for the systemctl command to
fail in a race condition.
glean shouldn't be enabling network.service during boot in
pre-networking phases (Ib2b618dd975ca44e9c6b0a2c9027642ffc46b9b0). I
have proposed I8319f1ed6498a9d447950c2b4b34bca59e7b97e4 to remove this
and document the behaviour.
This also bring across suse's version
(I20bffabd333ea290d8712ec2a467f2b2d5678f3a)
Change-Id: I89d9443cb61e287bd0d9da3f48315272218ee335
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
This patch introduces stevedore plugin mechanism for use
with the block device layer. This makes it possible that
other projects pass in their own block device plugins.
Change-Id: Id3ea56aaf75f5a20a4e1b6ac2a68adb12c56b574
Signed-off-by: Andreas Florath <andreas@florath.net>
diskimage-builder usually provides defaults that work out of the box.
One default that does not work outside of x86 land is Ubuntu distro
mirror url. Considering there are only two valid default options, we can
automatically choose a better default.
This patch changes behavior only for architectures known to be using
http://ports.ubuntu.com/ubuntu-ports. All others still would use
http://archive.ubuntu.com/ubuntu as default. It provides some guarantee
that we do not introduce a regression.
Change-Id: If95a64bac0c88f30736da4bae7f1fdce126c0bf6
We somewhat discussed skipping qcow2 generation previously in
I9372e195913798a851c96e62eee89029e067baa1. As recent issues with PPC
testing have shown, we are not actually testing the "vm" element and
hence the bootloader path in the functional tests.
I don't think we need to test this on every element; it overlaps
somewhat with the testing done by the nodepool jobs which build full
images and boot them. I also didn't want to introduce a separate run
for this. Thus it seems valuable to at least have one element
enhanced to do this installation and conversion in our default tests
for basic sanity.
This disables qcow generation by default, as per the other change, but
allows an element to drop a file that will override the output
formats. The Xenial element is modified to produce a qcow2 using
this, and also introduces a dependency on the "vm" element so it tries
to install the bootloader.
We now exit if the .qcow2 fails to build as well.
Change-Id: I1a6acefe52f8c696c39b2d592fdc7ae32a87e6fe
Add a default PPC block-device layout. I've extracted this into
separate yaml files for ease of editing and to facilitate things like
longer comments.
This is not sufficient to get PPC images working, but it is required.
Change-Id: I09e5d1ed92260bdb632333f5203dd7e70d512dc8
The stdout of the script is captured, so anything coming out from
these commands needs to be captured. Move to check_process and show
the output as part of an error log in failure case.
Change-Id: I1150375cdc479d4f19b8ddeb49a824ab16fdf831
Sphix 1.5 (I9e7261c4124b71eeb6bddd9e21747b61bbdc16fa) includes
"warning-is-error" which supersedes pbr's warnerrors. Enable this and
fix up the resulting failures
- trailing lines for lists in element_deps directive
- missing README's that are linked
- syntax error and highlighting in building instructions
Change-Id: I6549551b4a9bf47076c9811a7a38a666cbea2a50
99-squash-package-install in the package-installs element does not
know which python environments the requirements were installed into.
This can cause it to select the wrong python to run the
package-installs-squash script.
Co-Authored-By: Adam Harwell <flux.adam@gmail.com>
Change-Id: I5fab0e192c3a2dad8f60e821c184479e24e33bcd
execfile() has gone in python3, and if you google various
stackoverflow results, python-dev mailing list threads and other
projects it seems runpy is considered one of the better solutions.
This should be py2.7 safe too.
Change-Id: I18077ba9d603752492cc81f260e12710981f4dff
On Debian Jessie and Debian Stretch systemctl is in /bin.
If the package systemd-sysv is not installed the script
dib-init-system did not find the init system.
This patch fixes the problem: it also looks in /bin
for systemctl and if found decides for systemd.
Change-Id: I5a18052a070bad5e16b14672237a1e2b38513949
Signed-off-by: Andreas Florath <andreas@florath.net>
The architecture-emulation-binaries element does for a limited number
of architectures on Ubuntu exactly the same as `qemu-debootstrap`.
`qemu-debootstrap` comes in the qemu-user-static package that is
anyhow used when cross-building an image.
This patch replaces the complete architecture-emulation-binaries
element with a call to qemu-debootstrap.
Change-Id: Ib9667307bfd3ff7592444a2ec5b04aa5365a1872
Signed-off-by: Andreas Florath <andreas@florath.net>
Depending on the types of deployment (security, nfv...) some extra
kernel flags are needed on the images. This change exposes the
DIB_BOOTLOADER_DEFAULT_CMDLINE parameter, defaulting to the
existing 'nofb nomodeset vga=normal', that will allow to modify
these boot params.
Change-Id: I67d191fa5ca44a57f776cb9739a02dd71212969c
Closes-Bug: #1668890
The order of the partitions is important, it needs to be preserved.
If using a simple dict, this is not happening. As a consequence,
checks like 'primary partition being first' are failing because the
dictionary sorts the partitions randomly.
Switched to OrderedDict solved the problem, as it preserves the
ordering it gets from the yaml blob.
Change-Id: Icfa9bd95ffd0203d7c3f6af95de3a6f848c2a954
Now that the main partitioning refactor patch is merged, there is
a small relict of handling partitions still in the disk-image-create
main.
This patch moves the functionality from disk-image-create to the
block-device/partitioning module: it is mostly a rewrite of the
original bash code in python.
Change-Id: Ia73baeca74180a7bc9ea487da03ff56d6a3070ce
Signed-off-by: Andreas Florath <andreas@florath.net>
Looks that the special handling for Ubuntu is not needed any longer
(its a pity that there are no detailed comments...).
The grub2 element is a second implementation of the bootstrap element
- but because there are some features that come only here, e.g. efi
boot, it should be working as long as this is not implemented in the
bootloader element.
Change-Id: I74269116ea30b84f3259805720d5cd1616f960c5
Signed-off-by: Andreas Florath <andreas@florath.net>
Closes-Bug: #1627402
Currently there is no description of dependencies in the generated
documentation of the elements: therefore a user of an element does not
know which other elements are automatically included and e.g. which
configuration options are available. In addition there are some
copy&pastes of parts of the README.rst scattered thought different
Ubuntu and Debian specific elements.
This patch adds a semi-automatic generation of dependency information
of all elements. Nevertheless these are not automatically included.
The author of the element's README.rst can decide if and where the
dependency information should appear and can use the descriptor
.. element_deps::
for this.
This patch adds the dependency information for some Debian and
Ubuntu patches - and creates the base for later removing the
duplicated parts.
A call is added to element_dependencies._find_all_elements() to
populate reverse dependencies for Element objects.
(This is a reworking of I31d2b6050b6c46fefe37378698e9a330025db430 for
the feature/v2 branch)
Change-Id: Iebb83916fed71565071246baa550849eef40560b
In error scenarios there might be no loadable state available.
This patch adds some robustness handling this scenarios
by checking the result and possible bailing out.
Change-Id: I02e2731b14bec22c22db08d60d8d15db1911a84e
Signed-off-by: Andreas Florath <andreas@florath.net>
With the old configuration structure it was only possible
to use one image and one partition layout. The new
block-device configuration uses a list at top level;
therefore it is possible to use multiple instances
of each element type.
Change-Id: I9db4327486b676887d6ce09609994116dbebfc89
Signed-off-by: Andreas Florath <andreas@florath.net>
When merging feature/v2 we have re-enabled the centos-minimal test
which has a blank line in the element-deps. The modified get_elements
functions added in If97bcd21e45de1b5ed91023fdc441a4617051a6b were not
handling this.
Fixing this, we can extract this into some common code for
element-deps and element-provides.
Also, I noticed while debugging that the element list is an iterator
from reverse(), so printing it doesn't help. Fix up the debug output.
Change-Id: I8e9b079f586e8a36dba3797a0fc03ca1b2041a04
During the creation of a disk image (e.g. for a VM), there is the need
to create, setup, configure and afterwards detach some kind of storage
where the newly installed OS can be copied to or directly installed
in.
This patch implements partitioning handling.
Change-Id: I0ca6a4ae3a2684d473b44e5f332ee4225ee30f8c
Signed-off-by: Andreas Florath <andreas@florath.net>
expand_dependencies() was a public interface so we should try and
preserve backwards compat. However, since the interface is really
broken, add a new exported function "get_elements" that instack can
switch to. This returns the canonical list of elements without
duplicates, and gives the path to each element too.
This highlighted that the unit tests were really a bit wrong. They're
testing inner functions when we have an "API" in the get_elements()
function. Convert all unit-tests to use this function instead. Since
this is a library call, convert the sys.exit() calls to raised
exceptions.
Refactor the variable output into a separate function so we can do a
sanity check on it.
The added flake8 ignores are for the "over-indented for ... indent"
which happens a lot with these new longer lines. Most other projects
ignore them.
This is an alternative proposal to
I15609389c18adf3017220fc94552514d195b323a
Change-Id: If97bcd21e45de1b5ed91023fdc441a4617051a6b
We do not have the concept of "not installed" in v2, so remove the
obsolete code looking for a now non-existent variable.
Also, log the version at startup. This can help when
debugging from logs
Change-Id: I964c4cf207c10666afc5bc7ab9f2bfb9b1897c1e
Because we're still fundamentally a python program calling a
shell-script, there's some oddities like not having the virtualenv
bin/ in the $PATH if we call disk-image-create directly.
We can detect this, however, and activate the virtualenv before we
fork the disk-image-create shell script so everything "just works".
See also nodepool change I0537cbf167bb18edf26f84ac269cbd9c8a1ea6a2
Change-Id: Ibfea6cf6a6fd0c7f1e468d501c61ae0b58992042
Remove the x bit from lib/disk-image-create; because it's called
directly by the entry-point, it doesn't need to be exectuable.
This should also be clearer that you're not supposed to run it
by hand.
Remove some boilerplate from old file
Change-Id: Ibb6cdae613e6c9cf21dd6aecc8e1f739bc3a2643
Move dib-run-parts from dib-utils into diskimage-builder directly.
For calling outside the chroot, we provide a standard entry-point
script. However, as noted in the warning comment, the underlying
script is still copied directly into the chroot by the dib-run-parts
element. I believe this to be the KISS approach.
This removes the dependency on dib-utils. We have discussed this
previously and nobody seemed to think retiring dib-utils was going to
be an issue.
This also updates the documentation to not mention dib-utils, or using
disk-image-create via $PATH setup, but rather gives instructions on
installing from pip with a virtualenv.
Change-Id: Ic1e22ba498d2c368da7d72e2e2b70ff34324feb8
It has always been a weird thing that dib is a python package, but
is totally driven by the disk-image-create script. It creates this
strange division that is hard to explain.
This moves disk-image-create to a regular python entry-point
Currently, this simply exec()s the original disk-image-create script.
However, we now have a (private) interface between disk-image-create
written in python and the driver shell script. Here's some things we
could do, for example:
* Argument parsing is generally nicer in Python, and then end result
is mostly just setting environment variables to flag different things
in the shell script. I could see us moving the argument-parsing into
diskimage_builder.disk_image_create:main() and just setting things in
os.environ before the exec()).
* I7092e1845942f249175933d67ab121188f3511fd sets IMAGE_ELEMENT_YAML in
disk-image-create by calling-back to element-info. We can just call
element_dependencies.find_all_elements() in here an export is to
os.environ before disk-image-create starts.
* remove need for ramdisk-image-create symlink by just exporting
IS_RAMDISK based on sys.argv[1] value
* you could even unit test some of this :)
Change-Id: I69ca3d26fede0506a6353c077c69f735c8d84d28
Currently we have all our elements and library files in a top-level
directory and install them into
<root>/share/diskimage-builder/[elements|lib] (where root is either /
or the root of a virtualenv).
The problem with this is that editable/development installs (pip -e)
do *not* install data_files. Thus we have no canonical location to
look for elements -- leading to the various odd things we do such as a
whole bunch of guessing at the top of disk-image-create and having a
special test-loader in tests/test_elements.py so we can run python
unit tests on those elements that have it.
data_files is really the wrong thing to use for what are essentially
assets of the program. data_files install works well for things like
config-files, init.d files or dropping documentation files.
By moving the elements under the diskimage_builder package, we always
know where they are relative to where we import from. In fact,
pkg_resources has an api for this which we wrap in the new
diskimage_builder/paths.py helper [1].
We use this helper to find the correct path in the couple of places we
need to find the base-elements dir, and for the paths to import the
library shell functions.
Elements such as svc-map and pkg-map include python unit-tests, which
we do not need tests/test_elements.py to special-case load any more.
They just get found automatically by the normal subunit loader.
I have a follow-on change (I69ca3d26fede0506a6353c077c69f735c8d84d28)
to move disk-image-create to a regular python entry-point.
Unfortunately, this has to move to work with setuptools. You'd think
a symlink under diskimage_builder/[elements|lib] would work, but it
doesn't.
[1] this API handles stuff like getting files out of .zip archive
modules, which we don't do. Essentially for us it's returning
__file__.
Change-Id: I5e3e3c97f385b1a4ff2031a161a55b231895df5b
Move element-info from a wrapper script to a standard entry-point
console_script.
Update the documentation to explain how to run it for development. I
don't think we should support the idea that you can check-out the code
and run ./bin/disk-image-create -- it has dependencies (dib-utils,
etc) and needs to be run from a virtualenv (this is what CI in the
gate does). A follow-up can clean-up some of the path munging stuff
we have for this in disk-image-create.
Change-Id: Ic0c03995667f320a27ac30441279f3e6abb6bca8
