This was suggested in a review comment in
I8a5d62a076a5a50597f2f1df3a8615afba6dadb2. It works out quite nicely
because the BlockDevice() driver now doesn't need to know anything
about stevedore or plugins, and just works on the node list. It also
simplifies the unit testing by not having to call create_graph through
a BlockDevice object.
Change-Id: I98512f6cf42e256d2ea8225a0b496d303bf357b8
This completes the transitions started in
Ic5a61365ef0132476b11bdbf1dd96885e91c3cb6
The new file plugin.py is the place to start with this change. The
abstract base classes PluginBase and NodeBase are heavily documented.
NodeBase essentially replaces Digraph.Node
The changes in level?/*.py make no functional changes, but are just
refactoring to implement the plugin and node classes consistently.
Additionally we have added asserts during parsing & generation to
ensure plugins are implemented PluginBase, and get_nodes() is always
returning NodeBase objects for the graph.
Change-Id: Ie648e9224749491260dea65d7e8b8151a6824b9c
This switches the code to use networkx for the digraph implementation.
Note that the old implementation specifically isn't removed in this
change -- for review clarity. It will be replaced by a base class
that defines things properly to the API described below.
Plugins return a node object with three functions
get_name() : return the unique name of this node
get_nodes() : return a list of nodes for insertion into the graph.
Usually this is just "self". Some special things like partitioning
add extra nodes at this point, however.
get_edges() : return a tuple of two lists; edges_from and edges_to
As you would expect the first is a list of node names that points to
us, and the second is a list of node names we point to. Usually
this is only populated as ([self.base],[]) -- i.e. our "base" node
points to us. Some plugins, such as mounting, create links both to
and from themselves, however.
Plugins have been updated, some test cases added (error cases
specifically)
Change-Id: Ic5a61365ef0132476b11bdbf1dd96885e91c3cb6
This moves to a more generic config parser that doesn't have plugins
parsing part of the tree.
I understand why it ended up that way; we have "partitions" key which
has special semantics compared to others keys and there was a desire
to keep it isolated from core tree->graph code. But this isn't really
isolated; you have to reverse-engineer several module-crossing
boundaries, extras classes and repetitive recursive functions.
Ultimately, plugins should have access to the node graph, but not
participate in configuration parsing. This way we ensure that plugins
can't invent new methods of configuration parsing.
Note: unit tests produce the same tree -> graph conversion as the old
method. i.e. this is not intended to have a functional change.
Change-Id: I8a5d62a076a5a50597f2f1df3a8615afba6dadb2
Add a range of unit-testing for configuration parsing, graph
generation and mount-point generation. Unfortunately there's some
global variable hacks, and some stubs, but it's a start.
Change-Id: I9e4f950c2c2ea656fc0c1a14594059fb4c62fa35
There's an increasing amount of pydoc based documentation. Output the
module reference and link it into the developers main page.
One fixup to the rst needed
Change-Id: I1d43a1fe1c735eb4559e3d2b40834d1c8115c586
Move Partition() object creation into the actual Partition object,
rather than having the logic within the Partitioning() object
Change-Id: I833ed419a0fca38181a9e2db28e5af87500d8ba4
Split Partition() into it's own file for clarity. This will be
followed-on by less dependence between Partitions and Partition
Change-Id: I860f6a1787c0e4fe99f93919ac37cf7d80bfaae9
Moving the exception didn't cause problems in
I925ed62bdc808f0e07862f6e0905e80b50fbe942, but in later changes where
we split blockdevice.py up a bit more, we can get a bit tangled with
circular imports.
Change-Id: I8297483f64c4e1deecd5ec88ee40e9198bb83589
Because in some cases (e.g. partitioning) the order is needed,
add weights to the digraph to get an (somewhat) stable
topological sort.
Change-Id: I5ef1acc6338ac93c593faa0eafe26cbed42ed887
Signed-off-by: Andreas Florath <andreas@florath.net>
Per [1] this is the "official" CDN mirror, which I think is the most
appropriate for the default. I think this addresses the concerns
httpredir service, which I don't think ever quite got out of beta.
[1] https://wiki.debian.org/DebianGeoMirror
Change-Id: I55f2a00b8bbb0f0a20d3be229e4c2c32a7b69057
This new element will allow to regenerate dracut
on the produced images, to enable different modules. It
relies on a yaml blob to specify modules and packages
needed. It defaults to installing lvm and crypt.
Change-Id: I292fb70cde41ee6053b7b81a67931bcdaaa6d664
"log and throw" is arguably an anti-pattern; the error message either
bubles-up into the exception, or the handler figures it out. We have
an example where this logs, and then the handler in blockdevice.py
catches it and logs it again.
Less layers is better; just raise the exception, and use log.exception
to get tracebacks where handled.
Change-Id: I8efd94fbe52a3911253753f447afdb7565849185
Manifests files can release sensitive information and therefore should
have restrictive permissions.
Change-Id: I64d6c830217a7d8b0172df2dc774079dcd1e2a68
Related-Bug: #1671842
A majority of the "plugins" aren't implementing the plugin class.
Clearly we need some refactoring of the ideas here. Remove for
simplicity.
Change-Id: If399a371b171f4fd17cfa5856fe55daca4c86e60
To avoid failures with double unmount, skip unmounting
the mountpoints that are managed by block device.
Change-Id: I228779eb9bf544a27a53e5017c87573023fd375a
With new block device definition, where content of the image
can be mounted on different partitions, is not enough with
executing setfiles on root directory. Instead of that, expose
all the mountpoints on the image, and apply setfiles on them.
Change-Id: I153f979722eaec49eab93d7cd398c5589b9bfc44
This patch finalizes the block device refactoring. It moves the three
remaining levels (filesystem creation, mount and fstab handling) into
the new python module.
Now it is possible to use any number of disk images, any number of
partitions and used them mounted to different directories.
Notes:
* unmount_dir : modified to only unmount the subdirs mounted by
mount_proc_sys_dev(). dib-block-device unmounts
$TMP_MOUNT_PATH/mnt (see I85e01f3898d3c043071de5fad82307cb091a64a9)
Change-Id: I592c0b1329409307197460cfa8fd69798013f1f8
Signed-off-by: Andreas Florath <andreas@florath.net>
Closes-Bug: #1664924
The args agument was only used to find the symbol for the getval
command. Have the command pass the symbol to find in directly. We
can therefore remove the args paramater to the BlockDevice() creation.
Change-Id: I8e357131b70a00e4a2c4792c009f6058d1d5ae9e
Move argument parsing to subparsers, rather than positional arguments.
This better reflects the tool's role as a driver and allows
sub-commands to deal with arguments in a natural way.
Change-Id: Iae8c368e0f3fe47abfddb9e0a1558bd5b3423aee
I accidentally dropped the clearing of this file when it moved to
cmd.py during rebase of I1919f6e865acae14ee95cd025c9c7b75ca266a9c
Change-Id: Ibe9fcde594770cb51c732cc253987308dc038083
DIB_BLOCK_DEVICE_PARAMS_YAML should be exported, and the
dib-block-device will take this as the value of --params. Remove this
to simplify the command-line
Change-Id: I6764ed223ecd36f9d24e19f164b6a927380b410f
This creates a BlockDeviceCmd object to hold the main() function.
This doesn't really do anything different right now, but sets a base
for using argparse subparsers to handle the command-line
Change-Id: I4acf95ff4d554a3b4e7e2244ab1706631b98458f
This moves the YAML parameter parsing into the command-line driver.
It makes the argument optional so it can be taken from the environment
variable directly. The parsed YAML is passed to the BlockDevice
object.
Change-Id: I6fa5e5b7d1fccfc7cf47d6e4a1fa6e560734680d
To avoid any confusion, commands passed to exec_sudo() should be a
list of "str"s. Log a message if we see unicode issues.
This also adds a debug trace of all output. stderr is captured.
This is modified to raise CalledProcessError on failure, like
check_call(). Calls that are ok to fail will need to explicitly catch
and ignore this.
The two calls that we expect to fail are wrapped
We wish to try rolling back if one of these command raises an
exception. Modify the create handler to initiate rollback on all
exceptions.
Change-Id: Iee4fa41ffaf243e4728bf3a5eeec5c8fa8d2dadc
The await function is essentially a non-standard check_output call.
Let's use standard calls to increase maintainability.
Change-Id: I2c25e1cd7122791fcaa86b46bd801e661471bc9e
As this method can be introduced without any dependency,
provide it on an independent change to simplify reviews.
This is a partial refactor based on
I592c0b1329409307197460cfa8fd69798013f1f8
Change-Id: Idaf3d2b3b3e23d0b9d6bc071d67b961a829ae422
Co-Authored-By: Andreas Florath <andreas@florath.net>
This is a partial refactor from change
I592c0b1329409307197460cfa8fd69798013f1f8
Change-Id: I8822e68e41c4ebd47eea9ffed4557efc130a7bf7
Co-Authored-By: Andreas Florath <andreas@florath.net>
Add a new method in the block device library called
exec_sudo, so it can be reused.
This is a partial refactor of change
I592c0b1329409307197460cfa8fd69798013f1f8
Change-Id: Id621f6d029e1275a35c4fd3f19b57c8518076134
Co-Authored-By: Andreas Florath <andreas@florath.net>
As part of the final steps, refactor the bits belonging
to block device and functions. This is a partial refactor
from I3600c6a3d663c697b59d91bd3fbb5e408af345e4
Change-Id: I7aa4fe0466e44846d8fa3194575d446fe4b5b2e6
Co-Authored-By: Andreas Florath <andreas@florath.net>
Introducing the refactors of the block device to allow a tree-like
configuration, and start using it for the partitions level.
Based on patch I3600c6a3d663c697b59d91bd3fbb5e408af345e4
Change-Id: I58bb3c256a1dfd100d29266571c333c2d43334f7
Co-Authored-By: Andreas Florath <andreas@florath.net>
It seems that the redhat nodepool job is quite reliably geting a
"floating point" error during centos image build. This happens after
03-yum-cleanup which is pruning the locales. This might be a
red-herring, since the logs are full of
/bin/bash: warning: setlocale: LC_ALL: cannot change locale (C.UTF-8)
I think in our recent de-puppetisation of hosts, something might have
changed that is setting LC_ALL=C.UTF-8 for the jenkins user, at least
on Ubuntu. This is a problem for centos, as it doesn't have C.UTF-8
locale. I then think using the invalid locale is what leads the the
floating-point error when doing some maths in dib-run-parts to
calculate runtimes.
We are currently overriding LANG, but we really want LC_ALL to ensure
this applies globally.
Change-Id: I8e7cae093c4b32e0d20b73ae0086f14c7cc6a9cb
Add a new getval call that allows to retrieve values
from the block device. Also isolating the block device
information into a 'blockdev' dictionary entry, to better
return it with the getval command.
This is a refactor from the original code at
I3600c6a3d663c697b59d91bd3fbb5e408af345e4.
Change-Id: I93d33669a3a0ae644eab9f9b955bb5a9470cadeb
Co-Authored-By: Andreas Florath <andreas@florath.net>
