sig_cloud/diskimage-builder
2
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
f58bf252de
diskimage-builder/diskimage_builder/block_device/tests/test_config.py

159 lines
6.3 KiB
Python
Raw Normal View History

Adding unit testing for configuration Add a range of unit-testing for configuration parsing, graph generation and mount-point generation. Unfortunately there's some global variable hacks, and some stubs, but it's a start. Change-Id: I9e4f950c2c2ea656fc0c1a14594059fb4c62fa35
2017-05-23 01:21:02 +00:00
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import logging
Move create_graph into config.py This was suggested in a review comment in I8a5d62a076a5a50597f2f1df3a8615afba6dadb2. It works out quite nicely because the BlockDevice() driver now doesn't need to know anything about stevedore or plugins, and just works on the node list. It also simplifies the unit testing by not having to call create_graph through a BlockDevice object. Change-Id: I98512f6cf42e256d2ea8225a0b496d303bf357b8
2017-05-26 00:32:59 +00:00
from diskimage_builder.block_device.config import config_tree_to_graph
from diskimage_builder.block_device.config import create_graph
Add a more generic tree->graph parser This moves to a more generic config parser that doesn't have plugins parsing part of the tree. I understand why it ended up that way; we have "partitions" key which has special semantics compared to others keys and there was a desire to keep it isolated from core tree->graph code. But this isn't really isolated; you have to reverse-engineer several module-crossing boundaries, extras classes and repetitive recursive functions. Ultimately, plugins should have access to the node graph, but not participate in configuration parsing. This way we ensure that plugins can't invent new methods of configuration parsing. Note: unit tests produce the same tree -> graph conversion as the old method. i.e. this is not intended to have a functional change. Change-Id: I8a5d62a076a5a50597f2f1df3a8615afba6dadb2
2017-05-17 05:54:02 +00:00
from diskimage_builder.block_device.exception import \
BlockDeviceSetupException
Add state object, rename "results", add unit tests A couple of things going on, but I think it makes sense to do them atomically. The NodeBase.create() argument "results" is the global state dictionary that will be saved to "state.json", and re-loaded in later phases and passed to them as the argument "state". So for consistency, call this argument "state" (this fits with the change out to start building the state dictionary earlier in the PluginBase.__init__() calls). Since the "state" is a pretty important part of how everything works, move it into a separate object. This is treated as essentially a singleton. It bundles it nicely together for some added documentation [1]. We move instantiation of this object out of the generic BlockDevice.__init__() call and into the actual cmd_* drivers. This is because there's two distinct instantiation operations -- creating a new state (during cmd_create) and loading an existing state (other cmd_*). This is also safer -- since we know the cmd_* arguments are looking for an existing state.json, we will fail if it somehow goes missing. To more fully unit test this, some testing plugins and new entry-points are added. These add known state values which we check for. These should be a good basis for further tests. [1] as noted, we could probably do some fun things in the future like make this implement a dictionary and have some saftey features like r/o keys. Change-Id: I90eb711b3e9b1ce139eb34bdf3cde641fd06828f
2017-05-30 02:06:41 +00:00
from diskimage_builder.block_device.tests.test_base import TestBase
Adding unit testing for configuration Add a range of unit-testing for configuration parsing, graph generation and mount-point generation. Unfortunately there's some global variable hacks, and some stubs, but it's a start. Change-Id: I9e4f950c2c2ea656fc0c1a14594059fb4c62fa35
2017-05-23 01:21:02 +00:00
logger = logging.getLogger(__name__)
Add state object, rename "results", add unit tests A couple of things going on, but I think it makes sense to do them atomically. The NodeBase.create() argument "results" is the global state dictionary that will be saved to "state.json", and re-loaded in later phases and passed to them as the argument "state". So for consistency, call this argument "state" (this fits with the change out to start building the state dictionary earlier in the PluginBase.__init__() calls). Since the "state" is a pretty important part of how everything works, move it into a separate object. This is treated as essentially a singleton. It bundles it nicely together for some added documentation [1]. We move instantiation of this object out of the generic BlockDevice.__init__() call and into the actual cmd_* drivers. This is because there's two distinct instantiation operations -- creating a new state (during cmd_create) and loading an existing state (other cmd_*). This is also safer -- since we know the cmd_* arguments are looking for an existing state.json, we will fail if it somehow goes missing. To more fully unit test this, some testing plugins and new entry-points are added. These add known state values which we check for. These should be a good basis for further tests. [1] as noted, we could probably do some fun things in the future like make this implement a dictionary and have some saftey features like r/o keys. Change-Id: I90eb711b3e9b1ce139eb34bdf3cde641fd06828f
2017-05-30 02:06:41 +00:00
class TestConfig(TestBase):
Adding unit testing for configuration Add a range of unit-testing for configuration parsing, graph generation and mount-point generation. Unfortunately there's some global variable hacks, and some stubs, but it's a start. Change-Id: I9e4f950c2c2ea656fc0c1a14594059fb4c62fa35
2017-05-23 01:21:02 +00:00
"""Helper for setting up and reading a config"""
def setUp(self):
super(TestConfig, self).setUp()
# reset all globals for each test.
# XXX: remove globals :/
import diskimage_builder.block_device.level2.mkfs
diskimage_builder.block_device.level2.mkfs.file_system_labels = set()
import diskimage_builder.block_device.level3.mount
Refactor mount-point sorting Currently we keep a global list of mount-points defined in the configuration and automatically setup dependencies between mount nodes based on their global "mount order" (i.e. higher directories mount first). The current method for achieving this is roughly to add the mount points to a dictionary indexed my mount-point, then at "get_edge()" call build the sorted list ... unless it has already been built because this gets called for every node. It seems much simpler to simply keep a sorted list of the MountPointNode objects as we add them. We don't need to implement a sorting algorithm then, we can just use sort() and implement __lt__ for the nodes. I believe the existing mount-order unit testing is sufficient; I'm struggling to find a valid configuration where the mount-order is *not* correctly specified in the configuration graph. Change-Id: Idc05cdf42d95e230b9906773aa2b4a3b0f075598
2017-05-31 00:41:02 +00:00
diskimage_builder.block_device.level3.mount.sorted_mount_points = []
Adding unit testing for configuration Add a range of unit-testing for configuration parsing, graph generation and mount-point generation. Unfortunately there's some global variable hacks, and some stubs, but it's a start. Change-Id: I9e4f950c2c2ea656fc0c1a14594059fb4c62fa35
2017-05-23 01:21:02 +00:00
class TestGraphGeneration(TestConfig):
"""Extra helper class for testing graph generation"""
def setUp(self):
super(TestGraphGeneration, self).setUp()
self.fake_default_config = {
'build-dir': '/fake',
'image-size': '1000',
'image-dir': '/fake',
'mount-base': '/fake',
}
Add a more generic tree->graph parser This moves to a more generic config parser that doesn't have plugins parsing part of the tree. I understand why it ended up that way; we have "partitions" key which has special semantics compared to others keys and there was a desire to keep it isolated from core tree->graph code. But this isn't really isolated; you have to reverse-engineer several module-crossing boundaries, extras classes and repetitive recursive functions. Ultimately, plugins should have access to the node graph, but not participate in configuration parsing. This way we ensure that plugins can't invent new methods of configuration parsing. Note: unit tests produce the same tree -> graph conversion as the old method. i.e. this is not intended to have a functional change. Change-Id: I8a5d62a076a5a50597f2f1df3a8615afba6dadb2
2017-05-17 05:54:02 +00:00
class TestConfigParsing(TestConfig):
Adding unit testing for configuration Add a range of unit-testing for configuration parsing, graph generation and mount-point generation. Unfortunately there's some global variable hacks, and some stubs, but it's a start. Change-Id: I9e4f950c2c2ea656fc0c1a14594059fb4c62fa35
2017-05-23 01:21:02 +00:00
"""Test parsing config file into a graph"""
Use networkx for digraph This switches the code to use networkx for the digraph implementation. Note that the old implementation specifically isn't removed in this change -- for review clarity. It will be replaced by a base class that defines things properly to the API described below. Plugins return a node object with three functions get_name() : return the unique name of this node get_nodes() : return a list of nodes for insertion into the graph. Usually this is just "self". Some special things like partitioning add extra nodes at this point, however. get_edges() : return a tuple of two lists; edges_from and edges_to As you would expect the first is a list of node names that points to us, and the second is a list of node names we point to. Usually this is only populated as ([self.base],[]) -- i.e. our "base" node points to us. Some plugins, such as mounting, create links both to and from themselves, however. Plugins have been updated, some test cases added (error cases specifically) Change-Id: Ic5a61365ef0132476b11bdbf1dd96885e91c3cb6
2017-05-23 23:57:32 +00:00
# test an entry in the config not being a valid plugin
Adding unit testing for configuration Add a range of unit-testing for configuration parsing, graph generation and mount-point generation. Unfortunately there's some global variable hacks, and some stubs, but it's a start. Change-Id: I9e4f950c2c2ea656fc0c1a14594059fb4c62fa35
2017-05-23 01:21:02 +00:00
def test_config_bad_plugin(self):
Add a more generic tree->graph parser This moves to a more generic config parser that doesn't have plugins parsing part of the tree. I understand why it ended up that way; we have "partitions" key which has special semantics compared to others keys and there was a desire to keep it isolated from core tree->graph code. But this isn't really isolated; you have to reverse-engineer several module-crossing boundaries, extras classes and repetitive recursive functions. Ultimately, plugins should have access to the node graph, but not participate in configuration parsing. This way we ensure that plugins can't invent new methods of configuration parsing. Note: unit tests produce the same tree -> graph conversion as the old method. i.e. this is not intended to have a functional change. Change-Id: I8a5d62a076a5a50597f2f1df3a8615afba6dadb2
2017-05-17 05:54:02 +00:00
config = self.load_config_file('bad_plugin.yaml')
self.assertRaises(BlockDeviceSetupException,
config_tree_to_graph,
config)
Adding unit testing for configuration Add a range of unit-testing for configuration parsing, graph generation and mount-point generation. Unfortunately there's some global variable hacks, and some stubs, but it's a start. Change-Id: I9e4f950c2c2ea656fc0c1a14594059fb4c62fa35
2017-05-23 01:21:02 +00:00
Use networkx for digraph This switches the code to use networkx for the digraph implementation. Note that the old implementation specifically isn't removed in this change -- for review clarity. It will be replaced by a base class that defines things properly to the API described below. Plugins return a node object with three functions get_name() : return the unique name of this node get_nodes() : return a list of nodes for insertion into the graph. Usually this is just "self". Some special things like partitioning add extra nodes at this point, however. get_edges() : return a tuple of two lists; edges_from and edges_to As you would expect the first is a list of node names that points to us, and the second is a list of node names we point to. Usually this is only populated as ([self.base],[]) -- i.e. our "base" node points to us. Some plugins, such as mounting, create links both to and from themselves, however. Plugins have been updated, some test cases added (error cases specifically) Change-Id: Ic5a61365ef0132476b11bdbf1dd96885e91c3cb6
2017-05-23 23:57:32 +00:00
# test a config that has multiple keys for a top-level entry
def test_config_multikey_node(self):
config = self.load_config_file('multi_key_node.yaml')
self.assertRaisesRegexp(BlockDeviceSetupException,
"Config entry top-level should be a single "
"dict:",
config_tree_to_graph,
config)
Adding unit testing for configuration Add a range of unit-testing for configuration parsing, graph generation and mount-point generation. Unfortunately there's some global variable hacks, and some stubs, but it's a start. Change-Id: I9e4f950c2c2ea656fc0c1a14594059fb4c62fa35
2017-05-23 01:21:02 +00:00
# a graph should remain the same
def test_graph(self):
graph = self.load_config_file('simple_graph.yaml')
Add a more generic tree->graph parser This moves to a more generic config parser that doesn't have plugins parsing part of the tree. I understand why it ended up that way; we have "partitions" key which has special semantics compared to others keys and there was a desire to keep it isolated from core tree->graph code. But this isn't really isolated; you have to reverse-engineer several module-crossing boundaries, extras classes and repetitive recursive functions. Ultimately, plugins should have access to the node graph, but not participate in configuration parsing. This way we ensure that plugins can't invent new methods of configuration parsing. Note: unit tests produce the same tree -> graph conversion as the old method. i.e. this is not intended to have a functional change. Change-Id: I8a5d62a076a5a50597f2f1df3a8615afba6dadb2
2017-05-17 05:54:02 +00:00
parsed_graph = config_tree_to_graph(graph)
self.assertItemsEqual(parsed_graph, graph)
Adding unit testing for configuration Add a range of unit-testing for configuration parsing, graph generation and mount-point generation. Unfortunately there's some global variable hacks, and some stubs, but it's a start. Change-Id: I9e4f950c2c2ea656fc0c1a14594059fb4c62fa35
2017-05-23 01:21:02 +00:00
# equivalence of simple tree to graph
def test_simple_tree(self):
tree = self.load_config_file('simple_tree.yaml')
graph = self.load_config_file('simple_graph.yaml')
Add a more generic tree->graph parser This moves to a more generic config parser that doesn't have plugins parsing part of the tree. I understand why it ended up that way; we have "partitions" key which has special semantics compared to others keys and there was a desire to keep it isolated from core tree->graph code. But this isn't really isolated; you have to reverse-engineer several module-crossing boundaries, extras classes and repetitive recursive functions. Ultimately, plugins should have access to the node graph, but not participate in configuration parsing. This way we ensure that plugins can't invent new methods of configuration parsing. Note: unit tests produce the same tree -> graph conversion as the old method. i.e. this is not intended to have a functional change. Change-Id: I8a5d62a076a5a50597f2f1df3a8615afba6dadb2
2017-05-17 05:54:02 +00:00
parsed_graph = config_tree_to_graph(tree)
Adding unit testing for configuration Add a range of unit-testing for configuration parsing, graph generation and mount-point generation. Unfortunately there's some global variable hacks, and some stubs, but it's a start. Change-Id: I9e4f950c2c2ea656fc0c1a14594059fb4c62fa35
2017-05-23 01:21:02 +00:00
self.assertItemsEqual(parsed_graph, graph)
# equivalence of a deeper tree to graph
def test_deep_tree(self):
tree = self.load_config_file('deep_tree.yaml')
graph = self.load_config_file('deep_graph.yaml')
Add a more generic tree->graph parser This moves to a more generic config parser that doesn't have plugins parsing part of the tree. I understand why it ended up that way; we have "partitions" key which has special semantics compared to others keys and there was a desire to keep it isolated from core tree->graph code. But this isn't really isolated; you have to reverse-engineer several module-crossing boundaries, extras classes and repetitive recursive functions. Ultimately, plugins should have access to the node graph, but not participate in configuration parsing. This way we ensure that plugins can't invent new methods of configuration parsing. Note: unit tests produce the same tree -> graph conversion as the old method. i.e. this is not intended to have a functional change. Change-Id: I8a5d62a076a5a50597f2f1df3a8615afba6dadb2
2017-05-17 05:54:02 +00:00
parsed_graph = config_tree_to_graph(tree)
Adding unit testing for configuration Add a range of unit-testing for configuration parsing, graph generation and mount-point generation. Unfortunately there's some global variable hacks, and some stubs, but it's a start. Change-Id: I9e4f950c2c2ea656fc0c1a14594059fb4c62fa35
2017-05-23 01:21:02 +00:00
self.assertItemsEqual(parsed_graph, graph)
# equivalence of a complicated multi-partition tree to graph
def test_multipart_tree(self):
tree = self.load_config_file('multiple_partitions_tree.yaml')
graph = self.load_config_file('multiple_partitions_graph.yaml')
Add a more generic tree->graph parser This moves to a more generic config parser that doesn't have plugins parsing part of the tree. I understand why it ended up that way; we have "partitions" key which has special semantics compared to others keys and there was a desire to keep it isolated from core tree->graph code. But this isn't really isolated; you have to reverse-engineer several module-crossing boundaries, extras classes and repetitive recursive functions. Ultimately, plugins should have access to the node graph, but not participate in configuration parsing. This way we ensure that plugins can't invent new methods of configuration parsing. Note: unit tests produce the same tree -> graph conversion as the old method. i.e. this is not intended to have a functional change. Change-Id: I8a5d62a076a5a50597f2f1df3a8615afba6dadb2
2017-05-17 05:54:02 +00:00
parsed_graph = config_tree_to_graph(tree)
Adding unit testing for configuration Add a range of unit-testing for configuration parsing, graph generation and mount-point generation. Unfortunately there's some global variable hacks, and some stubs, but it's a start. Change-Id: I9e4f950c2c2ea656fc0c1a14594059fb4c62fa35
2017-05-23 01:21:02 +00:00
logger.debug(parsed_graph)
self.assertItemsEqual(parsed_graph, graph)
class TestCreateGraph(TestGraphGeneration):
Use networkx for digraph This switches the code to use networkx for the digraph implementation. Note that the old implementation specifically isn't removed in this change -- for review clarity. It will be replaced by a base class that defines things properly to the API described below. Plugins return a node object with three functions get_name() : return the unique name of this node get_nodes() : return a list of nodes for insertion into the graph. Usually this is just "self". Some special things like partitioning add extra nodes at this point, however. get_edges() : return a tuple of two lists; edges_from and edges_to As you would expect the first is a list of node names that points to us, and the second is a list of node names we point to. Usually this is only populated as ([self.base],[]) -- i.e. our "base" node points to us. Some plugins, such as mounting, create links both to and from themselves, however. Plugins have been updated, some test cases added (error cases specifically) Change-Id: Ic5a61365ef0132476b11bdbf1dd96885e91c3cb6
2017-05-23 23:57:32 +00:00
# Test a graph with bad edge pointing to an invalid node
def test_invalid_missing(self):
config = self.load_config_file('bad_edge_graph.yaml')
self.assertRaisesRegexp(BlockDeviceSetupException,
"Edge not defined: this_is_not_a_node",
Move create_graph into config.py This was suggested in a review comment in I8a5d62a076a5a50597f2f1df3a8615afba6dadb2. It works out quite nicely because the BlockDevice() driver now doesn't need to know anything about stevedore or plugins, and just works on the node list. It also simplifies the unit testing by not having to call create_graph through a BlockDevice object. Change-Id: I98512f6cf42e256d2ea8225a0b496d303bf357b8
2017-05-26 00:32:59 +00:00
create_graph,
Use networkx for digraph This switches the code to use networkx for the digraph implementation. Note that the old implementation specifically isn't removed in this change -- for review clarity. It will be replaced by a base class that defines things properly to the API described below. Plugins return a node object with three functions get_name() : return the unique name of this node get_nodes() : return a list of nodes for insertion into the graph. Usually this is just "self". Some special things like partitioning add extra nodes at this point, however. get_edges() : return a tuple of two lists; edges_from and edges_to As you would expect the first is a list of node names that points to us, and the second is a list of node names we point to. Usually this is only populated as ([self.base],[]) -- i.e. our "base" node points to us. Some plugins, such as mounting, create links both to and from themselves, however. Plugins have been updated, some test cases added (error cases specifically) Change-Id: Ic5a61365ef0132476b11bdbf1dd96885e91c3cb6
2017-05-23 23:57:32 +00:00
config, self.fake_default_config)
# Test a graph with bad edge pointing to an invalid node
def test_duplicate_name(self):
config = self.load_config_file('duplicate_name.yaml')
self.assertRaisesRegexp(BlockDeviceSetupException,
"Duplicate node name: "
"this_is_a_duplicate",
Move create_graph into config.py This was suggested in a review comment in I8a5d62a076a5a50597f2f1df3a8615afba6dadb2. It works out quite nicely because the BlockDevice() driver now doesn't need to know anything about stevedore or plugins, and just works on the node list. It also simplifies the unit testing by not having to call create_graph through a BlockDevice object. Change-Id: I98512f6cf42e256d2ea8225a0b496d303bf357b8
2017-05-26 00:32:59 +00:00
create_graph,
Use networkx for digraph This switches the code to use networkx for the digraph implementation. Note that the old implementation specifically isn't removed in this change -- for review clarity. It will be replaced by a base class that defines things properly to the API described below. Plugins return a node object with three functions get_name() : return the unique name of this node get_nodes() : return a list of nodes for insertion into the graph. Usually this is just "self". Some special things like partitioning add extra nodes at this point, however. get_edges() : return a tuple of two lists; edges_from and edges_to As you would expect the first is a list of node names that points to us, and the second is a list of node names we point to. Usually this is only populated as ([self.base],[]) -- i.e. our "base" node points to us. Some plugins, such as mounting, create links both to and from themselves, however. Plugins have been updated, some test cases added (error cases specifically) Change-Id: Ic5a61365ef0132476b11bdbf1dd96885e91c3cb6
2017-05-23 23:57:32 +00:00
config, self.fake_default_config)
Adding unit testing for configuration Add a range of unit-testing for configuration parsing, graph generation and mount-point generation. Unfortunately there's some global variable hacks, and some stubs, but it's a start. Change-Id: I9e4f950c2c2ea656fc0c1a14594059fb4c62fa35
2017-05-23 01:21:02 +00:00
# Test digraph generation from deep_graph config file
def test_deep_graph_generator(self):
config = self.load_config_file('deep_graph.yaml')
Move create_graph into config.py This was suggested in a review comment in I8a5d62a076a5a50597f2f1df3a8615afba6dadb2. It works out quite nicely because the BlockDevice() driver now doesn't need to know anything about stevedore or plugins, and just works on the node list. It also simplifies the unit testing by not having to call create_graph through a BlockDevice object. Change-Id: I98512f6cf42e256d2ea8225a0b496d303bf357b8
2017-05-26 00:32:59 +00:00
graph, call_order = create_graph(config, self.fake_default_config)
Adding unit testing for configuration Add a range of unit-testing for configuration parsing, graph generation and mount-point generation. Unfortunately there's some global variable hacks, and some stubs, but it's a start. Change-Id: I9e4f950c2c2ea656fc0c1a14594059fb4c62fa35
2017-05-23 01:21:02 +00:00
call_order_list = [n.name for n in call_order]
# manually created from deep_graph.yaml
# Note unlike below, the sort here is stable because the graph
# doesn't have multiple paths with only one partition
call_order_names = ['image0', 'root', 'mkfs_root',
'mount_mkfs_root',
'fstab_mount_mkfs_root']
self.assertListEqual(call_order_list, call_order_names)
# Test multiple parition digraph generation
def test_multiple_partitions_graph_generator(self):
config = self.load_config_file('multiple_partitions_graph.yaml')
Move create_graph into config.py This was suggested in a review comment in I8a5d62a076a5a50597f2f1df3a8615afba6dadb2. It works out quite nicely because the BlockDevice() driver now doesn't need to know anything about stevedore or plugins, and just works on the node list. It also simplifies the unit testing by not having to call create_graph through a BlockDevice object. Change-Id: I98512f6cf42e256d2ea8225a0b496d303bf357b8
2017-05-26 00:32:59 +00:00
graph, call_order = create_graph(config, self.fake_default_config)
Adding unit testing for configuration Add a range of unit-testing for configuration parsing, graph generation and mount-point generation. Unfortunately there's some global variable hacks, and some stubs, but it's a start. Change-Id: I9e4f950c2c2ea656fc0c1a14594059fb4c62fa35
2017-05-23 01:21:02 +00:00
call_order_list = [n.name for n in call_order]
# The sort creating call_order_list is unstable.
# We want to ensure we see the "partitions" object in
# root->var->var_log order
root_pos = call_order_list.index('root')
var_pos = call_order_list.index('var')
var_log_pos = call_order_list.index('var_log')
self.assertGreater(var_pos, root_pos)
self.assertGreater(var_log_pos, var_pos)
# Ensure mkfs happens after partition
mkfs_root_pos = call_order_list.index('mkfs_root')
self.assertLess(root_pos, mkfs_root_pos)
mkfs_var_pos = call_order_list.index('mkfs_var')
self.assertLess(var_pos, mkfs_var_pos)
mkfs_var_log_pos = call_order_list.index('mkfs_var_log')
self.assertLess(var_log_pos, mkfs_var_log_pos)
Reference in New Issue Copy Permalink