Commit Graph

12 Commits

Author SHA1 Message Date
Ian Wienand
2a25db9ee6
Start running dib-lint again
A recent change that didn't fail with hard-tabs made me realise we're
not running tox -e pep8 ... which means we're not running dib-lint
which should find this (and other things).

I couldn't pinpoint when this happened; maybe job config was never in
this repo.

Anyway, move the pylint and dib-lint/flake8 testing to the now
standard "linters" and update the linting job to
openstack-tox-linters.

It looks like pylint is very lightly used (came in with
I7e24d8348db3aef79e1395d12692199a1f80161a and we've never expanded any
testing).  Leave this alone for now, but probably it is not important
any more.

This revealed some issues; updated flake8
(Iaa19c36f8cab8482a01f764c588375db8e7d8be3) found some spacing issues
with keywords and an update to elrepo to match our standard bash
flags.

Change-Id: I45bf108c467f7c8190ca252e6c48450c2622aaf8
2022-09-21 07:56:05 +10:00
Steve Baker
833c5b8ceb Support LVM thin provisioning
This change extends the block device lvs attributes to allow creating
a volume which represents a thin pool, and to create volumes which are
allocated from this pool.

Change-Id: Ic58f55c36236cc8c6279fbcb708e27dc2982f2d5
2022-08-24 10:34:42 +12:00
Steve Baker
d090126c66 Parse block device lvm lvs size attributes
The block device lvm lvs `size` attribute was passed directly to
lvcreate, so using units M, G means base 2. All other block device
size values are parsed with accepted conventions of M, B being base 10
and MiB, GiB being base 2.

lvm lvs `size` attributes are now parsed the same as other size
attributes. This improves consistency and makes it practical to
calculate volume sizes to fill the partition size. This means existing
size values will now create slightly smaller volumes. Previous sizes
can be restored by changing the unit to MiB, GiB, or increasing the
value for a base 10 unit.

The impact on this change should be minimal, the only known uses of lvm
volumes (TripleO, and element block-device-efi-lvm) uses extents
percentage instead of size. The smaller sizes can always be increased
after deployment.

Requested sizes will also be rounded down to align with physical
extents (4MiB). Previously specifying a value which did not align on
4MiB would consume an extra extent which could unexpectedly consume
more than the partition size.

Change-Id: Ia109cc5105071d82cc895d8d9cb85bc47da20a7a
2022-07-06 11:27:42 +12:00
Zuul
ea6403388e Merge "Switch from unittest2 compat methods to Python 3.x methods" 2020-07-13 02:21:12 +00:00
melissaml
bf0dc265ae Switch from unittest2 compat methods to Python 3.x methods
With the removal of Python 2.x we can remove the unittest2 compat
wrappers and switch to assertCountEqual instead of assertItemsEqual

We have been able to use them since then, because
testtools required unittest2, which still included it. With testtools
removing Python 2.7 support [3][4], we will lose support for
assertItemsEqual, so we should switch to use assertCountEqual.

[1] - https://bugs.python.org/issue17866
[2] - https://hg.python.org/cpython/rev/d9921cb6e3cd
[3] - testing-cabal/testtools#286
[4] - testing-cabal/testtools#277

Change-Id: I870286a2557e41099597c22dc9747743e1077615
2020-07-07 11:11:28 +08:00
Simon Westphahl
4a424ecabb Use kpartx option to update partition mappings
Fix cases of 'mkfs' failing because the partitions never showed up. Partition
mappings will now be updated instead of just adding them with 'kpartx'. That
means that 'kpartx' will also remove devmappings for deleted partitions.

Traceback of failing mkfs call:

2020-05-11 22:03:25.523 | INFO diskimage_builder.block_device.utils [-] Calling [sudo sync]
2020-05-11 22:03:25.539 | INFO diskimage_builder.block_device.utils [-] Calling [sudo kpartx -avs /dev/loop0]
2020-05-11 22:03:25.581 | INFO diskimage_builder.block_device.utils [-] Calling [sudo mkfs -t ext4 -i 4096 -J size=64 -L cloudimg-rootfs -U 21c6f9eb-4d52-4e5c-b9b7-796735de8909 -q /dev/mapper/loop0p1]
2020-05-11 22:03:25.700 | ERROR diskimage_builder.block_device.blockdevice [-] Create failed; rollback initiated
2020-05-11 22:03:25.700 | Traceback (most recent call last):
2020-05-11 22:03:25.700 |   File "/home/zuul/dib/lib/python3.6/site-packages/diskimage_builder/block_device/blockdevice.py", line 406, in cmd_create
2020-05-11 22:03:25.700 |     node.create()
2020-05-11 22:03:25.700 |   File "/home/zuul/dib/lib/python3.6/site-packages/diskimage_builder/block_device/level2/mkfs.py", line 133, in create
2020-05-11 22:03:25.700 |     exec_sudo(cmd)
2020-05-11 22:03:25.700 |   File "/home/zuul/dib/lib/python3.6/site-packages/diskimage_builder/block_device/utils.py", line 143, in exec_sudo
2020-05-11 22:03:25.700 |     raise e
2020-05-11 22:03:25.700 | diskimage_builder.block_device.exception.BlockDeviceSetupException: exec_sudo failed
2020-05-11 22:03:25.700 | INFO diskimage_builder.block_device.level0.localloop [-] loopdev detach
2020-05-11 22:03:25.701 | INFO diskimage_builder.block_device.utils [-] Calling [sudo losetup -d /dev/loop0]
2020-05-11 22:03:25.732 | INFO diskimage_builder.block_device.level0.localloop [-] Remove image file [/tmp/dib_image.muyw7t1h/image0.raw]
2020-05-11 22:03:25.734 | ERROR diskimage_builder.block_device.blockdevice [-] Rollback complete, exiting
2020-05-11 22:03:25.740 | Traceback (most recent call last):
2020-05-11 22:03:25.740 |   File "/home/zuul/dib/bin/dib-block-device", line 8, in <module>
2020-05-11 22:03:25.740 |     sys.exit(main())
2020-05-11 22:03:25.740 |   File "/home/zuul/dib/lib/python3.6/site-packages/diskimage_builder/block_device/cmd.py", line 120, in main
2020-05-11 22:03:25.740 |     return bdc.main()
2020-05-11 22:03:25.740 |   File "/home/zuul/dib/lib/python3.6/site-packages/diskimage_builder/block_device/cmd.py", line 115, in main
2020-05-11 22:03:25.740 |     self.args.func()
2020-05-11 22:03:25.740 |   File "/home/zuul/dib/lib/python3.6/site-packages/diskimage_builder/block_device/cmd.py", line 36, in cmd_create
2020-05-11 22:03:25.740 |     self.bd.cmd_create()
2020-05-11 22:03:25.740 |   File "/home/zuul/dib/lib/python3.6/site-packages/diskimage_builder/block_device/blockdevice.py", line 406, in cmd_create
2020-05-11 22:03:25.740 |     node.create()
2020-05-11 22:03:25.740 |   File "/home/zuul/dib/lib/python3.6/site-packages/diskimage_builder/block_device/level2/mkfs.py", line 133, in create
2020-05-11 22:03:25.740 |     exec_sudo(cmd)
2020-05-11 22:03:25.740 |   File "/home/zuul/dib/lib/python3.6/site-packages/diskimage_builder/block_device/utils.py", line 143, in exec_sudo
2020-05-11 22:03:25.740 |     raise e
2020-05-11 22:03:25.740 | diskimage_builder.block_device.exception.BlockDeviceSetupException: exec_sudo failed

Change-Id: I374f7f22f9e93ef35eb5813712ca59e75f0733e8
Related-Bug: #1698337
2020-06-09 09:07:55 +02:00
Ian Wienand
28ebd24844 Uncap hacking
This causes problems for other projects incorporating dib; we don't
have a specific need for a cap.

Fix a few issues, mostly spacing or regex matches.  No functional
changes.

W503 and W504 relate to leaving artithmetic operators at the start or
end of lines, and are mutually exclusive and, due to "ignore"
overriding the defaults both get enabled.  It seems everyone gets this
wrong (https://gitlab.com/pycqa/flake8/issues/466).  Don't take a
position on this and ignore both.

Use double # around comments including YAML snippets using "# type: "
which now gets detected as PEP484/mypy type hints.

Change-Id: I8b7ce6dee02dcce31c82427a2441c931d136ef57
2020-02-24 10:34:46 +11:00
Yolanda Robla
64bb87f7b5 Only detach device if all partitions have been cleaned
Currently there is a bug, that tries to detach the device from a
partition at the first try, without considering that there may be
other partitions and volumes on it. Ensure that the detach is done
properly, and add a test to ensure that this happens correctly.

Change-Id: I35c5a473509f17a70270a2cbf5bf579faaeb123a
Fixes-Bug: #1777861
2018-07-30 16:24:57 +10:00
Ian Wienand
7302f38f97 Move LVM cleanup phase into cleanup
A recap -- we run umount phase then cleanup phase.

Currently we register a object to do the final LVM cleanup based on
the parent PV.  In light of I697bfbf042816c5ddf170bde9534cc4f0c7279ff,
I believe this should just be done in the cleanup phase.  Note there
was probably additional confusion because the partition removal was
done in the cleanup phase until
I7af3c5cf66afd81a481f454b5207af552ad52a32, where is was moved into the
umount phase.

Thus it is moved into the cleanup() function and this should now run,
per the comment, after everything is unmounted in umount phase.

This also exposes that we didn't have the cleanup phase in the unit
tests (because it wasn't doing anything I guess).  Add it.

Change-Id: I1c5f4ffc9619c774f78d21b918a81647b3dc28f5
2018-07-30 14:35:16 +10:00
Andreas Florath
f5736f3178 block-device lvm: fix umount phase
As described in blockdevice.py detachment and (most) resources
release must be done in the umount phase of a block device module.

Until now these jobs were done in the lvm cleanup() phase - which
is too late - especially when using nested LVMs.

This patch moves the functionality of the cleanup() phase to the
umount() phase for the lvm module.
It includes a test case that fails without applying the provided
source code changes.

Change-Id: I697bfbf042816c5ddf170bde9534cc4f0c7279ff
Signed-off-by: Andreas Florath <andreas@florath.net>
2018-06-28 15:21:59 +10:00
Andreas Florath
bb6cf52d85 Remove dd from LVM element
This patch removes the unneeded dd calls in the lvm block device
plugin.

After removing the underlying block device, there is the need to call
'pvscan --cache'.  This is done by a dedicated LVM cleanup node which
is cleaned up after the the underlying block device.

Change-Id: Id8eaede77fbdc107d2ba1035cd6b8eb5c10160c3
Signed-off-by: Andreas Florath <andreas@florath.net>
2017-10-08 17:21:21 +00:00
Yolanda Robla
c2dc3dc78e LVM support for dib-block-device
This provides a basic LVM support to dib-block-device.

Co-Authored-By: Ian Wienand <iwienand@redhat.com>

Change-Id: Ibd624d9f95ee68b20a15891f639ddd5b3188cdf9
2017-08-24 16:22:56 +10:00