Commit Graph

2939 Commits

Author SHA1 Message Date
Jenkins
c54e9fddfa Merge "Use fakelogger in test_blockdevice_mbr" 2017-05-17 23:59:18 +00:00
Jenkins
340738fb86 Merge "Only unmount directories that are mounted" 2017-05-17 17:32:11 +00:00
Ian Wienand
08e8471360 Use fakelogger in test_blockdevice_mbr
Capture logging instead of putting it on stdout

Change-Id: Ic0c9f25549a1984fecf8e912e5b14845e2363730
2017-05-17 18:49:12 +10:00
Ian Wienand
2d2b2725bd Remove PluginBase/NodePluginBase class
A majority of the "plugins" aren't implementing the plugin class.
Clearly we need some refactoring of the ideas here.  Remove for
simplicity.

Change-Id: If399a371b171f4fd17cfa5856fe55daca4c86e60
2017-05-17 09:03:42 +02:00
Yolanda Robla
5b305ffa7b Only unmount directories that are mounted
To avoid failures with double unmount, skip unmounting
the mountpoints that are managed by block device.

Change-Id: I228779eb9bf544a27a53e5017c87573023fd375a
2017-05-17 08:37:25 +02:00
OpenStack Proposal Bot
7cfc20eabc Updated from global requirements
Change-Id: Idac484e11d270b3dc2a4b4d97820f105270a7eaa
2017-05-17 03:47:03 +00:00
Jenkins
1c0a5d995a Merge "Apply setfiles on all mountpoints" 2017-05-16 08:10:25 +00:00
Yolanda Robla
6d0b9abc0f Apply setfiles on all mountpoints
With new block device definition, where content of the image
can be mounted on different partitions, is not enough with
executing setfiles on root directory. Instead of that, expose
all the mountpoints on the image, and apply setfiles on them.

Change-Id: I153f979722eaec49eab93d7cd398c5589b9bfc44
2017-05-16 07:51:48 +02:00
OpenStack Proposal Bot
72117965df Updated from global requirements
Change-Id: I27d0dbbc14a3d673d9449d0ed8a6f0c52e10257d
2017-05-15 00:42:53 +00:00
Andreas Florath
e4e23897a1 Refactor: block-device filesystem creation, mount and fstab
This patch finalizes the block device refactoring.  It moves the three
remaining levels (filesystem creation, mount and fstab handling) into
the new python module.

Now it is possible to use any number of disk images, any number of
partitions and used them mounted to different directories.

Notes:

 * unmount_dir : modified to only unmount the subdirs mounted by
   mount_proc_sys_dev().  dib-block-device unmounts
   $TMP_MOUNT_PATH/mnt (see I85e01f3898d3c043071de5fad82307cb091a64a9)

Change-Id: I592c0b1329409307197460cfa8fd69798013f1f8
Signed-off-by: Andreas Florath <andreas@florath.net>
Closes-Bug: #1664924
2017-05-12 13:52:02 +02:00
Ian Wienand
b9c065de28 Remove args from BlockDevice() init
The args agument was only used to find the symbol for the getval
command.  Have the command pass the symbol to find in directly.  We
can therefore remove the args paramater to the BlockDevice() creation.

Change-Id: I8e357131b70a00e4a2c4792c009f6058d1d5ae9e
2017-05-12 09:36:23 +10:00
Ian Wienand
c74ba2fe74 Move to subparsers
Move argument parsing to subparsers, rather than positional arguments.
This better reflects the tool's role as a driver and allows
sub-commands to deal with arguments in a natural way.

Change-Id: Iae8c368e0f3fe47abfddb9e0a1558bd5b3423aee
2017-05-11 21:03:33 +10:00
Jenkins
b39b5cb5a5 Merge "Clear __init__.py from cmd move" 2017-05-11 10:35:22 +00:00
Ian Wienand
f76d68020b Clear __init__.py from cmd move
I accidentally dropped the clearing of this file when it moved to
cmd.py during rebase of I1919f6e865acae14ee95cd025c9c7b75ca266a9c

Change-Id: Ibe9fcde594770cb51c732cc253987308dc038083
2017-05-11 18:52:14 +10:00
Ian Wienand
0368052a2f Take --params from environment
DIB_BLOCK_DEVICE_PARAMS_YAML should be exported, and the
dib-block-device will take this as the value of --params.  Remove this
to simplify the command-line

Change-Id: I6764ed223ecd36f9d24e19f164b6a927380b410f
2017-05-11 17:36:51 +10:00
Ian Wienand
ff5b30db8a Create BlockDeviceCmd object
This creates a BlockDeviceCmd object to hold the main() function.
This doesn't really do anything different right now, but sets a base
for using argparse subparsers to handle the command-line

Change-Id: I4acf95ff4d554a3b4e7e2244ab1706631b98458f
2017-05-11 15:48:47 +10:00
Ian Wienand
a06c610a8c Move YAML parsing into cmd.py; default to env
This moves the YAML parameter parsing into the command-line driver.
It makes the argument optional so it can be taken from the environment
variable directly.  The parsed YAML is passed to the BlockDevice
object.

Change-Id: I6fa5e5b7d1fccfc7cf47d6e4a1fa6e560734680d
2017-05-11 15:22:41 +10:00
Ian Wienand
a9d8e51ec2 Move dib-block-device implementation into cmd.py
Move the command-line driver components into cmd.py.  No functional
change.

Change-Id: I1919f6e865acae14ee95cd025c9c7b75ca266a9c
2017-05-11 15:22:41 +10:00
Ian Wienand
47140293b6 Move blockdevicesetupexception.py into blockdevice.py
Less is more when it comes to code :)

Change-Id: I925ed62bdc808f0e07862f6e0905e80b50fbe942
2017-05-11 15:22:41 +10:00
Jenkins
fae8484e6b Merge "block_device: reorder imports" 2017-05-11 04:53:40 +00:00
Jenkins
bd7d7b052a Merge "Remove unused val_else_none" 2017-05-11 03:24:35 +00:00
Ian Wienand
9a8184ab4c Remove unused val_else_none
This function is unused

Change-Id: I8fe3e5452b95a639618a37a02f34b5b9f63ca43a
2017-05-11 11:53:08 +10:00
Ian Wienand
2a185ec6b6 block_device: reorder imports
Reorder imports to hacking standard (stdlib, third-party, project) [1]

[1] https://docs.openstack.org/developer/hacking/#import-order-template

Change-Id: I4aa73321e1e796ef6b8b079e42f90bf5c75388fe
2017-05-11 10:38:55 +10:00
Ian Wienand
0d8c4270c0 exec_sudo: check cmd for str, log output and raise exception
To avoid any confusion, commands passed to exec_sudo() should be a
list of "str"s.  Log a message if we see unicode issues.

This also adds a debug trace of all output.  stderr is captured.

This is modified to raise CalledProcessError on failure, like
check_call().  Calls that are ok to fail will need to explicitly catch
and ignore this.

The two calls that we expect to fail are wrapped

We wish to try rolling back if one of these command raises an
exception.  Modify the create handler to initiate rollback on all
exceptions.

Change-Id: Iee4fa41ffaf243e4728bf3a5eeec5c8fa8d2dadc
2017-05-11 09:45:25 +10:00
Ian Wienand
6f90daca7f Use check_output
The await function is essentially a non-standard check_output call.
Let's use standard calls to increase maintainability.

Change-Id: I2c25e1cd7122791fcaa86b46bd801e661471bc9e
2017-05-09 15:49:39 +10:00
Yolanda Robla
59a1fc6546 Add sort_mount_point method
As this method can be introduced without any dependency,
provide it on an independent change to simplify reviews.
This is a partial refactor based on
I592c0b1329409307197460cfa8fd69798013f1f8

Change-Id: Idaf3d2b3b3e23d0b9d6bc071d67b961a829ae422
Co-Authored-By: Andreas Florath <andreas@florath.net>
2017-05-05 17:29:53 +00:00
Yolanda Robla
c3dda92b67 Refactor documentation on image creation
Add the bits of documentation that talk about image creation
at the scope of level 1.
This is a partial refactor of change
I592c0b1329409307197460cfa8fd69798013f1f8

Change-Id: I2619c9ebf3ecfeea67fe9063a169d8324d7ffdf2
Co-Authored-By: Andreas Florath <andreas@florath.net>
2017-05-05 17:29:42 +00:00
Yolanda Robla
fb70a49ba2 Add a test to validate we can build debian vms
This is a partial refactor from change
I592c0b1329409307197460cfa8fd69798013f1f8

Change-Id: I8822e68e41c4ebd47eea9ffed4557efc130a7bf7
Co-Authored-By: Andreas Florath <andreas@florath.net>
2017-05-05 19:17:39 +02:00
Yolanda Robla
9b75eda51a Introduce exec_sudo command
Add a new method in the block device library called
exec_sudo, so it can be reused.

This is a partial refactor of change
I592c0b1329409307197460cfa8fd69798013f1f8

Change-Id: Id621f6d029e1275a35c4fd3f19b57c8518076134
Co-Authored-By: Andreas Florath <andreas@florath.net>
2017-05-05 16:11:41 +02:00
Adam Harwell
196b44f38a Fix py3 error in block-device
This is breaking the octavia py3 gates.
Introduced by: I7aa4fe0466e44846d8fa3194575d446fe4b5b2e6

Change-Id: I3d86482a2999197a60a81d42afc5ef7a6e71e313
2017-05-04 15:55:22 -07:00
Yolanda Robla
e21935626b Refactor block-device base functions.
As part of the final steps, refactor the bits belonging
to block device and functions. This is a partial refactor
from I3600c6a3d663c697b59d91bd3fbb5e408af345e4

Change-Id: I7aa4fe0466e44846d8fa3194575d446fe4b5b2e6
Co-Authored-By: Andreas Florath <andreas@florath.net>
2017-05-04 19:54:18 +00:00
Jenkins
02d4f667ff Merge "Add refactor of tree-like vs graph" 2017-05-04 18:51:14 +00:00
Jenkins
20ec14c996 Merge "Add bzip2 to test install" 2017-05-04 06:29:40 +00:00
Ian Wienand
d129a9d19a Add bzip2 to test install
I think this has gone missing on test nodes during unpuppeting.  Add,
as the gentoo job grabs some bz2 tarballs for extraction.

Change-Id: Icb82930702124f7bf112b03a469d90d4163d398b
2017-05-04 14:59:14 +10:00
Yolanda Robla
08c36e4bf8 Add refactor of tree-like vs graph
Introducing the refactors of the block device to allow a tree-like
configuration, and start using it for the partitions level.
Based on patch I3600c6a3d663c697b59d91bd3fbb5e408af345e4

Change-Id: I58bb3c256a1dfd100d29266571c333c2d43334f7
Co-Authored-By: Andreas Florath <andreas@florath.net>
2017-05-03 05:27:43 +00:00
Ian Wienand
91ba21dd6c Set LC_ALL in disk-image-create
It seems that the redhat nodepool job is quite reliably geting a
"floating point" error during centos image build.  This happens after
03-yum-cleanup which is pruning the locales.  This might be a
red-herring, since the logs are full of

  /bin/bash: warning: setlocale: LC_ALL: cannot change locale (C.UTF-8)

I think in our recent de-puppetisation of hosts, something might have
changed that is setting LC_ALL=C.UTF-8 for the jenkins user, at least
on Ubuntu.  This is a problem for centos, as it doesn't have C.UTF-8
locale.  I then think using the invalid locale is what leads the the
floating-point error when doing some maths in dib-run-parts to
calculate runtimes.

We are currently overriding LANG, but we really want LC_ALL to ensure
this applies globally.

Change-Id: I8e7cae093c4b32e0d20b73ae0086f14c7cc6a9cb
2017-05-03 11:31:11 +10:00
Yolanda Robla
340451ea00 Improve documentation for image creation
Add the tree-like vs complete digraph configuration
for images. This is a partial refactor from
I3600c6a3d663c697b59d91bd3fbb5e408af345e4

Change-Id: Ia7a8321e63d59771fe47d8e262b9aacffd60d8d9
Co-Authored-By: Andreas Florath <andreas@florath.net>
2017-05-01 15:22:35 +00:00
Jenkins
629b81bc9b Merge "Refactor block_device: isolate the getval call" 2017-05-01 14:45:13 +00:00
Yolanda Robla
943f1ccf04 Refactor block_device: isolate the getval call
Add a new getval call that allows to retrieve values
from the block device. Also isolating the block device
information into a 'blockdev' dictionary entry, to better
return it with the getval command.

This is a refactor from the original code at
I3600c6a3d663c697b59d91bd3fbb5e408af345e4.

Change-Id: I93d33669a3a0ae644eab9f9b955bb5a9470cadeb
Co-Authored-By: Andreas Florath <andreas@florath.net>
2017-05-01 12:22:52 +02:00
Jenkins
83ea4a17b8 Merge "Clean out apt index caches at end of image build" 2017-05-01 08:09:07 +00:00
Jenkins
ead4a50c2a Merge "Make Gentoo package updates work more often" 2017-05-01 08:05:04 +00:00
Jenkins
c36ac99458 Merge "Turn off strict_id mode for Ec2 datasource" 2017-05-01 08:04:52 +00:00
Andreas Florath
803d40b0c6 Refactor block_device: passing command line parameters
The original approach was to pass each and every command
line parameter to the block device.  While the block device
functionality gets extended, this is not any longer practical.

Instead of passing in all the parameters separately this patch
collects these in a YAML file that is passed in to the block device
layer.

Change-Id: I9d07593a01441b62632234468ac25a982cf1a9f0
Signed-off-by: Andreas Florath <andreas@florath.net>
2017-04-23 09:51:18 +00:00
Matthew Thode
6f51fbe355
Make Gentoo package updates work more often
Some package updates are more complex and require things like --backtrack=99 to
be passed to emerge.  We also try harder to ensure the system is in a consistent
state as a last step.

Change-Id: Ia5d3514e8b2a6cb2d656ade997cebb798d9c0a47
2017-04-22 19:34:11 -05:00
Paul Belanger
1ce16a987b
Add yum-utils as EPEL dependency
With 8e822768f9 we added the ability to
disable the EPEL repository, however we need yum-utils to use
yum-config-manager.

Change-Id: Iea445f84494fd9a89fd93e9b35f920eb5e55211d
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2017-04-20 17:01:00 -04:00
Simon Leinen
0ff40cf63c Turn off strict_id mode for Ec2 datasource
Recent changes in the default configuration of cloud-init in Ubuntu
cause warnings when the Ec2 datasource is used on non-Amazon clouds,
see https://bugs.launchpad.net/cloud-init/+bug/1660385

We explicitly select the previous behavior when an Ec2 datasource is
desired.

Change-Id: Iebad8f6c0017fe08013dd5fe667c6132158b71cd
Closes-bug: 1683038
2017-04-19 09:47:56 +02:00
OpenStack Proposal Bot
f3cfeab5ec Updated from global requirements
Change-Id: I09b47aa75008bfd832ec8eec9108501f18223875
2017-04-12 04:10:50 +00:00
Ian Wienand
7a962e9d1c Skip python3-virtualenv on <= trusty
If DIB_PYTHON_VERSION is < 3 on the !redhat path, that means we're on
an older platform that may not have python3-virtualenv packages.  Skip
install.

Ensure the order of operations happens by forcing the installs

Also add a note about limited platform support (patches welcome :)

Change-Id: I18412767f0ebf946d557a0a126285369e96af159
2017-04-12 06:36:20 +10:00
Ian Wienand
79d4113cbe pip-and-virtualenv : install python2 & 3, and default to 2
Recent changes in project-config have shown that we leave the system
in an inconsistent state when installing from source.  On fedora, we
will have installed the python2 packages, but then used $DIB_PYTHON to
install python3 pip from source!

This tries to clarify the situation.  As described in the document,
with package installs, we just install the $DIB_PYTHON packaged
versions.

Source installs want to take over the global namespace.  This is the
price you pay for running the latest versions outside package managers
:) The only sane thing seems to be for us to normalise python2 &
python3 versions of pip, setuptools and virtualenv and then hacking
things such that "/usr/bin/pip" and "/usr/bin/virtalenv" remain
defaulted to python2 versions.

Documentation is added

Change-Id: Ibc6572b89e256d1f48b7fe7c672b8b9524dc704f
2017-04-11 18:59:11 +10:00
Ian Wienand
ffd4820d59 Install pip with python interpreter
Currently we install pip/virtualenv with "/usr/local/bin/dib-python".
This means that every time you create a virtualenv, the python
interpreter inside it is called "dib-python" which is confusing.

Add an env var DIB_PYTHON that points directly the to interpreter
available during build, for use when running scripts.

Change-Id: I88ad3c9eb958d58db4631d9b27bc2c592f970345
2017-04-11 18:59:09 +10:00