Commit graph

24 commits

Author SHA1 Message Date
Jenkins
1324f5b7db Merge "Remove use of 'which'." 2017-06-11 09:30:34 +00:00
Ian Wienand
6c394f5746 Pass all blockdevices to bootloader
Currently we only export "image-block-device" which is the loopback
device (/dev/loopX) for the underlying image.  This is the device we
install grub to (from inside the chroot ...)

This is ok for x86, but is insufficient for some platforms like PPC
which have a separate boot partition.  They do not want to install to
the loop device, but do things like dd special ELF files into special
boot partitions.

The first problem seems to be that in level1/partitioning.py we have a
whole bunch of different paths that either call partprobe on the loop
device, or kpartx.  We have _all_part_devices_exist() that gates the
kpartx for unknown reasons.  We have detach_loopback() that does not
seem to remove losetup created devices.  I don't think this does
cleanup if it uses kpartx correctly.  It is extremley unclear what's
going to be mapped where.

This moves to us *only* using kpartx to map the partitions of the loop
device.  We will *not* call partprobe and create the /dev/loopXpN
devices and will only have the devicemapper nodes kpartx creates.
This seems to be best.  Cleanup happens inside partitioning.py.
practice.  Deeper thinking about this, and more cleanup of the
variables will be welcome.

This adds "image-block-devices" (note the extra "s") which exports all
the block devices with name and path.  This is in a string format that
can be eval'd to an array (you can't export arrays).

This is then used in a follow-on
(I0918e8df8797d6dbabf7af618989ab7f79ee9580) to pick the right
partition on PPC.

Change-Id: If8e33106b4104da2d56d7941ce96ffcb014907bc
2017-06-08 17:14:22 +10:00
Jesse Keating
84d10dce57 Remove use of 'which'.
Instead, either use the bash built-in of type to ensure it exists. Since
which is an external dep, things can fail oddly in a constrained
environment.

Also add a dib-lint test for this.

Change-Id: I645029f5b5bfe1198c89ce10fd3246be8636e8af
Signed-off-by: Jesse Keating <omgjlk@us.ibm.com>
2017-05-19 12:43:36 -07:00
Yolanda Robla
6d0b9abc0f Apply setfiles on all mountpoints
With new block device definition, where content of the image
can be mounted on different partitions, is not enough with
executing setfiles on root directory. Instead of that, expose
all the mountpoints on the image, and apply setfiles on them.

Change-Id: I153f979722eaec49eab93d7cd398c5589b9bfc44
2017-05-16 07:51:48 +02:00
Andreas Florath
e4e23897a1 Refactor: block-device filesystem creation, mount and fstab
This patch finalizes the block device refactoring.  It moves the three
remaining levels (filesystem creation, mount and fstab handling) into
the new python module.

Now it is possible to use any number of disk images, any number of
partitions and used them mounted to different directories.

Notes:

 * unmount_dir : modified to only unmount the subdirs mounted by
   mount_proc_sys_dev().  dib-block-device unmounts
   $TMP_MOUNT_PATH/mnt (see I85e01f3898d3c043071de5fad82307cb091a64a9)

Change-Id: I592c0b1329409307197460cfa8fd69798013f1f8
Signed-off-by: Andreas Florath <andreas@florath.net>
Closes-Bug: #1664924
2017-05-12 13:52:02 +02:00
Ian Wienand
c74ba2fe74 Move to subparsers
Move argument parsing to subparsers, rather than positional arguments.
This better reflects the tool's role as a driver and allows
sub-commands to deal with arguments in a natural way.

Change-Id: Iae8c368e0f3fe47abfddb9e0a1558bd5b3423aee
2017-05-11 21:03:33 +10:00
Ian Wienand
0368052a2f Take --params from environment
DIB_BLOCK_DEVICE_PARAMS_YAML should be exported, and the
dib-block-device will take this as the value of --params.  Remove this
to simplify the command-line

Change-Id: I6764ed223ecd36f9d24e19f164b6a927380b410f
2017-05-11 17:36:51 +10:00
Yolanda Robla
e21935626b Refactor block-device base functions.
As part of the final steps, refactor the bits belonging
to block device and functions. This is a partial refactor
from I3600c6a3d663c697b59d91bd3fbb5e408af345e4

Change-Id: I7aa4fe0466e44846d8fa3194575d446fe4b5b2e6
Co-Authored-By: Andreas Florath <andreas@florath.net>
2017-05-04 19:54:18 +00:00
Ian Wienand
91ba21dd6c Set LC_ALL in disk-image-create
It seems that the redhat nodepool job is quite reliably geting a
"floating point" error during centos image build.  This happens after
03-yum-cleanup which is pruning the locales.  This might be a
red-herring, since the logs are full of

  /bin/bash: warning: setlocale: LC_ALL: cannot change locale (C.UTF-8)

I think in our recent de-puppetisation of hosts, something might have
changed that is setting LC_ALL=C.UTF-8 for the jenkins user, at least
on Ubuntu.  This is a problem for centos, as it doesn't have C.UTF-8
locale.  I then think using the invalid locale is what leads the the
floating-point error when doing some maths in dib-run-parts to
calculate runtimes.

We are currently overriding LANG, but we really want LC_ALL to ensure
this applies globally.

Change-Id: I8e7cae093c4b32e0d20b73ae0086f14c7cc6a9cb
2017-05-03 11:31:11 +10:00
Yolanda Robla
943f1ccf04 Refactor block_device: isolate the getval call
Add a new getval call that allows to retrieve values
from the block device. Also isolating the block device
information into a 'blockdev' dictionary entry, to better
return it with the getval command.

This is a refactor from the original code at
I3600c6a3d663c697b59d91bd3fbb5e408af345e4.

Change-Id: I93d33669a3a0ae644eab9f9b955bb5a9470cadeb
Co-Authored-By: Andreas Florath <andreas@florath.net>
2017-05-01 12:22:52 +02:00
Andreas Florath
803d40b0c6 Refactor block_device: passing command line parameters
The original approach was to pass each and every command
line parameter to the block device.  While the block device
functionality gets extended, this is not any longer practical.

Instead of passing in all the parameters separately this patch
collects these in a YAML file that is passed in to the block device
layer.

Change-Id: I9d07593a01441b62632234468ac25a982cf1a9f0
Signed-off-by: Andreas Florath <andreas@florath.net>
2017-04-23 09:51:18 +00:00
Jenkins
a076bf66d6 Merge "Move do_extra_package_install to run in install phase" 2017-04-10 01:24:55 +00:00
Noam Angel
95503b4201 Move do_extra_package_install to run in install phase
This change move "do_extra_package_install" from pre-install to install
phase.

Extra packages are added by user request using the flag "-p", This
package should not be something the elements depend on.
The reason behind this patch is to move the extra package install to
a proper phase, Also more reasonable if base element run package update
to be before we install extra packages.

Change-Id: I68cc773aba9aa01743f0dda9f4e635e4cac2a282
2017-04-10 08:32:47 +10:00
Adam Harwell
f06231ee08 Use DIB_PYTHON_EXEC to run commands
DIB_PYTHON_EXEC was added in I5fab0e192c3a2dad8f60e821c184479e24e33bcd
to export the python that disk-image-create is running under.  If dib
is installed with python3 then just calls "python" (python2) to run
sub-scripts, it fails to find itself.

This has been tested in devstack gates that use py3x

Change-Id: Ia1028972bfc0517b468b279aab9decdbcd7424ca
2017-04-07 07:40:58 +10:00
Dan Radez
06576a02f0 Adding aarch64 support for CentOS7
Using delorian and delorian-deps for aarch64
this will produce an overcloud-full image

Change-Id: Id0dc83d9c558b0c0bbfaed7727883541d140472c
2017-03-22 10:46:54 -04:00
Andreas Florath
866a06f92d Refactor: block-device partitioning cleanup
Now that the main partitioning refactor patch is merged, there is
a small relict of handling partitions still in the disk-image-create
main.

This patch moves the functionality from disk-image-create to the
block-device/partitioning module: it is mostly a rewrite of the
original bash code in python.

Change-Id: Ia73baeca74180a7bc9ea487da03ff56d6a3070ce
Signed-off-by: Andreas Florath <andreas@florath.net>
2017-03-07 18:43:09 +11:00
Ian Wienand
7a155e08bf Merge branch 'master' into merge-branch
Change-Id: I28e4c7837d84e8b66eff3d182666c5a87a9e3c9b
2017-02-09 13:35:53 +11:00
Ian Wienand
adf39c52cf Release 1.21.1
-----BEGIN PGP SIGNATURE-----
 
 iQEcBAABAgAGBQJYW2GoAAoJEBty/58O8cX8uSAH/15dJsglP6Zie7jSSJcR6k+e
 PJembHn9qrqrCjmJ5EwakojySaaLhwEJKvlP54OU9v7pmUXL9gJtK2OzW54LQ41g
 xBHIu0Pg4z7juyHm9+1P2Sr7Mzs1pVSEbsIYpDYUU19eghI1EAeIj3I1woKgajN7
 JlI61j3r67G6EAVtPOnmD1jvXS8CrtjiJ9wtWTH20pWfmksovg/GuXUCZrLkAAhO
 NcK35CdMii1Hkr7XOH424La/Ar+3qfUX18ZkbJY6yHzkq/ityTzzKOFjAaDl2Jg9
 WNc+SLCVYpPhPwgt7miTywamUNj3ZviA5/Hd8fuLXmtHSLQ23WOtBiaQMLtwXHs=
 =8dIl
 -----END PGP SIGNATURE-----

Merge tag '1.26.1' into merge-branch

Release 1.21.1

Change-Id: Ib9eb3dd1d384fc5b6a9846608216e056c57a173a
2017-02-02 20:36:23 +11:00
Ian Wienand
bfca36c772 Release 1.25.2
-----BEGIN PGP SIGNATURE-----
 
 iQEcBAABAgAGBQJYV1yqAAoJEBty/58O8cX8hLwIAKP66w6MdPN8PDgUOteui/Sx
 N0UFKJ9yR4GQOAP0NffPLjch5/g0iJLs3eFKOhtGC1LjbDjpVgjX8vW18ib8wBZK
 GemOZPF3uxg8FROrZF1vpoDy/cHgL1YV10hCnwdjN/r9rb8zOuSabqjW+Dennj2n
 fZ0SJfa8Owfudn3YxGuOymVb/wMtEloDmVGBEI1Y+h7osELCCDi3OXmwsA8qMsdl
 cTwbeugBs4PlOVbZUK/JKGuwIHKgPnDYzYu5KpXw77/MdjGT0fo5Tlq5AOBDI2sC
 9JOFEBDli4Ro05VwvI58ADMpvvOax+9EvOhLbB1dRPdZl21Iyb6gOdy2PUbFO0c=
 =aKxq
 -----END PGP SIGNATURE-----

Merge tag '1.25.2' into merge-branch

Release 1.25.2

Change-Id: I698bcf2e82117bd81649cd065a7af5cac85990c7
2017-02-02 11:20:00 +11:00
Andreas Florath
ec7f56c1b2 Refactor: block-device handling (partitioning)
During the creation of a disk image (e.g. for a VM), there is the need
to create, setup, configure and afterwards detach some kind of storage
where the newly installed OS can be copied to or directly installed
in.

This patch implements partitioning handling.

Change-Id: I0ca6a4ae3a2684d473b44e5f332ee4225ee30f8c
Signed-off-by: Andreas Florath <andreas@florath.net>
2017-01-24 19:59:10 +00:00
Ian Wienand
5adfc9365a Fix --version display
We do not have the concept of "not installed" in v2, so remove the
obsolete code looking for a now non-existent variable.

Also, log the version at startup.  This can help when
debugging from logs

Change-Id: I964c4cf207c10666afc5bc7ab9f2bfb9b1897c1e
2016-12-01 04:37:04 +00:00
Ian Wienand
7d5afecfd9 Merge remote-tracking branch 'origin/master' into merge-branch
Change-Id: Ibab1bb95521292ae818bd91f7073c3749a2cc0cb
2016-11-18 13:53:56 +11:00
Ian Wienand
3c6e589226 Remove execute perm from disk-image-create
Remove the x bit from lib/disk-image-create; because it's called
directly by the entry-point, it doesn't need to be exectuable.
This should also be clearer that you're not supposed to run it
by hand.

Remove some boilerplate from old file

Change-Id: Ibb6cdae613e6c9cf21dd6aecc8e1f739bc3a2643
2016-11-04 17:11:27 +11:00
Ian Wienand
7446c32197 Move diskimage-image-create to an entry point
It has always been a weird thing that dib is a python package, but
is totally driven by the disk-image-create script.  It creates this
strange division that is hard to explain.

This moves disk-image-create to a regular python entry-point

Currently, this simply exec()s the original disk-image-create script.

However, we now have a (private) interface between disk-image-create
written in python and the driver shell script.  Here's some things we
could do, for example:

* Argument parsing is generally nicer in Python, and then end result
  is mostly just setting environment variables to flag different things
  in the shell script.  I could see us moving the argument-parsing into
  diskimage_builder.disk_image_create:main() and just setting things in
  os.environ before the exec()).

* I7092e1845942f249175933d67ab121188f3511fd sets IMAGE_ELEMENT_YAML in
  disk-image-create by calling-back to element-info.  We can just call
  element_dependencies.find_all_elements() in here an export is to
  os.environ before disk-image-create starts.

* remove need for ramdisk-image-create symlink by just exporting
  IS_RAMDISK based on sys.argv[1] value

* you could even unit test some of this :)

Change-Id: I69ca3d26fede0506a6353c077c69f735c8d84d28
2016-11-02 05:12:11 +00:00
Renamed from bin/disk-image-create (Browse further)