Commit Graph

1288 Commits

Author SHA1 Message Date
Jenkins
cc3a28755d Merge "Move dib-run-parts into diskimage-builder" into feature/v2 2016-11-15 00:43:49 +00:00
Ian Wienand
3c6e589226 Remove execute perm from disk-image-create
Remove the x bit from lib/disk-image-create; because it's called
directly by the entry-point, it doesn't need to be exectuable.
This should also be clearer that you're not supposed to run it
by hand.

Remove some boilerplate from old file

Change-Id: Ibb6cdae613e6c9cf21dd6aecc8e1f739bc3a2643
2016-11-04 17:11:27 +11:00
Ian Wienand
d65678678e Move dib-run-parts into diskimage-builder
Move dib-run-parts from dib-utils into diskimage-builder directly.

For calling outside the chroot, we provide a standard entry-point
script.  However, as noted in the warning comment, the underlying
script is still copied directly into the chroot by the dib-run-parts
element.  I believe this to be the KISS approach.

This removes the dependency on dib-utils.  We have discussed this
previously and nobody seemed to think retiring dib-utils was going to
be an issue.

This also updates the documentation to not mention dib-utils, or using
disk-image-create via $PATH setup, but rather gives instructions on
installing from pip with a virtualenv.

Change-Id: Ic1e22ba498d2c368da7d72e2e2b70ff34324feb8
2016-11-04 17:07:37 +11:00
Ian Wienand
7446c32197 Move diskimage-image-create to an entry point
It has always been a weird thing that dib is a python package, but
is totally driven by the disk-image-create script.  It creates this
strange division that is hard to explain.

This moves disk-image-create to a regular python entry-point

Currently, this simply exec()s the original disk-image-create script.

However, we now have a (private) interface between disk-image-create
written in python and the driver shell script.  Here's some things we
could do, for example:

* Argument parsing is generally nicer in Python, and then end result
  is mostly just setting environment variables to flag different things
  in the shell script.  I could see us moving the argument-parsing into
  diskimage_builder.disk_image_create:main() and just setting things in
  os.environ before the exec()).

* I7092e1845942f249175933d67ab121188f3511fd sets IMAGE_ELEMENT_YAML in
  disk-image-create by calling-back to element-info.  We can just call
  element_dependencies.find_all_elements() in here an export is to
  os.environ before disk-image-create starts.

* remove need for ramdisk-image-create symlink by just exporting
  IS_RAMDISK based on sys.argv[1] value

* you could even unit test some of this :)

Change-Id: I69ca3d26fede0506a6353c077c69f735c8d84d28
2016-11-02 05:12:11 +00:00
Ian Wienand
97c01e48ed Move elements & lib relative to diskimage_builder package
Currently we have all our elements and library files in a top-level
directory and install them into
<root>/share/diskimage-builder/[elements|lib] (where root is either /
or the root of a virtualenv).

The problem with this is that editable/development installs (pip -e)
do *not* install data_files.  Thus we have no canonical location to
look for elements -- leading to the various odd things we do such as a
whole bunch of guessing at the top of disk-image-create and having a
special test-loader in tests/test_elements.py so we can run python
unit tests on those elements that have it.

data_files is really the wrong thing to use for what are essentially
assets of the program.  data_files install works well for things like
config-files, init.d files or dropping documentation files.

By moving the elements under the diskimage_builder package, we always
know where they are relative to where we import from.  In fact,
pkg_resources has an api for this which we wrap in the new
diskimage_builder/paths.py helper [1].

We use this helper to find the correct path in the couple of places we
need to find the base-elements dir, and for the paths to import the
library shell functions.

Elements such as svc-map and pkg-map include python unit-tests, which
we do not need tests/test_elements.py to special-case load any more.
They just get found automatically by the normal subunit loader.

I have a follow-on change (I69ca3d26fede0506a6353c077c69f735c8d84d28)
to move disk-image-create to a regular python entry-point.

Unfortunately, this has to move to work with setuptools.  You'd think
a symlink under diskimage_builder/[elements|lib] would work, but it
doesn't.

[1] this API handles stuff like getting files out of .zip archive
modules, which we don't do.  Essentially for us it's returning
__file__.

Change-Id: I5e3e3c97f385b1a4ff2031a161a55b231895df5b
2016-11-01 17:27:41 -07:00
Gregory Haynes
eff793fc2f Remove deprecated expand-dependencies arg
This argument has been deprecated in element-dependencies for some time.
Removing in v2.

Change-Id: I9e40ec7bef7afbebba3958043db46975ea7b24ff
2016-10-06 16:14:32 +00:00
Ian Wienand
91b431ce78 Move element-info to a standard entry-point
Move element-info from a wrapper script to a standard entry-point
console_script.

Update the documentation to explain how to run it for development.  I
don't think we should support the idea that you can check-out the code
and run ./bin/disk-image-create -- it has dependencies (dib-utils,
etc) and needs to be run from a virtualenv (this is what CI in the
gate does).  A follow-up can clean-up some of the path munging stuff
we have for this in disk-image-create.

Change-Id: Ic0c03995667f320a27ac30441279f3e6abb6bca8
2016-09-08 15:29:56 +10:00
Andreas Florath
3d48a528c1 Refactor: block-device handling (local loop)
Block device handling can be somewhat complex - especially
when taking things like md, lvm or encryption into account.

This patch factors out the creation and deletion of the local
loop image device handling into a python library.

The main propose of this patch is to implement the needed
infrastructure.  Based on this, more advanced functions can be added.
Example: (advanced) partitioning, LVM, handling different boot
scenarios (BIOS, UEFI, ...), possibility of handling multiple images
(local loop image, iSCSI, physical hard disk, ...), handling of
different filesystems for different partitions / LVs.

Change-Id: Ib626b36a00f8a5dc3dbde8df3e2619a2438eaaf1
Signed-off-by: Andreas Florath <andreas@florath.net>
2016-09-08 04:31:01 +00:00
Ian Wienand
37a53354ec Add IMAGE_ELEMENT_YAML and get_image_element_array
These new variables are a list of elements chosen for the build along
with their full paths.  For Python elements, IMAGE_ELEMENT_YAML is a
YAML formatted list that can be easily parsed.  For bash elements,
"get_image_element_array" will produce an associative-array of the
same (working around lack of array export in Bash).

This list is intended for consumption of elements who need to copy
files from other elements, such as pkg-map and svc-map.  As discussed
in I2a29861c67de2d25c595cb35d850e92807d26ac6, this list has already
been pruned and had overrides processed, so it is safe to simply walk
over this list with no further processing.

Since we're presenting the element list in a couple of different ways,
we combine it all into the element-info script.  It will output an
eval-able string that declares the appropriate variables.

I've added some inline documentation so they still appear in grep.
The documentation is updated with examples, and moved to a more
appropriate location as a sub-section of the element sytle guide.

To test this out, use the associative-array in generate_hooks, where
we can now find the element's directory without searching.

Change-Id: Ibbd07d082ec827441def2d3f6240df3efdc6eae3
2016-09-08 11:08:07 +10:00
Ian Wienand
274be6de55 Making element overriding explicit
This is a re-factor of element_dependencies to achieve two things --
centralising override policy and storing path names.

Firstly we want to make the override policy for elements completely
explicit.  Currently, elements that wish to copy parts of other
elements walk ELEMENTS_PATH themselves and look for elements in
IMAGE_ELEMENT.  How they handle duplicate elements can differ, leading
to inconsistent behaviour.

We introduce logic in element-info to find elements in each of the
directories in ELEMENT_PATHS in *reverse* order -- that is to say,
earlier entries in the paths will overwrite later ones.

For example

 ELEMENT_PATHS=foo:bar:baz

will mean that "foo/element" will override "baz/element", since "foo"
is first.  This should be sane to anyone familiar with $PATH.
Documentation is clarified around this point and a test-case is added.

The second thing is that we want to keep the complete path of the
elements we have chosen.  We want the aforementioned elements that
walk the element list to use these canonical paths to pickup files;
this way they don't need to make local decisions about element
overrides, but can simply iterate a list and copy/merge files if they
exist.

A follow-on change (I7092e1845942f249175933d67ab121188f3511fd) will
expose this data in a separate variable that can be parsed by elements
(a further follow-on I0a64b45e9f2cfa28e84b2859d76b065a6c4590f0
modifies the elements to use this information).  Thus this does not
change the status-quo -- elements that are walking ELEMENTS_PATH
themselves and can/will continue doing that.

Change-Id: I2a29861c67de2d25c595cb35d850e92807d26ac6
2016-09-08 10:58:19 +10:00
Ian Wienand
452f7b8d5a Clear up "already provided" message
I got quite confused what this was trying to tell me at first.  It's
saying that you requested an element but another element already
provides that element, so we don't know which one to choose.

To help clarify the situation to the user, keep track of what is
providing elements so we can describe where the conflict came from.

Change-Id: Ie7471ac900a8cbee5684c928badd1b8ce6d3e3cf
2016-07-01 11:53:07 +10:00
Ian Wienand
2b18513cad Convert element_dependencies to logging
Use standard logging module for output.  Add some basic testing of
error messages to the unit-tests.  Use the logging_config module to
setup the logging for interactive use.

Change-Id: Ia23722a7bd00aba336118edb155356a3b3ef6926
2016-06-28 16:02:45 +10:00
Ian Wienand
8661dd1a31 Introspect logging testing more
This adds some fixtures to ensure the log output is what we expect,
along with ensuring we test the operation of the debug environment
variables.

Change-Id: I01c36299539b15a633b9307da5a348a5ae4e2563
2016-06-16 13:54:52 +10:00
Andreas Florath
6ca6057981 Add python logger configuration
This adds a common usable logging configuration for DIB, that
can be used by different python tools or elements.

This change is a factored out patch from the block device
refactoring.

Change-Id: I34d44610f74030936e7c0f602340b183baec3f1b
Signed-off-by: Andreas Florath <andreas@florath.net>
2016-06-16 13:54:50 +10:00
Andreas Jaeger
63eb3ac06e Remove outdated translation files
This repo is not translated by default, so no need to have old
translation files in here that can be regenerated any time.
Remove also the English translation since English is the source
language, there's no need to have an extra translation at all.

Change-Id: I100d7bec0439307ecee5ea977aa6fce2386fa731
2016-01-31 20:31:29 +01:00
Ben Nemec
5ca944f7a0 Bump to hacking 0.10
Change-Id: Ib4f4e5d2e48717a514636cde906dc4f977021cc6
2015-02-11 15:28:28 -06:00
Gregory Haynes
c4bbb6f3bc Create docs site containing element READMEs
We currently do not have the ability to create a docs site which
outlines all the elements.

Change-Id: I77ccf61e0c4b1509b3e7ce9b8f15ea5ccfd50d9b
2015-02-10 11:45:35 -08:00
jodewey
8fc027227a Corrected element-info usage
Deprecated the `--expand-dependencies` flag from `element-info` usage.
The flag was required and not optional.  We can rely on argparse to exit non-0
when the required positional argument is not provided.

Change-Id: Iaf8eb962eb600760974bc33c30b809a07a23278e
Closes-Bug: 1265649
2015-01-29 13:43:48 -08:00
Ben Nemec
58d6c6478a Add unit test for cache-url
A sanity check that cache-url does the right thing when presented
with the appropriate curl return codes.  This change brings in the
test code for scripts from tripleo-image-elements, which needs to
be factored out into a place both projects can use it.

I'm stubbing out curl so we don't have a dep on curl in our unit
tests, and because I've seen some strange behavior out of curl in
the gate that caused random failures in this test.

Change-Id: I31e6b06b45415feec7285511d07e65eb78d0d045
2014-09-30 16:39:21 -05:00
Christian Berendt
3e48883ddc Bump hacking to 0.9.x series
Change-Id: If4a34faafcd37cf68c53525ccf5ec626a9f42fdb
2014-07-21 10:15:41 +02:00
Jenkins
786899ea6d Merge "Add test to ensure we don't duplicate filenames" 2014-06-13 19:40:08 +00:00
Gonéri Le Bouder
1011adf370 fail at startup with no operating-system element
Raise an error if there is no element with “operating-system”
in the element-provides file.

Change-Id: I2242537abc4c610252984c32c286bc6eb25de672
2014-06-12 11:19:20 +02:00
Dan Prince
b2c8ec3259 Add test to ensure we don't duplicate filenames
Add a new test_no_dup_filenames module to help ensure we don't have
duplicate names across elements.

Change-Id: I2af4a614f71fe58eba04e3fc50d3c0a70dd91f71
2014-06-11 08:16:41 -04:00
Monty Taylor
1c07d4ba1a Remove All Rights Reserved
The phrase is no longer needed as of August 23, 2000 with Nicaragua's
joining of the Berne Convention.

Additionally, in at least one instance,
elements/cache-url/bin/cache-url, its existence in the file between
Copyright lines is just weird and feels misleading, even though it is
not.

Remove all of the lines, because sanity.

Change-Id: I24fd76c2b4f66b8036010b5079db39ead729abee
2014-04-25 11:25:10 -07:00
Jon-Paul Sullivan
c357ca6634 Iterate over provided elements first
To ensure that we have the list of provided elements from the
command line before looking up dependencies, use a deque and
popleft() to take the elements on a first-in/first-out basis.

Change-Id: I0a2c21fa081763fd3cc8358be25dafeee1ed4718
Closes-Bug: #1303911
2014-04-07 18:04:00 +01:00
Jan Provaznik
604926b38f Adds "element-provides" support to element dependencies
An element can define a list of elements provided by itself. This allows
using an alternative element and keeping element dependencies at the
same time. This will be needed for example for mariadb element which will be
an alternative to mysql element - mysql dependency can be kept everywhere
but user can choose mariadb explicitly as a replacement by putting "mysql"
into "element-provides" file.

Change-Id: I7f193612f279dd71d9f1f1edfc60009838a00c5a
2014-03-25 15:35:52 +01:00
James Slagle
dee69b1810 Python code refactorings.
This commit refactors the python code in diskimage_builder slightly.  This is
in preparation for a larger review that adds more functionality to the python
code, namely the ability to apply elements to the current system as opposed to
a chroot.  Further, the refactorings can stand on their own for better clarity.
They include:
 - renaming elements.py to element_dependencies.py.  Adds clarity about the
   purpose of this module.
 - updating other code for this rename.
 - move tests into a tests submodule.

Change-Id: I5519cc52398e442b24e33802bae42070d64b0c1d
2013-11-12 10:47:57 -05:00
Ghe Rivero
5265f4cb02 Replace assertEquals with assertEqual
To comply with the new hacking>=0.8.0,<0.9 requirements,
assertEqual is used instead of assertEquals which has been deprecated,
giving the error H234.

Change-Id: I3cbef1353260aec1502114010e12d0e507376389
2013-11-06 09:56:10 +01:00
Jeremy Stanley
a0fa2749b1 Translations license statement correction
* .../locale/diskimage_builder.pot
* .../locale/en/LC_MESSAGES/diskimage_builder.po: Correct the
project name mentioned in the translation comment headers to avoid
downstream licensing confusion.

Change-Id: I6af9f2b3cda7462e21d22ff534e762381e6c73f5
2013-09-20 13:29:34 +00:00
Monty Taylor
6b74b65449 Fix hacking errors
As another step in aligning further with OpenStack practices, stop
ignoring the hacking style checks.

Change-Id: I16c9f0ca3be5790176467377303817249e7643ea
2013-07-27 12:17:02 -04:00
Jenkins
a37f7ca2a4 Merge "Enable Flake8 F*** checks." 2013-06-21 13:44:53 +00:00
Joe Gordon
a98c0fe282 Improve debugging of missing elements.
Print the checked directory when cannot find an element.

Change-Id: I2c1f74e4975e66b6ec9d4122c02067bc2c473620
2013-06-20 14:26:58 -07:00
Joe Gordon
fcd1a2e7fd Enable Flake8 F*** checks.
Fix and enable F*** Checks by removing unused imports.

Change-Id: I39863a4664abff1349911546f16bfea055104e63
2013-06-19 22:42:13 -07:00
Ghe Rivero
308eee2827 Use multiple locations for elements dir.
Using ELEMENTS_DIR env variable, you can specify multiple dirs containint your elements.
It must be a ":" separated elements list.

When an element is seen in one of those lists, it will be chosen and will continue with the next element.

Change-Id: I18eca27d943139cd6ca1ebd232b419e502d7b048
2013-02-27 10:48:55 +01:00
Tim Miller
6cfea30246 Prevent silent failure of element-info:
bin/element-info accesses it's library via a symlink:
  bin/diskimage_builder -> diskimage_builder

This causes the relative path in elements.py to be
incorrect.

element-info silently fails in this situation, because
it allows missing 'element-deps' files, so that these files
may be optional.

This change causes element-info to fail explicitly if $ELEMENTS_DIR
is not set, as it now is when called by diskimage-create, and
adds tests reflecting this behavior.

Change-Id: Iec50f934feb13dfed64d69297a3af6ac9f842677
2013-02-18 15:18:08 -08:00
Clint Byrum
c3ee071d73 Make it possible for openstack-CI to run tests
Per http://wiki.openstack.org/ProjectTestingInterface we enable all
of the commands except build_sphinx because we do not have any sphinx
documentation as of yet. Includes babel support though there are no
properly internationalized strings just yet.

Change-Id: Iae6e6b3f9e605106f4575196fa5527d2187255df
2013-02-04 22:26:17 -08:00
Clint Byrum
c8c33e3bb1 Add a simple implementation of element dependency
This adds a new optional file to the root of elements. The
file lists dependencies which will be added to the list
requested by the user during disk image creation.

Change-Id: Id71c3b333563604bbbaf90f9cf40e24fa9738fc8
2013-01-23 16:04:20 -08:00
Robert Collins
1f39f4c629 Add a test framework for testing elements.
Change-Id: I845ac3ec6bbcd212ee43151981a2bc5264f4a5b3
2012-12-14 20:17:39 +13:00