I think that generally this is a lot of noise in the logs, as the
internals of cache-url is well tested, so we don't need to trace log
by default.
Change-Id: I25b5a1ec0d8f99691b2b4b62b9fdd537e5a773e4
curl's "-v" is a bit too verbose for "-x", especially when what you're
downloading bounces through a few redirects as is common. Turn this
down and put it behind "-xx" or greater.
Change-Id: I6d91166bb237f2a1818cae7532e794ef0f01288b
When there is a hashsum mismatch diskimage-builder forces downloads
with the -f switch of cache-url. This is currently broken because bash
escapes the quotes in curl_opts. This tricks curl trying to download
'no-cache' instead of the url. This can be fixed by using an array for
curl_opts which does the correct thing here.
Change-Id: Id9f1579dda9a3e0a2b08dd5faaeef0e2e580d419
Avoids failing on the first attempt to download the image to cache as
mirrors hosting them can randomly go down, usually with a connection
refused.
Change-Id: I9de9f33c2cc16596d04b35c4eb92621e6a2c7511
It looks like we dropped running these probably when we moved the
elements around. For testtools to find the test scripts we need to
add the __init__.py files to make the directories look like modules.
Also prevent copying any .pyc or cache files in as hooks.
Change-Id: I66d5f6ee62cc4d9ee14c64e819b4db57d035d09f
In some cases cache-url can get pulled in without curl, causing it to
fail.
Co-Authored-By: Adam Harwell <flux.adam@gmail.com>
Change-Id: Ibd66c2ca4f8cc423783555d8a99b1184f43adff2
When a download redirector redirects to a broken mirror, timeout
quickly rather than waiting until the overall job is being timed out.
Change-Id: If7eb63d406aaf61f71aa9203cf708c474aa63fd0
Currently we have all our elements and library files in a top-level
directory and install them into
<root>/share/diskimage-builder/[elements|lib] (where root is either /
or the root of a virtualenv).
The problem with this is that editable/development installs (pip -e)
do *not* install data_files. Thus we have no canonical location to
look for elements -- leading to the various odd things we do such as a
whole bunch of guessing at the top of disk-image-create and having a
special test-loader in tests/test_elements.py so we can run python
unit tests on those elements that have it.
data_files is really the wrong thing to use for what are essentially
assets of the program. data_files install works well for things like
config-files, init.d files or dropping documentation files.
By moving the elements under the diskimage_builder package, we always
know where they are relative to where we import from. In fact,
pkg_resources has an api for this which we wrap in the new
diskimage_builder/paths.py helper [1].
We use this helper to find the correct path in the couple of places we
need to find the base-elements dir, and for the paths to import the
library shell functions.
Elements such as svc-map and pkg-map include python unit-tests, which
we do not need tests/test_elements.py to special-case load any more.
They just get found automatically by the normal subunit loader.
I have a follow-on change (I69ca3d26fede0506a6353c077c69f735c8d84d28)
to move disk-image-create to a regular python entry-point.
Unfortunately, this has to move to work with setuptools. You'd think
a symlink under diskimage_builder/[elements|lib] would work, but it
doesn't.
[1] this API handles stuff like getting files out of .zip archive
modules, which we don't do. Essentially for us it's returning
__file__.
Change-Id: I5e3e3c97f385b1a4ff2031a161a55b231895df5b
