Commit Graph

2982 Commits

Author SHA1 Message Date
Jamie Lennox
aa03e24c30 Use the dib python to do cleanup
In a system where python2 is not installed and /usr/bin/python is not
linked then the cleanup process will fail trying to invoke the python
script. Use the previously determined DIB_PYTHON_EXEC if it's available.

Change-Id: I128292808ccef92cc1803988b35caae5aa6fa541
2017-06-22 11:20:51 +10:00
Ian Wienand
18a0d970fa Move ironic-agent test to fedora-minimal
I'm uncertain as to why this is using the "fedora" element for testing
... but it requires downloading the fedora .qcow on every test which
has shown to be unreliable.  An easy thing to do is to switch it to
fedora-minimal; that will only involve downloads from local mirrors in
the gate.

Add redhat-rpm-config for minimal.  I admit I have not fully gone
through why this is not pulled in.  It's been an issue since
I459f2203fa145049dda185da952813118193d573 and there's all sorts of
bugs.

Change-Id: I37458e3926dae32a259bd5aa9efc645561b029a0
2017-06-21 15:05:36 +10:00
Ian Wienand
649f0b66d9 Start at using CI mirrors for fedora/centos
fedora/centos-minimal don't obey DIB_DISTRIBUTION_MIRROR currently.  I
don't really want them too -- we want to be able to separate the
mirrors used during the build process from those embedded into the
final image.  Add DIB_YUM_MINIMAL_BOOTSTRAP_REPOS which is a directory
with repo files to use during the install.

This introduces setup-gate-mirrors.sh which is intended to setup
repo/sources/whatever files in the openstack gate that point to the
local region mirror.  It pulls the info from the mirror_info.sh script
on each CI node.

The openstack-ci-mirrors element is updated to export these variables.
elements are updated to depend on it.  Tests are restored

Change-Id: I7604fc4d41cb1483be16b8d628a24e8fc764f515
2017-06-21 12:02:27 +10:00
Ian Wienand
f0b70211c6 Use local mirror for ubuntu-minimal jobs
This adds "openstack-ci-mirrors" element which performs various
settings to get builds using local mirrors.  As a first step, we
convert ubuntu-minimal jobs

The main trick is that since infra mirrors are created with rerepo
they are not signed (they are recreated, not cloned, and not signing
is seen as a feature in that it deters external use).  So we need to
instruct debootstrap to ignore signing and also turn it off for
in-chroot apt.  Other than that, the existing DIB_DISTRIBUTION_MIRROR
works to redirect installs.

Remove "restricted" as it's not mirrored, and I don't think we want it
in here by default.

(I think DIB_DISTRIBUTION_MIRROR is a bit of an anti-pattern, because
it leaves the mirrors in the final image -- just because you use them
to build, doesn't mean you want them at runtime).  But we don't need
to fix that now, and we don't use any created images.)

This pauses fedora testing until the next change, which moves to using
local mirrors for testing on fedora/centos

Change-Id: I778bd05a1e615c27edf1c9f0a1409119a6b3a850
2017-06-21 12:01:31 +10:00
Ian Wienand
0d37351031 Move apt-sources to ubuntu-minimal / move debian to skip list
The gate is currently extremley unstable, and these two issues are
causing most of the problems.  We need to commit them atomically so we
can get anything moving again

---

The gate is very unstable downloading the ubuntu tarballs from
upstream at the moment.  Move this to ubuntu-minimal which, in a later
change will source files from our local mirror.

We need a caching mechanism for these large files to avoid this
instability.  This is future work for the various image-based jobs.

---

Move debian to default skip lists

I don't know if it's mirrors being worked hard for the Stretch
release, but this is constantly failing the gate.  I will move this to
the -nv extras job

I am working on having the voting job use local mirrors for
everything.  Unfortunately debian infra mirrors don't have stretch yet
and we need to do some fiddling to get "stable" available.  Once we
have all this, we can consider making it voting again.

Change-Id: Iaf7b3888ef06c7aef63cbf76a94b33f96bc9c5c2
2017-06-21 10:34:53 +10:00
Ian Wienand
5d5fa06e5c Sync after writing partition table
We introduced the "settle" in
I90103b59357edebbac7a641e8980cb282d37561b thinking that maybe kpartx
had not finished writing the partition.  This probably wasn't a bad
first assumption, since we used to have this -- but is seems
insufficient.

The other failiure here seems to be if kpartx hasn't actually seen the
updated partition table in the image, so it has correctly (in it's
mind) not mounted the partition.

Looking at strace of fdisk run manually on a loopback, it will do a
fsync on the raw device after writing and then a global sync as it
exits.

This replicates this; we flush and fsync in mbr.py in the exit handler
after writing the partition, before closing the file (i've updated one
of the unit tests to double-check the call).  In the partitioning.py
caller we execute a sync call too.

Since it does seem unlikely the "-s" option of kpartx is not working,
I've removed the udev settle work-around too.

Change-Id: Ia77a0ffe4c76854b326ed76490479d9c691b49aa
Partial-Bug: #1698337
2017-06-19 17:13:36 +10:00
Ian Wienand
a0f747932d Install systemd earlier for Stretch
Debian Stretch released as stable recently, and the init system is
less tightly specified in the base dependencies (for some info, see
[1]).  It seems, probably unintentionally, that in the previous
release systemd-sysv was brought in by debootstrap, but that is no
longer happening.

Add systemd as an early dependency of debian-minimal.

Remove the package-installs.yaml as that happens too late (other
things need to know the init system to write out service files, etc
and probe for systemd utils before package-installs).  As mentioned, I
do not believe the "only install systemd on testing" idea was actually
working here, because it was being brought in during the initial
debootstrap.

Update some documentation to explain what's going on

[1] https://lists.debian.org/debian-boot/2015/05/msg00156.html

Change-Id: Id67c0cf08728407d234976f9807d3bd71d12f758
2017-06-19 13:27:33 +10:00
Michael Johnson
250aeb5d21 Fix mkfs failure when loop device is not ready
There was a race in diskimage-builder where the mkfs call after a
kpartx -avs for the loop device would fail because the device was
not yet ready.  This adds a udevadm settle call after the kpartx
to make sure the udev event queue has cleared.

Change-Id: I90103b59357edebbac7a641e8980cb282d37561b
Closes-Bug: #1698337
2017-06-17 09:00:13 +10:00
Jenkins
1324f5b7db Merge "Remove use of 'which'." 2017-06-11 09:30:34 +00:00
Ian Wienand
5ac8a98e9a PPC bootloader; install to boot partition
Using the newly exposed variables from the prior change, install the
ppc bootloader to the boot partition, not the underlying loopback
device.

Change-Id: I0918e8df8797d6dbabf7af618989ab7f79ee9580
2017-06-08 17:14:22 +10:00
Ian Wienand
6c394f5746 Pass all blockdevices to bootloader
Currently we only export "image-block-device" which is the loopback
device (/dev/loopX) for the underlying image.  This is the device we
install grub to (from inside the chroot ...)

This is ok for x86, but is insufficient for some platforms like PPC
which have a separate boot partition.  They do not want to install to
the loop device, but do things like dd special ELF files into special
boot partitions.

The first problem seems to be that in level1/partitioning.py we have a
whole bunch of different paths that either call partprobe on the loop
device, or kpartx.  We have _all_part_devices_exist() that gates the
kpartx for unknown reasons.  We have detach_loopback() that does not
seem to remove losetup created devices.  I don't think this does
cleanup if it uses kpartx correctly.  It is extremley unclear what's
going to be mapped where.

This moves to us *only* using kpartx to map the partitions of the loop
device.  We will *not* call partprobe and create the /dev/loopXpN
devices and will only have the devicemapper nodes kpartx creates.
This seems to be best.  Cleanup happens inside partitioning.py.
practice.  Deeper thinking about this, and more cleanup of the
variables will be welcome.

This adds "image-block-devices" (note the extra "s") which exports all
the block devices with name and path.  This is in a string format that
can be eval'd to an array (you can't export arrays).

This is then used in a follow-on
(I0918e8df8797d6dbabf7af618989ab7f79ee9580) to pick the right
partition on PPC.

Change-Id: If8e33106b4104da2d56d7941ce96ffcb014907bc
2017-06-08 17:14:22 +10:00
Ian Wienand
1d1e4ccb3e Move rollback into NodeBase object
Currently we pass a reference to a global "rollback" list to create()
to keep rollback functions.  Other nodes don't need to know about
global rollback state, and by passing by reference we're giving them
the chance to mess it up for everyone else.

Add a "add_rollback()" function in NodeBase for create() calls to
register rollback calls within themselves.  As they hit rollback
points they can add a new entry.  lambda v arguments is much of a
muchness -- but this is similar to the standard atexit() call so with
go with that pattern.  A new "rollback()" call is added that the
driver will invoke on each node as it works its way backwards in case
of failure.

On error, nodes will have rollback() called in reverse order (which
then calls registered rollbacks in reverse order).

A unit test is added to test rollback behaviour

Change-Id: I65214e72c7ef607dd08f750a6d32a0b10fe97ac3
2017-06-08 17:14:20 +10:00
Ian Wienand
09dee46579 Move global mount tracking into state
Keep track of the mount-point ordering in a state variable, rather
than a global.  This path is tested by existing unit tests.

Note a prior change inserted the MountNode objects directly into a
list in self.state, which makes sorting quite easy as it can just
implement __lt__.  Unfortunately we still json dump the state, and
thus we can't have aribtrary objects in it (future work may be to
check keys inserted into the status object...).  So we have to do a
bit of wrangling with tuple lists and comparision functions here, but
it's not too bad.

Change-Id: I0c51e0c53c4efdb7a65ab0efe09a6780cb1affa8
2017-06-08 17:13:28 +10:00
Ian Wienand
886f925b13 Use global state to check for duplicate fs labels
As we add file-systems, add them to global state and check the labels
are uniqiue.  Add a unit test and remove the old global value.

Bonus fixup to the length check, and a test for that too.

Change-Id: I0f5a96f687c92e000afc9c98a26c49c4b1d3f28d
2017-06-08 17:13:28 +10:00
Ian Wienand
b708918b85 Remove 'state' argument from later cmd_* calls
With I468dbf5134947629f125504513703d6f2cdace59 each node has a
reference to the global state object.  This means it gets pickled into
the node-list, which is loaded for later calls.  There is no need to
reload the state.json it and pass it for later cmd_* calls, as the
nodes can see it via the unpickled self.state

Change-Id: I9e2f8910f17599d92ee33e7df8e36d8ed4d44575
2017-06-08 17:13:28 +10:00
Ian Wienand
824a9e91c4 Add state to NodeBase class
Making the global state reference a defined part of the node makes
some parts of the block device processing easier and removes the need
for other global values.

The state is passed to PluginNodeBase.__init__() and expected to be
passed into all nodes as they are created.  NodeBase.__init__() is
updated with the new paramater 'state'.

The parameter is removed from the create() call as nodes can simply
reference it at any point as "self.state".

This is similar to 1cdc8b20373c5d582ea928cfd7334469ff36dbce, except it
is based on I68840594a34af28d41d9522addcfd830bd203b97 which loads the
node-list from pickled state for later cmd_* calls.  Thus we only
build the state *once*, at cmd_create() time as we build the node
list.

Change-Id: I468dbf5134947629f125504513703d6f2cdace59
2017-06-08 17:13:26 +10:00
Ian Wienand
e82e0097a9 Use picked nodes for later cmd_* calls
Currently the later cmd_* calls -- umount, cleanup, delete -- all
recreate the node graph by parsing the config file using
create_graph()

There is some need, however, to have a sense of global state when
building the node list.  The problem is, this is a one time operation
-- we do not want to rebuild that state for these later calls (see the
"loaded" checks in proposed
Ic3b805f9258128d5233b21ff25579c03487c7fcc).

An insight here seems to be that these cmd_* calls do not actually
want to re-parse the configuration file and rebuild the node list;
they just want to walk the node list in reverse with the state as
provided after cmd_create().

So, rather than re-creating the node list, we might as well just
pickle it, save it to disk along side the state dictionary dump and
reload it for cmd_*.

After this, I think we can safely have PluginBase.__init__() be passed
the state.  We will now know that this will only be called once,
during initial creation.

Change-Id: I68840594a34af28d41d9522addcfd830bd203b97
2017-06-08 17:10:10 +10:00
Ian Wienand
9a8b135267 Don't make image & loopdev functions static
You can't pickle a static method reference which complicates being
able to save the node graph when the "rollback" call-back wants to
hold references to these functions.  The outer module (localoop.py) is
small anyway, so from an organisation point of view the difference is
minimal.  Since these are really only called with parameters from the
containing class, they could be class methods with no parameters, at
the small expense of having to fiddle the mbr test-case a bit.

Change-Id: I6f9592a4295abe1b41294b79828bc2f3c2da01c6
2017-06-08 17:10:10 +10:00
Jenkins
60a5484ae8 Merge "Add env var to dump config graph" 2017-06-08 06:59:51 +00:00
Ian Wienand
d5c3863b87 Add env var to dump config graph
Make this a bit easier during debugging.  Add env var and some
developer instructions.

Change-Id: I34978ddb47d6642dfa22cae0f4c0543c0ba5475f
2017-06-08 05:04:58 +00:00
Ian Wienand
6fe1ef94f1 Use class as super() argument
Fix a few typos using the inherited class for super()

Change-Id: If9f2f423f136fb78ee93018d5c299d0dae603aad
2017-06-08 09:43:47 +10:00
Jenkins
e54ba47871 Merge "Move ppc block-device default to right $ARCH" 2017-06-07 10:02:17 +00:00
Jenkins
24a0890e4c Merge "Update tracing in block_device_create_config_file" 2017-06-07 08:18:10 +00:00
Jenkins
7447d59ec8 Merge "Add a keep-output flag for functional tests" 2017-06-07 06:50:36 +00:00
Jenkins
6dd20424ab Merge "Use https in docs links" 2017-06-07 04:16:19 +00:00
Ian Wienand
90b56b3aab Move ppc block-device default to right $ARCH
The supported ppc ${ARCH} is "ppc64el" (at least in the gate testing
...) so move the file to that, so gets picked up by
block_device_create_config_file

Change-Id: I9273f35cdbfb0a62404461cbc1df9b2a92155fb0
2017-06-07 13:30:38 +10:00
Ian Wienand
89a85f6fbb Update tracing in block_device_create_config_file
Something seems to be going on with the ppc matching in the gate test.
Small updates to see what's going on...

Change-Id: Ie48cd4ce1f983a58932a577a43746240f6866936
2017-06-07 13:30:38 +10:00
Ian Wienand
3886d5e206 Add a keep-output flag for functional tests
Add a -k flag that disables deleting of of ${destdir} for tests.  This
should allow examination of the resulting images if required.

Change-Id: I107c33e70100b21495a807f10762d3b6babe9bfe
2017-06-07 11:47:12 +10:00
OpenStack Proposal Bot
b39756580d Updated from global requirements
Change-Id: Ic1b5594e8dec2a476aa9a9ada54458350209fb70
2017-06-06 12:13:28 +00:00
Ian Wienand
7661da1341 Pad state dump
Because we append the function/line info after debug lines in the gate
logs, the pretty-print ends up not looking all that pretty.  Pad it.

Change-Id: Ice013428342614300cd51e8b7be56e79b75b31fc
2017-06-06 12:34:00 +10:00
Van Hung Pham
6143c3e956 Use https in docs links
Use https instead of http

Change-Id: I0a9e17142226046233d586a668e58e86a5a23098
2017-06-05 22:54:23 +07:00
Jenkins
ec70cb61f0 Merge "Trivial fix typos" 2017-06-05 05:54:50 +00:00
Jenkins
f9aa27e117 Merge "Adjust package installation for openSUSE" 2017-06-05 05:01:21 +00:00
Ian Wienand
7101c52620 Add missing test requirements, fixup pylint env
Add some missing test requirements.  I noticed these because pylint
was unhappy about the imports if you look closely.

Also, pylint shouldn't be in deps as it comes from the parent's
"test-requirements.txt" install.  We don't need the VIRTUAL_ENV
setting either.

Change-Id: Ie082a058a9d3d51164448410a00d0719b0b37c4a
2017-06-05 12:22:52 +10:00
Ian Wienand
cdb1a95be1 Move "functional" unit tests under block-device
This is code motion with some small changes to make follow-on's
easier.

test_blockdevice_mbr.py is moved alongside the other tests.  It is
modified slightly to use the standard base class and remove a lot of
repeated test setup; a fixture is used for the tempdir (so it doesn't
have to be torn-down, and is removed properly on error) and the partx
args are moved into the setUp() so each test doesn't have to create
it.  No functional change.  renamed test_mbr.py for shortness.

test_blockdevice_utils.py is merged with existing test_utils.py.  No
change to the tests.

test_blockdevice.py is removed.  It isn't doing anything currently; to
work it will need to take an approach based more on mocking of calls
that require elevated permissions.  It's in history if we need it.

Change-Id: I87b1ea94afaaa0b44e6a57b9d073f95a63a04cf0
2017-06-05 12:22:52 +10:00
Vu Cong Tuan
cae44c7eea Replace assertRaisesRegexp with assertRaisesRegex
assertRaisesRegexp was renamed to assertRaisesRegex in Py3.2
For more details, please check:
https://docs.python.org/3/library/
unittest.html#unittest.TestCase.assertRaisesRegex

Change-Id: I705c958c0dbf1daa409ed29ccbc038426298c306
Closes-Bug: #1436957
2017-06-03 13:27:37 +07:00
Jenkins
5a045e036d Merge "dhcp-all-interfaces.sh - Add support for InfiniBand interface DHCP" 2017-06-02 06:11:19 +00:00
Jenkins
80cc1d0ea4 Merge "Adjust package mapping for SUSE family" 2017-06-02 02:56:16 +00:00
Dirk Mueller
d0a398c167 Adjust package mapping for SUSE family
package-installs.yaml is installing python-dev, not python2-dev,
so we need to adjust the mapping accordingly.

In addition, zypper-minimal used an dpkg specific package name,
while there is a SUSE equivalent (and zypper-minimal is anyway
SUSE family specific)

Change-Id: Ia9dd061fa46a514781808d62e5e93b03f75c6745
2017-05-31 21:09:53 +02:00
Dirk Mueller
f58bf252de Drop support for Ubuntu precise
Ubuntu 12.04 LTS reached its regular End of Life on April 28, 2017.

Depends-On: I5e145095a10db112bb27516bfe652d2cdc052a61
Change-Id: I64af4c5183d77a75dcd062895d19b0a1330c8da8
2017-05-31 14:36:30 +02:00
Dirk Mueller
344b1bc5d3 Adjust package installation for openSUSE
On SUSE family distros the squash-tools are simply part of the
main package called "squashfs", so install that one instead. Without
this change bindep on SUSE hosts fails with:

  ERROR: These requested packages were not installed:
  squashfs-tools

Also adjust install_test_deps.sh to install required packages
on an openSUSE host.

Change-Id: I61dcd5314e78dbb1fb31e723799374edd456da99
2017-05-31 14:35:34 +02:00
Jenkins
016b1f1522 Merge "Make BlockDeviceState implement a dict" 2017-05-31 10:55:32 +00:00
Jenkins
d1c49c1ae4 Merge "Refactor mount-point sorting" 2017-05-31 10:50:26 +00:00
Jenkins
b312c06dbb Merge "Decode string to bytes in dracut-regenerate" 2017-05-31 10:49:51 +00:00
Jenkins
09543cf52b Merge "Add state object, rename "results", add unit tests" 2017-05-31 05:52:24 +00:00
Jenkins
d0e0714f71 Merge "Test openSUSE 42.2/42.3 image builds" 2017-05-31 04:37:57 +00:00
Vu Cong Tuan
6a72052108 Trivial fix typos
Change-Id: Ib86aa9938fd852610ec0a6d8d868181f87bd2f24
2017-05-31 11:17:05 +07:00
Jenkins
2bdc154df5 Merge "drop deprecated map-services/packages from zypper element" 2017-05-31 02:11:21 +00:00
Jenkins
05d64b99ce Merge "Remove ccache" 2017-05-31 01:48:01 +00:00
Ian Wienand
4253cab773 Make BlockDeviceState implement a dict
While plugins treat the state as just a dictionary, it's nice for the
driver functions to keep state related functions encapsulated in the
state object singleton.  Wrap the internal state dictionary so we can
pass the BlockDeviceState directly without dereferencing.

Change-Id: Ic0193c64d645ed1312f898cbfef87841f460799c
2017-05-31 11:24:55 +10:00