rewrite createhdds in python, make it shinier
Summary:
createhdds.sh was just too damn simple and understandable, so
I thought I'd make it three times longer, object oriented,
and hard to understand!
OK, OK, that's not great sales. Alright. The main thing was to
make it smarter. This rewrite lets it do these things:
* Only create the images that are missing (not rebuild all)
* Work out the releases to build images for
* Rename images when appropriate
* Rebuild images when they need rebuilding
* Remove old / abandoned images
It can figure out what images ought to be present - including
working out the 'next' release and figuring out from that what
releases it needs images for - and build only the missing ones.
There's a 'version' concept for images; if the existing image
is older than the version given in the data file, it'll be
rebuilt. The data file can list 'rename' pairs, allowing images
to be renamed (like when we move from a single image to multiple
label/filesystem variants). This code uses fedfind's ability
to find the current release version to figure out what releases
we need virtbuilder images for (so you don't have to pass it
in). And it can find image files that aren't in the 'currently
expected' set and wipe them. Images can also have a 'maxage',
triggering a rebuild when they exceed it - this is intended
for the virtbuilder images, so we get a rebuild with the
latest updates every so often (default is two weeks).
The point of all this is to help with unattended deployment/
maintenance, i.e. the ansible deployment we have in infra;
the idea is that we can just set that up to run the 'all'
subcommand every so often, and it'll remove old images, create
new ones, and rebuild ones that are outdated.
I kept the ability to build a single image (or a whole image
'group'), and included the ability to just run a check without
actually doing a rebuild. There's a few little weird things
and holes here as it's not really the focus of the tool.
Test Plan:
Build all images, do a full test run, and see if
it works OK. Test out all the variations of building single
images / image groups, and using the 'check' command.
Reviewers: jskladan, garretraziel
Reviewed By: jskladan, garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D687
2015-12-17 21:25:30 +00:00
|
|
|
{
|
|
|
|
|
"guestfs" : [
|
|
|
|
|
{
|
|
|
|
|
"name" : "full",
|
|
|
|
|
"size" : "10G",
|
|
|
|
|
"labels" : ["mbr", "gpt"],
|
|
|
|
|
"parts" : [
|
|
|
|
|
{
|
|
|
|
|
"filesystem" : "ext4",
|
|
|
|
|
"type" : "p",
|
|
|
|
|
"start" : "4096",
|
|
|
|
|
"end" : "10485760"
|
|
|
|
|
},
|
|
|
|
|
{
|
|
|
|
|
"filesystem" : "ext4",
|
|
|
|
|
"type" : "p",
|
|
|
|
|
"start" : "10485761",
|
|
|
|
|
"end" : "-4097"
|
|
|
|
|
}
|
|
|
|
|
],
|
|
|
|
|
"writes" : [
|
|
|
|
|
{
|
|
|
|
|
"part" : "1",
|
|
|
|
|
"path" : "/testfile",
|
|
|
|
|
"content" : "Hello, world!"
|
|
|
|
|
},
|
|
|
|
|
{
|
|
|
|
|
"part" : "2",
|
|
|
|
|
"path" : "/testfile",
|
|
|
|
|
"content" : "Oh, hi Mark"
|
|
|
|
|
}
|
|
|
|
|
]
|
|
|
|
|
},
|
|
|
|
|
{
|
|
|
|
|
"name" : "freespace",
|
|
|
|
|
"size" : "10G",
|
|
|
|
|
"labels" : ["mbr", "gpt"],
|
|
|
|
|
"parts" : [
|
|
|
|
|
{
|
|
|
|
|
"filesystem" : "ext4",
|
|
|
|
|
"type" : "p",
|
|
|
|
|
"start" : "4096",
|
|
|
|
|
"end" : "2097152"
|
|
|
|
|
}
|
|
|
|
|
],
|
|
|
|
|
"writes" : [
|
|
|
|
|
{
|
|
|
|
|
"part" : "1",
|
|
|
|
|
"path" : "/testfile",
|
|
|
|
|
"content" : "Hello, world!"
|
|
|
|
|
}
|
|
|
|
|
]
|
|
|
|
|
},
|
|
|
|
|
{
|
|
|
|
|
"name" : "ks",
|
|
|
|
|
"size" : "100M",
|
keep upload files locally, activate network in kickstarts
Summary:
I've come to dislike the approach where we source files that
get included in the disk images remotely; it's unnecessarily
complicated and needlessly exposes us to unexpected changes in
the remote files. So simply keep the files in this git repo
instead, none of them is very big. This also simplifies the
code.
Also, change all the kickstarts to specify --device=link and
--activate on their network lines. These arguments *should*
be included, according to the documentation (the first to
specify which device the config is for, the second to specify
that it should be activated in the installer environment), and
I think the lack of the second is actually now a problem for
the FreeIPA kickstart enrolment test (not sure exactly what
changed to make this a problem where it wasn't before, but
*something* has, and this fixes it).
Test Plan:
check that all tests still run properly. The
FreeIPA enrolment kickstart test should now get somewhat
further. Before this change, for F25 and Rawhide, it fails to
enrol at all during install. Now it should enrol properly -
that's what the kickstart changes fix, it was failing because
it wasn't using the right network config - but it still fails
when trying to log in as test1, due to RHBZ #1366403 .
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D968
2016-08-16 00:02:05 +00:00
|
|
|
"imgver" : "3",
|
rewrite createhdds in python, make it shinier
Summary:
createhdds.sh was just too damn simple and understandable, so
I thought I'd make it three times longer, object oriented,
and hard to understand!
OK, OK, that's not great sales. Alright. The main thing was to
make it smarter. This rewrite lets it do these things:
* Only create the images that are missing (not rebuild all)
* Work out the releases to build images for
* Rename images when appropriate
* Rebuild images when they need rebuilding
* Remove old / abandoned images
It can figure out what images ought to be present - including
working out the 'next' release and figuring out from that what
releases it needs images for - and build only the missing ones.
There's a 'version' concept for images; if the existing image
is older than the version given in the data file, it'll be
rebuilt. The data file can list 'rename' pairs, allowing images
to be renamed (like when we move from a single image to multiple
label/filesystem variants). This code uses fedfind's ability
to find the current release version to figure out what releases
we need virtbuilder images for (so you don't have to pass it
in). And it can find image files that aren't in the 'currently
expected' set and wipe them. Images can also have a 'maxage',
triggering a rebuild when they exceed it - this is intended
for the virtbuilder images, so we get a rebuild with the
latest updates every so often (default is two weeks).
The point of all this is to help with unattended deployment/
maintenance, i.e. the ansible deployment we have in infra;
the idea is that we can just set that up to run the 'all'
subcommand every so often, and it'll remove old images, create
new ones, and rebuild ones that are outdated.
I kept the ability to build a single image (or a whole image
'group'), and included the ability to just run a check without
actually doing a rebuild. There's a few little weird things
and holes here as it's not really the focus of the tool.
Test Plan:
Build all images, do a full test run, and see if
it works OK. Test out all the variations of building single
images / image groups, and using the 'check' command.
Reviewers: jskladan, garretraziel
Reviewed By: jskladan, garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D687
2015-12-17 21:25:30 +00:00
|
|
|
"parts" : [
|
|
|
|
|
{
|
|
|
|
|
"filesystem" : "ext4",
|
|
|
|
|
"type" : "p",
|
|
|
|
|
"start" : "4096",
|
|
|
|
|
"end" : "-1"
|
|
|
|
|
}
|
|
|
|
|
],
|
|
|
|
|
"uploads" : [
|
|
|
|
|
{
|
|
|
|
|
"part" : "1",
|
|
|
|
|
"target" : "/root-user-crypted-net.ks",
|
keep upload files locally, activate network in kickstarts
Summary:
I've come to dislike the approach where we source files that
get included in the disk images remotely; it's unnecessarily
complicated and needlessly exposes us to unexpected changes in
the remote files. So simply keep the files in this git repo
instead, none of them is very big. This also simplifies the
code.
Also, change all the kickstarts to specify --device=link and
--activate on their network lines. These arguments *should*
be included, according to the documentation (the first to
specify which device the config is for, the second to specify
that it should be activated in the installer environment), and
I think the lack of the second is actually now a problem for
the FreeIPA kickstart enrolment test (not sure exactly what
changed to make this a problem where it wasn't before, but
*something* has, and this fixes it).
Test Plan:
check that all tests still run properly. The
FreeIPA enrolment kickstart test should now get somewhat
further. Before this change, for F25 and Rawhide, it fails to
enrol at all during install. Now it should enrol properly -
that's what the kickstart changes fix, it was failing because
it wasn't using the right network config - but it still fails
when trying to log in as test1, due to RHBZ #1366403 .
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D968
2016-08-16 00:02:05 +00:00
|
|
|
"source" : "root-user-crypted-net.ks"
|
2016-05-04 18:55:22 +00:00
|
|
|
},
|
|
|
|
|
{
|
|
|
|
|
"part" : "1",
|
|
|
|
|
"target" : "/freeipa.ks",
|
keep upload files locally, activate network in kickstarts
Summary:
I've come to dislike the approach where we source files that
get included in the disk images remotely; it's unnecessarily
complicated and needlessly exposes us to unexpected changes in
the remote files. So simply keep the files in this git repo
instead, none of them is very big. This also simplifies the
code.
Also, change all the kickstarts to specify --device=link and
--activate on their network lines. These arguments *should*
be included, according to the documentation (the first to
specify which device the config is for, the second to specify
that it should be activated in the installer environment), and
I think the lack of the second is actually now a problem for
the FreeIPA kickstart enrolment test (not sure exactly what
changed to make this a problem where it wasn't before, but
*something* has, and this fixes it).
Test Plan:
check that all tests still run properly. The
FreeIPA enrolment kickstart test should now get somewhat
further. Before this change, for F25 and Rawhide, it fails to
enrol at all during install. Now it should enrol properly -
that's what the kickstart changes fix, it was failing because
it wasn't using the right network config - but it still fails
when trying to log in as test1, due to RHBZ #1366403 .
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D968
2016-08-16 00:02:05 +00:00
|
|
|
"source" : "freeipa.ks"
|
2016-05-04 18:55:22 +00:00
|
|
|
},
|
|
|
|
|
{
|
|
|
|
|
"part" : "1",
|
|
|
|
|
"target" : "/freeipaclient.ks",
|
keep upload files locally, activate network in kickstarts
Summary:
I've come to dislike the approach where we source files that
get included in the disk images remotely; it's unnecessarily
complicated and needlessly exposes us to unexpected changes in
the remote files. So simply keep the files in this git repo
instead, none of them is very big. This also simplifies the
code.
Also, change all the kickstarts to specify --device=link and
--activate on their network lines. These arguments *should*
be included, according to the documentation (the first to
specify which device the config is for, the second to specify
that it should be activated in the installer environment), and
I think the lack of the second is actually now a problem for
the FreeIPA kickstart enrolment test (not sure exactly what
changed to make this a problem where it wasn't before, but
*something* has, and this fixes it).
Test Plan:
check that all tests still run properly. The
FreeIPA enrolment kickstart test should now get somewhat
further. Before this change, for F25 and Rawhide, it fails to
enrol at all during install. Now it should enrol properly -
that's what the kickstart changes fix, it was failing because
it wasn't using the right network config - but it still fails
when trying to log in as test1, due to RHBZ #1366403 .
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D968
2016-08-16 00:02:05 +00:00
|
|
|
"source" : "freeipaclient.ks"
|
rewrite createhdds in python, make it shinier
Summary:
createhdds.sh was just too damn simple and understandable, so
I thought I'd make it three times longer, object oriented,
and hard to understand!
OK, OK, that's not great sales. Alright. The main thing was to
make it smarter. This rewrite lets it do these things:
* Only create the images that are missing (not rebuild all)
* Work out the releases to build images for
* Rename images when appropriate
* Rebuild images when they need rebuilding
* Remove old / abandoned images
It can figure out what images ought to be present - including
working out the 'next' release and figuring out from that what
releases it needs images for - and build only the missing ones.
There's a 'version' concept for images; if the existing image
is older than the version given in the data file, it'll be
rebuilt. The data file can list 'rename' pairs, allowing images
to be renamed (like when we move from a single image to multiple
label/filesystem variants). This code uses fedfind's ability
to find the current release version to figure out what releases
we need virtbuilder images for (so you don't have to pass it
in). And it can find image files that aren't in the 'currently
expected' set and wipe them. Images can also have a 'maxage',
triggering a rebuild when they exceed it - this is intended
for the virtbuilder images, so we get a rebuild with the
latest updates every so often (default is two weeks).
The point of all this is to help with unattended deployment/
maintenance, i.e. the ansible deployment we have in infra;
the idea is that we can just set that up to run the 'all'
subcommand every so often, and it'll remove old images, create
new ones, and rebuild ones that are outdated.
I kept the ability to build a single image (or a whole image
'group'), and included the ability to just run a check without
actually doing a rebuild. There's a few little weird things
and holes here as it's not really the focus of the tool.
Test Plan:
Build all images, do a full test run, and see if
it works OK. Test out all the variations of building single
images / image groups, and using the 'check' command.
Reviewers: jskladan, garretraziel
Reviewed By: jskladan, garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D687
2015-12-17 21:25:30 +00:00
|
|
|
}
|
|
|
|
|
]
|
|
|
|
|
},
|
|
|
|
|
{
|
|
|
|
|
"name" : "updates_img",
|
|
|
|
|
"size" : "100M",
|
2016-04-01 15:01:12 +00:00
|
|
|
"imgver" : "2",
|
rewrite createhdds in python, make it shinier
Summary:
createhdds.sh was just too damn simple and understandable, so
I thought I'd make it three times longer, object oriented,
and hard to understand!
OK, OK, that's not great sales. Alright. The main thing was to
make it smarter. This rewrite lets it do these things:
* Only create the images that are missing (not rebuild all)
* Work out the releases to build images for
* Rename images when appropriate
* Rebuild images when they need rebuilding
* Remove old / abandoned images
It can figure out what images ought to be present - including
working out the 'next' release and figuring out from that what
releases it needs images for - and build only the missing ones.
There's a 'version' concept for images; if the existing image
is older than the version given in the data file, it'll be
rebuilt. The data file can list 'rename' pairs, allowing images
to be renamed (like when we move from a single image to multiple
label/filesystem variants). This code uses fedfind's ability
to find the current release version to figure out what releases
we need virtbuilder images for (so you don't have to pass it
in). And it can find image files that aren't in the 'currently
expected' set and wipe them. Images can also have a 'maxage',
triggering a rebuild when they exceed it - this is intended
for the virtbuilder images, so we get a rebuild with the
latest updates every so often (default is two weeks).
The point of all this is to help with unattended deployment/
maintenance, i.e. the ansible deployment we have in infra;
the idea is that we can just set that up to run the 'all'
subcommand every so often, and it'll remove old images, create
new ones, and rebuild ones that are outdated.
I kept the ability to build a single image (or a whole image
'group'), and included the ability to just run a check without
actually doing a rebuild. There's a few little weird things
and holes here as it's not really the focus of the tool.
Test Plan:
Build all images, do a full test run, and see if
it works OK. Test out all the variations of building single
images / image groups, and using the 'check' command.
Reviewers: jskladan, garretraziel
Reviewed By: jskladan, garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D687
2015-12-17 21:25:30 +00:00
|
|
|
"parts" : [
|
|
|
|
|
{
|
|
|
|
|
"filesystem" : "ext4",
|
|
|
|
|
"label" : "UPDATES_IMG",
|
|
|
|
|
"type" : "p",
|
|
|
|
|
"start" : "4096",
|
|
|
|
|
"end" : "-1"
|
|
|
|
|
}
|
|
|
|
|
],
|
|
|
|
|
"uploads" : [
|
|
|
|
|
{
|
|
|
|
|
"part" : "1",
|
|
|
|
|
"target" : "/updates.img",
|
keep upload files locally, activate network in kickstarts
Summary:
I've come to dislike the approach where we source files that
get included in the disk images remotely; it's unnecessarily
complicated and needlessly exposes us to unexpected changes in
the remote files. So simply keep the files in this git repo
instead, none of them is very big. This also simplifies the
code.
Also, change all the kickstarts to specify --device=link and
--activate on their network lines. These arguments *should*
be included, according to the documentation (the first to
specify which device the config is for, the second to specify
that it should be activated in the installer environment), and
I think the lack of the second is actually now a problem for
the FreeIPA kickstart enrolment test (not sure exactly what
changed to make this a problem where it wasn't before, but
*something* has, and this fixes it).
Test Plan:
check that all tests still run properly. The
FreeIPA enrolment kickstart test should now get somewhat
further. Before this change, for F25 and Rawhide, it fails to
enrol at all during install. Now it should enrol properly -
that's what the kickstart changes fix, it was failing because
it wasn't using the right network config - but it still fails
when trying to log in as test1, due to RHBZ #1366403 .
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D968
2016-08-16 00:02:05 +00:00
|
|
|
"source" : "updates.img"
|
rewrite createhdds in python, make it shinier
Summary:
createhdds.sh was just too damn simple and understandable, so
I thought I'd make it three times longer, object oriented,
and hard to understand!
OK, OK, that's not great sales. Alright. The main thing was to
make it smarter. This rewrite lets it do these things:
* Only create the images that are missing (not rebuild all)
* Work out the releases to build images for
* Rename images when appropriate
* Rebuild images when they need rebuilding
* Remove old / abandoned images
It can figure out what images ought to be present - including
working out the 'next' release and figuring out from that what
releases it needs images for - and build only the missing ones.
There's a 'version' concept for images; if the existing image
is older than the version given in the data file, it'll be
rebuilt. The data file can list 'rename' pairs, allowing images
to be renamed (like when we move from a single image to multiple
label/filesystem variants). This code uses fedfind's ability
to find the current release version to figure out what releases
we need virtbuilder images for (so you don't have to pass it
in). And it can find image files that aren't in the 'currently
expected' set and wipe them. Images can also have a 'maxage',
triggering a rebuild when they exceed it - this is intended
for the virtbuilder images, so we get a rebuild with the
latest updates every so often (default is two weeks).
The point of all this is to help with unattended deployment/
maintenance, i.e. the ansible deployment we have in infra;
the idea is that we can just set that up to run the 'all'
subcommand every so often, and it'll remove old images, create
new ones, and rebuild ones that are outdated.
I kept the ability to build a single image (or a whole image
'group'), and included the ability to just run a check without
actually doing a rebuild. There's a few little weird things
and holes here as it's not really the focus of the tool.
Test Plan:
Build all images, do a full test run, and see if
it works OK. Test out all the variations of building single
images / image groups, and using the 'check' command.
Reviewers: jskladan, garretraziel
Reviewed By: jskladan, garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D687
2015-12-17 21:25:30 +00:00
|
|
|
}
|
|
|
|
|
]
|
|
|
|
|
},
|
|
|
|
|
{
|
|
|
|
|
"name" : "shrink",
|
|
|
|
|
"size" : "10G",
|
2018-12-01 17:02:59 +00:00
|
|
|
"labels" : ["mbr"],
|
rewrite createhdds in python, make it shinier
Summary:
createhdds.sh was just too damn simple and understandable, so
I thought I'd make it three times longer, object oriented,
and hard to understand!
OK, OK, that's not great sales. Alright. The main thing was to
make it smarter. This rewrite lets it do these things:
* Only create the images that are missing (not rebuild all)
* Work out the releases to build images for
* Rename images when appropriate
* Rebuild images when they need rebuilding
* Remove old / abandoned images
It can figure out what images ought to be present - including
working out the 'next' release and figuring out from that what
releases it needs images for - and build only the missing ones.
There's a 'version' concept for images; if the existing image
is older than the version given in the data file, it'll be
rebuilt. The data file can list 'rename' pairs, allowing images
to be renamed (like when we move from a single image to multiple
label/filesystem variants). This code uses fedfind's ability
to find the current release version to figure out what releases
we need virtbuilder images for (so you don't have to pass it
in). And it can find image files that aren't in the 'currently
expected' set and wipe them. Images can also have a 'maxage',
triggering a rebuild when they exceed it - this is intended
for the virtbuilder images, so we get a rebuild with the
latest updates every so often (default is two weeks).
The point of all this is to help with unattended deployment/
maintenance, i.e. the ansible deployment we have in infra;
the idea is that we can just set that up to run the 'all'
subcommand every so often, and it'll remove old images, create
new ones, and rebuild ones that are outdated.
I kept the ability to build a single image (or a whole image
'group'), and included the ability to just run a check without
actually doing a rebuild. There's a few little weird things
and holes here as it's not really the focus of the tool.
Test Plan:
Build all images, do a full test run, and see if
it works OK. Test out all the variations of building single
images / image groups, and using the 'check' command.
Reviewers: jskladan, garretraziel
Reviewed By: jskladan, garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D687
2015-12-17 21:25:30 +00:00
|
|
|
"filesystems" : ["ext4", "ntfs"],
|
|
|
|
|
"parts" : [
|
|
|
|
|
{
|
|
|
|
|
"type" : "p",
|
|
|
|
|
"start" : "4096",
|
|
|
|
|
"end" : "-4097"
|
|
|
|
|
}
|
|
|
|
|
],
|
|
|
|
|
"writes" : [
|
|
|
|
|
{
|
|
|
|
|
"part" : "1",
|
|
|
|
|
"path" : "/testfile",
|
|
|
|
|
"content" : "Hello, world!"
|
|
|
|
|
}
|
|
|
|
|
]
|
|
|
|
|
}
|
|
|
|
|
],
|
switch from virt-builder to virt-install (T813)
Summary:
We've kinda been having too much trouble with virt-builder
lately, mainly SELinux related issues due to how it does image
customization. It also produces images that differ in notable
ways from what a 'typical' install would give. virt-install
solves both these problems, and also gives us more flexibility
for storage configuration and post-install customization should
we need them in future.
The change isn't really too drastic, and the design is similar:
instead of virt-builder commands files, each image type now has
a kickstart file where all its customizations can be done.
There's also a single extra image dict key, 'variant', which
specifies which install tree variant to use for running the
install. It defaults to 'Everything' (for F24+) and 'Server'
(for <F24, as Everything wasn't installable until F24) but we
set it to 'Server' for the server images and 'Workstation' for
the desktop images, so those installs will use the correct
variant install class.
We run the installs in VNC. You can do it with a serial console
and log the output, but then anaconda gets clever and changes
several things in the installed system based on the fact that
you did the install over a serial console: it twiddles with
the kernel args and doesn't set graphical.target as the default.
We don't want any of that mess, so we do a VNC install.
The 'size' value is just a number of gigabytes for virt-install
images (as that's how the virt-install 'size' argument works).
This also drops some unused 32-bit images (we don't do 32-bit
KDE or Server upgrade tests, so there's no need to build those
images).
Test Plan:
Re-generate all affected images and re-run all tests
that use them, make sure they work. I am doing this on staging
at present. Note: this would render D911 unnecessary.
Reviewers: garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D917
2016-07-04 16:29:25 +00:00
|
|
|
"virtinstall" : [
|
rewrite createhdds in python, make it shinier
Summary:
createhdds.sh was just too damn simple and understandable, so
I thought I'd make it three times longer, object oriented,
and hard to understand!
OK, OK, that's not great sales. Alright. The main thing was to
make it smarter. This rewrite lets it do these things:
* Only create the images that are missing (not rebuild all)
* Work out the releases to build images for
* Rename images when appropriate
* Rebuild images when they need rebuilding
* Remove old / abandoned images
It can figure out what images ought to be present - including
working out the 'next' release and figuring out from that what
releases it needs images for - and build only the missing ones.
There's a 'version' concept for images; if the existing image
is older than the version given in the data file, it'll be
rebuilt. The data file can list 'rename' pairs, allowing images
to be renamed (like when we move from a single image to multiple
label/filesystem variants). This code uses fedfind's ability
to find the current release version to figure out what releases
we need virtbuilder images for (so you don't have to pass it
in). And it can find image files that aren't in the 'currently
expected' set and wipe them. Images can also have a 'maxage',
triggering a rebuild when they exceed it - this is intended
for the virtbuilder images, so we get a rebuild with the
latest updates every so often (default is two weeks).
The point of all this is to help with unattended deployment/
maintenance, i.e. the ansible deployment we have in infra;
the idea is that we can just set that up to run the 'all'
subcommand every so often, and it'll remove old images, create
new ones, and rebuild ones that are outdated.
I kept the ability to build a single image (or a whole image
'group'), and included the ability to just run a check without
actually doing a rebuild. There's a few little weird things
and holes here as it's not really the focus of the tool.
Test Plan:
Build all images, do a full test run, and see if
it works OK. Test out all the variations of building single
images / image groups, and using the 'check' command.
Reviewers: jskladan, garretraziel
Reviewed By: jskladan, garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D687
2015-12-17 21:25:30 +00:00
|
|
|
{
|
|
|
|
|
"name" : "minimal",
|
|
|
|
|
"releases" : {
|
2018-03-06 17:06:58 +00:00
|
|
|
"-1" : ["x86_64", "ppc64le", "ppc64", "aarch64"],
|
2018-08-31 20:36:15 +00:00
|
|
|
"-2" : ["x86_64", "ppc64le", "ppc64", "aarch64"],
|
|
|
|
|
"stable": ["x86_64"],
|
2018-08-31 20:39:11 +00:00
|
|
|
"branched": ["x86_64"]
|
switch from virt-builder to virt-install (T813)
Summary:
We've kinda been having too much trouble with virt-builder
lately, mainly SELinux related issues due to how it does image
customization. It also produces images that differ in notable
ways from what a 'typical' install would give. virt-install
solves both these problems, and also gives us more flexibility
for storage configuration and post-install customization should
we need them in future.
The change isn't really too drastic, and the design is similar:
instead of virt-builder commands files, each image type now has
a kickstart file where all its customizations can be done.
There's also a single extra image dict key, 'variant', which
specifies which install tree variant to use for running the
install. It defaults to 'Everything' (for F24+) and 'Server'
(for <F24, as Everything wasn't installable until F24) but we
set it to 'Server' for the server images and 'Workstation' for
the desktop images, so those installs will use the correct
variant install class.
We run the installs in VNC. You can do it with a serial console
and log the output, but then anaconda gets clever and changes
several things in the installed system based on the fact that
you did the install over a serial console: it twiddles with
the kernel args and doesn't set graphical.target as the default.
We don't want any of that mess, so we do a VNC install.
The 'size' value is just a number of gigabytes for virt-install
images (as that's how the virt-install 'size' argument works).
This also drops some unused 32-bit images (we don't do 32-bit
KDE or Server upgrade tests, so there's no need to build those
images).
Test Plan:
Re-generate all affected images and re-run all tests
that use them, make sure they work. I am doing this on staging
at present. Note: this would render D911 unnecessary.
Reviewers: garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D917
2016-07-04 16:29:25 +00:00
|
|
|
},
|
2019-01-18 02:15:15 +00:00
|
|
|
"size" : "9",
|
|
|
|
|
"imgver": "3"
|
rewrite createhdds in python, make it shinier
Summary:
createhdds.sh was just too damn simple and understandable, so
I thought I'd make it three times longer, object oriented,
and hard to understand!
OK, OK, that's not great sales. Alright. The main thing was to
make it smarter. This rewrite lets it do these things:
* Only create the images that are missing (not rebuild all)
* Work out the releases to build images for
* Rename images when appropriate
* Rebuild images when they need rebuilding
* Remove old / abandoned images
It can figure out what images ought to be present - including
working out the 'next' release and figuring out from that what
releases it needs images for - and build only the missing ones.
There's a 'version' concept for images; if the existing image
is older than the version given in the data file, it'll be
rebuilt. The data file can list 'rename' pairs, allowing images
to be renamed (like when we move from a single image to multiple
label/filesystem variants). This code uses fedfind's ability
to find the current release version to figure out what releases
we need virtbuilder images for (so you don't have to pass it
in). And it can find image files that aren't in the 'currently
expected' set and wipe them. Images can also have a 'maxage',
triggering a rebuild when they exceed it - this is intended
for the virtbuilder images, so we get a rebuild with the
latest updates every so often (default is two weeks).
The point of all this is to help with unattended deployment/
maintenance, i.e. the ansible deployment we have in infra;
the idea is that we can just set that up to run the 'all'
subcommand every so often, and it'll remove old images, create
new ones, and rebuild ones that are outdated.
I kept the ability to build a single image (or a whole image
'group'), and included the ability to just run a check without
actually doing a rebuild. There's a few little weird things
and holes here as it's not really the focus of the tool.
Test Plan:
Build all images, do a full test run, and see if
it works OK. Test out all the variations of building single
images / image groups, and using the 'check' command.
Reviewers: jskladan, garretraziel
Reviewed By: jskladan, garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D687
2015-12-17 21:25:30 +00:00
|
|
|
},
|
|
|
|
|
{
|
|
|
|
|
"name" : "desktop",
|
|
|
|
|
"releases" : {
|
2017-07-27 19:56:26 +00:00
|
|
|
"stable" : ["x86_64", "i686"],
|
2017-02-15 01:41:31 +00:00
|
|
|
"branched": ["x86_64"]
|
rewrite createhdds in python, make it shinier
Summary:
createhdds.sh was just too damn simple and understandable, so
I thought I'd make it three times longer, object oriented,
and hard to understand!
OK, OK, that's not great sales. Alright. The main thing was to
make it smarter. This rewrite lets it do these things:
* Only create the images that are missing (not rebuild all)
* Work out the releases to build images for
* Rename images when appropriate
* Rebuild images when they need rebuilding
* Remove old / abandoned images
It can figure out what images ought to be present - including
working out the 'next' release and figuring out from that what
releases it needs images for - and build only the missing ones.
There's a 'version' concept for images; if the existing image
is older than the version given in the data file, it'll be
rebuilt. The data file can list 'rename' pairs, allowing images
to be renamed (like when we move from a single image to multiple
label/filesystem variants). This code uses fedfind's ability
to find the current release version to figure out what releases
we need virtbuilder images for (so you don't have to pass it
in). And it can find image files that aren't in the 'currently
expected' set and wipe them. Images can also have a 'maxage',
triggering a rebuild when they exceed it - this is intended
for the virtbuilder images, so we get a rebuild with the
latest updates every so often (default is two weeks).
The point of all this is to help with unattended deployment/
maintenance, i.e. the ansible deployment we have in infra;
the idea is that we can just set that up to run the 'all'
subcommand every so often, and it'll remove old images, create
new ones, and rebuild ones that are outdated.
I kept the ability to build a single image (or a whole image
'group'), and included the ability to just run a check without
actually doing a rebuild. There's a few little weird things
and holes here as it's not really the focus of the tool.
Test Plan:
Build all images, do a full test run, and see if
it works OK. Test out all the variations of building single
images / image groups, and using the 'check' command.
Reviewers: jskladan, garretraziel
Reviewed By: jskladan, garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D687
2015-12-17 21:25:30 +00:00
|
|
|
},
|
switch from virt-builder to virt-install (T813)
Summary:
We've kinda been having too much trouble with virt-builder
lately, mainly SELinux related issues due to how it does image
customization. It also produces images that differ in notable
ways from what a 'typical' install would give. virt-install
solves both these problems, and also gives us more flexibility
for storage configuration and post-install customization should
we need them in future.
The change isn't really too drastic, and the design is similar:
instead of virt-builder commands files, each image type now has
a kickstart file where all its customizations can be done.
There's also a single extra image dict key, 'variant', which
specifies which install tree variant to use for running the
install. It defaults to 'Everything' (for F24+) and 'Server'
(for <F24, as Everything wasn't installable until F24) but we
set it to 'Server' for the server images and 'Workstation' for
the desktop images, so those installs will use the correct
variant install class.
We run the installs in VNC. You can do it with a serial console
and log the output, but then anaconda gets clever and changes
several things in the installed system based on the fact that
you did the install over a serial console: it twiddles with
the kernel args and doesn't set graphical.target as the default.
We don't want any of that mess, so we do a VNC install.
The 'size' value is just a number of gigabytes for virt-install
images (as that's how the virt-install 'size' argument works).
This also drops some unused 32-bit images (we don't do 32-bit
KDE or Server upgrade tests, so there's no need to build those
images).
Test Plan:
Re-generate all affected images and re-run all tests
that use them, make sure they work. I am doing this on staging
at present. Note: this would render D911 unnecessary.
Reviewers: garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D917
2016-07-04 16:29:25 +00:00
|
|
|
"size" : "20",
|
2017-10-21 00:37:38 +00:00
|
|
|
"imgver": "4",
|
switch from virt-builder to virt-install (T813)
Summary:
We've kinda been having too much trouble with virt-builder
lately, mainly SELinux related issues due to how it does image
customization. It also produces images that differ in notable
ways from what a 'typical' install would give. virt-install
solves both these problems, and also gives us more flexibility
for storage configuration and post-install customization should
we need them in future.
The change isn't really too drastic, and the design is similar:
instead of virt-builder commands files, each image type now has
a kickstart file where all its customizations can be done.
There's also a single extra image dict key, 'variant', which
specifies which install tree variant to use for running the
install. It defaults to 'Everything' (for F24+) and 'Server'
(for <F24, as Everything wasn't installable until F24) but we
set it to 'Server' for the server images and 'Workstation' for
the desktop images, so those installs will use the correct
variant install class.
We run the installs in VNC. You can do it with a serial console
and log the output, but then anaconda gets clever and changes
several things in the installed system based on the fact that
you did the install over a serial console: it twiddles with
the kernel args and doesn't set graphical.target as the default.
We don't want any of that mess, so we do a VNC install.
The 'size' value is just a number of gigabytes for virt-install
images (as that's how the virt-install 'size' argument works).
This also drops some unused 32-bit images (we don't do 32-bit
KDE or Server upgrade tests, so there's no need to build those
images).
Test Plan:
Re-generate all affected images and re-run all tests
that use them, make sure they work. I am doing this on staging
at present. Note: this would render D911 unnecessary.
Reviewers: garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D917
2016-07-04 16:29:25 +00:00
|
|
|
"variant": "Workstation"
|
2016-04-25 07:59:21 +00:00
|
|
|
},
|
2016-07-08 15:58:23 +00:00
|
|
|
{
|
|
|
|
|
"name" : "desktopencrypt",
|
|
|
|
|
"releases" : {
|
2018-05-02 22:44:14 +00:00
|
|
|
"stable" : ["x86_64"],
|
|
|
|
|
"branched" : ["x86_64"]
|
2016-07-08 15:58:23 +00:00
|
|
|
},
|
|
|
|
|
"size" : "20",
|
|
|
|
|
"variant": "Workstation"
|
|
|
|
|
},
|
2016-04-25 07:59:21 +00:00
|
|
|
{
|
|
|
|
|
"name" : "server",
|
|
|
|
|
"releases" : {
|
2018-03-06 17:06:58 +00:00
|
|
|
"stable" : ["x86_64", "ppc64le", "ppc64", "aarch64"],
|
|
|
|
|
"branched": ["x86_64", "ppc64le", "ppc64", "aarch64"]
|
2016-05-05 23:43:10 +00:00
|
|
|
},
|
switch from virt-builder to virt-install (T813)
Summary:
We've kinda been having too much trouble with virt-builder
lately, mainly SELinux related issues due to how it does image
customization. It also produces images that differ in notable
ways from what a 'typical' install would give. virt-install
solves both these problems, and also gives us more flexibility
for storage configuration and post-install customization should
we need them in future.
The change isn't really too drastic, and the design is similar:
instead of virt-builder commands files, each image type now has
a kickstart file where all its customizations can be done.
There's also a single extra image dict key, 'variant', which
specifies which install tree variant to use for running the
install. It defaults to 'Everything' (for F24+) and 'Server'
(for <F24, as Everything wasn't installable until F24) but we
set it to 'Server' for the server images and 'Workstation' for
the desktop images, so those installs will use the correct
variant install class.
We run the installs in VNC. You can do it with a serial console
and log the output, but then anaconda gets clever and changes
several things in the installed system based on the fact that
you did the install over a serial console: it twiddles with
the kernel args and doesn't set graphical.target as the default.
We don't want any of that mess, so we do a VNC install.
The 'size' value is just a number of gigabytes for virt-install
images (as that's how the virt-install 'size' argument works).
This also drops some unused 32-bit images (we don't do 32-bit
KDE or Server upgrade tests, so there's no need to build those
images).
Test Plan:
Re-generate all affected images and re-run all tests
that use them, make sure they work. I am doing this on staging
at present. Note: this would render D911 unnecessary.
Reviewers: garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D917
2016-07-04 16:29:25 +00:00
|
|
|
"size" : "6",
|
|
|
|
|
"imgver": "3",
|
|
|
|
|
"variant": "Server"
|
2016-04-25 07:59:21 +00:00
|
|
|
},
|
|
|
|
|
{
|
|
|
|
|
"name" : "kde",
|
|
|
|
|
"releases" : {
|
switch from virt-builder to virt-install (T813)
Summary:
We've kinda been having too much trouble with virt-builder
lately, mainly SELinux related issues due to how it does image
customization. It also produces images that differ in notable
ways from what a 'typical' install would give. virt-install
solves both these problems, and also gives us more flexibility
for storage configuration and post-install customization should
we need them in future.
The change isn't really too drastic, and the design is similar:
instead of virt-builder commands files, each image type now has
a kickstart file where all its customizations can be done.
There's also a single extra image dict key, 'variant', which
specifies which install tree variant to use for running the
install. It defaults to 'Everything' (for F24+) and 'Server'
(for <F24, as Everything wasn't installable until F24) but we
set it to 'Server' for the server images and 'Workstation' for
the desktop images, so those installs will use the correct
variant install class.
We run the installs in VNC. You can do it with a serial console
and log the output, but then anaconda gets clever and changes
several things in the installed system based on the fact that
you did the install over a serial console: it twiddles with
the kernel args and doesn't set graphical.target as the default.
We don't want any of that mess, so we do a VNC install.
The 'size' value is just a number of gigabytes for virt-install
images (as that's how the virt-install 'size' argument works).
This also drops some unused 32-bit images (we don't do 32-bit
KDE or Server upgrade tests, so there's no need to build those
images).
Test Plan:
Re-generate all affected images and re-run all tests
that use them, make sure they work. I am doing this on staging
at present. Note: this would render D911 unnecessary.
Reviewers: garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D917
2016-07-04 16:29:25 +00:00
|
|
|
"-1" : ["x86_64"],
|
|
|
|
|
"-2" : ["x86_64"]
|
2016-04-25 07:59:21 +00:00
|
|
|
},
|
switch from virt-builder to virt-install (T813)
Summary:
We've kinda been having too much trouble with virt-builder
lately, mainly SELinux related issues due to how it does image
customization. It also produces images that differ in notable
ways from what a 'typical' install would give. virt-install
solves both these problems, and also gives us more flexibility
for storage configuration and post-install customization should
we need them in future.
The change isn't really too drastic, and the design is similar:
instead of virt-builder commands files, each image type now has
a kickstart file where all its customizations can be done.
There's also a single extra image dict key, 'variant', which
specifies which install tree variant to use for running the
install. It defaults to 'Everything' (for F24+) and 'Server'
(for <F24, as Everything wasn't installable until F24) but we
set it to 'Server' for the server images and 'Workstation' for
the desktop images, so those installs will use the correct
variant install class.
We run the installs in VNC. You can do it with a serial console
and log the output, but then anaconda gets clever and changes
several things in the installed system based on the fact that
you did the install over a serial console: it twiddles with
the kernel args and doesn't set graphical.target as the default.
We don't want any of that mess, so we do a VNC install.
The 'size' value is just a number of gigabytes for virt-install
images (as that's how the virt-install 'size' argument works).
This also drops some unused 32-bit images (we don't do 32-bit
KDE or Server upgrade tests, so there's no need to build those
images).
Test Plan:
Re-generate all affected images and re-run all tests
that use them, make sure they work. I am doing this on staging
at present. Note: this would render D911 unnecessary.
Reviewers: garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D917
2016-07-04 16:29:25 +00:00
|
|
|
"size" : "20",
|
|
|
|
|
"imgver": "3"
|
createhdds: add a disk image for use by a 'support server'
Summary:
this will be used for tests like iSCSI and NFS which need a
server end, but where (unlike e.g. FreeIPA) we don't want set
up of the server end to be considered a test and run on the
compose being tested, but instead we just consider the server
end a 'test asset' and expect it to be as reliable as possible.
For now we just set it up to provide an iSCSI target, as that's
what I want to write a test for first.
Note: I was initially writing the actual iSCSI target config
file as part of image creation, but that turns out to be a bit
tricky because we can't safely write a relative or absolute path
for the source file into the commands file. We'd have to instead
allow specifying file injections in the JSON, similar to how we
do for guestfs images, then construct an absolute path in the
Python code using SCRIPTDIR, but it was easier to just write the
file in the openQA test instead, so I did that.
Test Plan:
check that creating the support image works (and if
you're being really thorough, boot it and check the package is
there).
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D883
2016-06-08 22:45:54 +00:00
|
|
|
},
|
|
|
|
|
{
|
|
|
|
|
"name" : "support",
|
|
|
|
|
"releases" : {
|
2018-03-06 17:06:58 +00:00
|
|
|
"current" : ["x86_64", "ppc64le", "ppc64", "aarch64"]
|
switch from virt-builder to virt-install (T813)
Summary:
We've kinda been having too much trouble with virt-builder
lately, mainly SELinux related issues due to how it does image
customization. It also produces images that differ in notable
ways from what a 'typical' install would give. virt-install
solves both these problems, and also gives us more flexibility
for storage configuration and post-install customization should
we need them in future.
The change isn't really too drastic, and the design is similar:
instead of virt-builder commands files, each image type now has
a kickstart file where all its customizations can be done.
There's also a single extra image dict key, 'variant', which
specifies which install tree variant to use for running the
install. It defaults to 'Everything' (for F24+) and 'Server'
(for <F24, as Everything wasn't installable until F24) but we
set it to 'Server' for the server images and 'Workstation' for
the desktop images, so those installs will use the correct
variant install class.
We run the installs in VNC. You can do it with a serial console
and log the output, but then anaconda gets clever and changes
several things in the installed system based on the fact that
you did the install over a serial console: it twiddles with
the kernel args and doesn't set graphical.target as the default.
We don't want any of that mess, so we do a VNC install.
The 'size' value is just a number of gigabytes for virt-install
images (as that's how the virt-install 'size' argument works).
This also drops some unused 32-bit images (we don't do 32-bit
KDE or Server upgrade tests, so there's no need to build those
images).
Test Plan:
Re-generate all affected images and re-run all tests
that use them, make sure they work. I am doing this on staging
at present. Note: this would render D911 unnecessary.
Reviewers: garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D917
2016-07-04 16:29:25 +00:00
|
|
|
},
|
2018-11-10 01:07:57 +00:00
|
|
|
"size" : "7",
|
|
|
|
|
"imgver" : "4"
|
rewrite createhdds in python, make it shinier
Summary:
createhdds.sh was just too damn simple and understandable, so
I thought I'd make it three times longer, object oriented,
and hard to understand!
OK, OK, that's not great sales. Alright. The main thing was to
make it smarter. This rewrite lets it do these things:
* Only create the images that are missing (not rebuild all)
* Work out the releases to build images for
* Rename images when appropriate
* Rebuild images when they need rebuilding
* Remove old / abandoned images
It can figure out what images ought to be present - including
working out the 'next' release and figuring out from that what
releases it needs images for - and build only the missing ones.
There's a 'version' concept for images; if the existing image
is older than the version given in the data file, it'll be
rebuilt. The data file can list 'rename' pairs, allowing images
to be renamed (like when we move from a single image to multiple
label/filesystem variants). This code uses fedfind's ability
to find the current release version to figure out what releases
we need virtbuilder images for (so you don't have to pass it
in). And it can find image files that aren't in the 'currently
expected' set and wipe them. Images can also have a 'maxage',
triggering a rebuild when they exceed it - this is intended
for the virtbuilder images, so we get a rebuild with the
latest updates every so often (default is two weeks).
The point of all this is to help with unattended deployment/
maintenance, i.e. the ansible deployment we have in infra;
the idea is that we can just set that up to run the 'all'
subcommand every so often, and it'll remove old images, create
new ones, and rebuild ones that are outdated.
I kept the ability to build a single image (or a whole image
'group'), and included the ability to just run a check without
actually doing a rebuild. There's a few little weird things
and holes here as it's not really the focus of the tool.
Test Plan:
Build all images, do a full test run, and see if
it works OK. Test out all the variations of building single
images / image groups, and using the 'check' command.
Reviewers: jskladan, garretraziel
Reviewed By: jskladan, garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D687
2015-12-17 21:25:30 +00:00
|
|
|
}
|
|
|
|
|
],
|
|
|
|
|
"renames" : []
|
|
|
|
|
}
|
