diskimage-builder/diskimage_builder/elements/package-installs/bin/package-installs-v2

143 lines
4.7 KiB
Plaintext
Raw Normal View History

#!/usr/local/bin/dib-python
# Copyright 2014 Hewlett-Packard Development Company, L.P.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
from __future__ import print_function
import argparse
import json
Fix package-installs-v2 output The current output for package-installs-v2 is inscrutable [1] The problem starts with process_output() which is not capturing stderr. This means that any stderr output is dislocated from any stdout output around it. This is *really* confusing as you get a bunch of seemingly meaningless stderr output from any calls before you see any stdout (e.g. in [1] you can see random yum error output that should have been with the yum call)). The simplest thing to do is to redirect stderr to stdout which keeps everything in sync. This causes a slight problem, however, because pkg-map outputs both status information and errors on stderr. To work around this but maintain compatibility, we add a "--prefix" argument that prepends mapped packages from pkg-map with a value we can match on. The existing status/debug output from pkg-map is low-value; modify the call so that it will be traced only at higher debug levels (e.g. -x -x). The current loop is also calling pkg-map for every package in every element (this is why in [1] the same message is repeated over and over). This is unnecessary; it only needs to pkg-map once for each element, giving the package list as the arguments. Create package lists by element and pass those to pkg-map. As a cleanup, there is no point in printing e.output if the process_output fails for the install because we are already tracing it; i.e. the output, even for failures, is already in the logs. Printing it again just duplicates the output. [2] is an extract showing what I feel is a much more understandable log output for a fairly complex install. [1] http://paste.openstack.org/show/595118/ [2] http://paste.openstack.org/show/595303/ Change-Id: Ia74602a5d2db032a476481caec0e45dab013d54f
2017-01-18 00:21:54 +00:00
import os
import subprocess
import sys
Fix package-installs-v2 output The current output for package-installs-v2 is inscrutable [1] The problem starts with process_output() which is not capturing stderr. This means that any stderr output is dislocated from any stdout output around it. This is *really* confusing as you get a bunch of seemingly meaningless stderr output from any calls before you see any stdout (e.g. in [1] you can see random yum error output that should have been with the yum call)). The simplest thing to do is to redirect stderr to stdout which keeps everything in sync. This causes a slight problem, however, because pkg-map outputs both status information and errors on stderr. To work around this but maintain compatibility, we add a "--prefix" argument that prepends mapped packages from pkg-map with a value we can match on. The existing status/debug output from pkg-map is low-value; modify the call so that it will be traced only at higher debug levels (e.g. -x -x). The current loop is also calling pkg-map for every package in every element (this is why in [1] the same message is repeated over and over). This is unnecessary; it only needs to pkg-map once for each element, giving the package list as the arguments. Create package lists by element and pass those to pkg-map. As a cleanup, there is no point in printing e.output if the process_output fails for the install because we are already tracing it; i.e. the output, even for failures, is already in the logs. Printing it again just duplicates the output. [2] is an extract showing what I feel is a much more understandable log output for a fairly complex install. [1] http://paste.openstack.org/show/595118/ [2] http://paste.openstack.org/show/595303/ Change-Id: Ia74602a5d2db032a476481caec0e45dab013d54f
2017-01-18 00:21:54 +00:00
from collections import defaultdict
# run a command, return output
# if follow is set, output will be echoed to stdout
def process_output(cmdline, follow=False):
proc = subprocess.Popen(cmdline, stdout=subprocess.PIPE,
Fix package-installs-v2 output The current output for package-installs-v2 is inscrutable [1] The problem starts with process_output() which is not capturing stderr. This means that any stderr output is dislocated from any stdout output around it. This is *really* confusing as you get a bunch of seemingly meaningless stderr output from any calls before you see any stdout (e.g. in [1] you can see random yum error output that should have been with the yum call)). The simplest thing to do is to redirect stderr to stdout which keeps everything in sync. This causes a slight problem, however, because pkg-map outputs both status information and errors on stderr. To work around this but maintain compatibility, we add a "--prefix" argument that prepends mapped packages from pkg-map with a value we can match on. The existing status/debug output from pkg-map is low-value; modify the call so that it will be traced only at higher debug levels (e.g. -x -x). The current loop is also calling pkg-map for every package in every element (this is why in [1] the same message is repeated over and over). This is unnecessary; it only needs to pkg-map once for each element, giving the package list as the arguments. Create package lists by element and pass those to pkg-map. As a cleanup, there is no point in printing e.output if the process_output fails for the install because we are already tracing it; i.e. the output, even for failures, is already in the logs. Printing it again just duplicates the output. [2] is an extract showing what I feel is a much more understandable log output for a fairly complex install. [1] http://paste.openstack.org/show/595118/ [2] http://paste.openstack.org/show/595303/ Change-Id: Ia74602a5d2db032a476481caec0e45dab013d54f
2017-01-18 00:21:54 +00:00
stderr=subprocess.STDOUT,
universal_newlines=True)
if follow:
print("Running command: %s" % cmdline)
out = ""
with proc.stdout:
for line in iter(proc.stdout.readline, ''):
out += line
print("> %s" % line, end="")
proc.wait()
print("returncode: %d" % proc.returncode)
else:
out = proc.communicate()[0]
if proc.returncode:
e = subprocess.CalledProcessError(proc.returncode, cmdline)
e.output = out
raise e
return out
def main():
Fix package-installs-v2 output The current output for package-installs-v2 is inscrutable [1] The problem starts with process_output() which is not capturing stderr. This means that any stderr output is dislocated from any stdout output around it. This is *really* confusing as you get a bunch of seemingly meaningless stderr output from any calls before you see any stdout (e.g. in [1] you can see random yum error output that should have been with the yum call)). The simplest thing to do is to redirect stderr to stdout which keeps everything in sync. This causes a slight problem, however, because pkg-map outputs both status information and errors on stderr. To work around this but maintain compatibility, we add a "--prefix" argument that prepends mapped packages from pkg-map with a value we can match on. The existing status/debug output from pkg-map is low-value; modify the call so that it will be traced only at higher debug levels (e.g. -x -x). The current loop is also calling pkg-map for every package in every element (this is why in [1] the same message is repeated over and over). This is unnecessary; it only needs to pkg-map once for each element, giving the package list as the arguments. Create package lists by element and pass those to pkg-map. As a cleanup, there is no point in printing e.output if the process_output fails for the install because we are already tracing it; i.e. the output, even for failures, is already in the logs. Printing it again just duplicates the output. [2] is an extract showing what I feel is a much more understandable log output for a fairly complex install. [1] http://paste.openstack.org/show/595118/ [2] http://paste.openstack.org/show/595303/ Change-Id: Ia74602a5d2db032a476481caec0e45dab013d54f
2017-01-18 00:21:54 +00:00
dbg_level = int(os.getenv('DIB_DEBUG_TRACE', '0'))
parser = argparse.ArgumentParser(
description="Install or uninstall packages for a specific phase based"
" on package-installs files.")
parser.add_argument('--phase', required=True,
help="Install phase to filter on. Valid options are"
" 'install.d' or pre-install.d")
parser.add_argument('--uninstall', action="store_true",
help="Only show packages to uninstall. By default only"
" packages to install are shown")
parser.add_argument('-n', '--noop', action="store_true",
help="Don't actually install, just print the command")
parser.add_argument('infile', help="File to process")
args = parser.parse_args()
packages = json.load(open(args.infile))
if args.uninstall:
install = "uninstall"
else:
install = "install"
pkgs = list()
if args.phase in packages and install in packages[args.phase]:
install_packages = packages[args.phase][install]
else:
print("Nothing to %s" % install)
sys.exit(0)
Fix package-installs-v2 output The current output for package-installs-v2 is inscrutable [1] The problem starts with process_output() which is not capturing stderr. This means that any stderr output is dislocated from any stdout output around it. This is *really* confusing as you get a bunch of seemingly meaningless stderr output from any calls before you see any stdout (e.g. in [1] you can see random yum error output that should have been with the yum call)). The simplest thing to do is to redirect stderr to stdout which keeps everything in sync. This causes a slight problem, however, because pkg-map outputs both status information and errors on stderr. To work around this but maintain compatibility, we add a "--prefix" argument that prepends mapped packages from pkg-map with a value we can match on. The existing status/debug output from pkg-map is low-value; modify the call so that it will be traced only at higher debug levels (e.g. -x -x). The current loop is also calling pkg-map for every package in every element (this is why in [1] the same message is repeated over and over). This is unnecessary; it only needs to pkg-map once for each element, giving the package list as the arguments. Create package lists by element and pass those to pkg-map. As a cleanup, there is no point in printing e.output if the process_output fails for the install because we are already tracing it; i.e. the output, even for failures, is already in the logs. Printing it again just duplicates the output. [2] is an extract showing what I feel is a much more understandable log output for a fairly complex install. [1] http://paste.openstack.org/show/595118/ [2] http://paste.openstack.org/show/595303/ Change-Id: Ia74602a5d2db032a476481caec0e45dab013d54f
2017-01-18 00:21:54 +00:00
# sort the list by element, this way we only do one pkg-map call
# per element
by_element = defaultdict(list)
for (pkg, element) in install_packages:
Fix package-installs-v2 output The current output for package-installs-v2 is inscrutable [1] The problem starts with process_output() which is not capturing stderr. This means that any stderr output is dislocated from any stdout output around it. This is *really* confusing as you get a bunch of seemingly meaningless stderr output from any calls before you see any stdout (e.g. in [1] you can see random yum error output that should have been with the yum call)). The simplest thing to do is to redirect stderr to stdout which keeps everything in sync. This causes a slight problem, however, because pkg-map outputs both status information and errors on stderr. To work around this but maintain compatibility, we add a "--prefix" argument that prepends mapped packages from pkg-map with a value we can match on. The existing status/debug output from pkg-map is low-value; modify the call so that it will be traced only at higher debug levels (e.g. -x -x). The current loop is also calling pkg-map for every package in every element (this is why in [1] the same message is repeated over and over). This is unnecessary; it only needs to pkg-map once for each element, giving the package list as the arguments. Create package lists by element and pass those to pkg-map. As a cleanup, there is no point in printing e.output if the process_output fails for the install because we are already tracing it; i.e. the output, even for failures, is already in the logs. Printing it again just duplicates the output. [2] is an extract showing what I feel is a much more understandable log output for a fairly complex install. [1] http://paste.openstack.org/show/595118/ [2] http://paste.openstack.org/show/595303/ Change-Id: Ia74602a5d2db032a476481caec0e45dab013d54f
2017-01-18 00:21:54 +00:00
by_element[element].append(pkg)
for element, packages in by_element.items():
print("Map %s for %s: %s" % (install, element, ', '.join(packages)))
# Only trace pkg-map for higher levels of debugging. Note
# that pkg-map debug output comes out on stderr, which is
# captured into the output by process_output. We filter by
# "prefix" so we don't think the debug lines are packages!
pkg_map_args = ['pkg-map', '--prefix', '-',
'--missing-ok', '--element', element]
if dbg_level > 1:
pkg_map_args.append('--debug')
pkg_map_args.extend(packages)
try:
Fix package-installs-v2 output The current output for package-installs-v2 is inscrutable [1] The problem starts with process_output() which is not capturing stderr. This means that any stderr output is dislocated from any stdout output around it. This is *really* confusing as you get a bunch of seemingly meaningless stderr output from any calls before you see any stdout (e.g. in [1] you can see random yum error output that should have been with the yum call)). The simplest thing to do is to redirect stderr to stdout which keeps everything in sync. This causes a slight problem, however, because pkg-map outputs both status information and errors on stderr. To work around this but maintain compatibility, we add a "--prefix" argument that prepends mapped packages from pkg-map with a value we can match on. The existing status/debug output from pkg-map is low-value; modify the call so that it will be traced only at higher debug levels (e.g. -x -x). The current loop is also calling pkg-map for every package in every element (this is why in [1] the same message is repeated over and over). This is unnecessary; it only needs to pkg-map once for each element, giving the package list as the arguments. Create package lists by element and pass those to pkg-map. As a cleanup, there is no point in printing e.output if the process_output fails for the install because we are already tracing it; i.e. the output, even for failures, is already in the logs. Printing it again just duplicates the output. [2] is an extract showing what I feel is a much more understandable log output for a fairly complex install. [1] http://paste.openstack.org/show/595118/ [2] http://paste.openstack.org/show/595303/ Change-Id: Ia74602a5d2db032a476481caec0e45dab013d54f
2017-01-18 00:21:54 +00:00
follow = True if dbg_level > 1 else False
map_output = process_output(pkg_map_args, follow=follow)
map_output = map_output.strip().split('\n')
map_output = [m[1:] for m in map_output if m.startswith('-')]
pkgs.extend(map_output)
except subprocess.CalledProcessError as e:
if e.returncode == 1:
if args.noop:
pkgs.append(pkg)
continue
else:
print("pkg-map failed")
sys.exit(1)
elif e.returncode == 2:
pkgs.append(pkg)
continue
install_args = ["install-packages"]
if args.uninstall:
install_args.append("-e")
install_args.extend(list(set(pkgs)))
if args.noop:
print(" ".join(install_args))
else:
try:
process_output(install_args, follow=True)
except subprocess.CalledProcessError as e:
Fix package-installs-v2 output The current output for package-installs-v2 is inscrutable [1] The problem starts with process_output() which is not capturing stderr. This means that any stderr output is dislocated from any stdout output around it. This is *really* confusing as you get a bunch of seemingly meaningless stderr output from any calls before you see any stdout (e.g. in [1] you can see random yum error output that should have been with the yum call)). The simplest thing to do is to redirect stderr to stdout which keeps everything in sync. This causes a slight problem, however, because pkg-map outputs both status information and errors on stderr. To work around this but maintain compatibility, we add a "--prefix" argument that prepends mapped packages from pkg-map with a value we can match on. The existing status/debug output from pkg-map is low-value; modify the call so that it will be traced only at higher debug levels (e.g. -x -x). The current loop is also calling pkg-map for every package in every element (this is why in [1] the same message is repeated over and over). This is unnecessary; it only needs to pkg-map once for each element, giving the package list as the arguments. Create package lists by element and pass those to pkg-map. As a cleanup, there is no point in printing e.output if the process_output fails for the install because we are already tracing it; i.e. the output, even for failures, is already in the logs. Printing it again just duplicates the output. [2] is an extract showing what I feel is a much more understandable log output for a fairly complex install. [1] http://paste.openstack.org/show/595118/ [2] http://paste.openstack.org/show/595303/ Change-Id: Ia74602a5d2db032a476481caec0e45dab013d54f
2017-01-18 00:21:54 +00:00
print("install-packages failed with returncode %d" % e.returncode)
sys.exit(1)
if __name__ == '__main__':
main()
# Tell emacs to use python-mode
# Local variables:
# mode: python
# End: