mirrors/u-boot
1
0
Fork 0
mirror of https://source.denx.de/u-boot/u-boot.git synced 2025-08-30 19:11:37 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity

Merge branch '2022-08-05-buildman-integrate-boardscfg'

Browse Source 
To quote Simon:
This series drops the need for the genboardscfg.py script, so that the
boards.cfg file is produced (and consumed) entirely within buildman. The
file is not entirely removed since it does have some uses and we need some
sort of cache for the information. The genboardscfg.py script is
effectively incorporated in buildman.

It also improves operation from an IDE with a new -I option and fixes up
some of the pylint warnings in buildman.

Finally, this series also fixes a bug which allows use to drop support for
CONFIG_SYS_EXTRA_OPTIONS which is long-standing desire. It also fixes a
minor bug that causes 'Invalid line' spam when checking for function bloat
with the -B option.
This commit is contained in:
Tom Rini 2022-08-05 13:22:44 -04:00
commit 56edbb5eaf
20 changed files with 958 additions and 1004 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
.azure-pipelines.yml
View File

@ -140,7 +140,7 @@ stages:
options: $(container_option) options: $(container_option)
steps: steps:
- script: | - script: |
if [ `./tools/genboardscfg.py -f 2>&1 | wc -l` -ne 0 ]; then exit 1; fi ./tools/buildman/buildman -R
- job: tools_only - job: tools_only
displayName: 'Ensure host tools build' displayName: 'Ensure host tools build'

2
.gitlab-ci.yml
View File

@ -180,7 +180,7 @@ sloccount:
Check for configs without MAINTAINERS entry: Check for configs without MAINTAINERS entry:
stage: testsuites stage: testsuites
script: script:
- if [ `./tools/genboardscfg.py -f 2>&1 | wc -l` -ne 0 ]; then exit 1; fi - ./tools/buildman/buildman -R
# Ensure host tools build # Ensure host tools build
Build tools-only: Build tools-only:

13
boot/Kconfig
View File

@ -452,19 +452,6 @@ config OF_STDOUT_VIA_ALIAS
incorrect when used with device tree as this option does not incorrect when used with device tree as this option does not
exist / should not be used. exist / should not be used.
config SYS_EXTRA_OPTIONS
string "Extra Options (DEPRECATED)"
help
The old configuration infrastructure (= mkconfig + boards.cfg)
provided the extra options field. If you have something like
"HAS_BAR,BAZ=64", the optional options
#define CONFIG_HAS
#define CONFIG_BAZ 64
will be defined in include/config.h.
This option was prepared for the smooth migration from the old
configuration to Kconfig. Since this option will be removed sometime,
new boards should not use this option.
config HAVE_SYS_TEXT_BASE config HAVE_SYS_TEXT_BASE
bool bool
depends on !NIOS2 && !XTENSA depends on !NIOS2 && !XTENSA

7
doc/README.kconfig
View File

@ -99,7 +99,6 @@ Kconfig. Each field of boards.cfg was converted as follows:
Vendor -> CONFIG_SYS_VENDOR defined by Kconfig Vendor -> CONFIG_SYS_VENDOR defined by Kconfig
Board -> CONFIG_SYS_BOARD defined by Kconfig Board -> CONFIG_SYS_BOARD defined by Kconfig
Target -> File name of defconfig (configs/<target>_defconfig) Target -> File name of defconfig (configs/<target>_defconfig)
Options -> CONFIG_SYS_EXTRA_OPTIONS defined by Kconfig
Maintainers -> "M:" entry of MAINTAINERS Maintainers -> "M:" entry of MAINTAINERS
@ -140,12 +139,6 @@ When removing an obsolete board, the following steps are generally needed:
TODO TODO
---- ----
- The option field of boards.cfg, which was used for the pre-Kconfig
configuration, moved to CONFIG_SYS_EXTRA_OPTIONS verbatim now.
Board maintainers are expected to implement proper Kconfig options
and switch over to them. Eventually CONFIG_SYS_EXTRA_OPTIONS will go away.
CONFIG_SYS_EXTRA_OPTIONS should not be used for new boards.
- In the pre-Kconfig, a single board had multiple entries in the boards.cfg - In the pre-Kconfig, a single board had multiple entries in the boards.cfg
file with differences in the option fields. The corresponding defconfig file with differences in the option fields. The corresponding defconfig
files were auto-generated when switching to Kconfig. Now we have too many files were auto-generated when switching to Kconfig. Now we have too many

3
doc/develop/moveconfig.rst
View File

@ -295,8 +295,7 @@ Available options
-y, --yes -y, --yes
Instead of prompting, automatically go ahead with all operations. This Instead of prompting, automatically go ahead with all operations. This
includes cleaning up headers, CONFIG_SYS_EXTRA_OPTIONS, the config whitelist includes cleaning up headers, the config whitelist and the README.
and the README.
To see the complete list of supported options, run:: To see the complete list of supported options, run::

4
scripts/Makefile.autoconf
View File

@ -112,10 +112,6 @@ vpl/include/autoconf.mk: vpl/u-boot.cfg
# Prior to Kconfig, it was generated by mkconfig. Now it is created here. # Prior to Kconfig, it was generated by mkconfig. Now it is created here.
define filechk_config_h define filechk_config_h
(echo "/* Automatically generated - do not edit */"; \ (echo "/* Automatically generated - do not edit */"; \
for i in $$(echo $(CONFIG_SYS_EXTRA_OPTIONS) | sed 's/,/ /g'); do \
echo \#define CONFIG_$$i \
| sed '/=/ {s/=/ /;q; } ; { s/$$/ 1/; }'; \
done; \
echo \#define CONFIG_BOARDDIR board/$(if $(VENDOR),$(VENDOR)/)$(BOARD);\ echo \#define CONFIG_BOARDDIR board/$(if $(VENDOR),$(VENDOR)/)$(BOARD);\
echo \#include \<config_uncmd_spl.h\>; \ echo \#include \<config_uncmd_spl.h\>; \
echo \#include \<configs/$(CONFIG_SYS_CONFIG_NAME).h\>; \ echo \#include \<configs/$(CONFIG_SYS_CONFIG_NAME).h\>; \

23
scripts/build-whitelist.sh
View File

@ -10,30 +10,13 @@
# #
export LC_ALL=C LC_COLLATE=C export LC_ALL=C LC_COLLATE=C
# There are two independent greps. The first pulls out the component parts # Looks for the rest of the CONFIG options, but exclude those in Kconfig and
# of CONFIG_SYS_EXTRA_OPTIONS. An example is: # defconfig files.
# #
# SUN7I_GMAC,AHCI,SATAPWR=SUNXI_GPB(8)
#
# We want this to produce:
# CONFIG_SUN7I_GMAC
# CONFIG_AHCI
# CONFIG_SATAPWR
#
# The second looks for the rest of the CONFIG options, but excludes those in
# Kconfig and defconfig files.
#
(
git grep CONFIG_SYS_EXTRA_OPTIONS |sed -n \
's/.*CONFIG_SYS_EXTRA_OPTIONS="\(.*\)"/\1/ p' \
| tr , '\n' \
| sed 's/ *\([A-Za-z0-9_]*\).*/CONFIG_\1/'
git grep CONFIG_ | \ git grep CONFIG_ | \
egrep -vi "(Kconfig:|defconfig:|README|\.py|\.pl:)" \ egrep -vi "(Kconfig:|defconfig:|README|\.py|\.pl:)" \
| tr ' \t' '\n\n' \ | tr ' \t' '\n\n' \
| sed -n 's/^\(CONFIG_[A-Za-z0-9_]*\).*/\1/p' | sed -n 's/^\(CONFIG_[A-Za-z0-9_]*\).*/\1/p' \
) \
|sort |uniq >scripts/config_whitelist.txt.tmp1; |sort |uniq >scripts/config_whitelist.txt.tmp1;
# Finally, we need a list of the valid Kconfig options to exclude these from # Finally, we need a list of the valid Kconfig options to exclude these from

1
scripts/pylint.base
View File

@ -201,7 +201,6 @@ tools_dtoc_test_fdt 6.88
tools_dtoc_test_src_scan 9.43 tools_dtoc_test_src_scan 9.43
tools_efivar 6.71 tools_efivar 6.71
tools_endian-swap 9.29 tools_endian-swap 9.29
tools_genboardscfg 7.95
tools_microcode-tool 7.25 tools_microcode-tool 7.25
tools_moveconfig 8.34 tools_moveconfig 8.34
tools_patman___init__ 0.00 tools_patman___init__ 0.00

40
tools/buildman/README
View File

@ -128,10 +128,10 @@ Selecting which boards to build
=============================== ===============================
Buildman lets you build all boards, or a subset. Specify the subset by passing Buildman lets you build all boards, or a subset. Specify the subset by passing
command-line arguments that list the desired board name, architecture name, command-line arguments that list the desired build target, architecture,
SOC name, or anything else in the boards.cfg file. Multiple arguments are CPU, board name, vendor, SoC or options. Multiple arguments are allowed. Each
allowed. Each argument will be interpreted as a regular expression, so argument will be interpreted as a regular expression, so behaviour is a superset
behaviour is a superset of exact or substring matching. Examples are: of exact or substring matching. Examples are:
* 'tegra20' All boards with a Tegra20 SoC * 'tegra20' All boards with a Tegra20 SoC
* 'tegra' All boards with any Tegra Soc (Tegra20, Tegra30, Tegra114...) * 'tegra' All boards with any Tegra Soc (Tegra20, Tegra30, Tegra114...)
@ -1054,7 +1054,6 @@ between one commit and the next.
For example: For example:
$ buildman -b squash brppt1 -sU $ buildman -b squash brppt1 -sU
boards.cfg is up to date. Nothing to do.
Summary of 2 commits for 3 boards (3 threads, 3 jobs per thread) Summary of 2 commits for 3 boards (3 threads, 3 jobs per thread)
01: Migrate bootlimit to Kconfig 01: Migrate bootlimit to Kconfig
02: Squashed commit of the following: 02: Squashed commit of the following:
@ -1092,6 +1091,21 @@ This will write the full build into /tmp/build including object files. You must
specify the output directory with -o when using -w. specify the output directory with -o when using -w.
Support for IDEs (Integrated Development Environments)
======================================================
Normally buildman summarises the output and shows information indicating the
meaning of each line of output. For example a '+' symbol appears at the start of
each error line. Also, buildman prints information about what it is about to do,
along with a summary at the end.
When using buildman from an IDE, it is helpful to drop this behaviour. Use the
-I/--ide option for that. You might find -W helpful also so that warnings do
not cause the build to fail:
buildman -o /tmp/build --board sandbox -wWI
Changing the configuration Changing the configuration
========================== ==========================
@ -1294,6 +1308,19 @@ Some options you might like are:
break anything. But note this does not check bisectability! break anything. But note this does not check bisectability!
Using boards.cfg
================
This file is no-longer needed by buildman but it is still generated in the
working directory. This helps avoid a delay on every build, since scanning all
the Kconfig files takes a few seconds. Use the -R flag to force regeneration
of the file - in that case buildman exits after writing the file. with exit code
2 if there was an error in the maintainer files.
You should use 'buildman -nv <criteria>' instead of greoing the boards.cfg file,
since it may be dropped altogether in future.
TODO TODO
==== ====
@ -1304,9 +1331,6 @@ scope for more though, e.g.:
- 'hunting' for problems, perhaps by building a few boards for each arch, or - 'hunting' for problems, perhaps by building a few boards for each arch, or
checking commits for changed files and building only boards which use those checking commits for changed files and building only boards which use those
files files
- using the same git repo for all threads instead of cloning it. Currently
it uses about 500MB per thread, so on a 64-thread machine this is 32GB for
the build.
Credits Credits

289
tools/buildman/board.py
View File

@ -1,78 +1,12 @@
# SPDX-License-Identifier: GPL-2.0+ # SPDX-License-Identifier: GPL-2.0+
# Copyright (c) 2012 The Chromium OS Authors. # Copyright (c) 2012 The Chromium OS Authors.
from collections import OrderedDict
import re
class Expr: """A single board which can be selected and built"""
"""A single regular expression for matching boards to build"""
def __init__(self, expr):
"""Set up a new Expr object.
Args:
expr: String cotaining regular expression to store
"""
self._expr = expr
self._re = re.compile(expr)
def Matches(self, props):
"""Check if any of the properties match the regular expression.
Args:
props: List of properties to check
Returns:
True if any of the properties match the regular expression
"""
for prop in props:
if self._re.match(prop):
return True
return False
def __str__(self):
return self._expr
class Term:
"""A list of expressions each of which must match with properties.
This provides a list of 'AND' expressions, meaning that each must
match the board properties for that board to be built.
"""
def __init__(self):
self._expr_list = []
self._board_count = 0
def AddExpr(self, expr):
"""Add an Expr object to the list to check.
Args:
expr: New Expr object to add to the list of those that must
match for a board to be built.
"""
self._expr_list.append(Expr(expr))
def __str__(self):
"""Return some sort of useful string describing the term"""
return '&'.join([str(expr) for expr in self._expr_list])
def Matches(self, props):
"""Check if any of the properties match this term
Each of the expressions in the term is checked. All must match.
Args:
props: List of properties to check
Returns:
True if all of the expressions in the Term match, else False
"""
for expr in self._expr_list:
if not expr.Matches(props):
return False
return True
class Board: class Board:
"""A particular board that we can build""" """A particular board that we can build"""
def __init__(self, status, arch, cpu, soc, vendor, board_name, target, options): def __init__(self, status, arch, cpu, soc, vendor, board_name, target, cfg_name):
"""Create a new board type. """Create a new board type.
Args: Args:
@ -83,7 +17,7 @@ class Board:
vendor: Name of vendor (e.g. armltd) vendor: Name of vendor (e.g. armltd)
board_name: Name of board (e.g. integrator) board_name: Name of board (e.g. integrator)
target: Target name (use make <target>_defconfig to configure) target: Target name (use make <target>_defconfig to configure)
options: board-specific options (e.g. integratorcp:CM1136) cfg_name: Config name
""" """
self.target = target self.target = target
self.arch = arch self.arch = arch
@ -91,220 +25,7 @@ class Board:
self.board_name = board_name self.board_name = board_name
self.vendor = vendor self.vendor = vendor
self.soc = soc self.soc = soc
self.options = options self.cfg_name = cfg_name
self.props = [self.target, self.arch, self.cpu, self.board_name, self.props = [self.target, self.arch, self.cpu, self.board_name,
self.vendor, self.soc, self.options] self.vendor, self.soc, self.cfg_name]
self.build_it = False self.build_it = False
class Boards:
"""Manage a list of boards."""
def __init__(self):
# Use a simple list here, sinc OrderedDict requires Python 2.7
self._boards = []
def AddBoard(self, board):
"""Add a new board to the list.
The board's target member must not already exist in the board list.
Args:
board: board to add
"""
self._boards.append(board)
def ReadBoards(self, fname):
"""Read a list of boards from a board file.
Create a board object for each and add it to our _boards list.
Args:
fname: Filename of boards.cfg file
"""
with open(fname, 'r', encoding='utf-8') as fd:
for line in fd:
if line[0] == '#':
continue
fields = line.split()
if not fields:
continue
for upto in range(len(fields)):
if fields[upto] == '-':
fields[upto] = ''
while len(fields) < 8:
fields.append('')
if len(fields) > 8:
fields = fields[:8]
board = Board(*fields)
self.AddBoard(board)
def GetList(self):
"""Return a list of available boards.
Returns:
List of Board objects
"""
return self._boards
def GetDict(self):
"""Build a dictionary containing all the boards.
Returns:
Dictionary:
key is board.target
value is board
"""
board_dict = OrderedDict()
for board in self._boards:
board_dict[board.target] = board
return board_dict
def GetSelectedDict(self):
"""Return a dictionary containing the selected boards
Returns:
List of Board objects that are marked selected
"""
board_dict = OrderedDict()
for board in self._boards:
if board.build_it:
board_dict[board.target] = board
return board_dict
def GetSelected(self):
"""Return a list of selected boards
Returns:
List of Board objects that are marked selected
"""
return [board for board in self._boards if board.build_it]
def GetSelectedNames(self):
"""Return a list of selected boards
Returns:
List of board names that are marked selected
"""
return [board.target for board in self._boards if board.build_it]
def _BuildTerms(self, args):
"""Convert command line arguments to a list of terms.
This deals with parsing of the arguments. It handles the '&'
operator, which joins several expressions into a single Term.
For example:
['arm & freescale sandbox', 'tegra']
will produce 3 Terms containing expressions as follows:
arm, freescale
sandbox
tegra
The first Term has two expressions, both of which must match for
a board to be selected.
Args:
args: List of command line arguments
Returns:
A list of Term objects
"""
syms = []
for arg in args:
for word in arg.split():
sym_build = []
for term in word.split('&'):
if term:
sym_build.append(term)
sym_build.append('&')
syms += sym_build[:-1]
terms = []
term = None
oper = None
for sym in syms:
if sym == '&':
oper = sym
elif oper:
term.AddExpr(sym)
oper = None
else:
if term:
terms.append(term)
term = Term()
term.AddExpr(sym)
if term:
terms.append(term)
return terms
def SelectBoards(self, args, exclude=[], boards=None):
"""Mark boards selected based on args
Normally either boards (an explicit list of boards) or args (a list of
terms to match against) is used. It is possible to specify both, in
which case they are additive.
If boards and args are both empty, all boards are selected.
Args:
args: List of strings specifying boards to include, either named,
or by their target, architecture, cpu, vendor or soc. If
empty, all boards are selected.
exclude: List of boards to exclude, regardless of 'args'
boards: List of boards to build
Returns:
Tuple
Dictionary which holds the list of boards which were selected
due to each argument, arranged by argument.
List of errors found
"""
result = OrderedDict()
warnings = []
terms = self._BuildTerms(args)
result['all'] = []
for term in terms:
result[str(term)] = []
exclude_list = []
for expr in exclude:
exclude_list.append(Expr(expr))
found = []
for board in self._boards:
matching_term = None
build_it = False
if terms:
match = False
for term in terms:
if term.Matches(board.props):
matching_term = str(term)
build_it = True
break
elif boards:
if board.target in boards:
build_it = True
found.append(board.target)
else:
build_it = True
# Check that it is not specifically excluded
for expr in exclude_list:
if expr.Matches(board.props):
build_it = False
break
if build_it:
board.build_it = True
if matching_term:
result[matching_term].append(board.target)
result['all'].append(board.target)
if boards:
remaining = set(boards) - set(found)
if remaining:
warnings.append('Boards not found: %s\n' % ', '.join(remaining))
return result, warnings

752
tools/buildman/boards.py Normal file
View File

@ -0,0 +1,752 @@
# SPDX-License-Identifier: GPL-2.0+
# Copyright (c) 2012 The Chromium OS Authors.
# Author: Simon Glass <sjg@chromium.org>
# Author: Masahiro Yamada <yamada.m@jp.panasonic.com>
"""Maintains a list of boards and allows them to be selected"""
from collections import OrderedDict
import errno
import fnmatch
import glob
import multiprocessing
import os
import re
import sys
import tempfile
import time
from buildman import board
from buildman import kconfiglib
### constant variables ###
OUTPUT_FILE = 'boards.cfg'
CONFIG_DIR = 'configs'
SLEEP_TIME = 0.03
COMMENT_BLOCK = f'''#
# List of boards
# Automatically generated by {__file__}: don't edit
#
# Status, Arch, CPU, SoC, Vendor, Board, Target, Config, Maintainers
'''
def try_remove(fname):
"""Remove a file ignoring 'No such file or directory' error.
Args:
fname (str): Filename to remove
Raises:
OSError: output file exists but could not be removed
"""
try:
os.remove(fname)
except OSError as exception:
# Ignore 'No such file or directory' error
if exception.errno != errno.ENOENT:
raise
def output_is_new(output):
"""Check if the output file is up to date.
Looks at defconfig and Kconfig files to make sure none is newer than the
output file. Also ensures that the boards.cfg does not mention any removed
boards.
Args:
output (str): Filename to check
Returns:
True if the given output file exists and is newer than any of
*_defconfig, MAINTAINERS and Kconfig*. False otherwise.
Raises:
OSError: output file exists but could not be opened
"""
# pylint: disable=too-many-branches
try:
ctime = os.path.getctime(output)
except OSError as exception:
if exception.errno == errno.ENOENT:
# return False on 'No such file or directory' error
return False
raise
for (dirpath, _, filenames) in os.walk(CONFIG_DIR):
for filename in fnmatch.filter(filenames, '*_defconfig'):
if fnmatch.fnmatch(filename, '.*'):
continue
filepath = os.path.join(dirpath, filename)
if ctime < os.path.getctime(filepath):
return False
for (dirpath, _, filenames) in os.walk('.'):
for filename in filenames:
if (fnmatch.fnmatch(filename, '*~') or
not fnmatch.fnmatch(filename, 'Kconfig*') and
not filename == 'MAINTAINERS'):
continue
filepath = os.path.join(dirpath, filename)
if ctime < os.path.getctime(filepath):
return False
# Detect a board that has been removed since the current board database
# was generated
with open(output, encoding="utf-8") as inf:
for line in inf:
if 'Options,' in line:
return False
if line[0] == '#' or line == '\n':
continue
defconfig = line.split()[6] + '_defconfig'
if not os.path.exists(os.path.join(CONFIG_DIR, defconfig)):
return False
return True
class Expr:
"""A single regular expression for matching boards to build"""
def __init__(self, expr):
"""Set up a new Expr object.
Args:
expr (str): String cotaining regular expression to store
"""
self._expr = expr
self._re = re.compile(expr)
def matches(self, props):
"""Check if any of the properties match the regular expression.
Args:
props (list of str): List of properties to check
Returns:
True if any of the properties match the regular expression
"""
for prop in props:
if self._re.match(prop):
return True
return False
def __str__(self):
return self._expr
class Term:
"""A list of expressions each of which must match with properties.
This provides a list of 'AND' expressions, meaning that each must
match the board properties for that board to be built.
"""
def __init__(self):
self._expr_list = []
self._board_count = 0
def add_expr(self, expr):
"""Add an Expr object to the list to check.
Args:
expr (Expr): New Expr object to add to the list of those that must
match for a board to be built.
"""
self._expr_list.append(Expr(expr))
def __str__(self):
"""Return some sort of useful string describing the term"""
return '&'.join([str(expr) for expr in self._expr_list])
def matches(self, props):
"""Check if any of the properties match this term
Each of the expressions in the term is checked. All must match.
Args:
props (list of str): List of properties to check
Returns:
True if all of the expressions in the Term match, else False
"""
for expr in self._expr_list:
if not expr.matches(props):
return False
return True
class KconfigScanner:
"""Kconfig scanner."""
### constant variable only used in this class ###
_SYMBOL_TABLE = {
'arch' : 'SYS_ARCH',
'cpu' : 'SYS_CPU',
'soc' : 'SYS_SOC',
'vendor' : 'SYS_VENDOR',
'board' : 'SYS_BOARD',
'config' : 'SYS_CONFIG_NAME',
# 'target' is added later
}
def __init__(self):
"""Scan all the Kconfig files and create a Kconfig object."""
# Define environment variables referenced from Kconfig
os.environ['srctree'] = os.getcwd()
os.environ['UBOOTVERSION'] = 'dummy'
os.environ['KCONFIG_OBJDIR'] = ''
self._tmpfile = None
self._conf = kconfiglib.Kconfig(warn=False)
def __del__(self):
"""Delete a leftover temporary file before exit.
The scan() method of this class creates a temporay file and deletes
it on success. If scan() method throws an exception on the way,
the temporary file might be left over. In that case, it should be
deleted in this destructor.
"""
if self._tmpfile:
try_remove(self._tmpfile)
def scan(self, defconfig):
"""Load a defconfig file to obtain board parameters.
Args:
defconfig (str): path to the defconfig file to be processed
Returns:
A dictionary of board parameters. It has a form of:
{
'arch': <arch_name>,
'cpu': <cpu_name>,
'soc': <soc_name>,
'vendor': <vendor_name>,
'board': <board_name>,
'target': <target_name>,
'config': <config_header_name>,
}
"""
# strip special prefixes and save it in a temporary file
outfd, self._tmpfile = tempfile.mkstemp()
with os.fdopen(outfd, 'w') as outf:
with open(defconfig, encoding='utf-8') as inf:
for line in inf:
colon = line.find(':CONFIG_')
if colon == -1:
outf.write(line)
else:
outf.write(line[colon + 1:])
self._conf.load_config(self._tmpfile)
try_remove(self._tmpfile)
self._tmpfile = None
params = {}
# Get the value of CONFIG_SYS_ARCH, CONFIG_SYS_CPU, ... etc.
# Set '-' if the value is empty.
for key, symbol in list(self._SYMBOL_TABLE.items()):
value = self._conf.syms.get(symbol).str_value
if value:
params[key] = value
else:
params[key] = '-'
defconfig = os.path.basename(defconfig)
params['target'], match, rear = defconfig.partition('_defconfig')
assert match and not rear, f'{defconfig} : invalid defconfig'
# fix-up for aarch64
if params['arch'] == 'arm' and params['cpu'] == 'armv8':
params['arch'] = 'aarch64'
return params
class MaintainersDatabase:
"""The database of board status and maintainers.
Properties:
database: dict:
key: Board-target name (e.g. 'snow')
value: tuple:
str: Board status (e.g. 'Active')
str: List of maintainers, separated by :
warnings (list of str): List of warnings due to missing status, etc.
"""
def __init__(self):
"""Create an empty database."""
self.database = {}
self.warnings = []
def get_status(self, target):
"""Return the status of the given board.
The board status is generally either 'Active' or 'Orphan'.
Display a warning message and return '-' if status information
is not found.
Args:
target (str): Build-target name
Returns:
str: 'Active', 'Orphan' or '-'.
"""
if not target in self.database:
self.warnings.append(f"WARNING: no status info for '{target}'")
return '-'
tmp = self.database[target][0]
if tmp.startswith('Maintained'):
return 'Active'
if tmp.startswith('Supported'):
return 'Active'
if tmp.startswith('Orphan'):
return 'Orphan'
self.warnings.append(f"WARNING: {tmp}: unknown status for '{target}'")
return '-'
def get_maintainers(self, target):
"""Return the maintainers of the given board.
Args:
target (str): Build-target name
Returns:
str: Maintainers of the board. If the board has two or more
maintainers, they are separated with colons.
"""
if not target in self.database:
self.warnings.append(f"WARNING: no maintainers for '{target}'")
return ''
return ':'.join(self.database[target][1])
def parse_file(self, fname):
"""Parse a MAINTAINERS file.
Parse a MAINTAINERS file and accumulate board status and maintainers
information in the self.database dict.
Args:
fname (str): MAINTAINERS file to be parsed
"""
targets = []
maintainers = []
status = '-'
with open(fname, encoding="utf-8") as inf:
for line in inf:
# Check also commented maintainers
if line[:3] == '#M:':
line = line[1:]
tag, rest = line[:2], line[2:].strip()
if tag == 'M:':
maintainers.append(rest)
elif tag == 'F:':
# expand wildcard and filter by 'configs/*_defconfig'
for item in glob.glob(rest):
front, match, rear = item.partition('configs/')
if not front and match:
front, match, rear = rear.rpartition('_defconfig')
if match and not rear:
targets.append(front)
elif tag == 'S:':
status = rest
elif line == '\n':
for target in targets:
self.database[target] = (status, maintainers)
targets = []
maintainers = []
status = '-'
if targets:
for target in targets:
self.database[target] = (status, maintainers)
class Boards:
"""Manage a list of boards."""
def __init__(self):
self._boards = []
def add_board(self, brd):
"""Add a new board to the list.
The board's target member must not already exist in the board list.
Args:
brd (Board): board to add
"""
self._boards.append(brd)
def read_boards(self, fname):
"""Read a list of boards from a board file.
Create a Board object for each and add it to our _boards list.
Args:
fname (str): Filename of boards.cfg file
"""
with open(fname, 'r', encoding='utf-8') as inf:
for line in inf:
if line[0] == '#':
continue
fields = line.split()
if not fields:
continue
for upto, field in enumerate(fields):
if field == '-':
fields[upto] = ''
while len(fields) < 8:
fields.append('')
if len(fields) > 8:
fields = fields[:8]
brd = board.Board(*fields)
self.add_board(brd)
def get_list(self):
"""Return a list of available boards.
Returns:
List of Board objects
"""
return self._boards
def get_dict(self):
"""Build a dictionary containing all the boards.
Returns:
Dictionary:
key is board.target
value is board
"""
board_dict = OrderedDict()
for brd in self._boards:
board_dict[brd.target] = brd
return board_dict
def get_selected_dict(self):
"""Return a dictionary containing the selected boards
Returns:
List of Board objects that are marked selected
"""
board_dict = OrderedDict()
for brd in self._boards:
if brd.build_it:
board_dict[brd.target] = brd
return board_dict
def get_selected(self):
"""Return a list of selected boards
Returns:
List of Board objects that are marked selected
"""
return [brd for brd in self._boards if brd.build_it]
def get_selected_names(self):
"""Return a list of selected boards
Returns:
List of board names that are marked selected
"""
return [brd.target for brd in self._boards if brd.build_it]
@classmethod
def _build_terms(cls, args):
"""Convert command line arguments to a list of terms.
This deals with parsing of the arguments. It handles the '&'
operator, which joins several expressions into a single Term.
For example:
['arm & freescale sandbox', 'tegra']
will produce 3 Terms containing expressions as follows:
arm, freescale
sandbox
tegra
The first Term has two expressions, both of which must match for
a board to be selected.
Args:
args (list of str): List of command line arguments
Returns:
list of Term: A list of Term objects
"""
syms = []
for arg in args:
for word in arg.split():
sym_build = []
for term in word.split('&'):
if term:
sym_build.append(term)
sym_build.append('&')
syms += sym_build[:-1]
terms = []
term = None
oper = None
for sym in syms:
if sym == '&':
oper = sym
elif oper:
term.add_expr(sym)
oper = None
else:
if term:
terms.append(term)
term = Term()
term.add_expr(sym)
if term:
terms.append(term)
return terms
def select_boards(self, args, exclude=None, brds=None):
"""Mark boards selected based on args
Normally either boards (an explicit list of boards) or args (a list of
terms to match against) is used. It is possible to specify both, in
which case they are additive.
If brds and args are both empty, all boards are selected.
Args:
args (list of str): List of strings specifying boards to include,
either named, or by their target, architecture, cpu, vendor or
soc. If empty, all boards are selected.
exclude (list of str): List of boards to exclude, regardless of
'args', or None for none
brds (list of Board): List of boards to build, or None/[] for all
Returns:
Tuple
Dictionary which holds the list of boards which were selected
due to each argument, arranged by argument.
List of errors found
"""
def _check_board(brd):
"""Check whether to include or exclude a board
Checks the various terms and decide whether to build it or not (the
'build_it' variable).
If it is built, add the board to the result[term] list so we know
which term caused it to be built. Add it to result['all'] also.
Keep a list of boards we found in 'found', so we can report boards
which appear in self._boards but not in brds.
Args:
brd (Board): Board to check
"""
matching_term = None
build_it = False
if terms:
for term in terms:
if term.matches(brd.props):
matching_term = str(term)
build_it = True
break
elif brds:
if brd.target in brds:
build_it = True
found.append(brd.target)
else:
build_it = True
# Check that it is not specifically excluded
for expr in exclude_list:
if expr.matches(brd.props):
build_it = False
break
if build_it:
brd.build_it = True
if matching_term:
result[matching_term].append(brd.target)
result['all'].append(brd.target)
result = OrderedDict()
warnings = []
terms = self._build_terms(args)
result['all'] = []
for term in terms:
result[str(term)] = []
exclude_list = []
if exclude:
for expr in exclude:
exclude_list.append(Expr(expr))
found = []
for brd in self._boards:
_check_board(brd)
if brds:
remaining = set(brds) - set(found)
if remaining:
warnings.append(f"Boards not found: {', '.join(remaining)}\n")
return result, warnings
@classmethod
def scan_defconfigs_for_multiprocess(cls, queue, defconfigs):
"""Scan defconfig files and queue their board parameters
This function is intended to be passed to multiprocessing.Process()
constructor.
Args:
queue (multiprocessing.Queue): The resulting board parameters are
written into this.
defconfigs (sequence of str): A sequence of defconfig files to be
scanned.
"""
kconf_scanner = KconfigScanner()
for defconfig in defconfigs:
queue.put(kconf_scanner.scan(defconfig))
@classmethod
def read_queues(cls, queues, params_list):
"""Read the queues and append the data to the paramers list"""
for que in queues:
while not que.empty():
params_list.append(que.get())
def scan_defconfigs(self, jobs=1):
"""Collect board parameters for all defconfig files.
This function invokes multiple processes for faster processing.
Args:
jobs (int): The number of jobs to run simultaneously
"""
all_defconfigs = []
for (dirpath, _, filenames) in os.walk(CONFIG_DIR):
for filename in fnmatch.filter(filenames, '*_defconfig'):
if fnmatch.fnmatch(filename, '.*'):
continue
all_defconfigs.append(os.path.join(dirpath, filename))
total_boards = len(all_defconfigs)
processes = []
queues = []
for i in range(jobs):
defconfigs = all_defconfigs[total_boards * i // jobs :
total_boards * (i + 1) // jobs]
que = multiprocessing.Queue(maxsize=-1)
proc = multiprocessing.Process(
target=self.scan_defconfigs_for_multiprocess,
args=(que, defconfigs))
proc.start()
processes.append(proc)
queues.append(que)
# The resulting data should be accumulated to this list
params_list = []
# Data in the queues should be retrieved preriodically.
# Otherwise, the queues would become full and subprocesses would get stuck.
while any(p.is_alive() for p in processes):
self.read_queues(queues, params_list)
# sleep for a while until the queues are filled
time.sleep(SLEEP_TIME)
# Joining subprocesses just in case
# (All subprocesses should already have been finished)
for proc in processes:
proc.join()
# retrieve leftover data
self.read_queues(queues, params_list)
return params_list
@classmethod
def insert_maintainers_info(cls, params_list):
"""Add Status and Maintainers information to the board parameters list.
Args:
params_list (list of dict): A list of the board parameters
Returns:
list of str: List of warnings collected due to missing status, etc.
"""
database = MaintainersDatabase()
for (dirpath, _, filenames) in os.walk('.'):
if 'MAINTAINERS' in filenames:
database.parse_file(os.path.join(dirpath, 'MAINTAINERS'))
for i, params in enumerate(params_list):
target = params['target']
params['status'] = database.get_status(target)
params['maintainers'] = database.get_maintainers(target)
params_list[i] = params
return database.warnings
@classmethod
def format_and_output(cls, params_list, output):
"""Write board parameters into a file.
Columnate the board parameters, sort lines alphabetically,
and then write them to a file.
Args:
params_list (list of dict): The list of board parameters
output (str): The path to the output file
"""
fields = ('status', 'arch', 'cpu', 'soc', 'vendor', 'board', 'target',
'config', 'maintainers')
# First, decide the width of each column
max_length = {f: 0 for f in fields}
for params in params_list:
for field in fields:
max_length[field] = max(max_length[field], len(params[field]))
output_lines = []
for params in params_list:
line = ''
for field in fields:
# insert two spaces between fields like column -t would
line += ' ' + params[field].ljust(max_length[field])
output_lines.append(line.strip())
# ignore case when sorting
output_lines.sort(key=str.lower)
with open(output, 'w', encoding="utf-8") as outf:
outf.write(COMMENT_BLOCK + '\n'.join(output_lines) + '\n')
def ensure_board_list(self, output, jobs=1, force=False, quiet=False):
"""Generate a board database file if needed.
Args:
output (str): The name of the output file
jobs (int): The number of jobs to run simultaneously
force (bool): Force to generate the output even if it is new
quiet (bool): True to avoid printing a message if nothing needs doing
Returns:
bool: True if all is well, False if there were warnings
"""
if not force and output_is_new(output):
if not quiet:
print(f'{output} is up to date. Nothing to do.')
return True
params_list = self.scan_defconfigs(jobs)
warnings = self.insert_maintainers_info(params_list)
for warn in warnings:
print(warn, file=sys.stderr)
self.format_and_output(params_list, output)
return not warnings

86
tools/buildman/builder.py
View File

@ -108,7 +108,7 @@ u-boot/ source directory
boards: List of Board objects which have line in the error/warning output boards: List of Board objects which have line in the error/warning output
errline: The text of the error line errline: The text of the error line
""" """
ErrLine = collections.namedtuple('ErrLine', 'char,boards,errline') ErrLine = collections.namedtuple('ErrLine', 'char,brds,errline')
# Possible build outcomes # Possible build outcomes
OUTCOME_OK, OUTCOME_WARNING, OUTCOME_ERROR, OUTCOME_UNKNOWN = list(range(4)) OUTCOME_OK, OUTCOME_WARNING, OUTCOME_ERROR, OUTCOME_UNKNOWN = list(range(4))
@ -213,6 +213,8 @@ class Builder:
threading is not being used threading is not being used
_terminated: Thread was terminated due to an error _terminated: Thread was terminated due to an error
_restarting_config: True if 'Restart config' is detected in output _restarting_config: True if 'Restart config' is detected in output
_ide: Produce output suitable for an Integrated Development Environment,
i.e. dont emit progress information and put errors/warnings on stderr
""" """
class Outcome: class Outcome:
"""Records a build outcome for a single make invocation """Records a build outcome for a single make invocation
@ -325,6 +327,7 @@ class Builder:
self.config_filenames = BASE_CONFIG_FILENAMES self.config_filenames = BASE_CONFIG_FILENAMES
self.work_in_output = work_in_output self.work_in_output = work_in_output
self.adjust_cfg = adjust_cfg self.adjust_cfg = adjust_cfg
self._ide = False
if not self.squash_config_y: if not self.squash_config_y:
self.config_filenames += EXTRA_CONFIG_FILENAMES self.config_filenames += EXTRA_CONFIG_FILENAMES
@ -382,7 +385,7 @@ class Builder:
show_detail=False, show_bloat=False, show_detail=False, show_bloat=False,
list_error_boards=False, show_config=False, list_error_boards=False, show_config=False,
show_environment=False, filter_dtb_warnings=False, show_environment=False, filter_dtb_warnings=False,
filter_migration_warnings=False): filter_migration_warnings=False, ide=False):
"""Setup display options for the builder. """Setup display options for the builder.
Args: Args:
@ -397,6 +400,8 @@ class Builder:
compiler compiler
filter_migration_warnings: Filter out any warnings about migrating filter_migration_warnings: Filter out any warnings about migrating
a board to driver model a board to driver model
ide: Create output that can be parsed by an IDE. There is no '+' prefix on
error lines and output on stderr stays on stderr.
""" """
self._show_errors = show_errors self._show_errors = show_errors
self._show_sizes = show_sizes self._show_sizes = show_sizes
@ -407,6 +412,7 @@ class Builder:
self._show_environment = show_environment self._show_environment = show_environment
self._filter_dtb_warnings = filter_dtb_warnings self._filter_dtb_warnings = filter_dtb_warnings
self._filter_migration_warnings = filter_migration_warnings self._filter_migration_warnings = filter_migration_warnings
self._ide = ide
def _AddTimestamp(self): def _AddTimestamp(self):
"""Add a new timestamp to the list and record the build period. """Add a new timestamp to the list and record the build period.
@ -535,6 +541,7 @@ class Builder:
line += '%s : ' % self._complete_delay line += '%s : ' % self._complete_delay
line += target line += target
if not self._ide:
terminal.print_clear() terminal.print_clear()
tprint(line, newline=False, limit_to_line=True) tprint(line, newline=False, limit_to_line=True)
@ -662,12 +669,10 @@ class Builder:
""" """
sym = {} sym = {}
for line in fd.readlines(): for line in fd.readlines():
try: line = line.strip()
if line.strip(): parts = line.split()
size, type, name = line[:-1].split() if line and len(parts) == 3:
except: size, type, name = line.split()
tprint("Invalid line in file '%s': '%s'" % (fname, line[:-1]))
continue
if type in 'tTdDbB': if type in 'tTdDbB':
# function names begin with '.' on 64-bit powerpc # function names begin with '.' on 64-bit powerpc
if '.' in name[1:]: if '.' in name[1:]:
@ -834,8 +839,9 @@ class Builder:
Returns: Returns:
Tuple: Tuple:
Dict containing boards which passed building this commit. Dict containing boards which built this commit:
keyed by board.target key: board.target
value: Builder.Outcome object
List containing a summary of error lines List containing a summary of error lines
Dict keyed by error line, containing a list of the Board Dict keyed by error line, containing a list of the Board
objects with that error objects with that error
@ -867,11 +873,11 @@ class Builder:
config = {} config = {}
environment = {} environment = {}
for board in boards_selected.values(): for brd in boards_selected.values():
outcome = self.GetBuildOutcome(commit_upto, board.target, outcome = self.GetBuildOutcome(commit_upto, brd.target,
read_func_sizes, read_config, read_func_sizes, read_config,
read_environment) read_environment)
board_dict[board.target] = outcome board_dict[brd.target] = outcome
last_func = None last_func = None
last_was_warning = False last_was_warning = False
for line in outcome.err_lines: for line in outcome.err_lines:
@ -886,29 +892,29 @@ class Builder:
if is_warning or (last_was_warning and is_note): if is_warning or (last_was_warning and is_note):
if last_func: if last_func:
AddLine(warn_lines_summary, warn_lines_boards, AddLine(warn_lines_summary, warn_lines_boards,
last_func, board) last_func, brd)
AddLine(warn_lines_summary, warn_lines_boards, AddLine(warn_lines_summary, warn_lines_boards,
line, board) line, brd)
else: else:
if last_func: if last_func:
AddLine(err_lines_summary, err_lines_boards, AddLine(err_lines_summary, err_lines_boards,
last_func, board) last_func, brd)
AddLine(err_lines_summary, err_lines_boards, AddLine(err_lines_summary, err_lines_boards,
line, board) line, brd)
last_was_warning = is_warning last_was_warning = is_warning
last_func = None last_func = None
tconfig = Config(self.config_filenames, board.target) tconfig = Config(self.config_filenames, brd.target)
for fname in self.config_filenames: for fname in self.config_filenames:
if outcome.config: if outcome.config:
for key, value in outcome.config[fname].items(): for key, value in outcome.config[fname].items():
tconfig.Add(fname, key, value) tconfig.Add(fname, key, value)
config[board.target] = tconfig config[brd.target] = tconfig
tenvironment = Environment(board.target) tenvironment = Environment(brd.target)
if outcome.environment: if outcome.environment:
for key, value in outcome.environment.items(): for key, value in outcome.environment.items():
tenvironment.Add(key, value) tenvironment.Add(key, value)
environment[board.target] = tenvironment environment[brd.target] = tenvironment
return (board_dict, err_lines_summary, err_lines_boards, return (board_dict, err_lines_summary, err_lines_boards,
warn_lines_summary, warn_lines_boards, config, environment) warn_lines_summary, warn_lines_boards, config, environment)
@ -963,9 +969,8 @@ class Builder:
board.target board.target
""" """
self._base_board_dict = {} self._base_board_dict = {}
for board in board_selected: for brd in board_selected:
self._base_board_dict[board] = Builder.Outcome(0, [], [], {}, {}, self._base_board_dict[brd] = Builder.Outcome(0, [], [], {}, {}, {})
{})
self._base_err_lines = [] self._base_err_lines = []
self._base_warn_lines = [] self._base_warn_lines = []
self._base_err_line_boards = {} self._base_err_line_boards = {}
@ -1209,14 +1214,14 @@ class Builder:
List of boards with that error line, or [] if the user has not List of boards with that error line, or [] if the user has not
requested such a list requested such a list
""" """
boards = [] brds = []
board_set = set() board_set = set()
if self._list_error_boards: if self._list_error_boards:
for board in line_boards[line]: for brd in line_boards[line]:
if not board in board_set: if not brd in board_set:
boards.append(board) brds.append(brd)
board_set.add(board) board_set.add(brd)
return boards return brds
def _CalcErrorDelta(base_lines, base_line_boards, lines, line_boards, def _CalcErrorDelta(base_lines, base_line_boards, lines, line_boards,
char): char):
@ -1319,8 +1324,7 @@ class Builder:
if err_lines: if err_lines:
out_list = [] out_list = []
for line in err_lines: for line in err_lines:
boards = '' names = [brd.target for brd in line.brds]
names = [board.target for board in line.boards]
board_str = ' '.join(names) if names else '' board_str = ' '.join(names) if names else ''
if board_str: if board_str:
out = self.col.build(colour, line.char + '(') out = self.col.build(colour, line.char + '(')
@ -1369,8 +1373,14 @@ class Builder:
better_warn, worse_warn = _CalcErrorDelta(self._base_warn_lines, better_warn, worse_warn = _CalcErrorDelta(self._base_warn_lines,
self._base_warn_line_boards, warn_lines, warn_line_boards, 'w') self._base_warn_line_boards, warn_lines, warn_line_boards, 'w')
# For the IDE mode, print out all the output
if self._ide:
outcome = board_dict[target]
for line in outcome.err_lines:
sys.stderr.write(line)
# Display results by arch # Display results by arch
if any((ok_boards, warn_boards, err_boards, unknown_boards, new_boards, elif any((ok_boards, warn_boards, err_boards, unknown_boards, new_boards,
worse_err, better_err, worse_warn, better_warn)): worse_err, better_err, worse_warn, better_warn)):
arch_list = {} arch_list = {}
self.AddOutcome(board_selected, arch_list, ok_boards, '', self.AddOutcome(board_selected, arch_list, ok_boards, '',
@ -1535,9 +1545,9 @@ class Builder:
# Get a list of boards that did not get built, if needed # Get a list of boards that did not get built, if needed
not_built = [] not_built = []
for board in board_selected: for brd in board_selected:
if not board in board_dict: if not brd in board_dict:
not_built.append(board) not_built.append(brd)
if not_built: if not_built:
tprint("Boards not built (%d): %s" % (len(not_built), tprint("Boards not built (%d): %s" % (len(not_built),
', '.join(not_built))) ', '.join(not_built)))
@ -1746,6 +1756,7 @@ class Builder:
self._PrepareWorkingSpace(min(self.num_threads, len(board_selected)), self._PrepareWorkingSpace(min(self.num_threads, len(board_selected)),
commits is not None) commits is not None)
self._PrepareOutputSpace() self._PrepareOutputSpace()
if not self._ide:
tprint('\rStarting build...', newline=False) tprint('\rStarting build...', newline=False)
self.SetupBuild(board_selected, commits) self.SetupBuild(board_selected, commits)
self.ProcessResult(None) self.ProcessResult(None)
@ -1753,7 +1764,7 @@ class Builder:
# Create jobs to build all commits for each board # Create jobs to build all commits for each board
for brd in board_selected.values(): for brd in board_selected.values():
job = builderthread.BuilderJob() job = builderthread.BuilderJob()
job.board = brd job.brd = brd
job.commits = commits job.commits = commits
job.keep_outputs = keep_outputs job.keep_outputs = keep_outputs
job.work_in_output = self.work_in_output job.work_in_output = self.work_in_output
@ -1773,6 +1784,7 @@ class Builder:
# Wait until we have processed all output # Wait until we have processed all output
self.out_queue.join() self.out_queue.join()
if not self._ide:
tprint() tprint()
msg = 'Completed: %d total built' % self.count msg = 'Completed: %d total built' % self.count

6
tools/buildman/builderthread.py
View File

@ -40,7 +40,7 @@ class BuilderJob:
"""Holds information about a job to be performed by a thread """Holds information about a job to be performed by a thread
Members: Members:
board: Board object to build brd: Board object to build
commits: List of Commit objects to build commits: List of Commit objects to build
keep_outputs: True to save build output files keep_outputs: True to save build output files
step: 1 to process every commit, n to process every nth commit step: 1 to process every commit, n to process every nth commit
@ -48,7 +48,7 @@ class BuilderJob:
don't write to a separate output directory. don't write to a separate output directory.
""" """
def __init__(self): def __init__(self):
self.board = None self.brd = None
self.commits = [] self.commits = []
self.keep_outputs = False self.keep_outputs = False
self.step = 1 self.step = 1
@ -491,7 +491,7 @@ class BuilderThread(threading.Thread):
Returns: Returns:
List of Result objects List of Result objects
""" """
brd = job.board brd = job.brd
work_dir = self.builder.GetThreadDir(self.thread_num) work_dir = self.builder.GetThreadDir(self.thread_num)
self.toolchain = None self.toolchain = None
if job.commits: if job.commits:

7
tools/buildman/cmdline.py
View File

@ -59,9 +59,8 @@ def ParseArgs():
parser.add_option('-i', '--in-tree', dest='in_tree', parser.add_option('-i', '--in-tree', dest='in_tree',
action='store_true', default=False, action='store_true', default=False,
help='Build in the source tree instead of a separate directory') help='Build in the source tree instead of a separate directory')
# -I will be removed after April 2021 parser.add_option('-I', '--ide', action='store_true', default=False,
parser.add_option('-I', '--incremental', action='store_true', help='Create build output that can be parsed by an IDE')
default=False, help='Deprecated, does nothing. See -m')
parser.add_option('-j', '--jobs', dest='jobs', type='int', parser.add_option('-j', '--jobs', dest='jobs', type='int',
default=None, help='Number of jobs to run at once (passed to make)') default=None, help='Number of jobs to run at once (passed to make)')
parser.add_option('-k', '--keep-outputs', action='store_true', parser.add_option('-k', '--keep-outputs', action='store_true',
@ -90,6 +89,8 @@ def ParseArgs():
default=False, help="Use full toolchain path in CROSS_COMPILE") default=False, help="Use full toolchain path in CROSS_COMPILE")
parser.add_option('-P', '--per-board-out-dir', action='store_true', parser.add_option('-P', '--per-board-out-dir', action='store_true',
default=False, help="Use an O= (output) directory per board rather than per thread") default=False, help="Use an O= (output) directory per board rather than per thread")
parser.add_option('-R', '--regen-board-list', action='store_true',
help='Force regeneration of the list of boards, like the old boards.cfg file')
parser.add_option('-s', '--summary', action='store_true', parser.add_option('-s', '--summary', action='store_true',
default=False, help='Show a build summary') default=False, help='Show a build summary')
parser.add_option('-S', '--show-sizes', action='store_true', parser.add_option('-S', '--show-sizes', action='store_true',

48
tools/buildman/control.py
View File

@ -8,7 +8,7 @@ import shutil
import subprocess import subprocess
import sys import sys
from buildman import board from buildman import boards
from buildman import bsettings from buildman import bsettings
from buildman import cfgutil from buildman import cfgutil
from buildman import toolchain from buildman import toolchain
@ -87,7 +87,7 @@ def ShowActions(series, why_selected, boards_selected, builder, options,
for warning in board_warnings: for warning in board_warnings:
print(col.build(col.YELLOW, warning)) print(col.build(col.YELLOW, warning))
def ShowToolchainPrefix(boards, toolchains): def ShowToolchainPrefix(brds, toolchains):
"""Show information about a the tool chain used by one or more boards """Show information about a the tool chain used by one or more boards
The function checks that all boards use the same toolchain, then prints The function checks that all boards use the same toolchain, then prints
@ -100,9 +100,9 @@ def ShowToolchainPrefix(boards, toolchains):
Return: Return:
None on success, string error message otherwise None on success, string error message otherwise
""" """
boards = boards.GetSelectedDict() board_selected = brds.get_selected_dict()
tc_set = set() tc_set = set()
for brd in boards.values(): for brd in board_selected.values():
tc_set.add(toolchains.Select(brd.arch)) tc_set.add(toolchains.Select(brd.arch))
if len(tc_set) != 1: if len(tc_set) != 1:
return 'Supplied boards must share one toolchain' return 'Supplied boards must share one toolchain'
@ -111,7 +111,7 @@ def ShowToolchainPrefix(boards, toolchains):
print(tc.GetEnvArgs(toolchain.VAR_CROSS_COMPILE)) print(tc.GetEnvArgs(toolchain.VAR_CROSS_COMPILE))
return None return None
def DoBuildman(options, args, toolchains=None, make_func=None, boards=None, def DoBuildman(options, args, toolchains=None, make_func=None, brds=None,
clean_dir=False, test_thread_exceptions=False): clean_dir=False, test_thread_exceptions=False):
"""The main control code for buildman """The main control code for buildman
@ -124,7 +124,7 @@ def DoBuildman(options, args, toolchains=None, make_func=None, boards=None,
to execute 'make'. If this is None, the normal function to execute 'make'. If this is None, the normal function
will be used, which calls the 'make' tool with suitable will be used, which calls the 'make' tool with suitable
arguments. This setting is useful for tests. arguments. This setting is useful for tests.
board: Boards() object to use, containing a list of available brds: Boards() object to use, containing a list of available
boards. If this is None it will be created and scanned. boards. If this is None it will be created and scanned.
clean_dir: Used for tests only, indicates that the existing output_dir clean_dir: Used for tests only, indicates that the existing output_dir
should be removed before starting the build should be removed before starting the build
@ -176,32 +176,25 @@ def DoBuildman(options, args, toolchains=None, make_func=None, boards=None,
print() print()
return 0 return 0
if options.incremental:
print(col.build(col.RED,
'Warning: -I has been removed. See documentation'))
if not options.output_dir: if not options.output_dir:
if options.work_in_output: if options.work_in_output:
sys.exit(col.build(col.RED, '-w requires that you specify -o')) sys.exit(col.build(col.RED, '-w requires that you specify -o'))
options.output_dir = '..' options.output_dir = '..'
# Work out what subset of the boards we are building # Work out what subset of the boards we are building
if not boards: if not brds:
if not os.path.exists(options.output_dir): if not os.path.exists(options.output_dir):
os.makedirs(options.output_dir) os.makedirs(options.output_dir)
board_file = os.path.join(options.output_dir, 'boards.cfg') board_file = os.path.join(options.output_dir, 'boards.cfg')
our_path = os.path.dirname(os.path.realpath(__file__))
genboardscfg = os.path.join(our_path, '../genboardscfg.py')
if not os.path.exists(genboardscfg):
genboardscfg = os.path.join(options.git, 'tools/genboardscfg.py')
status = subprocess.call([genboardscfg, '-q', '-o', board_file])
if status != 0:
# Older versions don't support -q
status = subprocess.call([genboardscfg, '-o', board_file])
if status != 0:
sys.exit("Failed to generate boards.cfg")
boards = board.Boards() brds = boards.Boards()
boards.ReadBoards(board_file) ok = brds.ensure_board_list(board_file,
options.threads or multiprocessing.cpu_count(),
force=options.regen_board_list,
quiet=not options.verbose)
if options.regen_board_list:
return 0 if ok else 2
brds.read_boards(board_file)
exclude = [] exclude = []
if options.exclude: if options.exclude:
@ -214,14 +207,14 @@ def DoBuildman(options, args, toolchains=None, make_func=None, boards=None,
requested_boards += b.split(',') requested_boards += b.split(',')
else: else:
requested_boards = None requested_boards = None
why_selected, board_warnings = boards.SelectBoards(args, exclude, why_selected, board_warnings = brds.select_boards(args, exclude,
requested_boards) requested_boards)
selected = boards.GetSelected() selected = brds.get_selected()
if not len(selected): if not len(selected):
sys.exit(col.build(col.RED, 'No matching boards found')) sys.exit(col.build(col.RED, 'No matching boards found'))
if options.print_prefix: if options.print_prefix:
err = ShowToolchainPrefix(boards, toolchains) err = ShowToolchainPrefix(brds, toolchains)
if err: if err:
sys.exit(col.build(col.RED, err)) sys.exit(col.build(col.RED, err))
return 0 return 0
@ -352,7 +345,7 @@ def DoBuildman(options, args, toolchains=None, make_func=None, boards=None,
builder.in_tree = options.in_tree builder.in_tree = options.in_tree
# Work out which boards to build # Work out which boards to build
board_selected = boards.GetSelectedDict() board_selected = brds.get_selected_dict()
if series: if series:
commits = series.commits commits = series.commits
@ -362,6 +355,7 @@ def DoBuildman(options, args, toolchains=None, make_func=None, boards=None,
else: else:
commits = None commits = None
if not options.ide:
tprint(GetActionSummary(options.summary, commits, board_selected, tprint(GetActionSummary(options.summary, commits, board_selected,
options)) options))
@ -372,7 +366,7 @@ def DoBuildman(options, args, toolchains=None, make_func=None, boards=None,
options.show_errors, options.show_sizes, options.show_detail, options.show_errors, options.show_sizes, options.show_detail,
options.show_bloat, options.list_error_boards, options.show_config, options.show_bloat, options.list_error_boards, options.show_config,
options.show_environment, options.filter_dtb_warnings, options.show_environment, options.filter_dtb_warnings,
options.filter_migration_warnings) options.filter_migration_warnings, options.ide)
if options.summary: if options.summary:
builder.ShowSummary(commits, board_selected) builder.ShowSummary(commits, board_selected)
else: else:

59
tools/buildman/func_test.py
View File

@ -9,6 +9,7 @@ import tempfile
import unittest import unittest
from buildman import board from buildman import board
from buildman import boards
from buildman import bsettings from buildman import bsettings
from buildman import cmdline from buildman import cmdline
from buildman import control from buildman import control
@ -35,7 +36,7 @@ chromeos_daisy=VBOOT=${chroot}/build/daisy/usr ${vboot}
chromeos_peach=VBOOT=${chroot}/build/peach_pit/usr ${vboot} chromeos_peach=VBOOT=${chroot}/build/peach_pit/usr ${vboot}
''' '''
boards = [ BOARDS = [
['Active', 'arm', 'armv7', '', 'Tester', 'ARM Board 1', 'board0', ''], ['Active', 'arm', 'armv7', '', 'Tester', 'ARM Board 1', 'board0', ''],
['Active', 'arm', 'armv7', '', 'Tester', 'ARM Board 2', 'board1', ''], ['Active', 'arm', 'armv7', '', 'Tester', 'ARM Board 2', 'board1', ''],
['Active', 'powerpc', 'powerpc', '', 'Tester', 'PowerPC board 1', 'board2', ''], ['Active', 'powerpc', 'powerpc', '', 'Tester', 'PowerPC board 1', 'board2', ''],
@ -187,14 +188,14 @@ class TestFunctional(unittest.TestCase):
self.setupToolchains() self.setupToolchains()
self._toolchains.Add('arm-gcc', test=False) self._toolchains.Add('arm-gcc', test=False)
self._toolchains.Add('powerpc-gcc', test=False) self._toolchains.Add('powerpc-gcc', test=False)
self._boards = board.Boards() self._boards = boards.Boards()
for brd in boards: for brd in BOARDS:
self._boards.AddBoard(board.Board(*brd)) self._boards.add_board(board.Board(*brd))
# Directories where the source been cloned # Directories where the source been cloned
self._clone_dirs = [] self._clone_dirs = []
self._commits = len(commit_shortlog.splitlines()) + 1 self._commits = len(commit_shortlog.splitlines()) + 1
self._total_builds = self._commits * len(boards) self._total_builds = self._commits * len(BOARDS)
# Number of calls to make # Number of calls to make
self._make_calls = 0 self._make_calls = 0
@ -220,13 +221,13 @@ class TestFunctional(unittest.TestCase):
return command.run_pipe([[self._buildman_pathname] + list(args)], return command.run_pipe([[self._buildman_pathname] + list(args)],
capture=True, capture_stderr=True) capture=True, capture_stderr=True)
def _RunControl(self, *args, boards=None, clean_dir=False, def _RunControl(self, *args, brds=None, clean_dir=False,
test_thread_exceptions=False): test_thread_exceptions=False):
"""Run buildman """Run buildman
Args: Args:
args: List of arguments to pass args: List of arguments to pass
boards: brds: Boards object
clean_dir: Used for tests only, indicates that the existing output_dir clean_dir: Used for tests only, indicates that the existing output_dir
should be removed before starting the build should be removed before starting the build
test_thread_exceptions: Uses for tests only, True to make the threads test_thread_exceptions: Uses for tests only, True to make the threads
@ -239,7 +240,7 @@ class TestFunctional(unittest.TestCase):
sys.argv = [sys.argv[0]] + list(args) sys.argv = [sys.argv[0]] + list(args)
options, args = cmdline.ParseArgs() options, args = cmdline.ParseArgs()
result = control.DoBuildman(options, args, toolchains=self._toolchains, result = control.DoBuildman(options, args, toolchains=self._toolchains,
make_func=self._HandleMake, boards=boards or self._boards, make_func=self._HandleMake, brds=brds or self._boards,
clean_dir=clean_dir, clean_dir=clean_dir,
test_thread_exceptions=test_thread_exceptions) test_thread_exceptions=test_thread_exceptions)
self._builder = control.builder self._builder = control.builder
@ -442,7 +443,7 @@ class TestFunctional(unittest.TestCase):
def testNoBoards(self): def testNoBoards(self):
"""Test that buildman aborts when there are no boards""" """Test that buildman aborts when there are no boards"""
self._boards = board.Boards() self._boards = boards.Boards()
with self.assertRaises(SystemExit): with self.assertRaises(SystemExit):
self._RunControl() self._RunControl()
@ -451,7 +452,7 @@ class TestFunctional(unittest.TestCase):
self.setupToolchains(); self.setupToolchains();
self._RunControl('-o', self._output_dir) self._RunControl('-o', self._output_dir)
lines = terminal.get_print_test_lines() lines = terminal.get_print_test_lines()
self.assertIn('Building current source for %d boards' % len(boards), self.assertIn('Building current source for %d boards' % len(BOARDS),
lines[0].text) lines[0].text)
def testBadBranch(self): def testBadBranch(self):
@ -467,7 +468,7 @@ class TestFunctional(unittest.TestCase):
# Buildman always builds the upstream commit as well # Buildman always builds the upstream commit as well
self.assertIn('Building %d commits for %d boards' % self.assertIn('Building %d commits for %d boards' %
(self._commits, len(boards)), lines[0].text) (self._commits, len(BOARDS)), lines[0].text)
self.assertEqual(self._builder.count, self._total_builds) self.assertEqual(self._builder.count, self._total_builds)
# Only sandbox should succeed, the others don't have toolchains # Only sandbox should succeed, the others don't have toolchains
@ -476,12 +477,12 @@ class TestFunctional(unittest.TestCase):
self.assertEqual(ret_code, 100) self.assertEqual(ret_code, 100)
for commit in range(self._commits): for commit in range(self._commits):
for board in self._boards.GetList(): for brd in self._boards.get_list():
if board.arch != 'sandbox': if brd.arch != 'sandbox':
errfile = self._builder.GetErrFile(commit, board.target) errfile = self._builder.GetErrFile(commit, brd.target)
fd = open(errfile) fd = open(errfile)
self.assertEqual(fd.readlines(), self.assertEqual(fd.readlines(),
['No tool chain for %s\n' % board.arch]) ['No tool chain for %s\n' % brd.arch])
fd.close() fd.close()
def testBranch(self): def testBranch(self):
@ -493,17 +494,17 @@ class TestFunctional(unittest.TestCase):
def testCount(self): def testCount(self):
"""Test building a specific number of commitst""" """Test building a specific number of commitst"""
self._RunControl('-b', TEST_BRANCH, '-c2', '-o', self._output_dir) self._RunControl('-b', TEST_BRANCH, '-c2', '-o', self._output_dir)
self.assertEqual(self._builder.count, 2 * len(boards)) self.assertEqual(self._builder.count, 2 * len(BOARDS))
self.assertEqual(self._builder.fail, 0) self.assertEqual(self._builder.fail, 0)
# Each board has a config, and then one make per commit # Each board has a config, and then one make per commit
self.assertEqual(self._make_calls, len(boards) * (1 + 2)) self.assertEqual(self._make_calls, len(BOARDS) * (1 + 2))
def testIncremental(self): def testIncremental(self):
"""Test building a branch twice - the second time should do nothing""" """Test building a branch twice - the second time should do nothing"""
self._RunControl('-b', TEST_BRANCH, '-o', self._output_dir) self._RunControl('-b', TEST_BRANCH, '-o', self._output_dir)
# Each board has a mrproper, config, and then one make per commit # Each board has a mrproper, config, and then one make per commit
self.assertEqual(self._make_calls, len(boards) * (self._commits + 1)) self.assertEqual(self._make_calls, len(BOARDS) * (self._commits + 1))
self._make_calls = 0 self._make_calls = 0
self._RunControl('-b', TEST_BRANCH, '-o', self._output_dir, clean_dir=False) self._RunControl('-b', TEST_BRANCH, '-o', self._output_dir, clean_dir=False)
self.assertEqual(self._make_calls, 0) self.assertEqual(self._make_calls, 0)
@ -516,19 +517,19 @@ class TestFunctional(unittest.TestCase):
self._make_calls = 0 self._make_calls = 0
self._RunControl('-b', TEST_BRANCH, '-f', '-o', self._output_dir, clean_dir=False) self._RunControl('-b', TEST_BRANCH, '-f', '-o', self._output_dir, clean_dir=False)
# Each board has a config and one make per commit # Each board has a config and one make per commit
self.assertEqual(self._make_calls, len(boards) * (self._commits + 1)) self.assertEqual(self._make_calls, len(BOARDS) * (self._commits + 1))
def testForceReconfigure(self): def testForceReconfigure(self):
"""The -f flag should force a rebuild""" """The -f flag should force a rebuild"""
self._RunControl('-b', TEST_BRANCH, '-C', '-o', self._output_dir) self._RunControl('-b', TEST_BRANCH, '-C', '-o', self._output_dir)
# Each commit has a config and make # Each commit has a config and make
self.assertEqual(self._make_calls, len(boards) * self._commits * 2) self.assertEqual(self._make_calls, len(BOARDS) * self._commits * 2)
def testMrproper(self): def testMrproper(self):
"""The -f flag should force a rebuild""" """The -f flag should force a rebuild"""
self._RunControl('-b', TEST_BRANCH, '-m', '-o', self._output_dir) self._RunControl('-b', TEST_BRANCH, '-m', '-o', self._output_dir)
# Each board has a mkproper, config and then one make per commit # Each board has a mkproper, config and then one make per commit
self.assertEqual(self._make_calls, len(boards) * (self._commits + 2)) self.assertEqual(self._make_calls, len(BOARDS) * (self._commits + 2))
def testErrors(self): def testErrors(self):
"""Test handling of build errors""" """Test handling of build errors"""
@ -580,10 +581,10 @@ class TestFunctional(unittest.TestCase):
def testWorkInOutput(self): def testWorkInOutput(self):
"""Test the -w option which should write directly to the output dir""" """Test the -w option which should write directly to the output dir"""
board_list = board.Boards() board_list = boards.Boards()
board_list.AddBoard(board.Board(*boards[0])) board_list.add_board(board.Board(*BOARDS[0]))
self._RunControl('-o', self._output_dir, '-w', clean_dir=False, self._RunControl('-o', self._output_dir, '-w', clean_dir=False,
boards=board_list) brds=board_list)
self.assertTrue( self.assertTrue(
os.path.exists(os.path.join(self._output_dir, 'u-boot'))) os.path.exists(os.path.join(self._output_dir, 'u-boot')))
self.assertTrue( self.assertTrue(
@ -599,15 +600,15 @@ class TestFunctional(unittest.TestCase):
self.assertFalse( self.assertFalse(
os.path.exists(os.path.join(self._output_dir, 'u-boot'))) os.path.exists(os.path.join(self._output_dir, 'u-boot')))
board_list = board.Boards() board_list = boards.Boards()
board_list.AddBoard(board.Board(*boards[0])) board_list.add_board(board.Board(*BOARDS[0]))
with self.assertRaises(SystemExit) as e: with self.assertRaises(SystemExit) as e:
self._RunControl('-b', self._test_branch, '-o', self._output_dir, self._RunControl('-b', self._test_branch, '-o', self._output_dir,
'-w', clean_dir=False, boards=board_list) '-w', clean_dir=False, brds=board_list)
self.assertIn("single commit", str(e.exception)) self.assertIn("single commit", str(e.exception))
board_list = board.Boards() board_list = boards.Boards()
board_list.AddBoard(board.Board(*boards[0])) board_list.add_board(board.Board(*BOARDS[0]))
with self.assertRaises(SystemExit) as e: with self.assertRaises(SystemExit) as e:
self._RunControl('-w', clean_dir=False) self._RunControl('-w', clean_dir=False)
self.assertIn("specify -o", str(e.exception)) self.assertIn("specify -o", str(e.exception))

45
tools/buildman/test.py
View File

@ -10,6 +10,7 @@ import time
import unittest import unittest
from buildman import board from buildman import board
from buildman import boards
from buildman import bsettings from buildman import bsettings
from buildman import builder from buildman import builder
from buildman import cfgutil from buildman import cfgutil
@ -94,7 +95,7 @@ commits = [
[errors[4]]], [errors[4]]],
] ]
boards = [ BOARDS = [
['Active', 'arm', 'armv7', '', 'Tester', 'ARM Board 1', 'board0', ''], ['Active', 'arm', 'armv7', '', 'Tester', 'ARM Board 1', 'board0', ''],
['Active', 'arm', 'armv7', '', 'Tester', 'ARM Board 2', 'board1', ''], ['Active', 'arm', 'armv7', '', 'Tester', 'ARM Board 2', 'board1', ''],
['Active', 'powerpc', 'powerpc', '', 'Tester', 'PowerPC board 1', 'board2', ''], ['Active', 'powerpc', 'powerpc', '', 'Tester', 'PowerPC board 1', 'board2', ''],
@ -131,10 +132,10 @@ class TestBuild(unittest.TestCase):
self.commits.append(comm) self.commits.append(comm)
# Set up boards to build # Set up boards to build
self.boards = board.Boards() self.brds = boards.Boards()
for brd in boards: for brd in BOARDS:
self.boards.AddBoard(board.Board(*brd)) self.brds.add_board(board.Board(*brd))
self.boards.SelectBoards([]) self.brds.select_boards([])
# Add some test settings # Add some test settings
bsettings.Setup(None) bsettings.Setup(None)
@ -176,7 +177,7 @@ class TestBuild(unittest.TestCase):
result.combined = result.stdout + result.stderr result.combined = result.stdout + result.stderr
return result return result
def assertSummary(self, text, arch, plus, boards, outcome=OUTCOME_ERR): def assertSummary(self, text, arch, plus, brds, outcome=OUTCOME_ERR):
col = self._col col = self._col
expected_colour = (col.GREEN if outcome == OUTCOME_OK else expected_colour = (col.GREEN if outcome == OUTCOME_OK else
col.YELLOW if outcome == OUTCOME_WARN else col.RED) col.YELLOW if outcome == OUTCOME_WARN else col.RED)
@ -184,8 +185,8 @@ class TestBuild(unittest.TestCase):
# TODO(sjg@chromium.org): If plus is '', we shouldn't need this # TODO(sjg@chromium.org): If plus is '', we shouldn't need this
expect += ' ' + col.build(expected_colour, plus) expect += ' ' + col.build(expected_colour, plus)
expect += ' ' expect += ' '
for board in boards: for brd in brds:
expect += col.build(expected_colour, ' %s' % board) expect += col.build(expected_colour, ' %s' % brd)
self.assertEqual(text, expect) self.assertEqual(text, expect)
def _SetupTest(self, echo_lines=False, threads=1, **kwdisplay_args): def _SetupTest(self, echo_lines=False, threads=1, **kwdisplay_args):
@ -203,7 +204,7 @@ class TestBuild(unittest.TestCase):
build = builder.Builder(self.toolchains, self.base_dir, None, threads, build = builder.Builder(self.toolchains, self.base_dir, None, threads,
2, checkout=False, show_unknown=False) 2, checkout=False, show_unknown=False)
build.do_make = self.Make build.do_make = self.Make
board_selected = self.boards.GetSelectedDict() board_selected = self.brds.get_selected_dict()
# Build the boards for the pre-defined commits and warnings/errors # Build the boards for the pre-defined commits and warnings/errors
# associated with each. This calls our Make() to inject the fake output. # associated with each. This calls our Make() to inject the fake output.
@ -217,7 +218,7 @@ class TestBuild(unittest.TestCase):
# We should get two starting messages, an update for every commit built # We should get two starting messages, an update for every commit built
# and a summary message # and a summary message
self.assertEqual(count, len(commits) * len(boards) + 3) self.assertEqual(count, len(commits) * len(BOARDS) + 3)
build.SetDisplayOptions(**kwdisplay_args); build.SetDisplayOptions(**kwdisplay_args);
build.ShowSummary(self.commits, board_selected) build.ShowSummary(self.commits, board_selected)
if echo_lines: if echo_lines:
@ -236,7 +237,7 @@ class TestBuild(unittest.TestCase):
filter_dtb_warnings: Adjust the check for output produced with the filter_dtb_warnings: Adjust the check for output produced with the
--filter-dtb-warnings flag --filter-dtb-warnings flag
""" """
def add_line_prefix(prefix, boards, error_str, colour): def add_line_prefix(prefix, brds, error_str, colour):
"""Add a prefix to each line of a string """Add a prefix to each line of a string
The training \n in error_str is removed before processing The training \n in error_str is removed before processing
@ -253,9 +254,9 @@ class TestBuild(unittest.TestCase):
lines = error_str.strip().splitlines() lines = error_str.strip().splitlines()
new_lines = [] new_lines = []
for line in lines: for line in lines:
if boards: if brds:
expect = self._col.build(colour, prefix + '(') expect = self._col.build(colour, prefix + '(')
expect += self._col.build(self._col.MAGENTA, boards, expect += self._col.build(self._col.MAGENTA, brds,
bright=False) bright=False)
expect += self._col.build(colour, ') %s' % line) expect += self._col.build(colour, ') %s' % line)
else: else:
@ -468,18 +469,18 @@ class TestBuild(unittest.TestCase):
def testBoardSingle(self): def testBoardSingle(self):
"""Test single board selection""" """Test single board selection"""
self.assertEqual(self.boards.SelectBoards(['sandbox']), self.assertEqual(self.brds.select_boards(['sandbox']),
({'all': ['board4'], 'sandbox': ['board4']}, [])) ({'all': ['board4'], 'sandbox': ['board4']}, []))
def testBoardArch(self): def testBoardArch(self):
"""Test single board selection""" """Test single board selection"""
self.assertEqual(self.boards.SelectBoards(['arm']), self.assertEqual(self.brds.select_boards(['arm']),
({'all': ['board0', 'board1'], ({'all': ['board0', 'board1'],
'arm': ['board0', 'board1']}, [])) 'arm': ['board0', 'board1']}, []))
def testBoardArchSingle(self): def testBoardArchSingle(self):
"""Test single board selection""" """Test single board selection"""
self.assertEqual(self.boards.SelectBoards(['arm sandbox']), self.assertEqual(self.brds.select_boards(['arm sandbox']),
({'sandbox': ['board4'], ({'sandbox': ['board4'],
'all': ['board0', 'board1', 'board4'], 'all': ['board0', 'board1', 'board4'],
'arm': ['board0', 'board1']}, [])) 'arm': ['board0', 'board1']}, []))
@ -487,20 +488,20 @@ class TestBuild(unittest.TestCase):
def testBoardArchSingleMultiWord(self): def testBoardArchSingleMultiWord(self):
"""Test single board selection""" """Test single board selection"""
self.assertEqual(self.boards.SelectBoards(['arm', 'sandbox']), self.assertEqual(self.brds.select_boards(['arm', 'sandbox']),
({'sandbox': ['board4'], ({'sandbox': ['board4'],
'all': ['board0', 'board1', 'board4'], 'all': ['board0', 'board1', 'board4'],
'arm': ['board0', 'board1']}, [])) 'arm': ['board0', 'board1']}, []))
def testBoardSingleAnd(self): def testBoardSingleAnd(self):
"""Test single board selection""" """Test single board selection"""
self.assertEqual(self.boards.SelectBoards(['Tester & arm']), self.assertEqual(self.brds.select_boards(['Tester & arm']),
({'Tester&arm': ['board0', 'board1'], ({'Tester&arm': ['board0', 'board1'],
'all': ['board0', 'board1']}, [])) 'all': ['board0', 'board1']}, []))
def testBoardTwoAnd(self): def testBoardTwoAnd(self):
"""Test single board selection""" """Test single board selection"""
self.assertEqual(self.boards.SelectBoards(['Tester', '&', 'arm', self.assertEqual(self.brds.select_boards(['Tester', '&', 'arm',
'Tester' '&', 'powerpc', 'Tester' '&', 'powerpc',
'sandbox']), 'sandbox']),
({'sandbox': ['board4'], ({'sandbox': ['board4'],
@ -511,19 +512,19 @@ class TestBuild(unittest.TestCase):
def testBoardAll(self): def testBoardAll(self):
"""Test single board selection""" """Test single board selection"""
self.assertEqual(self.boards.SelectBoards([]), self.assertEqual(self.brds.select_boards([]),
({'all': ['board0', 'board1', 'board2', 'board3', ({'all': ['board0', 'board1', 'board2', 'board3',
'board4']}, [])) 'board4']}, []))
def testBoardRegularExpression(self): def testBoardRegularExpression(self):
"""Test single board selection""" """Test single board selection"""
self.assertEqual(self.boards.SelectBoards(['T.*r&^Po']), self.assertEqual(self.brds.select_boards(['T.*r&^Po']),
({'all': ['board2', 'board3'], ({'all': ['board2', 'board3'],
'T.*r&^Po': ['board2', 'board3']}, [])) 'T.*r&^Po': ['board2', 'board3']}, []))
def testBoardDuplicate(self): def testBoardDuplicate(self):
"""Test single board selection""" """Test single board selection"""
self.assertEqual(self.boards.SelectBoards(['sandbox sandbox', self.assertEqual(self.brds.select_boards(['sandbox sandbox',
'sandbox']), 'sandbox']),
({'all': ['board4'], 'sandbox': ['board4']}, [])) ({'all': ['board4'], 'sandbox': ['board4']}, []))
def CheckDirs(self, build, dirname): def CheckDirs(self, build, dirname):

8
tools/buildman/toolchain.py
View File

@ -441,7 +441,7 @@ class Toolchains:
args = args[:m.start(0)] + value + args[m.end(0):] args = args[:m.start(0)] + value + args[m.end(0):]
return args return args
def GetMakeArguments(self, board): def GetMakeArguments(self, brd):
"""Returns 'make' arguments for a given board """Returns 'make' arguments for a given board
The flags are in a section called 'make-flags'. Flags are named The flags are in a section called 'make-flags'. Flags are named
@ -462,13 +462,13 @@ class Toolchains:
A special 'target' variable is set to the board target. A special 'target' variable is set to the board target.
Args: Args:
board: Board object for the board to check. brd: Board object for the board to check.
Returns: Returns:
'make' flags for that board, or '' if none 'make' flags for that board, or '' if none
""" """
self._make_flags['target'] = board.target self._make_flags['target'] = brd.target
arg_str = self.ResolveReferences(self._make_flags, arg_str = self.ResolveReferences(self._make_flags,
self._make_flags.get(board.target, '')) self._make_flags.get(brd.target, ''))
args = re.findall("(?:\".*?\"|\S)+", arg_str) args = re.findall("(?:\".*?\"|\S)+", arg_str)
i = 0 i = 0
while i < len(args): while i < len(args):

444
tools/genboardscfg.py
View File

@ -1,444 +0,0 @@
#!/usr/bin/env python3
# SPDX-License-Identifier: GPL-2.0+
#
# Author: Masahiro Yamada <yamada.m@jp.panasonic.com>
#
"""
Converter from Kconfig and MAINTAINERS to a board database.
Run 'tools/genboardscfg.py' to create a board database.
Run 'tools/genboardscfg.py -h' for available options.
"""
import errno
import fnmatch
import glob
import multiprocessing
import optparse
import os
import sys
import tempfile
import time
from buildman import kconfiglib
### constant variables ###
OUTPUT_FILE = 'boards.cfg'
CONFIG_DIR = 'configs'
SLEEP_TIME = 0.03
COMMENT_BLOCK = '''#
# List of boards
# Automatically generated by %s: don't edit
#
# Status, Arch, CPU, SoC, Vendor, Board, Target, Options, Maintainers
''' % __file__
### helper functions ###
def try_remove(f):
"""Remove a file ignoring 'No such file or directory' error."""
try:
os.remove(f)
except OSError as exception:
# Ignore 'No such file or directory' error
if exception.errno != errno.ENOENT:
raise
def check_top_directory():
"""Exit if we are not at the top of source directory."""
for f in ('README', 'Licenses'):
if not os.path.exists(f):
sys.exit('Please run at the top of source directory.')
def output_is_new(output):
"""Check if the output file is up to date.
Returns:
True if the given output file exists and is newer than any of
*_defconfig, MAINTAINERS and Kconfig*. False otherwise.
"""
try:
ctime = os.path.getctime(output)
except OSError as exception:
if exception.errno == errno.ENOENT:
# return False on 'No such file or directory' error
return False
else:
raise
for (dirpath, dirnames, filenames) in os.walk(CONFIG_DIR):
for filename in fnmatch.filter(filenames, '*_defconfig'):
if fnmatch.fnmatch(filename, '.*'):
continue
filepath = os.path.join(dirpath, filename)
if ctime < os.path.getctime(filepath):
return False
for (dirpath, dirnames, filenames) in os.walk('.'):
for filename in filenames:
if (fnmatch.fnmatch(filename, '*~') or
not fnmatch.fnmatch(filename, 'Kconfig*') and
not filename == 'MAINTAINERS'):
continue
filepath = os.path.join(dirpath, filename)
if ctime < os.path.getctime(filepath):
return False
# Detect a board that has been removed since the current board database
# was generated
with open(output, encoding="utf-8") as f:
for line in f:
if line[0] == '#' or line == '\n':
continue
defconfig = line.split()[6] + '_defconfig'
if not os.path.exists(os.path.join(CONFIG_DIR, defconfig)):
return False
return True
### classes ###
class KconfigScanner:
"""Kconfig scanner."""
### constant variable only used in this class ###
_SYMBOL_TABLE = {
'arch' : 'SYS_ARCH',
'cpu' : 'SYS_CPU',
'soc' : 'SYS_SOC',
'vendor' : 'SYS_VENDOR',
'board' : 'SYS_BOARD',
'config' : 'SYS_CONFIG_NAME',
'options' : 'SYS_EXTRA_OPTIONS'
}
def __init__(self):
"""Scan all the Kconfig files and create a Kconfig object."""
# Define environment variables referenced from Kconfig
os.environ['srctree'] = os.getcwd()
os.environ['UBOOTVERSION'] = 'dummy'
os.environ['KCONFIG_OBJDIR'] = ''
self._conf = kconfiglib.Kconfig(warn=False)
def __del__(self):
"""Delete a leftover temporary file before exit.
The scan() method of this class creates a temporay file and deletes
it on success. If scan() method throws an exception on the way,
the temporary file might be left over. In that case, it should be
deleted in this destructor.
"""
if hasattr(self, '_tmpfile') and self._tmpfile:
try_remove(self._tmpfile)
def scan(self, defconfig):
"""Load a defconfig file to obtain board parameters.
Arguments:
defconfig: path to the defconfig file to be processed
Returns:
A dictionary of board parameters. It has a form of:
{
'arch': <arch_name>,
'cpu': <cpu_name>,
'soc': <soc_name>,
'vendor': <vendor_name>,
'board': <board_name>,
'target': <target_name>,
'config': <config_header_name>,
'options': <extra_options>
}
"""
# strip special prefixes and save it in a temporary file
fd, self._tmpfile = tempfile.mkstemp()
with os.fdopen(fd, 'w') as f:
for line in open(defconfig):
colon = line.find(':CONFIG_')
if colon == -1:
f.write(line)
else:
f.write(line[colon + 1:])
self._conf.load_config(self._tmpfile)
try_remove(self._tmpfile)
self._tmpfile = None
params = {}
# Get the value of CONFIG_SYS_ARCH, CONFIG_SYS_CPU, ... etc.
# Set '-' if the value is empty.
for key, symbol in list(self._SYMBOL_TABLE.items()):
value = self._conf.syms.get(symbol).str_value
if value:
params[key] = value
else:
params[key] = '-'
defconfig = os.path.basename(defconfig)
params['target'], match, rear = defconfig.partition('_defconfig')
assert match and not rear, '%s : invalid defconfig' % defconfig
# fix-up for aarch64
if params['arch'] == 'arm' and params['cpu'] == 'armv8':
params['arch'] = 'aarch64'
# fix-up options field. It should have the form:
# <config name>[:comma separated config options]
if params['options'] != '-':
params['options'] = params['config'] + ':' + \
params['options'].replace(r'\"', '"')
elif params['config'] != params['target']:
params['options'] = params['config']
return params
def scan_defconfigs_for_multiprocess(queue, defconfigs):
"""Scan defconfig files and queue their board parameters
This function is intended to be passed to
multiprocessing.Process() constructor.
Arguments:
queue: An instance of multiprocessing.Queue().
The resulting board parameters are written into it.
defconfigs: A sequence of defconfig files to be scanned.
"""
kconf_scanner = KconfigScanner()
for defconfig in defconfigs:
queue.put(kconf_scanner.scan(defconfig))
def read_queues(queues, params_list):
"""Read the queues and append the data to the paramers list"""
for q in queues:
while not q.empty():
params_list.append(q.get())
def scan_defconfigs(jobs=1):
"""Collect board parameters for all defconfig files.
This function invokes multiple processes for faster processing.
Arguments:
jobs: The number of jobs to run simultaneously
"""
all_defconfigs = []
for (dirpath, dirnames, filenames) in os.walk(CONFIG_DIR):
for filename in fnmatch.filter(filenames, '*_defconfig'):
if fnmatch.fnmatch(filename, '.*'):
continue
all_defconfigs.append(os.path.join(dirpath, filename))
total_boards = len(all_defconfigs)
processes = []
queues = []
for i in range(jobs):
defconfigs = all_defconfigs[total_boards * i // jobs :
total_boards * (i + 1) // jobs]
q = multiprocessing.Queue(maxsize=-1)
p = multiprocessing.Process(target=scan_defconfigs_for_multiprocess,
args=(q, defconfigs))
p.start()
processes.append(p)
queues.append(q)
# The resulting data should be accumulated to this list
params_list = []
# Data in the queues should be retrieved preriodically.
# Otherwise, the queues would become full and subprocesses would get stuck.
while any([p.is_alive() for p in processes]):
read_queues(queues, params_list)
# sleep for a while until the queues are filled
time.sleep(SLEEP_TIME)
# Joining subprocesses just in case
# (All subprocesses should already have been finished)
for p in processes:
p.join()
# retrieve leftover data
read_queues(queues, params_list)
return params_list
class MaintainersDatabase:
"""The database of board status and maintainers."""
def __init__(self):
"""Create an empty database."""
self.database = {}
def get_status(self, target):
"""Return the status of the given board.
The board status is generally either 'Active' or 'Orphan'.
Display a warning message and return '-' if status information
is not found.
Returns:
'Active', 'Orphan' or '-'.
"""
if not target in self.database:
print("WARNING: no status info for '%s'" % target, file=sys.stderr)
return '-'
tmp = self.database[target][0]
if tmp.startswith('Maintained'):
return 'Active'
elif tmp.startswith('Supported'):
return 'Active'
elif tmp.startswith('Orphan'):
return 'Orphan'
else:
print(("WARNING: %s: unknown status for '%s'" %
(tmp, target)), file=sys.stderr)
return '-'
def get_maintainers(self, target):
"""Return the maintainers of the given board.
Returns:
Maintainers of the board. If the board has two or more maintainers,
they are separated with colons.
"""
if not target in self.database:
print("WARNING: no maintainers for '%s'" % target, file=sys.stderr)
return ''
return ':'.join(self.database[target][1])
def parse_file(self, file):
"""Parse a MAINTAINERS file.
Parse a MAINTAINERS file and accumulates board status and
maintainers information.
Arguments:
file: MAINTAINERS file to be parsed
"""
targets = []
maintainers = []
status = '-'
for line in open(file, encoding="utf-8"):
# Check also commented maintainers
if line[:3] == '#M:':
line = line[1:]
tag, rest = line[:2], line[2:].strip()
if tag == 'M:':
maintainers.append(rest)
elif tag == 'F:':
# expand wildcard and filter by 'configs/*_defconfig'
for f in glob.glob(rest):
front, match, rear = f.partition('configs/')
if not front and match:
front, match, rear = rear.rpartition('_defconfig')
if match and not rear:
targets.append(front)
elif tag == 'S:':
status = rest
elif line == '\n':
for target in targets:
self.database[target] = (status, maintainers)
targets = []
maintainers = []
status = '-'
if targets:
for target in targets:
self.database[target] = (status, maintainers)
def insert_maintainers_info(params_list):
"""Add Status and Maintainers information to the board parameters list.
Arguments:
params_list: A list of the board parameters
"""
database = MaintainersDatabase()
for (dirpath, dirnames, filenames) in os.walk('.'):
if 'MAINTAINERS' in filenames:
database.parse_file(os.path.join(dirpath, 'MAINTAINERS'))
for i, params in enumerate(params_list):
target = params['target']
params['status'] = database.get_status(target)
params['maintainers'] = database.get_maintainers(target)
params_list[i] = params
def format_and_output(params_list, output):
"""Write board parameters into a file.
Columnate the board parameters, sort lines alphabetically,
and then write them to a file.
Arguments:
params_list: The list of board parameters
output: The path to the output file
"""
FIELDS = ('status', 'arch', 'cpu', 'soc', 'vendor', 'board', 'target',
'options', 'maintainers')
# First, decide the width of each column
max_length = dict([ (f, 0) for f in FIELDS])
for params in params_list:
for f in FIELDS:
max_length[f] = max(max_length[f], len(params[f]))
output_lines = []
for params in params_list:
line = ''
for f in FIELDS:
# insert two spaces between fields like column -t would
line += ' ' + params[f].ljust(max_length[f])
output_lines.append(line.strip())
# ignore case when sorting
output_lines.sort(key=str.lower)
with open(output, 'w', encoding="utf-8") as f:
f.write(COMMENT_BLOCK + '\n'.join(output_lines) + '\n')
def gen_boards_cfg(output, jobs=1, force=False, quiet=False):
"""Generate a board database file.
Arguments:
output: The name of the output file
jobs: The number of jobs to run simultaneously
force: Force to generate the output even if it is new
quiet: True to avoid printing a message if nothing needs doing
"""
check_top_directory()
if not force and output_is_new(output):
if not quiet:
print("%s is up to date. Nothing to do." % output)
sys.exit(0)
params_list = scan_defconfigs(jobs)
insert_maintainers_info(params_list)
format_and_output(params_list, output)
def main():
try:
cpu_count = multiprocessing.cpu_count()
except NotImplementedError:
cpu_count = 1
parser = optparse.OptionParser()
# Add options here
parser.add_option('-f', '--force', action="store_true", default=False,
help='regenerate the output even if it is new')
parser.add_option('-j', '--jobs', type='int', default=min(cpu_count, 240),
help='the number of jobs to run simultaneously')
parser.add_option('-o', '--output', default=OUTPUT_FILE,
help='output file [default=%s]' % OUTPUT_FILE)
parser.add_option('-q', '--quiet', action="store_true", help='run silently')
(options, args) = parser.parse_args()
gen_boards_cfg(options.output, jobs=options.jobs, force=options.force,
quiet=options.quiet)
if __name__ == '__main__':
main()

65
tools/moveconfig.py
View File

@ -443,70 +443,6 @@ def cleanup_headers(configs, args):
cleanup_one_header(header_path, patterns, args) cleanup_one_header(header_path, patterns, args)
cleanup_empty_blocks(header_path, args) cleanup_empty_blocks(header_path, args)
def cleanup_one_extra_option(defconfig_path, configs, args):
"""Delete config defines in CONFIG_SYS_EXTRA_OPTIONS in one defconfig file.
Args:
defconfig_path: path to the cleaned defconfig file.
configs: A list of CONFIGs to remove.
args (Namespace): program arguments
"""
start = 'CONFIG_SYS_EXTRA_OPTIONS="'
end = '"'
lines = read_file(defconfig_path)
for i, line in enumerate(lines):
if line.startswith(start) and line.endswith(end):
break
else:
# CONFIG_SYS_EXTRA_OPTIONS was not found in this defconfig
return
old_tokens = line[len(start):-len(end)].split(',')
new_tokens = []
for token in old_tokens:
pos = token.find('=')
if not (token[:pos] if pos >= 0 else token) in configs:
new_tokens.append(token)
if new_tokens == old_tokens:
return
tolines = copy.copy(lines)
if new_tokens:
tolines[i] = start + ','.join(new_tokens) + end
else:
tolines.pop(i)
show_diff(lines, tolines, defconfig_path, args.color)
if args.dry_run:
return
write_file(defconfig_path, tolines)
def cleanup_extra_options(configs, args):
"""Delete config defines in CONFIG_SYS_EXTRA_OPTIONS in defconfig files.
Args:
configs: A list of CONFIGs to remove.
args (Namespace): program arguments
"""
if not confirm(args, 'Clean up CONFIG_SYS_EXTRA_OPTIONS?'):
return
configs = [ config[len('CONFIG_'):] for config in configs ]
defconfigs = get_all_defconfigs()
for defconfig in defconfigs:
cleanup_one_extra_option(os.path.join('configs', defconfig), configs,
args)
def cleanup_whitelist(configs, args): def cleanup_whitelist(configs, args):
"""Delete config whitelist entries """Delete config whitelist entries
@ -1803,7 +1739,6 @@ doc/develop/moveconfig.rst for documentation.'''
if configs: if configs:
cleanup_headers(configs, args) cleanup_headers(configs, args)
cleanup_extra_options(configs, args)
cleanup_whitelist(configs, args) cleanup_whitelist(configs, args)
cleanup_readme(configs, args) cleanup_readme(configs, args)