doc: pytest: Fix inline code and other formatting

Mainly fix inline code and some other formatting mistakes. Inline code
uses double backticks `` in reStructuredText instead of single backticks
as in Markdown.

Also fix some smaller formatting issues, such as excess colons before
literal blocks.

Signed-off-by: Martin Schwan <m.schwan@phytec.de>
This commit is contained in:
Martin Schwan 2025-10-01 09:44:37 +02:00 committed by Heinrich Schuchardt
parent 5a21f5b4c9
commit a8d451aa14

View File

@ -27,8 +27,8 @@ the appropriate time.
In order to run the test suite at a minimum we require that both Python 3 and In order to run the test suite at a minimum we require that both Python 3 and
pip for Python 3 are installed. All of the required python modules are pip for Python 3 are installed. All of the required python modules are
described in the requirements.txt file in the /test/py/ directory and can be described in the ``requirements.txt`` file in the ``/test/py/`` directory and
installed via the command can be installed via the command
.. code-block:: bash .. code-block:: bash
@ -62,18 +62,18 @@ The test script supports either:
physical board, attach to the board's console stream, and reset the board. physical board, attach to the board's console stream, and reset the board.
Further details are described later. Further details are described later.
The usage of the command 'sudo' is not allowed in tests. Using elevated The usage of the command ``sudo`` is not allowed in tests. Using elevated
priviledges can lead to security concerns. Furthermore not all users may have priviledges can lead to security concerns. Furthermore not all users may have
administrator rights. Therefore the command 'sudo' must not be used in tests. administrator rights. Therefore the command ``sudo`` must not be used in tests.
To create disk images we have helper functions located in To create disk images we have helper functions located in
`test/py/tests/fs_helper.py` which shall be used in any tests that require ``test/py/tests/fs_helper.py`` which shall be used in any tests that require
creating disk images. creating disk images.
Using a Python sandbox to provide requirements Using a Python sandbox to provide requirements
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The recommended way to run the test suite, in order to ensure reproducibility The recommended way to run the test suite, in order to ensure reproducibility
is to use a Python sandbox such as `python -m venv` to set up the necessary is to use a Python sandbox such as ``python -m venv`` to set up the necessary
environment. This can be done via the following commands: environment. This can be done via the following commands:
@ -95,19 +95,19 @@ application), simply execute:
./test/py/test.py --bd sandbox --build ./test/py/test.py --bd sandbox --build
The `--bd` option tells the test suite which board type is being tested. This The ``--bd`` option tells the test suite which board type is being tested. This
lets the test suite know which features the board has, and hence exactly what lets the test suite know which features the board has, and hence exactly what
can be tested. can be tested.
The `--build` option tells U-Boot to compile U-Boot. Alternatively, you may The ``--build`` option tells U-Boot to compile U-Boot. Alternatively, you may
omit this option and build U-Boot yourself, in whatever way you choose, before omit this option and build U-Boot yourself, in whatever way you choose, before
running the test script. running the test script.
The test script will attach to U-Boot, execute all valid tests for the board, The test script will attach to U-Boot, execute all valid tests for the board,
then print a summary of the test process. A complete log of the test session then print a summary of the test process. A complete log of the test session
will be written to `${build_dir}/test-log.html`. This is best viewed in a web will be written to ``${build_dir}/test-log.html``. This is best viewed in a web
browser, but may be read directly as plain text, perhaps with the aid of the browser, but may be read directly as plain text, perhaps with the aid of the
`html2text` utility. ``html2text`` utility.
If sandbox crashes (e.g. with a segfault) you will see message like this:: If sandbox crashes (e.g. with a segfault) you will see message like this::
@ -134,23 +134,23 @@ First install support for parallel tests::
sudo apt install python3-pytest-xdist sudo apt install python3-pytest-xdist
or::: or::
pip3 install pytest-xdist pip3 install pytest-xdist
Then run the tests in parallel using the -n flag:: Then run the tests in parallel using the ``-n`` flag::
test/py/test.py -B sandbox --build --build-dir /tmp/b/sandbox -q -k \ test/py/test.py -B sandbox --build --build-dir /tmp/b/sandbox -q -k \
'not slow and not bootstd and not spi_flash' -n16 'not slow and not bootstd and not spi_flash' -n16
You can also use `make pcheck` to run all tests in parallel. This uses a maximum You can also use ``make pcheck`` to run all tests in parallel. This uses a
of 16 threads, since the setup time is significant and there are under 1000 maximum of 16 threads, since the setup time is significant and there are under
tests. 1000 tests.
Note that the `test-log.html` output does not work correctly at present with Note that the ``test-log.html`` output does not work correctly at present with
parallel testing. All the threads write to it at once, so it is garbled. parallel testing. All the threads write to it at once, so it is garbled.
Note that the `tools/` tests still run each tool's tests once after the other, Note that the ``tools/`` tests still run each tool's tests once after the other,
although within that, they do run in parallel. So for example, the buildman although within that, they do run in parallel. So for example, the buildman
tests run in parallel, then the binman tests run in parallel. There would be a tests run in parallel, then the binman tests run in parallel. There would be a
significant advantage to running them all in parallel together, but that would significant advantage to running them all in parallel together, but that would
@ -163,12 +163,12 @@ Testing under a debugger
~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~
If you need to run sandbox under a debugger, you may pass the command-line If you need to run sandbox under a debugger, you may pass the command-line
option `--gdbserver COMM`. This causes two things to happens: option ``--gdbserver COMM``. This causes two things to happens:
- Instead of running U-Boot directly, it will be run under gdbserver, with - Instead of running U-Boot directly, it will be run under gdbserver, with
debug communication via the channel `COMM`. You can attach a debugger to the debug communication via the channel ``COMM``. You can attach a debugger to the
sandbox process in order to debug it. See `man gdbserver` and the example sandbox process in order to debug it. See ``man gdbserver`` and the example
below for details of valid values for `COMM`. below for details of valid values for ``COMM``.
- All timeouts in tests are disabled, allowing U-Boot an arbitrary amount of - All timeouts in tests are disabled, allowing U-Boot an arbitrary amount of
time to execute commands. This is useful if U-Boot is stopped at a breakpoint time to execute commands. This is useful if U-Boot is stopped at a breakpoint
during debugging. during debugging.
@ -187,7 +187,7 @@ Window 2:
gdb ./build-sandbox/u-boot -ex 'target remote localhost:1234' gdb ./build-sandbox/u-boot -ex 'target remote localhost:1234'
Alternatively, you could leave off the `-ex` option and type the command Alternatively, you could leave off the ``-ex`` option and type the command
manually into gdb once it starts. manually into gdb once it starts.
You can use any debugger you wish, as long as it speaks the gdb remote You can use any debugger you wish, as long as it speaks the gdb remote
@ -196,16 +196,17 @@ protocol, or any graphical wrapper around gdb.
Some tests deliberately cause the sandbox process to exit, e.g. to test the Some tests deliberately cause the sandbox process to exit, e.g. to test the
reset command, or sandbox's CTRL-C handling. When this happens, you will need reset command, or sandbox's CTRL-C handling. When this happens, you will need
to attach the debugger to the new sandbox instance. If these tests are not to attach the debugger to the new sandbox instance. If these tests are not
relevant to your debugging session, you can skip them using pytest's -k relevant to your debugging session, you can skip them using pytest's ``-k``
command-line option; see the next section. command-line option; see the next section.
Command-line options Command-line options
-------------------- --------------------
--board-type, --bd, -B --board-type, --bd, -B
set the type of the board to be tested. For example, `sandbox` or `seaboard`. set the type of the board to be tested. For example, ``sandbox`` or
``seaboard``.
--board-identity`, --id --board-identity, --id
sets the identity of the board to be tested. This allows differentiation sets the identity of the board to be tested. This allows differentiation
between multiple instances of the same type of physical board that are between multiple instances of the same type of physical board that are
attached to the same host machine. This parameter is not interpreted by th attached to the same host machine. This parameter is not interpreted by th
@ -215,15 +216,15 @@ Command-line options
--build --build
indicates that the test script should compile U-Boot itself before running indicates that the test script should compile U-Boot itself before running
the tests. If using this option, make sure that any environment variables the tests. If using this option, make sure that any environment variables
required by the build process are already set, such as `$CROSS_COMPILE`. required by the build process are already set, such as ``$CROSS_COMPILE``.
--buildman --buildman
indicates that `--build` should use buildman to build U-Boot. There is no need indicates that ``--build`` should use buildman to build U-Boot. There is no
to set $CROSS_COMPILE` in this case since buildman handles it. need to set ``$CROSS_COMPILE`` in this case since buildman handles it.
--build-dir --build-dir
sets the directory containing the compiled U-Boot binaries. If omitted, this sets the directory containing the compiled U-Boot binaries. If omitted, this
is `${source_dir}/build-${board_type}`. is ``${source_dir}/build-${board_type}``.
--result-dir --result-dir
sets the directory to write results, such as log files, into. sets the directory to write results, such as log files, into.
@ -266,10 +267,10 @@ Command-line options
<3.0s : 2.4s | 1 <3.0s : 2.4s | 1
<7.5s : 6.1s | 1 <7.5s : 6.1s | 1
`pytest` also implements a number of its own command-line options. Commonly used pytest also implements a number of its own command-line options. Commonly used
options are mentioned below. Please see `pytest` documentation for complete options are mentioned below. Please see pytest documentation for complete
details. Execute `py.test --version` for a brief summary. Note that U-Boot's details. Execute ``py.test --version`` for a brief summary. Note that U-Boot's
test.py script passes all command-line arguments directly to `pytest` for test.py script passes all command-line arguments directly to pytest for
processing. processing.
-k -k
@ -277,13 +278,13 @@ processing.
option takes a single argument which is used to filter test names. Simple option takes a single argument which is used to filter test names. Simple
logical operators are supported. For example: logical operators are supported. For example:
- `'-k ums'` runs only tests with "ums" in their name. - ``-k ums`` runs only tests with "ums" in their name.
- `'-k ut_dm'` runs only tests with "ut_dm" in their name. Note that in this - ``-k ut_dm`` runs only tests with "ut_dm" in their name. Note that in this
case, "ut_dm" is a parameter to a test rather than the test name. The full case, "ut_dm" is a parameter to a test rather than the test name. The full
test name is e.g. "test_ut[ut_dm_leak]". test name is e.g. "test_ut[ut_dm_leak]".
- `'-k not reset'` runs everything except tests with "reset" in their name. - ``-k not reset`` runs everything except tests with "reset" in their name.
- `'-k ut or hush'` runs only tests with "ut" or "hush" in their name. - ``-k ut or hush`` runs only tests with "ut" or "hush" in their name.
- `'-k not (ut or hush)'` runs everything except tests with "ut" or "hush" in - ``-k not (ut or hush)`` runs everything except tests with "ut" or "hush" in
their name. their name.
-s -s
@ -297,7 +298,7 @@ The tools and techniques used to interact with real hardware will vary
radically between different host and target systems, and the whims of the user. radically between different host and target systems, and the whims of the user.
For this reason, the test suite does not attempt to directly interact with real For this reason, the test suite does not attempt to directly interact with real
hardware in any way. Rather, it executes a standardized set of "hook" scripts hardware in any way. Rather, it executes a standardized set of "hook" scripts
via `$PATH`. These scripts implement certain actions on behalf of the test via ``$PATH``. These scripts implement certain actions on behalf of the test
suite. This keeps the test suite simple and isolated from system variances suite. This keeps the test suite simple and isolated from system variances
unrelated to U-Boot features. unrelated to U-Boot features.
@ -309,14 +310,14 @@ Environment variables
The following environment variables are set when running hook scripts: The following environment variables are set when running hook scripts:
- `UBOOT_BOARD_TYPE` the board type being tested. - ``UBOOT_BOARD_TYPE`` the board type being tested.
- `UBOOT_BOARD_IDENTITY` the board identity being tested, or `na` if none was - ``UBOOT_BOARD_IDENTITY`` the board identity being tested, or ``na`` if none
specified. was specified.
- `UBOOT_SOURCE_DIR` the U-Boot source directory. - ``UBOOT_SOURCE_DIR`` the U-Boot source directory.
- `UBOOT_TEST_PY_DIR` the full path to `test/py/` in the source directory. - ``UBOOT_TEST_PY_DIR`` the full path to ``test/py/`` in the source directory.
- `UBOOT_BUILD_DIR` the U-Boot build directory. - ``UBOOT_BUILD_DIR`` the U-Boot build directory.
- `UBOOT_RESULT_DIR` the test result directory. - ``UBOOT_RESULT_DIR`` the test result directory.
- `UBOOT_PERSISTENT_DATA_DIR` the test persistent data directory. - ``UBOOT_PERSISTENT_DATA_DIR`` the test persistent data directory.
u-boot-test-console u-boot-test-console
''''''''''''''''''' '''''''''''''''''''
@ -326,15 +327,15 @@ should be connected to the board's console. This process should continue to run
indefinitely, until killed. The test suite will run this script in parallel indefinitely, until killed. The test suite will run this script in parallel
with all other hooks. with all other hooks.
This script may be implemented e.g. by executing `cu`, `kermit`, `conmux`, etc. This script may be implemented e.g. by executing ``cu``, ``kermit``, ``conmux``,
via exec(). etc. via ``exec()``.
If you are able to run U-Boot under a hardware simulator such as QEMU, then If you are able to run U-Boot under a hardware simulator such as QEMU, then
you would likely spawn that simulator from this script. However, note that you would likely spawn that simulator from this script. However, note that
`u-boot-test-reset` may be called multiple times per test script run, and must ``u-boot-test-reset`` may be called multiple times per test script run, and must
cause U-Boot to start execution from scratch each time. Hopefully your cause U-Boot to start execution from scratch each time. Hopefully your
simulator includes a virtual reset button! If not, you can launch the simulator includes a virtual reset button! If not, you can launch the
simulator from `u-boot-test-reset` instead, while arranging for this console simulator from ``u-boot-test-reset`` instead, while arranging for this console
process to always communicate with the current simulator instance. process to always communicate with the current simulator instance.
u-boot-test-flash u-boot-test-flash
@ -357,7 +358,7 @@ the following cases:
- The board allows U-Boot to be downloaded directly into RAM, and executed - The board allows U-Boot to be downloaded directly into RAM, and executed
from there. Use of this feature will reduce wear on the board's flash, so from there. Use of this feature will reduce wear on the board's flash, so
may be preferable if available, and if cold boot testing of U-Boot is not may be preferable if available, and if cold boot testing of U-Boot is not
required. If this feature is used, the `u-boot-test-reset` script should required. If this feature is used, the ``u-boot-test-reset`` script should
perform this download, since the board could conceivably be reset multiple perform this download, since the board could conceivably be reset multiple
times in a single test run. times in a single test run.
@ -386,8 +387,8 @@ to flash, pulsing the board's reset signal is likely all this script needs to
do. However, in some scenarios, this script may perform other actions. For do. However, in some scenarios, this script may perform other actions. For
example, it may call out to some SoC- or board-specific vendor utility in order example, it may call out to some SoC- or board-specific vendor utility in order
to download the U-Boot binary directly into RAM and execute it. This would to download the U-Boot binary directly into RAM and execute it. This would
avoid the need for `u-boot-test-flash` to actually write U-Boot to flash, thus avoid the need for ``u-boot-test-flash1`` to actually write U-Boot to flash,
saving wear on the flash chip(s). thus saving wear on the flash chip(s).
u-boot-test-release u-boot-test-release
''''''''''''''''''' '''''''''''''''''''
@ -407,11 +408,11 @@ Board-type-specific configuration
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Each board has a different configuration and behaviour. Many of these Each board has a different configuration and behaviour. Many of these
differences can be automatically detected by parsing the `.config` file in the differences can be automatically detected by parsing the ``.config`` file in the
build directory. However, some differences can't yet be handled automatically. build directory. However, some differences can't yet be handled automatically.
For each board, an optional Python module `u_boot_board_${board_type}` may exist For each board, an optional Python module ``u_boot_board_${board_type}`` may
to provide board-specific information to the test script. Any global value exist to provide board-specific information to the test script. Any global value
defined in these modules is available for use by any test function. The data defined in these modules is available for use by any test function. The data
contained in these scripts must be purely derived from U-Boot source code. contained in these scripts must be purely derived from U-Boot source code.
Hence, these configuration files are part of the U-Boot source tree too. Hence, these configuration files are part of the U-Boot source tree too.
@ -425,14 +426,14 @@ U-Boot configuration may support USB device mode and USB Mass Storage, but this
can only be tested if a USB cable is connected between the board and the host can only be tested if a USB cable is connected between the board and the host
machine running the test script. machine running the test script.
For each board, optional Python modules `u_boot_boardenv_${board_type}` and For each board, optional Python modules ``u_boot_boardenv_${board_type}`` and
`u_boot_boardenv_${board_type}_${board_identity}` may exist to provide ``u_boot_boardenv_${board_type}_${board_identity}`` may exist to provide
board-specific and board-identity-specific information to the test script. Any board-specific and board-identity-specific information to the test script. Any
global value defined in these modules is available for use by any test global value defined in these modules is available for use by any test
function. The data contained in these is specific to a particular user's function. The data contained in these is specific to a particular user's
hardware configuration. Hence, these configuration files are not part of the hardware configuration. Hence, these configuration files are not part of the
U-Boot source tree, and should be installed outside of the source tree. Users U-Boot source tree, and should be installed outside of the source tree. Users
should set `$PYTHONPATH` prior to running the test script to allow these should set ``$PYTHONPATH`` prior to running the test script to allow these
modules to be loaded. modules to be loaded.
Board module parameter usage Board module parameter usage
@ -443,29 +444,29 @@ module:
- none at present - none at present
U-Boot `.config` feature usage U-Boot ``.config`` feature usage
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The test scripts rely on various U-Boot `.config` features, either directly in The test scripts rely on various U-Boot ``.config`` features, either directly in
order to test those features, or indirectly in order to query information from order to test those features, or indirectly in order to query information from
the running U-Boot instance in order to test other features. the running U-Boot instance in order to test other features.
One example is that testing of the `md` command requires knowledge of a RAM One example is that testing of the ``md`` command requires knowledge of a RAM
address to use for the test. This data is parsed from the output of the address to use for the test. This data is parsed from the output of the
`bdinfo` command, and hence relies on CONFIG_CMD_BDI being enabled. ``bdinfo`` command, and hence relies on CONFIG_CMD_BDI being enabled.
For a complete list of dependencies, please search the test scripts for For a complete list of dependencies, please search the test scripts for
instances of: instances of:
- `buildconfig.get(...` - ``buildconfig.get(...``
- `@pytest.mark.buildconfigspec(...` - ``@pytest.mark.buildconfigspec(...``
- `@pytest.mark.notbuildconfigspec(...` - ``@pytest.mark.notbuildconfigspec(...``
Complete invocation example Complete invocation example
~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~
Assuming that you have installed the hook scripts into $HOME/ubtest/bin, and Assuming that you have installed the hook scripts into ``$HOME/ubtest/bin``, and
any required environment configuration Python modules into $HOME/ubtest/py, any required environment configuration Python modules into ``$HOME/ubtest/py``,
then you would likely invoke the test script as follows: then you would likely invoke the test script as follows:
If U-Boot has already been built: If U-Boot has already been built:
@ -477,7 +478,7 @@ If U-Boot has already been built:
./test/py/test.py --bd seaboard ./test/py/test.py --bd seaboard
If you want the test script to compile U-Boot for you too, then you likely If you want the test script to compile U-Boot for you too, then you likely
need to set `$CROSS_COMPILE` to allow this, and invoke the test script as need to set ``$CROSS_COMPILE`` to allow this, and invoke the test script as
follows: follows:
.. code-block:: bash .. code-block:: bash
@ -501,47 +502,47 @@ Writing tests
Please refer to the pytest documentation for details of writing pytest tests. Please refer to the pytest documentation for details of writing pytest tests.
Details specific to the U-Boot test suite are described below. Details specific to the U-Boot test suite are described below.
A test fixture named `ubman` should be used by each test function. This A test fixture named ``ubman`` should be used by each test function. This
provides the means to interact with the U-Boot console, and retrieve board and provides the means to interact with the U-Boot console, and retrieve board and
environment configuration information. environment configuration information.
The function `ubman.run_command()` executes a shell command on the The function ``ubman.run_command()`` executes a shell command on the
U-Boot console, and returns all output from that command. This allows U-Boot console, and returns all output from that command. This allows
validation or interpretation of the command output. This function validates validation or interpretation of the command output. This function validates
that certain strings are not seen on the U-Boot console. These include shell that certain strings are not seen on the U-Boot console. These include shell
error messages and the U-Boot sign-on message (in order to detect unexpected error messages and the U-Boot sign-on message (in order to detect unexpected
board resets). See the source of `console_base.py` for a complete list of board resets). See the source of ``console_base.py`` for a complete list of
"bad" strings. Some test scenarios are expected to trigger these strings. Use "bad" strings. Some test scenarios are expected to trigger these strings. Use
`ubman.disable_check()` to temporarily disable checking for specific ``ubman.disable_check()`` to temporarily disable checking for specific
strings. See `test_unknown_cmd.py` for an example. strings. See ``test_unknown_cmd.py`` for an example.
Board- and board-environment configuration values may be accessed as sub-fields Board- and board-environment configuration values may be accessed as sub-fields
of the `ubman.config` object, for example of the ``ubman.config`` object, for example
`ubman.config.ram_base`. ``ubman.config.ram_base``.
Build configuration values (from `.config`) may be accessed via the dictionary Build configuration values (from ``.config``) may be accessed via the dictionary
`ubman.config.buildconfig`, with keys equal to the Kconfig variable ``ubman.config.buildconfig``, with keys equal to the Kconfig variable
names. names.
A required configuration setting can be defined via a buildconfigspec() A required configuration setting can be defined via a ``buildconfigspec()``
annotation. The name of the configuration option is specified in lower case. The annotation. The name of the configuration option is specified in lower case. The
following annotation for a test requires CONFIG_EFI_LOADER=y: following annotation for a test requires ``CONFIG_EFI_LOADER=y``:
.. code-block:: python .. code-block:: python
@pytest.mark.buildconfigspec('efi_loader') @pytest.mark.buildconfigspec('efi_loader')
Sometimes multiple configuration option supply the same functionality. If Sometimes multiple configuration option supply the same functionality. If
multiple arguments are passed to buildconfigspec(), only one of the multiple arguments are passed to ``buildconfigspec()``, only one of the
configuration options needs to be set. The following annotation requires that configuration options needs to be set. The following annotation requires that
either of CONFIG_NET or CONFIG_NET_LWIP is set: either of ``CONFIG_NET`` or ``CONFIG_NET_LWIP`` is set:
.. code-block:: python .. code-block:: python
@pytest.mark.buildconfigspec('net', 'net lwip') @pytest.mark.buildconfigspec('net', 'net lwip')
The notbuildconfigspec() annotation can be used to require a configuration The ``notbuildconfigspec()`` annotation can be used to require a configuration
option not to be set. The following annotation requires CONFIG_RISCV=n: option not to be set. The following annotation requires ``CONFIG_RISCV=n``:
.. code-block:: python .. code-block:: python