ci-automation/tapfile: enforce foreign keys, simplify

ci-automation/README.md

@@ -2,16 +2,39 @@
 
 Scripts in this directory aim to ease automation of Flatcar builds in continuous integration systems.
 
+Design goal of the automation scripts is to provide self-contained, context-aware automation with a low integration overhead.
+Each step takes its context from the repository (version to build etc.) and from the artifact of the previous build, with the aim of reducing the number of arguments to an absolute minimum.
+
 Each script represents a distinct build step; each step ingests the container image of the previous step and produces a new container image for the next step.
 Notable exceptions are "SDK Bootstrap" (`sdk.sh`) which only creates an SDK tarball, and "VMs build" which does not output a container but only VM (vendor) images.
+The container images are self-contained and aim for ease of reproducibility.
+All steps make use of a "build cache" server for pulling (https) build inputs and for pushing (rsync) artifacts.
+
+Test automation is provided alongside build automation, following the same design principles.
 
 Please refer to the individual scripts for prerequisites, input parameters, and outputs.
 
-All steps make use of a "build cache" server for pulling (https) and pushing (rsync) build inputs and artifacts.
 
 ## Build steps
 
 The build pipeline can be used to build everything from scratch, including the SDK (starting from 1. below) or to build a new OS image (starting from 3.).
+"From scratch" builds (i.e. builds which include a new SDK) are usually only done for the `main` branch (`main` can be considered `alpha-next`).
+Release / maintenance branches in the majority of cases do note build a new SDK but start with the OS image build.
+Release branches usually use the SDK introduced when the new major version was branched off `main` throughout the lifetime of the major version; i.e. release `stable-MMMM.mm.pp` would use `SDK-MMMM.0.0`.
+
+To reproduce any given build step, follow this pattern:
+```
+./checkout <build-tag> # Build tag from either SDK bootstrap pr Packages step
+source ci-automation/<step-script>.sh
+<step_function> <parameters>
+```
+
+For example, to rebuild the AMD64 OS image of build `main-3145.0.0-nightly-20220209-0139`, do
+```
+./checkout main-3145.0.0-nightly-20220209-0139
+source ci-automation/image.sh
+image_build amd64
+```
 
 ### SDK bootstrap build
 
@@ -69,6 +92,7 @@ The build pipeline can be used to build everything from scratch, including the S
             |                             `--------´                   |
             |<-- tag: alpha-3499.0.0-dev23 --´|`- sdk + OS packages -->|
             |                                 |    container image     |
+            |                                 |    torcx manifest      |
             |                           ______v_______                 |
             |                          ( publish pkgs )                |
             |                           `------------´                 |
@@ -82,3 +106,37 @@ The build pipeline can be used to build everything from scratch, including the S
                  alpha-3499.0.0-dev23       `---´                      |
                                               `- vendor OS images ---->|
 ```
+
+## Testing
+
+Testing follows the same design principles build automation adheres to - it's self-contained and context-aware, reducing required parameters to a minimum.
+The `test.sh` script needs exactly two parameters: the architecture, and the image type to be tested.
+Optionally, patterns matching a group of tests can be supplied (or simply a list of tests); this defaults to "all tests" of a given vendor / image.
+`test.sh` also supports re-running failed tests automatically to reduce the need for human interaction on flaky tests.
+
+Testing is implemented in two layers:
+1. `ci-automation/test.sh` is a generic test wrapper / stub to be called from CI.
+2. `ci-automation/vendor-testing/` contains low-level vendor-specific test wrappers around [`kola`](https://github.com/flatcar-linux/mantle/tree/flatcar-master/kola/), our test scenario orchestrator.
+
+Testing relies on the SDK container and will use tools / test suites from the SDK.
+The low-level vendor / image specific script (layer 2. in the list above) is run inside the SDK.
+Testing will use the vendor image published by `vms.sh` from buildcache, and the torcx manifest published by `packages`.
+
+Additionally, a script library is provided (at `ci-automation/tapfile_helper_lib.sh`) to help handling `.tap` test result files produced by test runs.
+Library functions may be used to merge the result of multiple test runs (e.g. for multiple image types / vendors) into a single test result report.
+The test runs are considered successful only if all tests succeeded for all vendors / images at least once.
+
+**Usage**
+```
+./checkout <version-to-test>
+source ci-automation/test.sh
+test_run <arch> <image-type>
+```
+
+E.g. for running qemu / amd64 tests on `main-3145.0.0-nightly-20220209-0139`:
+```
+./checkout main-3145.0.0-nightly-20220209-0139
+source ci-automation/test.sh
+test_run amd64 qemu
+```
+

ci-automation/tapfile_helper_lib.sh

@@ -19,11 +19,16 @@ TAPFILE_HELPER_DBNAME="results.sqlite3"
 
 # wrapper around sqlite3 w/ retries if DB is locked
 function __sqlite3_wrapper() {
-    local dbfile="$1"
+    local dbfile="${TAPFILE_HELPER_DBNAME}"
+
+    local params=""
+    while [[ "$1" == -* ]] ; do
+        params="$params $1"
         shift
+    done
 
     while true; do
-        sqlite3 "${dbfile}" "$@"
+        sqlite3 "${dbfile}" $params "PRAGMA foreign_keys = ON;$@"
         local ret="$?"
         if [ "$ret" -ne 5 ] ; then
             return $ret
@@ -37,9 +42,8 @@ function __sqlite3_wrapper() {
 
 # Initialise the DB if it wasn't yet.
 function __db_init() {
-    local dbname="${TAPFILE_HELPER_DBNAME}"
 
-    __sqlite3_wrapper "${dbname}" '
+    __sqlite3_wrapper '
     CREATE TABLE IF NOT EXISTS "test_case" (
         "id"    INTEGER,
         "name"  TEXT UNIQUE,
@@ -77,8 +81,6 @@ function tap_ingest_tapfile() {
     local vendor="${2}"
     local run="${3}"
 
-    local dbname="${TAPFILE_HELPER_DBNAME}"
-
     local result=""
     local test_name=""
     local error_message=""
@@ -138,15 +140,13 @@ function tap_ingest_tapfile() {
 
     local SQL="${SQL}COMMIT;"
 
-    __sqlite3_wrapper "${dbname}" "${SQL}"
+    __sqlite3_wrapper "${SQL}"
 }
 # --
 
 # Print a list of all vendors we've seen so far.
 function tap_list_vendors() {
-    local dbname="${TAPFILE_HELPER_DBNAME}"
+    __sqlite3_wrapper 'SELECT DISTINCT name from vendor;'
-
-    __sqlite3_wrapper "${dbname}" 'SELECT DISTINCT name from vendor;'
 }
 # --
 
@@ -156,9 +156,7 @@ function tap_list_vendors() {
 function tap_failed_tests_for_vendor() {
     local vendor="$1"
 
-    local dbname="${TAPFILE_HELPER_DBNAME}"
+    __sqlite3_wrapper "
-
-    __sqlite3_wrapper "${dbname}" "
 		SELECT failed.name FROM test_case AS failed
 		WHERE EXISTS (
 				SELECT * FROM test_run AS t, vendor AS v, test_case AS c
@@ -186,12 +184,10 @@ function tap_generate_report() {
     local version="$2"
     local full_error_report="${3:-false}"
 
-    local dbname="${TAPFILE_HELPER_DBNAME}"
-
     local count
-    count="$(__sqlite3_wrapper "${dbname}" 'SELECT count(name) FROM test_case;')"
+    count="$(__sqlite3_wrapper 'SELECT count(name) FROM test_case;')"
     local vendors
-    vendors="$(__sqlite3_wrapper "${dbname}" 'SELECT name FROM vendor;' | tr '\n' ' ')"
+    vendors="$(__sqlite3_wrapper 'SELECT name FROM vendor;' | tr '\n' ' ')"
 
     echo "1..$((count+1))"
     echo "ok - Version: ${version}, Architecture: ${arch}" 
@@ -201,13 +197,13 @@ function tap_generate_report() {
 
     # Print result line for every test, including platforms it succeeded on
     #  and transient failed runs.
-    __sqlite3_wrapper "${dbname}" 'SELECT DISTINCT name from test_case;' | \
+    __sqlite3_wrapper 'SELECT DISTINCT name from test_case;' | \
     while read -r test_name; do
 
         # "ok" if the test succeeded at least once for all vendors that run the test,
         #   "not ok" otherwise.
         local verdict
-        verdict="$(__sqlite3_wrapper "${dbname}" "
+        verdict="$(__sqlite3_wrapper "
         SELECT failed.name FROM vendor AS failed
         WHERE EXISTS (
                 SELECT * FROM test_run AS t, vendor AS v, test_case AS c
@@ -231,7 +227,7 @@ function tap_generate_report() {
         # Generate a list of vendors and respective runs, in a single line.
         function list_runs() {
             local res="$1"
-            __sqlite3_wrapper -csv "${dbname}" "
+            __sqlite3_wrapper -csv "
                 SELECT v.name, t.run FROM test_run AS t, vendor AS v, test_case AS c
                 WHERE t.vendor_id=v.id AND t.case_id=c.id
                     AND c.name='${test_name}'
@@ -262,7 +258,7 @@ function tap_generate_report() {
             echo "   Failed: ${failed}"
             if [ "${verdict}" = "not ok" -o "${full_error_report}" = "true" ] ; then
                 # generate diagnostic output, per failed run.
-                __sqlite3_wrapper -csv "${dbname}" "
+                __sqlite3_wrapper -csv "
                 SELECT v.name, t.run
                     FROM test_run AS t, vendor AS v, test_case AS c
                     WHERE t.vendor_id=v.id AND t.case_id=c.id
@@ -272,7 +268,7 @@ function tap_generate_report() {
                 sed 's/,/ /' | \
                 while read -r vendor run; do
                     echo "   Error messages for ${vendor}, run ${run}:"
-                    __sqlite3_wrapper -csv "${dbname}" "
+                    __sqlite3_wrapper -csv "
                     SELECT t.output FROM test_run AS t, test_case AS c
                         WHERE t.case_id=c.id
                         AND c.name='${test_name}'

ci-automation/test.sh

@@ -122,7 +122,7 @@ function test_run() {
             break
         fi
 
-        echo "########### Some tests failed and will be re-run. ###########"
+        echo "########### Some tests failed and will be re-run (${retry} / ${retries}). ###########"
         echo "Failed tests: $failed_tests"
         echo "-----------"
         set -- $failed_tests