The SDK can either be a release SDK or a dev build SDK which are stored
in different paths. DOWNLOAD_ROOT_SDK should be based on the
SDK_URL_PATH value which indicates whether it's a release or dev build
path.
bootstrap_sdk runs catalyst.sh which will try to download the SDK if the
verify digest fails.
Importing the DIGEST allows to skip this step and to continue with the
previously downloaded SDK.
Signed-off-by: Mathieu Tortuyaux <mtortuyaux@microsoft.com>
Otherwise, it was failing since we check for unbound variable:
```
/bin/bash: line 1: PORTAGE_BINHOST: unbound variable
```
Signed-off-by: Mathieu Tortuyaux <mtortuyaux@microsoft.com>
Otherwise, the variable is empty and it creates errors later. Default
value is `gs://flatcar-jenkins`. Not `GS_DEVEL_ROOT` because if we check
the previous behavior, `DOWNLOAD_ROOT` was hardcoded with:
```shell
DOWNLOAD_ROOT_SDK=https://storage.googleapis.com/flatcar-jenkins/sdk
```
Signed-off-by: Mathieu Tortuyaux <mtortuyaux@microsoft.com>
`$verify_key` actually holds `--verify-key=verify.asc` so of course
`systemd-nspawn` fails since it does not expect `--verify-key` value.
Signed-off-by: Mathieu Tortuyaux <mtortuyaux@microsoft.com>
The catalyst build uses the same SDK version as seed as the current SDK, but
will only reuse the cached tarball if a DIGESTS file exists and is correct.
Prefetch this file to prevent the build from trying to access google storage
anonymously.
Signed-off-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>
because we need to pass google credentials to update_chroot, and 'cork update'
doesn't support that.
Add --sdk-url-path to sdk.sh for new cork default.
in this commit we make sure to use GCS bucket for dev container tests by
providing the required credentials and the associated fetch command.
Signed-off-by: Mathieu Tortuyaux <mtortuyaux@microsoft.com>
This updates the default settings in build scripts to use
https://mirror.release.flatcar-linux.net/
instead of the google storage bucket if no binhost or FLATCAR_DEV_BUILDS
is specified.
Defaults are updated for
* update_chroot (runs at SDK initialisation time)
* setup_board (creates /boards/[ARCH]/) chroots
* the development container
* set_version
The cl.basic and cl.internet tests are different tests which wasn't
clear before. Also, the grep process returns an exit code of 1 if it
didn't find a match, causing the job to cancel. The list of tests is
space separated and should not be quoted but on the other hand, we
do have to handle a literal *.
Look for the right test and handle the grep exit code, and disable
globs for the subshell for preserving a literal *.
The Linux 5.10 stable kernel introduced a regression that we didn't
catch because we only run kola on one hardware type in Equinix Metal.
Validate that a simple network test works on various instance types of
the current hardware generation.
Now that GLSA metadata was updated as of 2021-09-03, we need to
add the following entries to the GLSA allow list, to avoid build
failures caused by `glsa-check -t all`.
202006-03: perl 5.26.2, only SDK, allowlist
202008-01: python 2.7.15 & 3.6.5, only SDK, allowlist
202101-18: python 2.7.15 & 3.6.5, only SDK, allowlist
202104-04: python 2.7.15 & 3.6.5, only SDK, allowlist
202105-22: samba 4.12.9, not affected, samba has no ldap flag, no smbd.
202105-34: bash 4.3, non-trivial to update
202107-31: polkit 0.113, in-progress
202107-48: systemd 247.9, backported the fixes to v247.9.
Included is a dockerfile that installs system deps of kola in an debian:11
image. For the test script, the control flow is:
qemu_uefi.sh
qemu_uefi_arm64.sh
(docker)
qemu_common.sh
qemu_common uses the 'NATIVE_ARM64' variable passed by the jenkins job to control the behavior.
The differences are:
* use git directly to fetch (and verify) the manifest
* setup some symlinks so that /var/tmp is on the same BTRFS partition as $PWD/tmp
* setup symlinks so that we don't have to fixup installation of mantle to chroot
* run things directly instead of in chroot through cork
The whole script is executed as root, because kola requires root privileges
anyway and making kvm and sudo work with an arbitrary host user inside the
container would require a custom entrypoint to setup groups.
Signed-off-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>
201904-13: git 2.26.3, so not affected
201909-08: dbus 1.12.20, so not affected
201911-01: openssh 8.6, so not affected
202003-12: sudo 1.9.5, so not affected
202003-20: systemd 246+, so not affected
202003-24: file 5.39, so not affected
202003-30: git 2.26.3, so not affected
202003-31: gdb 9.2, so not affected
202003-52: samba 4.12.9, so not affected
202004-10: openssl 1.1.1l, so not affected
202004-13: git 2.26.3, so not affected
202005-02: qemu 5.2, so not affected
disk_util sometimes bails out during build with an ASCII conversion
error:
Traceback (most recent call last):
File "/mnt/host/source/src/scripts/build_library/disk_util", line 1114, in <module>
main(sys.argv)
File "/mnt/host/source/src/scripts/build_library/disk_util", line 1110, in main
options.func(options)
File "/mnt/host/source/src/scripts/build_library/disk_util", line 779, in Verity
Tune2fsReadWrite(options, part, disable_rw=True)
File "/mnt/host/source/src/scripts/build_library/disk_util", line 716, in Tune2fsReadWrite
image.write(chr(flag_value))
UnicodeEncodeError: 'ascii' codec can't encode character '\xff' in position 0: ordinal not in range(128)
Curiously, the error does not reproduce every time (though the code
leading to the error is straightforward).
This change converts the integer to be written to a byte array (of size
1) instead of using chr(). Also, the file to be written is explicitly
opened in binary mode.
Signed-off-by: Thilo Fromm <thilo@kinvolk.io>
This file used to be imported by scripts coreos-base/cros-devutils,
which we have dropped. Now it is imported only from some other script
in build library so move it there. This leaves lib as a directory
where we keep shflags library.
The scripts were imported by some scripts from
coreos-base/cros-devutils, which we have dropped already. So the
scripts in the lib directory are currently an unused baggage.
This requires passing the --azure-hyper-v-generation=V2 argument to kola. The
vhd/image is the same as for azure gen1 vms, the azure_gen2 specifier is only
for jenkins usage.
Signed-off-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>
The scripts that invoked `cros_workon` without specifying a path to
the script were not calling the internal `cros_workon` directly, but
rather a copy installed in `/usr/bin/cros_workon`.
`/usr/bin/cros_workon` comes from the `coreos-base/cros-devutil` and
is a wrapper script that sources `common.sh` file to figure the
location of the `scripts` and finally invokes the internal
`cros_workon`. Curious thing is that the sourced `common.sh` comes
from the `/usr/lib/crosutils` directory and contents of the directory
come from the `dev-util/crosutils` package. And that `common.sh` is
different from the one in the scripts directory, but fortunately the
part that detects the path to the `scripts` directory is the same. I'm
not sure where where exactly the copy of `common.sh` in
`/usr/lib/crosutils` comes from - likely from somewhere in
`https://chromium.googlesource.com/chromiumos/platform/crosutils`.
Just cut the middle layers and call the internal copy of `cros_workon`
directly.
This is some fallout from converting scripts from python2 to
python3. Output received from the functions in subprocess module now
return bytearrays, but we operate on them as if they were a text. So
decode the bytearrays to strings. Otherwise we are either getting some
junk values passed to the command line utilities (for example:
`b'/dev/loop2'` instead of `/dev/loop2`), or exceptions are thrown,
because a function expected a string.
This is just a conversion done by 2to3 with a manual updates of
shebangs to mention python3 explicitly. The fixups for bytearray vs
string issues will follow up.
The script is potentially useful, but it seems to be unused anyway. We
can bring it back later if there's a need. Note that this will need
updating it to python3 first. Which is why I'm dropping it currently -
it's one python2 script less to port.
