Back-port for flatcar-2983 branch.
This change introduces a containerised SDK as a replacement for cork SDK
operations. It also simplifies versioning by removing the need for
manifest repos as well as usage of the "repo" tool by use of git
submodules for coreos-overlay and portage-stable.
The following feature scripts are added:
- run_sdk_container: Run a command in an SDK container, using the
current scripts repo + ebuild submodules.
current scripts repo + ebuild submodules.
- bootstrap_sdk_container / build_sdk_container_image: Bootstrap a new
SDK and create an SDK container from the resulting SDK tarball.
The following additions have been made to SDK scripts:
- setup_board: add --pkgdir parameter to use a custom binary packge
directory.
Currently the kubeadm tests fail on arm64 because the instance type
only offers 1 vCPU:
cluster.go:117: error execution phase preflight: [preflight] Some fatal errors occurred:
cluster.go:117: [ERROR NumCPU]: the number of available CPUs 1 is less than the required 2
Switch to the next larger instance type which has 2 vCPUS.
The SDK can either be a release SDK or a dev build SDK which are stored
in different paths. DOWNLOAD_ROOT_SDK should be based on the
SDK_URL_PATH value which indicates whether it's a release or dev build
path.
bootstrap_sdk runs catalyst.sh which will try to download the SDK if the
verify digest fails.
Importing the DIGEST allows to skip this step and to continue with the
previously downloaded SDK.
Signed-off-by: Mathieu Tortuyaux <mtortuyaux@microsoft.com>
Otherwise, it was failing since we check for unbound variable:
```
/bin/bash: line 1: PORTAGE_BINHOST: unbound variable
```
Signed-off-by: Mathieu Tortuyaux <mtortuyaux@microsoft.com>
Otherwise, the variable is empty and it creates errors later. Default
value is `gs://flatcar-jenkins`. Not `GS_DEVEL_ROOT` because if we check
the previous behavior, `DOWNLOAD_ROOT` was hardcoded with:
```shell
DOWNLOAD_ROOT_SDK=https://storage.googleapis.com/flatcar-jenkins/sdk
```
Signed-off-by: Mathieu Tortuyaux <mtortuyaux@microsoft.com>
`$verify_key` actually holds `--verify-key=verify.asc` so of course
`systemd-nspawn` fails since it does not expect `--verify-key` value.
Signed-off-by: Mathieu Tortuyaux <mtortuyaux@microsoft.com>
The catalyst build uses the same SDK version as seed as the current SDK, but
will only reuse the cached tarball if a DIGESTS file exists and is correct.
Prefetch this file to prevent the build from trying to access google storage
anonymously.
Signed-off-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>
because we need to pass google credentials to update_chroot, and 'cork update'
doesn't support that.
Add --sdk-url-path to sdk.sh for new cork default.
in this commit we make sure to use GCS bucket for dev container tests by
providing the required credentials and the associated fetch command.
Signed-off-by: Mathieu Tortuyaux <mtortuyaux@microsoft.com>
This updates the default settings in build scripts to use
https://mirror.release.flatcar-linux.net/
instead of the google storage bucket if no binhost or FLATCAR_DEV_BUILDS
is specified.
Defaults are updated for
* update_chroot (runs at SDK initialisation time)
* setup_board (creates /boards/[ARCH]/) chroots
* the development container
* set_version
The cl.basic and cl.internet tests are different tests which wasn't
clear before. Also, the grep process returns an exit code of 1 if it
didn't find a match, causing the job to cancel. The list of tests is
space separated and should not be quoted but on the other hand, we
do have to handle a literal *.
Look for the right test and handle the grep exit code, and disable
globs for the subshell for preserving a literal *.
The Linux 5.10 stable kernel introduced a regression that we didn't
catch because we only run kola on one hardware type in Equinix Metal.
Validate that a simple network test works on various instance types of
the current hardware generation.
Included is a dockerfile that installs system deps of kola in an debian:11
image. For the test script, the control flow is:
qemu_uefi.sh
qemu_uefi_arm64.sh
(docker)
qemu_common.sh
qemu_common uses the 'NATIVE_ARM64' variable passed by the jenkins job to control the behavior.
The differences are:
* use git directly to fetch (and verify) the manifest
* setup some symlinks so that /var/tmp is on the same BTRFS partition as $PWD/tmp
* setup symlinks so that we don't have to fixup installation of mantle to chroot
* run things directly instead of in chroot through cork
The whole script is executed as root, because kola requires root privileges
anyway and making kvm and sudo work with an arbitrary host user inside the
container would require a custom entrypoint to setup groups.
Signed-off-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>
disk_util sometimes bails out during build with an ASCII conversion
error:
Traceback (most recent call last):
File "/mnt/host/source/src/scripts/build_library/disk_util", line 1114, in <module>
main(sys.argv)
File "/mnt/host/source/src/scripts/build_library/disk_util", line 1110, in main
options.func(options)
File "/mnt/host/source/src/scripts/build_library/disk_util", line 779, in Verity
Tune2fsReadWrite(options, part, disable_rw=True)
File "/mnt/host/source/src/scripts/build_library/disk_util", line 716, in Tune2fsReadWrite
image.write(chr(flag_value))
UnicodeEncodeError: 'ascii' codec can't encode character '\xff' in position 0: ordinal not in range(128)
Curiously, the error does not reproduce every time (though the code
leading to the error is straightforward).
This change converts the integer to be written to a byte array (of size
1) instead of using chr(). Also, the file to be written is explicitly
opened in binary mode.
Signed-off-by: Thilo Fromm <thilo@kinvolk.io>
This file used to be imported by scripts coreos-base/cros-devutils,
which we have dropped. Now it is imported only from some other script
in build library so move it there. This leaves lib as a directory
where we keep shflags library.
The scripts were imported by some scripts from
coreos-base/cros-devutils, which we have dropped already. So the
scripts in the lib directory are currently an unused baggage.
This requires passing the --azure-hyper-v-generation=V2 argument to kola. The
vhd/image is the same as for azure gen1 vms, the azure_gen2 specifier is only
for jenkins usage.
Signed-off-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>
The scripts that invoked `cros_workon` without specifying a path to
the script were not calling the internal `cros_workon` directly, but
rather a copy installed in `/usr/bin/cros_workon`.
`/usr/bin/cros_workon` comes from the `coreos-base/cros-devutil` and
is a wrapper script that sources `common.sh` file to figure the
location of the `scripts` and finally invokes the internal
`cros_workon`. Curious thing is that the sourced `common.sh` comes
from the `/usr/lib/crosutils` directory and contents of the directory
come from the `dev-util/crosutils` package. And that `common.sh` is
different from the one in the scripts directory, but fortunately the
part that detects the path to the `scripts` directory is the same. I'm
not sure where where exactly the copy of `common.sh` in
`/usr/lib/crosutils` comes from - likely from somewhere in
`https://chromium.googlesource.com/chromiumos/platform/crosutils`.
Just cut the middle layers and call the internal copy of `cros_workon`
directly.
