Currently the os/sdk and os/toolchains job perform a chroot update whose
results are immediately discarded because the rest of the build uses a fresh
chroot and catalyst. Towards the end of a release period this can extend the
build time by about an hour (longer if rust is involved).
Introduce a `--setuponly` flag that bails after the chroot configuration, and
the skips chroot update.
Signed-off-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>
The SDK can either be a release SDK or a dev build SDK which are stored
in different paths. DOWNLOAD_ROOT_SDK should be based on the
SDK_URL_PATH value which indicates whether it's a release or dev build
path.
bootstrap_sdk runs catalyst.sh which will try to download the SDK if the
verify digest fails.
Importing the DIGEST allows to skip this step and to continue with the
previously downloaded SDK.
Signed-off-by: Mathieu Tortuyaux <mtortuyaux@microsoft.com>
Otherwise, the variable is empty and it creates errors later. Default
value is `gs://flatcar-jenkins`. Not `GS_DEVEL_ROOT` because if we check
the previous behavior, `DOWNLOAD_ROOT` was hardcoded with:
```shell
DOWNLOAD_ROOT_SDK=https://storage.googleapis.com/flatcar-jenkins/sdk
```
Signed-off-by: Mathieu Tortuyaux <mtortuyaux@microsoft.com>
because we need to pass google credentials to update_chroot, and 'cork update'
doesn't support that.
Add --sdk-url-path to sdk.sh for new cork default.
The logic of the inline bash scripts of each job was sometimes
separated into the flatcar-scripts/jenkins/*.sh helpers but mostly
part of the Groovy file. This coupling had its advantages but also
downsides when special cases needed to be added for different release
versions. Other issues were that the inline scripts needed the
backslash character to be escaped twice and Jenkins was not good in
terminating the child processes when stopping a job. Having inline
bash scripts in Groovy also mandated the use of Jenkins to build and
release Flatcar Container Linux which hinders test builds in other CI
platforms.
Move the inline bash scripts fully to to the files in
flatcar-scripts/jenkins/ and create new ones for job that didn't have
a script there yet. Also invoke them through a systemd-run wrapper
script which ensures that all child processes are terminated and also
sets up /opt/bin as additional path for the static lbzcat binary.
A workaround for bash 4 was needed to use a temporary file instead of
the <(cmd) bash feature which caused a strange syntax error, otherwise
the bash commands are moved as they are.
Before, we were relying on the toolchains job to build and upload
packages that were part of the SDK. With this change, all packages that
should be part of the SDK are built and uploaded by the SDK job. The
toolchains job only builds toolchain packages specific for the release.
This change includes several adjustments done to both the SDK and the
toolchains jobs to make this work:
* Make the SDK job build all cross toolchains, including Rust
* Stop building Rust in the toolchains job and use the one in the SDK
instead.
* In toolchain_util.sh: detect when the symlink folder for crossdev
packages is missing and run crossdev to create it during
update_chroot setup.
* Make it possible to build the SDK starting from stage 4 instead of
stage 1, to make the SDK building faster for PR branches / nightlies
(full build should still be done for releases / weeklies).
The dev build SDKs are not in $FLATCAR_DEV_BUILDS/sdk but published under
$FLATCAR_DEV_BUILDS/developer/sdk.
Add an environment variable to specify where the SDK is to be found
but default to $FLATCAR_DEV_BUILDS/sdk if it is not specified.
From Jenkins this variable is exported as DOWNLOAD_ROOT_SDK.
More space can be saved by removing things that get overwritten on
the next job run, but they are used after this script runs (e.g.
for fingerprinting). Drop the cleanup from these scripts and move
it all to the post-build pipeline stage.
We are going to restore the split-script setup from the old Jenkins
server. This ensures that the each version's release process is
actually running with scripts in the correct release branch. It
also allows branching the VM format lists.
Note that the scripts added here only cover the currently active
jobs in the main build pipeline. There is no reason to add other
jobs, since they are mostly just running a single command using a
mantle binary from its master branch.
The scripts in this repository pick up after Jenkins has set up an
environment with all parameters and credentials defined, and an SDK
was prepared and validated.
