Most of this hinges on the --upload option being passed, and it never is
any more. Much of it also uses Google Buckets, which we no longer use,
save for some GCE-specific bits.
Signed-off-by: James Le Cuirot <jlecuirot@microsoft.com>
Catalyst 4 has totally changed the way repositories are handled. It only
works when the name of the directory containing the repository matches
the configured name of that repository. This was not the case for us,
with the coreos repository residing in the coreos-overlay directory. We
wanted to move and rename our repositories anyway, but this is a big
change, so we'll do separately. For now, this just renames coreos to
coreos-overlay.
Catalyst 4 also ingests the main repository snapshot as a squashfs
rather than a tarball. It features a utility to generate such a
snapshot, but it doesn't fit Flatcar well, particularly because it
expects each ebuild repository to reside at the top level of its own git
repository. It was very easy to call tar2sqfs manually though.
Signed-off-by: James Le Cuirot <jlecuirot@microsoft.com>
Catalyst runs builds with copies of the portage/coreos overlays in a
chroot, which prevents us from accessing the git metadata necessary to
create provenance information. Copy some files over into the
root_overlay used by the toolchains catalyst build so that provenance
can be correctly captured.
ROOT_OVERLAY variable is defined in terms of TEMPDIR. The TEMPDIR
variable is set to an empty value by catalyst.sh, which the two
scripts import. So ROOT_OVERLAY always ended up being located in
toplevel directory (i.e. `/`). But the TEMPDIR variable gets a
meaningful value after calling the catalyst_init function, so define
the ROOT_OVERLAY after the function is called.
Before, we were relying on the toolchains job to build and upload
packages that were part of the SDK. With this change, all packages that
should be part of the SDK are built and uploaded by the SDK job. The
toolchains job only builds toolchain packages specific for the release.
This change includes several adjustments done to both the SDK and the
toolchains jobs to make this work:
* Make the SDK job build all cross toolchains, including Rust
* Stop building Rust in the toolchains job and use the one in the SDK
instead.
* In toolchain_util.sh: detect when the symlink folder for crossdev
packages is missing and run crossdev to create it during
update_chroot setup.
* Make it possible to build the SDK starting from stage 4 instead of
stage 1, to make the SDK building faster for PR branches / nightlies
(full build should still be done for releases / weeklies).
The Rust arm64 crossdev toolchain package was uploaded to
toolchain-arm64/rust… instead of toolchain-arm64/dev-lang/rust….
Upload the complete dev-lang folder as only Rust will be fetched
from there anyway.
Two Flatcar versions were used in /etc/portage/make.conf both in the SDK
and in the boards.
Use only a single version by default to get the expected results and not
something else when using binary packages.
The Rust crossdev package was never uploaded to /sdk/ and always
had to be compiled again.
Upload it in a separate toolchain-arm64 directory because /Packages in /crossdev/
doesn't refer to the Rust package and its use flags.
Right now there is some funky logic to either use a previous build as a
seed or the current SDK tarball if it happens to have been downloaded.
This is a bit confusing and doesn't work reliably since it is reasonable
for there to be neither a previous build or the current SDK available if
the SDK chroot was created some time ago. Fix this by using the new SDK
library and always use the latest SDK, downloading it if needed.
Currently we don't have a good way to upload packages from different
jobs to the same location. The 'Packages' index file is only generated
locally so the second upload would always win.
This replaces the cross-toolchain compile step in bootstrap_sdk and adds the
ability to build native toolchains using the cross toolchain. This is just
the first step towards actually providing the native toolchain in a container.
