Most of this hinges on the --upload option being passed, and it never is
any more. Much of it also uses Google Buckets, which we no longer use,
save for some GCE-specific bits.
Signed-off-by: James Le Cuirot <jlecuirot@microsoft.com>
From https://wiki.gentoo.org/wiki/Catalyst/Stage_Creation#Build_Stage3:
> It is not necessary to build stage2 in order to build stage3. Gentoo
> release engineering does not build stage2, and you should not need to
> unless you're intentionally building a stage2 as your goal.
We can now sync portage-stable/scripts with upstream because
bootstrap.sh is only used during stage2, and the changes we had are no
longer relevant. It seems likely the changes were already redundant
anyway.
Signed-off-by: James Le Cuirot <jlecuirot@microsoft.com>
This is what upstream Gentoo does. They would previously update the
entire seed, but this took a long time. Our seeds are much bigger, so we
kept repo snapshots to build stage1 against these instead. The new
method of only rebuilding packages with changed sub-slots is a good
compromise and removes the need to write stage1 hooks that selectively
catch the repository up.
This also avoids some conflicts by adding the `--ignore-world` option.
Gentoo seeds have nothing in @world. We have much more, but none of that
is needed for stage1.
This continues to exclude cross-*-cros-linux-gnu/* as that is not needed
for stage1. It now also excludes dev-lang/rust, because it is never a
DEPEND, so it would not break other packages in this way. It may fail to
run due to a sub-slot change in one of its own dependencies, but it is
also unlikely to be needed in stage1 and it is not configured to use the
system LLVM. If needs be, we could improve the behaviour of Portage's
@changed-subslot to respect `--with-bdeps`.
In my testing, it was unable to handle an SDK from 17 months ago, but
one from 7 months ago did work. In practise, we will always use a much
more recent one, which is far more likely to work.
Signed-off-by: James Le Cuirot <jlecuirot@microsoft.com>
Catalyst 4 has totally changed the way repositories are handled. It only
works when the name of the directory containing the repository matches
the configured name of that repository. This was not the case for us,
with the coreos repository residing in the coreos-overlay directory. We
wanted to move and rename our repositories anyway, but this is a big
change, so we'll do separately. For now, this just renames coreos to
coreos-overlay.
Catalyst 4 also ingests the main repository snapshot as a squashfs
rather than a tarball. It features a utility to generate such a
snapshot, but it doesn't fit Flatcar well, particularly because it
expects each ebuild repository to reside at the top level of its own git
repository. It was very easy to call tar2sqfs manually though.
Signed-off-by: James Le Cuirot <jlecuirot@microsoft.com>
Stage1 hooks will receive a path to a file as a third parameter. They
can use it to tell the bootstrap script to set up catalyst to perform
updates on seed SDK. Contents of the file are ignored - what counts is
that the file exists AND is not empty.
- Fix the snapshot name, it is not "portage-${VERSION}", but rather
"gentoo-${VERSION}".
- After building the snapshot, remove all the similar files from the
snapshots directory - Catalyst gets easily confused by them and
bails out.
- Extend the `build_snapshot` function to optionally accept the config
path and the snapshot name, so SDK's stage1 code can use this
function instead of duplicating parts of it.
Sometimes the modification to stage1 repos is necessary. For example
we will need to update baselayout package in stage1, so we get the
cleaned up handling of PATH and ROOTPATH, which will allow a newer
portage to succeed in calculating dependencies in stage2.
That way we can see a report of what emerge is going to do and the
status of the use flags for the installed packages. The downside is
that we are going to have reports about using deprecated and
unsupported profile in even more places.
Emerge flags are cryptic in general, but short flags even more so, so
expand them. While at it, I noticed some places where bash arrays
could be used, so convert those places too.
Stop relying on github redirects, they are a mixed blessing and using
them broke emerge-gitclone inside dev-container in silent way. The
script could not find a desired revision of portage-stable or
coreos-overlay, because it tried to pull from kinvolk instead of
flatcar-linux github org. The redirects seem to hinder fetching a
specific commit, so the script pulled something else (HEAD or main?).
Previously before https://github.com/kinvolk/flatcar-scripts/pull/134,
`bootstrap_sdk` was looking at the wrong path
(/usr/lib/rust-*/rustlib/aarch64-unknown-linux-gnu instead of
/usr/lib/rustlib/aarch64-unknown-linux-gnu). As a result, Rust got always
removed and rebuilt in `install_cross_rust`, which resulted in
`flatcar-sdk/crossdev/dev-lang/rust/rust-1.54.0-1.xpak` being created.
Now legitimate changes of https://github.com/kinvolk/flatcar-scripts/pull/134
prevent the rebuild from happening. The path already exists in a stage4
SDK build, because the seed stage already has cross-compilers so the
Rust upgrade has all the right cross-paths.
That's why SDK builds with only stage4 failed when it tries uploading Rust
packages like the following. On the other hand, full SDK builds with stage1
to 4 worked well, because in that case Rust is rebuilt anyway.
```
INFO bootstrap_sdk: Uploading cross toolchain packages to
gs://flatcar-jenkins/developer/sdk/amd64/2021.08.04+dev-flatcar-master-3209
CommandException: No URLs matched:
/mnt/host/source/src/build/catalyst/packages/flatcar-sdk/crossdev/*
CommandException: No URLs matched:
/tmp/tmp.xyjXbCFhUc//mnt/host/source/src/build/catalyst/packages/flatcar-sdk/crossdev/*.sig
CommandException: 2 files/objects could not be transferred.
```
To fix that, we have to skip uploading packages when the crossdev
directory does not exist.
Debugged and suggested by @jepio
When performing a full bootstrap (stage1-4), the stage1 results are currently
discarded because of the logic in catalyst_build: the first build stage uses
the "seed" and every following stage uses the previous stages results *but*
stage1 is built before catalyst_build. So from the point of view of
catalyst_build, stage2 is the first one and uses the seed tarball.
To make sure stage1 results are used if it was built, set the SEED variable to
the latest stage1 location.
Signed-off-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>
The arm64 profiles don't specify SYMLINK_LIB=yes, which makes sense
since arm64 systems don't support multilib in the way that we are used
to from x86. What this means is that build artifacts are installed into
separate lib and lib64 directories. The root overlay installed in stage4
needs to check for SYMLINK_LIB before trying to create a symlink,
otherwise it fails to be applied because it collides with the directory
in the rootfs.
This uncovered a second minor issues - the rust toolchain bootstrap
scripts checked for /usr/lib64/rust*, but the ebuild installs to
/usr/lib/rust.
Signed-off-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>
This change uses portage-stable and coreos-overlay from the local SDK
chroot (from /var/lib/gentoo/repos) in the stage 1 SDK bootstrap build.
This is part 2 of the SDK bootstrap stage 1 fix (part 1 is covered in
64d8a73ac0), which ensures stage 1 does
not introduce any changes in its ebuilds over the seed SDK.
The change also introduces an option to consciously divert from the
above enforcement by use of command line parameters:
--stage1_overlay_ref <gitref> will check out coreos-overlay and use
<gitref> for stage 1 instead of the
local SDK's
/var/lib/gentoo/repos/coreos-overlay
--stage1_portage_ref <gitref> will check out portage-stable and use
<gitref> for stage 1 instead of the
local SDK's
/var/lib/gentoo/repos/gentoo
Signed-off-by: Thilo Fromm <thilo@kinvolk.io>
This change updates the stage1 SDK bootstrap build to use local
("known good") package ebuilds only, preventing updated package ebuilds
to apply in stage 1. This fixes SDK build breakage we observed when
upgrading core libraries like readline.
The change also removes the seed update from stage 1 as it should not
be needed anymore now that we postpone any package updates to stage 2.
The following package ebuild repos are used for stage 1:
- for portage-stable, we simply copy /var/gentoo/repos/gentoo
from the SDK root.
- coreos-overlay is more complicated since ebuilds are missing from
the SDK. So we grok the version the SDK was built with from
/mnt/host/source/.repo/manifests/default.xml
and then we create a local stage 1 clone of
https://github.com/kinvolk/coreos-overlay.git
in which we then check out the revision noted in the default mnifest.
Signed-off-by: Thilo Fromm <thilo@kinvolk.io>
ROOT_OVERLAY variable is defined in terms of TEMPDIR. The TEMPDIR
variable is set to an empty value by catalyst.sh, which the two
scripts import. So ROOT_OVERLAY always ended up being located in
toplevel directory (i.e. `/`). But the TEMPDIR variable gets a
meaningful value after calling the catalyst_init function, so define
the ROOT_OVERLAY after the function is called.
Before, we were relying on the toolchains job to build and upload
packages that were part of the SDK. With this change, all packages that
should be part of the SDK are built and uploaded by the SDK job. The
toolchains job only builds toolchain packages specific for the release.
This change includes several adjustments done to both the SDK and the
toolchains jobs to make this work:
* Make the SDK job build all cross toolchains, including Rust
* Stop building Rust in the toolchains job and use the one in the SDK
instead.
* In toolchain_util.sh: detect when the symlink folder for crossdev
packages is missing and run crossdev to create it during
update_chroot setup.
* Make it possible to build the SDK starting from stage 4 instead of
stage 1, to make the SDK building faster for PR branches / nightlies
(full build should still be done for releases / weeklies).
This variable was semi-deprecated ages ago so `version.txt` could follow
a similar variable naming pattern to `os-release`. Finally drop usage of
it here in favor of `$COREOS_VERSION`.
The version of repos.conf/coreos.conf that catalyst needs isn't valid
for normal SDK chroots and causes env-update to spew errors when it is
run prior to update_chroot which configures portage properly.
SDK tarballs have a .DIGESTS file but it is created by catalyst instead
of the upload_image function. In order to support plain GPG signing but
not avoid re-generating .DIGESTS we need to move that code out of
upload_image to a new function. upload_files shouldn't do it itself
because it is also used for portage binary packages which shouldn't be
signed (there is no point, nothing would verify the signatures).
- Remove custom COREOS_* attributes from /etc/lsb-release
- Move dev image logic to dev_image_util
For extra fun fix detection of local host URL for devserver.
- Remove weirdly verbose "DESCRIPTION" format.
- Add COREOS_RELEASE_BOARD back to /usr/share/coreos/release
This is mostly just so update_engine and gmerge report the correct
board name to devserver, informative-only on prod images.
- Remove version info from /etc/gentoo-release
- Switch from 'track' to 'group' terminology.
Right now there is some funky logic to either use a previous build as a
seed or the current SDK tarball if it happens to have been downloaded.
This is a bit confusing and doesn't work reliably since it is reasonable
for there to be neither a previous build or the current SDK available if
the SDK chroot was created some time ago. Fix this by using the new SDK
library and always use the latest SDK, downloading it if needed.
Sync up bootstrap_sdk with other tools by using the common upload
functions. As part of this refactor release_util a bit to provide a
truly generic upload function.
