When running the build_packages script, one can encounter an error such as
'Error fetching binhost package info from.' This pertains to SDK packages (not
board packages). Since we have transitioned to the SDK container, the SDK
packages are no longer published independently from the container image.
The `dev-python/setuptools_scm` was renamed in first quarter of 2023
to `dev-python/setuptools-scm` (the underline was replaced with a
dash). To be safe, check for blockers with the new package name, just
in case.
That way we can see a report of what emerge is going to do and the
status of the use flags for the installed packages. The downside is
that we are going to have reports about using deprecated and
unsupported profile in even more places.
Emerge flags are cryptic in general, but short flags even more so, so
expand them. While at it, I noticed some places where bash arrays
could be used, so convert those places too.
Normally `ln -sf path/to/target at/name` will create a symlink at
`at/name` that points to `path/to/target`. But if `at/name` already
exists and is a directory or a symlink to some other directory, then
this command will create a symlink at `at/name/target` pointing to
`path/to/target`. There is an ambiguity between 1st and 3rd form of
`ln` (please refer to `man ln` for the available invocation forms). It
can be disambiguated by using the `-T` flag to force the 1st form.
In our case, if `/etc/portage/patches` symlink already existed and was
pointing to `<coreos-overlay>/coreos/user-patches`, we ended up with a
useless symlink at `<coreos-overlay>/coreos/user-patches/user-patches`
pointing to `<coreos-overlay>/coreos/user-patches`.
When setting up portage configuration, we set up a symlink in
${ROOT}/etc/portage/patches that points to the coreos/user-patches
directory inside the coreos-overlay. That way, we can add our custom
patches to coreos-overlay without the need for moving the packages
from portage-stable to coreos-overlay only to apply an extra patch.
Currently the os/sdk and os/toolchains job perform a chroot update whose
results are immediately discarded because the rest of the build uses a fresh
chroot and catalyst. Towards the end of a release period this can extend the
build time by about an hour (longer if rust is involved).
Introduce a `--setuponly` flag that bails after the chroot configuration, and
the skips chroot update.
Signed-off-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>
This updates the default settings in build scripts to use
https://mirror.release.flatcar-linux.net/
instead of the google storage bucket if no binhost or FLATCAR_DEV_BUILDS
is specified.
Defaults are updated for
* update_chroot (runs at SDK initialisation time)
* setup_board (creates /boards/[ARCH]/) chroots
* the development container
* set_version
The scripts that invoked `cros_workon` without specifying a path to
the script were not calling the internal `cros_workon` directly, but
rather a copy installed in `/usr/bin/cros_workon`.
`/usr/bin/cros_workon` comes from the `coreos-base/cros-devutil` and
is a wrapper script that sources `common.sh` file to figure the
location of the `scripts` and finally invokes the internal
`cros_workon`. Curious thing is that the sourced `common.sh` comes
from the `/usr/lib/crosutils` directory and contents of the directory
come from the `dev-util/crosutils` package. And that `common.sh` is
different from the one in the scripts directory, but fortunately the
part that detects the path to the `scripts` directory is the same. I'm
not sure where where exactly the copy of `common.sh` in
`/usr/lib/crosutils` comes from - likely from somewhere in
`https://chromium.googlesource.com/chromiumos/platform/crosutils`.
Just cut the middle layers and call the internal copy of `cros_workon`
directly.
The `--jobs` parameter that some scripts defined was not used anywhere
in jenkins or mantle. So the value of the parameter always ended up
being equal to `${NUM_JOBS}` set by `common.sh`. Also, even if the
`--jobs` parameter was used for some script, that script usually
didn't forward the jobs value to other scripts, so the other scripts
ended up using `${NUM_JOBS}` again. Also, the `${FLAGS_jobs}` variable
was used by some functions in the build library, and those functions
were sometimes invoked by scripts that didn't define the
`${FLAGS_jobs}` variable. It is tedious to track which script should
actually define the parameter, and where it should be forwarded.
Just get rid of this half-working pretense. If you want to affect how
many jobs `emerge` uses, export the `NUM_JOBS` environment variable
before calling any script.
For `EMERGE_FLAGS` and `REBUILD_FLAGS` we unconditionally specify the
`--jobs` flag's value to `${NUM_JOBS}` because they are passed to
`emerge`. On the other hand we drop the `--jobs` parameter from the
`UPDATE_ARGS` variable, because this variable passed to `setup_board`
or `update_chroot`, which don't have this flag any more.
Setting the invalid CCACHE_ variables resulted in strange failure
in projects depending on meson, newer version like 0.55.3. For example
systemd build fails like the following errors:
```
* ACCESS DENIED: utimes: /mnt/host/source/ccache
* ACCESS DENIED: utimes: /mnt/host/source/ccache
F: utimes
S: deny
P: /mnt/host/source/ccache
A: /mnt/host/source/ccache
R: /mnt/host/source/ccache
C: ccache cc /build/amd64-usr/var/tmp/portage/sys-apps/systemd-246/work/systemd-246-abi_x86_64.amd64/meson-private/sanitycheckc.c -o /build/amd64-usr/var/tmp/portage/sys-apps/systemd-246/work/systemd-246-abi_x86_64.amd64/meson-private/sanitycheckc.exe -O1 -pipe -pipe -D_FILE_OFFSET_BITS=64
```
We should not set up ccache at all, as it has been already disabled in
coreos-overlay repo.
The dev build SDKs are not in $FLATCAR_DEV_BUILDS/sdk but published under
$FLATCAR_DEV_BUILDS/developer/sdk.
Add an environment variable to specify where the SDK is to be found
but default to $FLATCAR_DEV_BUILDS/sdk if it is not specified.
From Jenkins this variable is exported as DOWNLOAD_ROOT_SDK.
Normally toolchains packages are prevented from upgrading. This
drops that restriction and explicitly removes old versions so that
conflicting tool profiles are not accidentally used.
This reverts commit 20975049b3.
This omits the toolchain packages' version-pinning flag for the
binutils package while it is being upgraded. It also removes older
versions installed in parallel that cause unwanted rebuilds.
When stable has the upgraded version, this can be reverted.
This seems to fix the ccache permission issues `update_chroot` hits
while building ninja.
The erroneous files were created as root:portage, so a umask of 002
should let other portage group members share them, which seems entirely
reasonable.
Instead of patching portage to support the `disabled` flag now we just
patch it to leave the `[gentoo]` section out of the default repos.conf.
Follow up to 585275b268
This variable was semi-deprecated ages ago so `version.txt` could follow
a similar variable naming pattern to `os-release`. Finally drop usage of
it here in favor of `$COREOS_VERSION`.
When --usepkgonly is mixed with the right update flags it will
re-install any binary packages that have a newer build. Since the full
set of SDK and board packages are rebuilt quite often this leads to
excessive reinstalling.
Before attempting to do @preserved-rebuild to fix linked against old
libraries remove any packages that no longer have corresponding ebuilds,
making them impossible to rebuild. This uses `--depclean`'s secondary
meaning: `--unmerge` but only remove packages without dependencies.
Rebuilds packages that are linked against old libraries that have been
upgraded or removed from the system. Skipping this can lead to shared
library checks looking ok in the build root but then built images have
broken library dependencies.
The distfiles cache is always under .cache in the repo tree but there is
a lot of extra logic to make that configurable along with compatibility
symlinks for previous locations. Just yank it all out.
A step in reducing the amount of initialization code required: drop
needless symlinks under /usr/local/portage to the portage trees. Just
configure portage to point directly at the source instead. Only crossdev
remains in that location because it is a locally managed overlay.
This is required for the eventual removal of `$PORTDIR` and
`$PORTDIR_OVERLAY` and ensures toolchain rebuilds/updates with
`./build_packages --nousepkg` don't erroniously try to use ebuilds from
`/usr/portage` inside of the SDK.
In order to fix up the build_toolchains script the crossdev overlay
needs to be setup properly, previously only setup_board did it.
Overall silences a lot of warnings and fixes an issue with crossdev:
/usr/bin/emerge-wrapper: line 48: /eclass/toolchain-funcs.eclass: No such file or directory
/usr/bin/emerge-wrapper: line 49: tc-arch: command not found
Normally Gentoo expects moving between major GCC releases to be a manual
step. In our case we want this to always be automatic, otherwise the GCC
version won't be switched at all.
