Currently the os/sdk and os/toolchains job perform a chroot update whose
results are immediately discarded because the rest of the build uses a fresh
chroot and catalyst. Towards the end of a release period this can extend the
build time by about an hour (longer if rust is involved).
Introduce a `--setuponly` flag that bails after the chroot configuration, and
the skips chroot update.
Signed-off-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>
This updates the default settings in build scripts to use
https://mirror.release.flatcar-linux.net/
instead of the google storage bucket if no binhost or FLATCAR_DEV_BUILDS
is specified.
Defaults are updated for
* update_chroot (runs at SDK initialisation time)
* setup_board (creates /boards/[ARCH]/) chroots
* the development container
* set_version
The scripts that invoked `cros_workon` without specifying a path to
the script were not calling the internal `cros_workon` directly, but
rather a copy installed in `/usr/bin/cros_workon`.
`/usr/bin/cros_workon` comes from the `coreos-base/cros-devutil` and
is a wrapper script that sources `common.sh` file to figure the
location of the `scripts` and finally invokes the internal
`cros_workon`. Curious thing is that the sourced `common.sh` comes
from the `/usr/lib/crosutils` directory and contents of the directory
come from the `dev-util/crosutils` package. And that `common.sh` is
different from the one in the scripts directory, but fortunately the
part that detects the path to the `scripts` directory is the same. I'm
not sure where where exactly the copy of `common.sh` in
`/usr/lib/crosutils` comes from - likely from somewhere in
`https://chromium.googlesource.com/chromiumos/platform/crosutils`.
Just cut the middle layers and call the internal copy of `cros_workon`
directly.
The `--jobs` parameter that some scripts defined was not used anywhere
in jenkins or mantle. So the value of the parameter always ended up
being equal to `${NUM_JOBS}` set by `common.sh`. Also, even if the
`--jobs` parameter was used for some script, that script usually
didn't forward the jobs value to other scripts, so the other scripts
ended up using `${NUM_JOBS}` again. Also, the `${FLAGS_jobs}` variable
was used by some functions in the build library, and those functions
were sometimes invoked by scripts that didn't define the
`${FLAGS_jobs}` variable. It is tedious to track which script should
actually define the parameter, and where it should be forwarded.
Just get rid of this half-working pretense. If you want to affect how
many jobs `emerge` uses, export the `NUM_JOBS` environment variable
before calling any script.
For `EMERGE_FLAGS` and `REBUILD_FLAGS` we unconditionally specify the
`--jobs` flag's value to `${NUM_JOBS}` because they are passed to
`emerge`. On the other hand we drop the `--jobs` parameter from the
`UPDATE_ARGS` variable, because this variable passed to `setup_board`
or `update_chroot`, which don't have this flag any more.
Setting the invalid CCACHE_ variables resulted in strange failure
in projects depending on meson, newer version like 0.55.3. For example
systemd build fails like the following errors:
```
* ACCESS DENIED: utimes: /mnt/host/source/ccache
* ACCESS DENIED: utimes: /mnt/host/source/ccache
F: utimes
S: deny
P: /mnt/host/source/ccache
A: /mnt/host/source/ccache
R: /mnt/host/source/ccache
C: ccache cc /build/amd64-usr/var/tmp/portage/sys-apps/systemd-246/work/systemd-246-abi_x86_64.amd64/meson-private/sanitycheckc.c -o /build/amd64-usr/var/tmp/portage/sys-apps/systemd-246/work/systemd-246-abi_x86_64.amd64/meson-private/sanitycheckc.exe -O1 -pipe -pipe -D_FILE_OFFSET_BITS=64
```
We should not set up ccache at all, as it has been already disabled in
coreos-overlay repo.
The dev build SDKs are not in $FLATCAR_DEV_BUILDS/sdk but published under
$FLATCAR_DEV_BUILDS/developer/sdk.
Add an environment variable to specify where the SDK is to be found
but default to $FLATCAR_DEV_BUILDS/sdk if it is not specified.
From Jenkins this variable is exported as DOWNLOAD_ROOT_SDK.
Normally toolchains packages are prevented from upgrading. This
drops that restriction and explicitly removes old versions so that
conflicting tool profiles are not accidentally used.
This reverts commit 20975049b3.
This omits the toolchain packages' version-pinning flag for the
binutils package while it is being upgraded. It also removes older
versions installed in parallel that cause unwanted rebuilds.
When stable has the upgraded version, this can be reverted.
This seems to fix the ccache permission issues `update_chroot` hits
while building ninja.
The erroneous files were created as root:portage, so a umask of 002
should let other portage group members share them, which seems entirely
reasonable.
Instead of patching portage to support the `disabled` flag now we just
patch it to leave the `[gentoo]` section out of the default repos.conf.
Follow up to 585275b268
This variable was semi-deprecated ages ago so `version.txt` could follow
a similar variable naming pattern to `os-release`. Finally drop usage of
it here in favor of `$COREOS_VERSION`.
When --usepkgonly is mixed with the right update flags it will
re-install any binary packages that have a newer build. Since the full
set of SDK and board packages are rebuilt quite often this leads to
excessive reinstalling.
Before attempting to do @preserved-rebuild to fix linked against old
libraries remove any packages that no longer have corresponding ebuilds,
making them impossible to rebuild. This uses `--depclean`'s secondary
meaning: `--unmerge` but only remove packages without dependencies.
Rebuilds packages that are linked against old libraries that have been
upgraded or removed from the system. Skipping this can lead to shared
library checks looking ok in the build root but then built images have
broken library dependencies.
The distfiles cache is always under .cache in the repo tree but there is
a lot of extra logic to make that configurable along with compatibility
symlinks for previous locations. Just yank it all out.
A step in reducing the amount of initialization code required: drop
needless symlinks under /usr/local/portage to the portage trees. Just
configure portage to point directly at the source instead. Only crossdev
remains in that location because it is a locally managed overlay.
This is required for the eventual removal of `$PORTDIR` and
`$PORTDIR_OVERLAY` and ensures toolchain rebuilds/updates with
`./build_packages --nousepkg` don't erroniously try to use ebuilds from
`/usr/portage` inside of the SDK.
In order to fix up the build_toolchains script the crossdev overlay
needs to be setup properly, previously only setup_board did it.
Overall silences a lot of warnings and fixes an issue with crossdev:
/usr/bin/emerge-wrapper: line 48: /eclass/toolchain-funcs.eclass: No such file or directory
/usr/bin/emerge-wrapper: line 49: tc-arch: command not found
Normally Gentoo expects moving between major GCC releases to be a manual
step. In our case we want this to always be automatic, otherwise the GCC
version won't be switched at all.
Using parallel_emerge has been disabled by default for all commands
except build_image for quite a while now, build_image kept it just
because it was still a bit faster than normal emerge. Keeping
parallel_emerge complicates future changes to build_image so it needs to
drop it entirely. Since that means nothing uses it by default we might
as well just rip out support for it entirely.
To behave more like setup_board/build_packages update_chroot should
fully configure portage to make sure everything is accurate.
Now binhosts are defined in make.conf.host_setup so the static config in
coreos-overlays doesn't need to refer to version.txt. setup_board
already made this change in 7a43a07f.
Define path locations to reduce dependency between static configs in
coreos-overlays and the behavior of the scripts repo. Spreading
configuration across two repos makes everything harder to understand.
Eventually everything should either be defined in profiles in
coreos-overlays or minimal auto-generated config files here in scripts.
The latest release of portage uses repos.conf, so generate that
file when making the chroot. The crossdev overlay also needed
repo_name and layout.conf files.
- Remove custom COREOS_* attributes from /etc/lsb-release
- Move dev image logic to dev_image_util
For extra fun fix detection of local host URL for devserver.
- Remove weirdly verbose "DESCRIPTION" format.
- Add COREOS_RELEASE_BOARD back to /usr/share/coreos/release
This is mostly just so update_engine and gmerge report the correct
board name to devserver, informative-only on prod images.
- Remove version info from /etc/gentoo-release
- Switch from 'track' to 'group' terminology.
I would like to phase out parallel_emerge so disable it for all commands
other than build_image which is the only one that shows a noticeable
benefit from it (~2 min with --fast, ~3 min with --nofast).
Soon nss-usrfiles will be required to resolve users and groups properly.
To avoid potentially breaking during the transition we need to make sure
the package is installed early during the chroot upgrade process.
