This change updates the stage1 SDK bootstrap build to use local
("known good") package ebuilds only, preventing updated package ebuilds
to apply in stage 1. This fixes SDK build breakage we observed when
upgrading core libraries like readline.
The change also removes the seed update from stage 1 as it should not
be needed anymore now that we postpone any package updates to stage 2.
The following package ebuild repos are used for stage 1:
- for portage-stable, we simply copy /var/gentoo/repos/gentoo
from the SDK root.
- coreos-overlay is more complicated since ebuilds are missing from
the SDK. So we grok the version the SDK was built with from
/mnt/host/source/.repo/manifests/default.xml
and then we create a local stage 1 clone of
https://github.com/kinvolk/coreos-overlay.git
in which we then check out the revision noted in the default mnifest.
Signed-off-by: Thilo Fromm <thilo@kinvolk.io>
I have no idea how this thing worked before - the repos never were in
/usr/portage nor in /usr/local/portage… But the newer version of
portage seems to be pretty picky about the validity of repos location,
so fix them.
Setting the invalid CCACHE_ variables resulted in strange failure
in projects depending on meson, newer version like 0.55.3. For example
systemd build fails like the following errors:
```
* ACCESS DENIED: utimes: /mnt/host/source/ccache
* ACCESS DENIED: utimes: /mnt/host/source/ccache
F: utimes
S: deny
P: /mnt/host/source/ccache
A: /mnt/host/source/ccache
R: /mnt/host/source/ccache
C: ccache cc /build/amd64-usr/var/tmp/portage/sys-apps/systemd-246/work/systemd-246-abi_x86_64.amd64/meson-private/sanitycheckc.c -o /build/amd64-usr/var/tmp/portage/sys-apps/systemd-246/work/systemd-246-abi_x86_64.amd64/meson-private/sanitycheckc.exe -O1 -pipe -pipe -D_FILE_OFFSET_BITS=64
```
We should not set up ccache at all, as it has been already disabled in
coreos-overlay repo.
The default update seed command does only specify gcc which leads to
an error because »The short ebuild name "gcc" is ambiguous«.
Choose the standard package name instead of the cross compiler packages
which are only known to emerge because we build them as part of an SDK
release now.
Install `virtual/rust` to avoid version conflicts that happen in case of
rust versions in the SDK being different from those in the new ebuilds.
`/usr/share/catalyst/targets/stage1/stage1-chroot.sh` installs gcc and
its dependencies, including `dev-lang/rust`, while `virtual/rust` does
not get updated. That results in version conflicts between
`virtual/rust` and `dev-lang/rust`. To avoid such an issue, we should
update also `virtual/rust` when building stage1. Since `virtual/rust`
automatically pulls in `dev-lang/rust`, we do not need to explicitly
specify `dev-lang/rust` here.
The Perl update will break SDK bootstrapping during seed update, so
disable it again. This can be reverted after bumping the SDK to a
version that includes the new Perl.
It provides no value when it works, and it's randomly causing
failures to build toolchains due to permissions problems after
certain releases. This also requires taking it out of FEATURES in
the portage profile (which is the SDK profile by default).
Test Jenkins runs of SDK and toolchains jobs both ran in the same
time as with ccache enabled.
workaround for bootstrap_sdk on an Ubuntu host where /dev/shm is a
symlink to /run/shm. Since we mount the hosts /dev (for losetup) this
interferes with building python 2.7. The workaround is to disable the
/dev/shm during python builds. A longer term fix would be to not mount
the hosts /dev. Thanks for marineam for suggesting the fix on IRC.
Instead of patching portage to support the `disabled` flag now we just
patch it to leave the `[gentoo]` section out of the default repos.conf.
Follow up to 585275b268
This variable was semi-deprecated ages ago so `version.txt` could follow
a similar variable naming pattern to `os-release`. Finally drop usage of
it here in favor of `$COREOS_VERSION`.
Now that ccache is turned on by default in the profile portage complains
a lot if ccache isn't actually installed, sleeping 5 seconds for each
error message. Since pkgcache is in use ccache isn't going to make that
much of a difference but getting rid of those 5 second sleeps will. :)
This is required for the eventual removal of `$PORTDIR` and
`$PORTDIR_OVERLAY` and ensures toolchain rebuilds/updates with
`./build_packages --nousepkg` don't erroniously try to use ebuilds from
`/usr/portage` inside of the SDK.
In order to fix up the build_toolchains script the crossdev overlay
needs to be setup properly, previously only setup_board did it.
Overall silences a lot of warnings and fixes an issue with crossdev:
/usr/bin/emerge-wrapper: line 48: /eclass/toolchain-funcs.eclass: No such file or directory
/usr/bin/emerge-wrapper: line 49: tc-arch: command not found
Disable ccache as it is causing issues in other builds so disable it
everywhere to be safe. Disable the autoresume feature because our build
process doesn't actually make use of it.
Adding the update step appears to break permissions on the distfiles
directory. Ensure the portage user is correct and set the permissions on
directories it needs to write to in advance.
When bootstrapping a SDK we need to update GCC dependencies to ensure
the GCC built for stage1 is linked against the same library versions as
those that are included in the stage1. Without this updating the mpc
library just results in a broken stage1.
This doesn't make things go faster and I am suspicious that it makes
things worse. For example /etc/xml/catalog winds up empty from time to
time and I wonder if this locking is related to that.
Right now there is some funky logic to either use a previous build as a
seed or the current SDK tarball if it happens to have been downloaded.
This is a bit confusing and doesn't work reliably since it is reasonable
for there to be neither a previous build or the current SDK available if
the SDK chroot was created some time ago. Fix this by using the new SDK
library and always use the latest SDK, downloading it if needed.
A number of places refer to these paths and that number is going to
grow. Since the standard pattern is to use environment variables for
commonly used paths it is time to add ones for these:
REPO_CACHE_DIR
REPO_MANIFESTS_DIR
Use 2*CPUs for the target load average but add load average throttling
to emerge in addition to make. Also work around how catalyst sets
FEATURES so we can disable extra locking for hopefully faster builds.
