The dev build SDKs are not in $FLATCAR_DEV_BUILDS/sdk but published under
$FLATCAR_DEV_BUILDS/developer/sdk.
Add an environment variable to specify where the SDK is to be found
but default to $FLATCAR_DEV_BUILDS/sdk if it is not specified.
From Jenkins this variable is exported as DOWNLOAD_ROOT_SDK.
We were chowning the host directory, not the one in the chroot.
Host gpg >= 2.1.13 puts the gpg-agent socket in /run/user/UID/gnupg,
which is bind-mounted into the chroot, but the SDK gpg was ignoring it
because /run/user/UID was not owned by UID. This broke tag signing with
YubiKeys.
COREOS_BUILD_ID is set to a default value in common.sh if unset in the
environment. When entering the chroot this default value should not then
get promoted into the environment. Doing so causes catalyst to re-use
stale builds and multiple build_image runs to conflict with each other.
Commit 09851b84 didn't do a recursive bind by mistake, so if the host
system has anything mounted under the chroot directory for some reason
the bind would hide those mounts. Recursive ensures existing mounts
remain exposed as they did before.
The path of $GNUPGHOME outside the chroot may not really make sense
inside the chroot. Although that's probably not a big deal there's no
need to keep the outside value. Instead just bind it to the usual spot.
When running under jenkins the $GNUPGHOME may be located under the
current build directory instead of $HOME to avoid conflicting with other
jobs on the same build host.
The distfiles cache is always under .cache in the repo tree but there is
a lot of extra logic to make that configurable along with compatibility
symlinks for previous locations. Just yank it all out.
The version of repos.conf/coreos.conf that catalyst needs isn't valid
for normal SDK chroots and causes env-update to spew errors when it is
run prior to update_chroot which configures portage properly.
A step in reducing the amount of initialization code required: drop
needless symlinks under /usr/local/portage to the portage trees. Just
configure portage to point directly at the source instead. Only crossdev
remains in that location because it is a locally managed overlay.
This code is not applicable to us, it predates CoreOS and is a weird
thing for common.sh to be doing as well. Instead always define
CHROOT_TRUNK_DIR to /mnt/host/source, create ~/trunk in make_chroot.
Currently building images on older kernels will fail because mkfs.btrfs
enables an incompatible feature 'extref' by default. We never really
made this requirement explicit and the SDK in general has continued to
maintain compatibility with older kernels. Make the requirement explicit
so users will get errors quicker and there is a clear line for what
kernel features can be used in the SDK.
Newer git ebuilds have decided that the "git-prompt" script isn't really
bash completion so stopped installing it via that mechanism. Instead it
installed it started installing it in /usr/share/docs which gets
compressed by default and the path is based on ebuild version. The path
changed again in 1.9.3 to /usr/share/git and didn't compress it so that
makes it actually possibly usable but 1.9.3 or later isn't stable yet.
We can re-enable it the next time git gets updated but not worth fussing
over the current brokenness right now.
Using parallel_emerge has been disabled by default for all commands
except build_image for quite a while now, build_image kept it just
because it was still a bit faster than normal emerge. Keeping
parallel_emerge complicates future changes to build_image so it needs to
drop it entirely. Since that means nothing uses it by default we might
as well just rip out support for it entirely.
Previously /etc/os-release was installed both by set_lsb_release and
the baselayout package. Now it is only installed by set_lsb_release but
when baselayout is upgraded it removes /etc/os-release. So the first
update_chroot works but the second detects the chroot's version
incorrectly and tries to apply the one time updates in this directory.
Both of them are very old so we can just delete them. The second run
will now fix up /etc/os-release and we can all move on and be happy.
The host system's PATH may not be match the one required by the SDK.
When going through the enter_chroot script it gets reset because bash is
invoked as a login shell but this doesn't happen when using the plain
old chroot command.
Fixes https://github.com/coreos/scripts/pull/290
There is no need to arbitrarily bind mount all of the host system's /run
into the chroot. In fact this causes issues when the host system's /run
isn't set up in a way this script anticipates. Namely the user runtime
directory in /run/usr/$UID is another tmpfs mount on my system, leaving
the underlying directory node that is bind-mounted in with the wrong
ownership. Behave a little more like a responsible container and use a
fresh /run but continue binding /run/shm for whatever versions of Ubuntu
that depended on that behavior. Not strictly needed but go ahead and
create the user runtime directory with the correct permissions.
- Don't copy known_hosts if it doesn't exist.
- Don't bother with copying *.pub, not sure what that was for.
- Don't rewrite .ssh/config to remove internal Google ssh options.
The main case here is /etc/hosts does not exist on CoreOS. In the
process combine related and duplicate code. Setting the timezone now
happens in entire_chroot like hosts and resolv.conf. Don't bother with
setting a default UTC time zone, that is already the default.
- Automated builds drop SDK and binary packages into
gs://builds.developer.core-os.net/ and the new download URL is
http://builds.developer.core-os.net/ (COREOS_DEV_BUILDS)
- Change default upload path to gs://users.developer.core-os.net/ for
misc developer builds. Official builds go elsewhere and will just be
configured in buildbot/jenkins so some COREOS_OFFICIAL stuff is gone.
- Automated builds of images go to a private bucket,
gs://builds.release.core-os.net which later gets copied to
gs://alpha.release.core-os.net and friends by core_promote.
To behave more like setup_board/build_packages update_chroot should
fully configure portage to make sure everything is accurate.
Now binhosts are defined in make.conf.host_setup so the static config in
coreos-overlays doesn't need to refer to version.txt. setup_board
already made this change in 7a43a07f.
Define path locations to reduce dependency between static configs in
coreos-overlays and the behavior of the scripts repo. Spreading
configuration across two repos makes everything harder to understand.
Eventually everything should either be defined in profiles in
coreos-overlays or minimal auto-generated config files here in scripts.
For the most part this doesn't influence anything. The one exception is
the custom configuration for using curl is dropped, just rely on the
portage defaults. It appears curl was only used to work around a wget
issue with Google's internal SSL certificates. We care not. :)
The commands useradd/usermod will silently skip adding users to
secondary groups that are not in /etc/group. The idea being that the
tools should not create groups that conflict with existing LDAP/NIS
groups but why trying to do so isn't a fatal error I don't know.
Overall the code is rather complicated and tries to modify instead of
add when possible to allow running the SDK as the 'core' user. To keep
things simple gut this code, make the 'core' user special, and add
secondary groups via the 'gpasswd' command so that errors are reported
instead of silently ignored.
One functional change: the default groups have changed to kvm and
portage. The old list excluded kvm and included lots of extra cruft.
Make it possible for other scripts to share the same value for our
release repository and equally easy to override with a custom value.
Also allow setting the root from the command line in addition to the
environment. Usually --upload_root is better to use than --upload_path.
This makes it possible to toggle parallel_emerge just as other scripts
do. In other scripts update the help string to be more specific, the
--jobs option can be used to control parallelism.
Instead of handling toolchain packages in make_chroot and telling
update_chroot to skip the toolchains just depend on update_chroot to do
it properly. Reduces our code duplication by a tiny but worthwhile bit.
The current logic for downloading SDK tarballs is in cros_sdk and
written in python which isn't super convenient for re-using in the rest
of our shell scripts. This is a start of rewriting that logic into a
re-usable library but does not yet replace the functionality in cros_sdk.
We've had trouble with eclean and equery vanishing in our SDKs from time
to time. Although I don't know the root cause it seemed to be some
confusion in the ebuild environment, perhaps a mis-match between the
eclasses, profiles, and ebuilds. Updating all of those seemed to resolve
the issue and to make sure other environments are ok force a re-install
of portage and gentoolkit to clean things up.
