The old chroot version system we inherited from Chromium OS always
assumes that a newly unpacked tarball is the latest and greatest but
since we version the SDK in the same way as target builds we can use
that version for these sorts of upgrade scripts and not make assumptions
about how late and great the starting tarball was.
The first upgrade script simply aborts to force the user to recreate
their chroot when moving from python 2.6 to 2.7.
Our SDK tarballs aren't compressed using pbzip2 so there is no advantage
to using pbzip2 to decompress them over bzip2, however lbzip2 does offer
a big advantage. Also trust that the portage config defines a valid
version of bzip2 since we have control over the tarball creation and can
make sure to always include required utilities.
The existing code seems to assume that the mounts inherited from the
system are private, the Linux default. However on our systems that
clearly isn't the case, all system mounts are set as shared. Considering
all of us have been have been seeing mounts leak out of the SDK despite
cros_sdk creating a new filesystem namespace via unshare I'm guessing
this is a systemd thing.
Instead force all system mounts to 'slave' mode in the SDK namespace so
global changes are still visible but no SDK mounts can leak out.
Already did this for catalyst builds but might as well do it for all.
With this competing builds on the same host should be a little
friendlier to each other.
Half of the current time is spent on calling locale-gen even when there is
nothing to be done (all locales already generated). Throw it into the bg
to unblock the main thread.
BUG=None
TEST=`cros_sdk` still works
TEST=`LANG=et_EE.UTF-8 cros_sdk` generates the new locale in the background
Change-Id: Ibe9a07bec60a59cab1cf4230358f7f8ff5b21c2e
Reviewed-on: https://gerrit.chromium.org/gerrit/58041
Reviewed-by: David James <davidjames@chromium.org>
Commit-Queue: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
All devpts mounts are actually shared, even if you do:
mount -t devpts none /dev/pts
mount -t devpts none /mnt/foo
mount -t devpts none /mnt/asdfasdf
These all provide the same data.
This is problematic because most distros mount their host devpts like so:
mount -t devpts devpts /dev/pts -o mode=620,gid=5
But when cros_sdk runs, it uses:
mount -t devpts none /dev/pts
We aren't specifying a mode/gid, so it ends up using the defaults, and
this resets the host devpts mount as well.
Since we've already assumed that the system has devpts available, it's
fine to also assume that the system has it mounted at /dev/pts and we
can simply bind mount it.
BUG=None
TEST=`cros_sdk` no longer messes up host perms on /dev/pts
Change-Id: Ib594fc5e47707f296d97ac1edce32659ed2b2273
Reviewed-on: https://gerrit.chromium.org/gerrit/48018
Reviewed-by: Steev Klimaszewski <threeway@gmail.com>
Reviewed-by: David James <davidjames@chromium.org>
Commit-Queue: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
I use a mount at src/build/images to stop image builds from repeatedly
filling up my SSD. The chroot needs to respect this.
TEST=cros_sdk
BUG=none
Change-Id: I5c7a26c3b4f263bd683d3a897e6edccb83187bda
Reviewed-on: https://gerrit.chromium.org/gerrit/47178
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Queue: Michael Spang <spang@chromium.org>
Tested-by: Michael Spang <spang@chromium.org>
Newer Gentoo builds have moved to /run which means /var/lock is a symlink
to /run/lock. But since that is an absolute symlink, it points outside of
the chroot which doesn't work for us. Use a stable path unrelated to the
chroot instead, but only with newer chroots.
We no longer have to worry about backwards compat because the code that
used to rely on this lock file (running sync processes) was punted a long
time ago.
BUG=chromium:218085
TEST=`cbuildbot chromiumos-sdk` passes
Change-Id: I38c6848dfb86386849050d7ccf3f90cbbe8e0e81
Reviewed-on: https://gerrit.chromium.org/gerrit/46231
Reviewed-by: David James <davidjames@chromium.org>
Commit-Queue: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
This patch installs "socat" and a proxy gateway script into
the chroot so that git can use a proxy to access "git://"
protocol urls. This is needed when performing builds from
behind a firewall that requires a proxy. The script reads
the proxy environment variables all_proxy (SOCKS),
https_proxy (CONNECT), and http_proxy (CONNECT), in order of
preference, and supports no_proxy as a whitelist of target
hosts that must NOT go through the proxy.
This also updates enter_chroot.sh to automatically use this
script as GIT_PROXY_COMMAND when it sees the proxy
environment variables set.
The "socat" program is added to hard-host-depends as a
separate patch. That handles socat installation in case of
building a chroot from scratch or upgrading.
The proxy-gw script is installed in the src/scripts/bin
directory which can be stably referenced within the chroot
as /mnt/host/source/src/scripts/bin/. The
"/mnt/host/source" portion of this path is obtained from the
CHROOT_TRUNK_DIR environment variable which is set to a
suitable value by preexisting logic in common.sh.
This change became necessary to unbreak builds behind
proxies with the recent addition of two ebuilds using
egit.eclass with repositories using git:// URLs.
Original patch by Paul Drews <paul.drews@intel.com>;
modified version by Josh Triplett <josh@joshtriplett.org>.
CQ-DEPENDS=I1b01bce6f3e6a562b87f748e61508d142af576d9
BUG=none
TEST=git clone git://nv-tegra.nvidia.com/tools/cbootimage.git
Change-Id: Ic7fc917d1aa24f408bef6f102b6458114dded694
Reviewed-on: https://gerrit.chromium.org/gerrit/41659
Tested-by: paul drews <paul.drews@intel.com>
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Queue: paul drews <paul.drews@intel.com>
In an Ubuntu Precise chroot on the Chromebook Pixel, /run/shm is a
symbolic link to /dev/shm, so bind-mounting /run/shm to /dev/shm
is really bind-mounting /dev/shm to itself, which causes a 'too many
levels of symbolic links' error. To fix this, we check for a symbolic
link prior to running this command.
BUG=none
TEST=cros_sdk no longer prints errors on Chromebook Pixel
Change-Id: Ib46cde2b4a0e00b69bd187488967e445b228ae80
Reviewed-on: https://gerrit.chromium.org/gerrit/45048
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Queue: David James <davidjames@chromium.org>
Tested-by: David James <davidjames@chromium.org>
If ~/.subversion doesn't exist, the code didn't run, but if it existed
there is no reason to re-create it, nor is it necessary to change its
permissions since they are inherited by the bind mount source.
However user_mkdir was trying to run chown as root which does not work
over NFS with root_squash or krb-nfs.
Therefore, the un-needed call to user_mkdir is removed.
(this is an issue because cros_sdk --replace does call this code path
multiple times).
BUG=None
TEST=Built the chroot, and the permission denied on 'install' went away.
Change-Id: I01e9a7baf51a99a96d790c9613e26e652379e6df
Reviewed-on: https://gerrit.chromium.org/gerrit/44880
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Queue: Marc MERLIN <merlin@chromium.org>
Tested-by: Marc MERLIN <merlin@chromium.org>
If our sdk has an /etc/mtab file already, then clobber it. This fixes
build problems where chromeos-base now installs /etc/mtab for us, but
the sdk build isn't expecting it leading to the error:
INFO cros_sdk:make_chroot: Running init_setup()...
ln: creating symbolic link `/b/cbuild/new-sdk-chroot/etc/mtab': File exists
Running ['/b/cbuild/src/scripts/sdk_lib/make_chroot.sh', '--stage3_path',
'/b/cbuild/built-sdk.tar.xz', '--chroot', '/b/cbuild/new-sdk-chroot',
'--cache_dir', '/b/cbuild/.cache', '--nousepkg'] failed!
BUG=None
TEST=`cros_sdk --chroot foo` still works
Change-Id: I539cf329e93e28534e6ff00577ce415d76918b85
Reviewed-on: https://gerrit.chromium.org/gerrit/43641
Reviewed-by: David James <davidjames@chromium.org>
Commit-Queue: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
As-is all of the various emerge wrapping scripts default to using
--getbinpkg whenever --usepkg is enabled. This means every single emerge
command made makes multiple synchronous HTTP requests to the upstream
binary package repository to get the latest package list. This gets
really frustrating when working remotely with limited network
connectivity. Using --usepkg with --nogetbinpkg will use locally cached
packages without making remote requests.
use the efunctions package for the /etc/init.d/functions.sh script
instead of backing up the old function.sh which doesn't work with the
new baselayout
we remove openrc which provides /etc/init.d/functions.sh. Unfortunatly
other things rely on this file. Stash it away in /tmp/ then restore it
for now.
Change-Id: I18a59e05ecdf08cc8a560b29049c8d25ac1bf5a3
This seems to be needed for acessing some of the chrome repositories.
Without it we get git clone hangs trying to sync.
BUG=chromium-os:38303
TEST=local entry into chroot
Change-Id: Ia68a6486022e8d230572bad0f9031c3e5d36197c
Reviewed-on: https://gerrit.chromium.org/gerrit/42140
Commit-Queue: Peter Mayo <petermayo@chromium.org>
Reviewed-by: Peter Mayo <petermayo@chromium.org>
Tested-by: Peter Mayo <petermayo@chromium.org>
After CL:39921, I get the following warning every time I enter the chroot:
ln: failed to create symbolic link `.../chroot/root/.boto': File exists
All bots get this error as well. This is caused because CL:39921, causes
~/trunk to no longer resolve outside the chroot, so it's invalid for processes
outside the chroot to try to resolve paths inside there. Fix cases where we do
this inside enter_chroot.sh.
BUG=chromium-os:37347
TEST=cros_sdk doesn't print warnings anymore.
Change-Id: Iaeb9b7407e12397bce1600bd51559be20f998fdf
Reviewed-on: https://gerrit.chromium.org/gerrit/41571
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Queue: David James <davidjames@chromium.org>
Tested-by: David James <davidjames@chromium.org>
Slipped past during rename of the chroot upgrade script from
49 to 50; name was slightly changed but full re-validation of the
rename wasn't done (thus the typo slipped past testing, and review).
Simplify the code via removal of invoking the upgrade script, instead
just doing the relevant commands (fixing chroot awareness issues in
the process).
BUG=None
TEST=manual cros_sdk invocation
Change-Id: I122de8b4cf7ec0845643e09e7919cbcdbd0bb79a
Reviewed-on: https://gerrit.chromium.org/gerrit/41202
Reviewed-by: Brian Harring <ferringb@chromium.org>
Tested-by: Brian Harring <ferringb@chromium.org>
Rather than having to find /home/${SUDO_USER:-${USER}}/trunk, instead
just look for /mnt/host/trunk (defined by common.sh as $CHROOT_TRUNK_DIR).
This simplifies code flow, and is a requirement for shoving chromite
into PYTHONPATH globally w/in the chroot.
BUG=chromium-os:37347
TEST=cros_sdk --replace; cros_sdk w/ chroot upgrade.
Change-Id: I9ee3e6556541a91193f49cbf74ffc5a8e090537f
Reviewed-on: https://gerrit.chromium.org/gerrit/39921
Tested-by: Brian Harring <ferringb@chromium.org>
Reviewed-by: David James <davidjames@chromium.org>
Over time, stale ssh agent dirs build up in /tmp. Have enter_chroot run
a simple rmdir to clean out any empty dirs. Since we mount over top the
dir, this shouldn't kill any valid mount points.
BUG=None
TEST=`cros_sdk` cleaned out empty ssh dirs in /tmp
Change-Id: Ib9f063f99db61825082818a39a39c5eb01f2d24e
Reviewed-on: https://gerrit.chromium.org/gerrit/39004
Reviewed-by: David James <davidjames@chromium.org>
Reviewed-by: Matt Tennant <mtennant@chromium.org>
Reviewed-by: Brian Harring <ferringb@chromium.org>
Commit-Ready: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
When running on NFS, the root user may not be able to access ~/.ssh and
~/.gitconfig, so it is necessary to fallback to SUDO_USER to access these
files.
To discourage users from using NFS homedirs, print warnings every time
cros_sdk is run with an NFS homedir.
BUG=chromium-os:36783
TEST=Try cros_sdk --replace and cros_sdk with and without NFS homedirs.
Change-Id: I4cdbceca485d3491656d6f743814da4ebcdd75ad
Reviewed-on: https://gerrit.chromium.org/gerrit/38953
Commit-Ready: David James <davidjames@chromium.org>
Reviewed-by: David James <davidjames@chromium.org>
Tested-by: David James <davidjames@chromium.org>
Explicitly build curl/openssl/git since the toolchain itself tries to
fetch over http with git.
BUG=None
TEST=`cros_sdk --bootstrap` works
TEST=`cbuildbot chromiumos-sdk` works
Change-Id: I50b3145732f8345d6ad6ada41325648cbea31b84
Reviewed-on: https://gerrit.chromium.org/gerrit/36995
Tested-by: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Han Shen <shenhan@chromium.org>
Tested-by: Han Shen <shenhan@chromium.org>
Commit-Ready: Mike Frysinger <vapier@chromium.org>
sudo takes 150ms per invocation on Goobuntu, and with 10 invocations in
enter_chroot.sh, this means that we're wasting a lot of time, every time
cros_sdk is invoked. Cutting these unnecessary invocations reduces the time
required to run enter_chroot.sh from 2.3s to 0.8s.
CL:36618 is the companion change that updates cros_sdk to invoke
sudo unshare -m prior to calling enter_chroot.sh.
Summary of changes:
1. Remove all calls to sudo and just run the commands directly.
- Remove the mount queue and any sudo_multi optimizations.
- Rename sudo_chroot -> bare_chroot because we don't run sudo anymore there.
- Remove code for validating sudo timestamp.
2. Allow the scripts to work as root:
- Ensure that files created by cros_sdk that previously were owned by the
user still are owned by the user (either using chown or cp -p).
- Use $SUDO_USER to find the user's account.
- Use $SUDO_HOME instead of $HOME to find the user's home dir.
- Remove outdated code for disabling automount on Lucid, which doesn't work
when run as root.
- Update code for calculating the user's git username to use sudo to switch
to the user. Also move it to make_chroot.sh so that this change doesn't
impact performance.
3. Cleanup
- Remove environment syncer process in favor of just syncing once when chroot
is entered.
- Remove teardown and instead rely on unshare to unmount the mounts. To make
sure that outside processes never notice the mounts, we use mount -n. This
also ensures that /etc/mtab never contains stale mounts.
- Remove path-overrides, since it is no longer needed.
BUG=chromium-os:35714, chromium-os:35679
TEST=Trybot runs.
CQ-DEPEND=CL:36618
Change-Id: I919a8aadb08fafde97348e8511573c28fdd47186
Reviewed-on: https://gerrit.chromium.org/gerrit/36619
Tested-by: David James <davidjames@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Ready: David James <davidjames@chromium.org>
Users sometimes want to run gclient inside the chroot, so we shouldn't
tell users that using it is a bad idea.
The original reason why this message was added is historical: Originally,
users had a newer version of SVN inside the chroot compared to on their
workstation, so if you ran SVN inside the chroot it would permanently upgrade
your working copy such that the version of SVN outside the chroot did not work
with it anymore. This isn't a problem anymore, so we can remove the message.
BUG=none
TEST=Run remote trybot runs of chromiumos-sdk
Change-Id: I7b82a5c94e29d5928f4bb296ae2d99cef397d365
Reviewed-on: https://gerrit.chromium.org/gerrit/36346
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Ready: David James <davidjames@chromium.org>
Tested-by: David James <davidjames@chromium.org>
The process of bootstrapping the chroot from sources was
failing for several reasons when run from behind a firewall
with proxies. The llvm build was failing due to inability
to checkout sources through subversion using the
subversion.eclass wrapper (the "normal" way to do this in
the ebuild environment). This was because the user's
subversion configuration (including proxy settings) was not
inherited from $HOME/.subversion into the in-chroot sandbox
used by subversion.eclass.
This change creates symbolic links in the subversion.eclass
sandboxes for host and target builds in the chroot to fix
any build that uses the normal subversion.eclass for
checkouts. The operation is done at enter_chroot time so
that it applies to both ordinary builds and chroot creation
(via early_enter_chroot).
BUG=none
TEST='cros_sdk --replace --enter' behind proxied firewall
Change-Id: I0af2128866bb95799dc07c728c75cf3f2a0af7a3
Reviewed-on: https://gerrit.chromium.org/gerrit/34291
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Ready: paul drews <paul.drews@intel.com>
Tested-by: paul drews <paul.drews@intel.com>
Building the chroot environment from sources using
"--bootstrap" currently runs into a circular dependency:
curl->openssl->git->curl
The openssl->git dependency comes indirectly from the fact
that the current version of openssl uses the "cros-workon"
ebuild package to assist in applying packages. The ebuild
system automatically and silently resolves this circular
dependency by reverting the openssl library to an earlier
version that does not use cros-workon based patching.
Unfortunately this older version of openssl has a bug that
causes it not to work when doing builds in a firewalled
environment: When curl (using this older version of openssl
library) attempts to fetch an "https" url, it authenticates
the target server against a bundle of certificate-authority
certificates it maintains. Finding the certificate fails
(although the validation succeeds if curl is told explicitly
what certificate to use). With the certificate not-found,
server authentication fails, the curl download fails, and
the build ultimately fails.
This patch breaks the circular dependency, allowing a
more-current version of openssl to be used in curl, making
the above build scenario work in a firewalled environment.
The circularity is broken by first building git without curl
support (and webdav that depends on curl). Then early
toolchain components up through and including curl are
built. This build of curl then uses a more up-to-date
version of openssl with the desired bug-fix. Once curl is
built, then git is re-built and re-installed with the
now-installed version of curl (re-)enabled.
BUG=None
TEST=create chroot with --bootstrap ; build_packages (behind firewall)
Change-Id: Iaa560fdb6623fcb73cde066a3b2bc2a342169c62
Reviewed-on: https://gerrit.chromium.org/gerrit/34292
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Ready: paul drews <paul.drews@intel.com>
Tested-by: paul drews <paul.drews@intel.com>
CL:33868 (7b6f377c58) introduced a
breakage in the "cros_sdk --replace --bootstrap" scenario.
The make_chroot.sh script invokes early_enter_chroot before
invoking init_setup. The chroot/etc/profiles.d directory is
created in init_setup, but the referenced change was
expecting to create a file in that directory in the context
of early_enter_chroot before the directory was created.
This led to a "no such file or directory" error when trying
to create the file.
This change does a "mkdir -p" of the referenced directory
before putting things in it in the context of
early_enter_chroot. The filename is also fixed to the name
expected elsewhere in the scripts.
BUG=none
TEST=cros_sdk --replace --bootstrap
Change-Id: I6ac0467117d7b0dd413695153469b367d56c256c
Reviewed-on: https://gerrit.chromium.org/gerrit/34958
Commit-Ready: Brian Harring <ferringb@chromium.org>
Reviewed-by: Brian Harring <ferringb@chromium.org>
Tested-by: Brian Harring <ferringb@chromium.org>
This is forced by cros_sdk; in conjunction w/ this,
drop --distfiles and mangle the chroot on during entrance
dropping a symlink in the old /var/cache/distfiles location
pointing to the new mounted cache_dir location.
Additionally, thread CHROMEOS_CACHEDIR down through the end.
Do this without relying on a version upgrade script- we can't
require they be run before entering, thus we exploit the fact
that cros_sdk explicitly forces a write lock to do the upgrade,
if we see the old form we know we can do the upgrade w/out
worrying about collisions.
CQ-DEPEND=CL:33871
BUG=chromium-os:34457
TEST=manual testing.
Change-Id: I6805266e3ec683f05d3ba615f9e8840642a28e48
Reviewed-on: https://gerrit.chromium.org/gerrit/33868
Commit-Ready: Brian Harring <ferringb@chromium.org>
Reviewed-by: Brian Harring <ferringb@chromium.org>
Tested-by: Brian Harring <ferringb@chromium.org>
enter_chroot.sh was not updating /etc/hosts from the out-of-chroot
environment. Make it do that.
BUG=None
TEST=locally
Change-Id: Ieaa337ae90dbc0700c42fa7e4b96faf12d3968cb
Reviewed-on: https://gerrit.chromium.org/gerrit/34226
Reviewed-by: David James <davidjames@chromium.org>
Commit-Ready: Ryan Cui <rcui@chromium.org>
Tested-by: Ryan Cui <rcui@chromium.org>
