As-is all of the various emerge wrapping scripts default to using
--getbinpkg whenever --usepkg is enabled. This means every single emerge
command made makes multiple synchronous HTTP requests to the upstream
binary package repository to get the latest package list. This gets
really frustrating when working remotely with limited network
connectivity. Using --usepkg with --nogetbinpkg will use locally cached
packages without making remote requests.
use the efunctions package for the /etc/init.d/functions.sh script
instead of backing up the old function.sh which doesn't work with the
new baselayout
we remove openrc which provides /etc/init.d/functions.sh. Unfortunatly
other things rely on this file. Stash it away in /tmp/ then restore it
for now.
Change-Id: I18a59e05ecdf08cc8a560b29049c8d25ac1bf5a3
This seems to be needed for acessing some of the chrome repositories.
Without it we get git clone hangs trying to sync.
BUG=chromium-os:38303
TEST=local entry into chroot
Change-Id: Ia68a6486022e8d230572bad0f9031c3e5d36197c
Reviewed-on: https://gerrit.chromium.org/gerrit/42140
Commit-Queue: Peter Mayo <petermayo@chromium.org>
Reviewed-by: Peter Mayo <petermayo@chromium.org>
Tested-by: Peter Mayo <petermayo@chromium.org>
After CL:39921, I get the following warning every time I enter the chroot:
ln: failed to create symbolic link `.../chroot/root/.boto': File exists
All bots get this error as well. This is caused because CL:39921, causes
~/trunk to no longer resolve outside the chroot, so it's invalid for processes
outside the chroot to try to resolve paths inside there. Fix cases where we do
this inside enter_chroot.sh.
BUG=chromium-os:37347
TEST=cros_sdk doesn't print warnings anymore.
Change-Id: Iaeb9b7407e12397bce1600bd51559be20f998fdf
Reviewed-on: https://gerrit.chromium.org/gerrit/41571
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Queue: David James <davidjames@chromium.org>
Tested-by: David James <davidjames@chromium.org>
Slipped past during rename of the chroot upgrade script from
49 to 50; name was slightly changed but full re-validation of the
rename wasn't done (thus the typo slipped past testing, and review).
Simplify the code via removal of invoking the upgrade script, instead
just doing the relevant commands (fixing chroot awareness issues in
the process).
BUG=None
TEST=manual cros_sdk invocation
Change-Id: I122de8b4cf7ec0845643e09e7919cbcdbd0bb79a
Reviewed-on: https://gerrit.chromium.org/gerrit/41202
Reviewed-by: Brian Harring <ferringb@chromium.org>
Tested-by: Brian Harring <ferringb@chromium.org>
Rather than having to find /home/${SUDO_USER:-${USER}}/trunk, instead
just look for /mnt/host/trunk (defined by common.sh as $CHROOT_TRUNK_DIR).
This simplifies code flow, and is a requirement for shoving chromite
into PYTHONPATH globally w/in the chroot.
BUG=chromium-os:37347
TEST=cros_sdk --replace; cros_sdk w/ chroot upgrade.
Change-Id: I9ee3e6556541a91193f49cbf74ffc5a8e090537f
Reviewed-on: https://gerrit.chromium.org/gerrit/39921
Tested-by: Brian Harring <ferringb@chromium.org>
Reviewed-by: David James <davidjames@chromium.org>
Over time, stale ssh agent dirs build up in /tmp. Have enter_chroot run
a simple rmdir to clean out any empty dirs. Since we mount over top the
dir, this shouldn't kill any valid mount points.
BUG=None
TEST=`cros_sdk` cleaned out empty ssh dirs in /tmp
Change-Id: Ib9f063f99db61825082818a39a39c5eb01f2d24e
Reviewed-on: https://gerrit.chromium.org/gerrit/39004
Reviewed-by: David James <davidjames@chromium.org>
Reviewed-by: Matt Tennant <mtennant@chromium.org>
Reviewed-by: Brian Harring <ferringb@chromium.org>
Commit-Ready: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
When running on NFS, the root user may not be able to access ~/.ssh and
~/.gitconfig, so it is necessary to fallback to SUDO_USER to access these
files.
To discourage users from using NFS homedirs, print warnings every time
cros_sdk is run with an NFS homedir.
BUG=chromium-os:36783
TEST=Try cros_sdk --replace and cros_sdk with and without NFS homedirs.
Change-Id: I4cdbceca485d3491656d6f743814da4ebcdd75ad
Reviewed-on: https://gerrit.chromium.org/gerrit/38953
Commit-Ready: David James <davidjames@chromium.org>
Reviewed-by: David James <davidjames@chromium.org>
Tested-by: David James <davidjames@chromium.org>
Explicitly build curl/openssl/git since the toolchain itself tries to
fetch over http with git.
BUG=None
TEST=`cros_sdk --bootstrap` works
TEST=`cbuildbot chromiumos-sdk` works
Change-Id: I50b3145732f8345d6ad6ada41325648cbea31b84
Reviewed-on: https://gerrit.chromium.org/gerrit/36995
Tested-by: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Han Shen <shenhan@chromium.org>
Tested-by: Han Shen <shenhan@chromium.org>
Commit-Ready: Mike Frysinger <vapier@chromium.org>
sudo takes 150ms per invocation on Goobuntu, and with 10 invocations in
enter_chroot.sh, this means that we're wasting a lot of time, every time
cros_sdk is invoked. Cutting these unnecessary invocations reduces the time
required to run enter_chroot.sh from 2.3s to 0.8s.
CL:36618 is the companion change that updates cros_sdk to invoke
sudo unshare -m prior to calling enter_chroot.sh.
Summary of changes:
1. Remove all calls to sudo and just run the commands directly.
- Remove the mount queue and any sudo_multi optimizations.
- Rename sudo_chroot -> bare_chroot because we don't run sudo anymore there.
- Remove code for validating sudo timestamp.
2. Allow the scripts to work as root:
- Ensure that files created by cros_sdk that previously were owned by the
user still are owned by the user (either using chown or cp -p).
- Use $SUDO_USER to find the user's account.
- Use $SUDO_HOME instead of $HOME to find the user's home dir.
- Remove outdated code for disabling automount on Lucid, which doesn't work
when run as root.
- Update code for calculating the user's git username to use sudo to switch
to the user. Also move it to make_chroot.sh so that this change doesn't
impact performance.
3. Cleanup
- Remove environment syncer process in favor of just syncing once when chroot
is entered.
- Remove teardown and instead rely on unshare to unmount the mounts. To make
sure that outside processes never notice the mounts, we use mount -n. This
also ensures that /etc/mtab never contains stale mounts.
- Remove path-overrides, since it is no longer needed.
BUG=chromium-os:35714, chromium-os:35679
TEST=Trybot runs.
CQ-DEPEND=CL:36618
Change-Id: I919a8aadb08fafde97348e8511573c28fdd47186
Reviewed-on: https://gerrit.chromium.org/gerrit/36619
Tested-by: David James <davidjames@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Ready: David James <davidjames@chromium.org>
Users sometimes want to run gclient inside the chroot, so we shouldn't
tell users that using it is a bad idea.
The original reason why this message was added is historical: Originally,
users had a newer version of SVN inside the chroot compared to on their
workstation, so if you ran SVN inside the chroot it would permanently upgrade
your working copy such that the version of SVN outside the chroot did not work
with it anymore. This isn't a problem anymore, so we can remove the message.
BUG=none
TEST=Run remote trybot runs of chromiumos-sdk
Change-Id: I7b82a5c94e29d5928f4bb296ae2d99cef397d365
Reviewed-on: https://gerrit.chromium.org/gerrit/36346
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Ready: David James <davidjames@chromium.org>
Tested-by: David James <davidjames@chromium.org>
The process of bootstrapping the chroot from sources was
failing for several reasons when run from behind a firewall
with proxies. The llvm build was failing due to inability
to checkout sources through subversion using the
subversion.eclass wrapper (the "normal" way to do this in
the ebuild environment). This was because the user's
subversion configuration (including proxy settings) was not
inherited from $HOME/.subversion into the in-chroot sandbox
used by subversion.eclass.
This change creates symbolic links in the subversion.eclass
sandboxes for host and target builds in the chroot to fix
any build that uses the normal subversion.eclass for
checkouts. The operation is done at enter_chroot time so
that it applies to both ordinary builds and chroot creation
(via early_enter_chroot).
BUG=none
TEST='cros_sdk --replace --enter' behind proxied firewall
Change-Id: I0af2128866bb95799dc07c728c75cf3f2a0af7a3
Reviewed-on: https://gerrit.chromium.org/gerrit/34291
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Ready: paul drews <paul.drews@intel.com>
Tested-by: paul drews <paul.drews@intel.com>
Building the chroot environment from sources using
"--bootstrap" currently runs into a circular dependency:
curl->openssl->git->curl
The openssl->git dependency comes indirectly from the fact
that the current version of openssl uses the "cros-workon"
ebuild package to assist in applying packages. The ebuild
system automatically and silently resolves this circular
dependency by reverting the openssl library to an earlier
version that does not use cros-workon based patching.
Unfortunately this older version of openssl has a bug that
causes it not to work when doing builds in a firewalled
environment: When curl (using this older version of openssl
library) attempts to fetch an "https" url, it authenticates
the target server against a bundle of certificate-authority
certificates it maintains. Finding the certificate fails
(although the validation succeeds if curl is told explicitly
what certificate to use). With the certificate not-found,
server authentication fails, the curl download fails, and
the build ultimately fails.
This patch breaks the circular dependency, allowing a
more-current version of openssl to be used in curl, making
the above build scenario work in a firewalled environment.
The circularity is broken by first building git without curl
support (and webdav that depends on curl). Then early
toolchain components up through and including curl are
built. This build of curl then uses a more up-to-date
version of openssl with the desired bug-fix. Once curl is
built, then git is re-built and re-installed with the
now-installed version of curl (re-)enabled.
BUG=None
TEST=create chroot with --bootstrap ; build_packages (behind firewall)
Change-Id: Iaa560fdb6623fcb73cde066a3b2bc2a342169c62
Reviewed-on: https://gerrit.chromium.org/gerrit/34292
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Ready: paul drews <paul.drews@intel.com>
Tested-by: paul drews <paul.drews@intel.com>
CL:33868 (7b6f377c58) introduced a
breakage in the "cros_sdk --replace --bootstrap" scenario.
The make_chroot.sh script invokes early_enter_chroot before
invoking init_setup. The chroot/etc/profiles.d directory is
created in init_setup, but the referenced change was
expecting to create a file in that directory in the context
of early_enter_chroot before the directory was created.
This led to a "no such file or directory" error when trying
to create the file.
This change does a "mkdir -p" of the referenced directory
before putting things in it in the context of
early_enter_chroot. The filename is also fixed to the name
expected elsewhere in the scripts.
BUG=none
TEST=cros_sdk --replace --bootstrap
Change-Id: I6ac0467117d7b0dd413695153469b367d56c256c
Reviewed-on: https://gerrit.chromium.org/gerrit/34958
Commit-Ready: Brian Harring <ferringb@chromium.org>
Reviewed-by: Brian Harring <ferringb@chromium.org>
Tested-by: Brian Harring <ferringb@chromium.org>
This is forced by cros_sdk; in conjunction w/ this,
drop --distfiles and mangle the chroot on during entrance
dropping a symlink in the old /var/cache/distfiles location
pointing to the new mounted cache_dir location.
Additionally, thread CHROMEOS_CACHEDIR down through the end.
Do this without relying on a version upgrade script- we can't
require they be run before entering, thus we exploit the fact
that cros_sdk explicitly forces a write lock to do the upgrade,
if we see the old form we know we can do the upgrade w/out
worrying about collisions.
CQ-DEPEND=CL:33871
BUG=chromium-os:34457
TEST=manual testing.
Change-Id: I6805266e3ec683f05d3ba615f9e8840642a28e48
Reviewed-on: https://gerrit.chromium.org/gerrit/33868
Commit-Ready: Brian Harring <ferringb@chromium.org>
Reviewed-by: Brian Harring <ferringb@chromium.org>
Tested-by: Brian Harring <ferringb@chromium.org>
enter_chroot.sh was not updating /etc/hosts from the out-of-chroot
environment. Make it do that.
BUG=None
TEST=locally
Change-Id: Ieaa337ae90dbc0700c42fa7e4b96faf12d3968cb
Reviewed-on: https://gerrit.chromium.org/gerrit/34226
Reviewed-by: David James <davidjames@chromium.org>
Commit-Ready: Ryan Cui <rcui@chromium.org>
Tested-by: Ryan Cui <rcui@chromium.org>
This change was coopted from http://codereview.chromium.org/5331009/,
originally written by hungte@. And the coopted commit message:
It would be helpful if we could share some directories inside/outside the
chroot (e.g. editor configuration or the default Downloads directory). This
CL reads .local_mounts (just like .default_boards) from the "src/scripts"
folder, and mounts the directories whenever you do cros_sdk.
For safety concern, and to prevent the developer from accidentally deleting
their mounted files, the mounts are made read-only.
.local_mounts has a very simple syntax:
mount_path
or source_path(outside chroot) destination_path(inside chroot)
or # comments.
Examples:
/usr/share/vim/google
/home/XXX/Downloads /outside
BUG=chromium-os:34561
TEST=Manually:
1. Create ~/trunk/src/scripts/.local_mounts with following content:
# comment here
/usr/share/vim/google # test
/home/XXX/Downloads /outside
2. cros_sdk
3. ls -l /usr/share/vim/google/ # ensure dir is mounted correctly
ls -l /outside/ # ensure dir is mounted correctly
4. exit
5. mount | grep chroot # ensure nothing is left
Change-Id: I6f3400a436a825e8cdfcb18b788afe96ebba6757
Reviewed-on: https://gerrit.chromium.org/gerrit/33585
Tested-by: Michael Krebs <mkrebs@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Ready: Michael Krebs <mkrebs@chromium.org>
These are a new type of credential baked into chrome/chromium for
developers without internal copies of ChromeOS, and not building internal
versions of Chrome.
We automatically move .googleapikeys into the chroot each time.
We don't overwrite the destination, so that people can configure keys the
way they want. If they just don't want to be bothered, the best thing happens
the easiest way. Get Keys, put them in home. Keep working.
BUG=chromium-os:34438
TEST=local
Change-Id: I08e5970c6092f7b789aa5efef52db93841996d8f
Reviewed-on: https://gerrit.chromium.org/gerrit/33771
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Ready: Peter Mayo <petermayo@chromium.org>
Tested-by: Peter Mayo <petermayo@chromium.org>
Since sudo changes $HOME to /root, sudoed invocations of gsutil/boto
won't find the necessary credentials. This solves the problem by
installing a symlink at /root/.boto to the correct credentials file,
similar to how it's done for /home/$USER/.boto.
BUG=None
TEST=/root/.boto symlink created upon entering the chroot
Change-Id: I541556f836fa5d0b9708e5604218058401563fb3
Reviewed-on: https://gerrit.chromium.org/gerrit/32430
Reviewed-by: David James <davidjames@chromium.org>
Reviewed-by: Ryan Cui <rcui@chromium.org>
Commit-Ready: Gilad Arnold <garnold@chromium.org>
Tested-by: Gilad Arnold <garnold@chromium.org>
Do this via ensuring that any common.sh invoker
of raw umount (say a root script) sees our umount
path.
Additionally, inject into default profiles our override,
and via an upgrade scriptlet.
This is round two; originally appeared as CL:32088, was
reverted due to:
https://uberchromegw.corp.google.com/i/chromiumos/builders/chromiumos%20sdk/builds/2314/steps/BuildBoard/logs/stdio
The fix however is just adding a single sudo mkdir. :/
BUG=chromium-os:23443
TEST=cros_sdk --replace --bootstrap
TEST=cros_sdk --replace
Change-Id: I0dc7522a9c623f40081d4f138cea0c2c45171fea
Reviewed-on: https://gerrit.chromium.org/gerrit/32365
Commit-Ready: Brian Harring <ferringb@chromium.org>
Tested-by: Brian Harring <ferringb@chromium.org>
Reviewed-by: Chris Sosa <sosa@chromium.org>
Do this via ensuring that any common.sh invoker
of raw umount (say a root script) sees our umount
path.
Additionally, inject into default profiles our override,
and via an upgrade scriptlet.
BUG=chromium-os:23443
TEST=manual validation, trybot.
Change-Id: Ie2514f6e8d2e10a19ab8d11c8056177bc1a2fb4d
Reviewed-on: https://gerrit.chromium.org/gerrit/32088
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Ready: Brian Harring <ferringb@chromium.org>
Tested-by: Brian Harring <ferringb@chromium.org>
In particular, put the sudoers.d setup into one script (making
updates to it easier in the future if necessary), and
centralize the proxied vars into a const in common.sh.
Thanks to Kevin McCray/Josh Triplett/Alexander Kanevsky for
pointing out the missing proxy variables, and fixes/cleanup.
BUG=None
TEST=https_proxy=blah cros_sdk -- bash -c 'echo $https_proxy'
TEST=build_packages behind a proxy.
TEST=cros_sdk --replace && \
RSYNC_PROXY=blah cros_sdk -- bash -c 'echo $RSYNC_PROXY'
Change-Id: I3165882dfd9c8b52d25c2b26d7ff9242c84c91bd
Reviewed-on: https://gerrit.chromium.org/gerrit/31185
Tested-by: Brian Harring <ferringb@chromium.org>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Ready: Josh Triplett <josh@joshtriplett.org>
The code has been in here long enough - most people should be
transitioned over who are using gerrit-source. We've also already
removed the chrome projects from the default manifest, and things look
good so far.
BUG=chromium-os:32963
TEST=remote trybot
Change-Id: Idd5e3a2ad77ea86c7316a9d50f5da1a5fdf01d8b
Reviewed-on: https://gerrit.chromium.org/gerrit/31161
Reviewed-by: Brian Harring <ferringb@chromium.org>
Commit-Ready: Ryan Cui <rcui@chromium.org>
Tested-by: Ryan Cui <rcui@chromium.org>
This is used to build toolchains with specific env variables
BUG=chromium-os:33240
TEST=trybot x86-generic-toolchain-minor
Change-Id: I2bbdd7d013a15c57c590a0d660a210e0ae2a6695
Reviewed-on: https://gerrit.chromium.org/gerrit/30645
Tested-by: Zdenek Behan <zbehan@chromium.org>
Reviewed-by: David James <davidjames@chromium.org>
Commit-Ready: Zdenek Behan <zbehan@chromium.org>
Don't run .bash_logout after invocation of the hook, which clears the
screen, sending unnecessary escape characters.
BUG=None
TEST=Ran locally.
Change-Id: I6c466040e7169d304b892b85be6a5b0d578e7714
Reviewed-on: https://gerrit.chromium.org/gerrit/29645
Reviewed-by: David James <davidjames@chromium.org>
Commit-Ready: Ryan Cui <rcui@chromium.org>
Tested-by: Ryan Cui <rcui@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Make the following commands send stats to chromiumos-build-stats.appspot.com:
build_image
run_chroot_version_hooks
make_chroot
setup_board
update_chroot
BUG=chromium-os:33088
TEST=`cbuildbot --remote -p chromiumos/platform/crosutils alex-paladin`
confirm in the log that uploads succeded, and see them show up in queries
at chromiumos-build-stats.appspot.com.
Change-Id: I0280f91a3e7e0a0483c01c87072bc589003dbe95
Reviewed-on: https://gerrit.chromium.org/gerrit/28969
Tested-by: Matt Tennant <mtennant@chromium.org>
Reviewed-by: David James <davidjames@chromium.org>
Commit-Ready: Matt Tennant <mtennant@chromium.org>
See the bug for details.
BUG=chromium-os:32963.
TEST=Locally, remote trybots.
Change-Id: I33f5c42b36f3e06139036c299c2fc2c2ff026411
Reviewed-on: https://gerrit.chromium.org/gerrit/28543
Reviewed-by: David James <davidjames@chromium.org>
Commit-Ready: Ryan Cui <rcui@chromium.org>
Tested-by: Ryan Cui <rcui@chromium.org>
Originally, I patched enter_chroot.sh to stop the gvfs daemons to work
around an issue where these daemons would prevent loop devices from being
unmounted. See https://bugzilla.gnome.org/show_bug.cgi?id=677648
Unfortunately, temporarily stopping gvfs daemons has a bad side effects:
other GUI applications that rely on these daemons responding start hanging.
This can be reproduced, for example, by starting 'gedit'.
To fix these hangs, I'm just reverting my patches to enter_chroot.sh and
restoring the scripts to where they were before.
This reverts the following patches:
1. Stop gvfs daemons earlier during enter_chroot.
This reverts commit 0079158f73.
2. Revert "Stop the gvfsd-trash daemon during enter_chroot."
This reverts commit 654a00bd61.
3. Revert "Stop the automounting daemon whenever we're inside the chroot."
This reverts commit fae0a59e8b.
4. Revert "Clean up update_bootloaders.sh to avoid sleeping."
This reverts commit 0103b59138.
BUG=chromium-os:23443
TEST=Trybot run.
Change-Id: Ie9ff222fe5fc7232fd1fc39af129cc18531118c6
Reviewed-on: https://gerrit.chromium.org/gerrit/26922
Reviewed-by: Chris Wolfe <cwolfe@chromium.org>
Reviewed-by: Jon Kliegman <kliegs@chromium.org>
Reviewed-by: Brian Harring <ferringb@chromium.org>
Commit-Ready: David James <davidjames@chromium.org>
Tested-by: David James <davidjames@chromium.org>
The killall commands for stopping gvfs weren't working
consistently for two reasons:
1) They ran too late, after it already picked up the
mounts in cros_sdk.
2) killall sometimes can only access the first 15 characters
of a process name, so we should only match on these characters.
BUG=chromium-os:23443
TEST=Verify gvfs is properly stopped when entering the chroot
on precise systems.
Change-Id: I16aff4b0d9ac101083b63e06e55d50869479a152
Reviewed-on: https://gerrit.chromium.org/gerrit/26369
Reviewed-by: Pawel Osciak <posciak@google.com>
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Ready: David James <davidjames@chromium.org>
Tested-by: David James <davidjames@chromium.org>
For fresh sdk builds, gcc won't update and automatically see the
ccache tree for us. So make sure the perms are sane when people
enter the chroot. This will also automatically fix perms if/when
people manually delete the ccache dir (which sometimes happens on
the buildbots when people try to free up space).
BUG=None
TEST=`rm -rf distfiles/ccache/; cros_sdk` and see ccache dir get setup nicely
Change-Id: I5bcc86ebf696549b142a7ceb312eb8ec4be5e2bf
Reviewed-on: https://gerrit.chromium.org/gerrit/26257
Reviewed-by: David James <davidjames@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
Commit-Ready: Mike Frysinger <vapier@chromium.org>
This changelist adds code in common.sh to support collecting command
statistics before calling upload_command_stats to upload those stats
to an appspot instance.
The presence of a file at ~/.disable_build_stats_upload will disable
all uploading of build command stats.
BUG=chromium-os:27355
TEST=`build_packages --board=x86-generic` with missing appspot instance
shows upload error but still goes through with build.
TEST=`build_packages --board=x86-generic` with real appspot instance
completes upload with all expected stats and does not affect build.
TEST=From outside chroot:
`touch ~/.disable_build_stats_upload`
`cros_sdk`
'build_packages --board=x86-generic`
Nothing uploaded due to .disable file
TEST=Verified that putting 'set -u' in my fake build_packages
caused an exit in print_time_elapsed, then fixed unbound variable
in print_time_elapsed, then rerun passed.
TEST=`cbuildbot -g <cl> --lkgm mario-paladin` passed
TEST=`cbuildbot -g <cl> --lkgm link-release` passed
Change-Id: Ieb714522cb32d7558b661e4ee1a197d1fce2c516
Reviewed-on: https://gerrit.chromium.org/gerrit/26084
Tested-by: Matt Tennant <mtennant@chromium.org>
Reviewed-by: David James <davidjames@chromium.org>
Commit-Ready: Matt Tennant <mtennant@chromium.org>
This reverts commit 615fe57ff0.
This CL broke mod_image_for_recovery.sh with the following failure:
common.sh: line 684: $2: unbound variable
This bug is easy to fix, but suggests the CL needs more testing, so
we're going to revert for now and re-submit the CL once it's been
verified to pass on release trybots.
Example failure:
http://chromegw.corp.google.com/i/chromeos/builders/x86-mario%20canary/builds/2214/steps/Archive/logs/stdio
BUG=chromium-os:27355
TEST=None, since this is reverting a change that broke the tree.
Change-Id: I61d182e3dcee267a8d9dea3b547fa6a75140d974
Reviewed-on: https://gerrit.chromium.org/gerrit/26077
Reviewed-by: Doug Anderson <dianders@chromium.org>
Tested-by: David James <davidjames@chromium.org>
This changelist adds code in common.sh to support collecting command
statistics before calling upload_command_stats to upload those stats
to an appspot instance.
The presence of a file at ~/.disable_build_stats_upload will disable
all uploading of build command stats.
BUG=chromium-os:27355
TEST=`build_packages --board=x86-generic` with missing appspot instance
shows upload error but still goes through with build.
TEST=`build_packages --board=x86-generic` with real appspot instance
completes upload with all expected stats and does not affect build.
TEST=From outside chroot:
`touch ~/.disable_build_stats_upload`
`cros_sdk`
'build_packages --board=x86-generic`
Nothing uploaded due to .disable file
Change-Id: Iac071d1cc55a44335fc7c846960c7ae45fc93ed8
Reviewed-on: https://gerrit.chromium.org/gerrit/19401
Tested-by: Matt Tennant <mtennant@chromium.org>
Reviewed-by: Matt Tennant <mtennant@chromium.org>
Commit-Ready: Matt Tennant <mtennant@chromium.org>
This places a simple hook in cros_sdk by executing commands
in $HOME/.cros_chroot_init when a chroot is built. This
lets users copy files to the chroot (for instance, scripts), as
well as add lines to .bash_profile and other rc files.
BUG=chromium-os:31295
TEST=tested manually and verified .cros_chroot_init is executed when present
Change-Id: I2a5b070a9827272f7bb7b3d340ad6937b0bef329
Reviewed-on: https://gerrit.chromium.org/gerrit/23668
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Tested-by: Luigi Semenzato <semenzato@chromium.org>
Commit-Ready: Luigi Semenzato <semenzato@chromium.org>
The gvfsd-trash daemon interferes with mounting and unmounting
images, so it should also be disabled for user chroots.
The gvfsd-trash daemon automagically creates 'trash' subdirectory inside
any directory you mount and holds on to it for a little while, preventing
you from being able to unmount your mount point.
BUG=chromium-os:23443
TEST=Verify that this fix fixes cros_make_image_bootable for daisy on
Ubuntu precise.
Change-Id: I8df4c999ed57d7025b63c971390448c93d404e83
Reviewed-on: https://gerrit.chromium.org/gerrit/25243
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Anush Elangovan <anush@google.com>
Tested-by: David James <davidjames@chromium.org>
If automounting is enabled while we're inside the chroot, simple mount /
unmount commands will fail, causing image building commands to fail flakily.
This allows for quick mount / unmount.
BUG=chromium-os:23443
TEST='cros_sdk true' works now and does not print warnings on systems that
previously had race conditions in this case.
Change-Id: I18d725324cc42ab7c527d3b52479b6a90bc4172c
Reviewed-on: https://gerrit.chromium.org/gerrit/25166
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Tested-by: David James <davidjames@chromium.org>
Commit-Ready: David James <davidjames@chromium.org>
The only way to trigger this is if references were set up, but
never instantiated.
BUG=chromium-os:31682
TEST=manual permission checking.
Change-Id: Ibfc9dadf838f554cd10411753b5769117b1b1d42
Reviewed-on: https://gerrit.chromium.org/gerrit/24932
Reviewed-by: Brian Harring <ferringb@chromium.org>
Tested-by: Brian Harring <ferringb@chromium.org>
Commit-Ready: Brian Harring <ferringb@chromium.org>
When bootstrapping a chroot, the test for "is this a private source
tree repo" looked under the source tree in ~/trunk/src. That test
was reliable inside the chroot (during update_chroot), but from
outside the chroot (during bootstrapping) it depended on whether the
user had a ~/trunk/src with a certain file.
This change enforces correct behavior outside the chroot regardless
of the contents of the user's home directory.
BUG=chromium-os:31602
TEST=run the test case described in the bug report
Change-Id: I2150347fbad9c84af537f8c572908e6e5ce312b4
Reviewed-on: https://gerrit.chromium.org/gerrit/24659
Tested-by: Richard Barnette <jrbarnette@chromium.org>
Reviewed-by: David James <davidjames@chromium.org>
Commit-Ready: Richard Barnette <jrbarnette@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>
BUG=chromium-os:19287
TEST=try building a chroot both using .tbz2, .tar.bz2 and .tar.xz
Change-Id: Idfb13b691201b65c1fa1d5f8597f2aaa401a4051
Reviewed-on: https://gerrit.chromium.org/gerrit/23964
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Ready: Zdenek Behan <zbehan@chromium.org>
Tested-by: Zdenek Behan <zbehan@chromium.org>
For the builders/goobuntu, they're running pbzip2 ~1.0.5 w/ tar 1.22;
for whatever reason, that configuration reproducibly limits to single
core for:
tar -I /usr/bin/pbzip2 -xf /the/sdk
This is annoying; 2 minutes instead of 10s for 48 core builder for
example. Thus does *not* occur w/in the chroot (differing versions),
nor for tar=1.26 w/ pbzip2 1.1.6. The changelogs for both programs
are a bit spartan, but I'm suspecting tar just wasn't feeding it
particularly well (pbzip2 1.0.5 will parallelize if stdin is a pipe).
Regardless, we either try to force everyone to upgrade, or we just
use a form that behaves fine, which is what this CL does.
BUG=chromium-os:31320
TEST=manual validation of it.
Change-Id: I77a434bd2c70873459cbf373192fe73feadb2547
Reviewed-on: https://gerrit.chromium.org/gerrit/23811
Tested-by: Brian Harring <ferringb@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Ready: Brian Harring <ferringb@chromium.org>
Now that this is part of hard-host-depends, an we don't build the chroot
itself with ccache, so there's no need to force it in early.
BUG=None
TEST=`cbuildbot chromiumos-sdk` passed
Change-Id: I8b7c2a8c6f6df5eedac0c06ebb847f3011eb86d0
Reviewed-on: https://gerrit.chromium.org/gerrit/22954
Reviewed-by: Anush Elangovan <anush@google.com>
Commit-Ready: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
When bootstrapping for the first time, files in chroot/etc/ might not
exist, so we can't run `find` on them. This manifests itself currently
by spitting out the warning on all initial sdk boots:
find: `.../chroot/etc/resolv.conf': No such file or directory
People can find this confusing and cause sheriffs to waste time on the
wrong thing, so rework the code to avoid this.
BUG=None
TEST=`cros_sdk --delete ; cros_sdk` no longer warns about resolv.conf
Change-Id: I83f892e325e63e682aeb370a9dfc33e284e059d2
Reviewed-on: https://gerrit.chromium.org/gerrit/22845
Reviewed-by: Brian Harring <ferringb@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
Commit-Ready: Mike Frysinger <vapier@chromium.org>
Since /etc/mtab could be stale, use /proc/mounts instead.
BUG=None
TEST=`cros_sdk` in diff terminals still works
Change-Id: I526e5173581820c6983fe3702493a0349c1232c3
Reviewed-on: https://gerrit.chromium.org/gerrit/22860
Reviewed-by: David James <davidjames@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
Commit-Ready: Mike Frysinger <vapier@chromium.org>
Resubmit of If62b4f3973f02fd8e1deed35864c824a02ab0c22
This will be safe to land after
I2c4e21ec7e8c0c0cf58947e2b0a3a9edf7617a09
The breakage was a timing issue paired with people not always syncing
the complete tree. No changes to the CL are needed.
It is now used for:
- make_chroot (cros_sdk --bootstrap)
- update_chroot
setup_board is stripped of redundant code which was deprecated by this.
Also stripped is some usepkg logic in make_chroot, as that is now
exclusively source-only.
BUG=chromium-os:23032
TEST=trybot chromiumos-sdk
Change-Id: Ib888cf2886218622d9cfeebb17b9cd4462d06c89
Reviewed-on: https://gerrit.chromium.org/gerrit/22578
Tested-by: Zdenek Behan <zbehan@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Reviewed-by: asharif <asharif@chromium.org>
Commit-Ready: Zdenek Behan <zbehan@chromium.org>
The "function" keyword is superfluous, not in POSIX, is inconsistent
between bash files, and generally makes me angry. So convert every
instance to the form:
foo() {
BUG=None
TEST=`cbuildbot x86-generic-paladin` works
Change-Id: I97f5ca30a3edfef7222b1e08ac23917dc613b556
Reviewed-on: https://gerrit.chromium.org/gerrit/22467
Reviewed-by: David James <davidjames@chromium.org>
Commit-Ready: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
Currently, if set -e spots a nonzero exit we basically have
no real debug information- it just stops immediately without stating
where or why. This forces our scripts to be stupidly verbose so
we can track roughly where they were, thus when they fail we can
use that information to localize the rough exit point.
Instead we should be traping that set -e induced exit and
outputing necessary debug information to run it down. This includes
outputing the relevant stack trace, or at least what we can get of
it.
The 'die' function is now enhanced to automatically dump the trace
that lead to it. For most consumers this is desired- however for
commandline parsing induced dies ("--board is missing" for example),
the trace is noise. For those cases, a 'die_notrace' function was
added that retains the original non-backtrace behaviour.
Example output via instrumenting cros_generate_breakpad_symbols
w/ the failing command '/bin/false' (nonzero exit code).
Before:
./cros_generate_breakpad_symbols monkeys --board=x86-alex
<no output at all, just exit code 1>
With this CL:
./cros_generate_breakpad_symbols monkeys --board=x86-alex
ERROR : script called: ./cros_generate_breakpad_symbols 'monkeys' '--board=x86-alex'
ERROR : Backtrace: (most recent call is last)
ERROR : file cros_generate_breakpad_symbols, line 207, called: main 'monkeys' '--board=x86-alex'
ERROR : file cros_generate_breakpad_symbols, line 163, called: die_err_trap '/bin/false' '1'
ERROR :
ERROR : Command failed:
ERROR : Command '/bin/false' exited with nonzero code: 1
BUG=chromium-os:30598
TEST=inject a failing command into a script, verify the output.
TEST=inject a 'command not found', verify the output
TEST=cbuildbot x86-generic-full --remote
TEST=cbuildbot arm-tegra2-full --remote
TEST=cbuildbot chromiumos-sdk --remote
Change-Id: I517ffde4d1bb7e2310a74f5a6455b53ba2dea86c
Reviewed-on: https://gerrit.chromium.org/gerrit/17225
Reviewed-by: Brian Harring <ferringb@chromium.org>
Tested-by: Brian Harring <ferringb@chromium.org>
Commit-Ready: Brian Harring <ferringb@chromium.org>
Resubmit of If62b4f3973f02fd8e1deed35864c824a02ab0c22
This will be safe to land after I2c4e21ec7e8c0c0cf58947e2b0a3a9edf7617a09
It is now used for:
- make_chroot (cros_sdk --bootstrap)
- update_chroot
setup_board is stripped of redundant code which was deprecated by this.
Also stripped is some usepkg logic in make_chroot, as that is now
exclusively source-only.
BUG=chromium-os:23032
TEST=trybot chromiumos-sdk
Change-Id: Ic908eac712ac097e5c2062d3be70177e172aa924
Reviewed-on: https://gerrit.chromium.org/gerrit/20191
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Ready: Zdenek Behan <zbehan@chromium.org>
Tested-by: Zdenek Behan <zbehan@chromium.org>
The private chromeos-overlay has shared credentials for read-only
access to the archive of CrOS images. When entering the chroot check
whether these credentials are available, and install them to ~/.boto
within the chroot unless that file already exists.
BUG=None
TEST=Applied patch and entered fresh private and public chroots.
Manually ran 'gsutil config' to replace the credentials;
this works as long as there is no ~/.boto.bak already.
Re-entered the chroot with manually-configured credentials.
Ran 'gsutil ls' with BOTO_CONFIG set to a board-specific boto.
Change-Id: Ib62dc28f90bce692a833b5fc9d4c56981acc98d1
Reviewed-on: https://gerrit.chromium.org/gerrit/21480
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Tested-by: Chris Wolfe <cwolfe@chromium.org>
Commit-Ready: Chris Wolfe <cwolfe@chromium.org>
It is now used for:
- make_chroot (cros_sdk --bootstrap)
- update_chroot
setup_board is stripped of redundant code which was deprecated by this.
Also stripped is some usepkg logic in make_chroot, as that is now
exclusively source-only.
BUG=chromium-os:23032
TEST=trybot chromiumos-sdk
Change-Id: If62b4f3973f02fd8e1deed35864c824a02ab0c22
Reviewed-on: https://gerrit.chromium.org/gerrit/17910
Reviewed-by: Zdenek Behan <zbehan@chromium.org>
Tested-by: Zdenek Behan <zbehan@chromium.org>
Commit-Ready: Zdenek Behan <zbehan@chromium.org>
Make sure our chroot has pbzip2 as early as possible so we can leverage
it later on in the build process as the default compressor.
BUG=None
TEST=semi-manual:
- remove pbzip2 as soon as we've unpacked the chroot in make_chroot
- run `cros_sdk` and see no pbzip2 warnings
Change-Id: I27d00cdb97c5ffcde79eb04cfabbe6e82a72c706
Reviewed-on: https://gerrit.chromium.org/gerrit/19469
Reviewed-by: David James <davidjames@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
Commit-Ready: Mike Frysinger <vapier@chromium.org>
Now that the newer build-docbook-catalog supports ROOT, we don't need
to manually run this ourselves.
BUG=chromium-os:18036
TEST=`cbuildbot chromiumos-sdk` works
Change-Id: I3a89701a49623f9e50e651af9d8d505608fdc615
Reviewed-on: https://gerrit.chromium.org/gerrit/19252
Tested-by: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Zdenek Behan <zbehan@chromium.org>
Commit-Ready: Mike Frysinger <vapier@chromium.org>
To build recovery images, the initramfs package needs fonts for
various languages. The Japanese fonts to use depend on whether
it's an internal (Chrome OS) or external (Chromium OS) build.
Including the internal Japanese fonts requires additional settings
in /etc/make.conf in the chroot. This change updates make_chroot
and update_chroot to ensure that the necessary settings are present
when appropriate.
BUG=chromium-os:26757
TEST=confirm proper packages after chroot creation with private manifest
TEST=confirm proper packages after chroot creation with public manifest
TEST=confirm proper packages after chroot upgrade with private manifest
TEST=confirm proper packages after chroot upgrade with public manifest
CQ-DEPEND=I32c4561c47752d96d2769c5429c5e44fe630f8f9
Change-Id: I2b29b32bf4e6b891a2228dc7f93ae0bb3f85e37f
Reviewed-on: https://gerrit.chromium.org/gerrit/17372
Tested-by: Richard Barnette <jrbarnette@chromium.org>
Reviewed-by: David James <davidjames@chromium.org>
Commit-Ready: Richard Barnette <jrbarnette@chromium.org>
There's no reason to always show the full build output when updating the
chroot, so use the --quiet flag. If there is a failure, emerge will dump
the full log like normal.
BUG=None
TEST=`cros_sdk --delete && cros_sdk` showed much less output
Change-Id: I062900325d0005db150dbd1267048a7c19df2ac5
Reviewed-on: https://gerrit.chromium.org/gerrit/16977
Reviewed-by: David James <davidjames@chromium.org>
Reviewed-by: Brian Harring <ferringb@chromium.org>
Commit-Ready: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
We are changing the default emerge options to use --oneshot, so make sure
our scripts select the packages we care about for @world.
BUG=None
TEST=setup_board+build_packages for tegra2 adds pkgs to world
Change-Id: I5fc68c538a1a1c846bd9724f27ef717029d11d42
Reviewed-on: https://gerrit.chromium.org/gerrit/16838
Reviewed-by: David James <davidjames@chromium.org>
Commit-Ready: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
BUG=None
TEST=`cros_sdk --delete && cros_sdk` still works
Change-Id: I9a5d5e5eedb950dcfaf9108c22087a5a7fa7cd47
Reviewed-on: https://gerrit.chromium.org/gerrit/16825
Reviewed-by: Brian Harring <ferringb@chromium.org>
Reviewed-by: David James <davidjames@chromium.org>
Commit-Ready: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
The logic of IGNORE_PREFLIGHT_PREBUILTS is currently inverted in make_chroot,
causing preflight prebuilts to be ignored by default, but then used in the one
situation where they should be ignored. This CL fixes that.
BUG=chromium-os:26834
TEST=Verify preflight prebuilts are now used by default in make_chroot.
Change-Id: I626319d6de79b6f201f993a05e34610e944b161e
Reviewed-on: https://gerrit.chromium.org/gerrit/16614
Reviewed-by: Richard Barnette <jrbarnette@chromium.org>
Commit-Ready: David James <davidjames@chromium.org>
Tested-by: David James <davidjames@chromium.org>
The --jobs=<n> option wasn't actually being passed to parallel_emerge --
which meant it didn't do anything.
BUG=chromium-os:26827
TEST=Ran with --jobs=N and made sure only N parallel_emerge processes were
started.
Change-Id: I581fc5588b54e246acaefd0c7e528e55adf9ba8a
Reviewed-on: https://gerrit.chromium.org/gerrit/16570
Reviewed-by: Michael Krebs <mkrebs@chromium.org>
Commit-Ready: David James <davidjames@chromium.org>
Tested-by: David James <davidjames@chromium.org>
When we create a chroot for the first time, we see:
...
Exiting: Nothing left to do; exiting. :)
Elapsed time: 5m10s
cros_sdk: All set up. To enter the chroot, run:"
cros_sdk: $ cros_sdk --enter "
CAUTION: Do *NOT* rm -rf the chroot directory; if there are stale bind
mounts you may end up deleting your source tree too. To unmount and
delete the chroot cleanly, use:
$ cros_sdk --delete
(cr) (v) vapier@vapier ~/trunk/src/scripts $
Kill the spurious double quotes, add new lines to make the text easier
to pick out from the previous command/new prompt, and only show the
sdk prefix on the first line.
BUG=None
TEST=`cros_sdk --delete && cros_sdk` looks nice
Change-Id: Iba0ee68035fffe6fc946e9b3476cc0d0b55abfc5
Reviewed-on: https://gerrit.chromium.org/gerrit/16075
Reviewed-by: David James <davidjames@chromium.org>
Commit-Ready: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
The gcc-config file name could be different from the gcc atom that creates it.
Changed the logic to first find out the binary path of gcc and then to query
portage to see what package owns that path.
BUG=none
TEST=cros_sdk --bootstrap --replace
cros_sdk -- ./setup_board --board=x86-zgb --nousepkg
Change-Id: I3ccfa9948aaba6ba28107f150523c8a5d7cf1260
Reviewed-on: https://gerrit.chromium.org/gerrit/15556
Reviewed-by: David James <davidjames@chromium.org>
Reviewed-by: Zdenek Behan <zbehan@chromium.org>
Tested-by: asharif <asharif@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Ready: asharif <asharif@chromium.org>
The refresh-packages builder needs to start using auth tokens for logging into
Google Docs, rather than email/password each time, so this copies ~/.gdata_token
into the chroot in the same way as ~/gdata_cred.txt.
BUG=chromium-os:23819
TEST=Put a file at ~/.gdata_token outside chroot, then run:
`cros_sdk true ; cmp {chroot/home/$USER/,~/}.gdata_token`, which passes.
Change-Id: Ib688a01aa88e0fd0f0211236bb0354813ef1fbb3
Reviewed-on: https://gerrit.chromium.org/gerrit/15550
Commit-Ready: Matt Tennant <mtennant@chromium.org>
Reviewed-by: Matt Tennant <mtennant@chromium.org>
Tested-by: Matt Tennant <mtennant@chromium.org>
First, add a enable_strict_sudo helpers that scripts that are
sudo strict can invoke. This does a sanity check bailing immediately
if invoked from a non sudo-keep-alive context.
Second, update enter_chroot.sh and make_chroot.sh to be strict.
While this is strict, that's the point. The aim of this is to block
the previous sudo interactive mess for spreading, let alone reappearing
in scripts/code that has been cleansed.
BUG=chromium-os:18393
TEST=cros_sdk --replace; in the midst of it, do sudo -k.
cros_sdk should thus bail out w/ an appropriate error.
CQ-DEPEND=I01bb1466cf027401fa387af7fad15e42fd33aea4
Change-Id: I76c5b87a812cc78c30a2eb1a0c56b9e438f4a98f
Reviewed-on: https://gerrit.chromium.org/gerrit/15294
Commit-Ready: Brian Harring <ferringb@chromium.org>
Reviewed-by: Brian Harring <ferringb@chromium.org>
Tested-by: Brian Harring <ferringb@chromium.org>
Via distfiles bind mounting, chrome build output was being cached
across chroot replacements. This potentially is desirable, but
wasn't explicitly planned for and violates an assumption of the
canaries.
As such move the content to it's new location.
BUG=None
TEST=emerge-<board> chromeos-chrome
CQ-DEPEND=Ibfc5d42e74861ff498dd2cb6cc2d7be6ec0ded60
Change-Id: I18e384f0fc1cedb3c70ab85178103765322370eb
Reviewed-on: https://gerrit.chromium.org/gerrit/15383
Reviewed-by: David James <davidjames@chromium.org>
Tested-by: Brian Harring <ferringb@chromium.org>
Purpose of this is to allow us to avoid re-downloading everything
every time we rebuild the chroot.
This maintains two directories; host and target. Future enhancement
involves collapsing this into one- this requires some host work however,
and has some potential gotchas in doing so.
Meanwhile, we now store distfiles in repo/distfiles/{host,target},
and mount bind repo/distfiles into /var/cache/distfiles.
An upgrade script in turn optimistically tries to move the content
into the new location; if it can't complete the move, it wipes the
content and the user has to redownload it (acceptable, if annoying).
BUG=chromium-os:13115
TEST=cbuildbot x86-generic-full
Change-Id: Iea96429df0e1fdc4ac0860fbce0daabc90c4c2a3
Reviewed-on: https://gerrit.chromium.org/gerrit/15189
Reviewed-by: Brian Harring <ferringb@chromium.org>
Tested-by: Brian Harring <ferringb@chromium.org>
Commit-Ready: Brian Harring <ferringb@chromium.org>
Now that make_chroot.sh lives alongside enter_chroot, we're able
to invoke it directly and use internal options not exposed through
cros_sdk. Thus enter_chroot grows a --early_make_chroot flag.
This flag bypasses the normal sudo usage, and runs the command
as root. It is needed and used by make_chroot for when sudo
may not yet exist (--bootstrap via stage3 lacks sudo).
For where sudo exists and we need access to the source tree,
distfiles, or profile sourcing, enter_chroot is directly used.
For all other invocations we use a single sudo chroot call to
bypass the overhead of enter_chroot.
This is a complete fix to chromium-os:25697 while addressing
all refactoring requests that came up in I8a6c5a26.
Finally, convert all comments into complete sentences, upper
case the leading word, etc to keep reviewers happy.
BUG=chromium-os:25697
TEST=Within a trybot, cros_sdk --bootstrap --replace
TEST=cros_sdk --replace
TEST=cros_sdk --delete
TEST=cros_sdk
Change-Id: I270ee7fc325ef0bea74c61505d25cdbb49a9a333
Reviewed-on: https://gerrit.chromium.org/gerrit/15322
Commit-Ready: Brian Harring <ferringb@chromium.org>
Reviewed-by: Brian Harring <ferringb@chromium.org>
Tested-by: Brian Harring <ferringb@chromium.org>
Imported as of 09d3117c483c60f55f552694b84b0964dfc17477.
Only modification is so that it can find common.sh.
BUG=chromium-os:25697
TEST=None
Change-Id: Idb43f1ecff3dff10a61396629a344c8bd10f505e
Reviewed-on: https://gerrit.chromium.org/gerrit/15317
Tested-by: Brian Harring <ferringb@chromium.org>
Reviewed-by: David James <davidjames@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Ready: Brian Harring <ferringb@chromium.org>
This adds repo --references awareness (which maps down to git shared object
pools; see git clone -s); specifically, it sets up the appropriate indirection
bindings back to the actual shared source for use from within the chroot.
Note that the approach is a bit complex, but it's required complexity- via
this approach the references are accessible both from within, and without
the chroot.
BUG=chromium-os:19939
TEST=cbuildbot x86-generic-full --clobber --notests --nouprev
TEST=cbuildbot x86-generic-full --notests --nouprev
CQ-DEPEND=I986f17503dc154234ecadd90f6975d7164117cdb
Change-Id: I8498863010cefe0bf5b8f20350fa45a5f2a093d3
Reviewed-on: https://gerrit.chromium.org/gerrit/13467
Tested-by: Brian Harring <ferringb@chromium.org>
Reviewed-by: David James <davidjames@chromium.org>
Commit-Ready: Brian Harring <ferringb@chromium.org>
On long running systems, the ssh-agent dir might go stale. This should
not prevent entering the cros chroot though, so skip it if it's invalid.
BUG=None
TEST=`cros_sdk` enters with ssh-agent dir bind mounted
TEST=`rm -rf /tmp/ssh-*/; cros_sdk` enters with ssh-agent dir skipped
Change-Id: I923b031612c0d37a896437c7355cac6c448eef82
Reviewed-on: https://gerrit.chromium.org/gerrit/12758
Reviewed-by: David James <davidjames@chromium.org>
Commit-Ready: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
The current instructions for golo access include additions to the
~/.ssh/config file that are specific to Google distros of ssh. The
filtering mechanism in enter_chroot needed an update to include
these new options to filter out.
BUG=chromium-os:23322
TEST=With a golo-configured .ssh/config, enter chroot and test ssh:
> cros_sdk
> git remote update
Fetching ...
...
Change-Id: Ida9fcdff37ff0eb24c811ad789f82136dca7252f
Reviewed-on: https://gerrit.chromium.org/gerrit/12037
Tested-by: Matt Tennant <mtennant@chromium.org>
Reviewed-by: David James <davidjames@chromium.org>
Reviewed-by: Zdenek Behan <zbehan@chromium.org>
Commit-Ready: Matt Tennant <mtennant@chromium.org>
Every invocation of `sudo` delays things, so merge all of the mounts
into a single `sudo` command when possible. This saves over 1 second
on initial execution (out of ~4 seconds total).
BUG=None
TEST=`cros_sdk --enter true` still mounts & unmounts properly
Change-Id: Ibc66507dc21250a81207d2f940645eaebe93c79c
Reviewed-on: https://gerrit.chromium.org/gerrit/10901
Reviewed-by: David James <davidjames@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
Commit-Ready: Mike Frysinger <vapier@chromium.org>
The current code that syncs the user's ssh files from ~/.ssh/ to the
chroot assumes the user has these files in the first place. For the
fresh user and/or install, these might not exist. So ignore errors
from attempting to copy them. It's not like we attempt to "recover"
anyways, and this sync is more of a "let's make things smoother" than
a "we need this to run properly".
BUG=chromium-os:15005
TEST=`mv ~/.ssh ~/.ssh.bak; mkdir ~/.ssh; cros_sdk` and see no more warnings
Change-Id: I9cac2ecad7c64c088fbdfe55377c0429b3b7ab03
Reviewed-on: https://gerrit.chromium.org/gerrit/10922
Reviewed-by: David James <davidjames@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
Commit-Ready: Mike Frysinger <vapier@chromium.org>
Ubuntu 11.10, for whatever reason, moved /dev/shm to /run/shm. Since
/run is not mounted in the chroot, /dev/shm is a dangling symlink and
various things that require shared memory start failing.
If /run and /run/shm exist, bind-mount them in the chroot.
Because /run doesn't yet exist in the SDK tarballs, try to create it
(and /run/shm if necessary) as root if it and can't be created as a
normal user.
BUG=chromium-os:19871
TEST=Run cros_sdk, verify that /dev/shm has the expected contents
Change-Id: I61583c1c0d409c1234fa8d8930a9b64544c9a8e7
Reviewed-on: https://gerrit.chromium.org/gerrit/10222
Tested-by: Aaron Plattner <aplattner@nvidia.com>
Reviewed-by: David James <davidjames@chromium.org>
Commit-Ready: David James <davidjames@chromium.org>
The sync pid file is created without using sudo which means we should be
able to delete it without using sudo. By default, run `rm` directly and
if it fails, fall back to sudo like we historically have.
BUG=None
TEST=`cros_sdk --enter true`; see pid file deleted
Change-Id: I26d898f6d594eb9ea4652335468345dd11303122
Reviewed-on: http://gerrit.chromium.org/gerrit/8644
Commit-Ready: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
We call `readlink -f` on the chroot mount point to resolve any symlinks
in the path (since that is what `mount` records). But the only paths
that can have symlinks are in the base chroot mount. Everything below
that we know are not going to be symlinks (since we've set up the paths
ourselves). So process the chroot mount point with readlink once and
reuse that value everywhere else.
BUG=None
TEST=`cros_sdk --enter true` still mounts & unmounts properly
Change-Id: Id1e734d20c0cb766f5490583b793930af77b3b14
Reviewed-on: http://gerrit.chromium.org/gerrit/8645
Reviewed-by: David James <davidjames@chromium.org>
Commit-Ready: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
The chrome root code doesn't attempt to mount the tree if there is no
source root configured. So re-order the code from:
- is chrome source mounted ?
- does a chrome source exist ?
- mount chrome source
to the more logical:
- does a chrome source exist ?
- use ensure_mounted to check+mount
This lets us use the mount cache for the chrome mounting and unifies
duplicated mount/checking logic.
BUG=None
TEST=`cros_sdk --enter` still works with simultaneous runs
TEST=`cros_sdk --enter -- ls` still works
Change-Id: I7e6af9dd7f65cefa04438c2862c931f06237060a
Reviewed-on: http://gerrit.chromium.org/gerrit/8032
Commit-Ready: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
The depot_tools mount code is exactly like ensure_mounted except that it
prints a warning instead of dying when things can't be mounted. So take
the current api and extend it slightly to support warning or dying. This
also lets us re-use the existing mount cache and avoid the forks as well
as clean up duplicated code.
BUG=None
TEST=`cros_sdk --enter` still works with simultaneous runs
TEST=`cros_sdk --enter -- ls` still works
Change-Id: I89336778b6aa16191e79d900a51774929cadf06b
Reviewed-on: http://gerrit.chromium.org/gerrit/8031
Commit-Ready: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
The cros_sdk tool runs a lot of helper programs which can be replaced
with bash builtins, or condensed multiple calls into a single one. By
themselves they aren't that slow, but add them all up and run them a
whole lot, and it starts to make a difference.
External programs to bash internals:
- dirname $f -> ${f%/*}
- f=$(cat $f) -> f=$(<$f)
- which f -> type -P f
Simpler expressions:
- [[ ( ... ) ]] -> [[ ... ]]
- eval v=\$$f -> v=${!f}
Common/clearer expressions:
- ! var=$(cmd) -> var=$(cmd || :)
Condensed tools:
- sort | uniq -> uniq -u
BUG=None
TEST=`cros_sdk --enter` still works with simultaneous runs
TEST=`cros_sdk --enter -- ls` still works
Change-Id: Ice5e5e252237082a2249990644e051895d61d1fd
Reviewed-on: http://gerrit.chromium.org/gerrit/8029
Commit-Ready: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
Most of the time the files in the chroot /etc/ are already changed to
the proper owner. Only on the first run do they need to be changed.
So check the owner before doing the sudo as that is a bit faster. If
we do end up needing to run sudo, bundle the commands up to avoid having
to execute sudo multiple times.
BUG=None
TEST=`sudo chown root chroot/etc/resolv.conf && cros_sdk --enter`; see /etc/resolv.conf owned by me
Change-Id: Ifd974ace168ab309fdc2e7583d135ee23576a5bb
Reviewed-on: http://gerrit.chromium.org/gerrit/8418
Reviewed-by: David James <davidjames@chromium.org>
Commit-Ready: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
Since we grab a lock before mounting anything, we know that the relevant
list of mounts won't change (since we haven't changed it yet). So we can
cache the output of `mount` and re-use that in every subsequent check.
BUG=None
TEST=`cros_sdk --enter` still works with simultaneous runs
TEST=`cros_sdk --enter -- ls` still works
Change-Id: I27c98c923761fb777686632fe6e6604ca543cfb6
Reviewed-on: http://gerrit.chromium.org/gerrit/8030
Commit-Ready: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
Checking the permission of files is faster than assuming they're broken
and then running `sudo chmod`, so do just that.
BUG=None
TEST=run `cros_sdk --enter` with bad perms and see them fixed
TEST=run `cros_sdk --enter` with correct perms and see sudo skipped
TEST=the buildbot failure was due to other changes, not this one
Change-Id: Ie69b4e766e2e3652944e4723d091ce589d07a4f6
Reviewed-on: http://gerrit.chromium.org/gerrit/8028
Commit-Ready: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
No need to call `git var | sed` multiple times when we can call it once
and then use bash string parsing routines to extract the relevant info.
BUG=None
TEST=delete .gitconfig in chroot; enter chroot; verify .gitconfig is up-to-date
Change-Id: I900e4241a5c53bc46c90eb0f2a01152f4bfba577
Reviewed-on: http://gerrit.chromium.org/gerrit/8027
Commit-Ready: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
First let's add a chroot upgrade hook that'll declare a small list of
locales to speed up glibc upgrades.
The other blocker was that `cros_sdk --enter` took an inordinate amount
of time to execute (`locale-gen -u` specifically) when the number of
existing locales is huge. This seems to be a bug in the bash/glibc
stack which is resolved in newer versions, but we can workaround the
issue by forcing locale-gen to run in a C locale. The tool itself does
not care about its locale and we silence its output.
BUG=chromium-os:20378
TEST=`cros_sdk --enter`; see no locales in /etc/locale.gen; run `./build_packages` and see upgrade hook work
TEST=generate all 400 locales; `cros_sdk --enter`; see that it was quick
Change-Id: I8fcc5e26bd8e1bcfd52b6a6c7ef3cacf0a252081
Reviewed-on: http://gerrit.chromium.org/gerrit/7806
Reviewed-by: David James <davidjames@chromium.org>
Reviewed-by: Zdenek Behan <zbehan@chromium.org>
Commit-Ready: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
Running `sudo` at all is a bit poky, so let's avoid modprobing fuse when
we know it already exists because it's listed in /proc/filesystems.
The relative speed:
time sudo modprobe fuse -> 0.164s
time grep -q fuse /proc/filesystems -> 0.002s
BUG=None
TEST=enable `set -x`; run `sudo modprobe fuse`; run `cros_sdk true`; see modprobe skipped
Change-Id: I07213e7b0607ca65df2a8a3ae22eb4a49c8bb888
Reviewed-on: http://gerrit.chromium.org/gerrit/7824
Commit-Ready: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
It doesn't make much sense to run gconftool when the default automount
value is what we already want: false. So in this common case, skip the
save/restore logic altogether since we aren't actually changing things.
Further, we can merge the `which` check in by checking the exit status
of the `gconftool` run. If it fails, the tool doesn't exist.
BUG=None
TEST=enable `set -x`; run `cros_sdk true`; see gconftool run once in common case
Change-Id: I1c6aee945218a8a9df533d9ef207c750f909c252
Reviewed-on: http://gerrit.chromium.org/gerrit/7823
Commit-Ready: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
The teardown_env is a little slow when unmounting things because it runs
`sudo umount` once per mount point. This is so that when things go wrong,
it can easily recover. However, this slows down the common case at the
expense of the uncommon.
Refactor the code so that in the common case, we run one `sudo umount`.
When things do fail, we're a bit slower as we reparse the entire mount,
list, but that's fine as it's an error case.
BUG=None
TEST=enable `set -x`; run `cros_sdk true`; see all mount points unmounted in one shot
Change-Id: Iec98a7b9f51a77e90c30e6f6acae26e528b8c50d
Reviewed-on: http://gerrit.chromium.org/gerrit/7822
Tested-by: Mike Frysinger <vapier@chromium.org>
Reviewed-by: David James <davidjames@chromium.org>
Works around:
http://code.google.com/p/chromium-os/issues/detail?id=20378
which is breaking the tree.
BUG=chromium-os:20378
TEST=cros_sdk --enter
Change-Id: I0d5dc2dd6466efe9baafcc404648d2c53309a3b7
Reviewed-on: http://gerrit.chromium.org/gerrit/7679
Reviewed-by: David James <davidjames@chromium.org>
Reviewed-by: Thieu Le <thieule@chromium.org>
Tested-by: David James <davidjames@chromium.org>
The process of updating the Portage package status spreadsheet on
buildbots requires a credentials file at ~/.gdata_cred.txt in the
chroot. All buildbot VMs have this file outside the chroot now,
so this change adds that file to the list of things created in
the user's homedir when entering the chroot.
BUG=None
TEST=Run cros_sdk in these scenarios:
~/.gdata_cred.txt exists -> Same file should be at $HOME in chroot
~/.gdata_cred.txt does not exist -> No file created in chroot
Change-Id: I5c0f333a9308f5efa5324ce2e202a7c9e9fdb48b
Reviewed-on: http://gerrit.chromium.org/gerrit/6911
Reviewed-by: Zdenek Behan <zbehan@chromium.org>
Tested-by: Matt Tennant <mtennant@chromium.org>
BUG=None
TEST=run cros_sdk
Change-Id: I39fafd58c7cc9fd536fe9b75f314f9970766a483
Reviewed-on: http://gerrit.chromium.org/gerrit/6414
Tested-by: Richard Barnette <jrbarnette@chromium.org>
Reviewed-by: Chris Sosa <sosa@chromium.org>
Reviewed-by: David James <davidjames@chromium.org>
The buildbot code forces the en_US.UTF8 locale itself, and does so inside
of the chroot, so our auto-detection code doesn't catch it. Always create
these locales that the buildbot uses.
BUG=None
TEST=cleared out locales, emptied out active locale env vars, ran enter_chroot, saw that en_US{,UTF8} were created
Change-Id: I9ad65007a340333e19743985c9cbeea9403823fa
Reviewed-on: http://gerrit.chromium.org/gerrit/6168
Reviewed-by: David James <davidjames@chromium.org>
Tested-by: David James <davidjames@chromium.org>
Since we no longer force LC_ALL=C on everyone, the few times that perl
does get run by people, it spews the expected "Setting locale failed"
warnings. This isn't as big a deal as with Debian systems as perl usage
is uncommon in Gentoo.
This is also highlights the long-existing small issue of only specific
locales being available in the chroot. So for non US english speaking
users, they've been having non-optimal experiences.
So parse the user's active locale and automatically generate the missing
ones in the chroot so that when they do enter, things "just work".
BUG=chromium-os:19139
TEST=set locale to non-existent ones, enter_chroot, & verify locales are created
Change-Id: I43f809a1ee1472e4797edab0f32cecf582ea8b48
Reviewed-on: http://gerrit.chromium.org/gerrit/5986
Reviewed-by: David James <davidjames@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
* Removed boilerplate, simplified search code.
* Fixed one too long line
This will unfortunately kill all outstanding CLs into enter_chroot.
BUG=chromium-os:18750
TEST=run it
Change-Id: I39c45fa8163d92487b512e7e8d298ce9231f4bd2
Reviewed-on: http://gerrit.chromium.org/gerrit/5830
Tested-by: Zdenek Behan <zbehan@chromium.org>
Reviewed-by: Chris Sosa <sosa@chromium.org>
Reviewed-by: Anush Elangovan <anush@chromium.org>
Reviewed-by: Richard Barnette <jrbarnette@chromium.org>
Reviewed-by: David James <davidjames@chromium.org>
