Instead of handling toolchain packages in make_chroot and telling
update_chroot to skip the toolchains just depend on update_chroot to do
it properly. Reduces our code duplication by a tiny but worthwhile bit.
When a user creates a chroot and as a common primary group such as
'users' the groupadd command fails. Instead treat this the same as users
and only fail if the group exists but has a different (such as the
'users' group not using GID 100). Hopefully this works better.
If the user already exists check that the UID and GID are correct and
modify it (setting shell and home directory) to match what the SDK
expects. This avoids needlessly failing if the user calling cros_sdk is
the 'core' user on a CoreOS machine.
Change new-user creation to copy the user's full name and group instead
of using a generic name and Google's 'eng' group. Also remove the
default password for the account, it isn't needed and uses perl.
Our SDK tarballs aren't compressed using pbzip2 so there is no advantage
to using pbzip2 to decompress them over bzip2, however lbzip2 does offer
a big advantage. Also trust that the portage config defines a valid
version of bzip2 since we have control over the tarball creation and can
make sure to always include required utilities.
If our sdk has an /etc/mtab file already, then clobber it. This fixes
build problems where chromeos-base now installs /etc/mtab for us, but
the sdk build isn't expecting it leading to the error:
INFO cros_sdk:make_chroot: Running init_setup()...
ln: creating symbolic link `/b/cbuild/new-sdk-chroot/etc/mtab': File exists
Running ['/b/cbuild/src/scripts/sdk_lib/make_chroot.sh', '--stage3_path',
'/b/cbuild/built-sdk.tar.xz', '--chroot', '/b/cbuild/new-sdk-chroot',
'--cache_dir', '/b/cbuild/.cache', '--nousepkg'] failed!
BUG=None
TEST=`cros_sdk --chroot foo` still works
Change-Id: I539cf329e93e28534e6ff00577ce415d76918b85
Reviewed-on: https://gerrit.chromium.org/gerrit/43641
Reviewed-by: David James <davidjames@chromium.org>
Commit-Queue: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
As-is all of the various emerge wrapping scripts default to using
--getbinpkg whenever --usepkg is enabled. This means every single emerge
command made makes multiple synchronous HTTP requests to the upstream
binary package repository to get the latest package list. This gets
really frustrating when working remotely with limited network
connectivity. Using --usepkg with --nogetbinpkg will use locally cached
packages without making remote requests.
use the efunctions package for the /etc/init.d/functions.sh script
instead of backing up the old function.sh which doesn't work with the
new baselayout
we remove openrc which provides /etc/init.d/functions.sh. Unfortunatly
other things rely on this file. Stash it away in /tmp/ then restore it
for now.
Change-Id: I18a59e05ecdf08cc8a560b29049c8d25ac1bf5a3
Slipped past during rename of the chroot upgrade script from
49 to 50; name was slightly changed but full re-validation of the
rename wasn't done (thus the typo slipped past testing, and review).
Simplify the code via removal of invoking the upgrade script, instead
just doing the relevant commands (fixing chroot awareness issues in
the process).
BUG=None
TEST=manual cros_sdk invocation
Change-Id: I122de8b4cf7ec0845643e09e7919cbcdbd0bb79a
Reviewed-on: https://gerrit.chromium.org/gerrit/41202
Reviewed-by: Brian Harring <ferringb@chromium.org>
Tested-by: Brian Harring <ferringb@chromium.org>
Rather than having to find /home/${SUDO_USER:-${USER}}/trunk, instead
just look for /mnt/host/trunk (defined by common.sh as $CHROOT_TRUNK_DIR).
This simplifies code flow, and is a requirement for shoving chromite
into PYTHONPATH globally w/in the chroot.
BUG=chromium-os:37347
TEST=cros_sdk --replace; cros_sdk w/ chroot upgrade.
Change-Id: I9ee3e6556541a91193f49cbf74ffc5a8e090537f
Reviewed-on: https://gerrit.chromium.org/gerrit/39921
Tested-by: Brian Harring <ferringb@chromium.org>
Reviewed-by: David James <davidjames@chromium.org>
When running on NFS, the root user may not be able to access ~/.ssh and
~/.gitconfig, so it is necessary to fallback to SUDO_USER to access these
files.
To discourage users from using NFS homedirs, print warnings every time
cros_sdk is run with an NFS homedir.
BUG=chromium-os:36783
TEST=Try cros_sdk --replace and cros_sdk with and without NFS homedirs.
Change-Id: I4cdbceca485d3491656d6f743814da4ebcdd75ad
Reviewed-on: https://gerrit.chromium.org/gerrit/38953
Commit-Ready: David James <davidjames@chromium.org>
Reviewed-by: David James <davidjames@chromium.org>
Tested-by: David James <davidjames@chromium.org>
Explicitly build curl/openssl/git since the toolchain itself tries to
fetch over http with git.
BUG=None
TEST=`cros_sdk --bootstrap` works
TEST=`cbuildbot chromiumos-sdk` works
Change-Id: I50b3145732f8345d6ad6ada41325648cbea31b84
Reviewed-on: https://gerrit.chromium.org/gerrit/36995
Tested-by: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Han Shen <shenhan@chromium.org>
Tested-by: Han Shen <shenhan@chromium.org>
Commit-Ready: Mike Frysinger <vapier@chromium.org>
sudo takes 150ms per invocation on Goobuntu, and with 10 invocations in
enter_chroot.sh, this means that we're wasting a lot of time, every time
cros_sdk is invoked. Cutting these unnecessary invocations reduces the time
required to run enter_chroot.sh from 2.3s to 0.8s.
CL:36618 is the companion change that updates cros_sdk to invoke
sudo unshare -m prior to calling enter_chroot.sh.
Summary of changes:
1. Remove all calls to sudo and just run the commands directly.
- Remove the mount queue and any sudo_multi optimizations.
- Rename sudo_chroot -> bare_chroot because we don't run sudo anymore there.
- Remove code for validating sudo timestamp.
2. Allow the scripts to work as root:
- Ensure that files created by cros_sdk that previously were owned by the
user still are owned by the user (either using chown or cp -p).
- Use $SUDO_USER to find the user's account.
- Use $SUDO_HOME instead of $HOME to find the user's home dir.
- Remove outdated code for disabling automount on Lucid, which doesn't work
when run as root.
- Update code for calculating the user's git username to use sudo to switch
to the user. Also move it to make_chroot.sh so that this change doesn't
impact performance.
3. Cleanup
- Remove environment syncer process in favor of just syncing once when chroot
is entered.
- Remove teardown and instead rely on unshare to unmount the mounts. To make
sure that outside processes never notice the mounts, we use mount -n. This
also ensures that /etc/mtab never contains stale mounts.
- Remove path-overrides, since it is no longer needed.
BUG=chromium-os:35714, chromium-os:35679
TEST=Trybot runs.
CQ-DEPEND=CL:36618
Change-Id: I919a8aadb08fafde97348e8511573c28fdd47186
Reviewed-on: https://gerrit.chromium.org/gerrit/36619
Tested-by: David James <davidjames@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Ready: David James <davidjames@chromium.org>
Users sometimes want to run gclient inside the chroot, so we shouldn't
tell users that using it is a bad idea.
The original reason why this message was added is historical: Originally,
users had a newer version of SVN inside the chroot compared to on their
workstation, so if you ran SVN inside the chroot it would permanently upgrade
your working copy such that the version of SVN outside the chroot did not work
with it anymore. This isn't a problem anymore, so we can remove the message.
BUG=none
TEST=Run remote trybot runs of chromiumos-sdk
Change-Id: I7b82a5c94e29d5928f4bb296ae2d99cef397d365
Reviewed-on: https://gerrit.chromium.org/gerrit/36346
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Ready: David James <davidjames@chromium.org>
Tested-by: David James <davidjames@chromium.org>
Building the chroot environment from sources using
"--bootstrap" currently runs into a circular dependency:
curl->openssl->git->curl
The openssl->git dependency comes indirectly from the fact
that the current version of openssl uses the "cros-workon"
ebuild package to assist in applying packages. The ebuild
system automatically and silently resolves this circular
dependency by reverting the openssl library to an earlier
version that does not use cros-workon based patching.
Unfortunately this older version of openssl has a bug that
causes it not to work when doing builds in a firewalled
environment: When curl (using this older version of openssl
library) attempts to fetch an "https" url, it authenticates
the target server against a bundle of certificate-authority
certificates it maintains. Finding the certificate fails
(although the validation succeeds if curl is told explicitly
what certificate to use). With the certificate not-found,
server authentication fails, the curl download fails, and
the build ultimately fails.
This patch breaks the circular dependency, allowing a
more-current version of openssl to be used in curl, making
the above build scenario work in a firewalled environment.
The circularity is broken by first building git without curl
support (and webdav that depends on curl). Then early
toolchain components up through and including curl are
built. This build of curl then uses a more up-to-date
version of openssl with the desired bug-fix. Once curl is
built, then git is re-built and re-installed with the
now-installed version of curl (re-)enabled.
BUG=None
TEST=create chroot with --bootstrap ; build_packages (behind firewall)
Change-Id: Iaa560fdb6623fcb73cde066a3b2bc2a342169c62
Reviewed-on: https://gerrit.chromium.org/gerrit/34292
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Ready: paul drews <paul.drews@intel.com>
Tested-by: paul drews <paul.drews@intel.com>
This is forced by cros_sdk; in conjunction w/ this,
drop --distfiles and mangle the chroot on during entrance
dropping a symlink in the old /var/cache/distfiles location
pointing to the new mounted cache_dir location.
Additionally, thread CHROMEOS_CACHEDIR down through the end.
Do this without relying on a version upgrade script- we can't
require they be run before entering, thus we exploit the fact
that cros_sdk explicitly forces a write lock to do the upgrade,
if we see the old form we know we can do the upgrade w/out
worrying about collisions.
CQ-DEPEND=CL:33871
BUG=chromium-os:34457
TEST=manual testing.
Change-Id: I6805266e3ec683f05d3ba615f9e8840642a28e48
Reviewed-on: https://gerrit.chromium.org/gerrit/33868
Commit-Ready: Brian Harring <ferringb@chromium.org>
Reviewed-by: Brian Harring <ferringb@chromium.org>
Tested-by: Brian Harring <ferringb@chromium.org>
These are a new type of credential baked into chrome/chromium for
developers without internal copies of ChromeOS, and not building internal
versions of Chrome.
We automatically move .googleapikeys into the chroot each time.
We don't overwrite the destination, so that people can configure keys the
way they want. If they just don't want to be bothered, the best thing happens
the easiest way. Get Keys, put them in home. Keep working.
BUG=chromium-os:34438
TEST=local
Change-Id: I08e5970c6092f7b789aa5efef52db93841996d8f
Reviewed-on: https://gerrit.chromium.org/gerrit/33771
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Ready: Peter Mayo <petermayo@chromium.org>
Tested-by: Peter Mayo <petermayo@chromium.org>
Do this via ensuring that any common.sh invoker
of raw umount (say a root script) sees our umount
path.
Additionally, inject into default profiles our override,
and via an upgrade scriptlet.
This is round two; originally appeared as CL:32088, was
reverted due to:
https://uberchromegw.corp.google.com/i/chromiumos/builders/chromiumos%20sdk/builds/2314/steps/BuildBoard/logs/stdio
The fix however is just adding a single sudo mkdir. :/
BUG=chromium-os:23443
TEST=cros_sdk --replace --bootstrap
TEST=cros_sdk --replace
Change-Id: I0dc7522a9c623f40081d4f138cea0c2c45171fea
Reviewed-on: https://gerrit.chromium.org/gerrit/32365
Commit-Ready: Brian Harring <ferringb@chromium.org>
Tested-by: Brian Harring <ferringb@chromium.org>
Reviewed-by: Chris Sosa <sosa@chromium.org>
Do this via ensuring that any common.sh invoker
of raw umount (say a root script) sees our umount
path.
Additionally, inject into default profiles our override,
and via an upgrade scriptlet.
BUG=chromium-os:23443
TEST=manual validation, trybot.
Change-Id: Ie2514f6e8d2e10a19ab8d11c8056177bc1a2fb4d
Reviewed-on: https://gerrit.chromium.org/gerrit/32088
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Ready: Brian Harring <ferringb@chromium.org>
Tested-by: Brian Harring <ferringb@chromium.org>
In particular, put the sudoers.d setup into one script (making
updates to it easier in the future if necessary), and
centralize the proxied vars into a const in common.sh.
Thanks to Kevin McCray/Josh Triplett/Alexander Kanevsky for
pointing out the missing proxy variables, and fixes/cleanup.
BUG=None
TEST=https_proxy=blah cros_sdk -- bash -c 'echo $https_proxy'
TEST=build_packages behind a proxy.
TEST=cros_sdk --replace && \
RSYNC_PROXY=blah cros_sdk -- bash -c 'echo $RSYNC_PROXY'
Change-Id: I3165882dfd9c8b52d25c2b26d7ff9242c84c91bd
Reviewed-on: https://gerrit.chromium.org/gerrit/31185
Tested-by: Brian Harring <ferringb@chromium.org>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Ready: Josh Triplett <josh@joshtriplett.org>
This is used to build toolchains with specific env variables
BUG=chromium-os:33240
TEST=trybot x86-generic-toolchain-minor
Change-Id: I2bbdd7d013a15c57c590a0d660a210e0ae2a6695
Reviewed-on: https://gerrit.chromium.org/gerrit/30645
Tested-by: Zdenek Behan <zbehan@chromium.org>
Reviewed-by: David James <davidjames@chromium.org>
Commit-Ready: Zdenek Behan <zbehan@chromium.org>
Make the following commands send stats to chromiumos-build-stats.appspot.com:
build_image
run_chroot_version_hooks
make_chroot
setup_board
update_chroot
BUG=chromium-os:33088
TEST=`cbuildbot --remote -p chromiumos/platform/crosutils alex-paladin`
confirm in the log that uploads succeded, and see them show up in queries
at chromiumos-build-stats.appspot.com.
Change-Id: I0280f91a3e7e0a0483c01c87072bc589003dbe95
Reviewed-on: https://gerrit.chromium.org/gerrit/28969
Tested-by: Matt Tennant <mtennant@chromium.org>
Reviewed-by: David James <davidjames@chromium.org>
Commit-Ready: Matt Tennant <mtennant@chromium.org>
This places a simple hook in cros_sdk by executing commands
in $HOME/.cros_chroot_init when a chroot is built. This
lets users copy files to the chroot (for instance, scripts), as
well as add lines to .bash_profile and other rc files.
BUG=chromium-os:31295
TEST=tested manually and verified .cros_chroot_init is executed when present
Change-Id: I2a5b070a9827272f7bb7b3d340ad6937b0bef329
Reviewed-on: https://gerrit.chromium.org/gerrit/23668
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Tested-by: Luigi Semenzato <semenzato@chromium.org>
Commit-Ready: Luigi Semenzato <semenzato@chromium.org>
BUG=chromium-os:19287
TEST=try building a chroot both using .tbz2, .tar.bz2 and .tar.xz
Change-Id: Idfb13b691201b65c1fa1d5f8597f2aaa401a4051
Reviewed-on: https://gerrit.chromium.org/gerrit/23964
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Ready: Zdenek Behan <zbehan@chromium.org>
Tested-by: Zdenek Behan <zbehan@chromium.org>
For the builders/goobuntu, they're running pbzip2 ~1.0.5 w/ tar 1.22;
for whatever reason, that configuration reproducibly limits to single
core for:
tar -I /usr/bin/pbzip2 -xf /the/sdk
This is annoying; 2 minutes instead of 10s for 48 core builder for
example. Thus does *not* occur w/in the chroot (differing versions),
nor for tar=1.26 w/ pbzip2 1.1.6. The changelogs for both programs
are a bit spartan, but I'm suspecting tar just wasn't feeding it
particularly well (pbzip2 1.0.5 will parallelize if stdin is a pipe).
Regardless, we either try to force everyone to upgrade, or we just
use a form that behaves fine, which is what this CL does.
BUG=chromium-os:31320
TEST=manual validation of it.
Change-Id: I77a434bd2c70873459cbf373192fe73feadb2547
Reviewed-on: https://gerrit.chromium.org/gerrit/23811
Tested-by: Brian Harring <ferringb@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Ready: Brian Harring <ferringb@chromium.org>
Now that this is part of hard-host-depends, an we don't build the chroot
itself with ccache, so there's no need to force it in early.
BUG=None
TEST=`cbuildbot chromiumos-sdk` passed
Change-Id: I8b7c2a8c6f6df5eedac0c06ebb847f3011eb86d0
Reviewed-on: https://gerrit.chromium.org/gerrit/22954
Reviewed-by: Anush Elangovan <anush@google.com>
Commit-Ready: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
Resubmit of If62b4f3973f02fd8e1deed35864c824a02ab0c22
This will be safe to land after
I2c4e21ec7e8c0c0cf58947e2b0a3a9edf7617a09
The breakage was a timing issue paired with people not always syncing
the complete tree. No changes to the CL are needed.
It is now used for:
- make_chroot (cros_sdk --bootstrap)
- update_chroot
setup_board is stripped of redundant code which was deprecated by this.
Also stripped is some usepkg logic in make_chroot, as that is now
exclusively source-only.
BUG=chromium-os:23032
TEST=trybot chromiumos-sdk
Change-Id: Ib888cf2886218622d9cfeebb17b9cd4462d06c89
Reviewed-on: https://gerrit.chromium.org/gerrit/22578
Tested-by: Zdenek Behan <zbehan@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Reviewed-by: asharif <asharif@chromium.org>
Commit-Ready: Zdenek Behan <zbehan@chromium.org>
The "function" keyword is superfluous, not in POSIX, is inconsistent
between bash files, and generally makes me angry. So convert every
instance to the form:
foo() {
BUG=None
TEST=`cbuildbot x86-generic-paladin` works
Change-Id: I97f5ca30a3edfef7222b1e08ac23917dc613b556
Reviewed-on: https://gerrit.chromium.org/gerrit/22467
Reviewed-by: David James <davidjames@chromium.org>
Commit-Ready: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
Currently, if set -e spots a nonzero exit we basically have
no real debug information- it just stops immediately without stating
where or why. This forces our scripts to be stupidly verbose so
we can track roughly where they were, thus when they fail we can
use that information to localize the rough exit point.
Instead we should be traping that set -e induced exit and
outputing necessary debug information to run it down. This includes
outputing the relevant stack trace, or at least what we can get of
it.
The 'die' function is now enhanced to automatically dump the trace
that lead to it. For most consumers this is desired- however for
commandline parsing induced dies ("--board is missing" for example),
the trace is noise. For those cases, a 'die_notrace' function was
added that retains the original non-backtrace behaviour.
Example output via instrumenting cros_generate_breakpad_symbols
w/ the failing command '/bin/false' (nonzero exit code).
Before:
./cros_generate_breakpad_symbols monkeys --board=x86-alex
<no output at all, just exit code 1>
With this CL:
./cros_generate_breakpad_symbols monkeys --board=x86-alex
ERROR : script called: ./cros_generate_breakpad_symbols 'monkeys' '--board=x86-alex'
ERROR : Backtrace: (most recent call is last)
ERROR : file cros_generate_breakpad_symbols, line 207, called: main 'monkeys' '--board=x86-alex'
ERROR : file cros_generate_breakpad_symbols, line 163, called: die_err_trap '/bin/false' '1'
ERROR :
ERROR : Command failed:
ERROR : Command '/bin/false' exited with nonzero code: 1
BUG=chromium-os:30598
TEST=inject a failing command into a script, verify the output.
TEST=inject a 'command not found', verify the output
TEST=cbuildbot x86-generic-full --remote
TEST=cbuildbot arm-tegra2-full --remote
TEST=cbuildbot chromiumos-sdk --remote
Change-Id: I517ffde4d1bb7e2310a74f5a6455b53ba2dea86c
Reviewed-on: https://gerrit.chromium.org/gerrit/17225
Reviewed-by: Brian Harring <ferringb@chromium.org>
Tested-by: Brian Harring <ferringb@chromium.org>
Commit-Ready: Brian Harring <ferringb@chromium.org>
