This code is not applicable to us, it predates CoreOS and is a weird
thing for common.sh to be doing as well. Instead always define
CHROOT_TRUNK_DIR to /mnt/host/source, create ~/trunk in make_chroot.
Currently building images on older kernels will fail because mkfs.btrfs
enables an incompatible feature 'extref' by default. We never really
made this requirement explicit and the SDK in general has continued to
maintain compatibility with older kernels. Make the requirement explicit
so users will get errors quicker and there is a clear line for what
kernel features can be used in the SDK.
Newer git ebuilds have decided that the "git-prompt" script isn't really
bash completion so stopped installing it via that mechanism. Instead it
installed it started installing it in /usr/share/docs which gets
compressed by default and the path is based on ebuild version. The path
changed again in 1.9.3 to /usr/share/git and didn't compress it so that
makes it actually possibly usable but 1.9.3 or later isn't stable yet.
We can re-enable it the next time git gets updated but not worth fussing
over the current brokenness right now.
Using parallel_emerge has been disabled by default for all commands
except build_image for quite a while now, build_image kept it just
because it was still a bit faster than normal emerge. Keeping
parallel_emerge complicates future changes to build_image so it needs to
drop it entirely. Since that means nothing uses it by default we might
as well just rip out support for it entirely.
Previously /etc/os-release was installed both by set_lsb_release and
the baselayout package. Now it is only installed by set_lsb_release but
when baselayout is upgraded it removes /etc/os-release. So the first
update_chroot works but the second detects the chroot's version
incorrectly and tries to apply the one time updates in this directory.
Both of them are very old so we can just delete them. The second run
will now fix up /etc/os-release and we can all move on and be happy.
The host system's PATH may not be match the one required by the SDK.
When going through the enter_chroot script it gets reset because bash is
invoked as a login shell but this doesn't happen when using the plain
old chroot command.
Fixes https://github.com/coreos/scripts/pull/290
There is no need to arbitrarily bind mount all of the host system's /run
into the chroot. In fact this causes issues when the host system's /run
isn't set up in a way this script anticipates. Namely the user runtime
directory in /run/usr/$UID is another tmpfs mount on my system, leaving
the underlying directory node that is bind-mounted in with the wrong
ownership. Behave a little more like a responsible container and use a
fresh /run but continue binding /run/shm for whatever versions of Ubuntu
that depended on that behavior. Not strictly needed but go ahead and
create the user runtime directory with the correct permissions.
- Don't copy known_hosts if it doesn't exist.
- Don't bother with copying *.pub, not sure what that was for.
- Don't rewrite .ssh/config to remove internal Google ssh options.
The main case here is /etc/hosts does not exist on CoreOS. In the
process combine related and duplicate code. Setting the timezone now
happens in entire_chroot like hosts and resolv.conf. Don't bother with
setting a default UTC time zone, that is already the default.
- Automated builds drop SDK and binary packages into
gs://builds.developer.core-os.net/ and the new download URL is
http://builds.developer.core-os.net/ (COREOS_DEV_BUILDS)
- Change default upload path to gs://users.developer.core-os.net/ for
misc developer builds. Official builds go elsewhere and will just be
configured in buildbot/jenkins so some COREOS_OFFICIAL stuff is gone.
- Automated builds of images go to a private bucket,
gs://builds.release.core-os.net which later gets copied to
gs://alpha.release.core-os.net and friends by core_promote.
To behave more like setup_board/build_packages update_chroot should
fully configure portage to make sure everything is accurate.
Now binhosts are defined in make.conf.host_setup so the static config in
coreos-overlays doesn't need to refer to version.txt. setup_board
already made this change in 7a43a07f.
Define path locations to reduce dependency between static configs in
coreos-overlays and the behavior of the scripts repo. Spreading
configuration across two repos makes everything harder to understand.
Eventually everything should either be defined in profiles in
coreos-overlays or minimal auto-generated config files here in scripts.
For the most part this doesn't influence anything. The one exception is
the custom configuration for using curl is dropped, just rely on the
portage defaults. It appears curl was only used to work around a wget
issue with Google's internal SSL certificates. We care not. :)
The commands useradd/usermod will silently skip adding users to
secondary groups that are not in /etc/group. The idea being that the
tools should not create groups that conflict with existing LDAP/NIS
groups but why trying to do so isn't a fatal error I don't know.
Overall the code is rather complicated and tries to modify instead of
add when possible to allow running the SDK as the 'core' user. To keep
things simple gut this code, make the 'core' user special, and add
secondary groups via the 'gpasswd' command so that errors are reported
instead of silently ignored.
One functional change: the default groups have changed to kvm and
portage. The old list excluded kvm and included lots of extra cruft.
Make it possible for other scripts to share the same value for our
release repository and equally easy to override with a custom value.
Also allow setting the root from the command line in addition to the
environment. Usually --upload_root is better to use than --upload_path.
This makes it possible to toggle parallel_emerge just as other scripts
do. In other scripts update the help string to be more specific, the
--jobs option can be used to control parallelism.
Instead of handling toolchain packages in make_chroot and telling
update_chroot to skip the toolchains just depend on update_chroot to do
it properly. Reduces our code duplication by a tiny but worthwhile bit.
The current logic for downloading SDK tarballs is in cros_sdk and
written in python which isn't super convenient for re-using in the rest
of our shell scripts. This is a start of rewriting that logic into a
re-usable library but does not yet replace the functionality in cros_sdk.
We've had trouble with eclean and equery vanishing in our SDKs from time
to time. Although I don't know the root cause it seemed to be some
confusion in the ebuild environment, perhaps a mis-match between the
eclasses, profiles, and ebuilds. Updating all of those seemed to resolve
the issue and to make sure other environments are ok force a re-install
of portage and gentoolkit to clean things up.
When a user creates a chroot and as a common primary group such as
'users' the groupadd command fails. Instead treat this the same as users
and only fail if the group exists but has a different (such as the
'users' group not using GID 100). Hopefully this works better.
If the user already exists check that the UID and GID are correct and
modify it (setting shell and home directory) to match what the SDK
expects. This avoids needlessly failing if the user calling cros_sdk is
the 'core' user on a CoreOS machine.
Change new-user creation to copy the user's full name and group instead
of using a generic name and Google's 'eng' group. Also remove the
default password for the account, it isn't needed and uses perl.
The old chroot version system we inherited from Chromium OS always
assumes that a newly unpacked tarball is the latest and greatest but
since we version the SDK in the same way as target builds we can use
that version for these sorts of upgrade scripts and not make assumptions
about how late and great the starting tarball was.
The first upgrade script simply aborts to force the user to recreate
their chroot when moving from python 2.6 to 2.7.
Our SDK tarballs aren't compressed using pbzip2 so there is no advantage
to using pbzip2 to decompress them over bzip2, however lbzip2 does offer
a big advantage. Also trust that the portage config defines a valid
version of bzip2 since we have control over the tarball creation and can
make sure to always include required utilities.
The existing code seems to assume that the mounts inherited from the
system are private, the Linux default. However on our systems that
clearly isn't the case, all system mounts are set as shared. Considering
all of us have been have been seeing mounts leak out of the SDK despite
cros_sdk creating a new filesystem namespace via unshare I'm guessing
this is a systemd thing.
Instead force all system mounts to 'slave' mode in the SDK namespace so
global changes are still visible but no SDK mounts can leak out.
Already did this for catalyst builds but might as well do it for all.
With this competing builds on the same host should be a little
friendlier to each other.
Half of the current time is spent on calling locale-gen even when there is
nothing to be done (all locales already generated). Throw it into the bg
to unblock the main thread.
BUG=None
TEST=`cros_sdk` still works
TEST=`LANG=et_EE.UTF-8 cros_sdk` generates the new locale in the background
Change-Id: Ibe9a07bec60a59cab1cf4230358f7f8ff5b21c2e
Reviewed-on: https://gerrit.chromium.org/gerrit/58041
Reviewed-by: David James <davidjames@chromium.org>
Commit-Queue: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
All devpts mounts are actually shared, even if you do:
mount -t devpts none /dev/pts
mount -t devpts none /mnt/foo
mount -t devpts none /mnt/asdfasdf
These all provide the same data.
This is problematic because most distros mount their host devpts like so:
mount -t devpts devpts /dev/pts -o mode=620,gid=5
But when cros_sdk runs, it uses:
mount -t devpts none /dev/pts
We aren't specifying a mode/gid, so it ends up using the defaults, and
this resets the host devpts mount as well.
Since we've already assumed that the system has devpts available, it's
fine to also assume that the system has it mounted at /dev/pts and we
can simply bind mount it.
BUG=None
TEST=`cros_sdk` no longer messes up host perms on /dev/pts
Change-Id: Ib594fc5e47707f296d97ac1edce32659ed2b2273
Reviewed-on: https://gerrit.chromium.org/gerrit/48018
Reviewed-by: Steev Klimaszewski <threeway@gmail.com>
Reviewed-by: David James <davidjames@chromium.org>
Commit-Queue: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
I use a mount at src/build/images to stop image builds from repeatedly
filling up my SSD. The chroot needs to respect this.
TEST=cros_sdk
BUG=none
Change-Id: I5c7a26c3b4f263bd683d3a897e6edccb83187bda
Reviewed-on: https://gerrit.chromium.org/gerrit/47178
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Queue: Michael Spang <spang@chromium.org>
Tested-by: Michael Spang <spang@chromium.org>
Newer Gentoo builds have moved to /run which means /var/lock is a symlink
to /run/lock. But since that is an absolute symlink, it points outside of
the chroot which doesn't work for us. Use a stable path unrelated to the
chroot instead, but only with newer chroots.
We no longer have to worry about backwards compat because the code that
used to rely on this lock file (running sync processes) was punted a long
time ago.
BUG=chromium:218085
TEST=`cbuildbot chromiumos-sdk` passes
Change-Id: I38c6848dfb86386849050d7ccf3f90cbbe8e0e81
Reviewed-on: https://gerrit.chromium.org/gerrit/46231
Reviewed-by: David James <davidjames@chromium.org>
Commit-Queue: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
This patch installs "socat" and a proxy gateway script into
the chroot so that git can use a proxy to access "git://"
protocol urls. This is needed when performing builds from
behind a firewall that requires a proxy. The script reads
the proxy environment variables all_proxy (SOCKS),
https_proxy (CONNECT), and http_proxy (CONNECT), in order of
preference, and supports no_proxy as a whitelist of target
hosts that must NOT go through the proxy.
This also updates enter_chroot.sh to automatically use this
script as GIT_PROXY_COMMAND when it sees the proxy
environment variables set.
The "socat" program is added to hard-host-depends as a
separate patch. That handles socat installation in case of
building a chroot from scratch or upgrading.
The proxy-gw script is installed in the src/scripts/bin
directory which can be stably referenced within the chroot
as /mnt/host/source/src/scripts/bin/. The
"/mnt/host/source" portion of this path is obtained from the
CHROOT_TRUNK_DIR environment variable which is set to a
suitable value by preexisting logic in common.sh.
This change became necessary to unbreak builds behind
proxies with the recent addition of two ebuilds using
egit.eclass with repositories using git:// URLs.
Original patch by Paul Drews <paul.drews@intel.com>;
modified version by Josh Triplett <josh@joshtriplett.org>.
CQ-DEPENDS=I1b01bce6f3e6a562b87f748e61508d142af576d9
BUG=none
TEST=git clone git://nv-tegra.nvidia.com/tools/cbootimage.git
Change-Id: Ic7fc917d1aa24f408bef6f102b6458114dded694
Reviewed-on: https://gerrit.chromium.org/gerrit/41659
Tested-by: paul drews <paul.drews@intel.com>
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Queue: paul drews <paul.drews@intel.com>
In an Ubuntu Precise chroot on the Chromebook Pixel, /run/shm is a
symbolic link to /dev/shm, so bind-mounting /run/shm to /dev/shm
is really bind-mounting /dev/shm to itself, which causes a 'too many
levels of symbolic links' error. To fix this, we check for a symbolic
link prior to running this command.
BUG=none
TEST=cros_sdk no longer prints errors on Chromebook Pixel
Change-Id: Ib46cde2b4a0e00b69bd187488967e445b228ae80
Reviewed-on: https://gerrit.chromium.org/gerrit/45048
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Queue: David James <davidjames@chromium.org>
Tested-by: David James <davidjames@chromium.org>
If ~/.subversion doesn't exist, the code didn't run, but if it existed
there is no reason to re-create it, nor is it necessary to change its
permissions since they are inherited by the bind mount source.
However user_mkdir was trying to run chown as root which does not work
over NFS with root_squash or krb-nfs.
Therefore, the un-needed call to user_mkdir is removed.
(this is an issue because cros_sdk --replace does call this code path
multiple times).
BUG=None
TEST=Built the chroot, and the permission denied on 'install' went away.
Change-Id: I01e9a7baf51a99a96d790c9613e26e652379e6df
Reviewed-on: https://gerrit.chromium.org/gerrit/44880
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Queue: Marc MERLIN <merlin@chromium.org>
Tested-by: Marc MERLIN <merlin@chromium.org>
If our sdk has an /etc/mtab file already, then clobber it. This fixes
build problems where chromeos-base now installs /etc/mtab for us, but
the sdk build isn't expecting it leading to the error:
INFO cros_sdk:make_chroot: Running init_setup()...
ln: creating symbolic link `/b/cbuild/new-sdk-chroot/etc/mtab': File exists
Running ['/b/cbuild/src/scripts/sdk_lib/make_chroot.sh', '--stage3_path',
'/b/cbuild/built-sdk.tar.xz', '--chroot', '/b/cbuild/new-sdk-chroot',
'--cache_dir', '/b/cbuild/.cache', '--nousepkg'] failed!
BUG=None
TEST=`cros_sdk --chroot foo` still works
Change-Id: I539cf329e93e28534e6ff00577ce415d76918b85
Reviewed-on: https://gerrit.chromium.org/gerrit/43641
Reviewed-by: David James <davidjames@chromium.org>
Commit-Queue: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
As-is all of the various emerge wrapping scripts default to using
--getbinpkg whenever --usepkg is enabled. This means every single emerge
command made makes multiple synchronous HTTP requests to the upstream
binary package repository to get the latest package list. This gets
really frustrating when working remotely with limited network
connectivity. Using --usepkg with --nogetbinpkg will use locally cached
packages without making remote requests.
use the efunctions package for the /etc/init.d/functions.sh script
instead of backing up the old function.sh which doesn't work with the
new baselayout
we remove openrc which provides /etc/init.d/functions.sh. Unfortunatly
other things rely on this file. Stash it away in /tmp/ then restore it
for now.
Change-Id: I18a59e05ecdf08cc8a560b29049c8d25ac1bf5a3
This seems to be needed for acessing some of the chrome repositories.
Without it we get git clone hangs trying to sync.
BUG=chromium-os:38303
TEST=local entry into chroot
Change-Id: Ia68a6486022e8d230572bad0f9031c3e5d36197c
Reviewed-on: https://gerrit.chromium.org/gerrit/42140
Commit-Queue: Peter Mayo <petermayo@chromium.org>
Reviewed-by: Peter Mayo <petermayo@chromium.org>
Tested-by: Peter Mayo <petermayo@chromium.org>
After CL:39921, I get the following warning every time I enter the chroot:
ln: failed to create symbolic link `.../chroot/root/.boto': File exists
All bots get this error as well. This is caused because CL:39921, causes
~/trunk to no longer resolve outside the chroot, so it's invalid for processes
outside the chroot to try to resolve paths inside there. Fix cases where we do
this inside enter_chroot.sh.
BUG=chromium-os:37347
TEST=cros_sdk doesn't print warnings anymore.
Change-Id: Iaeb9b7407e12397bce1600bd51559be20f998fdf
Reviewed-on: https://gerrit.chromium.org/gerrit/41571
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Queue: David James <davidjames@chromium.org>
Tested-by: David James <davidjames@chromium.org>
Slipped past during rename of the chroot upgrade script from
49 to 50; name was slightly changed but full re-validation of the
rename wasn't done (thus the typo slipped past testing, and review).
Simplify the code via removal of invoking the upgrade script, instead
just doing the relevant commands (fixing chroot awareness issues in
the process).
BUG=None
TEST=manual cros_sdk invocation
Change-Id: I122de8b4cf7ec0845643e09e7919cbcdbd0bb79a
Reviewed-on: https://gerrit.chromium.org/gerrit/41202
Reviewed-by: Brian Harring <ferringb@chromium.org>
Tested-by: Brian Harring <ferringb@chromium.org>
Rather than having to find /home/${SUDO_USER:-${USER}}/trunk, instead
just look for /mnt/host/trunk (defined by common.sh as $CHROOT_TRUNK_DIR).
This simplifies code flow, and is a requirement for shoving chromite
into PYTHONPATH globally w/in the chroot.
BUG=chromium-os:37347
TEST=cros_sdk --replace; cros_sdk w/ chroot upgrade.
Change-Id: I9ee3e6556541a91193f49cbf74ffc5a8e090537f
Reviewed-on: https://gerrit.chromium.org/gerrit/39921
Tested-by: Brian Harring <ferringb@chromium.org>
Reviewed-by: David James <davidjames@chromium.org>
Over time, stale ssh agent dirs build up in /tmp. Have enter_chroot run
a simple rmdir to clean out any empty dirs. Since we mount over top the
dir, this shouldn't kill any valid mount points.
BUG=None
TEST=`cros_sdk` cleaned out empty ssh dirs in /tmp
Change-Id: Ib9f063f99db61825082818a39a39c5eb01f2d24e
Reviewed-on: https://gerrit.chromium.org/gerrit/39004
Reviewed-by: David James <davidjames@chromium.org>
Reviewed-by: Matt Tennant <mtennant@chromium.org>
Reviewed-by: Brian Harring <ferringb@chromium.org>
Commit-Ready: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
When running on NFS, the root user may not be able to access ~/.ssh and
~/.gitconfig, so it is necessary to fallback to SUDO_USER to access these
files.
To discourage users from using NFS homedirs, print warnings every time
cros_sdk is run with an NFS homedir.
BUG=chromium-os:36783
TEST=Try cros_sdk --replace and cros_sdk with and without NFS homedirs.
Change-Id: I4cdbceca485d3491656d6f743814da4ebcdd75ad
Reviewed-on: https://gerrit.chromium.org/gerrit/38953
Commit-Ready: David James <davidjames@chromium.org>
Reviewed-by: David James <davidjames@chromium.org>
Tested-by: David James <davidjames@chromium.org>
Explicitly build curl/openssl/git since the toolchain itself tries to
fetch over http with git.
BUG=None
TEST=`cros_sdk --bootstrap` works
TEST=`cbuildbot chromiumos-sdk` works
Change-Id: I50b3145732f8345d6ad6ada41325648cbea31b84
Reviewed-on: https://gerrit.chromium.org/gerrit/36995
Tested-by: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Han Shen <shenhan@chromium.org>
Tested-by: Han Shen <shenhan@chromium.org>
Commit-Ready: Mike Frysinger <vapier@chromium.org>
sudo takes 150ms per invocation on Goobuntu, and with 10 invocations in
enter_chroot.sh, this means that we're wasting a lot of time, every time
cros_sdk is invoked. Cutting these unnecessary invocations reduces the time
required to run enter_chroot.sh from 2.3s to 0.8s.
CL:36618 is the companion change that updates cros_sdk to invoke
sudo unshare -m prior to calling enter_chroot.sh.
Summary of changes:
1. Remove all calls to sudo and just run the commands directly.
- Remove the mount queue and any sudo_multi optimizations.
- Rename sudo_chroot -> bare_chroot because we don't run sudo anymore there.
- Remove code for validating sudo timestamp.
2. Allow the scripts to work as root:
- Ensure that files created by cros_sdk that previously were owned by the
user still are owned by the user (either using chown or cp -p).
- Use $SUDO_USER to find the user's account.
- Use $SUDO_HOME instead of $HOME to find the user's home dir.
- Remove outdated code for disabling automount on Lucid, which doesn't work
when run as root.
- Update code for calculating the user's git username to use sudo to switch
to the user. Also move it to make_chroot.sh so that this change doesn't
impact performance.
3. Cleanup
- Remove environment syncer process in favor of just syncing once when chroot
is entered.
- Remove teardown and instead rely on unshare to unmount the mounts. To make
sure that outside processes never notice the mounts, we use mount -n. This
also ensures that /etc/mtab never contains stale mounts.
- Remove path-overrides, since it is no longer needed.
BUG=chromium-os:35714, chromium-os:35679
TEST=Trybot runs.
CQ-DEPEND=CL:36618
Change-Id: I919a8aadb08fafde97348e8511573c28fdd47186
Reviewed-on: https://gerrit.chromium.org/gerrit/36619
Tested-by: David James <davidjames@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Ready: David James <davidjames@chromium.org>
Users sometimes want to run gclient inside the chroot, so we shouldn't
tell users that using it is a bad idea.
The original reason why this message was added is historical: Originally,
users had a newer version of SVN inside the chroot compared to on their
workstation, so if you ran SVN inside the chroot it would permanently upgrade
your working copy such that the version of SVN outside the chroot did not work
with it anymore. This isn't a problem anymore, so we can remove the message.
BUG=none
TEST=Run remote trybot runs of chromiumos-sdk
Change-Id: I7b82a5c94e29d5928f4bb296ae2d99cef397d365
Reviewed-on: https://gerrit.chromium.org/gerrit/36346
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Ready: David James <davidjames@chromium.org>
Tested-by: David James <davidjames@chromium.org>
The process of bootstrapping the chroot from sources was
failing for several reasons when run from behind a firewall
with proxies. The llvm build was failing due to inability
to checkout sources through subversion using the
subversion.eclass wrapper (the "normal" way to do this in
the ebuild environment). This was because the user's
subversion configuration (including proxy settings) was not
inherited from $HOME/.subversion into the in-chroot sandbox
used by subversion.eclass.
This change creates symbolic links in the subversion.eclass
sandboxes for host and target builds in the chroot to fix
any build that uses the normal subversion.eclass for
checkouts. The operation is done at enter_chroot time so
that it applies to both ordinary builds and chroot creation
(via early_enter_chroot).
BUG=none
TEST='cros_sdk --replace --enter' behind proxied firewall
Change-Id: I0af2128866bb95799dc07c728c75cf3f2a0af7a3
Reviewed-on: https://gerrit.chromium.org/gerrit/34291
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Ready: paul drews <paul.drews@intel.com>
Tested-by: paul drews <paul.drews@intel.com>
Building the chroot environment from sources using
"--bootstrap" currently runs into a circular dependency:
curl->openssl->git->curl
The openssl->git dependency comes indirectly from the fact
that the current version of openssl uses the "cros-workon"
ebuild package to assist in applying packages. The ebuild
system automatically and silently resolves this circular
dependency by reverting the openssl library to an earlier
version that does not use cros-workon based patching.
Unfortunately this older version of openssl has a bug that
causes it not to work when doing builds in a firewalled
environment: When curl (using this older version of openssl
library) attempts to fetch an "https" url, it authenticates
the target server against a bundle of certificate-authority
certificates it maintains. Finding the certificate fails
(although the validation succeeds if curl is told explicitly
what certificate to use). With the certificate not-found,
server authentication fails, the curl download fails, and
the build ultimately fails.
This patch breaks the circular dependency, allowing a
more-current version of openssl to be used in curl, making
the above build scenario work in a firewalled environment.
The circularity is broken by first building git without curl
support (and webdav that depends on curl). Then early
toolchain components up through and including curl are
built. This build of curl then uses a more up-to-date
version of openssl with the desired bug-fix. Once curl is
built, then git is re-built and re-installed with the
now-installed version of curl (re-)enabled.
BUG=None
TEST=create chroot with --bootstrap ; build_packages (behind firewall)
Change-Id: Iaa560fdb6623fcb73cde066a3b2bc2a342169c62
Reviewed-on: https://gerrit.chromium.org/gerrit/34292
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Ready: paul drews <paul.drews@intel.com>
Tested-by: paul drews <paul.drews@intel.com>
CL:33868 (7b6f377c58) introduced a
breakage in the "cros_sdk --replace --bootstrap" scenario.
The make_chroot.sh script invokes early_enter_chroot before
invoking init_setup. The chroot/etc/profiles.d directory is
created in init_setup, but the referenced change was
expecting to create a file in that directory in the context
of early_enter_chroot before the directory was created.
This led to a "no such file or directory" error when trying
to create the file.
This change does a "mkdir -p" of the referenced directory
before putting things in it in the context of
early_enter_chroot. The filename is also fixed to the name
expected elsewhere in the scripts.
BUG=none
TEST=cros_sdk --replace --bootstrap
Change-Id: I6ac0467117d7b0dd413695153469b367d56c256c
Reviewed-on: https://gerrit.chromium.org/gerrit/34958
Commit-Ready: Brian Harring <ferringb@chromium.org>
Reviewed-by: Brian Harring <ferringb@chromium.org>
Tested-by: Brian Harring <ferringb@chromium.org>
This is forced by cros_sdk; in conjunction w/ this,
drop --distfiles and mangle the chroot on during entrance
dropping a symlink in the old /var/cache/distfiles location
pointing to the new mounted cache_dir location.
Additionally, thread CHROMEOS_CACHEDIR down through the end.
Do this without relying on a version upgrade script- we can't
require they be run before entering, thus we exploit the fact
that cros_sdk explicitly forces a write lock to do the upgrade,
if we see the old form we know we can do the upgrade w/out
worrying about collisions.
CQ-DEPEND=CL:33871
BUG=chromium-os:34457
TEST=manual testing.
Change-Id: I6805266e3ec683f05d3ba615f9e8840642a28e48
Reviewed-on: https://gerrit.chromium.org/gerrit/33868
Commit-Ready: Brian Harring <ferringb@chromium.org>
Reviewed-by: Brian Harring <ferringb@chromium.org>
Tested-by: Brian Harring <ferringb@chromium.org>
enter_chroot.sh was not updating /etc/hosts from the out-of-chroot
environment. Make it do that.
BUG=None
TEST=locally
Change-Id: Ieaa337ae90dbc0700c42fa7e4b96faf12d3968cb
Reviewed-on: https://gerrit.chromium.org/gerrit/34226
Reviewed-by: David James <davidjames@chromium.org>
Commit-Ready: Ryan Cui <rcui@chromium.org>
Tested-by: Ryan Cui <rcui@chromium.org>
This change was coopted from http://codereview.chromium.org/5331009/,
originally written by hungte@. And the coopted commit message:
It would be helpful if we could share some directories inside/outside the
chroot (e.g. editor configuration or the default Downloads directory). This
CL reads .local_mounts (just like .default_boards) from the "src/scripts"
folder, and mounts the directories whenever you do cros_sdk.
For safety concern, and to prevent the developer from accidentally deleting
their mounted files, the mounts are made read-only.
.local_mounts has a very simple syntax:
mount_path
or source_path(outside chroot) destination_path(inside chroot)
or # comments.
Examples:
/usr/share/vim/google
/home/XXX/Downloads /outside
BUG=chromium-os:34561
TEST=Manually:
1. Create ~/trunk/src/scripts/.local_mounts with following content:
# comment here
/usr/share/vim/google # test
/home/XXX/Downloads /outside
2. cros_sdk
3. ls -l /usr/share/vim/google/ # ensure dir is mounted correctly
ls -l /outside/ # ensure dir is mounted correctly
4. exit
5. mount | grep chroot # ensure nothing is left
Change-Id: I6f3400a436a825e8cdfcb18b788afe96ebba6757
Reviewed-on: https://gerrit.chromium.org/gerrit/33585
Tested-by: Michael Krebs <mkrebs@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Ready: Michael Krebs <mkrebs@chromium.org>
These are a new type of credential baked into chrome/chromium for
developers without internal copies of ChromeOS, and not building internal
versions of Chrome.
We automatically move .googleapikeys into the chroot each time.
We don't overwrite the destination, so that people can configure keys the
way they want. If they just don't want to be bothered, the best thing happens
the easiest way. Get Keys, put them in home. Keep working.
BUG=chromium-os:34438
TEST=local
Change-Id: I08e5970c6092f7b789aa5efef52db93841996d8f
Reviewed-on: https://gerrit.chromium.org/gerrit/33771
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Ready: Peter Mayo <petermayo@chromium.org>
Tested-by: Peter Mayo <petermayo@chromium.org>
Since sudo changes $HOME to /root, sudoed invocations of gsutil/boto
won't find the necessary credentials. This solves the problem by
installing a symlink at /root/.boto to the correct credentials file,
similar to how it's done for /home/$USER/.boto.
BUG=None
TEST=/root/.boto symlink created upon entering the chroot
Change-Id: I541556f836fa5d0b9708e5604218058401563fb3
Reviewed-on: https://gerrit.chromium.org/gerrit/32430
Reviewed-by: David James <davidjames@chromium.org>
Reviewed-by: Ryan Cui <rcui@chromium.org>
Commit-Ready: Gilad Arnold <garnold@chromium.org>
Tested-by: Gilad Arnold <garnold@chromium.org>
Do this via ensuring that any common.sh invoker
of raw umount (say a root script) sees our umount
path.
Additionally, inject into default profiles our override,
and via an upgrade scriptlet.
This is round two; originally appeared as CL:32088, was
reverted due to:
https://uberchromegw.corp.google.com/i/chromiumos/builders/chromiumos%20sdk/builds/2314/steps/BuildBoard/logs/stdio
The fix however is just adding a single sudo mkdir. :/
BUG=chromium-os:23443
TEST=cros_sdk --replace --bootstrap
TEST=cros_sdk --replace
Change-Id: I0dc7522a9c623f40081d4f138cea0c2c45171fea
Reviewed-on: https://gerrit.chromium.org/gerrit/32365
Commit-Ready: Brian Harring <ferringb@chromium.org>
Tested-by: Brian Harring <ferringb@chromium.org>
Reviewed-by: Chris Sosa <sosa@chromium.org>
Do this via ensuring that any common.sh invoker
of raw umount (say a root script) sees our umount
path.
Additionally, inject into default profiles our override,
and via an upgrade scriptlet.
BUG=chromium-os:23443
TEST=manual validation, trybot.
Change-Id: Ie2514f6e8d2e10a19ab8d11c8056177bc1a2fb4d
Reviewed-on: https://gerrit.chromium.org/gerrit/32088
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Ready: Brian Harring <ferringb@chromium.org>
Tested-by: Brian Harring <ferringb@chromium.org>
In particular, put the sudoers.d setup into one script (making
updates to it easier in the future if necessary), and
centralize the proxied vars into a const in common.sh.
Thanks to Kevin McCray/Josh Triplett/Alexander Kanevsky for
pointing out the missing proxy variables, and fixes/cleanup.
BUG=None
TEST=https_proxy=blah cros_sdk -- bash -c 'echo $https_proxy'
TEST=build_packages behind a proxy.
TEST=cros_sdk --replace && \
RSYNC_PROXY=blah cros_sdk -- bash -c 'echo $RSYNC_PROXY'
Change-Id: I3165882dfd9c8b52d25c2b26d7ff9242c84c91bd
Reviewed-on: https://gerrit.chromium.org/gerrit/31185
Tested-by: Brian Harring <ferringb@chromium.org>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Ready: Josh Triplett <josh@joshtriplett.org>
The code has been in here long enough - most people should be
transitioned over who are using gerrit-source. We've also already
removed the chrome projects from the default manifest, and things look
good so far.
BUG=chromium-os:32963
TEST=remote trybot
Change-Id: Idd5e3a2ad77ea86c7316a9d50f5da1a5fdf01d8b
Reviewed-on: https://gerrit.chromium.org/gerrit/31161
Reviewed-by: Brian Harring <ferringb@chromium.org>
Commit-Ready: Ryan Cui <rcui@chromium.org>
Tested-by: Ryan Cui <rcui@chromium.org>
This is used to build toolchains with specific env variables
BUG=chromium-os:33240
TEST=trybot x86-generic-toolchain-minor
Change-Id: I2bbdd7d013a15c57c590a0d660a210e0ae2a6695
Reviewed-on: https://gerrit.chromium.org/gerrit/30645
Tested-by: Zdenek Behan <zbehan@chromium.org>
Reviewed-by: David James <davidjames@chromium.org>
Commit-Ready: Zdenek Behan <zbehan@chromium.org>
Don't run .bash_logout after invocation of the hook, which clears the
screen, sending unnecessary escape characters.
BUG=None
TEST=Ran locally.
Change-Id: I6c466040e7169d304b892b85be6a5b0d578e7714
Reviewed-on: https://gerrit.chromium.org/gerrit/29645
Reviewed-by: David James <davidjames@chromium.org>
Commit-Ready: Ryan Cui <rcui@chromium.org>
Tested-by: Ryan Cui <rcui@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Make the following commands send stats to chromiumos-build-stats.appspot.com:
build_image
run_chroot_version_hooks
make_chroot
setup_board
update_chroot
BUG=chromium-os:33088
TEST=`cbuildbot --remote -p chromiumos/platform/crosutils alex-paladin`
confirm in the log that uploads succeded, and see them show up in queries
at chromiumos-build-stats.appspot.com.
Change-Id: I0280f91a3e7e0a0483c01c87072bc589003dbe95
Reviewed-on: https://gerrit.chromium.org/gerrit/28969
Tested-by: Matt Tennant <mtennant@chromium.org>
Reviewed-by: David James <davidjames@chromium.org>
Commit-Ready: Matt Tennant <mtennant@chromium.org>
See the bug for details.
BUG=chromium-os:32963.
TEST=Locally, remote trybots.
Change-Id: I33f5c42b36f3e06139036c299c2fc2c2ff026411
Reviewed-on: https://gerrit.chromium.org/gerrit/28543
Reviewed-by: David James <davidjames@chromium.org>
Commit-Ready: Ryan Cui <rcui@chromium.org>
Tested-by: Ryan Cui <rcui@chromium.org>
Originally, I patched enter_chroot.sh to stop the gvfs daemons to work
around an issue where these daemons would prevent loop devices from being
unmounted. See https://bugzilla.gnome.org/show_bug.cgi?id=677648
Unfortunately, temporarily stopping gvfs daemons has a bad side effects:
other GUI applications that rely on these daemons responding start hanging.
This can be reproduced, for example, by starting 'gedit'.
To fix these hangs, I'm just reverting my patches to enter_chroot.sh and
restoring the scripts to where they were before.
This reverts the following patches:
1. Stop gvfs daemons earlier during enter_chroot.
This reverts commit 0079158f73.
2. Revert "Stop the gvfsd-trash daemon during enter_chroot."
This reverts commit 654a00bd61.
3. Revert "Stop the automounting daemon whenever we're inside the chroot."
This reverts commit fae0a59e8b.
4. Revert "Clean up update_bootloaders.sh to avoid sleeping."
This reverts commit 0103b59138.
BUG=chromium-os:23443
TEST=Trybot run.
Change-Id: Ie9ff222fe5fc7232fd1fc39af129cc18531118c6
Reviewed-on: https://gerrit.chromium.org/gerrit/26922
Reviewed-by: Chris Wolfe <cwolfe@chromium.org>
Reviewed-by: Jon Kliegman <kliegs@chromium.org>
Reviewed-by: Brian Harring <ferringb@chromium.org>
Commit-Ready: David James <davidjames@chromium.org>
Tested-by: David James <davidjames@chromium.org>
The killall commands for stopping gvfs weren't working
consistently for two reasons:
1) They ran too late, after it already picked up the
mounts in cros_sdk.
2) killall sometimes can only access the first 15 characters
of a process name, so we should only match on these characters.
BUG=chromium-os:23443
TEST=Verify gvfs is properly stopped when entering the chroot
on precise systems.
Change-Id: I16aff4b0d9ac101083b63e06e55d50869479a152
Reviewed-on: https://gerrit.chromium.org/gerrit/26369
Reviewed-by: Pawel Osciak <posciak@google.com>
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Ready: David James <davidjames@chromium.org>
Tested-by: David James <davidjames@chromium.org>
For fresh sdk builds, gcc won't update and automatically see the
ccache tree for us. So make sure the perms are sane when people
enter the chroot. This will also automatically fix perms if/when
people manually delete the ccache dir (which sometimes happens on
the buildbots when people try to free up space).
BUG=None
TEST=`rm -rf distfiles/ccache/; cros_sdk` and see ccache dir get setup nicely
Change-Id: I5bcc86ebf696549b142a7ceb312eb8ec4be5e2bf
Reviewed-on: https://gerrit.chromium.org/gerrit/26257
Reviewed-by: David James <davidjames@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
Commit-Ready: Mike Frysinger <vapier@chromium.org>
This changelist adds code in common.sh to support collecting command
statistics before calling upload_command_stats to upload those stats
to an appspot instance.
The presence of a file at ~/.disable_build_stats_upload will disable
all uploading of build command stats.
BUG=chromium-os:27355
TEST=`build_packages --board=x86-generic` with missing appspot instance
shows upload error but still goes through with build.
TEST=`build_packages --board=x86-generic` with real appspot instance
completes upload with all expected stats and does not affect build.
TEST=From outside chroot:
`touch ~/.disable_build_stats_upload`
`cros_sdk`
'build_packages --board=x86-generic`
Nothing uploaded due to .disable file
TEST=Verified that putting 'set -u' in my fake build_packages
caused an exit in print_time_elapsed, then fixed unbound variable
in print_time_elapsed, then rerun passed.
TEST=`cbuildbot -g <cl> --lkgm mario-paladin` passed
TEST=`cbuildbot -g <cl> --lkgm link-release` passed
Change-Id: Ieb714522cb32d7558b661e4ee1a197d1fce2c516
Reviewed-on: https://gerrit.chromium.org/gerrit/26084
Tested-by: Matt Tennant <mtennant@chromium.org>
Reviewed-by: David James <davidjames@chromium.org>
Commit-Ready: Matt Tennant <mtennant@chromium.org>
This reverts commit 615fe57ff0.
This CL broke mod_image_for_recovery.sh with the following failure:
common.sh: line 684: $2: unbound variable
This bug is easy to fix, but suggests the CL needs more testing, so
we're going to revert for now and re-submit the CL once it's been
verified to pass on release trybots.
Example failure:
http://chromegw.corp.google.com/i/chromeos/builders/x86-mario%20canary/builds/2214/steps/Archive/logs/stdio
BUG=chromium-os:27355
TEST=None, since this is reverting a change that broke the tree.
Change-Id: I61d182e3dcee267a8d9dea3b547fa6a75140d974
Reviewed-on: https://gerrit.chromium.org/gerrit/26077
Reviewed-by: Doug Anderson <dianders@chromium.org>
Tested-by: David James <davidjames@chromium.org>
This changelist adds code in common.sh to support collecting command
statistics before calling upload_command_stats to upload those stats
to an appspot instance.
The presence of a file at ~/.disable_build_stats_upload will disable
all uploading of build command stats.
BUG=chromium-os:27355
TEST=`build_packages --board=x86-generic` with missing appspot instance
shows upload error but still goes through with build.
TEST=`build_packages --board=x86-generic` with real appspot instance
completes upload with all expected stats and does not affect build.
TEST=From outside chroot:
`touch ~/.disable_build_stats_upload`
`cros_sdk`
'build_packages --board=x86-generic`
Nothing uploaded due to .disable file
Change-Id: Iac071d1cc55a44335fc7c846960c7ae45fc93ed8
Reviewed-on: https://gerrit.chromium.org/gerrit/19401
Tested-by: Matt Tennant <mtennant@chromium.org>
Reviewed-by: Matt Tennant <mtennant@chromium.org>
Commit-Ready: Matt Tennant <mtennant@chromium.org>
