This changelist adds code in common.sh to support collecting command
statistics before calling upload_command_stats to upload those stats
to an appspot instance.
The presence of a file at ~/.disable_build_stats_upload will disable
all uploading of build command stats.
BUG=chromium-os:27355
TEST=`build_packages --board=x86-generic` with missing appspot instance
shows upload error but still goes through with build.
TEST=`build_packages --board=x86-generic` with real appspot instance
completes upload with all expected stats and does not affect build.
TEST=From outside chroot:
`touch ~/.disable_build_stats_upload`
`cros_sdk`
'build_packages --board=x86-generic`
Nothing uploaded due to .disable file
TEST=Verified that putting 'set -u' in my fake build_packages
caused an exit in print_time_elapsed, then fixed unbound variable
in print_time_elapsed, then rerun passed.
TEST=`cbuildbot -g <cl> --lkgm mario-paladin` passed
TEST=`cbuildbot -g <cl> --lkgm link-release` passed
Change-Id: Ieb714522cb32d7558b661e4ee1a197d1fce2c516
Reviewed-on: https://gerrit.chromium.org/gerrit/26084
Tested-by: Matt Tennant <mtennant@chromium.org>
Reviewed-by: David James <davidjames@chromium.org>
Commit-Ready: Matt Tennant <mtennant@chromium.org>
This reverts commit 615fe57ff0.
This CL broke mod_image_for_recovery.sh with the following failure:
common.sh: line 684: $2: unbound variable
This bug is easy to fix, but suggests the CL needs more testing, so
we're going to revert for now and re-submit the CL once it's been
verified to pass on release trybots.
Example failure:
http://chromegw.corp.google.com/i/chromeos/builders/x86-mario%20canary/builds/2214/steps/Archive/logs/stdio
BUG=chromium-os:27355
TEST=None, since this is reverting a change that broke the tree.
Change-Id: I61d182e3dcee267a8d9dea3b547fa6a75140d974
Reviewed-on: https://gerrit.chromium.org/gerrit/26077
Reviewed-by: Doug Anderson <dianders@chromium.org>
Tested-by: David James <davidjames@chromium.org>
This changelist adds code in common.sh to support collecting command
statistics before calling upload_command_stats to upload those stats
to an appspot instance.
The presence of a file at ~/.disable_build_stats_upload will disable
all uploading of build command stats.
BUG=chromium-os:27355
TEST=`build_packages --board=x86-generic` with missing appspot instance
shows upload error but still goes through with build.
TEST=`build_packages --board=x86-generic` with real appspot instance
completes upload with all expected stats and does not affect build.
TEST=From outside chroot:
`touch ~/.disable_build_stats_upload`
`cros_sdk`
'build_packages --board=x86-generic`
Nothing uploaded due to .disable file
Change-Id: Iac071d1cc55a44335fc7c846960c7ae45fc93ed8
Reviewed-on: https://gerrit.chromium.org/gerrit/19401
Tested-by: Matt Tennant <mtennant@chromium.org>
Reviewed-by: Matt Tennant <mtennant@chromium.org>
Commit-Ready: Matt Tennant <mtennant@chromium.org>
The "function" keyword is superfluous, not in POSIX, is inconsistent
between bash files, and generally makes me angry. So convert every
instance to the form:
foo() {
BUG=None
TEST=`cbuildbot x86-generic-paladin` works
Change-Id: I97f5ca30a3edfef7222b1e08ac23917dc613b556
Reviewed-on: https://gerrit.chromium.org/gerrit/22467
Reviewed-by: David James <davidjames@chromium.org>
Commit-Ready: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
Specifically, do this via env for the time being since each program
doesn't necessarily have an option (nor warrant one).
BUG=None
TEST=None
Change-Id: I26e7f06ad5d6a44a7826bfa8465b34154d21b6a3
Reviewed-on: https://gerrit.chromium.org/gerrit/22295
Tested-by: Brian Harring <ferringb@chromium.org>
Commit-Ready: Brian Harring <ferringb@chromium.org>
Reviewed-by: David James <davidjames@chromium.org>
Currently, if set -e spots a nonzero exit we basically have
no real debug information- it just stops immediately without stating
where or why. This forces our scripts to be stupidly verbose so
we can track roughly where they were, thus when they fail we can
use that information to localize the rough exit point.
Instead we should be traping that set -e induced exit and
outputing necessary debug information to run it down. This includes
outputing the relevant stack trace, or at least what we can get of
it.
The 'die' function is now enhanced to automatically dump the trace
that lead to it. For most consumers this is desired- however for
commandline parsing induced dies ("--board is missing" for example),
the trace is noise. For those cases, a 'die_notrace' function was
added that retains the original non-backtrace behaviour.
Example output via instrumenting cros_generate_breakpad_symbols
w/ the failing command '/bin/false' (nonzero exit code).
Before:
./cros_generate_breakpad_symbols monkeys --board=x86-alex
<no output at all, just exit code 1>
With this CL:
./cros_generate_breakpad_symbols monkeys --board=x86-alex
ERROR : script called: ./cros_generate_breakpad_symbols 'monkeys' '--board=x86-alex'
ERROR : Backtrace: (most recent call is last)
ERROR : file cros_generate_breakpad_symbols, line 207, called: main 'monkeys' '--board=x86-alex'
ERROR : file cros_generate_breakpad_symbols, line 163, called: die_err_trap '/bin/false' '1'
ERROR :
ERROR : Command failed:
ERROR : Command '/bin/false' exited with nonzero code: 1
BUG=chromium-os:30598
TEST=inject a failing command into a script, verify the output.
TEST=inject a 'command not found', verify the output
TEST=cbuildbot x86-generic-full --remote
TEST=cbuildbot arm-tegra2-full --remote
TEST=cbuildbot chromiumos-sdk --remote
Change-Id: I517ffde4d1bb7e2310a74f5a6455b53ba2dea86c
Reviewed-on: https://gerrit.chromium.org/gerrit/17225
Reviewed-by: Brian Harring <ferringb@chromium.org>
Tested-by: Brian Harring <ferringb@chromium.org>
Commit-Ready: Brian Harring <ferringb@chromium.org>
Using /proc/mounts is safer because mtab may in rare cases get desynced.
The only significant difference in output is "X Y" instead of "X on Y".
BUG=chromium-os:30249
TEST=create a chroot; enter a chroot; exit a chroot
TEST=assortment of manual tests
Change-Id: I392290e6f52a677ee2d77d77e025ef60240b11b5
Reviewed-on: https://gerrit.chromium.org/gerrit/21499
Tested-by: Zdenek Behan <zbehan@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Ready: Zdenek Behan <zbehan@chromium.org>
Upstream Gentoo renamed this virtual, so follow suite.
BUG=None
TEST=`cbuildbot amd64-generic-full` worked
TEST=`cbuildbot arm-tegra2-full` worked
TEST=`cbuildbot x86-generic-full` worked
Change-Id: I2721c85fe83f4ee8a90533eda14813697430e98e
Reviewed-on: https://gerrit.chromium.org/gerrit/18781
Reviewed-by: Olof Johansson <olofj@chromium.org>
Reviewed-by: Sonny Rao <sonnyrao@chromium.org>
Commit-Ready: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
This CL adds uinput module in INSTALL_MASK so that the module
is not installed to non-test image. A mod_image_for_test script
is implemented to install the module in the test image if the
uinput module exists.
BUG=chromium-os:26707
TEST=Build a base image and a test image. Check the directory
/lib/modules/<version>/kernel/drivers/input/misc
where current <version> is 3.2.7.
In the base image, there should be no uinput.ko,
while in the test image, there should be uinput.ko.
CQ-DEPEND=Ie96242c4d56403866a2298db2ba3bd6459248c1b
Change-Id: I0ca6599f80b9bb72cdc044fc97cdf990ce550edc
Reviewed-on: https://gerrit.chromium.org/gerrit/19032
Reviewed-by: Kees Cook <keescook@chromium.org>
Reviewed-by: Mandeep Singh Baines <msb@chromium.org>
Tested-by: Joseph Shyh-In Hwang <josephsih@chromium.org>
Commit-Ready: Joseph Shyh-In Hwang <josephsih@chromium.org>
This CL adds uinput module in INSTALL_MASK so that the module
is not installed to non-test image. A mod_image_for_test script
is implemented to install the module in test image.
BUG=chromium-os:26707
TEST=Build a base image and a test image. Check the directory
/lib/modules/<version>/kernel/drivers/input/misc
where current <version> is 3.2.7.
In the base image, there should be no uinput.ko,
while in the test image, there should be uinput.ko.
Change-Id: I02b557466a56e11c5dcc1649a9275d0c2a896f09
Reviewed-on: https://gerrit.chromium.org/gerrit/17209
Reviewed-by: Kees Cook <keescook@chromium.org>
Reviewed-by: Mandeep Singh Baines <msb@chromium.org>
Tested-by: Joseph Shyh-In Hwang <josephsih@chromium.org>
Commit-Ready: Joseph Shyh-In Hwang <josephsih@chromium.org>
For all scripts, --build_root defaults to /build and is never used. To
remove option clutter, I've deleted this option.
The only script which still references build_root in src/scripts is
mod_image_for_pyauto.sh, which seems to support it for grabbing pyauto
dependencies from a location other than /build/*. This might conceivably
be useful, so I haven't touched that script.
BUG=chromium-os:27364
TEST=Remote trybot run. git grep for references to build_root.
Change-Id: I502f7df0123a598fc62a4ef4ed847ceb182f65b8
Reviewed-on: https://gerrit.chromium.org/gerrit/18283
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Tested-by: David James <davidjames@chromium.org>
Commit-Ready: David James <davidjames@chromium.org>
1. Remove options that are already deprecated:
--chromefromsource --chromebuild --chromebase
--crosbug12352_arm_kernel_signing
2. Remove the --retries option and associated function eretry.
3. Move seldomly used options out of --help to avoid confusing
developers. Developers who need these options can read the
source.
4. Alphabetize all options.
5. Add description of each script to --help.
BUG=chromium-os:27364
TEST=Run --help with each script. Full canary trybot run.
Change-Id: I95675b069781f7e950d75d32dbad744adce6b830
Reviewed-on: https://gerrit.chromium.org/gerrit/18194
Reviewed-by: David James <davidjames@chromium.org>
Tested-by: David James <davidjames@chromium.org>
Commit-Ready: David James <davidjames@chromium.org>
Using these flags directly was deprecated a while back and is causing confusion
which way is the right way. This CL removes all these FLAGS from build_image
(though keeps support of them in mod_image_for_test/image_to_usb invocations
in common.sh). In this change I've also defaulted build_image to build the
developer image (and only the developer image).
BUG=chromium-os:27362
TEST=Been busy testing. Have built the following combinations and verified them:
build_image base
build_image
build_image base dev test
build_image dev
build_image dev factory_test
build_image factory_install
Change-Id: Ie534c276a9ec571926964320ac176daa91b12a81
Reviewed-on: https://gerrit.chromium.org/gerrit/17386
Tested-by: Chris Sosa <sosa@chromium.org>
Reviewed-by: Richard Barnette <jrbarnette@chromium.org>
Commit-Ready: Chris Sosa <sosa@chromium.org>
With more interactive features appearing in the scripts, we need to
avoid hanging buildbots waiting for user input. This changes adds
utility functions to test whether and ensure that the execution is
interactive, and adds an assertion to the 'choose' function.
is_interactive: Check that both stdin and stderr are terminals.
assert_interactive: Die with an error if not interactive
Also tweaking 'choose' so that its menu is printed to stderr,
for consistency with read -p and select.
BUG=None
TEST=Modified setup_board to log the value of is_interactive, and
then (unconditionally) attempt to display a menu.
Running setup_board in a terminal showed interactive and the menu.
Manually run cbuildbot x86-generic-pfq showed non-interactive and
the assertion failure.
Change-Id: I768a37b8b85ce036edac7c9bb0fd1e0a2b92ecd5
Reviewed-on: https://gerrit.chromium.org/gerrit/16817
Commit-Ready: Chris Wolfe <cwolfe@chromium.org>
Reviewed-by: Chris Wolfe <cwolfe@chromium.org>
Tested-by: Chris Wolfe <cwolfe@chromium.org>
The --jobs=<n> option wasn't actually being passed to parallel_emerge --
which meant it didn't do anything.
BUG=chromium-os:26827
TEST=Ran with --jobs=N and made sure only N parallel_emerge processes were
started.
Change-Id: I581fc5588b54e246acaefd0c7e528e55adf9ba8a
Reviewed-on: https://gerrit.chromium.org/gerrit/16570
Reviewed-by: Michael Krebs <mkrebs@chromium.org>
Commit-Ready: David James <davidjames@chromium.org>
Tested-by: David James <davidjames@chromium.org>
Defined a bash function choose(), which allows to present a menu and
prompting for a selection, with support for default choice. Usage is
commented appropriately.
Originally reviewed in CL I0d2f20dc8d62ce5fa18c10d9f8b51a46b2ddca5d.
BUG=chromium-os:26010
TEST=Tested in conjunction with a use in image_to_usb.sh, works fine.
Change-Id: I53a42a46a3c90fd486fead578bfbae248f64cfc2
Reviewed-on: https://gerrit.chromium.org/gerrit/15586
Reviewed-by: Gilad Arnold <garnold@chromium.org>
Tested-by: Gilad Arnold <garnold@chromium.org>
Commit-Ready: Gilad Arnold <garnold@chromium.org>
First, add a enable_strict_sudo helpers that scripts that are
sudo strict can invoke. This does a sanity check bailing immediately
if invoked from a non sudo-keep-alive context.
Second, update enter_chroot.sh and make_chroot.sh to be strict.
While this is strict, that's the point. The aim of this is to block
the previous sudo interactive mess for spreading, let alone reappearing
in scripts/code that has been cleansed.
BUG=chromium-os:18393
TEST=cros_sdk --replace; in the midst of it, do sudo -k.
cros_sdk should thus bail out w/ an appropriate error.
CQ-DEPEND=I01bb1466cf027401fa387af7fad15e42fd33aea4
Change-Id: I76c5b87a812cc78c30a2eb1a0c56b9e438f4a98f
Reviewed-on: https://gerrit.chromium.org/gerrit/15294
Commit-Ready: Brian Harring <ferringb@chromium.org>
Reviewed-by: Brian Harring <ferringb@chromium.org>
Tested-by: Brian Harring <ferringb@chromium.org>
This is a preparatory step for changes to image_to_usb.sh and
mod_image_for_recovery.sh. In general, by migrating the default
recovery image name to common.sh we can improve the interoperability
between the different scripts.
BUG=chromium-os:26010
TEST=Obvious change; tested to see the mod_image_for_recovery.sh and
image_to_usb.sh still work.
Change-Id: I765d665b656d5d25b2d9b0e40bf5c05048c3f255
Reviewed-on: https://gerrit.chromium.org/gerrit/15507
Commit-Ready: Gilad Arnold <garnold@chromium.org>
Reviewed-by: Gilad Arnold <garnold@chromium.org>
Tested-by: Gilad Arnold <garnold@chromium.org>
When we source the local copy of shflags, we don't check for errors like
we do when sourcing the external one. Add a `die` call for that too.
BUG=None
TEST=`rm lib/shflags/shflags && (. ./common.sh)` now exits properly
Change-Id: I62c9b33dd96f2934691c43d53dea012097116ca1
Reviewed-on: https://gerrit.chromium.org/gerrit/15443
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
Commit-Ready: Mike Frysinger <vapier@chromium.org>
The shflags sourcing logic tries to use `die` before we've defined it, so
if there's an issue, we end up with:
bash: die: command not found
and the shflags code doesn't actually get loaded.
So relocate these small helpers to the top so we don't have to worry about
when it's safe to use these things.
BUG=None
TEST=`rm lib/shflags/shflags && (. ./common.sh)` now exits properly
Change-Id: Ibdc268e6c081aa07679dc9fce76e5603b7217b20
Reviewed-on: https://gerrit.chromium.org/gerrit/15442
Reviewed-by: Brian Harring <ferringb@chromium.org>
Commit-Ready: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
The factory test image was created by using rsync from build artifacts
in chroot, which has some concerns:
- Runtime dependencies of autotest-factory won't be picked into image.
- If a developer skips build_package and builds image by using only pre-built
binary packages, he will get nothing in factory test image.
- It's hard for developers to figure out how and when his changes will be
merged to next build_image (cros_workon does not really work).
- Output image will be definitely different for every developers, also the
official build bots.
- If developers never wipes his chroot (setup_board), the factory test image
will grow until out of space. (For example, my environment outputs a 825M
image while the official buildbot generates only 563M for same ToT source).
This CL changes image build command to using portage emerge, so that output
image can be prepared faster and smaller, and easier for maintenance.
BUG=chromium-os:3335
TEST=./build_packages; ./build_image --factory # Image starts factory UI successfully
# Also tried tests in test_list.all, seems fine.
time ./mod_image_for_test.sh --factory --force_copy --no_inplace
# time: 3m2s => 1m55s, factory test image data: 825/563M => 378M
./build_image --factory_install # factory install shim is also fine
Change-Id: I82b4505c74cd31e718aaff4a319d50b69b2c852c
Reviewed-on: https://gerrit.chromium.org/gerrit/14473
Tested-by: Hung-Te Lin <hungte@chromium.org>
Reviewed-by: Richard Barnette <jrbarnette@chromium.org>
Commit-Ready: Hung-Te Lin <hungte@chromium.org>
If people have whitespace in their .default_board setting, then the cros
utils will act weirdly. Some work the same as if there was no whitespace
(probably because the variable is used unquoted) while others error out
with weird messages (because the variable is used quoted).
Update the helper function to only allow certain characters in the name.
BUG=None
TEST=`printf 'x86-alex\n' > .default_board; cros_workon list --all` works
TEST=`printf 'x86-alex\t\n' > .default_board; cros_workon list --all` errors out
Change-Id: Id83794c13bfddb7fb56b7f8ed8a375eefe6096e0
Reviewed-on: https://gerrit.chromium.org/gerrit/13151
Reviewed-by: Jason Glasgow <jglasgow@chromium.org>
Reviewed-by: David James <davidjames@chromium.org>
Commit-Ready: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
Every invocation of `sudo` delays things, so merge all of the mounts
into a single `sudo` command when possible. This saves over 1 second
on initial execution (out of ~4 seconds total).
BUG=None
TEST=`cros_sdk --enter true` still mounts & unmounts properly
Change-Id: Ibc66507dc21250a81207d2f940645eaebe93c79c
Reviewed-on: https://gerrit.chromium.org/gerrit/10901
Reviewed-by: David James <davidjames@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
Commit-Ready: Mike Frysinger <vapier@chromium.org>
This partially reverts and partially clarifies the code in the above
commit. It was still in use, just in an obscure place, so a reference
to that has been added.
BUG=chromium-os:21971
TEST=below
1) build_lib/generate-au-zip.py
2) cd /tmp/au-generator/; unzip au-generator.zip
3) exit chroot; cd $chroot/tmp/au-generator/
4) LD_LIBRARY_PATH="$(pwd)" PATH="${PATH}:$(pwd)" ./cros_generate_update_payload --image something_i_had --output x --outside_chroot
Change-Id: Idfc03937c4f3517c8567efcdebd1cb8ce134fc8e
Reviewed-on: https://gerrit.chromium.org/gerrit/11019
Tested-by: Zdenek Behan <zbehan@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Ready: Zdenek Behan <zbehan@chromium.org>
Fixed all callers in my big CL to no longer have to wrap poorly using
\ and start from the beginning of the next line but rather pass in
an array of args i.e.
info "tacos" " are" " delicious".
BUG=None
TEST=Ran to see the errors in parse_build_image and manual eyeing.
Change-Id: I5eac8a5ae7a8d314dbc4e821ee33cf88213711d0
Reviewed-on: https://gerrit.chromium.org/gerrit/10823
Commit-Ready: Chris Sosa <sosa@chromium.org>
Reviewed-by: Chris Sosa <sosa@chromium.org>
Tested-by: Chris Sosa <sosa@chromium.org>
This CL does two things.
First it introduces an argument interface that works with all the following
combinations:
factory_install | [base|dev[test|factory_test]] rather than use --test etc.
This does not build extra images. If you just want a test image you can
run build_image --board=<board> test and all that's in your latest_dir is
chromiumos_test_image.bin.
Second, we only build what we ask and only finalize what we ask so on
an invocation of build_image test we only actually run cros_make_image_bootable
once saving 2 minutes on my machine (from 6:50->4:50 on multiple iterations).
BUG=chromium-os:22048
TEST=build_image test, all possible combinations of args through to start of
build. Built factory_install using new and old methods and dev test with both
methods. Also verified mod_image_for_test still works as advertises stand-alone.
Change-Id: I9fe2feb50a941c007214decd9ba1627012c050af
Reviewed-on: https://gerrit.chromium.org/gerrit/10621
Commit-Ready: Chris Sosa <sosa@chromium.org>
Reviewed-by: Chris Sosa <sosa@chromium.org>
Tested-by: Chris Sosa <sosa@chromium.org>
Since we need this anyways (can't assume the host outside of the chroot
provides shflags), always source the local copy rather than searching
for versions provided by the chroot.
BUG=chromium-os:21742
TEST=`cros_sdk --enter` works
TEST=`./build_packages --board=amd64-generic` works
Change-Id: Ia390042131f59948472b124cfe4e273483eada6a
Reviewed-on: http://gerrit.chromium.org/gerrit/10231
Reviewed-by: David James <davidjames@chromium.org>
Reviewed-by: Chris Sosa <sosa@chromium.org>
Commit-Ready: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
POSIX says that the first arg to substr() counts from 1 which means
that awk implementations interpret a value of 0 in their own special
way. With gawk, it does the logical thing (imo), but with mawk, it
does it by one less.
Example:
echo abc | mawk '{print substr($0, 0, 2)}' -> a
echo abc | gawk '{print substr($0, 0, 2)}' -> ab
echo abc | mawk '{print substr($0, 1, 2)}' -> ab
echo abc | gawk '{print substr($0, 1, 2)}' -> ab
So stick to the POSIX spec and count from 1.
While we're here, using "len" rather than "l" to improve readability.
URL=http://pubs.opengroup.org/onlinepubs/9699919799/utilities/awk.html
BUG=chrome-os-partner:6026
TEST=set /usr/bin/awk to mawk; run `cros_sdk --enter -- true`; see everything unmounted
TEST=set /usr/bin/awk to gawk; run `cros_sdk --enter -- true`; see everything unmounted
Change-Id: I48f57d642730a0b3d909079027edf6ebf1bb9459
Reviewed-on: http://gerrit.chromium.org/gerrit/8163
Reviewed-by: David James <davidjames@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
Commit-Ready: Mike Frysinger <vapier@chromium.org>
Normally safe_umount_tree isn't called if there are no mount points in the
chroot, but there are some edge cases (like early setup errors) where it
may be possible. So add a simple check to make sure that there are mount
points for us to work on before calling umount.
BUG=None
TEST=add early `die` before mounts; see no umount error output
Change-Id: I57e5022c3242c635097bceed44772f10eecf6a64
Reviewed-on: http://gerrit.chromium.org/gerrit/8026
Reviewed-by: Raymes Khoury <raymes@chromium.org>
Reviewed-by: David James <davidjames@chromium.org>
Commit-Ready: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
The teardown_env is a little slow when unmounting things because it runs
`sudo umount` once per mount point. This is so that when things go wrong,
it can easily recover. However, this slows down the common case at the
expense of the uncommon.
Refactor the code so that in the common case, we run one `sudo umount`.
When things do fail, we're a bit slower as we reparse the entire mount,
list, but that's fine as it's an error case.
BUG=None
TEST=enable `set -x`; run `cros_sdk true`; see all mount points unmounted in one shot
Change-Id: Iec98a7b9f51a77e90c30e6f6acae26e528b8c50d
Reviewed-on: http://gerrit.chromium.org/gerrit/7822
Tested-by: Mike Frysinger <vapier@chromium.org>
Reviewed-by: David James <davidjames@chromium.org>
Also drop find_*_component, because it is not used anywhere anymore.
BUG=chrome-os-partner:5459
TEST=none
Change-Id: Iea49a81538c22fee4b91c8b3e5582b5174282151
Reviewed-on: http://gerrit.chromium.org/gerrit/7425
Tested-by: Stefan Reinauer <reinauer@chromium.org>
Reviewed-by: Anton Staaf <robotboy@chromium.org>
"cros_sdk" is a drop-in replacement of enter_chroot.sh. The only
important difference is in the calling path, specifically that
cros_sdk is in path but has to be called from $(pwd) being inside the
repo checkout, while enter_chroot.sh is called by explicit path.
Invariably, "./enter_chroot.sh" can be replaced, as pwd=src/scripts.
Calling by absolute path can be replaced by first changing directory
anywhere into the repo checkout, and then calling cros_sdk.
BUG=chromium-os:18750
TEST=run them
Change-Id: Ieff91a27bb419e1121361d5b3a11e4c87ff7a087
Reviewed-on: http://gerrit.chromium.org/gerrit/6273
Tested-by: Zdenek Behan <zbehan@chromium.org>
Reviewed-by: Zdenek Behan <zbehan@chromium.org>
Much of this file quotes paths which could possibly contain breaking
chars (e.g. spaces), but a few are missed. In general, this probably
doesn't break as no one is adventurous enough to use spaces, but might
as well do it for the fun of it. And to be consistent. One of those.
BUG=None
TEST=ran make_chroot/enter_chroot/build_packages; seemed to be OK
Change-Id: Ic64d60790f50bb812c87a4672e2a24fafbccd749
Reviewed-on: http://gerrit.chromium.org/gerrit/5759
Tested-by: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>
If you trace any script with `set -x` enabled, bash will dump the color
variables and all of your output will be colored. So move the "reset"
color to last so that the bleeding is localized to a few lines rather
than a few hundred. There should be no functional difference.
BUG=None
TEST=ran setup_board with and w/out -x to create x86-generic to check result
Change-Id: I1e598cc4a38e9f217f2f670a00bd676bf70c2338
Reviewed-on: http://gerrit.chromium.org/gerrit/5804
Reviewed-by: Gaurav Shah <gauravsh@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
The recovery kernel is different from the regular kernel, and
should be kept sandboxed so that it does not mess with our
build directory.
We also need to give the recovery kernel a different PKGDIR so
that it does not overwrite our prebuilts for the ordinary
kernel.
BUG=chromium-os:18691
TEST=build_image --factory_install, mod_image_for_recovery.sh
Change-Id: I678a94305d5f30bc2c95bf4e53cfe81b2ae9d8ef
Reviewed-on: http://gerrit.chromium.org/gerrit/5305
Reviewed-by: David James <davidjames@chromium.org>
Tested-by: David James <davidjames@chromium.org>
This makes it possible for chromeos-kernel to include a source tree,
without the tree being installed to CrOS images.
BUG=None
TEST=Manual with modified chromeos-kernel
Change-Id: I9f1193277638e828ffa116c5a397609a3246c80b
Reviewed-on: http://gerrit.chromium.org/gerrit/4760
Reviewed-by: Chris Wolfe <cwolfe@chromium.org>
Tested-by: Chris Wolfe <cwolfe@chromium.org>
This is the first in three stages of changes to transition from separate
"u-boot" and "coreboot" output directories to a single "firmware"
directory. This change/stage will give scripts a facility to easily be
compatible with both schemes. The second stage will transition all the
scripts to use the mechanism, and then finally the third stage will
switch the directories over. Some additional work may be needed if there
are file name clashes, although those sorts of problems would likely
already be present. These new functions could also optionally be taken back
out after the transition.
This change adds find_fw_component, find_u_boot_component, and
find_coreboot_component functions to common.sh. The first function provides
the bulk of the functionality, while the second and third wrap the first and
pass it parameters appropriate for u-boot or coreboot. The first function
accepts an override which it returns if set, and if not it checks for a file
in each of the directories it's passed in the second parameter one by one and
returns the first one it finds. If it doesn't find any it defaults to the last
one.
BUG=chromium-os:18107
TEST=Modified some u-boot related ebuilds to put some files in the firmware
directory. Ran the ebuilds and verified the files showed up where expected.
Modified the cros_write_firmware.sh script to use these new functions. Ran
cros_write_firmware and saw it find the files in their new location.
Also verified that the firmware on the target was updated as expected.
Change-Id: Ic664e0e1c4a5b27efc9fb892ff5bd91cf1b5be56
Signed-off-by: Gabe Black <gabeblack@google.com>
Reviewed-on: http://gerrit.chromium.org/gerrit/5170
Tested-by: Gabe Black <gabeblack@chromium.org>
Reviewed-by: Doug Anderson <dianders@chromium.org>
This also removes all the tput noise we are seeing in buildbot logs.
("tput: No value for $TERM and no -T specified", "tput: unknown terminal
"unknown", etc.)
BUG=none
TEST=manual below
src/platform/scripts$ TERM=unknown; ./enter_chroot (No more tput errors)
Change-Id: Ic74a0bd66d7e0cedcf7b65ec3a5f43b61b8276f9
Reviewed-on: http://gerrit.chromium.org/gerrit/4763
Tested-by: Gaurav Shah <gauravsh@chromium.org>
Reviewed-by: Richard Barnette <jrbarnette@chromium.org>
BUG=None
TEST=setup_board, build_packages, build_image
TEST=grep all sources for the unused functions
Change-Id: Iaac5589aaba6f6f48aa00cdab76a7765f8d931ce
Reviewed-on: http://gerrit.chromium.org/gerrit/4580
Tested-by: Richard Barnette <jrbarnette@chromium.org>
Reviewed-by: Chris Sosa <sosa@chromium.org>
These hacks are all at least three months old, and our official policy is to
remove hacks after two weeks. For future hacks, the plan is to use the chroot
upgrader script so that old chroots can be supported indefinitely.
Thanks to the Gerrit transition, there are probably few old chroots around
anyway that aren't going to be recreated anyway.
BUG=chromium-os:16151
TEST=Verify enter_chroot.sh still works.
Change-Id: Ie0c777aabbd70d2a0c88b59ddd71e9b67965ca94
Reviewed-on: http://gerrit.chromium.org/gerrit/2158
Reviewed-by: Raymes Khoury <raymes@chromium.org>
Reviewed-by: Zdenek Behan <zbehan@chromium.org>
Tested-by: David James <davidjames@chromium.org>
For run_remote_tests this required "fixing" restart_in_chroot if required. Right now that script ->only<- works if its run on a scripts being called from src/scripts. This change makes it more generic by doing this for any script called from within our source tree.
I borrow reinterpret_path_for_chroot from image_to_live. Other CL to remove it from there.
Change-Id: If717beccd777ac178366d58b91521b9a62d55d85
BUG=chromium-os:11172
TEST=Ran them
Review URL: http://codereview.chromium.org/6730012
prepare_test_image would prepare the correct image (test or factory)
but its return value would always specify the test image. This meant
that image_to_usb.sh would pick up the wrong image.
This commit fixes this.
BUG=chromium-os:12377
TEST=./image_to_usb --from=... --to=/dev/sdb
also with --factory
also with --test
Make sure it picks up the correct image in each case.
Change-Id: I165b604b448a6c938a47e57a667bd5f001626efe
Review URL: http://codereview.chromium.org/6609016
BUG=chromium-os:11981
TEST=run cros_sign_bootstub with and without CROS_LOG_PREFIX set.
Change-Id: I320622d4161479dcc29ebdee68e160d94f1d3196
Review URL: http://codereview.chromium.org/6596108
These functions make it easy to do testing in scripts for required
files and tools and provide useful error messages when the files
or tools are not found.
BUG=chromium-os:11981
TEST=run cros_sign_bootstub in configurations that test all paths in
these functions.
Change-Id: Ia63cfabd1eecba6473331b756486745d71e6b699
Review URL: http://codereview.chromium.org/6591056
The build bots run things a little different it seems. I'll look into
what it is that they do that a normal run doesn't do that causes
this change to fail there. But for now I'm reverting it.
This reverts commit 6d65c0f1b3.
BUG=None
TEST=None
Review URL: http://codereview.chromium.org/6594062
Change-Id: I7cdfb25c28f1c139981aaf7382cd3e9dbfd4446b
BUG=chromium-os:11981
TEST=run cros_sign_bootstub with and without CROS_LOG_PREFIX set.
Change-Id: I320622d4161479dcc29ebdee68e160d94f1d3196
Review URL: http://codereview.chromium.org/6597053
into mod_image_for_test.sh rather than slightly different versions
of the same in image_to_usb.sh and image_to_vm.sh
Added a function to get a test image into common.sh
Added --inplace option to mod_image_for_test, which is the default,
and preserves the original behaviour. But using --noinplace it will
now do the copy for you.
Found that chromiumos_image.bin appears throughout the scripts, so added it and the test variant to common.sh
BUG=chromiumos-10126
TEST=run mod_image_for_test.sh with and without --noinplace
run image_to_usb.sh and image_to_vm.sh with both options
test on Seaboard that correct image is provided
Really we should have automated testing for these scripts
Change-Id: I5cfa91792c7fded35e7f4ca8f8f27c6b270817fb
Committed: http://chrome-svn/viewvc/chromeos?view=rev&revision=4fc5227
Review URL: http://codereview.chromium.org/5271010
from within the chroot.
It also fixes a number of style issues.
It changes the meaning of cros_workon "list-all" to list all available
packages, and adds "list-live" to list all live packages.
It changes things that load chromeos-common.sh from the installer to
load it from /usr/lib/installer.
BUG=chromium-os:4230
TEST=synced, rebuilt chroot, made packages, made images, built chrome
from source, and wrote an image to a USB stick.
Review URL: http://codereview.chromium.org/6240018
Change-Id: I90c34420af1a64020402bafef8e9e77f56837c02
If we are inside the chroot then we assume that the "gclient" root
is ~/trunk. If we are outside the chroot we continue with the
previous search mechanism to find the "gclient" root.
Change-Id: Ia40de609ea596228fec2644ff3046e376b112b06
BUG=chromium-os:4230
TEST=run "src/scripts/cros_overlay_list --board tegra2 --variant seaboard" inside and outside the chroot, and under sudo.
Review URL: http://codereview.chromium.org/6265028
Review URL: http://codereview.chromium.org/6344016
If we are inside the chroot then we assume that the "gclient" root
is ~/trunk. If we are outside the chroot we continue with the
previous search mechanism to find the "gclient" root.
Change-Id: Ia40de609ea596228fec2644ff3046e376b112b06
BUG=chromium-os:4230
TEST=run "src/scripts/cros_overlay_list --board tegra2 --variant seaboard" inside and outside the chroot.
Review URL: http://codereview.chromium.org/6265028
These are used to run virtual or nested X servers, which we
don't do. Removing the binaries saves 3.4 MB.
Change-Id: I528231a781f8a7a0202ca109f8064c3b11ff172b
BUG=chromium-os:8073
TEST=built an image and checked that the files were gone
Review URL: http://codereview.chromium.org/5990003
This reverts the main change of
84e72fab23
and undoes the breakage of scripts that are symlinks to
...chromeos/scripts..., but at the cost of re-breaking scripts under
scripts/bin. The correct fix is a combination of removing all the
symlinks to ...chromeos/scripts... and a more robust way of finding
the chroot path of the executable. But that will have to wait until
someone else does it next week or I return from vacation
TEST=Ran all of these outside chroot. None seemed more broken than when run in the chroot. /setup_board --board=x86-generic, ./cros_workon list --all --board=x86-mario, ./start_devserver , ./build_image --help, ./build_packages --help, ./cros_upgrade_chroot --help, ./enter_chroot.sh , ./customize_rootfs --help, ./run_remote_tests.sh , ./verify_rootfs_chksum.sh , ./set_shared_user_password.sh
BUG=n0ne
Change-Id: Ic2b7484e53e6a977ed6ae675fe99e109b385020d
Review URL: http://codereview.chromium.org/5946003
into mod_image_for_test.sh rather than slightly different versions
of the same in image_to_usb.sh and image_to_vm.sh
Added a function to get a test image into common.sh
Added --inplace option to mod_image_for_test, which is the default,
and preserves the original behaviour. But using --noinplace it will
now do the copy for you.
Found that chromiumos_image.bin appears throughout the scripts, so added it and the test variant to common.sh
BUG=chromiumos-10126
TEST=run mod_image_for_test.sh with and without --noinplace
run image_to_usb.sh and image_to_vm.sh with both options
test on Seaboard that correct image is provided
Really we should have automated testing for these scripts
Change-Id: I5cfa91792c7fded35e7f4ca8f8f27c6b270817fb
Review URL: http://codereview.chromium.org/5271010
BUG=chromium-os:9877
TEST=ran bin/cros_workon_now, cros_workon, set_shared_user_password
Note---this introduces another bashism to common.sh. However, despite
comments to the contrary, common.sh does not actually parse under dash
without this change
Change-Id: I1e6b9751afa8341c100f3b8e0b96284835a53d17
Review URL: http://codereview.chromium.org/5444002
The --fast support for build_packages is stable now, and should significantly
improve the speed of our official builders. We should turn it on so that
builders can finish their builds in less than 8 hours.
BUG=chromium-os:6706
TEST=Run ./enter_chroot.sh --official_build and verify that --fast is on by
default now
Change-Id: I6ad126b9b6ce16ffc9887a7af22c2e3f85afbf42
Review URL: http://codereview.chromium.org/3418001
Change-Id: Ifc47ed28dc3efc0e7ebd018f6703b36913ffd39c
BUG=8716
TEST=Ran the script inside the chroot to makesure it is generating the package.
Review URL: http://codereview.chromium.org/4425004
BUG=8544
TEST=build image with --factory_install, ensure that image does not have data in /usr/local/autotest-pkgs
Change-Id: I5ed17d794392ec654bc122fa15f7ab8a7abbba2d
Review URL: http://codereview.chromium.org/4756001
2. Source shflags, if they are in the current directory
Change-Id: Id22e597a6fb71905f2d0ca5c4a1d94ce5a8a931f
Review URL: http://codereview.chromium.org/4177005
echo -ne isn't portable but printf with octals is so this
makes the switch.
In addition, the current offset is off by 3 and is updating the UUID space.
This was from style cleanup and all functional test passed - of course.
This change fixes that too.
TEST=built and installed and checked ro enforcement works
BUG=chromium-os:7972
Change-Id: Ifb3cb2c6f3219af819baff5750453439e1ba6edf
Review URL: http://codereview.chromium.org/3997001
This change makes more of the root filesyste metadata static across builds, but
more can be done there. It also changes the root filesystem to use ext2 as
we don't need journaling in normal mode. Optionally we could use ext3 for
non-verified if desired (it's an easy change).
In particular, this change cleans up the following:
- clears the rootfs uuid
- labels it C-ROOT (instead of C-KEYFOB)
- removes reserved inodes and blocks
The major feature of this change, however, is that it adds two simple
helpers to common.sh: disable_rw_mount and enable_rw_mount. They will
set high order byte (le) in the ro compat field to be 0xff. This will
tell the kernel that the filesystem uses features R24-R31 which are safe
for use on the running kernel iff the filesystem is mounted read-only.
These functions are called in cros_make_image_bootable and
mount_gpt_image, respectively. mount_gpt_image will always
enable_rw_mount and cros_make_image_bootable will disable_rw_mount if
--enable_rootfs_verification is true.
The approach is ugly but reasonably well contained. If ext2 ever gets a
new revision and new features in the same range are introduced, then we
would be getting inconsistent behavior. That said, it is unlikely that
that churmn will happen and if the impact is negative, it will ideally
show up during testing.
N.B., this will likely result in changes needing to be made to the
signing scripts in vboot_reference to ensure that rw mounting is
enabled/disabled in the same way (E.g., during stamping).
BUG=chromium-os:7468
TEST=- built x86-generic, imaged to usb stick, attempted to mount rw /dev/sdc3 on the host and was properly bounced.
- booted to the image just fine on a dogfood device.
- mod'd for recovery, then installed and booted.
- mod_image_for_test runs with no errors; booted the resulting image as well
- booted a factory_install with the pending dm changes
- BVT passed with build_image x86-generic (vboot enabled)
- [in progress] autotest that checks if the rootdev = /dev/dm-0 and
then does a dumpe2fs | grep -q FEATURE_R31
Review URL: http://codereview.chromium.org/3916002
Change-Id: If4dcba7568a110f4e32627c916d9e5741e5e5414
This function is now called by enter_chroot.sh. A separate change
will call the function from make_chroot
Change-Id: I4fc07c413e56db3e5e7617428f20f2ffc02790d7
BUG=chromium-os:7072
TEST=Took root out of sudoers and ran enter_chroot.sh script; saw that it was re-added.
Review URL: http://codereview.chromium.org/3909001
It's sometimes useful to run a command in chroot and redirect/pipe stdout. Any objections to having enter_chroot send its chatter to stderr?
Review URL: http://codereview.chromium.org/3160024
Autotest installs stuff to the image via ssh, so there's no need to include
all of this stuff to the image. This saves disk space. Note that
/usr/local/autotest-chrome is currently already excluded from the image
because of a hack in the chrome ebuild, and that /usr/local/autotest is
usually also excluded from the image because autotest is not included
as a dependency of any packages that are installed.
In the factory setting, it looks like the factory has its own code for
installing autotests via rsync. Presumbably, if the factory tests are
managing the autotest directly, they won't want emerge mucking with the
directory as well on the image. Currently, emerge doesn't muck with the same
directory, but we probably want to ensure that it doesn't start doing that in
future to ensure sanity.
TEST=Built images. Modded for factory test. Verified that (a) build directory contains autotests, (b) regular images do not contain autotests; and (c) images modified for factory test do contain autotests
Review URL: http://codereview.chromium.org/3124010
BUG=none
TEST=run build_image, happen to finish on a time with less than 10 seconds in it, watch the missing 0-padding
Review URL: http://codereview.chromium.org/3050047
This change checks for and sources shflags from /usr/lib before
depending on the gclient-style path.
TEST=None
BUG=4230
Review URL: http://codereview.chromium.org/3010045
Change-Id: I3085c2356c40c691a2a72bf79c1d9167162d6dca
The fast build is stable enough now that we can enable it for everybody.
I ran the build constantly all weekend on two machines and I only ran into
one (rare) error, which I've fixed in a separate patch.
See <http://codereview.chromium.org/2856048/show>
TEST=Ran lots and lots of fast builds
BUG=none
Review URL: http://codereview.chromium.org/3059001
parallel_emerge prints a message every 30 seconds now so that buildbot doesn't
kill us early. Per discussion with cmasone, reverting the revert.
This reverts commit debaa3d8cf.
This reverts commit f0fb864e8d.
This change needs to be reverted because it suppresses logging output
from the build process. Thus, when something goes wrong...the
sherriff is left completely helpless in terms of debugging the issue.
TBR: davidjames@chromium.org
Also update eretry to not pass in $EMERGE_JOBS anymore, in coordination with my previous change, which passed it in directly instead of depending on eretry to do so: http://codereview.chromium.org/2944004/show
TEST=Ran build_image --jobs 2 and build_packages --jobs 2
BUG=none
Review URL: http://codereview.chromium.org/2914004
TEST=Ran unittests with 2 test failures (powerd and update engine).
Will check with authors to fix these before moving the default over
to this.
Review URL: http://codereview.chromium.org/2837004
TEST=Tested with building a new image, looking in the output directory, running
the image and running vi.
Review URL: http://codereview.chromium.org/2075019
I verified that if I whack /usr/lib/debug folder and run build_image, it works fine. This is important since I plan to check in this issue first and then the fix to add debug flags.
Review URL: http://codereview.chromium.org/2113007
Temporary approach to using INSTALL_MASK to suppress some non-relevant paths.
It reduces the size from ~426MB to ~163MB. Eventually, will remove it and
create a clean and separated ebuild for chromeos-factoryinstall.
Review URL: http://codereview.chromium.org/2084001
In addition unifies changes in mount_gpt_image and one-offs in mod_image_for_test
to consolidate gpt / var mounting.
Review URL: http://codereview.chromium.org/2064001
Since we're using upstart, don't let packages drop startup scripts into
/etc/init.d.
BUG=none
TEST=Build image, inspect contents of /etc/init.d (or lack thereof, after this patch)
Review URL: http://codereview.chromium.org/2023006
Add restart_in_chroot_if_needed to common.sh, and modify the build scripts which referred to assert_inside_chroot to use it instead. The effect is that you don't ever have to explicitly enter_chroot.sh to build (still can, it work's fine).
Update mod_image_for_test.sh to use restart_in_chroot_if_needed
Review URL: http://codereview.chromium.org/1736025
The safe_unmount function first tries a regular unmount, If that
fails it warns and tries a lazy unmount. If the lazy unmount fails
it dies.
Both unmounts take the -d option in case the mount is a loop device.
BUG=None
TEST=Entered and exited multiple chroots
Review URL: http://codereview.chromium.org/1593021
and "cat" stuff and pipe it to something that can write to a
file as root. For example:
echo "foo" | sudo_append /tmp/bar
will append "foo" to the file /tmp/bar as root. While
echo "foo" | sudo_clobber /tmp/bar
will truncate /tmp/bar and then write "foo" to the file.
Review URL: http://codereview.chromium.org/1610021
This is necessary for two reasons:
1) It's nice to get an error message when mount/unmount fails
2) set -e mode doesn't have any effect when you're in a subshell
Note that these mount/unmount failures do happen regularly in development,
so folks who depended on mount/umount failing silently will no longer be
able to rely on this and will have to kill the mounts manually.
Also fix subtle bugs in regexes for matching mount paths. (E.g. where the
regex for $HOME/chroot also matches $HOME/chroot2).
TEST=Tested mount/unmount with concurrently open shells.
Tested mount/unmount when mount is being used by a process but the
lock file does not reflect this.
BUG=none
Review URL: http://codereview.chromium.org/1211001
Use our karmic mirror for creating the build chroot, removing the need for
make_local_repo.sh and the repo_list_*.txt files. If a local repo exists
when build_chroot.sh is run it will be used, but it's no longer necessary.
This change does not remove make_local_repo.sh or the repo_list_*.txt files.
Review URL: http://codereview.chromium.org/548083
This change switches to mastering an image without using debootstrap.
We turn on the previously experimental bits that install a small
set of packages manually before handing things over to apt. In both
cases we use apt to download the packages so that it can populate
it's local package cache.
With this change we will no longer depend on the local_repo when
mastering an image. Developers will not have to rebuild their
local repo when repo_list_image.txt changes. Instead we will use
and lazy-fill the local apt-cache. The first time you build_image.sh
it will be slow since it needs to download the packages. Subsequent
runs should be as before since it will use the local cache. If
packages are added they should be lazily fetched in the next image
build.
Until we have a fully populated external mirror, developers will
still have to add packages to repo_list_image.txt. Also, until
make_chroot is switched over to use the external repo then
developers will have to redo their local repo when the
repo_list_dev.txt changes.
Review URL: http://codereview.chromium.org/521073
