Using parallel_emerge has been disabled by default for all commands
except build_image for quite a while now, build_image kept it just
because it was still a bit faster than normal emerge. Keeping
parallel_emerge complicates future changes to build_image so it needs to
drop it entirely. Since that means nothing uses it by default we might
as well just rip out support for it entirely.
To behave more like setup_board/build_packages update_chroot should
fully configure portage to make sure everything is accurate.
Now binhosts are defined in make.conf.host_setup so the static config in
coreos-overlays doesn't need to refer to version.txt. setup_board
already made this change in 7a43a07f.
Define path locations to reduce dependency between static configs in
coreos-overlays and the behavior of the scripts repo. Spreading
configuration across two repos makes everything harder to understand.
Eventually everything should either be defined in profiles in
coreos-overlays or minimal auto-generated config files here in scripts.
The latest release of portage uses repos.conf, so generate that
file when making the chroot. The crossdev overlay also needed
repo_name and layout.conf files.
- Remove custom COREOS_* attributes from /etc/lsb-release
- Move dev image logic to dev_image_util
For extra fun fix detection of local host URL for devserver.
- Remove weirdly verbose "DESCRIPTION" format.
- Add COREOS_RELEASE_BOARD back to /usr/share/coreos/release
This is mostly just so update_engine and gmerge report the correct
board name to devserver, informative-only on prod images.
- Remove version info from /etc/gentoo-release
- Switch from 'track' to 'group' terminology.
I would like to phase out parallel_emerge so disable it for all commands
other than build_image which is the only one that shows a noticeable
benefit from it (~2 min with --fast, ~3 min with --nofast).
Soon nss-usrfiles will be required to resolve users and groups properly.
To avoid potentially breaking during the transition we need to make sure
the package is installed early during the chroot upgrade process.
This makes it possible to toggle parallel_emerge just as other scripts
do. In other scripts update the help string to be more specific, the
--jobs option can be used to control parallelism.
This doesn't currently have any real impact but seems like a better
ordering since update scripts may need to emerge things.
Add some friendly version info logging.
The old chroot version system we inherited from Chromium OS always
assumes that a newly unpacked tarball is the latest and greatest but
since we version the SDK in the same way as target builds we can use
that version for these sorts of upgrade scripts and not make assumptions
about how late and great the starting tarball was.
The first upgrade script simply aborts to force the user to recreate
their chroot when moving from python 2.6 to 2.7.
As-is all of the various emerge wrapping scripts default to using
--getbinpkg whenever --usepkg is enabled. This means every single emerge
command made makes multiple synchronous HTTP requests to the upstream
binary package repository to get the latest package list. This gets
really frustrating when working remotely with limited network
connectivity. Using --usepkg with --nogetbinpkg will use locally cached
packages without making remote requests.
The current logic accidentally sets TOOLCHAIN_FLAGS when the --nousepkg
flag is used rather than appending it leading to possible flag loss.
While we're at it, use arrays.
BUG=None
TEST=`./setup_board --board=x32-generic --nousepkg` works now
Change-Id: I33627ec088afb649b4fdde42b933a4af1521675b
Reviewed-on: https://gerrit.chromium.org/gerrit/31150
Reviewed-by: Zdenek Behan <zbehan@chromium.org>
Commit-Ready: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
Make the following commands send stats to chromiumos-build-stats.appspot.com:
build_image
run_chroot_version_hooks
make_chroot
setup_board
update_chroot
BUG=chromium-os:33088
TEST=`cbuildbot --remote -p chromiumos/platform/crosutils alex-paladin`
confirm in the log that uploads succeded, and see them show up in queries
at chromiumos-build-stats.appspot.com.
Change-Id: I0280f91a3e7e0a0483c01c87072bc589003dbe95
Reviewed-on: https://gerrit.chromium.org/gerrit/28969
Tested-by: Matt Tennant <mtennant@chromium.org>
Reviewed-by: David James <davidjames@chromium.org>
Commit-Ready: Matt Tennant <mtennant@chromium.org>
If the board we're building includes extra toolchains that aren't part
of our official set, we need to make sure cros_setup_toolchains still
tries to set them up for us.
BUG=None
TEST=`cbuildbot chromiumos-sdk` works
TEST=`./setup_board --board=x32-generic` installs an extra toolchain
Change-Id: I85f70177f82bdf295eb2cd8668a031317a3b6e61
Reviewed-on: https://gerrit.chromium.org/gerrit/24336
Tested-by: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Zdenek Behan <zbehan@chromium.org>
Commit-Ready: Mike Frysinger <vapier@chromium.org>
Previously, if you were working on a cros_workon package on the host, it
would only get updated once -- after that, the package would never build
again.
With this change, we now rebuild host workon packages any time their
timestamp is modified. This replicates the same feature that we already
have for board packages.
BUG=chromium-os:12771
TEST=Verify that host workon packages are rebuilt when the timestamps change,
and only when the timestamps change.
Change-Id: I31ef1d83dc591161a7cb55c4af806ee4a4212cdd
Reviewed-on: https://gerrit.chromium.org/gerrit/24782
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Ready: David James <davidjames@chromium.org>
Tested-by: David James <davidjames@chromium.org>
Resubmit of If62b4f3973f02fd8e1deed35864c824a02ab0c22
This will be safe to land after
I2c4e21ec7e8c0c0cf58947e2b0a3a9edf7617a09
The breakage was a timing issue paired with people not always syncing
the complete tree. No changes to the CL are needed.
It is now used for:
- make_chroot (cros_sdk --bootstrap)
- update_chroot
setup_board is stripped of redundant code which was deprecated by this.
Also stripped is some usepkg logic in make_chroot, as that is now
exclusively source-only.
BUG=chromium-os:23032
TEST=trybot chromiumos-sdk
Change-Id: Ib888cf2886218622d9cfeebb17b9cd4462d06c89
Reviewed-on: https://gerrit.chromium.org/gerrit/22578
Tested-by: Zdenek Behan <zbehan@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Reviewed-by: asharif <asharif@chromium.org>
Commit-Ready: Zdenek Behan <zbehan@chromium.org>
Currently, if set -e spots a nonzero exit we basically have
no real debug information- it just stops immediately without stating
where or why. This forces our scripts to be stupidly verbose so
we can track roughly where they were, thus when they fail we can
use that information to localize the rough exit point.
Instead we should be traping that set -e induced exit and
outputing necessary debug information to run it down. This includes
outputing the relevant stack trace, or at least what we can get of
it.
The 'die' function is now enhanced to automatically dump the trace
that lead to it. For most consumers this is desired- however for
commandline parsing induced dies ("--board is missing" for example),
the trace is noise. For those cases, a 'die_notrace' function was
added that retains the original non-backtrace behaviour.
Example output via instrumenting cros_generate_breakpad_symbols
w/ the failing command '/bin/false' (nonzero exit code).
Before:
./cros_generate_breakpad_symbols monkeys --board=x86-alex
<no output at all, just exit code 1>
With this CL:
./cros_generate_breakpad_symbols monkeys --board=x86-alex
ERROR : script called: ./cros_generate_breakpad_symbols 'monkeys' '--board=x86-alex'
ERROR : Backtrace: (most recent call is last)
ERROR : file cros_generate_breakpad_symbols, line 207, called: main 'monkeys' '--board=x86-alex'
ERROR : file cros_generate_breakpad_symbols, line 163, called: die_err_trap '/bin/false' '1'
ERROR :
ERROR : Command failed:
ERROR : Command '/bin/false' exited with nonzero code: 1
BUG=chromium-os:30598
TEST=inject a failing command into a script, verify the output.
TEST=inject a 'command not found', verify the output
TEST=cbuildbot x86-generic-full --remote
TEST=cbuildbot arm-tegra2-full --remote
TEST=cbuildbot chromiumos-sdk --remote
Change-Id: I517ffde4d1bb7e2310a74f5a6455b53ba2dea86c
Reviewed-on: https://gerrit.chromium.org/gerrit/17225
Reviewed-by: Brian Harring <ferringb@chromium.org>
Tested-by: Brian Harring <ferringb@chromium.org>
Commit-Ready: Brian Harring <ferringb@chromium.org>
Resubmit of If62b4f3973f02fd8e1deed35864c824a02ab0c22
This will be safe to land after I2c4e21ec7e8c0c0cf58947e2b0a3a9edf7617a09
It is now used for:
- make_chroot (cros_sdk --bootstrap)
- update_chroot
setup_board is stripped of redundant code which was deprecated by this.
Also stripped is some usepkg logic in make_chroot, as that is now
exclusively source-only.
BUG=chromium-os:23032
TEST=trybot chromiumos-sdk
Change-Id: Ic908eac712ac097e5c2062d3be70177e172aa924
Reviewed-on: https://gerrit.chromium.org/gerrit/20191
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Ready: Zdenek Behan <zbehan@chromium.org>
Tested-by: Zdenek Behan <zbehan@chromium.org>
It is now used for:
- make_chroot (cros_sdk --bootstrap)
- update_chroot
setup_board is stripped of redundant code which was deprecated by this.
Also stripped is some usepkg logic in make_chroot, as that is now
exclusively source-only.
BUG=chromium-os:23032
TEST=trybot chromiumos-sdk
Change-Id: If62b4f3973f02fd8e1deed35864c824a02ab0c22
Reviewed-on: https://gerrit.chromium.org/gerrit/17910
Reviewed-by: Zdenek Behan <zbehan@chromium.org>
Tested-by: Zdenek Behan <zbehan@chromium.org>
Commit-Ready: Zdenek Behan <zbehan@chromium.org>
The latest gcc-config will accept the CHOST setting from the host env,
so export that variable since we already spent the time running portageq
to get it. This allows gcc-config to never execute portageq, and since
we run it multiple times, it helps in every invocation.
Along those lines, make sure we run gcc-config with `sudo -E` to pass
through the setting.
Since awk can handle regexps, merge the grep|awk into a single awk call.
Finally, there was a slight logic typo where we would always re-run
gcc-config even when the profiles were the same. This does not match
the comments, nor the spirit of this code, so fix the operator typo.
BUG=None
TEST=`./update_chroot` still works and doesn't re-select the toolchain
Change-Id: If73df81c014219f8f9ab5895e59d055696add777
Reviewed-on: https://gerrit.chromium.org/gerrit/19164
Reviewed-by: Zdenek Behan <zbehan@chromium.org>
Commit-Ready: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
1. Remove options that are already deprecated:
--chromefromsource --chromebuild --chromebase
--crosbug12352_arm_kernel_signing
2. Remove the --retries option and associated function eretry.
3. Move seldomly used options out of --help to avoid confusing
developers. Developers who need these options can read the
source.
4. Alphabetize all options.
5. Add description of each script to --help.
BUG=chromium-os:27364
TEST=Run --help with each script. Full canary trybot run.
Change-Id: I95675b069781f7e950d75d32dbad744adce6b830
Reviewed-on: https://gerrit.chromium.org/gerrit/18194
Reviewed-by: David James <davidjames@chromium.org>
Tested-by: David James <davidjames@chromium.org>
Commit-Ready: David James <davidjames@chromium.org>
Update setup_board and build_packages to use common.sh from source
directory, so that changes in common.sh show up immediately.
BUG=chromium-os:27364
TEST=Run each script, testing that changes to common.sh are picked up.
Change-Id: I6847bbf8c486d14c58dfcbb9361cd606c11c817b
Reviewed-on: https://gerrit.chromium.org/gerrit/18245
Reviewed-by: David James <davidjames@chromium.org>
Tested-by: David James <davidjames@chromium.org>
Commit-Ready: David James <davidjames@chromium.org>
To build recovery images, the initramfs package needs fonts for
various languages. The Japanese fonts to use depend on whether
it's an internal (Chrome OS) or external (Chromium OS) build.
Including the internal Japanese fonts requires additional settings
in /etc/make.conf in the chroot. This change updates make_chroot
and update_chroot to ensure that the necessary settings are present
when appropriate.
BUG=chromium-os:26757
TEST=confirm proper packages after chroot creation with private manifest
TEST=confirm proper packages after chroot creation with public manifest
TEST=confirm proper packages after chroot upgrade with private manifest
TEST=confirm proper packages after chroot upgrade with public manifest
CQ-DEPEND=I32c4561c47752d96d2769c5429c5e44fe630f8f9
Change-Id: I2b29b32bf4e6b891a2228dc7f93ae0bb3f85e37f
Reviewed-on: https://gerrit.chromium.org/gerrit/17372
Tested-by: Richard Barnette <jrbarnette@chromium.org>
Reviewed-by: David James <davidjames@chromium.org>
Commit-Ready: Richard Barnette <jrbarnette@chromium.org>
There's no reason to always show the full build output when updating the
perl modules, so use the --quiet flag. If there is a failure, emerge will
dump the full log like normal.
BUG=None
TEST=`./update_chroot` showed much less output when updating perl modules
Change-Id: I0265443bca0f42a79cc3e44a7f8e9bfeab69f568
Reviewed-on: https://gerrit.chromium.org/gerrit/17080
Reviewed-by: David James <davidjames@chromium.org>
Commit-Ready: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
We are changing the default emerge options to use --oneshot, so make sure
our scripts select the packages we care about for @world.
BUG=None
TEST=setup_board+build_packages for tegra2 adds pkgs to world
Change-Id: I5fc68c538a1a1c846bd9724f27ef717029d11d42
Reviewed-on: https://gerrit.chromium.org/gerrit/16838
Reviewed-by: David James <davidjames@chromium.org>
Commit-Ready: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
The logic of IGNORE_PREFLIGHT_PREBUILTS is currently inverted in make_chroot,
causing preflight prebuilts to be ignored by default, but then used in the one
situation where they should be ignored. This CL fixes that.
BUG=chromium-os:26834
TEST=Verify preflight prebuilts are now used by default in make_chroot.
Change-Id: I626319d6de79b6f201f993a05e34610e944b161e
Reviewed-on: https://gerrit.chromium.org/gerrit/16614
Reviewed-by: Richard Barnette <jrbarnette@chromium.org>
Commit-Ready: David James <davidjames@chromium.org>
Tested-by: David James <davidjames@chromium.org>
The --jobs=<n> option wasn't actually being passed to parallel_emerge --
which meant it didn't do anything.
BUG=chromium-os:26827
TEST=Ran with --jobs=N and made sure only N parallel_emerge processes were
started.
Change-Id: I581fc5588b54e246acaefd0c7e528e55adf9ba8a
Reviewed-on: https://gerrit.chromium.org/gerrit/16570
Reviewed-by: Michael Krebs <mkrebs@chromium.org>
Commit-Ready: David James <davidjames@chromium.org>
Tested-by: David James <davidjames@chromium.org>
* This is now safe to do, since the only protected files are exactly
the ones that have been autogenerated during chroot creation and may
not be overwritten.
BUG=chromium-os:13987
TEST=update the chroot
Change-Id: Ica4d8328b21089b23e2b0e8a29530cedddabc299
Reviewed-on: https://gerrit.chromium.org/gerrit/10166
Commit-Ready: Zdenek Behan <zbehan@chromium.org>
Reviewed-by: Zdenek Behan <zbehan@chromium.org>
Tested-by: Zdenek Behan <zbehan@chromium.org>
This variable is intended to be used by cbuildbot to allow it to ignore
the preflight binhost when clobbering the preflight builder itself, to
ensure it doesn't pick up the old preflight prebuilts again.
BUG=chromium-os:21788
TEST=Set IGNORE_PREFLIGHT_BINHOST and see that preflight binhosts are ignored.
Change-Id: I2a0ca38ad8d35b34d2f14a980d5cd67ab66c0f07
Reviewed-on: https://gerrit.chromium.org/gerrit/11088
Reviewed-by: Ryan Cui <rcui@chromium.org>
Tested-by: Ryan Cui <rcui@chromium.org>
Commit-Ready: David James <davidjames@chromium.org>
BUG=chromium-os:21474
TEST=run the script, see my chroot being updated, cases below:
1) run normally, see the latest gcc being selected
2) run while having an older gcc manually selected, see no updates
3) set an invalid native profile, see it being fixed
Change-Id: Ic84187b8acf39fba11f2e39f36457e6f696ad7e4
Reviewed-on: http://gerrit.chromium.org/gerrit/9832
Tested-by: Zdenek Behan <zbehan@chromium.org>
Reviewed-by: David James <davidjames@chromium.org>
Commit-Ready: Zdenek Behan <zbehan@chromium.org>
Whenever perl is upgraded, it is a good idea to clean out any old
modules and recompile them. This check ensures that perl-cleaner is run
once and only once after each major perl upgrade.
BUG=chromium-os:19244
TEST=Run preflight queue with perl upgrade and old sdk. Verify that old
perl modules are removed in the first run. Verify perl-cleaner is
not run again in subsequent runs. Run sdk builder and confirm sdk
builder runs perl-cleaner every time since it starts with an old
version of perl.
Change-Id: Ib14f9d73122d5ff2c7a23afc3f56905e30ff2cbc
Reviewed-on: http://gerrit.chromium.org/gerrit/6149
Reviewed-by: Anush Elangovan <anush@chromium.org>
Tested-by: David James <davidjames@chromium.org>
The problem here is that most were doing their exiting w/in a subshell;
exit within a subshell kills the subshell, not the parent. Not all scripts
were using set -e (which would pick up the failing subshell); as such
just rewriting them to remove the potential via eliminateing the subshelling.
Beyond that, removed a couple of custom (working, although non-standard)
approaches, and removed a duplicate common.sh sourc'ing w/in mk_memento_images.sh
TEST=force 'find_common_sh' to fail, note the scripts fails to exit
BUG=none
Change-Id: Ia1108a091a6399ad6aedd3cade4a107f4411686c
Reviewed-on: http://gerrit.chromium.org/gerrit/3905
Reviewed-by: Brian Harring <ferringb@chromium.org>
Tested-by: Brian Harring <ferringb@chromium.org>
New toolchains should only be rolled out to developers running with
binary packages when the sdk-builder has completed and uploaded the
prebuilts. This ensures that we've vetted the toolchain properly
before it rolls out and also speeds up binary builds for developers.
This also provides consistency with the behavior of setup_board.
BUG=chromium-os:16363
TEST=Verify these 3 packages are only updated in --usepkg builds
when the prebuilts are also updated.
Change-Id: I6be7c6c98c12130751d3d4440bec7c71afb1d4c2
Reviewed-on: http://gerrit.chromium.org/gerrit/2391
Reviewed-by: Raymes Khoury <raymes@chromium.org>
Tested-by: David James <davidjames@chromium.org>
BUG=quickfix for bot break
TEST=run ./update_chroot
Change-Id: I459d50f3e3374c1ff8ff8fc692c8ba7f3582bbd5
Reviewed-on: http://gerrit.chromium.org/gerrit/743
Tested-by: David James <davidjames@chromium.org>
Reviewed-by: David James <davidjames@chromium.org>
script
BUG=none
TEST=run build_packages, setup_board, make_chroot, with the paths fixed
to this script (http://gerrit.chromium.org/gerrit/700), see things work
Change-Id: I5a3950ddd5949fcf900a609f86092fce33db8cbd
Reviewed-on: http://gerrit.chromium.org/gerrit/720
Reviewed-by: David James <davidjames@chromium.org>
Tested-by: Zdenek Behan <zbehan@chromium.org>
