This file used to be imported by scripts coreos-base/cros-devutils,
which we have dropped. Now it is imported only from some other script
in build library so move it there. This leaves lib as a directory
where we keep shflags library.
The scripts were imported by some scripts from
coreos-base/cros-devutils, which we have dropped already. So the
scripts in the lib directory are currently an unused baggage.
Remove the catalystrc code that updated /etc/locale.gen, it wasn't
generally effective since catalyst installs things without
CONFIG_PROTECT and therefore locale.gen always got overwritten just
before it was going to be used. Add --load-average to MAKEOPTS so
builds better manage their parallelism when multiple independent jobs
are compiling code. Disable some locking to speed up builds that use
lots of binary packages.
The previous logic here only skipped the bootstrap if the toolchain
packages were already installed (which won't happen on the build host)
but we can also skip the bootstrap if binary packages are available
(which will happen on the build host). This will help avoid needless gcc
rebuilds :)
We aren't taking full advantage of seedcache and untarring things isn't
a big burden on us anyway so don't bother and hopefully avoid leaving
oodles of files behind.
A few things here:
- Source manifests/version.txt directly instead of coreos-version.sh
- Remove Chrome branch from target image directory names.
- Use proper version instead of timestap for catalyst builds.
- Move lsb_release script from coreos-overlay to build_library.
When running bootstrap_sdk for the first time on a host the default of
using a previous catalyst run as a seed won't work but this may be a
fresh SDK install so that tarball is probably around somewhere and will
work as a default seed for most things.
By default emerge will not install build dependencies if it doesn't need
them (i.e. installing a binary package) but we want to make sure
everything gets included in stage4 no matter how it was installed.
This uses Gentoo's catalyst for very thoroughly building images from
scratch. Using images based on this will eliminate some of the hackery
in make_chroot.sh for building up the sdk from a stock stage3 tarball.
For reference the procedure it performs is this:
1. snapshot: Grab a snapshot of portage-stable. Note that overalys are
not snapshotted.
2. stage1: Using a "seed" tarball as a build environment, build a
minimal root file system into a clean directory using ROOT=...
and USE=-* The restricted USE flags are key be small and avoid
circular dependencies.
3. stage2: Run portage-stable/scripts/bootstrap.sh
This rebuilds the toolchain. Probably not strictly necessary most of
the time but does super-duper-promise that the toolchain isn't linked
to or otherwise influenced by whatever was in the "seed" tarball.
4. stage3: Run emerge -e system to rebuild everything using the fresh
toolchain using the normal USE flags provided by the profile. This
will also pull in assorted base system packages that weren't included
in the minimal environment stage1 created.
5. stage4: Install any extra packages or other desired tweaks. For the
sdk we just install all the packages normally make_chroot.sh does.
crosutils cros_build_lib is deprecated in favor of
chromite.lib.cros_build_lib. Remove unused functions to encourage folks
to migrate over to chromite.lib.cros_build_lib instead.
Info and GetCallerName are only used internally so we add an underscore
prefix to discourage others from using it.
BUG=chromium-os:24660, chromium-os:21287
TEST=pylint. Unit tests. Trybot runs.
CQ-DEPEND=CL:40300, CL:40311
Change-Id: Ia34d2770db2c2678b8966f6dceb2164b0a3333a2
Reviewed-on: https://gerrit.chromium.org/gerrit/40301
Tested-by: David James <davidjames@chromium.org>
Reviewed-by: Brian Harring <ferringb@chromium.org>
Commit-Queue: David James <davidjames@chromium.org>
Moving to the cirrus driver (instead of std) so we can support
cirrus-fb, in preparation for nonroot-X support.
BUG=chromium-os:388
TEST=local build, run with x86-generic
Change-Id: Ic7d5776f673ac3c15680d62331345d55985f1d73
Reviewed-on: https://gerrit.chromium.org/gerrit/39900
Reviewed-by: Stéphane Marchesin <marcheu@chromium.org>
Reviewed-by: Chris Sosa <sosa@chromium.org>
Tested-by: John Sheu <sheu@chromium.org>
Commit-Queue: John Sheu <sheu@chromium.org>
BUG=chromium-os:34126
TEST=Ran image_to_vm with full line from buildbot.
Change-Id: I7b579500e80f3b3882f7e9aa62f73e89bf11ceba
Reviewed-on: https://gerrit.chromium.org/gerrit/32429
Reviewed-by: Chris Sosa <sosa@chromium.org>
Tested-by: Chris Sosa <sosa@chromium.org>
BUG=chromium-os:34126
TEST=image_to_vm with new size and ran desktopui_NaClSanity
Change-Id: I5553de776cb5dec3d2b0dde362ff3560926a6472
Reviewed-on: https://gerrit.chromium.org/gerrit/32424
Reviewed-by: Rahul Chaturvedi <rkc@chromium.org>
Tested-by: Chris Sosa <sosa@chromium.org>
BUG=None
TEST=manual, rename kvm to qemu-kvm, else run it on a gentoo host.
Change-Id: I4971336a1be4c07fe2c8799643f52cd094d5caf1
Reviewed-on: https://gerrit.chromium.org/gerrit/32093
Tested-by: Brian Harring <ferringb@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Ready: Brian Harring <ferringb@chromium.org>
The "function" keyword is superfluous, not in POSIX, is inconsistent
between bash files, and generally makes me angry. So convert every
instance to the form:
foo() {
BUG=None
TEST=`cbuildbot x86-generic-paladin` works
Change-Id: I97f5ca30a3edfef7222b1e08ac23917dc613b556
Reviewed-on: https://gerrit.chromium.org/gerrit/22467
Reviewed-by: David James <davidjames@chromium.org>
Commit-Ready: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
This allows KVM stage to run significantly faster. On
my machine, with this change I was able to run the pre-flight
queue VM update tests in ~369 seconds, compared to the normal
time of >1000 seconds.
BUG=chromium-os:23112, chromium-os:22784
TEST=Verify speed increase, trybot runs.
Change-Id: Ic27dd1719573fb50a412a0cfe39adf5408493ea8
Reviewed-on: https://gerrit.chromium.org/gerrit/11835
Commit-Ready: David James <davidjames@chromium.org>
Reviewed-by: David James <davidjames@chromium.org>
Tested-by: David James <davidjames@chromium.org>
This CL is the counter part of https://gerrit.chromium.org/gerrit/11190
For creating portable factory archives and simplify scripts folder, we should
move all factory scripts to a tandalone folder. Symlink is set to ease
transition until we've updated all factory documents.
archive_factory was designed for creating bundle so not required anymore.
make_universal_factory_shim and mk_memento_images are only invoked by
make_factory_packages, so they're also simply removed without symlinks.
BUG=chrome-os-partner:5979
TEST=./make_factory_package.sh --config PATH_TO_CONFIG # pass
Change-Id: I1489b012aaf1ade4a9ead91abbb7b625b967dc0f
Reviewed-on: https://gerrit.chromium.org/gerrit/10630
Reviewed-by: Nick Sanders <nsanders@chromium.org>
Reviewed-by: Zdenek Behan <zbehan@chromium.org>
Reviewed-by: David James <davidjames@chromium.org>
Commit-Ready: Hung-Te Lin <hungte@chromium.org>
Tested-by: Hung-Te Lin <hungte@chromium.org>
BUG=chromium-os:21610
TEST=Ran cros_start_vm/cros_stop_vm ... start_vm with same pid after manually
closing it, etc.
Change-Id: If4062c452a135d39a280374cbbebfa852e744022
Reviewed-on: http://gerrit.chromium.org/gerrit/9974
Tested-by: Chris Sosa <sosa@chromium.org>
Reviewed-by: Don Garrett <dgarrett@chromium.org>
Commit-Ready: Chris Sosa <sosa@chromium.org>
We try to set the cache type to unsafe, which is a new kvm feature,
else we set the cache type to writeback. The hope is that these cache
types will perform better on the bots, especially in the case of the
guest trying to flush data to disk.
The CL includes code to determine which version of kvm is running, and
the cache type is chosen based on that.
BUG=chromium-os:20305
TEST=trybot
Change-Id: Icf7dbb8f0056a330282ebeb81d0902b3d54f34c5
Reviewed-on: http://gerrit.chromium.org/gerrit/8360
Reviewed-by: David James <davidjames@chromium.org>
Tested-by: Andrew de los Reyes <adlr@chromium.org>
This CL allows factory related scripts to be executed in a prepared and limited
environment which can be made by extracting a factory bundle:
- always load scripts from $SCRIPT_ROOT
- allow running bundled binary programs from $SCRIPT_ROOT/../bin
- override GCLIENT_ROOT for the layout of a bundle
BUG=chrome-os-partner:5979
TEST=(outside cros environment)
./make_factory_package --config mp_factory.conf # test omaha
./make_factory_package --config rma.conf # test --usbimg
./make_factory_package --config mp_ssd.conf # test --diskimg
Change-Id: Ibf85fa267f6fe53ae88e41fd7f62ef7110b2b0bf
Reviewed-on: http://gerrit.chromium.org/gerrit/7905
Tested-by: Hung-Te Lin <hungte@chromium.org>
Reviewed-by: Nick Sanders <nsanders@chromium.org>
We need more flexible image creation tool for various layout, like "RMA with
multiple release images" which needs 4 images (install, test, release A,
release B).
To make the creation of image more flexible, this CL changes
make_universal_factory_shim to using a "layout file" that can describe any disk
image format we want.
Image creation speed is also improved by aligning partition offsets and
restricting oflag=dsync applied only when being used for block devices so that
make_universal_factory_shim (outputs to a normal temporary file) becomes faster
by utilizing system cache.
For a system with every images already cached:
- Time for building 3 in 1 factory instal shim:
Before = 8 seconds, After = 5 seconds.
- Time for full RMA shim creation:
Before = 1m25s, After = 40-50 seconds.
BUG=chrome-os-partner:4108
TEST=./make_universal_factory_shim.sh image1 image2 image3 image4
# boots correctly after being imaged to SD card.
./make_factory_package.sh --usbimg RMA ....
# RMA image can install images and works correctly
Change-Id: I645196d6d6e0a24d3dfa4c413a338279df4c0d5b
Reviewed-on: http://gerrit.chromium.org/gerrit/6893
Tested-by: Hung-Te Lin <hungte@chromium.org>
Reviewed-by: Nick Sanders <nsanders@chromium.org>
"cros_sdk" is a drop-in replacement of enter_chroot.sh. The only
important difference is in the calling path, specifically that
cros_sdk is in path but has to be called from $(pwd) being inside the
repo checkout, while enter_chroot.sh is called by explicit path.
Invariably, "./enter_chroot.sh" can be replaced, as pwd=src/scripts.
Calling by absolute path can be replaced by first changing directory
anywhere into the repo checkout, and then calling cros_sdk.
BUG=chromium-os:18750
TEST=run them
Change-Id: Ieff91a27bb419e1121361d5b3a11e4c87ff7a087
Reviewed-on: http://gerrit.chromium.org/gerrit/6273
Tested-by: Zdenek Behan <zbehan@chromium.org>
Reviewed-by: Zdenek Behan <zbehan@chromium.org>
In order to support building arbitrary image, the partition copying scripts has
been changed to support "copying partitions in same size" and "overwriting
partitions in different size", and "copying partition from external file".
We need these APIs to create disk/usb image with release images that is using
partition with different size (ex, recovery images).
Image copying buffer selection and disk image creation time are also improved.
BUG=chromium-os:15050
TEST=./make_factory_package.sh ... --diskimg preimage.bin
./make_factory_package.sh ... --usbimg rma.bin
./make_factory_package.sh ... # omaha mode
Change-Id: I6a4c820abf59e780985c95dc35f9340b347bd952
Reviewed-on: http://gerrit.chromium.org/gerrit/5981
Reviewed-by: Nick Sanders <nsanders@chromium.org>
Tested-by: Hung-Te Lin <hungte@chromium.org>
The release image parameter must be an image signed for SSD booting. This CL
adds detection code and allows on-the-fly conversion from recovery to SSD image.
BUG=chromium-os:15050
TEST=./make_factory_package --release RECOVERY --factory FACTORY # success
# Seeing: INFO : Image type is [recovery]:...
./make_factory_package --release USB --factory FACTORY # success
# Seeing: Image type is [usb]:...
./make_factory_package --release SSD --factory FACTORY # success
# Seeing: Image type is [ssd]:...
./make_factory_package --release GARBAGE --factory FACTORY # failure
# Seeing: Image type is [invalid]:...
./make_factory_package --release GARBAGE --factory FACTORY --nodetect_release_image # success
# No image type messages
Change-Id: I8530b3f58574a4298b4d6d904a12bb92636c7365
Reviewed-on: http://gerrit.chromium.org/gerrit/5965
Tested-by: Hung-Te Lin <hungte@chromium.org>
Reviewed-by: Nick Sanders <nsanders@chromium.org>
When we want to separate kernel and rootfs, there must be a more flexible syntax
to assign input source.
The partition info parsing code is also improved to detect errors.
BUG=chromium-os:15050
TEST=./make_factory_package --release RECOVERY --factory FACTORY
Change-Id: Ie74b3e23117480a7f503488b39dedceadbfb41e3
Reviewed-on: http://gerrit.chromium.org/gerrit/5962
Reviewed-by: Nick Sanders <nsanders@chromium.org>
Tested-by: Hung-Te Lin <hungte@chromium.org>
RunCommand should only pass in stdin, stdout, and stderr to processes,
not other file handles. Fixing this prevents issues where child
processes can cause the parent to hang by forgetting to close the
file handle when launching daemons.
BUG=chromium-os:18104
TEST=Buildbot run with smoke suite. Unit tests.
Change-Id: Ib85095e1fb592ef05a972a5412348363049e6d86
Reviewed-on: http://gerrit.chromium.org/gerrit/4673
Reviewed-by: Brian Harring <ferringb@chromium.org>
Reviewed-by: Don Garrett <dgarrett@chromium.org>
Tested-by: David James <davidjames@chromium.org>
It seems like we are seeing whacky errors with sys.stderr
dying when trying to print out a process and its exception.
I think controlling how the output is printed in RunCommand
is better than passing it with the exception.
Change-Id: I3711575e5d9e710c3aea6ff19d43d149fd83f44c
BUG=chromium-os:14398
TEST=Ran it with au_test_harness
Review URL: http://codereview.chromium.org/6902129
Change-Id: If46fd1558a00498ce4721dd756532e916f49bc19
BUG=chromium-os:12950
TEST=Ran with autestharness and ran unittests.
Review URL: http://codereview.chromium.org/6877015
Cleaned up _InsertKeyIntoImage to be more general (able to remove key if no key set). Also made it much more intelligent ... only doing things when it needs to.
Moved mount / unmount code to cros_lib.
BUG=chromium-os:12684
TEST=Ran it a lot ... inspected images after and saw tests pass.
Review URL: http://codereview.chromium.org/6698017
Change-Id: Icd96d1178eeabf45a2d5916fbcab8bf7ffba7e21
