This reverts commit b97cfe126f.
The minor device numbers of loop partitions are allocated dynamically
which significantly complicates dunning under Docker which uses a static
/dev. Rolling this back until we can rely on /dev being dynamic.
This avoids the need to dd individual filesystem images into a complete
disk image, just mount the partitions directly from a loop device
covering the whole image. This does add the requirement that mkfs run as
root but that isn't a problem.
These are just cluttering things and adding an element of "how does this
work?" because base_image_util was defaulting to the "usb" layout in
some places and "base" in others.
This change removes /usr/sbin/write_gpt.sh from images which we have no
use for. This allows us to drop the indirection of writing partition
tables by first writing out a script to call. Now cgpt.py can call cgpt
directly to initialize the partition layout. This opens the way for
further improvements to how disk images are created.
As of Linux 3.2 loopback supports discard by punching holes in the
underlying file. This doesn't actually seem to impact things right now
since we are writing to fresh filesystems but might as well do this to
prevent wasted space from sneaking in later on.
Enable sparse files for all dd and cp commands and replace some dd
commands that are really better off being truncate commands.
While in the neighborhood there were a number of useless sudo commands
for things that just happen to be in sbin. Call them directly instead.
/mnt/stateful_partition was already a little unruly with
/mnt/stateful_partition/home and /mnt/stateful_partition/var_overlay
serving similar functional purposes.
Then we needed to also add /opt and /srv overlays.
I also have wanted to get rid of the ugly and weird
/mnt/stateful_partition name so lets just have one big move.
/mnt/stateful_partition -> /media/state
/mnt/stateful_partition/var_overlay -> /media/overlays/var
/mnt/stateful_partition/home -> /media/overlays/home
From there we add /media/overlays/srv and /media/overlays/opt
The basic system directory structure including the lib symlinks were
fixed for sysroot in the following commits but the image build uses an
entirely different bit of code to do the exact same set of hacks. Port
those changes to the image building code to hopefully make all happy.
2ae0c30f4eac931bd088
A few things here:
- Source manifests/version.txt directly instead of coreos-version.sh
- Remove Chrome branch from target image directory names.
- Use proper version instead of timestap for catalyst builds.
- Move lsb_release script from coreos-overlay to build_library.
During builds var_overlay is always mounted over /var. We want to do the
same at run time but we also want to ensure everything expected to be
there always does. After emerge completes gen_tmpfiles.py will scan /var
for any .keep files that were installed and records their parent
directories' permissions and ownership to /usr/lib/tmpfiles.d. On each
boot systemd will automatically recreate anything that goes missing.
This also means that going forward any ebuild that needs a directory in
/var (or anywhere else the stateful partition is bound) can simply rely
on the 'keepdir' ebuild function instead of adding things to
coreos_startup.
As outlined here we need a new partition layout, this patch makes the
necessary changes:
https://groups.google.com/forum/#!topic/coreos-dev/bA7gwGGoTng
The first big change is making all of the scripts obey partition numbers
based on labels in the disk_layout.json. This makes it much easier to
change later on.
The second big change is in the layout itself. The json file was updated
to reflect the document above.
And finally the grub boot configuration needed for pv-grub and pygrub
were added to the create_legacy_bootloader_templates.sh library utlity.
Everything seems to work and boot now.
The latest coreutils introduces a status=none option which suppresses
all information which is more than the current status=noxfer.
BUG=None
TEST=`./build_image` no longer spews any dd info
Change-Id: I3cfefed1d38e7e5ff52342c98e6a306d9a48950a
Reviewed-on: https://gerrit.chromium.org/gerrit/41930
Reviewed-by: David James <davidjames@chromium.org>
Commit-Queue: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
CrOS could be used in devices that dont have custom firmware to
interpret the Kernelblock (Part2,4). In such devices skip trying
to make the kernelblock
BUG=none
TEST=./build_image --board=chronos test and verify we dont build the kernel block
Change-Id: I3e9a8dd765ea00db2ebf112553d96c08960e544f
Reviewed-on: https://gerrit.chromium.org/gerrit/40219
Commit-Queue: Anush Elangovan <anush@chromium.org>
Reviewed-by: Anush Elangovan <anush@chromium.org>
Tested-by: Anush Elangovan <anush@chromium.org>
Now that we have an ebuild for factory install shim, let's move to that
instead of base image.
BUG=chrome-os-partner:16712
TEST=Test network boot, install shim, RMA shim on Link.
Test install shim on Snow.
CQ-DEPEND=CL:39952
Change-Id: Ib6a7a598087647b44a4aeca970e293ea72023890
Signed-off-by: Vic Yang <victoryang@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/39955
Reviewed-by: Richard Barnette <jrbarnette@chromium.org>
Reviewed-by: Hung-Te Lin <hungte@chromium.org>
This removes files only used at link time, or when manually debugging,
or by glibc's charset routines (gconv) which are not used by Chromium.
BUG=chromium-os:22939
BUG=chromium-os:23105
TEST=build_image for daisy works
TEST=`cbuildbot {x86,x86_64,arm}-generic-full` worked
TEST=`cbuildbot chromiumos-sdk` worked
TEST=build_image boots & runs fine on an alex
Original-Change-Id: I68d848c2e307c98d53a8faa73924f57571f7887b
Change-Id: I25ffda36d28fd114715053c3a2590cc2248688bf
Reviewed-on: https://gerrit.chromium.org/gerrit/39214
Reviewed-by: David James <davidjames@chromium.org>
Commit-Ready: Darren Krahn <dkrahn@chromium.org>
Tested-by: Darren Krahn <dkrahn@chromium.org>
This removes files only used at link time, or when manually debugging,
or by glibc's charset routines (gconv) which are not used by Chromium.
BUG=chromium-os:22939
BUG=chromium-os:23105
TEST=build_image for daisy works
TEST=`cbuildbot {x86,x86_64,arm}-generic-full` worked
TEST=`cbuildbot chromiumos-sdk` worked
TEST=build_image boots & runs fine on an alex
Change-Id: I68d848c2e307c98d53a8faa73924f57571f7887b
Reviewed-on: https://gerrit.chromium.org/gerrit/37378
Tested-by: Mike Frysinger <vapier@chromium.org>
Reviewed-by: David James <davidjames@chromium.org>
Commit-Ready: Mike Frysinger <vapier@chromium.org>
Changes to build_image to allow enabling of bootcache.
./build_image --board=$BOARD --enable_bootcache
A board can be configured to use the bootcache by
the following lines in private-overlays/overlay-<board>-private/scripts
if [[ ${FLAGS_bootcache_use_board_default} -eq ${FLAGS_TRUE} ]]; then
FLAGS_enable_bootcache=${FLAGS_TRUE}
fi
Setting --noenable_bootcache or --enable_bootcache on the comand
line will override the default.
BUG=chromium-os:25441
TEST=built and installed snow, amd-64(latitude), stumpy
Change-Id: Ie081ef94f4799b0071b53e0587d89f1247b4a11f
Reviewed-on: https://gerrit.chromium.org/gerrit/38414
Tested-by: Paul Taysom <taysom@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Ready: Paul Taysom <taysom@chromium.org>
This makes a follow up commit easier to document excludes.
BUG=chromium-os:22939
TEST=build_image for daisy installs same set of files
Change-Id: I09a9b3fe6f8c1d1fd9dd4d094f2fb7c81ce24880
Reviewed-on: https://gerrit.chromium.org/gerrit/37377
Reviewed-by: David James <davidjames@chromium.org>
Commit-Ready: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
BUG=None
TEST=build_image for daisy worked
Change-Id: If2343bc1f13a23bc1b6bc0eae9a4ed8938b53eb4
Reviewed-on: https://gerrit.chromium.org/gerrit/37361
Reviewed-by: David James <davidjames@chromium.org>
Commit-Ready: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
Install shim images are broken when booting from legacy x86 BIOS. The
install shim relies upon "cros_factory_install" being passed as a boot
flag. This flag is never passed to create_legacy_bootloader_templates,
so the install shim flow is broken.
This change passes the boot args flags to create_legacy_bootloader... so
the install shim will function normally.
TEST=Create factory_install image, verify correct boot w/ x86 BIOS.
BUG=chrome-os-partner:15661
Change-Id: I46b2be188f48b7626bfd3235d5788410c7488c42
Reviewed-on: https://gerrit.chromium.org/gerrit/36590
Tested-by: Shawn Nematbakhsh <shawnn@google.com>
Reviewed-by: Hung-Te Lin <hungte@chromium.org>
Reviewed-by: Liam McLoughlin <lmcloughlin@chromium.org>
Commit-Ready: Shawn Nematbakhsh <shawnn@google.com>
If the image happens to be full but didn't run out of space,
then don't dump the filesystem debug output. Only do it when
we're erroring out.
BUG=chromium-os:35083
TEST=`./build_image --board=x86-alex` still worked
Change-Id: Ia585b43273cc891aaaebe0fe08aedec78c91055e
Reviewed-on: https://gerrit.chromium.org/gerrit/35885
Reviewed-by: David James <davidjames@chromium.org>
Reviewed-by: Peter Mayo <petermayo@chromium.org>
Commit-Ready: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
Make it possible to run chromeos-install without root
BUG=none
TEST=Build image, chromeos-install, verify ROOT-B is full size
Change-Id: Id506f1e5a6f8b8ee03ea1bdd621aaab1239bca2c
Reviewed-on: https://gerrit.chromium.org/gerrit/34081
Reviewed-by: Don Garrett <dgarrett@chromium.org>
Commit-Ready: Chris Masone <cmasone@chromium.org>
Tested-by: Chris Masone <cmasone@chromium.org>
This change adds support for building the disk layout from a
configuration file. It also cleans up much of the image creation
code.
install_gpt no longer exists, and has been replaced by cgpt.py's
write action. This spits out a file that has two functions that can
be called to write a partition layout to a disk/file. This gets rid
of the gigantic nest of calculations that built the layout previously.
All instances of partition/filesystem sizes in build scripts should now
be gone in favour of calls to the cgpt.py tool.
create_boot_desc has moved inside the base image creation, in an effort
to simplify build_image.
load_kernel_test is gone since it's apparently not supposed to be called
here anyway (asked wfrichar/rspangler about this one).
Base image creation now uses files rather than loop devices when
building an image. This means we can simply umount them once we're
done and not worry about cleaning up the loop device, since it's
been done for us.
Hash pad calculation has been removed. This is now set manually inside
the partition config file.
Hybrid MBR creation is gone, since it's now possible to do that in a board
specific hook (see overlay-beaglebone/scripts/board_specific_setup.sh).
OEM partition now has a filesystem, which is mounted at /usr/share/oem
during emerge so that packages can stash files here.
root_fs_dir and friends are still globals, but the long-term idea
is to make this not the case.
BUG=chromium-os:33817
TEST=All types of images and their respective flows
(VM, recovery, test, factory etc)
Change-Id: I8a596728a4d1845c930e837bea627f5b6a11c098
Reviewed-on: https://gerrit.chromium.org/gerrit/29931
Commit-Ready: Liam McLoughlin <lmcloughlin@chromium.org>
Reviewed-by: Liam McLoughlin <lmcloughlin@chromium.org>
Tested-by: Liam McLoughlin <lmcloughlin@chromium.org>
When we run out of space in the target image, it can be unclear that this
is the actual issue. Detect this case and issue an appropriate warning,
and dump disk usage to help people triage things.
BUG=chromium-os:34167
TEST=`./build_image --board=amd64-generic` passed normally
TEST=`./build_image --board=amd64-generic factory_install` triggered the out-of-space check
Change-Id: I3052892a8c8bc386c7f08e1df26432eea2285563
Reviewed-on: https://gerrit.chromium.org/gerrit/32250
Reviewed-by: Peter Mayo <petermayo@chromium.org>
Reviewed-by: Brian Harring <ferringb@chromium.org>
Commit-Ready: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
BUG=none
TEST=Build a beaglebone image then verify that the beaglebone
bootloader was installed into the ESP
Change-Id: I1037ee6a61d409e8fd2d66ec6746048a54ce8f14
Reviewed-on: https://gerrit.chromium.org/gerrit/26738
Tested-by: Liam McLoughlin <lmcloughlin@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Ready: Liam McLoughlin <lmcloughlin@chromium.org>
No need to dump all these mkfs/dd details early on.
BUG=None
TEST=`./build_image` still works, and doesn't spew as much
Change-Id: Ia8113d2ce2c0e6a8a13535b67b37372f0f146398
Reviewed-on: https://gerrit.chromium.org/gerrit/29366
Reviewed-by: Richard Barnette <jrbarnette@chromium.org>
Commit-Ready: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
To avoid needing to migrate fresh dev images's /var contents when
using the encrypted partition, move the /var bits out of the old
stateful_partition/var location into stateful_partition/var_overlay.
The (initially empty) system /var will continue to either live
in stateful_partition/var (in the unencrypted case), or in
stateful_partition/encrypted/var (in the encrypted case).
The contents needed for gmerge will be symlinked into place at runtime
(via CL I6e68b1f334f5d5b3c4d2977008435bd929191ce7). While the installer
already makes sure that /var is not shipped on an image, this change
additionally make sure that the other contents installed by the ebuilds
will not show up in the actual /var either.
BUG=chromium-os:22172
TEST=link build, boot, install, manual testing.
Change-Id: Ie6480a59929818fe5d36a46abf533b648fb78850
Signed-off-by: Kees Cook <keescook@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/26355
Reviewed-by: Chris Sosa <sosa@chromium.org>
This allows booting on boards that require their bootloader be on
the first partition of a MBR formatted disk
BUG=chromium-os:32150
TEST=Build an image using the --hybrid_mbr_hack flag, verify that
the ESP is visible on a system with GPT support
Change-Id: I4b137ef672b9ed7327bd42ec0a260d82a8c9d470
Reviewed-on: https://gerrit.chromium.org/gerrit/26071
Tested-by: Liam McLoughlin <lmcloughlin@chromium.org>
Reviewed-by: Gilad Arnold <garnold@chromium.org>
Commit-Ready: Liam McLoughlin <lmcloughlin@chromium.org>
Reviewed-by: Liam McLoughlin <lmcloughlin@chromium.org>
Currently, if set -e spots a nonzero exit we basically have
no real debug information- it just stops immediately without stating
where or why. This forces our scripts to be stupidly verbose so
we can track roughly where they were, thus when they fail we can
use that information to localize the rough exit point.
Instead we should be traping that set -e induced exit and
outputing necessary debug information to run it down. This includes
outputing the relevant stack trace, or at least what we can get of
it.
The 'die' function is now enhanced to automatically dump the trace
that lead to it. For most consumers this is desired- however for
commandline parsing induced dies ("--board is missing" for example),
the trace is noise. For those cases, a 'die_notrace' function was
added that retains the original non-backtrace behaviour.
Example output via instrumenting cros_generate_breakpad_symbols
w/ the failing command '/bin/false' (nonzero exit code).
Before:
./cros_generate_breakpad_symbols monkeys --board=x86-alex
<no output at all, just exit code 1>
With this CL:
./cros_generate_breakpad_symbols monkeys --board=x86-alex
ERROR : script called: ./cros_generate_breakpad_symbols 'monkeys' '--board=x86-alex'
ERROR : Backtrace: (most recent call is last)
ERROR : file cros_generate_breakpad_symbols, line 207, called: main 'monkeys' '--board=x86-alex'
ERROR : file cros_generate_breakpad_symbols, line 163, called: die_err_trap '/bin/false' '1'
ERROR :
ERROR : Command failed:
ERROR : Command '/bin/false' exited with nonzero code: 1
BUG=chromium-os:30598
TEST=inject a failing command into a script, verify the output.
TEST=inject a 'command not found', verify the output
TEST=cbuildbot x86-generic-full --remote
TEST=cbuildbot arm-tegra2-full --remote
TEST=cbuildbot chromiumos-sdk --remote
Change-Id: I517ffde4d1bb7e2310a74f5a6455b53ba2dea86c
Reviewed-on: https://gerrit.chromium.org/gerrit/17225
Reviewed-by: Brian Harring <ferringb@chromium.org>
Tested-by: Brian Harring <ferringb@chromium.org>
Commit-Ready: Brian Harring <ferringb@chromium.org>
With recent changes (transitioning to 64 bit x86 kernels and the 3.2
kernel) the factory install image EFI partition gets overflown,
resulting in build failures.
This change makes sure that factory shim image gets allocated twice
the room for the EFI partition (32M).
BUG=chromium-os:27639
TEST=manual
run the following commands while in chroot:
cd ~/trunk/src/scripts
./build_image --board=lumpy
./build_image --board=lumpy --replace --symlink=factory_shim --build_attempt=3 factory_install
cd ~/trunk/src/build/images/lumpy/factory_shim
./unpack_partitions.sh factory_install_shim.bin
ls -l part12
cd ~/trunk/src/build/images/lumpy/latest
./unpack_partitions.sh chromiumos_image.bin
ls -l part12
. observe that in factory shim case the part_12 partition size is
33554432 bytes, and in the regular image case the part_12
partition size is 16777216
. try using the factory_shim USB stick in recovery mode, observe the
system come up (did not have a server set up, so the full install
process was not verified).
Change-Id: Ibe001ec37c752dca90ec30ae056a67610e39a8fb
Signed-off-by: Vadim Bendebury <vbendeb@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/17907
Using these flags directly was deprecated a while back and is causing confusion
which way is the right way. This CL removes all these FLAGS from build_image
(though keeps support of them in mod_image_for_test/image_to_usb invocations
in common.sh). In this change I've also defaulted build_image to build the
developer image (and only the developer image).
BUG=chromium-os:27362
TEST=Been busy testing. Have built the following combinations and verified them:
build_image base
build_image
build_image base dev test
build_image dev
build_image dev factory_test
build_image factory_install
Change-Id: Ie534c276a9ec571926964320ac176daa91b12a81
Reviewed-on: https://gerrit.chromium.org/gerrit/17386
Tested-by: Chris Sosa <sosa@chromium.org>
Reviewed-by: Richard Barnette <jrbarnette@chromium.org>
Commit-Ready: Chris Sosa <sosa@chromium.org>
We have helper funcs already for displaying messages, so convert
some raw `echo` calls to them.
BUG=None
TEST=build_image still works and boots
Change-Id: Ie66cc59d0362ef6aa19011fa6cb0bc64a6a4fce8
Reviewed-on: https://gerrit.chromium.org/gerrit/14478
Reviewed-by: Richard Barnette <jrbarnette@chromium.org>
Commit-Ready: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
This is final step in ext4 conversion. We have been running with
the ext4 file system but ext3 format for several weeks. This changes
uses the ext4 format for the stateful partition.
Removed make_developer_script_runner.sh as per review.
BUG=chromium-os:20012
TEST=Ran ext4 on all platforms
Change-Id: I8e1564bfa576e9a0ad810879c18223a6c68b18e1
Reviewed-on: https://gerrit.chromium.org/gerrit/11220
Commit-Ready: Paul Taysom <taysom@google.com>
Reviewed-by: Paul Taysom <taysom@google.com>
Tested-by: Paul Taysom <taysom@google.com>
There appear to be race conditions in some environments around mounting
and unmounting a file system quickly failing due to a device in use error.
This change removes the extra umount/mount/rm/umount sequence around
rootfs/filler which was done to ensure the file was being synced to disk.
Instead the dd command has added conv=fdatasync which causes it to sync
the file to disk on completion.
This also fixes a possible loopback device leak where the unmount would fail
and then build output directory deletion would fail because the rootfs was
still mounted, leaking both disk space and loopback devices.
BUG=chromium-os:22308
TEST=Ran filefrag on filler, validated that rootfs.image has blocks of 0 bytes
in each position
Compressed chromiumos_base_image.bin, compared size to previous build_image
Booted image on tegra2 device, confirmed it still boots
Change-Id: I097440b7abefd4a0d25f743e968f33a1531a7a21
Reviewed-on: https://gerrit.chromium.org/gerrit/11487
Reviewed-by: Thieu Le <thieule@chromium.org>
Tested-by: Jon Kliegman <kliegs@chromium.org>
Reviewed-by: Paul Taysom <taysom@google.com>
Reviewed-by: David James <davidjames@chromium.org>
Commit-Ready: Jon Kliegman <kliegs@chromium.org>
Reviewed-by: Jon Kliegman <kliegs@chromium.org>
This also simplifies the factory_install logic to all be in the same place :).
BUG=chromium-os:22371
TEST=Ran trybot with this CL + tegra2 build that used make_netboot.sh that uses
the factory install which failed on the buildbot previously.
Change-Id: Id0847b85c061cf230d0e6cc5bb40de2eba0992b9
Reviewed-on: https://gerrit.chromium.org/gerrit/10986
Tested-by: Chris Sosa <sosa@chromium.org>
Reviewed-by: Richard Barnette <jrbarnette@chromium.org>
Commit-Ready: Chris Sosa <sosa@chromium.org>
Fixed all callers in my big CL to no longer have to wrap poorly using
\ and start from the beginning of the next line but rather pass in
an array of args i.e.
info "tacos" " are" " delicious".
BUG=None
TEST=Ran to see the errors in parse_build_image and manual eyeing.
Change-Id: I5eac8a5ae7a8d314dbc4e821ee33cf88213711d0
Reviewed-on: https://gerrit.chromium.org/gerrit/10823
Commit-Ready: Chris Sosa <sosa@chromium.org>
Reviewed-by: Chris Sosa <sosa@chromium.org>
Tested-by: Chris Sosa <sosa@chromium.org>
This CL does two things.
First it introduces an argument interface that works with all the following
combinations:
factory_install | [base|dev[test|factory_test]] rather than use --test etc.
This does not build extra images. If you just want a test image you can
run build_image --board=<board> test and all that's in your latest_dir is
chromiumos_test_image.bin.
Second, we only build what we ask and only finalize what we ask so on
an invocation of build_image test we only actually run cros_make_image_bootable
once saving 2 minutes on my machine (from 6:50->4:50 on multiple iterations).
BUG=chromium-os:22048
TEST=build_image test, all possible combinations of args through to start of
build. Built factory_install using new and old methods and dev test with both
methods. Also verified mod_image_for_test still works as advertises stand-alone.
Change-Id: I9fe2feb50a941c007214decd9ba1627012c050af
Reviewed-on: https://gerrit.chromium.org/gerrit/10621
Commit-Ready: Chris Sosa <sosa@chromium.org>
Reviewed-by: Chris Sosa <sosa@chromium.org>
Tested-by: Chris Sosa <sosa@chromium.org>
Verified boot is not supported yet for USB+EFI boot, which is the case
of factory install shim. We need to select normal boot in legacy boot
loaders when building factory install shim.
BUG=chrome-os-partner:6358
TEST=build_image --factory_install
# boots successfully under ZGB H2O firmware
Change-Id: Ic4645a1766514bf7e0c31f8a4df21452bf195705
Reviewed-on: http://gerrit.chromium.org/gerrit/10013
Tested-by: Hung-Te Lin <hungte@chromium.org>
Reviewed-by: Richard Barnette <jrbarnette@chromium.org>
Reviewed-by: Nick Sanders <nsanders@chromium.org>
