This replaces the cross-toolchain compile step in bootstrap_sdk and adds the
ability to build native toolchains using the cross toolchain. This is just
the first step towards actually providing the native toolchain in a container.
We don't need to reserve space on disk just to reserve partition
numbers. And now that partitions are aligned these blanks spots grew
from 512 bytes to 1MB which is not much but still silly.
When using anything other than classic spinning disks with 512 sectors
it is generally best to maintain some alignment with the underlying
physical sector or erase block size. The default alignment most
partitioning tools use these days is 1MB (2048 sectors). Also sometimes
qemu-img requires disk sizes to be aligned to 64KB.
The existing code arbitrarily multiplies START_SECTOR by 512 converting
from blocks/sectors to bytes, but blocks was the correct unit to begin
with. Also the secondary GPT area is not considered but that was OK
because the bogus unit conversion oversized our disks by almost 16MB.
Instead of relying on bugs properly reserve 34 sectors at each end of
the disk. (Well, we could get away with only 33 at the end since it
doesn't have a MBR but meh.)
Previously shifts were added into the getopts loop to work around
differences between different sh implementations but that causes getopts
to end the loop early. Instead use an intermediate variable to work
around inconsistent OPTIND behavior and explicitly check for the --
separator. Tested in bash, dash, and ash.
We don't have any particular reason for the weird hackery required to
install packages into /usr/local instead of root. The rootfs image is
already being modified a little might as well modify it a lot. :)
Vagrant users are accustomed to much larger disk sizes so lets give it
to them. I'm leaving the others as-is since it is easier to grow than
shrink disks if anyone has a particular size they need.
Use the smaller base format for 'raw' disk images since these will
usually be dd'd to a block device to create AMIs and what not. For
images using qcow2 and vmdk stick with the larger vm size.
This reverts commit b97cfe126f.
The minor device numbers of loop partitions are allocated dynamically
which significantly complicates dunning under Docker which uses a static
/dev. Rolling this back until we can rely on /dev being dynamic.
If git is installed via coreos-dev in the STATE partition it will need
some help finding its install location since it was built thinking it
would be installed in /usr rather than /usr/local.
This avoids the need to dd individual filesystem images into a complete
disk image, just mount the partitions directly from a loop device
covering the whole image. This does add the requirement that mkfs run as
root but that isn't a problem.
These are just cluttering things and adding an element of "how does this
work?" because base_image_util was defaulting to the "usb" layout in
some places and "base" in others.
This change removes /usr/sbin/write_gpt.sh from images which we have no
use for. This allows us to drop the indirection of writing partition
tables by first writing out a script to call. Now cgpt.py can call cgpt
directly to initialize the partition layout. This opens the way for
further improvements to how disk images are created.
This currently does nothing because our state partition is not partition
number 1. Even if it did we don't really needed it since we rely on
expanding on boot instead.
Remove --verity_*: Unused, we don't support verity
Remove --usb_disk: Unused, we use PARTUUID now.
Remove --enable_serial: Unused, and serial is enabled for syslinux
Right now the initial (pre image_to_vm) images oversize the root
partitions, creating the expected 1GB filesystem in a 2GB partition.
image_to_vm later shrinks the partition back down to match. Just start
out with 1GB partitions to begin with instead.
This one is more automagical and sets up ssh keys from ssh-agent and the
user's home directory by default. Also adds an option for setting the
ssh port so it can be something other than 2222. Script should be
sufficiently portable, tested in bash, dash, and ash.
Useful for qemu -nographic or any any other situation where serial is
easier to get at than VGA. It may be possible that in some setups ttyS0
isn't appropriate but we can figure out a way to customize kernel
options if/when that ever comes up.
Remove unused dev/dm-0 vs dm-1 logic from verity and the associated
rootwait option it required (meaningless with our initrd). Move old
cros_legacy to common instead of using it in every command line option.
We should remove it entirely soon since it isn't useful for us. Remove
unneeded intel graphics modeset option.
I want to start including version info in SDK builds as an alternative
scheme to the existing "chroot_version_hooks" system which always
assumes freshly unpacked SDKs are the latest regardless of what version
they actually were.
The recommended command using the config file was triggering a massive
memory leak in qemu because it was adding both the default virtual
hardware nic as well as the virtio nic. This could be worked around by
adding something like -net none or moving all the -net commands from the
file to the command line but eh. Clearly qemu config files are used and
tested by nobody else so lets just use a trusty script instead.
Vagrant will need the virtualbox ovf plus its own Vagrantfile config.
After this we will need an optional "package" step to this script to
take these files and bundle them into a .box tarball. This could also be
used to switch from .ovf+vmdk for plain virtualbox images to a bundled
.ova archive which combines the two.
Trying to include version info by adding the directory name to VM image
names didn't work and a better solution is for build_image to write out
a version.txt file. This should also fix an issue where uploading from
image_to_vm.sh didn't always go to the same location as the images
uploaded from build_image did.
As of Linux 3.2 loopback supports discard by punching holes in the
underlying file. This doesn't actually seem to impact things right now
since we are writing to fresh filesystems but might as well do this to
prevent wasted space from sneaking in later on.
Enable sparse files for all dd and cp commands and replace some dd
commands that are really better off being truncate commands.
While in the neighborhood there were a number of useless sudo commands
for things that just happen to be in sbin. Call them directly instead.
Its single use is in build_common and even then having a little progress
bar for copying images isn't that interesting, they just get lost in the
noise of the emerge output. Keep it simple, use cp.
/mnt/stateful_partition was already a little unruly with
/mnt/stateful_partition/home and /mnt/stateful_partition/var_overlay
serving similar functional purposes.
Then we needed to also add /opt and /srv overlays.
I also have wanted to get rid of the ugly and weird
/mnt/stateful_partition name so lets just have one big move.
/mnt/stateful_partition -> /media/state
/mnt/stateful_partition/var_overlay -> /media/overlays/var
/mnt/stateful_partition/home -> /media/overlays/home
From there we add /media/overlays/srv and /media/overlays/opt
The basic system directory structure including the lib symlinks were
fixed for sysroot in the following commits but the image build uses an
entirely different bit of code to do the exact same set of hacks. Port
those changes to the image building code to hopefully make all happy.
2ae0c30f4eac931bd088
Add --production_track argument to set_lsb_release and
cros_make_image_bootable to support using the production update service
on developer builds of the 'prod' image. This replaces the previous hack
of setting COREOS_OFFICIAL=1 in the middle of the build.
Since lsb-release doesn't exist prior to the first call to
set_lsb_release switch to sudo_clobber instead of append. That way if it
is called a second time later the contents aren't duplicated.
Write the info to gentoo-release and os-release as well so everything
gets the same information.
Last minute bug slipped in because of a line I commented out since the
current coreos kernel doesn't support virtio block devices (that change
coming soon). Qemu doesn't tolerate any spaces before # in comments.
The old script was heading towards spaghetti code realm. This breaks up
all the image variations such as hybrid MBR, OEM packages, etc into
configuration options and small functions that actually do the work.
All this is in the new vm_image_util.sh library but the command line
parsing and overall procedure remains in image_to_vm.sh
As part of this we gain support for putting some qemu options in a
config file as well as Xen virtual machines using pygrub and pvgrub.
Lots of generally unused options have been removed to simplify things
and keep output file names consistent.
reintroduce unique A/B menu.lsts to work around the kexec problems that
we have. Essentially instead of always using boot_kernel on pvgrub
systems use the A/B kernels installed at update time to the boot
partition.
Switching the toolchain to upstream Gentoo brought this directory back
and based on the Chromium OS history keeping this directory out of the
builds is a bit tedious. Keeping image sizes down isn't *that* important
right now so just let it be.
This adds the boot_kernel to the build boot partiton and updates the
relevant config files. Mission accomplished.
TODO: Update the installer to not worry about moving files around
anymore
Sync up bootstrap_sdk with other tools by using the common upload
functions. As part of this refactor release_util a bit to provide a
truly generic upload function.
This will be used to upload the latest images built from master, we
don't need every build so we just want to upload to a 'master'
directory, not one named for the current version.
Add core_upload_update to au-generator.zip which requires some extra
logic to make it runnable anywhere it may be. To organize the code a
little better all the delta_generator calls have been moved to
cros_generate_update_payload. core_upload_update is now just a wrapper
around cros_generate_update_payload and core-admin.
Cgpt was moved and a symlink based wrapper was added. That wrapper will
be improved soon, when when that's true we'll need to change this back.
A specific note... cgpt is currently statically linked. If that wrapper does
not remain statically linked, then a simple revert won't be enough.
BUG=chromium-os:39814
TEST=Manual au-generate.zip creation.
Change-Id: I2705b1eddd8ef28c7eb099512513daf80f586218
Reviewed-on: https://gerrit.chromium.org/gerrit/45128
Reviewed-by: Chris Sosa <sosa@chromium.org>
Commit-Queue: Don Garrett <dgarrett@chromium.org>
Tested-by: Don Garrett <dgarrett@chromium.org>
Make use of the new partition UUIDs for ROOT-A and ROOT-B in the root=
kernel parameters provided by the legacy (non-kexec) bootloaders. This
makes all of our images bootable as-is without having to pass them
through image_to_vm.sh. :-D
Before we can switch from using device names in root= to partition table
UUIDs we need some values that will remain consistent across upgrades
since the partition table is not updated when filesystems are.
vboot_reference now recognizes coreos-reserved and coreos-rootfs. Use
these prefixes so we stop using the chromeos GUIDs.
Test-plan: Tested on a VM and it boots and updates.
A few things here:
- Source manifests/version.txt directly instead of coreos-version.sh
- Remove Chrome branch from target image directory names.
- Use proper version instead of timestap for catalyst builds.
- Move lsb_release script from coreos-overlay to build_library.
During builds var_overlay is always mounted over /var. We want to do the
same at run time but we also want to ensure everything expected to be
there always does. After emerge completes gen_tmpfiles.py will scan /var
for any .keep files that were installed and records their parent
directories' permissions and ownership to /usr/lib/tmpfiles.d. On each
boot systemd will automatically recreate anything that goes missing.
This also means that going forward any ebuild that needs a directory in
/var (or anywhere else the stateful partition is bound) can simply rely
on the 'keepdir' ebuild function instead of adding things to
coreos_startup.
As outlined here we need a new partition layout, this patch makes the
necessary changes:
https://groups.google.com/forum/#!topic/coreos-dev/bA7gwGGoTng
The first big change is making all of the scripts obey partition numbers
based on labels in the disk_layout.json. This makes it much easier to
change later on.
The second big change is in the layout itself. The json file was updated
to reflect the document above.
And finally the grub boot configuration needed for pv-grub and pygrub
were added to the create_legacy_bootloader_templates.sh library utlity.
Everything seems to work and boot now.
this is a bit of a hack but I wanted to see if it had any utility during
development before making it all pretty. Essentially this is a copy of
build_image but instead of building up an entire image it simply puts
the files into directories on disk to be ran with systemd-nspawn/lxc/etc
so it is a bit complicated but essentially gtest pulls in python which
pulls in pyton-updater which wants portage so portage gets installed in
teh real root not the dev one. Just leave it for now.
on Fedora 18 on Gnome 3.0 something is making the first attempt at
unmounting return busy. Unfortunatly, the return code is 32 everytime
so we have to parse the output of umount :( :( :(
Change-Id: I7f94bf6c2059c7e7cb4fb173d9ffbabd59f2b24f
Change the order of partitions to given better performance for spinning disks.
However, this change affects all devices, not just those with spinning disks.
We will be able to remove the per-board overrides from several devices if/
when this change is accepted.
BUG=chromium-os:38122
TEST=image_to_live, (Will run trybots, and perform a recovery).
Change-Id: Ibe60e384e0392936f9f8ceb326618951367974df
Reviewed-on: https://gerrit.chromium.org/gerrit/41792
Reviewed-by: Chris Sosa <sosa@chromium.org>
Tested-by: Don Garrett <dgarrett@chromium.org>
Commit-Queue: Don Garrett <dgarrett@chromium.org>
Suspecting this CL is killing devices in the lab: crosbug.com/38483
This reverts commit 38fe561a92
Change-Id: I6a6f0c170d5410de79ad0d2b96c7b9c95046cae4
Reviewed-on: https://gerrit.chromium.org/gerrit/42452
Reviewed-by: Scott Zawalski <scottz@chromium.org>
Tested-by: Simran Basi <sbasi@chromium.org>
This is needed for vmtests to have enough space, because default size,
although expandable, is too small in vm images.
BUG=chromium-os:38088
TEST=cbuildbot x86-generic-asan amd-64-generic-asan --remote -g I07d31d8b -gI18fab01e -gI9f55fcd3
Change-Id: I9f55fcd3707859dca8304b8e0c78ebd39afea566
Reviewed-on: https://gerrit.chromium.org/gerrit/42390
Tested-by: Denis Glotov <glotov@chromium.org>
Reviewed-by: Don Garrett <dgarrett@chromium.org>
Commit-Queue: Denis Glotov <glotov@chromium.org>
This is to be used with build_image and image_to_vm by cros buildbots
that need more rootfs size.
Unlike 2gb-rootfs, it has both ROOT-A and ROOT-B enlarged to 2gb, so
that image update test (1_update in SimpleTestsUpdateAndVerify) is possible.
BUG=chromium-os:38088
TEST=cbuildbot x86-generic-asan amd-64-generic-asan --remote -g I6f497214 -g I07d31d8b
Change-Id: I6f497214f54f4457cf2b40610beb9272622d1a47
Reviewed-on: https://gerrit.chromium.org/gerrit/41814
Tested-by: Denis Glotov <glotov@chromium.org>
Reviewed-by: Don Garrett <dgarrett@chromium.org>
Commit-Queue: Denis Glotov <glotov@chromium.org>
The latest coreutils introduces a status=none option which suppresses
all information which is more than the current status=noxfer.
BUG=None
TEST=`./build_image` no longer spews any dd info
Change-Id: I3cfefed1d38e7e5ff52342c98e6a306d9a48950a
Reviewed-on: https://gerrit.chromium.org/gerrit/41930
Reviewed-by: David James <davidjames@chromium.org>
Commit-Queue: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
Enable the serial port from the build_image command line.
Currenly, you have to edit build_kernel_image.sh to enable
the serial port for kernel debugging. Now:
./build_image --board=${BOARD} --enable_serial=ttyS0
Will enable sending printks to the specified serial port.
BUG=chromium-os:38026
TEST=built images with and without serial enabled
Change-Id: I9ef4f2a20f0d451e132371339c4eba1faf4c94de
Reviewed-on: https://gerrit.chromium.org/gerrit/41638
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Queue: Paul Taysom <taysom@chromium.org>
Tested-by: Paul Taysom <taysom@chromium.org>
We don't need to support POSIX shell, so convert to bash.
BUG=None
TEST=`cbuildbot {amd64,x86,arm}-generic-full chromiumos-sdk daisy-release` worked
Change-Id: I33ad25d2310c593f3e346d955e3aa27da41091fc
Reviewed-on: https://gerrit.chromium.org/gerrit/41271
Reviewed-by: David James <davidjames@chromium.org>
Commit-Queue: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
By default, pv automatically infers and uses the full width of the
terminal. This generally makes sense for a console application, but in
the case of pv it just causes the progress bar to be ridiculously wide
when run on wide terminals, to the point it's hard to read. This CL is
setting it to 80 characters, the widely accepted standard width for
a terminal, in cases where the terminal appears to be larger than
80 columns. Note that:
* Even with -w, pv appears to be resizing the progress bar as the
terminal width changes midway through the run. This means that if
a user widens the window, then the progress bar will go wide again and
there's nothing to be done about it.
* Theoretically, in very rare cases this may lead to a progress bar the
exceeds the width of the terminal (i.e. set to 80 columns on
a terminal that has just shrunk to fewer columns). The odds for such
timing are close to nil and even then the damage is minimal.
* This will work for non-terminal runs, or otherwise runs where stty
does not produce any output.
* To avoid the initialization overhead for all common.sh inclusion,
replacing the variable with a function that prints the pv/cat command.
BUG=None
TEST=Ran ./image_to_usb on wide and narrow terminal windows, it works.
Change-Id: I549df1dd29e93909ea646ae9b9e09d9a588ad382
Reviewed-on: https://gerrit.chromium.org/gerrit/40937
Commit-Queue: Gilad Arnold <garnold@chromium.org>
Reviewed-by: Gilad Arnold <garnold@chromium.org>
Tested-by: Gilad Arnold <garnold@chromium.org>
CrOS could be used in devices that dont have custom firmware to
interpret the Kernelblock (Part2,4). In such devices skip trying
to make the kernelblock
BUG=none
TEST=./build_image --board=chronos test and verify we dont build the kernel block
Change-Id: I3e9a8dd765ea00db2ebf112553d96c08960e544f
Reviewed-on: https://gerrit.chromium.org/gerrit/40219
Commit-Queue: Anush Elangovan <anush@chromium.org>
Reviewed-by: Anush Elangovan <anush@chromium.org>
Tested-by: Anush Elangovan <anush@chromium.org>
Now that we have an ebuild for factory install shim, let's move to that
instead of base image.
BUG=chrome-os-partner:16712
TEST=Test network boot, install shim, RMA shim on Link.
Test install shim on Snow.
CQ-DEPEND=CL:39952
Change-Id: Ib6a7a598087647b44a4aeca970e293ea72023890
Signed-off-by: Vic Yang <victoryang@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/39955
Reviewed-by: Richard Barnette <jrbarnette@chromium.org>
Reviewed-by: Hung-Te Lin <hungte@chromium.org>
When changing the size of the stateful partition, we must be able to
operate on any image, with any arrangement of partitions. The static
templates cannot be used because we don't know what we're starting with.
As such, this change makes the update function walk the list of
partitions, duplicating all their details, and moves any located after
stateful by the change in size, and copies in the new stateful contents.
BUG=chromium-os:37080
TEST=link build and decryption recovery tested
Change-Id: I1131dd8ee91e5db2556bdf8f7ca12b08f35da32b
Signed-off-by: Kees Cook <keescook@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/39424
Reviewed-by: Will Drewry <wad@chromium.org>
Reviewed-by: Liam McLoughlin <lmcloughlin@chromium.org>
Tested-by: Liam McLoughlin <lmcloughlin@chromium.org>
check_deps checks for libraries using RPATH, but does not take the
similar RUNPATH into account. For the purposes of chasing down
library dependencies, finding a library on either path suffices.
BUG=None
TEST=Build an image that includes binaries and associated libraries in a
non-standard location, with the binaries and libraries setting an
RUNPATH (and not an RPATH) pointing at that non-standard location.
Change-Id: Ic930bbacbe5c8ddeb367c39960dadea8aaba0cb2
Reviewed-on: https://gerrit.chromium.org/gerrit/39397
Tested-by: Josh Triplett <josh@joshtriplett.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Ready: Josh Triplett <josh@joshtriplett.org>
This removes files only used at link time, or when manually debugging,
or by glibc's charset routines (gconv) which are not used by Chromium.
BUG=chromium-os:22939
BUG=chromium-os:23105
TEST=build_image for daisy works
TEST=`cbuildbot {x86,x86_64,arm}-generic-full` worked
TEST=`cbuildbot chromiumos-sdk` worked
TEST=build_image boots & runs fine on an alex
Original-Change-Id: I68d848c2e307c98d53a8faa73924f57571f7887b
Change-Id: I25ffda36d28fd114715053c3a2590cc2248688bf
Reviewed-on: https://gerrit.chromium.org/gerrit/39214
Reviewed-by: David James <davidjames@chromium.org>
Commit-Ready: Darren Krahn <dkrahn@chromium.org>
Tested-by: Darren Krahn <dkrahn@chromium.org>
BUG=chromium-os:36862
TEST=./build_image factory_install --board=parrot
Change-Id: If1e59d09eec45340e49b2711b4799eb1f83ea007
Reviewed-on: https://gerrit.chromium.org/gerrit/39133
Tested-by: Paul Taysom <taysom@chromium.org>
Reviewed-by: David James <davidjames@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Ready: Paul Taysom <taysom@chromium.org>
