This just sets the code file size to the var file size, so it gets
zero-padding without having to pipe commands together.
From: David Michael <david.michael@coreos.com>
[Rebased]
Signed-off-by: Geoff Levand <geoff@infradead.org>
This reverts the vagrant image back to using oem-vagrant because we
don't want to break the existing images. It moves the new,
Ignition-powered virtualbox flavor of vagrant into a new image.
This changes the oem-package for vagrant to vagrant-virtualbox,
which uses ignition instead of cloud-clonfig and sets the oem id
to "virtualbox" so that ignition can handle the machine correctly
This adds the option --torcx_store to specify the path to a
directory containing torcx images to be baked into the OS image. A
blank string can be given instead of a path to restore the previous
behavior and leave an empty vendor store.
The default value is the default path created by build_torcx_store,
which is used when build_packages updates torcx images. This means
that the current pattern "./build_packages && ./build_image prod"
should result in a fully updated OS image with all torcx images
available in the vendor store.
When OVA template is not being used, the default dhcp value yes will
trigger cloud-init to generate a 00-.network file, which will break
network connectivity Intermittently. Please see the details here:
https://github.com/coreos/bugs/issues/1802#issuecomment-297847614
When building dev images, the PORTAGE_BINHOST value during build
time is written to the image's make.conf. This breaks the default
binary package setup, since Jenkins is using gs:// URLs for signed
package verification and authenticated downloads, and the make.conf
doesn't inherit the GS_* variables to handle those schemes.
This should be reverted when signed packages are properly supported
by default in the dev images.
This changes the format from:
sys-apps/systemd-212-r8::coreos GPL-2 LGPL-2.1 MIT public-domain
to a JSON structure:
[
{
"project": "sys-apps/systemd-212-r8::coreos",
"license": ["GPL-2", "LGPL-2.1", "MIT", "public-domain"]
}
]
We don't have to worry about the changing format because the previous
format was never published. This is designed to match the
bill-of-materials [1] format so that it can be consumed by the site.
[1]: https://github.com/coreos/license-bill-of-materials
INFO build_oem_aci: Writing coreos_oem_gce_aci_stage_packages.txt
awk: cmd. line:1: fatal: cannot open file `/build/amd64-usr/var/db/pkg//DEPEND' for reading (No such file or directory)
INFO build_oem_aci: Writing coreos_oem_gce_aci_stage_licenses.txt
awk: cmd. line:1: fatal: cannot open file `/build/amd64-usr/var/db/pkg//DEPEND' for reading (No such file or directory)
We've always generated these license manifests (detailing which ebuilds
are covered by which license), but never published them. This adds these
manifests to the list of published files so that they are publicly
available.
os-release is requested in bug reports, and knowing which board
the problem occurred on is often helpful.
Signed-off-by: Geoff Levand <geoff@infradead.org>
Detect first boot based on the existence of a coreos/first_boot file
in the EFI partition, and set "coreos.first_boot=detected" command line
argument when found. We use "detected" rather than "1" so the initramfs
knows that it should mount the ESP and delete the file. This lets us
defer clearing the first-boot flag until Ignition has run successfully,
without having to change the disk GUID after filesystems are mounted.
Continue detecting the first-boot disk GUID and adding the command-line
argument to randomize it, since we still want unique disk GUIDs
regardless of Ignition.
This is useful for emerges that are meant for incomplete rootfs's, such
as ACI building emerges. There are cases where the #! check is expected
to fail while doing those.
This reverts commit a7ffba9a9f.
The build_image script can build multiple formats. When our
releases and automated builds are creating developer containers and
production images from the same command, the verity flag would be
disabled while building the container and remain disabled when building
the production image. This resulted in no verity in all our builds.
Allow "coreos-install -o vmware_raw" to install Container Linux with
the vmware OEM.
Use base DISK_LAYOUT to reduce the minimum disk size.
Fixescoreos/bugs#359.
They're not in the root fs, but they are in the initramfs. Handle this
by augmenting the package list with packages that are both
- build dependencies of coreos-kernel, and
- configured to cause rebuilds of coreos-kernel when their sub-slot
changes.
To clean things up and prepare for arrm64 support move
all the enable_rootfs_verification processing into one
location and add some comments.
Signed-off-by: Geoff Levand <geoff@infradead.org>
To make verity work both enable_rootfs_verification and enable_verity
need to be set. Without one verity just gets half enabled. Remove
the enable_verity flag and do the full verity setup when
enable_rootfs_verification is set.
Signed-off-by: Geoff Levand <geoff@infradead.org>
The disable_read_write variable was just a copy of FLAGS_enable_rootfs_verification,
so to make things less confusing just use FLAGS_enable_rootfs_verification.
Signed-off-by: Geoff Levand <geoff@infradead.org>
Add a new grub variable extra_options, the contents of which is
added to the linux command line. Use extra_options to set
the ACPI options needed for arm64.
Signed-off-by: Geoff Levand <geoff@infradead.org>
Two new image types have been added:
1. parallels - this produces VM images with extension pvm.tgz that can be loaded directly into Parallels Desktop
2. vagrant_parallels - this produces a Vagrant box that works with parallels vagrant provider (http://parallels.github.io/vagrant-parallels/)
Just like vmdk and others we rely on qemu-img to convert raw images. Support for Parallels disk images was added to qemu-img in version 2.4.
I also removed the box files from the actual image since there are not needed in /usr/share/oem.
Signed-off-by: Bassam Tabbara <bassam.tabbara@quantum.com>
The ACI root is created by reusing the create_prod_image function
to install a base meta-package. It then runs a script to customize
the file structure as required by agent software (if necessary),
writes a manifest file from a supplied template, and then packages
it all into a tar file.
The Xen loader in GRUB never received support for our hacky scheme of
adding the verity hash to the kernel cmdline. Disable till that's fixed.
Partially reverts 2016567 and 533b1b9.
Consolidates two very similar flags into one and fix an issue where
verity could get enabled in the GRUB config when rootfs verification was
turned off (e.g. on arm64 which cannot use verity yet).
workaround for bootstrap_sdk on an Ubuntu host where /dev/shm is a
symlink to /run/shm. Since we mount the hosts /dev (for losetup) this
interferes with building python 2.7. The workaround is to disable the
/dev/shm during python builds. A longer term fix would be to not mount
the hosts /dev. Thanks for marineam for suggesting the fix on IRC.
If the gptprio.next command fails to give us something to boot we
shouldn't try! In order to diagnose why the failure happened halt
immediately so the user can see the error message.
Once we've built the packages, verify against the Gentoo Linux Security
Advisories to ensure that we're not shipping anything with known
vulnerabilities.
Instead of patching portage to support the `disabled` flag now we just
patch it to leave the `[gentoo]` section out of the default repos.conf.
Follow up to 585275b268
PROD_IMAGE is a flag that indicates a production image should be
built, and will be set for dev builds if the user specifies that
both dev and prod images should be built. build_image was
incorrectly using the PROD_IMAGE variable to conditionaly do some
setup depending on the image type.
Add a new variable IMAGE_BUILD_TYPE that can be tested for the type
of image currently being built and replace the PROD_IMAGE usage.
Signed-off-by: Geoff Levand <geoff@infradead.org>
A bunch of packages install PAM configuration fragments in /etc. Rather than
modify them all to install into /usr/lib, just move the entire directory at
image build time.
We need to ship some PCR measurements alongside images in order to make it
easier for admins to provide an appropriate policy. Add some tooling to
generate the appropriate hashes during build, pack those into a zip file
and upload it.
profile is already set up to source /usr/share/baselayout/profile.env
but it never has because I forgot to add this line during the migration
to amd64-usr images. Sure took us a while to notice that one... :(
This resolves two issues:
- Large dependencies are *never* built during image_to_vm,
build_packages must now handle that.
- Since build_packages can't resonably do the oem-* packages (they all
conflict with eachother) we do want to build them from the ebuild.
This is now enforced so a old binpkg is never used. This resolves
confusing issues people have always had while when editing oem
ebuilds but getting a stale build instead.
Allows build_image to be used without first running build_packages.
Note: setup_board --force is required before build_packages will work
properly after doing this since baselayout won't be installed otherwise.
- May be sourced early, so explicitly die if source fails.
- Add a function for getting the latest version of a package.
- Read PROVIDES metadata using portageq, enabling data to be read from
binary packages in addition to installed packages. The performance
issue is not an issue here and needed to support empty build roots.
Most vm images have an expanded root partiton to make them practical to
use as-is. Some deployments may not want such a large root, putting most
storage on other volumes.
This variable was semi-deprecated ages ago so `version.txt` could follow
a similar variable naming pattern to `os-release`. Finally drop usage of
it here in favor of `$COREOS_VERSION`.
The one-liner `[[ -z ${PIPESTATUS[*]#0} ]]` no longer works because the
expansion still includes spaces even if all the values are zero. Somehow
that didn't matter in bash 4.2 but it does mater in 4.3 to be consistent
with the general behavior of variables in [[ tests.
The console often contains very useful information in the event of a
hard crash, in such situations there's no ability to unblank the console
via keypress because the kernel won't handle the interrupt.
Since CoreOS is a server/cluster operating system, there won't generally
be monitors connected benefitting from a blanked console. Disabling the
blanking altogether allows the frame buffer contents to always be
visible, even when the kernel can't handle keypresses.
coreos.first_boot=1 will no longer trigger disk-guid randomization, so
manual ignition triggers in diskless/pxe scenarios may succeed. Instead
we explicitly request the randomization when first_boot=1 was added by
grub finding the 00000000-0000-0000-0000-000000000001 disk-guid.
In order to boot properly we need `rootflags=rw mount.usrflags=ro` on
the command line. These have been build into the kernel directly but for
arm64 builds the built in options seem to be ignored.
Add the necessary variables in grub.cfg and populate the EFI
partition with arm64 efi executable and modules.
Signed-off-by: Andrej Rosano <andrej@inversepath.com>
Now that ccache is turned on by default in the profile portage complains
a lot if ccache isn't actually installed, sleeping 5 seconds for each
error message. Since pkgcache is in use ccache isn't going to make that
much of a difference but getting rid of those 5 second sleeps will. :)
Failing to explicitly set the selinux policy store to operate on may
result in semodule installing the policy in an incorrect location. Pass
it on the command line in order to avoid this.
ldconfig does not work for non-native arches. Create a new
build_image routine run_ldconfig that uses qemu user emulation
to run the board ldconfig on the board rootfs when the board and
SDK arches are different.
See: http://code.google.com/p/chromium/issues/detail?id=378377
Prior to calling run_ldconfig the board rootfs must have ldconfig
installed. To arrange this move the call of run_ldconfig to after
the base package install.
Fixes build_image errors like these when building for arm64:
/sbin/ldconfig: /lib64/libXXX is for unknown machine 183.
Signed-off-by: Geoff Levand <geoff@infradead.org>
This reverts commit 39bb800f16.
This change disabled a number of features so it isn't suitable for the
generic VMware templates. We need to re-trace our steps to list exactly
what tools/systems weren't accepting the linux26 type.
The new python script check_root uses data that portage already
maintains on what shared libraries packages need or provide instead of
re-scanning whatever ELF files that can be found. This is much more
comprehensive but there is a bit of a transition issue for folks with
long-lived SDKs: packages built with portage older than 2.2.18 do not
include this data. As such for now the check is non-fatal and provides a
command you can use to refresh locally installed packages.
The code checking for conflicts between top level directories and /usr
has also been rewritten. Both tests now are considerably faster.
SDK tarballs have a .DIGESTS file but it is created by catalyst instead
of the upload_image function. In order to support plain GPG signing but
not avoid re-generating .DIGESTS we need to move that code out of
upload_image to a new function. upload_files shouldn't do it itself
because it is also used for portage binary packages which shouldn't be
signed (there is no point, nothing would verify the signatures).
The grub configuration needs some updates to handle dealing with booting
the kernel from the ESP rather than from inside the image. We also want to
be able to avoid dealing with signing the config file, so build it into the
binary. Finally, rather than having to cope with signing grub modules, build
the ones we need to boot into the grub image.
Once we're signing the root filesystem, we're not going to be able to boot
the kernel from there. Copy the kernel out to the EFI System Partition and
sign it.
Add qemu_uefi_secure target for building Secure Boot images. These are
identical to qemu_uefi images with the exception that the test keys have
been installed into the flash image, enabling Secure Boot by default. In
addition, sign the grub binary with the test keys during build when
producing unofficial images.
Adds CROSS_PROFILES, BOARD_CHOSTS, and BOARD_PROFILES definitions to support a
generic arm64-usr board.
get_portage_arch() is updated to convert aarch64 correctly.
Signed-off-by: Geoff Levand <geoff@infradead.org>
This is required for the eventual removal of `$PORTDIR` and
`$PORTDIR_OVERLAY` and ensures toolchain rebuilds/updates with
`./build_packages --nousepkg` don't erroniously try to use ebuilds from
`/usr/portage` inside of the SDK.
In order to fix up the build_toolchains script the crossdev overlay
needs to be setup properly, previously only setup_board did it.
Overall silences a lot of warnings and fixes an issue with crossdev:
/usr/bin/emerge-wrapper: line 48: /eclass/toolchain-funcs.eclass: No such file or directory
/usr/bin/emerge-wrapper: line 49: tc-arch: command not found
The portage CBUILD and HOSTCC variables need to be set to the SDK host to get
a proper cross build when building target binaries.
Change _configure_sysroot to use the CBUILD environment variable to set the
CBUILD and HOSTCC variables of ${ROOT}/etc/portage/make.conf. Also, fix up all
calls to _configure_sysroot to set the CBUILD environment variable.
Fixes setup_board failure when the host and target architectures differ.
Signed-off-by: Geoff Levand <geoff@infradead.org>
[marineam: fixed a copy/paste error]
Previously fsck output was suppressed to reduce the amount of noise in
build logs on the assumption that fsck really shouldn't have a reason to
fail. The filesystem is freshly created after all. However some users
have reported that fsck is failing but without error messages we don't
know why.
There isn't a sane way for users to know the licenses of individual
packages in CoreOS images in built images. The information is hidden
away back in the original ebuilds. This extends our existing package
list with a new file that also includes licenses:
```
app-admin/flannel-0.3.0-r3::coreos Apache-2.0
app-admin/fleet-0.9.1::coreos Apache-2.0
app-admin/locksmith-0.2.3::coreos Apache-2.0
app-admin/sdnotify-proxy-0.1.0::coreos Apache-2.0
app-admin/sudo-1.8.10_p2::portage-stable ISC BSD
app-admin/toolbox-0.0.0-r4::coreos Apache-2.0
app-arch/bzip2-1.0.6-r6::portage-stable BZIP2
app-arch/gzip-1.5::portage-stable GPL-3
app-arch/tar-1.27.1-r2::portage-stable GPL-3+
...
```
- "./build_image prod" already has the ability to specify which package will specify all the packages that should be pulled in and built into an image by specifying a package name using the --base_pkg command line flag. This creates an equivalent option for "./build_image dev" creating a --base_dev_pkg flag that passes a package name into the create_dev_img() function in dev_image_util.sh the same way that --base_pkg is passed into create_prod_image() inside prod_image_util.sh.
This change changes the default 'bytes-per-inode' ration from 16K to 4K,
the block size. To prevent this from wasting too much space change the
inode size from the default 256 to the minimum size, 128. Larger inodes
are used to store extended attributes more efficiently but since we do
not use SELinux the majority of files do not have security attributes.
These defaults may be modified via the new `bytes_per_inode` and
`inode_size` options.
Fix parsing the following output:
[ebuild N ] dev-libs/gmp-5.1.3-r1 to /usr/x86_64-cros-linux-gnu/
[ebuild UD] sys-libs/timezone-data-2013d [2014i-r1] to /usr/x86_64-cros-linux-gnu/
The previous regex did not account for upgrades and got confused by the
`[2014i-r1]` listing and goobbled up too much of the string. I am not
sure *why* portage is reporting an upgrade when --emptytree is also used
but there it is. Match all not-] characters instead.
Disable ccache as it is causing issues in other builds so disable it
everywhere to be safe. Disable the autoresume feature because our build
process doesn't actually make use of it.
Adding the update step appears to break permissions on the distfiles
directory. Ensure the portage user is correct and set the permissions on
directories it needs to write to in advance.
When bootstrapping a SDK we need to update GCC dependencies to ensure
the GCC built for stage1 is linked against the same library versions as
those that are included in the stage1. Without this updating the mpc
library just results in a broken stage1.
Probing all filesystem types on all block devices appears to hang
booting Amazon EC2 HVM instances. The console output is unreliably
buffered so there is no information on what the failure actually is. On
the up side we can work around it easily by only searching the GPT which
appears to be safe.
To aid testing things under Xen it helps to have a machine locally that
actually runs Xen! This isn't a particularly great setup but it works
well enough to simplify my own testing. Must be used with a developer
image and packages built with `USE=vm-testing` set to include the Xen
userspace tools.
This uses our new GRUB2 features to handle GPT priority partition
selection, terminal selection, OEM tweaks, etc. The old SYSLINUX and
PV-GRUB configs are now unused except for maintaining compatibility
with older installs. Of the old configs only the ones that
coreos-postinst copies are needed. The new setup supports using GRUB2
under Xen, giving us automatic fallback support on all of our platforms
for the very first time!
Since grub.cfg is copied into place instead of generated, build_image's
--boot_args option is no longer supported. It could be re-added later
with some sed goo but for now it is easy enough to just edit grub.cfg.
Mark the initial copy of CoreOS as 'successful' and with a non-zero
priority. Required to boot with a stricter interpretation of the
partition selection scheme which ignores partitions that have a priority
of zero. The new grub implementation follows this rule and is what the
original ChromeOS spec used too.
For the sake of completeness if multiple partitions are configured in
the json file with this feature they will be prioritized in disk-order.
The VHD format actually uses 2MB blocks internally so the 1MB alignment
used in e77e4e54 wasn't sufficent to prevent other tools from further
adjusting the image size to align it. Additionally a 1MB alignment may
be triggering a bug in OpenStack or XenServer disk resizing that renders
that partial block at the end of the old image size unmapped/unavailabe.
So far the default iteration order of python dicts has mostly matched
the order that we want the partitions on disk but this is not always the
case. I caught the BIOS-BOOT partition being ordered on disk after the
USR-A partition. Nothing bad came of this but consistancy is good.
The new disk size alignment left too much extra space at the end of the
disk which would lead to pointless resizing on first boot. Fill in the
extra space so that no more than 1MB is left unused.
The VHD disk format internally includes CHS addressing and qemu-img
respectfully aligns disk images to the common 16 heads 63 sectors
geometry when possible. This is unfortunate since images uploaded to
Azure must also be aligned to 1MB we normally do.
Since qemu-img doesn't have a way to handle this well right now adjust
our existing alignment logic to create disk images aligned to both.
I am unsure exactly what situation is causing the loopback partition
device node to not exist when it is being mounted but this should help
work around the situation and log loudly about it so we can hopefully
figure out where to dig further.
Version 4 is too low. Some VMware products even crash trying to
upgrade it to a greater version (VMware Fusion 6 Pro). Having at
least 7 will allow us to use some modern features in most VMware
products, such as enabling vmxnet3 virtual network adapters or adding
much more memory and cpu cores to virtual machines.
Pruning files via INSTALL_MASK in the profile is a bit more apropriate
since it allows us to keep most of that info in one place. The only
parts that need to be deleted or adjusted here are inputs and outputs of
`env-update` which has to be run after everything is installed.
Previously we didn't actually clean up `env.d` at all which lead at
least one user to think they should edit those files and run
`env-update` themselves but we don't ship that tool on prod images.
This sets the IMG_FORCE_OEM_PACKAGE variable to the supplied string. If a
':' is present, what follows it gets put in the IMG_FORCE_OEM_USE variable
and what precedes in the former.
_get_vm_opt() has been modified to generally support forced overrides such
as this one, simply set variables named IMG_FORCE_$opt.
Now you can do things like:
for fmt in cloudstack \
digitalocean \
ec2-compat:ec2 \
ec2-compat:openstack \
ec2-compat:brightbox \
exoscale \
gce \
hyperv \
rackspace \
rackspace-onmetal; do
./image_to_vm.sh --format=qemu --oem_pkg=$fmt
../build/images/amd64-usr/latest/coreos_developer_qemu.sh -curses
done
rather than having to modify build_library/vm_image_util.sh to test oem
builds in qemu.
