Recently we had some problems with read-only filesystems, that pop up
in different places. It may be easier to catch if the debugging info
is printed in an error case instead of just one specific location.
Use FLATCAR_DEV_BUILDS only for setting up binpkg URLs, ceding the SDK
tarball URL role to the new FLATCAR_SDK_SERVERS variable. That way we
can still set up binpkg URLs the way we used to do so far and set up
SDK tarball URLs differently.
For two-phase SDK build, we would like to use intermediate SDK as a
seed. This SDK is only available on bincache, but previously only
nightly builds could use bincache as the source of SDK tarballs. Now,
with the URL split, we can set up the builds to use both bincache and
the release server, where release builds will prioritize release
server over bincache, and developer builds - bincache over release
server.
We need to fix another symlink created by gcc-config, so extend the
function that was doing it for some other links and rename it - it's
not about only liblto any more.
This change adds a job for publishing binary packages to the build cache
server to the ci automation.
Also, setup_board is updated to use the buildcache package cache if a
nightly build version is detected.
Signed-off-by: flatcar-ci <infra+ci@flatcar-linux.org>
This updates the default settings in build scripts to use
https://mirror.release.flatcar-linux.net/
instead of the google storage bucket if no binhost or FLATCAR_DEV_BUILDS
is specified.
Defaults are updated for
* update_chroot (runs at SDK initialisation time)
* setup_board (creates /boards/[ARCH]/) chroots
* the development container
* set_version
The subshell was printing the error backtrace, but apparently it was
of no consequences. Just assume that listing may fail, so we don't get
any confusing backtraces.
The environment variables FLATCAR_DEV_BUILDS and FLATCAR_DEV_BUILDS_SDK
define where the base URL for the binary package store of the board
packages and the SDK packages. To set it, they were either exported in
the chroot or passed each time as parameter but this was only available
for the SDK packages in the tricky order
"update_chroot --dev_builds_sdk URL" (or --binhost) and then
"./build_packages --skip_chroot_upgrade".
The defining information for binary package URL comes from version.txt
but it lacks the base URL for a convenient use. Add the base URLs for
SDK and board packages there to be able to get binary packages without
manual tricks and errors. Any changes to
~/trunk/.repo/manifests/version.txt will directly be picked up
from ./build_packages and friends to set the correct URLs in
/etc/portage/make.conf and /build/BOARD/etc/portage/make.conf
so that even a manual "emerge(-BOARD) --usepkg --getbinpkg" uses them.
A helper script "./set_version" is provided to ease modifying
~/trunk/.repo/manifests/version.txt by resolving the latest nightly
version.
Later this functionality can also be used to simplify the Jenkins code
which currently sets these variables (but the special case of the
Release Base download being different from the upload holds).
Before, we were relying on the toolchains job to build and upload
packages that were part of the SDK. With this change, all packages that
should be part of the SDK are built and uploaded by the SDK job. The
toolchains job only builds toolchain packages specific for the release.
This change includes several adjustments done to both the SDK and the
toolchains jobs to make this work:
* Make the SDK job build all cross toolchains, including Rust
* Stop building Rust in the toolchains job and use the one in the SDK
instead.
* In toolchain_util.sh: detect when the symlink folder for crossdev
packages is missing and run crossdev to create it during
update_chroot setup.
* Make it possible to build the SDK starting from stage 4 instead of
stage 1, to make the SDK building faster for PR branches / nightlies
(full build should still be done for releases / weeklies).
We only support amd64-usr at this point, so this removes a required
step when setting up a new SDK. If the default board is specified
normally or through the environment, it will override this value.
5a76e4e5e9 started exporting COREOS_BUILD_ID
whenever it was found in version.txt, even if its value was blank. Because
COREOS_BUILD_ID is in ENVIRONMENT_WHITELIST, this caused generated build IDs
to be propagated into the SDK chroot environment and reused for every build
in a "cork enter" session. Stop exporting COREOS_BUILD_ID when we set it
ourselves.
See also 8e754f9c2b.
Change the setting of COREOS_BUILD_ID so that its value, in order of
preference, is set to
A value set in the environment.
A value provided in manifest's version.txt.
A fall back value of the current time-date.
Signed-off-by: Geoff Levand <geoff@infradead.org>
The one-liner `[[ -z ${PIPESTATUS[*]#0} ]]` no longer works because the
expansion still includes spaces even if all the values are zero. Somehow
that didn't matter in bash 4.2 but it does mater in 4.3 to be consistent
with the general behavior of variables in [[ tests.
The generation of version.txt was the only thing depending on sourcing
the deprecated BUILD, BRANCH, and PATCH values from version.txt which
common.sh no longer does since 0b6acf86. Derive them instead.
When running under jenkins the $GNUPGHOME may be located under the
current build directory instead of $HOME to avoid conflicting with other
jobs on the same build host.
By copying and removing the relevant qemu static executable the
functions enable and disable the chroot environment for arm64 rootfs.
Signed-off-by: Andrej Rosano <andrej@inversepath.com>
This code is not applicable to us, it predates CoreOS and is a weird
thing for common.sh to be doing as well. Instead always define
CHROOT_TRUNK_DIR to /mnt/host/source, create ~/trunk in make_chroot.
Currently building images on older kernels will fail because mkfs.btrfs
enables an incompatible feature 'extref' by default. We never really
made this requirement explicit and the SDK in general has continued to
maintain compatibility with older kernels. Make the requirement explicit
so users will get errors quicker and there is a clear line for what
kernel features can be used in the SDK.
Using parallel_emerge has been disabled by default for all commands
except build_image for quite a while now, build_image kept it just
because it was still a bit faster than normal emerge. Keeping
parallel_emerge complicates future changes to build_image so it needs to
drop it entirely. Since that means nothing uses it by default we might
as well just rip out support for it entirely.
- Automated builds drop SDK and binary packages into
gs://builds.developer.core-os.net/ and the new download URL is
http://builds.developer.core-os.net/ (COREOS_DEV_BUILDS)
- Change default upload path to gs://users.developer.core-os.net/ for
misc developer builds. Official builds go elsewhere and will just be
configured in buildbot/jenkins so some COREOS_OFFICIAL stuff is gone.
- Automated builds of images go to a private bucket,
gs://builds.release.core-os.net which later gets copied to
gs://alpha.release.core-os.net and friends by core_promote.
This image type is the same as the developer image except that it is a
single root filesystem and is bootable via systemd-nspawn. This may
become obsolete eventually when it becomes possible to boot the normal
disk images under nspawn but it is useful for testing until then.
The partition type is defined by the Discoverable Partitions Spec.
http://www.freedesktop.org/wiki/Specifications/DiscoverablePartitionsSpec/
We need some more control over exactly what lands in dev vs prod images
which will require letting them diverge in what is currently the common
base image step. There isn't any real need for the base image in the
first place other than to speed up building both dev and prod images at
the same time but that isn't common enough to worry about.
As part of this cleanup also remove references to CHROMEOS_* variables
and the recovery image that never actually existed in CoreOS.
The existing version.txt is kinda annoying. The common case of referring
to the current version requires joining three values and the names of
those values only make sense in ChromeOS. Instead just use version as a
string, using VERSION, VERSION_ID, and BUILD_ID just as they appear in
os-release. It is up to the few scripts that need the individual parts
to break the version apart.
The old values remain for the sake of compatibility.
I would like to phase out parallel_emerge so disable it for all commands
other than build_image which is the only one that shows a noticeable
benefit from it (~2 min with --fast, ~3 min with --nofast).
Make it possible for other scripts to share the same value for our
release repository and equally easy to override with a custom value.
Also allow setting the root from the command line in addition to the
environment. Usually --upload_root is better to use than --upload_path.
For multi-file uploads we should explicitly declare what the name of the
.DIGESTS file should be instead of using the first file name. Relying on
the ordering was subtle and easy to break.
