This change removes Flatcar specific builds of docker[-cli], containerd,
runc, and cri-tools and instead switches to upstream Gentoo ebuilds
added to portage-stable.
The change updates docker to 24.0.6.
NOTE that there currently is no upstream ebuild for containerd-1.7.7, so
this change adds that ebuild based on the upstream containerd-1.7.6
ebuild.
Flatcar customisations like systemd units etc. are now applied in the
manglefs script of the respective sysexts, based on file system trees in
coreos-overlay/coreos/sysext/(containerd|docker).
The build_sysext script has been extended by an option to strip all
binaries in a sysext; the option is active by default. This takes care
of removing debug symbols from docker and containerd - which are not
removed by the default Gentoo build. The overall size of both containerd
and docker sysext is reduced by ~50%.
Lastly, the sysext command line syntax of build_image has been extended
to allow specifying multiple packages for a sysext. This was necessary
because docker-cli and docker do not have any runtime relationships and
therefore must both be specified for installation to correctly mirror
Flatcar's own docker packaging.
Signed-off-by: Thilo Fromm <thilofromm@microsoft.com>
- updated github actions for runc, containerd, and docker to not handle
nonexistent ebuilds in app-torcx/ anymore
- removed spurious package_run_dependencies from build_image_util.sh
- build_sysext: generate pkginfo before mangle script runs
use zstd for compression; add cli flag to select compression
- ci_automation_common.sh: remove spurious `/` from match string
- coreos, board-packages, bootengine: bump ebuild revisions
- kernel commonconfig: add squashfs zstd support
Signed-off-by: Thilo Fromm <thilofromm@microsoft.com>
This change refactors base OS sysext builds to use a separate build
script `build_library/sysext_prod_builder`, which is called from
`build_library/prod_image_util.sh` when `build_image` runs.
This allows for better separation of cleanup traps: prod image sysext
builds need its own trap / cleanup function for temporary build
directories and loopback mounts.
Prod sysext builds properly generate lincense and SBOM information, and
provide detailed file listings and disk space usage stats.
- SBOM / licenses JSON now include all packages of the
final image, i.e. a combined list of base image and all base OS
sysexts.
- Packages lists, files list and detailed files list include the sysext
squashfs files for the base image, and separate sections with files /
packages lists for each sysext.
- Disk usage contains both final disk image usage as well as usage of
each individual sysext squashfs.
This change refactors sysext builds during build_image and generalises
the code (no hard-coded containerd and docker anymore).
A command line option is added to build_image for sysexts to include in
the OS image. It defaults to containerd and docker but may be set to
arbitrary packages. The command line supports simple depenencies, i.e.
the "docker" sysext will re-use package information from the
"containerd" sysext and not include another containerd.
Signed-off-by: Thilo Fromm <thilofromm@microsoft.com>
This change removes torcx libraries, references, and commandline options
from build automation scripts and from build_library/.
Containerd and docker are shipped via sysexts which are included in the
base image.
Signed-off-by: Thilo Fromm <thilofromm@microsoft.com>
Apparently successful `[[ -b "${LOOP_DEV}p1" ]]` check is not enough -
the mount can still fail. So instead of doing those checks, try
mounting and reprobing in the loop with some small exponential
backoffs.
- remove explicit "-multilib" from prefix keywordsas it is set in
profile
- split heredoc for generating emerge wrapper so we don't need to
escape
- add sys-apps/bubblewrap and virtual/tmpfiles to package update
automation list
- use prefix build fix for libgpg-error from upstream
Signed-off-by: Thilo Fromm <thilofromm@microsoft.com>
This change adds experimental prefix builds to the Flatcar SDK.
Prefix builds use a custom sys prefix path and emerge all binaries and
runtime dependencies into that prefix.
This path can then e.g. be shipped as a portable sysext since it
includes all dependencies, and has libraries at a custom path so these
do not conflict with libraries on target systems.
Prefix uses a staging environment (path) featuring a full-blown
development environment, and a "final" environment for installing.
Staging and final need to be created using setup_prefix first,
which will also create an emerge wrapper to emerge ebuilds into staging
and subsequently final. The root fs in final may then e.g. be used to
create a distro independent, portable sysext.
Co-authored-by: James Le Cuirot <chewi@gentoo.org>
Co-authored-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>
Co-authored-by: Thilo Fromm <thilofromm@microsoft.com>
The compression feature of btrfs allows us to store more in the
size-limited /usr and OEM partitions. The size should of course still
be monitored to not bloat the image but more headroom helps to try
things out quickly without hitting the hard limit which fails the
build.
Use btrfs with zstd compression for the /usr partition. While for ext2
a hack exists to force read-only mounts by manipulating some bytes of
the filesystem, on btrfs we can use the subvolume read-only flag
instead which also works for the default top level subvolume. However,
it also makes also sense to mount the filesystem with the "norecovery"
mount option to prevent any write attempts even when the "ro" option is
set (not needed when using dm-verity in read-only mode but when
directly mounting without dm-verity). A new subvolumes is not created
because subvolumes don't offer anything special as long as we use the
A/B partition update mechanism (but they could be an alternative for
that). Note that switching to the btrfs on the /usr partition is only
possible when the Flatcar Stable release has all patches in
update-engine and seismograph's rootdev.
The `localedef` tool expects `/usr/lib/locale` directory to
exist. This directory used to be created by the `sys-libs/glibc`
package (with the `keepdir` directive), but after the update of the
package, the locale generation stuff (and the `keepdir` directive )was
moved to the `sys-libs/locale-gen` package. This package is not
installed in the production images, so the `/usr/lib/locale` directory
was not created. In such a situation, calling localedef to generate
C.UTF-8 locale resulted in an error like:
cannot create temporary file: ${SOME_ROOTFS}/usr/lib/locale/locale-archive.ufpG15: No such file or directory
Create the directory before calling localedef to fix the problem.
The OEM sysext image file in the OEM partition had the version variable
name being part of the filename instead of the substituted version value
because of wrong quoting when the fixed string got replaced by ${…}.
The vendor tools on the OEM partition weren't updated. We now want to
ship them as systemd-sysext images which we can easily update. This
change extends the Flatcar A/B update mechanism to cover the OEM
systemd-sysext images. The same mechanism is also able to support
"official" Flatcar extensions, e.g., a ZFS extension.
When running the build_packages script, one can encounter an error such as
'Error fetching binhost package info from.' This pertains to SDK packages (not
board packages). Since we have transitioned to the SDK container, the SDK
packages are no longer published independently from the container image.
This change improves build_sysext by sourcing a missing lib dependency,
adding a number of comfort / quality-of-life options, and updating the
output of '--help' accordingly.
The OEM sysext finction in build_library/vm_image_util.sh is also
updated to use new command line format.
1. Include missing dependency toolchain_util.sh to fix an error in
board_options.sh (get_board_arch undefined).
2. Use positional parameters for mandatory arguments.
build_dir and sysext_name are mandatory and are now positional
arguments instead of options.
binary_package is the third positional argument but can be omitted
if --metapkgs was specified.
3. --squashfs_base is now guessed better and will use the most recent
build by default.
4. A new boolean flag --ignore_version_mismatch for the more daring
developer was added. The flag will cause the script to continue if a
version mismatch between SDK board packages and squashfs base is
detected.
5. Error messages were improved for when mandatory parameters were not
provided.
6. The '--help' message was improved and adjusted to the new parameters.
Signed-off-by: Thilo Fromm <thilofromm@microsoft.com>
