Apparently successful `[[ -b "${LOOP_DEV}p1" ]]` check is not enough -
the mount can still fail. So instead of doing those checks, try
mounting and reprobing in the loop with some small exponential
backoffs.
- remove explicit "-multilib" from prefix keywordsas it is set in
profile
- split heredoc for generating emerge wrapper so we don't need to
escape
- add sys-apps/bubblewrap and virtual/tmpfiles to package update
automation list
- use prefix build fix for libgpg-error from upstream
Signed-off-by: Thilo Fromm <thilofromm@microsoft.com>
This change adds experimental prefix builds to the Flatcar SDK.
Prefix builds use a custom sys prefix path and emerge all binaries and
runtime dependencies into that prefix.
This path can then e.g. be shipped as a portable sysext since it
includes all dependencies, and has libraries at a custom path so these
do not conflict with libraries on target systems.
Prefix uses a staging environment (path) featuring a full-blown
development environment, and a "final" environment for installing.
Staging and final need to be created using setup_prefix first,
which will also create an emerge wrapper to emerge ebuilds into staging
and subsequently final. The root fs in final may then e.g. be used to
create a distro independent, portable sysext.
Co-authored-by: James Le Cuirot <chewi@gentoo.org>
Co-authored-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>
Co-authored-by: Thilo Fromm <thilofromm@microsoft.com>
The `localedef` tool expects `/usr/lib/locale` directory to
exist. This directory used to be created by the `sys-libs/glibc`
package (with the `keepdir` directive), but after the update of the
package, the locale generation stuff (and the `keepdir` directive )was
moved to the `sys-libs/locale-gen` package. This package is not
installed in the production images, so the `/usr/lib/locale` directory
was not created. In such a situation, calling localedef to generate
C.UTF-8 locale resulted in an error like:
cannot create temporary file: ${SOME_ROOTFS}/usr/lib/locale/locale-archive.ufpG15: No such file or directory
Create the directory before calling localedef to fix the problem.
The OEM sysext image file in the OEM partition had the version variable
name being part of the filename instead of the substituted version value
because of wrong quoting when the fixed string got replaced by ${…}.
The vendor tools on the OEM partition weren't updated. We now want to
ship them as systemd-sysext images which we can easily update. This
change extends the Flatcar A/B update mechanism to cover the OEM
systemd-sysext images. The same mechanism is also able to support
"official" Flatcar extensions, e.g., a ZFS extension.
When running the build_packages script, one can encounter an error such as
'Error fetching binhost package info from.' This pertains to SDK packages (not
board packages). Since we have transitioned to the SDK container, the SDK
packages are no longer published independently from the container image.
This change improves build_sysext by sourcing a missing lib dependency,
adding a number of comfort / quality-of-life options, and updating the
output of '--help' accordingly.
The OEM sysext finction in build_library/vm_image_util.sh is also
updated to use new command line format.
1. Include missing dependency toolchain_util.sh to fix an error in
board_options.sh (get_board_arch undefined).
2. Use positional parameters for mandatory arguments.
build_dir and sysext_name are mandatory and are now positional
arguments instead of options.
binary_package is the third positional argument but can be omitted
if --metapkgs was specified.
3. --squashfs_base is now guessed better and will use the most recent
build by default.
4. A new boolean flag --ignore_version_mismatch for the more daring
developer was added. The flag will cause the script to continue if a
version mismatch between SDK board packages and squashfs base is
detected.
5. Error messages were improved for when mandatory parameters were not
provided.
6. The '--help' message was improved and adjusted to the new parameters.
Signed-off-by: Thilo Fromm <thilofromm@microsoft.com>
Included a script to enable generating systemd-sysexts. Successfully
tested sysext generation with a fresh Flatcar image (e.g., Python and
Neofetch system extension). Part of my internship work.
The current OS images we provide are not OK as base for flatcar specific
sysext images: it lacks the package metadata and portage configuration,
in order to keep end user OS image clean. This script retains this
information and allows you to produce systemd-sysexts to extend the
system. This script can be used to build a Flatcar sysext image.
Recommended to run from image build folder.
Signed-off-by: Krish Jain <kjain7@u.rochester.edu>
After changes to the inode size, the sysext installation runs out of
space because the installation happens on a mounted production image.
This is problematic because the /usr partition is only 1024MB in size
and gets full. Mount a temporary overlay so that we can use that for
installation, and discard it afterwards.
This also means we no longer need to disable verity and in fact could
live without copying the prod image. I won't make that change since
we're working on a new script to automate building of sysexts using the
overlay approach.
Signed-off-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>
Inode sizes smaller than 256:
- don't support extended metadata (nanosecond timestamp resolution)
- cannot handle dates beyond 2038
- are deprecated
Change the default from 128 to 256. There is no way to apply this change on a
mounted filesystem so this change will only apply to new deployments.
Fixes: flatcar/flatcar#1082
Signed-off-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>
For release tests and updating a machine to a dev build we already have
the dev-key-signed generic update payload but not yet the OEM sysext
update payload.
Generate the dev-key-signed OEM sysext update payload during build and
upload it.
The initial MVP of the OEM sysext usage we release won't have updates
for the sysext image and, therefore, it is not bound to the OS version.
The special name suffix instead of the version hints bootengine at using
it if no matching version is found. The name will also be used at hint
for update-engine to clean it up when versioned sysext images arrive.
We don't have an update process of the OEM sysexts implemented yet, so
use a fake "initial" version for them and make them independent from
OS version.
It isn't doing much as nothing QEMU-specific was being installed into
the OEM partition.
With that done, we opt into building an OEM sysext image for QEMU
platform.
This package will be used for the sysext image, instead of for
installing files into /usr/share/oem. This means that we can drop some
files or move them elsewhere. The systemd service file is not needed,
because it is installed by the app-emulation/wa-linux-agent package
now. This also means that the ignition file as lost its purpose. The
grub.cfg and oem-release must be installed in /usr/share/oem, next to
the sysext raw image file, so handling of these files is moved to the
newly added coreos-base/common-oem-files package. `eject` symlink to
`/usr/bin/true` is installed in the newly added manglefs.sh script.
With this done, we also opt into building an OEM sysext image for
Azure platform.
Will come in handy when generating OEM sysexts. We can mount the
generic image, put the image database back into the image and emerge
extra packages without the need to drop all DEPENDS and BDEPENDS from
the ebuilds.
- Fix the snapshot name, it is not "portage-${VERSION}", but rather
"gentoo-${VERSION}".
- After building the snapshot, remove all the similar files from the
snapshots directory - Catalyst gets easily confused by them and
bails out.
- Extend the `build_snapshot` function to optionally accept the config
path and the snapshot name, so SDK's stage1 code can use this
function instead of duplicating parts of it.
The temporary /etc backup created during emerging packages should only
contain empty files that will make sure that the symlinks pointing to
files within the /etc backup won't dangle at any time.
We used to create a base_image_var.conf tmpfiles config file that
contained information about directories under /var that weren't
covered by any other tmpfiles config file. Recently some package
update started installing a directory under /var that belonged to a
user/group not found directly in passwd/group file in /etc. This
user/group was defined in passwd/group in /usr/share/baselayout, but
at the early boot, these are not yet checked for user/group
information, so systemd-tmpfiles running inside initrd failed when
trying to create such an entry using the base_image_var.conf tmpfiles
config file.
Split the base_image_var.conf into two files - base_image_var.conf and
base_image_var_late.conf. The former will only contain entries owned
by user/group that are supposed to exist very early in the boot, while
the latter will contain the rest of directories - those will be
created later during the boot.
This will generate tmpfiles config only for directories that are owned
by an allowed user and group if such are passed. Not passing any
allowed users or groups, allows any user or group.
The `list` command of `equery` will exit with status 3 if a package is
not found and `--quiet` is in effect. This results in some non-fatal
noise during the SDK build:
INFO update_chroot: Maybe removing some hard blocks
ERROR update_chroot: script called: update_chroot '--toolchain_boards=arm64-usr' '--usepkg' '--nousepkgonly' '--getbinpkg'
ERROR update_chroot: Backtrace: (most recent call is last)
ERROR update_chroot: file update_chroot, line 250, called: remove_hard_blocks 'sudo_e_emerge' 'equery' 'dev-python/setuptools_scm:2'
ERROR update_chroot: file update_chroot_util.sh, line 49, called: get_versions_from_equery 'equery' 'dev-python/setuptools_scm'
ERROR update_chroot: file update_chroot_util.sh, line 9, called: die_err_trap '"${equery_cmd}" --quiet --no-color list --format='${version} ${fullversion}' "${pkg}"' '3'
ERROR update_chroot:
ERROR update_chroot: Command failed:
ERROR update_chroot: Command '"${equery_cmd}" --quiet --no-color list --format='${version} ${fullversion}' "${pkg}"' exited with nonzero code: 3
INFO update_chroot: No hard blockers to remove
Shut the noise up. If package is not found, then there is simply
nothing to do.
We already drop tmpfile rules that we don't need because we ship the
files through our /etc overlay. However, some rules weren't dropped
because they used tabs and not spaces (/etc/selinux/, /etc/iscsi and
/etc/ssl/*).
Drop rule lines for /etc that use tabs. Also rules modifiers like ! to
only do it during boot or - to allow failure will be removed but those
with + or = will stay as they to explicit recreation.
The existing tmpfile logic took care of folders that the ebuild keepdir
directive wanted to exist on the OS. However, files and symlinks were
not created, causing them to be missing if we didn't explicitly modify
the ebuild files in coreos-overlay to use tmpfiles or patching of
paths to be in /usr. We need a logic to provide /etc files from the
current /usr partition without getting stale. This can be done best
with an overlay mount which requires to keep the original /etc files
under /usr.
Move the final /etc folder of the image build to /usr/share/flatcar/etc
to serve as lower layer in the overlay. Also remove any state from the
rootfs to make sure that we don't rely on it when testing our images
before the release. What we get with an overlay mount is essentially a
similar behavior to a 3-way merge because as long as the user didn't
change the files, the old version is replaced with the new version and
as soon as the user did changes, that file is frozen and wins over the
provided old (in case of a rollback) or new versions from /usr. It does
not work on file lines but on whole file contents, yet that is also
what rpm-ostree does to my knowledge. Also, run tmpfiles once and do
the SELinux labeling to prevent files being created in the upperdir
because they were missing in the lowerdir, or because they had missing
SELinux labels.
