When running the build_packages script, one can encounter an error such as
'Error fetching binhost package info from.' This pertains to SDK packages (not
board packages). Since we have transitioned to the SDK container, the SDK
packages are no longer published independently from the container image.
This change improves build_sysext by sourcing a missing lib dependency,
adding a number of comfort / quality-of-life options, and updating the
output of '--help' accordingly.
The OEM sysext finction in build_library/vm_image_util.sh is also
updated to use new command line format.
1. Include missing dependency toolchain_util.sh to fix an error in
board_options.sh (get_board_arch undefined).
2. Use positional parameters for mandatory arguments.
build_dir and sysext_name are mandatory and are now positional
arguments instead of options.
binary_package is the third positional argument but can be omitted
if --metapkgs was specified.
3. --squashfs_base is now guessed better and will use the most recent
build by default.
4. A new boolean flag --ignore_version_mismatch for the more daring
developer was added. The flag will cause the script to continue if a
version mismatch between SDK board packages and squashfs base is
detected.
5. Error messages were improved for when mandatory parameters were not
provided.
6. The '--help' message was improved and adjusted to the new parameters.
Signed-off-by: Thilo Fromm <thilofromm@microsoft.com>
Included a script to enable generating systemd-sysexts. Successfully
tested sysext generation with a fresh Flatcar image (e.g., Python and
Neofetch system extension). Part of my internship work.
The current OS images we provide are not OK as base for flatcar specific
sysext images: it lacks the package metadata and portage configuration,
in order to keep end user OS image clean. This script retains this
information and allows you to produce systemd-sysexts to extend the
system. This script can be used to build a Flatcar sysext image.
Recommended to run from image build folder.
Signed-off-by: Krish Jain <kjain7@u.rochester.edu>
After changes to the inode size, the sysext installation runs out of
space because the installation happens on a mounted production image.
This is problematic because the /usr partition is only 1024MB in size
and gets full. Mount a temporary overlay so that we can use that for
installation, and discard it afterwards.
This also means we no longer need to disable verity and in fact could
live without copying the prod image. I won't make that change since
we're working on a new script to automate building of sysexts using the
overlay approach.
Signed-off-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>
Inode sizes smaller than 256:
- don't support extended metadata (nanosecond timestamp resolution)
- cannot handle dates beyond 2038
- are deprecated
Change the default from 128 to 256. There is no way to apply this change on a
mounted filesystem so this change will only apply to new deployments.
Fixes: flatcar/flatcar#1082
Signed-off-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>
For release tests and updating a machine to a dev build we already have
the dev-key-signed generic update payload but not yet the OEM sysext
update payload.
Generate the dev-key-signed OEM sysext update payload during build and
upload it.
The initial MVP of the OEM sysext usage we release won't have updates
for the sysext image and, therefore, it is not bound to the OS version.
The special name suffix instead of the version hints bootengine at using
it if no matching version is found. The name will also be used at hint
for update-engine to clean it up when versioned sysext images arrive.
We don't have an update process of the OEM sysexts implemented yet, so
use a fake "initial" version for them and make them independent from
OS version.
It isn't doing much as nothing QEMU-specific was being installed into
the OEM partition.
With that done, we opt into building an OEM sysext image for QEMU
platform.
This package will be used for the sysext image, instead of for
installing files into /usr/share/oem. This means that we can drop some
files or move them elsewhere. The systemd service file is not needed,
because it is installed by the app-emulation/wa-linux-agent package
now. This also means that the ignition file as lost its purpose. The
grub.cfg and oem-release must be installed in /usr/share/oem, next to
the sysext raw image file, so handling of these files is moved to the
newly added coreos-base/common-oem-files package. `eject` symlink to
`/usr/bin/true` is installed in the newly added manglefs.sh script.
With this done, we also opt into building an OEM sysext image for
Azure platform.
Will come in handy when generating OEM sysexts. We can mount the
generic image, put the image database back into the image and emerge
extra packages without the need to drop all DEPENDS and BDEPENDS from
the ebuilds.
- Fix the snapshot name, it is not "portage-${VERSION}", but rather
"gentoo-${VERSION}".
- After building the snapshot, remove all the similar files from the
snapshots directory - Catalyst gets easily confused by them and
bails out.
- Extend the `build_snapshot` function to optionally accept the config
path and the snapshot name, so SDK's stage1 code can use this
function instead of duplicating parts of it.
The temporary /etc backup created during emerging packages should only
contain empty files that will make sure that the symlinks pointing to
files within the /etc backup won't dangle at any time.
We used to create a base_image_var.conf tmpfiles config file that
contained information about directories under /var that weren't
covered by any other tmpfiles config file. Recently some package
update started installing a directory under /var that belonged to a
user/group not found directly in passwd/group file in /etc. This
user/group was defined in passwd/group in /usr/share/baselayout, but
at the early boot, these are not yet checked for user/group
information, so systemd-tmpfiles running inside initrd failed when
trying to create such an entry using the base_image_var.conf tmpfiles
config file.
Split the base_image_var.conf into two files - base_image_var.conf and
base_image_var_late.conf. The former will only contain entries owned
by user/group that are supposed to exist very early in the boot, while
the latter will contain the rest of directories - those will be
created later during the boot.
This will generate tmpfiles config only for directories that are owned
by an allowed user and group if such are passed. Not passing any
allowed users or groups, allows any user or group.
The `list` command of `equery` will exit with status 3 if a package is
not found and `--quiet` is in effect. This results in some non-fatal
noise during the SDK build:
INFO update_chroot: Maybe removing some hard blocks
ERROR update_chroot: script called: update_chroot '--toolchain_boards=arm64-usr' '--usepkg' '--nousepkgonly' '--getbinpkg'
ERROR update_chroot: Backtrace: (most recent call is last)
ERROR update_chroot: file update_chroot, line 250, called: remove_hard_blocks 'sudo_e_emerge' 'equery' 'dev-python/setuptools_scm:2'
ERROR update_chroot: file update_chroot_util.sh, line 49, called: get_versions_from_equery 'equery' 'dev-python/setuptools_scm'
ERROR update_chroot: file update_chroot_util.sh, line 9, called: die_err_trap '"${equery_cmd}" --quiet --no-color list --format='${version} ${fullversion}' "${pkg}"' '3'
ERROR update_chroot:
ERROR update_chroot: Command failed:
ERROR update_chroot: Command '"${equery_cmd}" --quiet --no-color list --format='${version} ${fullversion}' "${pkg}"' exited with nonzero code: 3
INFO update_chroot: No hard blockers to remove
Shut the noise up. If package is not found, then there is simply
nothing to do.
We already drop tmpfile rules that we don't need because we ship the
files through our /etc overlay. However, some rules weren't dropped
because they used tabs and not spaces (/etc/selinux/, /etc/iscsi and
/etc/ssl/*).
Drop rule lines for /etc that use tabs. Also rules modifiers like ! to
only do it during boot or - to allow failure will be removed but those
with + or = will stay as they to explicit recreation.
The existing tmpfile logic took care of folders that the ebuild keepdir
directive wanted to exist on the OS. However, files and symlinks were
not created, causing them to be missing if we didn't explicitly modify
the ebuild files in coreos-overlay to use tmpfiles or patching of
paths to be in /usr. We need a logic to provide /etc files from the
current /usr partition without getting stale. This can be done best
with an overlay mount which requires to keep the original /etc files
under /usr.
Move the final /etc folder of the image build to /usr/share/flatcar/etc
to serve as lower layer in the overlay. Also remove any state from the
rootfs to make sure that we don't rely on it when testing our images
before the release. What we get with an overlay mount is essentially a
similar behavior to a 3-way merge because as long as the user didn't
change the files, the old version is replaced with the new version and
as soon as the user did changes, that file is frozen and wins over the
provided old (in case of a rollback) or new versions from /usr. It does
not work on file lines but on whole file contents, yet that is also
what rpm-ostree does to my knowledge. Also, run tmpfiles once and do
the SELinux labeling to prevent files being created in the upperdir
because they were missing in the lowerdir, or because they had missing
SELinux labels.
With PORTDIR and PORTDIR_OVERLAY environment variables being gone as
overrides, setting up a profile for the developer container broke. The
overrides were a hack already, as eselect does not seem to have
support for setting a profile based on repos.conf with repo locations
that are valid only after chrooting into the root directory. So
instead of invoking eselect, we set up the symlink ourselves.
That way we can see a report of what emerge is going to do and the
status of the use flags for the installed packages. The downside is
that we are going to have reports about using deprecated and
unsupported profile in even more places.
Emerge flags are cryptic in general, but short flags even more so, so
expand them. While at it, I noticed some places where bash arrays
could be used, so convert those places too.
When adding a mask or accept keywords entry for some version of a
toolchain package (gcc, libc, gdb, binutils or kernel headers), it
can't be done by just doing it, for example, for sys-devel/gcc. Both
cross-{x86_64,aarch64}-cros-linux-gnu/gcc needs to be
masked/keyworded, otherwise crossdev will pick up the latest stable
version for cross-{x86_64,aarch64}-cros-linux-gnu/gcc and this choice
is not affected by masks or accept keywords of sys-devel/gcc.
This situation does not happen all that often, but when it happens,
it's usually hard to remember to handle also the cross toolchain
packages. Forgetting to do so leads to weird issues. So instead of
telling crossdev to use the latest stable versions of cross toolchain
packages, we will tell it to use specific versions that match the
version of plain packages.
Timestamp and user/group information are out, in are device ID and
inode number. That way, the file can be used for accounting size
differences of files/image.
This was a place I missed where /etc/portage is set up. Because of it,
user patches for sys-devel/gcc were not picked up.
Also stop using deprecated PORTDIR and PORTDIR_OVERLAY getters. We
still set those variables, but we will drop them eventually.
Normally `ln -sf path/to/target at/name` will create a symlink at
`at/name` that points to `path/to/target`. But if `at/name` already
exists and is a directory or a symlink to some other directory, then
this command will create a symlink at `at/name/target` pointing to
`path/to/target`. There is an ambiguity between 1st and 3rd form of
`ln` (please refer to `man ln` for the available invocation forms). It
can be disambiguated by using the `-T` flag to force the 1st form.
In our case, if `/etc/portage/patches` symlink already existed and was
pointing to `<coreos-overlay>/coreos/user-patches`, we ended up with a
useless symlink at `<coreos-overlay>/coreos/user-patches/user-patches`
pointing to `<coreos-overlay>/coreos/user-patches`.
When setting up portage configuration, we set up a symlink in
${ROOT}/etc/portage/patches that points to the coreos/user-patches
directory inside the coreos-overlay. That way, we can add our custom
patches to coreos-overlay without the need for moving the packages
from portage-stable to coreos-overlay only to apply an extra patch.
Flatcar has net-fs/samba 4.15.4-r3, greater than 4.13.8, so it is
not necessary to keep GLSA 202105-22 in GLSA_ALLOWLIST.
Allow 202209-12 for now, as update to grub 2.06 is still in progress.
We need to fix another symlink created by gcc-config, so extend the
function that was doing it for some other links and rename it - it's
not about only liblto any more.
Some packages are currently missing from the /usr/share/SLSA directory
compared to flatcar_production_image_packages.txt. For torcx packages,
extract the reports from the torcx bundle when adding it to the rootfs.
For initramfs packages, as a substitute we enumerate build dependencies
of coreos-kernel (image_packages_implicit()). At this time these are
bootengine and intel-microcode.
Prod images need libstdc++.so and other libraries produced by
sys-devel/gcc build, but because we don't want all of gcc in the image,
the binpkg is manually unpacked instead of installed with emerge. Make
sure to preserve SLSA metadata when unpacking as well.
`./setup_board --nousepkg --nogetbinpkg` currently fails with a
circular dependency due to pulling in the whole systemd-cryptsetup-udev
dependency chain. This is due to several issue:
* `emerge --root=$ROOT --emptytree` considers ROOT=/ to also be empty,
so it pulls in all host packages. This must've not always been the case.
So we need to pipe the dependency package list through `egrep $ROOT`
to filter only those that would get installed into the desired ROOT
* if SYSROOT=/ and not SYSROOT=ROOT, then virtual/os-headers is missing
from $ROOT package list
* the final filter expression tries to previously looked like this:
(=sys-devel/gcc|sys-devel/binutils-0.9) which also matches
sys-devel/gcc-config and sys-devel/binutils-config, which are
necessary dependencies. Rework the match expression to not filter
those out.
This made no difference back when lib was a symlink to lib64, but now that they are separate,
libs belongs in /usr/lib64. This mostly doesn't show up because ldconfig configures the ld.so cache
to include both locations, but when updating from an older release ld.so.cache is out of date.
Unfortunately ld.so.cache does not get updated until after multipathd, which causes
multipathd to dump core. This may also affect other packages that need access to
libgcc early.
See also: https://github.com/flatcar-linux/Flatcar/issues/809
Since v0.51.0 syft supports generating parsing the gentoo package
database. This is a first go at integrating that into our image build
process. This doesn't yet include packages inside torcx packages, or the
kernel, or initramfs-only packages.
There is some cruft left after grub hashes generation. After the
contents are zipped into archive, they don't need to be around any
more.
Try to remove the rootfs directory after unmounting the
image. disk_util can recreate it again if there is a need for it.
Remove the build directory used for generating ACI images - it's not
needed after successful installation.
The new build pipeline compresses images already but uploaded both the
compressed and uncompressed files because the whole build folder gets
uploaded.
Add a new flag "--only_store_compressed" to the image generation which
deletes the uncompressed file after compression is done. Uncompressed
images are still supported if specified in the flag
"image_compression_formats".
Closes https://github.com/flatcar-linux/Flatcar/issues/793
Make qemu_template.sh work on MacOS
Line 14; The nproc command is only available on systems with GNU coreutils installed. The getconf _NPROCESSORS_ONLN alternative will work on a wider range of UNIX systems.
Line 114; The mktemp syntax used only works on GNU implementation.
Line 159; added hvf (MacOS) and tcg (no acceleration) options as a fallback. By doing this qemu-system-x86_64 will try to use kvm, but when it fails try hvf, and when that fails switch to the tcg accelerator.
This was already being done for gcc but not for binutils. Binutils is
also slotted and when we run the sdk (stage3/stage4) job in the CI, the
seed sdk already contains crossdev packages that we may want to update.
The os-release file was not only accessible through /usr/lib/ but
also through /usr/lib64 because "lib" was just a symlink.
Now that we split them up into two directories, add a compatibility
symlink in case /usr/lib64 was used to access os-release. A check
is added to also work without the split which is useful if the split
is not done for the SDK at the same time.
The standard location is /usr/lib/modules but on Flatcar "lib" was a
symlink to "lib64". Now this is going to be split up in separate
directories but with compatibility symlinks.
Add the new location to the ignore list.
Sysext images have a compatibility matching mechanism that searches for
the matching OS version or custom sysext level setting. On Flatcar
there is just the OS version set in /etc/os-release until now which
means that sysext images can't easily be used together with autoupdates
that change the OS version.
Define a sysext level for Flatcar so that users can refer to it instead
of the OS version when they have images that don't rely on a particular
Flatcar version.
Here an example of the now possible metadata:
/etc/extensions/NAME/usr/lib64/extension-release.d/extension-release.NAME
ID=flatcar
SYSEXT_LEVEL=1.0
and a symlink /etc/extensions/NAME/usr/lib → /etc/extensions/NAME/usr/lib64
to work around the problem that using lib/ as path destroys Flatcar's
lib → lib64 symlink.
In the future the matching logic hopefully gets more flexible because
now it is just a string comparison. Also, the architecture is not
matched either for now - we should work with upstream to improve this.
Closes: https://github.com/flatcar-linux/Flatcar/issues/643
Rename sztd to zst and amend the changelog. The zstd binary generates a
compressed file with the .zst extension by default.
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
This change adds a new flag called --image_compression_format which
allows us to output the final VM image as one of the supported formats:
bz2 (default), gz, zip or none
if the compression format is "none" or "", the image will not be compressed.
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
Disable ebuild-locks for the emerge command that creates the image.
Ebuild-locks protect unsandboxed ebuild phases from running
concurrently, but also slow things down greatly when a lot of
concurrency would otherwise be possible. The image build phase merges a
big amount of binary packages, and I am not aware of us having any
phases that risk concurrently modifying shared files.
I have been testing this for the last months and have not seen any
failures. The time savings are significant: this cuts image build time
from 20m to 10m for me.
Signed-off-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>
Package users nowadays get created through systemd-sysuser files.
Gentoo uses the acct-user|groups packages to allocate stable IDs for
these users. Since they get created at runtime, we have the problem
that they end up in /etc/passwd at boot time which would be fine if
they follow the acct-user allocations but it could also be that there
is a package that uses its own sysuser files, leading to dynamic ID
allocation which we can't control and may result in ugly user ID
mismatches that are hard to resolve again. Normally we intend to ship
all system users under /usr/share/baselayout/passwd so that /etc/passwd
is really left to the user's own entries.
Generate the /etc/passwd sysuser entries at image build time and move
these entries over to /usr/share/baselayout/passwd so that all
system users reside in this database. We should still ensure to have
acct-user packages for all system users or at least hardcoded user
IDs, therefore, add a check for that.
201908-24: polkit 0.120-r2, so not affected
201909-01: perl 5.34.0, so not affected
202003-26: python 3.9.8, so not affected
202005-09: python 3.9.8, so not affected
202006-03: perl 5.34.0, so not affected
202008-01: python 3.9.8, so not affected
202101-18: python 3.9.8, so not affected
202104-04: python 3.9.8, so not affected
202105-34: bash 5.1_p8, so not affected
202107-31: polkit 0.120-r2, so not affected
202107-48: systemd 250.3, so not affected
Azure requires disks to be fixed-size VHD files when uploading to blob storage
in order to create image/gallery objects from them. This is documented here[1].
To prevent mistakes from happening create disks in that format directly so that
any azure compatible tool can upload them, though azcopy is recommend because
it handles their sparseness best.
This has not been an issue for us so far because kola uses code from an older
utility that transparently handled the dynamic-to-fixed-size conversion for VHD
files (azure-vhd-utils). But people working with these things for the first
time fall into this trap.
[1]: https://docs.microsoft.com/en-us/azure/virtual-machines/linux/create-upload-generic#resizing-vhds.
Signed-off-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>
source_on_disk() so far relied on the 'sourcePackage' field, which contains the
primary dependency of a torcx packge (app-torcx/docker ->
app-emulation/docker). Now the 'metaPackage' field (app-torcx/docker) is used,
which lets us look at RDEPENDS and figure out all packages that are indirectly
installed when installing a torcx package. torcx_dependencies() does just that,
so move it's definition to torcx_manifest.sh.
Signed-off-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>
The torcx_manifest.json file currently has a 'sourcePackage' field which is
extracted from the first runtime dependency of the torcx package ebuild. This
is a convention, and causes sourcePackage to hold 'app-emulation/docker' for
the 'app-torcx/docker' package. This does not carry enough information to be
able to figure out what other packages are part of the torcx package.
Store an additional field, 'metaPackage', in the manifest which contains the
name of the torcx package. With the right ebuild it is then possible to figure
out what other packages are part of a given torcx package. This can then be
used to add that information to the image packages list.
Signed-off-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>
The default image group is already encoded in
/usr/share/flatcar/update.conf but it was written to
/etc/flatcar/update.conf as well. This can cause problems when the user
switches channels by forcing an update to a specific release from the
different channel (e.g., through the flatcar-update tool) as it leaves
the file under /etc/flatcar/update.conf out of sync with the new
channel version in /usr/share/flatcar/update.conf.
Since we don't really need to write a specific channel to /etc on new
images as we can rely on the value from /usr, we now leave any possible
overwriting of the value in /etc entirely to the user.
Since the update of dev-python/certifi, running the command
`./image_to_vm.sh --format gce --board=amd64-usr` fails due to a
dangling symlink. This symlink is located in
/usr/lib64/python3.9/site-packages and is not supposed to be installed
in the first place because of this INSTALL_MASK entry in
coreos-overlay/profiles/coreos/targets/generic/oem-aci/make.defaults:
INSTALL_MASK="${INSTALL_MASK}
/usr/*/python3*
"
There is an open upstream bug that INSTALL_MASK doesn't work correctly on
symlinks (https://bugs.gentoo.org/678462).
The best we can do at this time is to ignore the dangling symlink.
Signed-off-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>
This change adds a job for publishing binary packages to the build cache
server to the ci automation.
Also, setup_board is updated to use the buildcache package cache if a
nightly build version is detected.
Signed-off-by: flatcar-ci <infra+ci@flatcar-linux.org>
Stop relying on github redirects, they are a mixed blessing and using
them broke emerge-gitclone inside dev-container in silent way. The
script could not find a desired revision of portage-stable or
coreos-overlay, because it tried to pull from kinvolk instead of
flatcar-linux github org. The redirects seem to hinder fetching a
specific commit, so the script pulled something else (HEAD or main?).
