Apparently `local -x FOO` does not locally export an already existing
variable, but rather does some whole weird lot of nothing - it shadows
an existing variable with a new unset one, but it won't export it
until it gets assigned.
We only want to do the signing in Azure, not the whole image job. This
new job downloads the unsigned image, signs it, and replaces it.
Signed-off-by: James Le Cuirot <jlecuirot@microsoft.com>
The --extract_update option used to do exactly that, just extract the
USR-A partition for updates and no more. Now it does the same thing as
--generate_update, except it names the file flatcar_test_update.gz
rather than flatcar_production_update.gz. --generate_update is never
actually used because official update payloads are manually generated
with the generate_payload script later on.
Resolve this confusion by deduplicating the common code between them.
Any update payload produced during this stage of the build is only
useful for testing, so change --generate_update to always create
flatcar_test_update.gz. --generate_update now implies --extract_update
and both are enabled by default.
Signed-off-by: James Le Cuirot <jlecuirot@microsoft.com>
We were supposed to collect allowed users and allowed groups into
separate arrays. Due to the copy-paste mistake, we overwrote allowed
users array with allowed groups while leaving the array for allowed
groups empty, so we ended up passing only allowed groups instead of
both.
Most of this hinges on the --upload option being passed, and it never is
any more. Much of it also uses Google Buckets, which we no longer use,
save for some GCE-specific bits.
Signed-off-by: James Le Cuirot <jlecuirot@microsoft.com>
We don't want to be blocked from doing releases in the meantime. Revert
this commit when ready.
Signed-off-by: James Le Cuirot <jlecuirot@microsoft.com>
This is what upstream Gentoo does. They would previously update the
entire seed, but this took a long time. Our seeds are much bigger, so we
kept repo snapshots to build stage1 against these instead. The new
method of only rebuilding packages with changed sub-slots is a good
compromise and removes the need to write stage1 hooks that selectively
catch the repository up.
This also avoids some conflicts by adding the `--ignore-world` option.
Gentoo seeds have nothing in @world. We have much more, but none of that
is needed for stage1.
This continues to exclude cross-*-cros-linux-gnu/* as that is not needed
for stage1. It now also excludes dev-lang/rust, because it is never a
DEPEND, so it would not break other packages in this way. It may fail to
run due to a sub-slot change in one of its own dependencies, but it is
also unlikely to be needed in stage1 and it is not configured to use the
system LLVM. If needs be, we could improve the behaviour of Portage's
@changed-subslot to respect `--with-bdeps`.
In my testing, it was unable to handle an SDK from 17 months ago, but
one from 7 months ago did work. In practise, we will always use a much
more recent one, which is far more likely to work.
Signed-off-by: James Le Cuirot <jlecuirot@microsoft.com>
The cros_workon tool has been replaced with a simpler flatcar_workon
tool based around git-r3.
Signed-off-by: James Le Cuirot <jlecuirot@microsoft.com>
The removal of files in the overlay present in the lowerdir creates
whiteout entries that mask the lowerdir entries. For those files that
have a tmpfile rule for creation, a reboot would cause the file to be
created in the upperdir, meaning this file is not updated from the
lowerdir when it changes. In addition we have filtered out some tmpfile
rules that caused upcopies (symlinks and directories) which meant that
removing the /etc/resolv.conf symlink didn't bring it back after reboot.
To make files from the lowerdir show up if they have a tmpfile rule that
normally would recreate them we keep a list of whiteout entries that we
clean up on boot. This also prevents freezing files because
systemd-tmpfiles does not need to recreate them in the upperdir.
- updated github actions for runc, containerd, and docker to not handle
nonexistent ebuilds in app-torcx/ anymore
- removed spurious package_run_dependencies from build_image_util.sh
- build_sysext: generate pkginfo before mangle script runs
use zstd for compression; add cli flag to select compression
- ci_automation_common.sh: remove spurious `/` from match string
- coreos, board-packages, bootengine: bump ebuild revisions
- kernel commonconfig: add squashfs zstd support
Signed-off-by: Thilo Fromm <thilofromm@microsoft.com>
This change refactors base OS sysext builds to use a separate build
script `build_library/sysext_prod_builder`, which is called from
`build_library/prod_image_util.sh` when `build_image` runs.
This allows for better separation of cleanup traps: prod image sysext
builds need its own trap / cleanup function for temporary build
directories and loopback mounts.
Prod sysext builds properly generate lincense and SBOM information, and
provide detailed file listings and disk space usage stats.
- SBOM / licenses JSON now include all packages of the
final image, i.e. a combined list of base image and all base OS
sysexts.
- Packages lists, files list and detailed files list include the sysext
squashfs files for the base image, and separate sections with files /
packages lists for each sysext.
- Disk usage contains both final disk image usage as well as usage of
each individual sysext squashfs.
This change refactors sysext builds during build_image and generalises
the code (no hard-coded containerd and docker anymore).
A command line option is added to build_image for sysexts to include in
the OS image. It defaults to containerd and docker but may be set to
arbitrary packages. The command line supports simple depenencies, i.e.
the "docker" sysext will re-use package information from the
"containerd" sysext and not include another containerd.
Signed-off-by: Thilo Fromm <thilofromm@microsoft.com>
This change removes torcx libraries, references, and commandline options
from build automation scripts and from build_library/.
Containerd and docker are shipped via sysexts which are included in the
base image.
Signed-off-by: Thilo Fromm <thilofromm@microsoft.com>
The `localedef` tool expects `/usr/lib/locale` directory to
exist. This directory used to be created by the `sys-libs/glibc`
package (with the `keepdir` directive), but after the update of the
package, the locale generation stuff (and the `keepdir` directive )was
moved to the `sys-libs/locale-gen` package. This package is not
installed in the production images, so the `/usr/lib/locale` directory
was not created. In such a situation, calling localedef to generate
C.UTF-8 locale resulted in an error like:
cannot create temporary file: ${SOME_ROOTFS}/usr/lib/locale/locale-archive.ufpG15: No such file or directory
Create the directory before calling localedef to fix the problem.
The temporary /etc backup created during emerging packages should only
contain empty files that will make sure that the symlinks pointing to
files within the /etc backup won't dangle at any time.
We used to create a base_image_var.conf tmpfiles config file that
contained information about directories under /var that weren't
covered by any other tmpfiles config file. Recently some package
update started installing a directory under /var that belonged to a
user/group not found directly in passwd/group file in /etc. This
user/group was defined in passwd/group in /usr/share/baselayout, but
at the early boot, these are not yet checked for user/group
information, so systemd-tmpfiles running inside initrd failed when
trying to create such an entry using the base_image_var.conf tmpfiles
config file.
Split the base_image_var.conf into two files - base_image_var.conf and
base_image_var_late.conf. The former will only contain entries owned
by user/group that are supposed to exist very early in the boot, while
the latter will contain the rest of directories - those will be
created later during the boot.
We already drop tmpfile rules that we don't need because we ship the
files through our /etc overlay. However, some rules weren't dropped
because they used tabs and not spaces (/etc/selinux/, /etc/iscsi and
/etc/ssl/*).
Drop rule lines for /etc that use tabs. Also rules modifiers like ! to
only do it during boot or - to allow failure will be removed but those
with + or = will stay as they to explicit recreation.
The existing tmpfile logic took care of folders that the ebuild keepdir
directive wanted to exist on the OS. However, files and symlinks were
not created, causing them to be missing if we didn't explicitly modify
the ebuild files in coreos-overlay to use tmpfiles or patching of
paths to be in /usr. We need a logic to provide /etc files from the
current /usr partition without getting stale. This can be done best
with an overlay mount which requires to keep the original /etc files
under /usr.
Move the final /etc folder of the image build to /usr/share/flatcar/etc
to serve as lower layer in the overlay. Also remove any state from the
rootfs to make sure that we don't rely on it when testing our images
before the release. What we get with an overlay mount is essentially a
similar behavior to a 3-way merge because as long as the user didn't
change the files, the old version is replaced with the new version and
as soon as the user did changes, that file is frozen and wins over the
provided old (in case of a rollback) or new versions from /usr. It does
not work on file lines but on whole file contents, yet that is also
what rpm-ostree does to my knowledge. Also, run tmpfiles once and do
the SELinux labeling to prevent files being created in the upperdir
because they were missing in the lowerdir, or because they had missing
SELinux labels.
Emerge flags are cryptic in general, but short flags even more so, so
expand them. While at it, I noticed some places where bash arrays
could be used, so convert those places too.
Timestamp and user/group information are out, in are device ID and
inode number. That way, the file can be used for accounting size
differences of files/image.
We need to fix another symlink created by gcc-config, so extend the
function that was doing it for some other links and rename it - it's
not about only liblto any more.
Some packages are currently missing from the /usr/share/SLSA directory
compared to flatcar_production_image_packages.txt. For torcx packages,
extract the reports from the torcx bundle when adding it to the rootfs.
For initramfs packages, as a substitute we enumerate build dependencies
of coreos-kernel (image_packages_implicit()). At this time these are
bootengine and intel-microcode.
Since v0.51.0 syft supports generating parsing the gentoo package
database. This is a first go at integrating that into our image build
process. This doesn't yet include packages inside torcx packages, or the
kernel, or initramfs-only packages.
There is some cruft left after grub hashes generation. After the
contents are zipped into archive, they don't need to be around any
more.
Try to remove the rootfs directory after unmounting the
image. disk_util can recreate it again if there is a need for it.
Remove the build directory used for generating ACI images - it's not
needed after successful installation.
