The qemu and qemu_uefi_secure images have the same contents as the
qemu_uefi image which wastes space on the release server. A similar
case is the PXE vmlinuz which is the same as the regular one, too.
Set up symlinks for same images, and also detect this when compressing
to set up symlinks there as well. To reduce complexity, the qemu and
qemu_uefi_secure images are not supported anymore and the Jenkins or
GitHub CI will skip over them if specified. Users that build their own
images need to adapt, though.
Add support for Gen 2 Hyper-V VMs.
`./image_to_vm` tool has now a new supported format: `hyperv_vhdx`,
that produces .vhdx dynamic disks.
How to use:
```bash
./image_to_vm.sh --from ../build/images/amd64-usr/developer-latest/ --format hyperv_vhdx
```
See: https://github.com/flatcar/Flatcar/issues/1009
Uses PR: https://github.com/flatcar/bootengine/pull/92
Signed-off-by: Adrian Vladu <avladu@cloudbasesolutions.com>
With the PXE script it is easy to boot different versions from one
folder without any copies because the kernel and PXE initrd are always
"fresh".
Instead of only supporting hardcoded file names, support parameters for
the kernel and initrd file to be used.
The qemu UEFI and regular qemu script only differ by having a default
value for the firmware. If one tries to switch between different
firmwares one normally would modify the script.
Make it easier to switch boot modes and use custom firmwares by
supporting a flag to set the pflash contents.
For testing TPM2-backed rootfs encryption it is handy to have a software
TPM option for the qemu script.
Add a flag for a software TPM with swtpm like kola also does. The user
has to specify a folder for the secret state and this won't be removed
because the same store should be able to be passed when booting the VM
again after shutdown.
While Flatcar itself runs fine with 1 GB, many workloads do not and
having to debug this is time consuming when one forgets to bump the VM
memory, e.g., in the Qemu script.
Default to 2 GB as known-good setting for things like Kubernetes or
setting up LUKS devices.
When testing multiple images one always has to copy them to the
expected file name, and when trying to run two VMs this means one needs
to either use separate directories or modify the qemu script. One also
needs to modify the qemu script to bump the memory for K8s or for LUKS.
Support parameters for the VM image name and the VM memory.
The ACI OEM container image was used with rkt for GCE. For long time
this has been migrated to systemd-nspawn and now systemd-sysext.
Remove the unused build library code.
Mksquashfs running against a btrfs filesystem tries to capture btrfs specific
xattrs (btrfs.compression) generating a lot of spam. Remove the spam by
ignoring btrfs xattrs.
Signed-off-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>
Putting things in a sysexts subdir results in the same layout on
bincache and does not follow the expected url schema for fetching the
sysext.
Signed-off-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>
This allows it to be sourced from other scripts and used e.g. in
generating an image changes summary.
Signed-off-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>
There are two challenges with the sysext: it needs config files in /etc
and it needs udev rules for mounting during boot to work. The etc files
are placed in the standard flatcar etc overlay path but the overlay is
mounted from the initrd. So instead, we create a tmpfiles.d rule that
symlinks the best important files over. For the udev issue, we create a
drop-in in /etc that ensures udev runs after systemd-sysext.
We also can't rely on systemd presets to work, so instead parse the
preset file and statically create the service dependencies. For the
primary zfs.target we rely on an Upholds entry. Users can still disabled
unwanted services if they want.
We also removed unnecessary files:
- development files
- initramfs related scripts
Signed-off-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>
This function is meant to prebuild certain sysexts to be released along
with each release. These will not be built into the image, but instead
can be fetched by the user on demand.
The command to build sysexts would be:
./build_image prod sysext
Signed-off-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>
It looks like 'norecovery' is deprecated and has been removed in the v6.8-rc1
kernel. Replace it with 'rescue=nologreplay', which is a replacement
implemented since v5.9. The standalone 'nologreplay' option is also deprecated.
Signed-off-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>
We have an existing qemu_uefi_secure format definition, but it is
necessary to update it so that it actually works. Qemu needs to be
passed the correct flags to enable SMM, we need to switch to the Q35
machine, and we need to copy over the secboot variant of the OVMF
firmware.
This is just the contents of the section, but the section
itself is written by grub-mkimage. sbat.csv needs to be passed
with --sbat.
Signed-off-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>
Signed-off-by: Sayan Chowdhury <schowdhury@microsoft.com>
The removal of files in the overlay present in the lowerdir creates
whiteout entries that mask the lowerdir entries. For those files that
have a tmpfile rule for creation, a reboot would cause the file to be
created in the upperdir, meaning this file is not updated from the
lowerdir when it changes. In addition we have filtered out some tmpfile
rules that caused upcopies (symlinks and directories) which meant that
removing the /etc/resolv.conf symlink didn't bring it back after reboot.
To make files from the lowerdir show up if they have a tmpfile rule that
normally would recreate them we keep a list of whiteout entries that we
clean up on boot. This also prevents freezing files because
systemd-tmpfiles does not need to recreate them in the upperdir.
Recently we had some problems with read-only filesystems, that pop up
in different places. It may be easier to catch if the debugging info
is printed in an error case instead of just one specific location.
We see occasional failures due to a read-only filesystem on GitHub
Action runners:
mkdir: cannot create directory ‘/home/sdk/trunk/src/scripts/artifacts/amd64-usr/developer-3790.0.0+nightly-20231116-2100-5-g49eb1a4c07-a1/rootfs/usr/share/flatcar/sysext’: Read-only file system
Add commands for a debug output.
The special Brightbox image uses the OpenStack userdata in Ignition but
lacked Afterburn usage. It actually works to use the OpenStack image and
directly which also enables Afterburn, thus we can drop the special
image.
Don't build a special image for Brightbox but recommend to use OpenStack
images directly. A symlink is added to help with the download of
hardcoded user scripts.
This change removes Flatcar specific builds of docker[-cli], containerd,
runc, and cri-tools and instead switches to upstream Gentoo ebuilds
added to portage-stable.
The change updates docker to 24.0.6.
NOTE that there currently is no upstream ebuild for containerd-1.7.7, so
this change adds that ebuild based on the upstream containerd-1.7.6
ebuild.
Flatcar customisations like systemd units etc. are now applied in the
manglefs script of the respective sysexts, based on file system trees in
coreos-overlay/coreos/sysext/(containerd|docker).
The build_sysext script has been extended by an option to strip all
binaries in a sysext; the option is active by default. This takes care
of removing debug symbols from docker and containerd - which are not
removed by the default Gentoo build. The overall size of both containerd
and docker sysext is reduced by ~50%.
Lastly, the sysext command line syntax of build_image has been extended
to allow specifying multiple packages for a sysext. This was necessary
because docker-cli and docker do not have any runtime relationships and
therefore must both be specified for installation to correctly mirror
Flatcar's own docker packaging.
Signed-off-by: Thilo Fromm <thilofromm@microsoft.com>
- updated github actions for runc, containerd, and docker to not handle
nonexistent ebuilds in app-torcx/ anymore
- removed spurious package_run_dependencies from build_image_util.sh
- build_sysext: generate pkginfo before mangle script runs
use zstd for compression; add cli flag to select compression
- ci_automation_common.sh: remove spurious `/` from match string
- coreos, board-packages, bootengine: bump ebuild revisions
- kernel commonconfig: add squashfs zstd support
Signed-off-by: Thilo Fromm <thilofromm@microsoft.com>
This change refactors base OS sysext builds to use a separate build
script `build_library/sysext_prod_builder`, which is called from
`build_library/prod_image_util.sh` when `build_image` runs.
This allows for better separation of cleanup traps: prod image sysext
builds need its own trap / cleanup function for temporary build
directories and loopback mounts.
Prod sysext builds properly generate lincense and SBOM information, and
provide detailed file listings and disk space usage stats.
- SBOM / licenses JSON now include all packages of the
final image, i.e. a combined list of base image and all base OS
sysexts.
- Packages lists, files list and detailed files list include the sysext
squashfs files for the base image, and separate sections with files /
packages lists for each sysext.
- Disk usage contains both final disk image usage as well as usage of
each individual sysext squashfs.
This change refactors sysext builds during build_image and generalises
the code (no hard-coded containerd and docker anymore).
A command line option is added to build_image for sysexts to include in
the OS image. It defaults to containerd and docker but may be set to
arbitrary packages. The command line supports simple depenencies, i.e.
the "docker" sysext will re-use package information from the
"containerd" sysext and not include another containerd.
Signed-off-by: Thilo Fromm <thilofromm@microsoft.com>
This change removes torcx libraries, references, and commandline options
from build automation scripts and from build_library/.
Containerd and docker are shipped via sysexts which are included in the
base image.
Signed-off-by: Thilo Fromm <thilofromm@microsoft.com>
