The docker and containerd copy files from the repository, which are owned by
the sdk user. This ownership leaks into the final image, which means the first
created user could edit systemd files. This is bad.
Modify the cp invocation to copy files without preserving ownership. The
sysext-mangle script is called by build_sysext, which is executed using sudo.
Signed-off-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>
build_sysext uses a base squashfs (basically a full snapshot of the
Flatcar OS image) to build custom sysexts on top. Before building it
ensures the base image actualy matches the OS version in the repository
root.
The version string includes a BUILD_ID which might be auto-generated (by
including common.sh) if it is not present in the version file - e.g.
when the version is an official release (tag). This build ID
auto-generation causes issues with the version check when image build
and sysext build scripts run independently - each will generate its own
build ID, and this will cause build_sysext's version check to fail.
build_sysext will now use the build id from the base squashfs when it is
not set in the source tree's version.txt to work around that issue. This
is a more general solution than 361eda220b
(which this patch reverts) as it directly addresses the issue in
build_sysext instead of working around it in sysext_prod_builder.
Signed-off-by: Thilo Fromm <thilofromm@microsoft.com>
It is not clear why this was forked originally. One reason was to avoid
the sys-apps/lsb-release dependency, but it probably wasn't just that.
It seems likely that the upstream package did not support cross targets
at the time. Now it does.
It appears that LTO was previously enabled by us following Gentoo rather
than through an explicit decision. They now disable it by default, so we
do likewise. It previously used "fat" LTO, which makes Rust especially
slow to build and reportedly made rustc slower than with "thin" LTO!
There seems little benefit in using thin LTO given that we rebuild Rust
almost as much as the packages that use it, plus we don't enable LTO
anywhere else.
We still avoid rustdoc to keep the size down using INSTALL_MASK. This
isn't as good as not building it in the first place, but this alone
isn't worth keeping a fork.
Cross targets are now handled via the admittedly experimental
RUST_CROSS_TARGETS support. This has been in place for a while, and I
think it is fairly widely used now. If it does disappear, it would
almost certainly be for something even better.
This also updates Rust from 1.80.0 to 1.80.1.
Signed-off-by: James Le Cuirot <jlecuirot@microsoft.com>
It will patch gcc to respect ESYSROOT when cross-compiling, effectively
adding the --sysroot flag without the use of flags or wrappers. This
hasn't been merged into Gentoo yet, but it has been given the nod. When
it does get merged, it was only be for newer gcc versions than we're
currently using, so we'll need this user patch in the meantime
regardless.
Signed-off-by: James Le Cuirot <jlecuirot@microsoft.com>
From https://wiki.gentoo.org/wiki/Catalyst/Stage_Creation#Build_Stage3:
> It is not necessary to build stage2 in order to build stage3. Gentoo
> release engineering does not build stage2, and you should not need to
> unless you're intentionally building a stage2 as your goal.
We can now sync portage-stable/scripts with upstream because
bootstrap.sh is only used during stage2, and the changes we had are no
longer relevant. It seems likely the changes were already redundant
anyway.
Signed-off-by: James Le Cuirot <jlecuirot@microsoft.com>
These flags normally need to be temporarily forced during stage1, but we
already force them permanently in our profiles.
Removing this appears to make build_library/portage redundant, but it
will later be used to allow building under QEMU with Catalyst, and it
could have other uses too.
Signed-off-by: James Le Cuirot <jlecuirot@microsoft.com>
The changes to support Catalyst 4 are not backwards compatible and we
need a seamless transition for builds in CI.
Signed-off-by: James Le Cuirot <jlecuirot@microsoft.com>
This is what upstream Gentoo does. They would previously update the
entire seed, but this took a long time. Our seeds are much bigger, so we
kept repo snapshots to build stage1 against these instead. The new
method of only rebuilding packages with changed sub-slots is a good
compromise and removes the need to write stage1 hooks that selectively
catch the repository up.
This also avoids some conflicts by adding the `--ignore-world` option.
Gentoo seeds have nothing in @world. We have much more, but none of that
is needed for stage1.
This continues to exclude cross-*-cros-linux-gnu/* as that is not needed
for stage1. It now also excludes dev-lang/rust, because it is never a
DEPEND, so it would not break other packages in this way. It may fail to
run due to a sub-slot change in one of its own dependencies, but it is
also unlikely to be needed in stage1 and it is not configured to use the
system LLVM. If needs be, we could improve the behaviour of Portage's
@changed-subslot to respect `--with-bdeps`.
In my testing, it was unable to handle an SDK from 17 months ago, but
one from 7 months ago did work. In practise, we will always use a much
more recent one, which is far more likely to work.
Signed-off-by: James Le Cuirot <jlecuirot@microsoft.com>
Catalyst 4 has totally changed the way repositories are handled. It only
works when the name of the directory containing the repository matches
the configured name of that repository. This was not the case for us,
with the coreos repository residing in the coreos-overlay directory. We
wanted to move and rename our repositories anyway, but this is a big
change, so we'll do separately. For now, this just renames coreos to
coreos-overlay.
Catalyst 4 also ingests the main repository snapshot as a squashfs
rather than a tarball. It features a utility to generate such a
snapshot, but it doesn't fit Flatcar well, particularly because it
expects each ebuild repository to reside at the top level of its own git
repository. It was very easy to call tar2sqfs manually though.
Signed-off-by: James Le Cuirot <jlecuirot@microsoft.com>
This change fixes a version mismatch of FLATCAR_BUILD_ID when performing
a dev build of an existing release tag. The build ID is part of the
version string of dev builds, separated by a "+" from the main version.
If common.sh detects a dev build (COREOS_OFFICIAL != 1) and
FLATCAR_BUILD_ID is empty, common.sh will generate a new ID based on a
timestamp.
For official releases, FLATCAR_BUILD_ID is not set in version.txt. A dev
build of a release tag would make common.sh generate a new ID each time
it is sourced by different processes. build_image sources common.sh
first, and writes the resulting version string the OS image's
os-release file. build_sysext runs later and also sources common.sh,
leading its version check to fail as its own VERSION now differs from
the version of the OS image it's supposed to generate sysexts for.
This change reads BUILD_ID from the OS image rootfs in
sysext_prod_builder and exports FLATCAR_BUILD_ID accordingly before
calling build_sysext. Hence FLATCAR_BUILD_ID is not empty, so common.sh
in build_sysext will not re-generate it.
Signed-off-by: Thilo Fromm <thilofromm@microsoft.com>
The cros_workon tool has been replaced with a simpler flatcar_workon
tool based around git-r3.
Signed-off-by: James Le Cuirot <jlecuirot@microsoft.com>
Since we build them into the grub executable, they are not needed on
disk. The only case I am unsure of is legacy BIOS boot, so left those
on disk.
Signed-off-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>
Signed-off-by: Adrian Vladu <avladu@cloudbasesolutions.com>
We currently carry multiple copies of the same grub core.elf or core.efi
on the boot partition. Save some space by removing duplicates that are
never used at runtime. CPIO build needed to be adapted because it
publishes grub efi files.
Signed-off-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>
Enabled user session dbus in base image to support podman rootless mode.
Extension images can now be created from multiple packages by seperating
them with a comma. The podman sysext includes app-containers/podman and
net-misc/passt.
It can be enabled by adding podman to /etc/flatcar/enabled-sysext.conf.
Potential TODO: gpgme had to be added as BDEPEND to podman ebuild.
As Ignition supports KubeVirt, add a custom oem for it and also the
required parts to be able to build an image in .qcow2 format that
is already using internal .qcow2 gzip compression.
Fixes: https://github.com/flatcar/Flatcar/issues/1358
Signed-off-by: Adrian Vladu <avladu@cloudbasesolutions.com>
For importing Scaleway images, extension needs to be '.qcow2'
See: https://www.scaleway.com/en/docs/compute/instances/how-to/snapshot-import-export-feature/
> Make sure that the QCOW / QCOW2 image file you want to import,
> uses the file extension .qcow or .qcow2 to avoid issues while importing the image.
Signed-off-by: Mathieu Tortuyaux <mtortuyaux@microsoft.com>
This variable allows to override the disk extension which is initially
based on the DISK_FORMAT.
Signed-off-by: Mathieu Tortuyaux <mtortuyaux@microsoft.com>
The OEM partition is on a btrfs file system and grub has to be able to
read grub.cfg from there so it makes sense to include btrfs in core
modules. This avoids all other fs modules from being autoloaded during
boot.
Signed-off-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>
In the context of load.cfg in the memdisk, root is set to memdisk, so
passing it as a hint to search is not helpful. While we don't know for
sure whether hd0 is the boot disk, it's a safe hint for most situations.
Signed-off-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>
The move to symlinking to the qemu-uefi image also resulted in the
qemu-uefi image being referenced in the qemu-bios and qemu-uefi-secure
scripts instead of referencing the image symlinks. Same for the VM name
shown in the qemu window title.
When generating the qemu scripts, use the original qemu image name and
VM name.
When mksquashfs encounters btrfs attributes which indicate that the file
is transparently compressed, it can't embed this info in the squashfs
and prints a warning about that.
Silence the warnings by excluding btrfs attributes as done already in
the rest of the code base. This aligns the mksquashfs invocation in this
regard.
The qemu and qemu_uefi_secure images have the same contents as the
qemu_uefi image which wastes space on the release server. A similar
case is the PXE vmlinuz which is the same as the regular one, too.
Set up symlinks for same images, and also detect this when compressing
to set up symlinks there as well. To reduce complexity, the qemu and
qemu_uefi_secure images are not supported anymore and the Jenkins or
GitHub CI will skip over them if specified. Users that build their own
images need to adapt, though.
Add support for Gen 2 Hyper-V VMs.
`./image_to_vm` tool has now a new supported format: `hyperv_vhdx`,
that produces .vhdx dynamic disks.
How to use:
```bash
./image_to_vm.sh --from ../build/images/amd64-usr/developer-latest/ --format hyperv_vhdx
```
See: https://github.com/flatcar/Flatcar/issues/1009
Uses PR: https://github.com/flatcar/bootengine/pull/92
Signed-off-by: Adrian Vladu <avladu@cloudbasesolutions.com>
With the PXE script it is easy to boot different versions from one
folder without any copies because the kernel and PXE initrd are always
"fresh".
Instead of only supporting hardcoded file names, support parameters for
the kernel and initrd file to be used.
The qemu UEFI and regular qemu script only differ by having a default
value for the firmware. If one tries to switch between different
firmwares one normally would modify the script.
Make it easier to switch boot modes and use custom firmwares by
supporting a flag to set the pflash contents.
For testing TPM2-backed rootfs encryption it is handy to have a software
TPM option for the qemu script.
Add a flag for a software TPM with swtpm like kola also does. The user
has to specify a folder for the secret state and this won't be removed
because the same store should be able to be passed when booting the VM
again after shutdown.
While Flatcar itself runs fine with 1 GB, many workloads do not and
having to debug this is time consuming when one forgets to bump the VM
memory, e.g., in the Qemu script.
Default to 2 GB as known-good setting for things like Kubernetes or
setting up LUKS devices.
When testing multiple images one always has to copy them to the
expected file name, and when trying to run two VMs this means one needs
to either use separate directories or modify the qemu script. One also
needs to modify the qemu script to bump the memory for K8s or for LUKS.
Support parameters for the VM image name and the VM memory.
The ACI OEM container image was used with rkt for GCE. For long time
this has been migrated to systemd-nspawn and now systemd-sysext.
Remove the unused build library code.
Mksquashfs running against a btrfs filesystem tries to capture btrfs specific
xattrs (btrfs.compression) generating a lot of spam. Remove the spam by
ignoring btrfs xattrs.
Signed-off-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>
Putting things in a sysexts subdir results in the same layout on
bincache and does not follow the expected url schema for fetching the
sysext.
Signed-off-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>
