Commit Graph

24810 Commits

Author SHA1 Message Date
Flatcar Buildbot
40a31c4eca dev-libs/protobuf: Sync with Gentoo
It's from Gentoo commit d674a844b38ef779323f6b445f58db22c6586c19.
2023-10-25 15:05:33 +02:00
Flatcar Buildbot
f15bf139fa dev-lang/python: Sync with Gentoo
It's from Gentoo commit 0bcd2ab51bbbac3397629a3b7a7bdd566857fb89.
2023-10-25 15:05:33 +02:00
Flatcar Buildbot
51d6d222b7 dev-lang/lua: Sync with Gentoo
It's from Gentoo commit 7c4d1adc59340281050875aaf8cad24b0e5b3cf0.
2023-10-25 15:05:33 +02:00
Flatcar Buildbot
6a8975ff68 app-portage/gentoolkit: Sync with Gentoo
It's from Gentoo commit 122ee4d7e65a6a1ba60a508a99f4d93632bda5dd.
2023-10-25 15:05:33 +02:00
Flatcar Buildbot
5b21af3c87 app-emulation/qemu: Sync with Gentoo
It's from Gentoo commit 196685c68adecedeb3c335e551924cab4aee55ea.
2023-10-25 15:05:32 +02:00
Krzesimir Nowak
81457884fc run_sdk_container: Allow mounting custom volumes into SDK container 2023-10-25 14:51:51 +02:00
Krzesimir Nowak
6b326744d8 sdk_lib,run_sdk_container: Modernize a bit
- Make cosmetic fixes in help output.

- There is usually no need for putting variables inside quotes in
  assignments.

- Use [[ ]] to avoid putting everything into strings.

- Use arrays instead of relying on strings to be split on whitespace
  as it was the case for invoking docker and getting GPG volume flags
  for docker.

- Make sure that some cleanup and trap strings quote variables
  properly.

- Add a "call_docker" function to avoid dealing with "docker" and a
  new "docker_a" variables when willing to invoke docker. The "docker"
  variable rather shouldn't be used, but it is still there in case
  some other scripts were using it.
2023-10-25 14:49:35 +02:00
flatcar-ci
5ec3b7bd96 New version: main-3767.0.0-nightly-20231024-2100 2023-10-24 21:00:32 +00:00
Thilo Fromm
ac811ab50f
Merge pull request #1216 from flatcar/contrib/torcx-deprecation-docker-sysext
Contrib: Deprecate torcx, ship containerd / docker as sysexts
2023-10-24 09:33:44 +02:00
flatcar-ci
4deb3ba157 New version: main-3766.0.0-nightly-20231023-2100 2023-10-23 21:00:25 +00:00
Thilo Fromm
f81bbebddf build_sysext: run package inventorisation before mangle script
Signed-off-by: Thilo Fromm <thilofromm@microsoft.com>
2023-10-23 16:05:46 +02:00
Thilo Fromm
65bf2bfd0c Update run_local_tests.sh: improve quoting
Co-authored-by: Krzesimir Nowak <knowak@microsoft.com>
2023-10-23 16:05:46 +02:00
Thilo Fromm
6dd2f203f1 run_local_tests.sh: un-break variable quoting
Turns out using ${var@Q} instead of ${var} ends up with paths like
/work/foo/'amd64'-usr/...
instead of
/work/foo/amd64-usr/...
which breaks the script. So we revert it.

Signed-off-by: Thilo Fromm <thilofromm@microsoft.com>
2023-10-23 16:05:46 +02:00
Thilo Fromm
d744d32733 build_sysext: optimise squashfs for size
Signed-off-by: Thilo Fromm <thilofromm@microsoft.com>
2023-10-23 16:05:46 +02:00
Thilo Fromm
37cc05ce11 coreos/base/package.use: squashfs-tools: enable all compression formats
This change enables USE flags for all supported compression formats.
zstd specifically is required to build zstd sysexts.
2023-10-23 16:05:46 +02:00
Thilo Fromm
c2d1c13abd Apply bash scripting suggestions from code review
Thank you Krzesimir!

Co-authored-by: Krzesimir Nowak <knowak@microsoft.com>
2023-10-23 16:05:45 +02:00
Thilo Fromm
b775036a62 torcx removal: address PR comments
- updated github actions for runc, containerd, and docker to not handle
  nonexistent ebuilds in app-torcx/ anymore
- removed spurious package_run_dependencies from build_image_util.sh
- build_sysext: generate pkginfo before mangle script runs
  use zstd for compression; add cli flag to select compression
- ci_automation_common.sh: remove spurious `/` from match string
- coreos, board-packages, bootengine: bump ebuild revisions
- kernel commonconfig: add squashfs zstd support

Signed-off-by: Thilo Fromm <thilofromm@microsoft.com>
2023-10-23 16:05:45 +02:00
Thilo Fromm
57ed77ea91 ci-automation: fix local image detection for nightly SDKs
Signed-off-by: Thilo Fromm <thilofromm@microsoft.com>
2023-10-23 16:05:45 +02:00
Thilo Fromm
22582dca7c changelog: add torcx removal, sysext migration
Signed-off-by: Thilo Fromm <thilofromm@microsoft.com>
2023-10-23 16:05:45 +02:00
Thilo Fromm
6512ef04ca overlay coreos-base/update_engine: remove torcx sanity check
Include PR https://github.com/flatcar/update_engine/pull/30 to un-break
updates when torcx was removed in favour of sysext.

Signed-off-by: Thilo Fromm <thilofromm@microsoft.com>
2023-10-23 16:05:45 +02:00
Thilo Fromm
428503083f run_sdk_container: Add -U flag to use the version file unchanged
This change adds a -U flag to run_sdk_container. If provided, the script
will not regenerate version.txt but instead use the existing file as-is.

Signed-off-by: Thilo Fromm <thilofromm@microsoft.com>
2023-10-23 16:05:45 +02:00
Thilo Fromm
919d368688 mantle-container: bump image ref to support torcx removal
This change bumps the image ref of the mantle container to
ghcr.io/flatcar/mantle:git-20a2f8ffee8c8a1a042b1da99f0f59312110f285.
This version includes 2 PRs (https://github.com/flatcar/mantle/pull/465
and https://github.com/flatcar/mantle/pull/466) which add support for
sysext docker / torcx removal in the OS image.

Signed-off-by: Thilo Fromm <thilofromm@microsoft.com>
2023-10-23 16:05:45 +02:00
Thilo Fromm
8ef820ad20 vendor-testing/qemu_update.sh: make update payload configurable
This change makes QEMU_UPDATE_PAYLOAD configurable via
ci-automation/settings.env where it was hard-wired before.

The change also fixes fall-out in qemu_update.sh by ensuring a local tmp
directory is created before it is used by the test.

Signed-off-by: Thilo Fromm <thilofromm@microsoft.com>
2023-10-23 16:05:45 +02:00
Thilo Fromm
bd54e3a0ce .github: removed torcx artifact handling from github CI
Signed-off-by: Thilo Fromm <thilofromm@microsoft.com>
2023-10-23 16:05:45 +02:00
Thilo Fromm
35779f1ca3 run_local_tests.sh: script for running tests on local builds
This change adds run_local_tests.sh, a script to run tests on local
builds. It's a comfort wrapper around ci-automation scripts and uses
the latest local build.

Signed-off-by: Thilo Fromm <thilofromm@microsoft.com>
2023-10-23 16:05:45 +02:00
Thilo Fromm
6f651a298c build scripts: separate txt inventory; official naming for sysexts
Signed-off-by: Thilo Fromm <thilofromm@microsoft.com>
2023-10-23 16:05:45 +02:00
Thilo Fromm
e85ca4984c ci-automation: remove torcx artifact handling and tests 2023-10-23 16:05:45 +02:00
Thilo Fromm
89555ed2bd base OS sysexts: separate build script, inventory generation
This change refactors base OS sysext builds to use a separate build
script `build_library/sysext_prod_builder`, which is called from
`build_library/prod_image_util.sh` when `build_image` runs.

This allows for better separation of cleanup traps: prod image sysext
builds need its own trap / cleanup function for temporary build
directories and loopback mounts.

Prod sysext builds properly generate lincense and SBOM information, and
provide detailed file listings and disk space usage stats.

- SBOM / licenses JSON now include all packages of the
  final image, i.e. a combined list of base image and all base OS
  sysexts.
- Packages lists, files list and detailed files list include the sysext
  squashfs files for the base image, and separate sections with files /
  packages lists for each sysext.
- Disk usage contains both final disk image usage as well as usage of
  each individual sysext squashfs.
2023-10-23 16:05:45 +02:00
Thilo Fromm
c589fb8e56 coreos-overlay: removed all remaining app-torcx artifacts
Signed-off-by: Thilo Fromm <thilofromm@microsoft.com>
2023-10-23 16:05:45 +02:00
Thilo Fromm
8e01a2c8f9 build_image: add sysext command line option
This change refactors sysext builds during build_image and generalises
the code (no hard-coded containerd and docker anymore).

A command line option is added to build_image for sysexts to include in
the OS image. It defaults to containerd and docker but may be set to
arbitrary packages. The command line supports simple depenencies, i.e.
the "docker" sysext will re-use package information from the
"containerd" sysext and not include another containerd.

Signed-off-by: Thilo Fromm <thilofromm@microsoft.com>
2023-10-23 16:05:45 +02:00
Thilo Fromm
8f8f262f19 torcx: remove from scripts, use docker+containerd sysexts
This change removes torcx libraries, references, and commandline options
from build automation scripts and from build_library/.

Containerd and docker are shipped via sysexts which are included in the
base image.

Signed-off-by: Thilo Fromm <thilofromm@microsoft.com>
2023-10-23 16:05:45 +02:00
Thilo Fromm
aae3fd8260 prod_image_util.sh: remove pkgdb tarball (replaced by sysext)
Signed-off-by: Thilo Fromm <thilofromm@microsoft.com>
2023-10-23 16:05:45 +02:00
Thilo Fromm
9837c35ec4 build_sysext: add sysext dependency handling
This change adds 2 optional command line parameters to build_sysext to
handle dependencies in stacked sysexts. The command line parameters
allow exporting portage package db information into a separate squashfs
image as well as using package db information exported by a previous
sysext build.

--generate_pkginfo will generate a separate squashfs
   <sysextname>_pkginfo.raw which contains the sysext's /var/db/pkgs.

--base_pkginfo=<pkginfo>[:<pkginfo>[:...]] will use the supplied paths
   to pkginfo squashfses as additional lowerdirs when creating a sysext.

Useage example:
./build_sysext --generate_pkginfo containerd app-containers/containerd
./build_sysext --base_pkginfo=containerd_pkginfo.raw docker app-containers/docker

will create a containerd.raw sysext and a dependent docker.raw sysext
that does not have the containerd dependency installed. Both sysexts
must be merged together in order for docker to work.

Signed-off-by: Thilo Fromm <thilofromm@microsoft.com>
2023-10-23 16:05:45 +02:00
Thilo Fromm
9cede02f86 dev_container_util.sh: remove spurious DEVCONTAINER=1
Signed-off-by: Thilo Fromm <thilofromm@microsoft.com>
2023-10-23 16:05:45 +02:00
Krish Jain
cc029bb5ca use DEVCONTAINER=1 2023-10-23 16:05:45 +02:00
Krish Jain
3c4639b0ce use sudo_clobber function 2023-10-23 16:05:45 +02:00
Krish Jain
298e283944 fix permissions issue 2023-10-23 16:05:45 +02:00
Krish Jain
4633cc3e7c Fix syntax errors, quoting 2023-10-23 16:05:45 +02:00
Krish Jain
f6f5c32417 add information about the packages contained in the built sysext 2023-10-23 16:05:45 +02:00
Krish Jain
fe949f5900 Make sure binary packages are built in build_packages already 2023-10-23 16:05:45 +02:00
Krish Jain
4e8b276319 Remove duplicate making directory root_fs_dir/etc/extensions 2023-10-23 16:05:45 +02:00
Krish Jain
abe8bf683a Use helper function query_available_package() 2023-10-23 16:05:45 +02:00
Krish Jain
4b00bc14a4 tail, not head 2023-10-23 16:05:45 +02:00
Krish Jain
9709a64239 incorporate suggestions from Jeremi 2023-10-23 16:05:45 +02:00
Krish Jain
e6bb5470c4 Add mangle script for containerd 2023-10-23 16:05:45 +02:00
Krish Jain
e5f1ecbae3 Only keep docker stuff in manglefs_docker 2023-10-23 16:05:45 +02:00
Krish Jain
2be2e884b3 Add manglefs script for containerd 2023-10-23 16:05:45 +02:00
Krish Jain
3d17f65088 Create seperate docker and containerd sysext 2023-10-23 16:05:45 +02:00
krishjainx
c026d9ed53 Remove app-arch/torcx from coreos-overlay 2023-10-23 16:05:45 +02:00
krishjainx
851009c6cc Install docker as a system extension, use manglefs script ( convert the unit dependencies from wants to upholds); one sysext for all docker for now 2023-10-23 16:05:45 +02:00