Move and rename repos, upgrade to Catalyst 4, support SDK on arm64

Sorry for doing all this in one giant commit, but it was hard to
separate it out.

We had no arm64 SDK, so some cross-compiling or emulation was most
likely going to be needed to produce one. Catalyst 4 adds support for
building with QEMU, so I looked into upgrading. This turned out to be
very much slower than emulating the amd64 SDK on arm64, where an arm64
build could then be mostly run without emulation. We can't stay on
Catalyst 3 forever though, so I continued with the upgrade.

Despite being slow, I have kept support for building with QEMU using
Catalyst since it requires little code and may be useful to somebody.

Catalyst 4 has totally changed the way repositories are handled. It only
works when the name of the directory containing the repository matches
the configured name of that repository. This was not the case for us,
with the coreos repository residing in the coreos-overlay directory. We
wanted to move and rename our repositories anyway, so they are now known
as gentoo-subset and flatcar-overlay, and they live under scripts/repos.
Using the same name as upstream Gentoo would have been problematic, and
just "flatcar" would have looked awkward in documentation.

Catalyst 4 also ingests the main repository snapshot as a squashfs
rather than a tarball. It features a utility to generate such a
snapshot, but it doesn't fit Flatcar well, particularly because it
expects each ebuild repository to reside at the top level of its own git
repository. It was very easy to call tar2sqfs manually though.

There were several places where we assumed that amd64 was native and
arm64 required emulation via QEMU. The scripts are now more
architecture-agnostic, paving the way for riscv support later.

We no longer set QEMU_LD_PREFIX because it prevents the SDK itself from
being emulated. It also assumes there is only one non-native target,
which won't be the case soon. bubblewrap does a better job of running
binaries under QEMU.

Signed-off-by: James Le Cuirot <jlecuirot@microsoft.com>

.github/workflows/common.sh

@@ -14,9 +14,8 @@ if [[ ! -d "${WORK_SCRIPTS_DIR:-}" ]]; then
 fi
 
 readonly SDK_OUTER_TOPDIR="${WORK_SCRIPTS_DIR}"
-readonly SDK_OUTER_OVERLAY="${SDK_OUTER_TOPDIR}/sdk_container/src/third_party/coreos-overlay"
-readonly SDK_INNER_SRCDIR="/mnt/host/source/src"
-readonly SDK_INNER_OVERLAY="${SDK_INNER_SRCDIR}/third_party/coreos-overlay"
+readonly SDK_OUTER_OVERLAY="${SDK_OUTER_TOPDIR}/repos/flatcar-overlay"
+readonly SDK_INNER_OVERLAY="/mnt/host/source/src/scripts/repos/flatcar-overlay"
 
 readonly BUILDBOT_USERNAME="Flatcar Buildbot"
 readonly BUILDBOT_USEREMAIL="buildbot@flatcar-linux.org"

.github/workflows/image_changes.sh

@@ -25,8 +25,8 @@ function github_ricj_callback() {
     show_changes_env+=(
         # Override the default locations of repositories.
         "SCRIPTS_REPO=."
-        "COREOS_OVERLAY_REPO=../coreos-overlay"
-        "PORTAGE_STABLE_REPO=../portage-stable"
+        "FLATCAR_OVERLAY_REPO=../flatcar-overlay"
+        "GENTOO_SUBSET_REPO=../gentoo-subset"
     )
     show_changes_params+=(
         # We may not have a tag handy, so we tell show-changes

.github/workflows/update-gentoo-subset-packages-from-list.yaml

@@ -1,4 +1,4 @@
-name: Keep portage-stable packages updated
+name: Keep gentoo-subset packages updated
 on:
   schedule:
     - cron:  '0 7 * * 1'
@@ -35,10 +35,10 @@ jobs:
           git config --global user.name "Flatcar Buildbot"
           git config --global user.email "buildbot@flatcar-linux.org"
           old_head=$(git -C scripts rev-parse HEAD)
-          packages_list=$(realpath scripts/.github/workflows/portage-stable-packages-list)
+          packages_list=$(realpath scripts/.github/workflows/gentoo-subset-packages-list)
           gentoo_repo=$(realpath gentoo)
           build_scripts=$(realpath flatcar-build-scripts)
-          pushd scripts/sdk_container/src/third_party/portage-stable
+          pushd scripts/repos/gentoo-subset
           while read -r package; do
               if [[ ! -e "${package}" ]]; then
                   # If this happens, it means that the package was moved to overlay
@@ -50,7 +50,7 @@ jobs:
                   # If this happens, it means that the package was obsoleted or moved
                   # in Gentoo. The obsoletion needs to be handled in the case-by-case
                   # manner, while move should be handled by doing the same move
-                  # in portage-stable. The build should not break because of the move,
+                  # in gentoo-subset. The build should not break because of the move,
                   # because most likely it's already reflected in the profiles/updates
                   # directory.
                   echo "::warning title=${package}::Obsolete or moved package"
@@ -73,10 +73,10 @@ jobs:
         with:
           token: ${{ secrets.GITHUB_TOKEN }}
           path: scripts
-          branch: buildbot/weekly-portage-stable-package-updates-${{steps.update-listed-packages.outputs.TODAYDATE }}
+          branch: buildbot/weekly-gentoo-subset-package-updates-${{steps.update-listed-packages.outputs.TODAYDATE }}
           delete-branch: true
           base: main
-          title: Weekly portage-stable package updates ${{steps.update-listed-packages.outputs.TODAYDATE }}
+          title: Weekly gentoo-subset package updates ${{steps.update-listed-packages.outputs.TODAYDATE }}
           body: |
             CI: TODO
 

.github/workflows/update-metadata-glsa.yaml

@@ -15,7 +15,7 @@ jobs:
       - name: Update GLSA metadata
         id: update-glsa-metadata
         run: |
-          glsa_dir='sdk_container/src/third_party/portage-stable/metadata/glsa'
+          glsa_dir='repos/gentoo-subset/metadata/glsa'
           rm -rf "${glsa_dir}"
           mkdir --parents "${glsa_dir}"
           rsync --archive rsync://rsync.gentoo.org/gentoo-portage/metadata/glsa/* "${glsa_dir}"
@@ -30,6 +30,6 @@ jobs:
           base: main
           title: Monthly GLSA metadata ${{steps.update-glsa-metadata.outputs.TODAYDATE }}
           body: Updated GLSA metadata
-          commit-message: "portage-stable/metadata: Monthly GLSA metadata updates"
+          commit-message: "gentoo-subset/metadata: Monthly GLSA metadata updates"
           author: Flatcar Buildbot <buildbot@flatcar-linux.org>
           labels: main

PREFIX.md

@@ -35,7 +35,7 @@ Prefix uses a _staging environment_ to build binary packages, then installs thes
 The _staging environment_ contains toolchains and all build tools required to create binary packages (a full `@system`).
 The _final environment_ only contains run-time dependencies.
 
-Packages are built from ebuilds in coreos-overlay, portage-stable, and prefix-overlay.
+Packages are built from ebuilds in flatcar-overlay, gentoo-subset, and prefix-overlay.
 
 A QoL `emerge` wrapper is included to install packages to the prefix.
 

README.md

@@ -14,20 +14,18 @@ The SDK can be used to
 # Using the scripts repository
 
 The repository is meant to be the entry point for Flatcar builds and development.
-Ebuilds for all packages reside in one of 2 subdirectories - [coreos-overlay](sdk_container/src/third_party/coreos-overlay) and [portage-stable](sdk_container/src/third_party/portage-stable/):
+Ebuilds for all packages reside in one of 2 subdirectories - [flatcar-overlay](repos/flatcar-overlay) and [gentoo-subset](repos/gentoo-subset/):
 ```
 scripts
-   +--sdk_container
-          +---------src
-                     +--third_party
-                             +------coreos-overlay
-                             +------portage-stable
+   +--repos
+          +--flatcar-overlay
+          +--gentoo-subset
 ```
 
-`portage-stable` is kept in alignment with upstream Gentoo and should not contain any modifications (with only minor, well-justified exceptions).
+`gentoo-subset` is kept in alignment with upstream Gentoo and should not contain any modifications (with only minor, well-justified exceptions).
 Consider it a small sub-set of Gentoo.
 
-`coreos-overlay` contains significantly modified or even entirely self-written ebuilds.
+`flatcar-overlay` contains significantly modified or even entirely self-written ebuilds.
 
 The `scripts` repository makes ample use of tags to mark releases.
 Sometimes, local and origin tags can diverge (e.g. when re-tagging something locally to test a build).
@@ -52,7 +50,7 @@ While work on a native ARM64 native SDK is ongoing, it's unfortunately not ready
 
 The container can be run in one of two ways - "standalone", or integrated with the [scripts](https://github.com/flatcar/scripts) repo:
 * Standalone mode will use no host volumes and will allow you to play with the SDK in a sandboxed throw-away environment. In standalone mode, you interface with Docker directly to use the SDK container.
-* Integrated mode will closely integrate with the scripts repo directory and bind-mount it as well as the portage-stable and coreos-overlay directories into the container. Integrated mode uses wrapper scripts to interact with the SDK container. This is the recommended way for developing patches for Flatcar.
+* Integrated mode will closely integrate with the scripts repo directory and bind-mount it as well as the gentoo-subset and flatcar-overlay directories into the container. Integrated mode uses wrapper scripts to interact with the SDK container. This is the recommended way for developing patches for Flatcar.
 
 ## Standalone mode
 
@@ -82,7 +80,7 @@ To start a container in privileged mode with `/dev` available use:
 
 This is the preferred mode of working with the SDK.
 Interaction with the container happens via wrapper scripts from the scripts repository.
-Both the host's scripts repo as well as the ebuild paths (portage-stable and coreos-overlay) are made available in the container, allowing for work on these directly.
+Both the host's scripts repo as well as the ebuild paths (gentoo-subset and flatcar-overlay) are made available in the container, allowing for work on these directly.
 The wrapper scripts will re-use existing containers instead of creating new ones to preserve your work in the container, enabling consistency.
 
 To clone the scripts repo and pick a version:

bootstrap_sdk

@@ -7,11 +7,11 @@
 # This uses Gentoo's catalyst for very thoroughly building images from
 # scratch. Using images based on this will eliminate some of the hackery
 # in make_chroot.sh for building up the sdk from a stock stage3 tarball.
-# 
+#
 # For reference the procedure it performs is this:
-# 
-# 1. snapshot: Grab a snapshot of the portage-stable repo from
-#     the current SDK's /var/lib/gentoo/repos/gentoo.
+#
+# 1. snapshot: Grab a snapshot of the gentoo-subset repo from
+#     the current SDK's /var/gentoo/repos/gentoo-subset.
 #     Alternatively, check out a git ref specified via --portage-ref.
 #
 # 2. stage1: Using a "seed" tarball as a build environment, build a
@@ -22,20 +22,20 @@
 #      will be linked against the SEED SDK libraries, NOT against libraries
 #      built in stage 1. See "stage_repo()" documentation further below for more.
 #     This stage uses:
-#     - portage-stable from the SDK's /var/lib/gentoo/repos/gentoo
+#     - gentoo-subset from the SDK's /var/gentoo/repos/gentoo-subset
 #        or a custom path via --stage1_portage_path command line option
-#     - coreos-overlay  from the SDK's /var/lib/gentoo/repos/coreos-overlay
+#     - flatcar-overlay  from the SDK's /var/gentoo/repos/flatcar-overlay
 #        or a custom path via --stage1_overlay_path command line option
 #     Command line option refs need caution though, since
 #     stage1 must not contain updated ebuilds (see build_stage1 below).
 #
-# 3. stage2: Run portage-stable/scripts/bootstrap.sh
+# 3. stage2: Run gentoo-subset/scripts/bootstrap.sh
 #     This rebuilds the toolchain using Gentoo bootstrapping, ensuring it's not linked
 #     to or otherwise influenced by whatever was in the "seed" tarball.
 #     The toolchain rebuild may contain updated package ebuilds from
-#     third_party/(portage-stable|coreos-overlay).
-#     This and all following stages use portage-stable and coreos-overlay
-#     from third_party/... (see 1.)
+#     repos/(gentoo-subset|flatcar-overlay).
+#     This and all following stages use gentoo-subset and flatcar-overlay
+#     from repos/... (see 1.)
 #
 # 4. stage3: Run emerge -e system to rebuild everything using the fresh updated
 #     toolchain from 3., using the normal USE flags provided by the profile. This
@@ -68,15 +68,14 @@ DEFINE_string stage1_overlay_path "" \
 ## Define the stage4 config template
 catalyst_stage4() {
 cat <<EOF
-target: stage4
 pkgcache_path: $BINPKGS
 stage4/packages: coreos-devel/sdk-depends
 stage4/fsscript: ${BUILD_LIBRARY_DIR}/catalyst_sdk.sh
 stage4/root_overlay: ${ROOT_OVERLAY}
-stage4/empty: /etc/portage/repos.conf /root /usr/portage /var/cache/edb
+stage4/empty: /root /var/cache/edb
 stage4/rm: /etc/machine-id /etc/resolv.conf
 EOF
-catalyst_stage_default
+catalyst_stage_default 4
 }
 
 # Switch to HTTP because early boostrap stages do not have SSL support.
@@ -113,12 +112,12 @@ chmod 1777 "${ROOT_OVERLAY}/tmp"
 cp "${BUILD_LIBRARY_DIR}/toolchain_util.sh" "${ROOT_OVERLAY}/tmp"
 
 
-# Stage 1 uses "known-good" ebuilds (from both coreos-overlay and portage-stable)
+# Stage 1 uses "known-good" ebuilds (from both flatcar-overlay and gentoo-subset)
 #  to build a minimal toolchain (USE="-*") for stage 2.
 #
-# No package updates must happen in stage 1, so we use the portage-stable and
-#  coreos-overlay paths included with the current SDK (from the SDK chroot's
-#  /var/lib/gentoo/repos/). "Current SDK" refers to the SDK we entered with
+# No package updates must happen in stage 1, so we use the gentoo-subset and
+#  flatcar-overlay paths included with the current SDK (from the SDK chroot's
+#  /var/gentoo/repos/). "Current SDK" refers to the SDK we entered with
 #  'cork enter', i.e. the SDK we run ./bootstrap_sdk in.
 #
 # Using ebuilds from the above mentioned sources will ensure that stage 1 builds
@@ -148,43 +147,38 @@ stage_repo() {
     local path=${2}
     local dest=${3}
     local update_seed_file=${4}
-    local gitname="$repo"
-
-    if [ "$gitname" = "gentoo" ] ; then
-        gitname="portage-stable"
-    fi
 
     if [ -z "$path" ]; then
-        cp -R "/var/gentoo/repos/${repo}" "$dest"
-        info "Using local SDK's ebuild repo '$repo' ('$gitname') in stage 1."
+        cp -R --reflink=auto "/var/gentoo/repos/${repo}" "$dest"
+        info "Using local SDK's ebuild repo '$repo' in stage 1."
     else
         mkdir "$dest/$repo"
-        cp -R "${path}/"* "$dest/${repo}/"
-        info "Using custom path '$path' for ebuild repo '$repo' ('$gitname') in stage 1."
+        cp -R --reflink=auto "${path}/"* "$dest/${repo}/"
+        info "Using custom path '$path' for ebuild repo '$repo' in stage 1."
         info "This may break stage 2. YOU HAVE BEEN WARNED. You break it, you keep it."
     fi
     (
         set -euo pipefail
         local repo_var hook name
 
-        # FLAGS_coreos_overlay for gitname coreos-overlay
-        repo_var="FLAGS_${gitname//-/_}"
+        # FLAGS_flatcar_overlay for flatcar-overlay
+        repo_var="FLAGS_${repo//-/_}"
         shopt -s nullglob
-        for hook in "${FLAGS_coreos_overlay}/coreos/stage1_hooks/"*"-${gitname}.sh"; do
+        for hook in "${FLAGS_flatcar_overlay}/coreos/stage1_hooks/"*"-${repo}.sh"; do
             name=${hook##*/}
-            name=${name%"-${gitname}.sh"}
-            info "Invoking stage1 ${gitname} hook ${name} on ${dest}/${repo}"
+            name=${name%"-${repo}.sh"}
+            info "Invoking stage1 ${repo} hook ${name} on ${dest}/${repo}"
             "${hook}" "${dest}/${repo}" "${!repo_var}" "${update_seed_file}"
         done
     )
 }
 
 build_stage1() {
-    # First, write out the default 4-stage catalyst configuration files
+    # First, write out the default catalyst configuration files
     write_configs
 
-    # Prepare local copies of both the "known-good" portage-stable and the
-    #  "known-good" coreos-overlay ebuild repos
+    # Prepare local copies of both the "known-good" gentoo-subset and the
+    #  "known-good" flatcar-overlay ebuild repos
     local stage1_repos="$TEMPDIR/stage1-ebuild-repos"
     info "Creating stage 1 ebuild repos and stage 1 snapshot in '$stage1_repos'"
     rm -rf "$stage1_repos"
@@ -196,22 +190,16 @@ build_stage1() {
 
     # prepare ebuild repos for stage 1, either from the local SDK (default)
     #  or from custom paths specified via command line flags
-    stage_repo "gentoo" "${FLAGS_stage1_portage_path}" "$stage1_repos" "${update_seed_file}"
-    stage_repo "coreos-overlay" "${FLAGS_stage1_overlay_path}" "$stage1_repos" "${update_seed_file}"
+    stage_repo "gentoo-subset" "${FLAGS_stage1_portage_path}" "$stage1_repos" "${update_seed_file}"
+    stage_repo "flatcar-overlay" "${FLAGS_stage1_overlay_path}" "$stage1_repos" "${update_seed_file}"
 
-    # Create a snapshot of "known-good" portage-stable repo copy for use in stage 1
-    #  This requires us to create a custom catalyst config to point it to the
-    #  repo copy we just created, for snapshotting.
-    catalyst_conf > "$TEMPDIR/catalyst-stage1.conf"
-    sed -i "s:^portdir.*:portdir=\"$stage1_repos/gentoo\":" \
-         "$TEMPDIR/catalyst-stage1.conf"
-    # take the "portage directory" (portage-stable copy) snapshot
-    build_snapshot "${TEMPDIR}/catalyst-stage1.conf" "${FLAGS_version}-stage1"
+    # take the "portage directory" (gentoo-subset copy) snapshot
+    build_snapshot "$stage1_repos/gentoo-subset" "${FLAGS_version}-stage1"
 
-    # Update the stage 1 spec to use the "known-good" portage-stable snapshot
-    #  and coreos-overlay copy repository versions from above.
-    sed -i -e "s/^snapshot:.*/snapshot: $FLAGS_version-stage1/" \
-           -e "s,^portage_overlay:.*,portage_overlay: $stage1_repos/coreos-overlay," \
+    # Update the stage 1 spec to use the "known-good" gentoo-subset snapshot
+    #  and flatcar-overlay copy repository versions from above.
+    sed -i -e "s/^snapshot_treeish:.*/snapshot_treeish: $FLAGS_version-stage1/" \
+           -e "s,^repos:.*,repos: $stage1_repos/flatcar-overlay," \
         "$TEMPDIR/stage1.spec"
 
     # If we are to use a custom path for either ebuild repo we want to update the stage1 seed SDK
@@ -220,10 +208,11 @@ build_stage1() {
         echo "update_seed_command: --update --deep --newuse --complete-graph --rebuild-if-new-ver --rebuild-exclude cross-*-cros-linux-gnu/* sys-devel/gcc " \
             >>"$TEMPDIR/stage1.spec"
     fi
+
     rm -f "${update_seed_file}"
 
     # Finally, build stage 1
-    build_stage stage1 "$SEED" "$TEMPDIR/catalyst-stage1.conf"
+    build_stage stage1
 }
 
 if [[ "$STAGES" =~ stage1 ]]; then

build_library/build_image_util.sh

@@ -134,32 +134,13 @@ generate_update() {
   upload_legacy_digests "${update}.DIGESTS" compressed_images
 }
 
-# ldconfig cannot generate caches for non-native arches.
-# Use qemu & the native ldconfig to work around that.
-# http://code.google.com/p/chromium/issues/detail?id=378377
 run_ldconfig() {
-  local root_fs_dir=$1
-  case ${ARCH} in
-  arm64)
-    sudo qemu-aarch64 "${root_fs_dir}"/usr/sbin/ldconfig -r "${root_fs_dir}";;
-  x86|amd64)
-    sudo ldconfig -r "${root_fs_dir}";;
-  *)
-    die "Unable to run ldconfig for ARCH ${ARCH}"
-  esac
+  # This wrapper is created by setup_board.
+  sudo "ldconfig-${BOARD}" -r "$1"
 }
 
 run_localedef() {
-  local root_fs_dir="$1" loader=()
-  case ${ARCH} in
-  arm64)
-    loader=( qemu-aarch64 -L "${root_fs_dir}" );;
-  amd64)
-    loader=( "${root_fs_dir}/usr/lib64/ld-linux-x86-64.so.2" \
-               --library-path "${root_fs_dir}/usr/lib64" );;
-  *)
-    die "Unable to run localedef for ARCH ${ARCH}";;
-  esac
+  local root_fs_dir="$1"
   info "Generating C.UTF-8 locale..."
   local i18n="${root_fs_dir}/usr/share/i18n"
   # localedef will silently fall back to /usr/share/i18n if missing so
@@ -167,8 +148,8 @@ run_localedef() {
   [[ -f "${i18n}/charmaps/UTF-8.gz" ]] || die
   [[ -f "${i18n}/locales/C" ]] || die
   sudo mkdir -p "${root_fs_dir}/usr/lib/locale"
-  sudo I18NPATH="${i18n}" "${loader[@]}" "${root_fs_dir}/usr/bin/localedef" \
-      --prefix="${root_fs_dir}" --charmap=UTF-8 --inputfile=C C.UTF-8
+  sudo I18NPATH="${i18n}" "bwrap-${BOARD}" "${root_fs_dir}" /usr/bin/localedef \
+      --charmap=UTF-8 --inputfile=C C.UTF-8
 }
 
 # Basic command to emerge binary packages into the target image.
@@ -365,7 +346,7 @@ get_metadata() {
             local mirror="$(echo "${v}" | grep mirror:// | cut -d '/' -f 3)"
             if [ -n "${mirror}" ]; then
                 # Take only first mirror, those not working should be removed
-                local location="$(grep "^${mirror}"$'\t' /var/gentoo/repos/gentoo/profiles/thirdpartymirrors | cut -d $'\t' -f 2- | cut -d ' ' -f 1 | tr -d $'\t')"
+                local location="$(grep "^${mirror}"$'\t' /var/gentoo/repos/gentoo-subset/profiles/thirdpartymirrors | cut -d $'\t' -f 2- | cut -d ' ' -f 1 | tr -d $'\t')"
                 v="$(echo "${v}" | sed "s#mirror://${mirror}/#${location}#g")"
             fi
             new_val+="${v} "
@@ -489,8 +470,8 @@ EOF
     local license_list # define before assignment because it would mask any error
     license_list="$(jq -r '.[] | "\(.licenses | .[])"' "${json_input}" | sort | uniq)"
     local license_dirs=(
-        "/mnt/host/source/src/third_party/coreos-overlay/licenses/"
-        "/mnt/host/source/src/third_party/portage-stable/"
+        "/mnt/host/source/src/scripts/repos/flatcar-overlay/licenses/"
+        "/mnt/host/source/src/scripts/repos/gentoo-subset/licenses/"
         "/var/gentoo/repos/gentoo/licenses/"
         "none"
     )

build_library/catalyst.sh

@@ -25,13 +25,14 @@ BINPKGS=
 DISTDIR=
 TEMPDIR=
 STAGES=
+unset QEMU
 
 DEFINE_string catalyst_root "${DEFAULT_CATALYST_ROOT}" \
     "Path to directory for all catalyst images and other files."
-DEFINE_string portage_stable "${SRC_ROOT}/third_party/portage-stable" \
-    "Path to the portage-stable git checkout."
-DEFINE_string coreos_overlay "${SRC_ROOT}/third_party/coreos-overlay" \
-    "Path to the coreos-overlay git checkout."
+DEFINE_string gentoo_subset "${SCRIPTS_DIR}/repos/gentoo-subset" \
+    "Path to the gentoo-subset git checkout."
+DEFINE_string flatcar_overlay "${SCRIPTS_DIR}/repos/flatcar-overlay" \
+    "Path to the flatcar-overlay git checkout."
 DEFINE_string seed_tarball "${DEFAULT_SEED}" \
     "Path to an existing stage tarball to start from."
 DEFINE_string version "${FLATCAR_VERSION}" \
@@ -55,17 +56,15 @@ DEFINE_boolean debug ${FLAGS_FALSE} "Enable verbose output from catalyst."
 catalyst_conf() {
 cat <<EOF
 # catalyst.conf
-contents="auto"
-digests="md5 sha1 sha512 whirlpool"
-hash_function="crc32"
-options="pkgcache"
+digests=["md5", "sha1", "sha512", "blake2b"]
+options=["pkgcache"]
 sharedir="/usr/share/catalyst"
 storedir="$CATALYST_ROOT"
 distdir="$DISTDIR"
 envscript="$TEMPDIR/catalystrc"
 port_logdir="$CATALYST_ROOT/log"
-portdir="$FLAGS_portage_stable"
-snapshot_cache="$CATALYST_ROOT/tmp/snapshot_cache"
+repo_basedir="/mnt/host/source/src/scripts/repos"
+repo_name="gentoo-subset"
 EOF
 }
 
@@ -82,61 +81,49 @@ export ac_cv_posix_semaphores_enabled=yes
 EOF
 }
 
-repos_conf() {
-cat <<EOF
-[DEFAULT]
-main-repo = portage-stable
-
-[coreos]
-location = /var/gentoo/repos/local
-
-[portage-stable]
-location = /var/gentoo/repos/gentoo
-EOF
-}
-
 # Common values for all stage spec files
 catalyst_stage_default() {
 cat <<EOF
+target: stage$1
 subarch: $ARCH
 rel_type: $TYPE
 portage_confdir: $TEMPDIR/portage
-portage_overlay: $FLAGS_coreos_overlay
+repos: $FLAGS_flatcar_overlay
+keep_repos: gentoo-subset flatcar-overlay
 profile: $FLAGS_profile
-snapshot: $FLAGS_version
+snapshot_treeish: $FLAGS_version
 version_stamp: $FLAGS_version
 cflags: -O2 -pipe
 cxxflags: -O2 -pipe
 ldflags: -Wl,-O2 -Wl,--as-needed
+source_subpath: ${SEED}
+${QEMU+interpreter: $(type -P "${QEMU}")}
 EOF
 }
 
 # Config values for each stage
 catalyst_stage1() {
 cat <<EOF
-target: stage1
 # stage1 packages aren't published, save in tmp
 pkgcache_path: ${TEMPDIR}/stage1-${ARCH}-packages
 update_seed: no
 EOF
-catalyst_stage_default
+catalyst_stage_default 1
 }
 
 catalyst_stage2() {
 cat <<EOF
-target: stage2
 # stage2 packages aren't published, save in tmp
 pkgcache_path: ${TEMPDIR}/stage2-${ARCH}-packages
 EOF
-catalyst_stage_default
+catalyst_stage_default 2
 }
 
 catalyst_stage3() {
 cat <<EOF
-target: stage3
 pkgcache_path: $BINPKGS
 EOF
-catalyst_stage_default
+catalyst_stage_default 3
 }
 
 catalyst_stage4() {
@@ -207,8 +194,8 @@ catalyst_init() {
     # so far so good, expand path to work with weird comparison code below
     FLAGS_seed_tarball=$(readlink -f "$FLAGS_seed_tarball")
 
-    if [[ ! "$FLAGS_seed_tarball" =~ .*\.tar\.bz2 ]]; then
-        die_notrace "Seed tarball doesn't end in .tar.bz2 :-/"
+    if [[ ! "$FLAGS_seed_tarball" =~ .\.tar\.(bz2|xz) ]]; then
+        die_notrace "Seed tarball doesn't end in .tar.bz2 or .tar.xz :-/"
     fi
 
     # catalyst is obnoxious and wants the $TYPE/stage3-$VERSION part of the
@@ -216,49 +203,54 @@ catalyst_init() {
     # directory under $TEMPDIR instead, aka the SEEDCACHE feature.)
     if [[ "$FLAGS_seed_tarball" =~ "$CATALYST_ROOT/builds/".* ]]; then
         SEED="${FLAGS_seed_tarball#$CATALYST_ROOT/builds/}"
-        SEED="${SEED%.tar.bz2}"
+        SEED="${SEED%.tar.*}"
     else
         mkdir -p "$CATALYST_ROOT/builds/seed"
         cp -n "$FLAGS_seed_tarball" "$CATALYST_ROOT/builds/seed"
         SEED="seed/${FLAGS_seed_tarball##*/}"
-        SEED="${SEED%.tar.bz2}"
+        SEED="${SEED%.tar.*}"
+    fi
+
+    # Emulate the build, if needed. Note the SDK itself may already be emulated,
+    # so check the requested arch against the kernel's real arch, not uname -m.
+    if [[ ${ARCH} != $(get_portage_arch "$(< /proc/sys/kernel/arch)") ]]; then
+        case "${ARCH}" in
+            amd64) QEMU=qemu-x86_64 ;;
+            arm64) QEMU=qemu-aarch64 ;;
+            riscv) QEMU=qemu-riscv64 ;;
+        esac
     fi
 }
 
 write_configs() {
     info "Creating output directories..."
-    mkdir -m 775 -p "$TEMPDIR/portage/repos.conf" "$DISTDIR"
+    mkdir -m 775 -p "$DISTDIR"
     chown portage:portage "$DISTDIR"
     info "Writing out catalyst configs..."
     info "    catalyst.conf"
     catalyst_conf > "$TEMPDIR/catalyst.conf"
     info "    catalystrc"
     catalystrc > "$TEMPDIR/catalystrc"
-    info "    portage/repos.conf/coreos.conf"
-    repos_conf > "$TEMPDIR/portage/repos.conf/coreos.conf"
     info "    stage1.spec"
     catalyst_stage1 > "$TEMPDIR/stage1.spec"
-    info "    stage2.spec"
-    catalyst_stage2 > "$TEMPDIR/stage2.spec"
-    info "    stage3.spec"
-    catalyst_stage3 > "$TEMPDIR/stage3.spec"
-    info "    stage4.spec"
-    catalyst_stage4 > "$TEMPDIR/stage4.spec"
-    info "Putting a symlink to user patches..."
-    ln -sfT '/var/gentoo/repos/local/coreos/user-patches' \
-        "$TEMPDIR/portage/patches"
+
+    info "Configuring Portage..."
+    cp -r "${BUILD_LIBRARY_DIR}"/portage/ "${TEMPDIR}/"
+
+    ln -sfT '/var/gentoo/repos/flatcar-overlay/coreos/user-patches' \
+        "${TEMPDIR}"/portage/patches
+
+    [[ -n ${QEMU} ]] ||
+        rm "${TEMPDIR}"/portage/package.env/qemu
 }
 
 build_stage() {
-    local stage srcpath catalyst_conf target_tarball
+    local stage catalyst_conf target_tarball
 
     stage="$1"
-    srcpath="$2"
-    catalyst_conf="$3"
+    catalyst_conf="$TEMPDIR/catalyst.conf"
     target_tarball="${stage}-${ARCH}-${FLAGS_version}.tar.bz2"
 
-    [ -z "$catalyst_conf" ] && catalyst_conf="$TEMPDIR/catalyst.conf"
-
     if [[ -f "$BUILDS/${target_tarball}" && $FLAGS_rebuild == $FLAGS_FALSE ]]
     then
         info "Skipping $stage, $target_tarball already exists."
@@ -270,8 +262,7 @@ build_stage() {
         "${DEBUG[@]}" \
         --verbose \
         --config "$TEMPDIR/catalyst.conf" \
-        --file "$TEMPDIR/${stage}.spec" \
-        --cli "source_subpath=$srcpath"
+        --file "$TEMPDIR/${stage}.spec"
     # Catalyst does not clean up after itself...
     rm -rf "$TEMPDIR/$stage-${ARCH}-${FLAGS_version}"
     ln -sf "$stage-${ARCH}-${FLAGS_version}.tar.bz2" \
@@ -280,46 +271,19 @@ build_stage() {
 }
 
 build_snapshot() {
-    local catalyst_conf snapshot snapshots_dir snapshot_base snapshot_path
+    local repo_dir snapshot snapshots_dir snapshot_path
 
-    catalyst_conf=${1:-"${TEMPDIR}/catalyst.conf"}
+    repo_dir=${1:-"${FLAGS_gentoo_subset}"}
     snapshot=${2:-"${FLAGS_version}"}
     snapshots_dir="${CATALYST_ROOT}/snapshots"
-    snapshot_base="${snapshots_dir}/gentoo-${snapshot}"
-    snapshot_path="${snapshot_base}.tar.bz2"
-    if [[ -f "${snapshot_path}" && $FLAGS_rebuild == $FLAGS_FALSE ]]
+    snapshot_path="${snapshots_dir}/gentoo-subset-${snapshot}.sqfs"
+    if [[ -f ${snapshot_path} && $FLAGS_rebuild == $FLAGS_FALSE ]]
     then
         info "Skipping snapshot, ${snapshot_path} exists"
     else
         info "Creating snapshot ${snapshot_path}"
-        catalyst \
-            "${DEBUG[@]}" \
-            --verbose \
-            --config "${catalyst_conf}" \
-            --snapshot "${snapshot}"
-    fi
-    local f
-    local to_remove=()
-    # This will expand to at least our just built snapshot tarball, so
-    # no nullglob is needed here.
-    for f in "${snapshot_base}".*; do
-        case "${f}" in
-            "${snapshot_path}")
-                # Our snapshot, keep it as is.
-                :
-                ;;
-            *.CONTENTS|*.CONTENTS.gz|*.DIGESTS)
-                # These can stay, catalyst is not bothered by those.
-                :
-                ;;
-            *)
-                to_remove+=("${f}")
-                ;;
-        esac
-    done
-    if [[ ${#to_remove[@]} -gt 0 ]]; then
-        info "$(printf '%s\n' 'Found spurious files in snapshots directory that may confuse Catalyst, removing them:' "${to_remove[@]}")"
-        rm -rf "${to_remove[@]}"
+        mkdir -p "${snapshot_path%/*}"
+        tar -c -C "${repo_dir}" . | tar2sqfs "${snapshot_path}" -q -f -j1 -c gzip
     fi
 }
 
@@ -335,7 +299,7 @@ catalyst_build() {
 
     used_seed=0
     if [[ "$STAGES" =~ stage1 ]]; then
-        build_stage stage1 "$SEED"
+        build_stage stage1
         used_seed=1
     fi
 
@@ -343,7 +307,9 @@ catalyst_build() {
         if [[ $used_seed -eq 1 ]]; then
             SEED="${TYPE}/stage1-${ARCH}-latest"
         fi
-        build_stage stage2 "$SEED"
+        info "    stage2.spec"
+        catalyst_stage2 > "$TEMPDIR/stage2.spec"
+        build_stage stage2
         used_seed=1
     fi
 
@@ -351,7 +317,9 @@ catalyst_build() {
         if [[ $used_seed -eq 1 ]]; then
             SEED="${TYPE}/stage2-${ARCH}-latest"
         fi
-        build_stage stage3 "$SEED"
+        info "    stage3.spec"
+        catalyst_stage3 > "$TEMPDIR/stage3.spec"
+        build_stage stage3
         used_seed=1
     fi
 
@@ -359,10 +327,12 @@ catalyst_build() {
         if [[ $used_seed -eq 1 ]]; then
             SEED="${TYPE}/stage3-${ARCH}-latest"
         fi
-        build_stage stage4 "$SEED"
+        info "    stage4.spec"
+        catalyst_stage4 > "$TEMPDIR/stage4.spec"
+        build_stage stage4
         used_seed=1
     fi
 
     # Cleanup snapshots, we don't use them
-    rm -rf "$CATALYST_ROOT/snapshots/gentoo-${FLAGS_version}.tar.bz2"*
+    rm -rf "$CATALYST_ROOT/snapshots/${FLAGS_gentoo_subset##*/}-${FLAGS_version}.sqfs"*
 }

build_library/catalyst_sdk.sh

@@ -11,7 +11,7 @@ echo "Setting the default Python interpreter"
 eselect python update
 
 echo "Building cross toolchain for the SDK."
-configure_crossdev_overlay / /tmp/crossdev
+configure_crossdev_overlay / /usr/local/portage/crossdev
 
 for cross_chost in $(get_chost_list); do
     echo "Building cross toolchain for ${cross_chost}"
@@ -21,10 +21,10 @@ for cross_chost in $(get_chost_list); do
         install_cross_rust "${cross_chost}" ${clst_myemergeopts}
 done
 
-echo "Saving snapshot of coreos-overlay repo for future SDK bootstraps"
-# Copy coreos-overlay, which is in /var/gentoo/repos/local/, into a
-# local directory.  /var/gentoo/repos/local/ is removed before archiving
-# and we want to keep a snapshot. This snapshot is used - alongside
-# /var/gentoo/repos/gentoo - by stage 1 of future bootstraps.
-mkdir -p /var/gentoo/repos/coreos-overlay
-cp -R /var/gentoo/repos/local/* /var/gentoo/repos/coreos-overlay
+echo "Saving snapshot of repos for future SDK bootstraps"
+# Copy gentoo-subset and flatcar-overlay, which are under
+# /mnt/host/source/src/scripts/repos, into a local directory because they are
+# removed before archiving and we want to keep snapshots. These snapshots are
+# used by stage 1 in future bootstraps.
+mkdir -p /var/gentoo/repos
+cp -R /mnt/host/source/src/scripts/repos/{gentoo-subset,flatcar-overlay}/ /var/gentoo/repos/

build_library/catalyst_toolchains.sh

@@ -37,7 +37,7 @@ build_target_toolchain() {
         run_merge -u --root="$ROOT" --sysroot="$ROOT" "${TOOLCHAIN_PKGS[@]}"
 }
 
-configure_crossdev_overlay / /tmp/crossdev
+configure_crossdev_overlay / /usr/local/portage/crossdev
 
 for board in $(get_board_list); do
     echo "Building native toolchain for ${board}"

build_library/dev_container_util.sh

@@ -20,11 +20,11 @@ configure_dev_portage() {
 
     # Need profiles at the bare minimum
     local repo
-    for repo in portage-stable coreos-overlay; do
+    for repo in gentoo-subset flatcar-overlay; do
         sudo mkdir -p "${root_fs_dir}/var/lib/portage/${repo}"
         sudo rsync -rtl --exclude=md5-cache \
-            "${SRC_ROOT}/third_party/${repo}/metadata" \
-            "${SRC_ROOT}/third_party/${repo}/profiles" \
+            "${SRC_ROOT}/scripts/repos/${repo}/metadata" \
+            "${SRC_ROOT}/scripts/repos/${repo}/profiles" \
             "${root_fs_dir}/var/lib/portage/${repo}"
     done
 
@@ -42,24 +42,26 @@ PORTAGE_BINHOST="$(get_binhost_url "${binhost}" "${update_group}" 'pkgs')
 $(get_binhost_url "${binhost}" "${update_group}" 'toolchain')"
 EOF
 
-    sudo_clobber "${root_fs_dir}/etc/portage/repos.conf/coreos.conf" <<EOF
+    sudo_clobber "${root_fs_dir}/etc/portage/repos.conf/gentoo-subset.conf" <<EOF
 [DEFAULT]
-main-repo = portage-stable
+main-repo = gentoo-subset
 
-[coreos]
-location = /var/lib/portage/coreos-overlay
+[gentoo-subset]
+location = /var/lib/portage/gentoo-subset
+EOF
 
-[portage-stable]
-location = /var/lib/portage/portage-stable
+    sudo_clobber "${root_fs_dir}/etc/portage/repos.conf/flatcar-overlay.conf" <<EOF
+[flatcar-overlay]
+location = /var/lib/portage/flatcar-overlay
 EOF
 
     # Now set the correct profile, we do not use the eselect tool - it
     # does not seem to be usable outside of the chroot without using
     # deprecated PORTDIR and PORTDIR_OVERLAY environment variables.
     local profile_name=$(get_board_profile "${BOARD}")
-    # Turn coreos:coreos/amd64/generic into coreos/amd64/generic/dev
+    # Turn flatcar-overlay:coreos/amd64/generic into coreos/amd64/generic/dev
     profile_name="${profile_name#*:}/dev"
-    local profile_directory="${root_fs_dir}/var/lib/portage/coreos-overlay/profiles/${profile_name}"
+    local profile_directory="${root_fs_dir}/var/lib/portage/flatcar-overlay/profiles/${profile_name}"
     if [[ ! -d "${profile_directory}" ]]; then
         die "Not a valid profile: ${profile_name}"
     fi

build_library/portage/env/releng/qemu

@@ -0,0 +1 @@
+FEATURES="-pid-sandbox -network-sandbox -ipc-sandbox"

build_library/portage/package.env/qemu

@@ -0,0 +1 @@
+*/* releng/qemu

build_library/portage/profile/package.use.force/releng/alternatives

@@ -0,0 +1,10 @@
+# need this here because of stage1
+app-alternatives/awk gawk
+app-alternatives/bc gnu
+app-alternatives/bzip2 reference
+app-alternatives/cpio gnu
+app-alternatives/gzip reference
+app-alternatives/lex flex
+app-alternatives/sh bash
+app-alternatives/tar gnu
+app-alternatives/yacc bison

build_library/prefix_util.sh

@@ -67,7 +67,7 @@ function setup_prefix_dirs() {
   sudo cp -vR "${prefix_repo}/skel/etc/portage" "${STAGINGROOT}${EPREFIX}/etc/"
   sudo cp -vR "${prefix_repo}/skel/etc/portage" "${FINALROOT}${EPREFIX}/etc/"
 
-  local profile="/mnt/host/source/src/third_party/portage-stable/profiles/default/linux"
+  local profile="/mnt/host/source/src/scripts/repos/gentoo-subset/profiles/default/linux"
   case "${PREFIX_BOARD}" in
     amd64-usr) profile="${profile}/amd64/17.1/no-multilib/prefix/kernel-3.2+";;
     arm64-usr) profile="${profile}/arm64/17.0/prefix/kernel-3.2+";;

build_library/sysext_mangle_containerd-flatcar

@@ -9,7 +9,7 @@ echo ">>> NOTICE: $0: removing 'gen-manpages', 'containerd-stress' from sysext"
 rm -f "${rootfs}/usr/bin/gen-manpages" "${rootfs}/usr/bin/containerd-stress"
 
 script_root="$(cd "$(dirname "$0")/../"; pwd)"
-files_dir="${script_root}/sdk_container/src/third_party/coreos-overlay/coreos/sysext/containerd"
+files_dir="${script_root}/repos/flatcar-overlay/coreos/sysext/containerd"
 
 echo ">>> NOTICE $0: installing extra files from '${files_dir}'"
 cp -va "${files_dir}/"* "${rootfs}"

build_library/sysext_mangle_docker-flatcar

@@ -8,7 +8,7 @@ echo ">>> NOTICE: $0: removing '/usr/lib/debug/', '/usr/share/docker/contrib' fr
 rm -rf "${rootfs}/usr/lib/debug/" "${rootfs}/usr/share/docker/contrib/"
 
 script_root="$(cd "$(dirname "$0")/../"; pwd)"
-files_dir="${script_root}/sdk_container/src/third_party/coreos-overlay/coreos/sysext/docker"
+files_dir="${script_root}/repos/flatcar-overlay/coreos/sysext/docker"
 
 echo ">>> NOTICE $0: installing extra files from '${files_dir}'"
 cp -va "${files_dir}/"* "${rootfs}"

build_library/toolchain_util.sh

@@ -14,18 +14,18 @@ TOOLCHAIN_PKGS=(
 # This is only used as an intermediate step to be able to use the cross
 # compiler to build a full native toolchain. Packages are not uploaded.
 declare -A CROSS_PROFILES
-CROSS_PROFILES["x86_64-cros-linux-gnu"]="coreos:coreos/amd64/generic"
-CROSS_PROFILES["aarch64-cros-linux-gnu"]="coreos:coreos/arm64/generic"
+CROSS_PROFILES["x86_64-cros-linux-gnu"]="flatcar-overlay:coreos/amd64/generic"
+CROSS_PROFILES["aarch64-cros-linux-gnu"]="flatcar-overlay:coreos/arm64/generic"
 
 # Map board names to CHOSTs and portage profiles. This is the
 # definitive list, there is assorted code new and old that either
 # guesses or hard-code these. All that should migrate to this list.
 declare -A BOARD_CHOSTS BOARD_PROFILES
 BOARD_CHOSTS["amd64-usr"]="x86_64-cros-linux-gnu"
-BOARD_PROFILES["amd64-usr"]="coreos:coreos/amd64/generic"
+BOARD_PROFILES["amd64-usr"]="flatcar-overlay:coreos/amd64/generic"
 
 BOARD_CHOSTS["arm64-usr"]="aarch64-cros-linux-gnu"
-BOARD_PROFILES["arm64-usr"]="coreos:coreos/arm64/generic"
+BOARD_PROFILES["arm64-usr"]="flatcar-overlay:coreos/arm64/generic"
 
 BOARD_NAMES=( "${!BOARD_CHOSTS[@]}" )
 
@@ -169,7 +169,7 @@ get_sdk_arch() {
 }
 
 get_sdk_profile() {
-    echo "coreos:coreos/$(get_sdk_arch)/sdk"
+    echo "flatcar-overlay:coreos/$(get_sdk_arch)/sdk"
 }
 
 get_sdk_libdir() {
@@ -244,7 +244,7 @@ configure_crossdev_overlay() {
     echo "x-crossdev" | \
         "${sudo[@]}" tee "${root}${location}/profiles/repo_name" > /dev/null
     "${sudo[@]}" tee "${root}${location}/metadata/layout.conf" > /dev/null <<EOF
-masters = portage-stable coreos
+masters = gentoo-subset flatcar-overlay
 use-manifests = true
 thin-manifests = true
 EOF
@@ -271,7 +271,7 @@ _get_dependency_list() {
 
 # Configure a new ROOT
 # Values are copied from the environment or the current host configuration.
-# Usage: CBUILD=foo-bar-linux-gnu ROOT=/foo/bar SYSROOT=/foo/bar configure_portage coreos:some/profile
+# Usage: CBUILD=foo-bar-linux-gnu ROOT=/foo/bar SYSROOT=/foo/bar configure_portage flatcar-overlay:some/profile
 # Note: if using portageq to get CBUILD it must be called before CHOST is set.
 _configure_sysroot() {
     local profile="$1"
@@ -287,7 +287,7 @@ _configure_sysroot() {
     "${sudo[@]}" eselect profile set --force "$profile"
 
     local coreos_path
-    coreos_path=$(portageq get_repo_path "${ROOT}" coreos)
+    coreos_path=$(portageq get_repo_path "${ROOT}" flatcar-overlay)
     "${sudo[@]}" ln -sfT "${coreos_path}/coreos/user-patches" "${ROOT}/etc/portage/patches"
 
     echo "Writing make.conf for the sysroot ${SYSROOT}, root ${ROOT}"

build_library/vm_image_util.sh

@@ -553,7 +553,7 @@ install_oem_sysext() {
         --metapkgs="${metapkg}"
     )
     local overlay_path mangle_fs
-    overlay_path=$(portageq get_repo_path / coreos)
+    overlay_path=$(portageq get_repo_path / flatcar-overlay)
     mangle_fs="${overlay_path}/${metapkg}/files/manglefs.sh"
     if [[ -x "${mangle_fs}" ]]; then
         build_sysext_flags+=(

build_sysext

@@ -65,7 +65,7 @@ metapackage 'coreos-base/oem-azure' for the arm64 squashfs base at
 sudo build_sysext \\
      --board=arm64-usr \\
      --metapkgs=coreos-base/oem-azure \\
-     --mangle_fs=sdk_container/src/third_party/coreos-overlay/coreos-base/oem-azure/files/manglefs.sh \\
+     --mangle_fs=repos/flatcar-overlay/coreos-base/oem-azure/files/manglefs.sh \\
      --squashfs_base=build/artifacts/flatcar_production_image_sysext.squashfs \\
      --image_builddir=oem-images \\
      oem-azure
@@ -244,7 +244,7 @@ ROOT="${BUILD_DIR}/install-root" PORTAGE_CONFIGROOT="${BUILD_DIR}/install-root"
 if [[ "${FLAGS_strip_binaries}" = "${FLAGS_TRUE}" ]]; then
     chost="$("portageq-${BOARD}" envvar CHOST)"
     strip="${chost}-strip"
-    
+
     info "Stripping all non-stripped binaries in sysext using '${strip}'"
 
     # Find all non-stripped binaries, remove ':' from filepath, and strip 'em

build_toolchains

@@ -18,13 +18,12 @@ FORCE_STAGES="stage4"
 ## Define the stage4 config template
 catalyst_stage4() {
 cat <<EOF
-target: stage4
 pkgcache_path: $BINPKGS
 stage4/packages: @system
 stage4/fsscript: ${BUILD_LIBRARY_DIR}/catalyst_toolchains.sh
 stage4/root_overlay: ${ROOT_OVERLAY}
 EOF
-catalyst_stage_default
+catalyst_stage_default 4
 }
 create_provenance_overlay() {
   local root_overlay="$1"

checkout

@@ -17,13 +17,13 @@ usage() {
     echo "  Usage:"
     echo "  $0 [-t|-b] [-n] [-u <strategy>] <tag-or-branch>"
     echo "       Check our a tag or branch and synchronise git submodules"
-    echo "          coreos-overlay and portage-stable."
+    echo "          flatcar-overlay and gentoo-subset."
     echo "       By default, $0 tries to auto-detect whether to handle a tag or a branch,"
     echo "         with tags being prioritised over branches of the same name."
     echo
     echo "       <tag-or-branch> Name of the tag or branch to check out."
-    echo "                       BRANCH: Check out the branch in scripts, coreos-overlay"
-    echo "                                and portage-stable, and fast-forward to the"
+    echo "                       BRANCH: Check out the branch in scripts, flatcar-overlay"
+    echo "                                and gentoo-subset, and fast-forward to the"
     echo "                                latest changes by default (but see '-u' below)."
     echo "                       TAG:    Check out the tag in scripts, and update submodules."
     echo
@@ -90,8 +90,8 @@ esac
 
 # make sure submodules are initialised
 git submodule init
-for dir in sdk_container/src/third_party/coreos-overlay \
-           sdk_container/src/third_party/portage-stable ; do
+for dir in repos/flatcar-overlay \
+           repos/gentoo-subset ; do
     if [ ! -f "$dir"/.git ] ; then
         git submodule update -N "$dir"
     fi
@@ -111,10 +111,10 @@ function check_all() {
         return
     fi
 
-    local overlay="$(git -C sdk_container/src/third_party/coreos-overlay $gitcmd \
+    local overlay="$(git -C repos/flatcar-overlay $gitcmd \
                         | sed -e 's/^[[:space:]]*//' -e 's:remotes/[^/]\+/::' \
                         | grep -m1 -E "^$name\$")"
-    local portage="$(git -C sdk_container/src/third_party/portage-stable $gitcmd \
+    local portage="$(git -C repos/gentoo-subset $gitcmd \
                         | sed -e 's/^[[:space:]]*//' -e 's:remotes/[^/]\+/::' \
                         | grep -m1 -E "^$name\$")"
 
@@ -129,8 +129,8 @@ function check_all() {
 #
 if ! $force_branch; then
     for dir in . \
-        sdk_container/src/third_party/coreos-overlay \
-        sdk_container/src/third_party/portage-stable ; do
+        repos/flatcar-overlay \
+        repos/gentoo-subset ; do
         git -C "$dir" fetch --tags --force --prune --prune-tags
     done
     tag="$(check_all 'tag -l' "$name")"
@@ -166,8 +166,8 @@ echo
 echo "Checking out BRANCH '$branch'"
 echo "----------------------------------"
 for dir in . \
-    sdk_container/src/third_party/coreos-overlay \
-    sdk_container/src/third_party/portage-stable ; do
+    repos/flatcar-overlay \
+    repos/gentoo-subset ; do
 
     git -C "$dir" checkout "$branch"
     if [ -n "$update_strategy" ] ; then
@@ -175,4 +175,3 @@ for dir in . \
         git -C "$dir" pull "$update_strategy"
     fi
 done
-

ci-automation/README.md

@@ -39,7 +39,7 @@ image_build amd64
 ### SDK bootstrap build
 
 1. SDK Bootstrap (`sdk.sh`): Use a seed SDK tarball and seed SDK container image to build a new SDK tarball.
-   The resulting SDK tarball will ship packages and versions from the updated coreos-overlay and portage-stable ebuild directories.
+   The resulting SDK tarball will ship packages and versions from the updated flatcar-overlay and gentoo-subset ebuild directories.
    This step updates the versionfile, recording the SDK container version just built.
    It will generate and push a new version tag to the scripts repo.
 2. SDK container build (`sdk_container.sh`) : use SDK tarball to build an SDK container image.

ci-automation/image_changes.sh

@@ -81,8 +81,8 @@ function ricj_callback() {
         "PATH=${PATH}:${PWD}/ci-automation/python-bin"
         # Override the default locations of repositories.
         "SCRIPTS_REPO=."
-        "COREOS_OVERLAY_REPO=../coreos-overlay"
-        "PORTAGE_STABLE_REPO=../portage-stable"
+        "FLATCAR_OVERLAY_REPO=../flatcar-overlay"
+        "GENTOO_SUBSET_REPO=../gentoo-subset"
     )
     show_changes_params+=(
         # The show-changes script expects a tag name, so using git tag
@@ -251,7 +251,7 @@ function get_oem_id_list() {
     arch=${1}; shift
     list_var_name=${1}; shift
 
-    local -a ebuilds=("${scripts_repo}/sdk_container/src/third_party/coreos-overlay/coreos-base/common-oem-files/common-oem-files-"*'.ebuild')
+    local -a ebuilds=("${scripts_repo}/repos/flatcar-overlay/coreos-base/common-oem-files/common-oem-files-"*'.ebuild')
     if [[ ${#ebuilds[@]} -eq 0 ]] || [[ ! -e ${ebuilds[0]} ]]; then
         echo "No coreos-base/common-oem-files ebuilds?!" >&2
         exit 1

common.sh

@@ -51,9 +51,6 @@ fi
 # Turn on bash debug support if available for backtraces.
 shopt -s extdebug 2>/dev/null
 
-# Source qemu library path
-. /etc/profile.d/qemu-aarch64.sh 2> /dev/null || true
-
 # Output a backtrace all the way back to the raw invocation, suppressing
 # only the _dump_trace frame itself.
 _dump_trace() {
@@ -992,38 +989,3 @@ BOAT
   echo -e "${V_VIDOFF}"
   die "$* failed"
 }
-
-# The binfmt_misc support in the kernel is required.
-# The aarch64 binaries should be executed through
-# "/usr/bin/qemu-aarch64-static"
-setup_qemu_static() {
-  local root_fs_dir="$1"
-  case "${BOARD}" in
-    amd64-usr) return 0;;
-    arm64-usr)
-      if [[ -f "${root_fs_dir}/sbin/ldconfig" ]]; then
-        sudo cp /usr/bin/qemu-aarch64 "${root_fs_dir}"/usr/bin/qemu-aarch64-static
-        echo export QEMU_LD_PREFIX=\"/build/arm64-usr/\" | sudo tee /etc/profile.d/qemu-aarch64.sh
-        . /etc/profile.d/qemu-aarch64.sh
-      else
-        die "Missing basic layout in target rootfs"
-      fi
-    ;;
-    *) die "Unsupported arch" ;;
-  esac
-}
-
-clean_qemu_static() {
-  local root_fs_dir="$1"
-  case "${BOARD}" in
-    amd64-usr) return 0;;
-    arm64-usr)
-      if [[ -f "${root_fs_dir}/usr/bin/qemu-aarch64-static" ]]; then
-        sudo rm "${root_fs_dir}"/usr/bin/qemu-aarch64-static
-      else
-        die "File not found"
-      fi
-    ;;
-    *) die "Unsupported arch" ;;
-  esac
-}

find_overlay_dups

@@ -1,14 +1,14 @@
 #!/bin/bash
 
-# Prints packages which are in both portage-stable and coreos-overlay
+# Prints packages which are in both gentoo-subset and flatcar-overlay
 
 SCRIPT_ROOT=$(dirname $(readlink -f "$0"))
 . "${SCRIPT_ROOT}/common.sh" || exit 1
 
-DEFINE_string overlay_path "${SRC_ROOT}/third_party/coreos-overlay" \
+DEFINE_string overlay_path "${SCRIPT_ROOT}/repos/flatcar-overlay" \
     "Directory containing the overlay"
-DEFINE_string portage_stable_path "${SRC_ROOT}/third_party/portage-stable" \
-    "Path to portage-stable"
+DEFINE_string gentoo_subset_path "${SCRIPT_ROOT}/repos/gentoo-subset" \
+    "Path to gentoo-subset"
 
 # Parse flags
 FLAGS "$@" || exit 1
@@ -19,10 +19,10 @@ function get_tree_packages() {
 	find "$1" -maxdepth 3 -type f -name "*.ebuild" -printf "%P\n" | xargs dirname | sort | uniq
 }
 
-portage_stable_packages=$(get_tree_packages ${FLAGS_portage_stable_path})
+gentoo_subset_packages=$(get_tree_packages ${FLAGS_gentoo_subset_path})
 overlay_packages=$(get_tree_packages ${FLAGS_overlay_path})
 
-all_packages="$portage_stable_packages $overlay_packages"
+all_packages="$gentoo_subset_packages $overlay_packages"
 dups=$(sort <<< "$all_packages" | uniq -D | uniq)
 
 if [[ -z "$dups" ]]; then

generate_payload

@@ -388,7 +388,7 @@ for d in ./data/*/*; do
     for FILE_PATH in "${DATA_DIR}"/*.sig; do
       gpg --verify "${FILE_PATH}"
     done
-    
+
     echo "Generating extension payloads for ${DATA_DIR}"
     shopt -s nullglob
     for EXTENSION_PATH in "${DATA_DIR}/flatcar-"*.raw "${DATA_DIR}/oem-"*.raw; do
@@ -401,7 +401,7 @@ for d in ./data/*/*; do
           --image "${EXTENSION_PATH}" \
           --output "${OUTPUT_PATH}" \
           --private_keys "${PRIVATE_KEYS}" \
-          --public_keys "/mnt/host/source/src/scripts/sdk_container/src/third_party/coreos-overlay/coreos-base/coreos-au-key/files/official-v2.pub.pem" \
+          --public_keys "/mnt/host/source/src/scripts/repos/flatcar-overlay/coreos-base/coreos-au-key/files/official-v2.pub.pem" \
           --keys_separator "+"
       else
         echo "ERROR: Found update payload already: ${OUTPUT_PATH}."
@@ -409,10 +409,10 @@ for d in ./data/*/*; do
       fi
     done
     shopt -u nullglob
-    
+
     echo "Extracting flatcar_production_update.bin.bz2 for ${DATA_DIR}"
     bunzip2 -f -k "${DATA_DIR}/flatcar_production_update.bin.bz2"
-    
+
     echo "Generating generic update payload for ${DATA_DIR}"
     OUTPUT_PATH="${DATA_DIR}/flatcar_production_update.gz"
     if [ ! -f "${OUTPUT_PATH}" ]; then
@@ -422,12 +422,12 @@ for d in ./data/*/*; do
             --kernel "${DATA_DIR}/flatcar_production_image.vmlinuz" \
             --output "${OUTPUT_PATH}" \
             --private_keys "${PRIVATE_KEYS}" \
-            --public_keys "/mnt/host/source/src/scripts/sdk_container/src/third_party/coreos-overlay/coreos-base/coreos-au-key/files/official-v2.pub.pem" \
+            --public_keys "/mnt/host/source/src/scripts/repos/flatcar-overlay/coreos-base/coreos-au-key/files/official-v2.pub.pem" \
             --keys_separator "+"
     else
         echo "ERROR: Found update payload already: ${OUTPUT_PATH}."
         exit 1
     fi
-    
+
     echo "Payload generated: ${OUTPUT_PATH}"
 done

jenkins/manifest.sh

@@ -117,11 +117,11 @@ then
 fi
 if [[ -n "${OVERLAY_REF}" ]]
 then
-        setup_manifest_ref coreos-overlay "${OVERLAY_REF}"
+        setup_manifest_ref flatcar-overlay "${OVERLAY_REF}"
 fi
 if [[ -n "${PORTAGE_REF}" ]]
 then
-        setup_manifest_ref portage-stable "${PORTAGE_REF}"
+        setup_manifest_ref gentoo-subset "${PORTAGE_REF}"
 fi
 
 ln -fns "${FLATCAR_BUILD_ID}.xml" manifest/default.xml

repos/flatcar-overlay/README.md

@@ -3,7 +3,7 @@
 This overlay contains Container Linux specific packages and Gentoo packages
 that differ from their upstream Gentoo versions.
 
-See the [portage-stable](https://github.com/coreos/portage-stable) repo
+See the [gentoo-subset](https://github.com/flatcar/scripts/repos/gentoo-subset) repo
 for packages which do not have Container Linux specific changes.
 
 Licensing information can be found in the respective files, so consult
@@ -29,7 +29,7 @@ development or production image, plus any OEM specific packages.
 that belong in the OEM partition. Any RDEPENDS from these packages should
 be copied to the RDEPENDS in `board-packages` to ensure they are built.
 
-`coreos-base/coreos-oem-*` are metapackages for OEM specific ACIs. 
+`coreos-base/coreos-oem-*` are metapackages for OEM specific ACIs.
 
 # Updating
 
@@ -39,7 +39,7 @@ To update follow the following steps:
   upstream Gentoo, not only resetting the ebuild file but also any additional
   files like patches or downstream additions under `files`.
 * Run `~/trunk/src/scripts/update_ebuilds --portage_stable . CATEGORY/PACKAGE`
-  in the `coreos-overlay` folder to import a new version from upstream Gentoo.
+  in the `flatcar-overlay` folder to import a new version from upstream Gentoo.
   Drop the ebuild files that you don't plan to use.
 * Commit the changes with a message like `CATEGORY/PACKAGE: Sync from Gentoo`,
   and mention the the commit ID in the body (`git show update_ebuilds/master`).

repos/flatcar-overlay/app-containers/docker-cli/README.md

@@ -0,0 +1 @@
+This is a temporary overlay over gentoo-subset until Gentoo catches up with package versions.

repos/flatcar-overlay/app-containers/docker/README.md

@@ -0,0 +1 @@
+This is a temporary overlay over gentoo-subset until Gentoo catches up with package versions.