mirrors/flatcar-scripts
1
0
Fork 0
mirror of https://github.com/flatcar/scripts.git synced 2025-09-23 06:31:18 +02:00
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #147 from flatcar-linux/t-lo/sdk-container

Browse Source 
SDK container: Add support for SDK containers.
This commit is contained in:
Thilo Fromm 2021-12-07 18:56:03 +01:00 committed by GitHub
commit f996c974fe
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
34 changed files with 1912 additions and 4 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
.dockerignore Normal file
View File

@ -0,0 +1,2 @@
__build__
sdk_container/.cache

12
.gitignore vendored
View File

@ -8,3 +8,15 @@
*.pyc *.pyc
*~ *~
/cbuildbot_package.list /cbuildbot_package.list
# Flatcar SDK tarballs
*.tar.bz2
# SDK container env passing helpers
sdk_container/.env
sdk_container/.sdkenv
# build cache / artefacts directories
__build__/
sdk_container/.cache
sdk_container/.config

6
.gitmodules vendored Normal file
View File

@ -0,0 +1,6 @@
[submodule "sdk_container/src/third_party/coreos-overlay"]
path = sdk_container/src/third_party/coreos-overlay
url = https://github.com/flatcar-linux/coreos-overlay.git
[submodule "sdk_container/src/third_party/portage-stable"]
path = sdk_container/src/third_party/portage-stable
url = https://github.com/flatcar-linux/portage-stable.git

127
README.md
View File

@ -2,4 +2,131 @@
Welcome to the scripts repo, your starting place for most things here in the Flatcar Container Linux SDK. To get started you can find our documentation on [the Flatcar docs website][flatcar-docs]. Welcome to the scripts repo, your starting place for most things here in the Flatcar Container Linux SDK. To get started you can find our documentation on [the Flatcar docs website][flatcar-docs].
The SDK can be used to
* Patch or update applications or libraries included in the Flatcar OS image
* Add or remove applications and / or libraries
* Modify the kernel configuration and add or remove kernel modules included with Flatcar
* Build OS images for a variety of targets (qemu, bare metal, AWS, Azure, VMWare, etc.)
* And lastly, the SDK can be used to upgrade SDK packages and to build new SDKs
[flatcar-docs]: https://docs.flatcar-linux.org/os/sdk-modifying-flatcar/ [flatcar-docs]: https://docs.flatcar-linux.org/os/sdk-modifying-flatcar/
# Using the scripts repository: submodules and tags
The repository is meant to be the entry point for Flatcar builds and development.
For building packages, there are 2 additional repositories, [coreos-overlay](https://github.com/flatcar-linux/) and [portage-stable](https://github.com/flatcar-linux/portage-stable), which contain all packages' `ebuild` (build configuration) files.
These repositories are included in `scripts` via git submodules and are used by the SDK container wrapper scripts detailed on further below.
The submodules reside in:
```
scripts
+--sdk_container
+---------src
+--third_party
+------coreos-overlay
+------portage-stable
```
When working with the scripts repo always make sure to initialise and to update these submodules; otherwise builds will break because build configuration is missing:
```bash
$ git clone --recurse-submodules https://github.com/flatcar-linux/scripts.git
```
The `scripts` repository makes ample use of tags to mark releases.
Sometimes, local and origin tags can diverge (e.g. when re-tagging something locally to test a build).
Also, `git pull` and `git fetch` do not automatically pull new tags, so long-standing local sourcetrees may lack newer versions.
To fetch and update all tags and to remove tags locally which have been deleted upstream, do
```
$ git pull --all --tags --prune --prune-tags
```
If upstream retagged (of if a tag was changed locally) the corresponding upstream tag will not be pulled so the local tag remains.
In order to override local tags with upstream, run
```
$ git pull --all --tags --prune --prune-tags --force
```
# Using the SDK container
We provide a containerised SDK via https://github.com/orgs/flatcar-linux/packages. The container comes in 3 flavours:
* Full SDK initialised with both architectures supported by Flatcar (amd64 and arm64). This is the largest container, it's about 8GB in size (~3 GB compressed).
* AMD64 SDK initialised for building AMD64 OS images. About 6GB in size (2GB compressed).
* ARM64 SDK initialised for building ARM64 OS images on AMD64 hosts. Also about 6GB in size.
While work on a native ARM64 native SDK is ongoing, it's unfortunately not ready yet. If you want to help, patches are welcome!
The container can be run in one of two ways - "standalone", or integrated with the [scripts](https://github.com/flatcar-linux/scripts) repo:
* Standalone mode will use no host volumes and will allow you to play with the SDK in a sandboxed throw-away environment. In standalone mode, you interface with Docker directly to use the SDK container.
* Integrated mode will closely integrate with the scripts repo directory and bind-mount it as well as the portage-stable and coreos-overlay gitmodules into the container. Integrated mode uses wrapper scripts to interact with the SDK container. This is the recommended way for developing patches for Flatcar.
## Standalone mode
In standalone mode, the SDK is just another Docker container. Interaction with the container happens via use of `docker` directly. Use for experimenting and for throw-away work only, otherwise please use integrated mode (see below).
* Check the list of available versions and pick a version to use. The SDK Major versions correspond to Flatcar Major release versions.
List of images: `https://github.com/orgs/flatcar-linux/packages/container/package/flatcar-sdk-all`
For the purpose of this example we'll use version `3033.0.0`.
* Fetch the container image: `docker pull ghcr.io/flatcar-linux/flatcar-sdk-all:3033.0.0`
* Start the image in interactive (tty) mode: `docker run -ti ghcr.io/flatcar-linux/flatcar-sdk-all:3033.0.0`
You are now inside the SDK container (the hostname will likely differ):
`sdk@f236fda982a4 ~/trunk/src/scripts $`
* Initialise the SDK in self-contained mode. This needs to be done once per container and will check out the scripts, coreos-overlay, and portage-stable repositories into the container.
`sdk@f236fda982a4 ../sdk_init_selfcontained.sh`
You can now work with the SDK container.
### Privileged mode when building images
In order to build OS images (via `./build_image` and `./image_to_vm`) the SDK tooling requires privileged access to `/dev`.
This is necessary because the SDK currently employs loop devices to create and to partition OS images.
To start a container in privileged mode with `/dev` available use:
* `docker run -ti --privileged -v /dev:/dev ghcr.io/flatcar-linux/flatcar-sdk-all:3033.0.0`
## Integrated mode
This is the preferred mode of working with the SDK.
Interaction with the container happens via wrapper scripts from the scripts repository.
Both the host's scripts repo as well as its submodules (portage-stable and coreos-overlay) are made available in the container, allowing for work on these repos directly.
The wrapper scripts will re-use existing containers instead of creating new ones to preserve your work in the container, enabling consistency.
To clone the scripts repo and pick a version:
* Clone the scripts repo: `git clone https://github.com/flatcar-linux/scripts.git`
* Optionally, check out a release tag to base your work on
* list releases (e.g. all Alpha releases): `git tag -l alpha-*`
* check out the release version, e.g. `3033.0.0`: `git checkout 3033.0.0`
* Make sure to initialise and fetch git submodules - Flatcar's ebuilds are in 2 separate repositories, connected to `scripts` via submodules.
* `git submodule init; git submodule update`
To use the SDK container:
* Fetch image and start the SDK container: `./run_sdk_container -t`
This will fetch the container image of the "scripts" repo's release version you checked out.
The `-t` command line flag will allocate a TTY, which is preferred for interactive use.
The command will put you into the SDK container:
`sdk@sdk-container ~/trunk/src/scripts $`
* Alternatively, you can run individual commands in the SDK container using `./run_sdk_container <command>` (e.g. `./run_sdk_container ./build_packages`).
Subsequent calls to `./run_sdk_container` will re-use the container (as long as the local release version check-out the scripts repo does not change).
Check out `docker container ls --all` and you'll see something like
```
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
19ea3b6d00ad ghcr.io/flatcar-linux/flatcar-sdk-all:3033.0.0 "/bin/sh -c /home/sd…" 4 hours ago Exited (0) About an hour ago flatcar-sdk-all-3033.0.0_os-3033.0.0
```
Re-use of containers happens on a per-name basis. The above example's container name `flatcar-sdk-all-3033.0.0_os-3033.0.0` is generated automatically. Using `docker container rm` the container can be discarded - a subsequent call to `./run_sdk_container` will create a new one. Custom containers can be created by use of the `-n <name>` command line option; these will be re-used in subsequent calls to `./run_sdk_container` when using the same `<name>`.
The local sourcetree can also be used with an entirely custom SDK container image. Users must ensure that the image is either fetch-able or present locally. The custom image can be specified using `-C <custom-image>`. This option is useful e.g. for building the local sourcetree with different SDK versions.
Check out `./run_sdk_container -h` for more information on command line options.
# Building a new SDK container
Building an SDK container is done using `./build_sdk_container_image <tarball>`.
The tarball input is the result of an SDK bootstrap (see below). Version information for both OS as well as for the SDK will be extracted from the tarball name.
The version file will be updated accordingly before the SDK container is built.
During the build, toolchain packages will be built and installed into the SDK container image. Both supported boards (`amd64-usr` and `arm64-usr`) will be initialised in the container image.
# Bootstrapping a new SDK tarball using the SDK container
The script `./bootstrap_sdk_container` bootstraps a new SDK tarball using an existing SDK container and seed tarball. Specifying the seed version as well as the designated new SDK version is required for this script.
# Automation stubs for continuous integration
Script stubs for various build stages can be found in the [ci-automation](ci-automation) folder. These are helpful for gluing Flatcar Container Linux builds to a continuous integration system.

78
bootstrap_sdk_container Executable file
View File

@ -0,0 +1,78 @@
#!/bin/bash
#
# Copyright (c) 2021 The Flatcar Maintainers.
# Use of this source code is governed by a BSD-style license that can be
# found in the LICENSE file.
set -eu
cd $(dirname "$0")
source sdk_lib/sdk_container_common.sh
seed_version=""
target_version=""
vernum="$(strip_version_prefix "$target_version")"
if is_official "$vernum" ; then
official="true"
else
official="false"
fi
declare -a cleanup
# --
usage() {
echo " Usage:"
echo " $0 <seed-sdk-version> <new-sdk-version> [-x <cleanup-script>]"
echo
echo " This script will bootstrap a new SDK tarball using an SDK container."
echo " '$sdk_container_common_versionfile' will be updated to the target version."
echo
echo " <seed-sdk-vernum> - SDK version number (e.g. '3005.0.0') to use for bootstrapping."
echo " The SDK container will be pulled and the tarball"
echo " downloaded if necessary."
echo " <new-sdk-vernum> - SDK version number (e.g. '3027.0.0') of the new SDK."
echo " -x <cleanup-script> - For each resource generated during build (container etc.)"
echo " add a cleanup line to <script> which, when run, will free"
echo " the resource. Useful for CI."
echo " -h - Print this help."
echo
}
# --
while [ 0 -lt $# ] ; do
case "$1" in
-h) usage; exit 0;;
-x) cleanup=("-x" "$2"); shift; shift;;
*) if [ -z "$seed_version" ] ; then
seed_version="$1"
elif [ -z "$target_version" ] ; then
target_version="$1"
else
echo "ERROR: Spurious positional parameter '$1'"
usage; exit 1;
fi
shift;;
esac
done
if [ -z "$seed_version" -o -z "$target_version" ] ; then
echo "ERROR: Missing seed and /or target SDK version."
usage
exit 1
fi
# --
yell "\n######\n###### Bootstrapping SDK version $target_version from seed ($seed_version)"
if $official; then
export COREOS_OFFICIAL=1
fi
# bootstrap_sdk needs FLATCAR_SDK_VERSION set to the seed version
./run_sdk_container "${cleanup[@]}" -V "$seed_version" -v "$target_version" \
sudo -E ./bootstrap_sdk
# Update versionfile to the actual SDK version
create_versionfile "${target_version}"

4
build_image
View File

@ -36,7 +36,7 @@ DEFINE_string base_dev_pkg "coreos-base/coreos-dev" \
DEFINE_string torcx_manifest "${DEFAULT_BUILD_ROOT}/torcx/${DEFAULT_BOARD}/latest/torcx_manifest.json" \ DEFINE_string torcx_manifest "${DEFAULT_BUILD_ROOT}/torcx/${DEFAULT_BOARD}/latest/torcx_manifest.json" \
"The torcx manifest describing torcx packages for this image (or blank for none)" "The torcx manifest describing torcx packages for this image (or blank for none)"
DEFINE_string torcx_root "${DEFAULT_BUILD_ROOT}/torcx" \ DEFINE_string torcx_root "${DEFAULT_BUILD_ROOT}/torcx" \
"Directory in which torcx packages can be found" "Directory in which torcx packages can be found. Will update the default --torcx_manifest if set."
DEFINE_string output_root "${DEFAULT_BUILD_ROOT}/images" \ DEFINE_string output_root "${DEFAULT_BUILD_ROOT}/images" \
"Directory in which to place image result directories (named by version)" "Directory in which to place image result directories (named by version)"
DEFINE_string disk_layout "" \ DEFINE_string disk_layout "" \
@ -93,7 +93,7 @@ check_gsutil_opts
# Patch around default values not being able to depend on other flags. # Patch around default values not being able to depend on other flags.
if [ "x${FLAGS_torcx_manifest}" = "x${DEFAULT_BUILD_ROOT}/torcx/${DEFAULT_BOARD}/latest/torcx_manifest.json" ]; then if [ "x${FLAGS_torcx_manifest}" = "x${DEFAULT_BUILD_ROOT}/torcx/${DEFAULT_BOARD}/latest/torcx_manifest.json" ]; then
FLAGS_torcx_manifest="${DEFAULT_BUILD_ROOT}/torcx/${FLAGS_board}/latest/torcx_manifest.json" FLAGS_torcx_manifest="${FLAGS_torcx_root}/${FLAGS_board}/latest/torcx_manifest.json"
fi fi
# If downloading packages is enabled ensure the board is configured properly. # If downloading packages is enabled ensure the board is configured properly.

4
build_packages
View File

@ -36,6 +36,8 @@ DEFINE_boolean skip_toolchain_update "${FLAGS_FALSE}" \
"Don't update toolchain automatically." "Don't update toolchain automatically."
DEFINE_boolean skip_chroot_upgrade "${FLAGS_FALSE}" \ DEFINE_boolean skip_chroot_upgrade "${FLAGS_FALSE}" \
"Don't run the chroot upgrade automatically; use with care." "Don't run the chroot upgrade automatically; use with care."
DEFINE_string torcx_output_root "${DEFAULT_BUILD_ROOT}" \
"Directory in which to place torcx stores and manifests (named by board/version)"
DEFINE_boolean skip_torcx_store "${FLAGS_FALSE}" \ DEFINE_boolean skip_torcx_store "${FLAGS_FALSE}" \
"Don't build a new torcx store from the updated sysroot." "Don't build a new torcx store from the updated sysroot."
@ -301,7 +303,7 @@ upload_packages
# Build a new torcx store with the updated packages, passing flags through. # Build a new torcx store with the updated packages, passing flags through.
if [ "${FLAGS_skip_torcx_store}" -eq "${FLAGS_FALSE}" ]; then if [ "${FLAGS_skip_torcx_store}" -eq "${FLAGS_FALSE}" ]; then
"${SCRIPTS_DIR}"/build_torcx_store --board="${BOARD}" "${SCRIPTS_DIR}"/build_torcx_store --board="${BOARD}" --output_root="${FLAGS_torcx_output_root}"
fi fi
info "Builds complete" info "Builds complete"

245
build_sdk_container_image Executable file
View File

@ -0,0 +1,245 @@
#!/bin/bash
#
# Copyright (c) 2021 The Flatcar Maintainers.
# Use of this source code is governed by a BSD-style license that can be
# found in the LICENSE file.
# This script will generate an SDK container image from an SDK tarball
# (which in turn is generated by bootstrap_sdk[_container]).
#
# It uses a number of intermediate build steps:
# 1. Import the SDK tarball into a "tarball" container image.
# 2. Build a "plain" SDK container image which creates the "sdk" user
# and runs "update_chroot" to initialise the x86 and aarch64 SDK cross env.
# This step uses sdk_lib/Dockerfile.sdk-import.
# 3. Using the "plain" SDK image, start a temporary "toolchain" container
# to build toolchain binary packages.
# 4. Inheriting from the "plain" SDK image and using the toolchain packages,
# build a "full" SDK container image, with board support for both amd64-usr and arm64-usr.
# A temporary HTTP server on the Docker interface IP is spun up in this step
# to serve the toolchain binpkgs to the build container.
# This step uses sdk_lib/Dockerfile.sdk-build.
# 5. In a last step, all unnecessary binaries are removed from the "full" image and 3
# output SDK images are produced:
# - "all", with both amd64 and arm64 target support, and
# - "amd64" and "arm64, with only amd64 or arm64 target support respectively.
# This step uses sdk_lib/Dockerfile.lean-arch.
set -eu
cd $(dirname "$0")
source sdk_lib/sdk_container_common.sh
arch="amd64"
official="0"
tarball=""
os_version=""
keep="false"
cleanup=""
usage() {
echo " $0 - Create SDK container image from an SDK tarball"
echo " This script will set up a new SDK container from a tarball."
echo " The resulting container comes in 3 flavours:"
echo " 1. flatcar-sdk-all - includes both ARM64 and AMD64 support"
echo " 2.+3. flatcar-sdk-(amd64|arm64) - only includes support for one target."
echo " Usage:"
echo " $0 <tarball>] [-k] [-v <os-version>] [-x <script>]"
echo
echo " <tarball> - Local tarball to build SDK from."
echo " The tarball must follow the format"
echo " flatcar-sdk-(amd64|arm64)-<version>.tar.bz2."
echo " -v <version> - Use custom OS version (defaults to tarball's SDK version)."
echo " -k - Keep intermediate build containers (sdk-import / sdk-tarball)."
echo " (Useful for sdk container build script development.)"
echo " -x <script> - For each resource generated during build (container etc.)"
echo " add a cleanup line to <script> which, when run, will free"
echo " the resource. Useful for CI."
echo " -h - Print this help."
echo
}
# --
while [ 0 -lt $# ] ; do
case "$1" in
-h) usage; exit 0;;
-k) keep="true"; shift;;
-v) os_version="$2"; shift; shift;;
-x) cleanup="$2"; shift; shift;;
*) if [ -z "$tarball" ] ; then
tarball="$1"; shift
else
echo "ERROR: spurious positional parameter '$@'."
usage
exit 1
fi;;
esac
done
if [ -z "$tarball" -o ! -s "$tarball" ] ; then
echo "ERROR: missing / invalid SDK tarball argument"
exit 1
fi
# --
# Grok version / arch from tarball name, set official if version is a release version
version="$(echo "$tarball" | sed -n 's/.*flatcar-sdk-\(arm64\|amd64\)-\(.\+\)\.tar\.bz2/\2/p')"
arch="$(echo "$tarball" | sed -n 's/.*flatcar-sdk-\(arm64\|amd64\)-.*\.tar\.bz2/\1/p')"
if [ -z "$version" -o -z "$arch" ]; then
echo "ERROR: Unable to determine version / arch from '$tarball'"
exit 1
fi
if [ -z "${os_version}" ] ; then
os_version="${version}"
fi
if is_official "$version" && [ "${version}" = "${os_version}" ] ; then
official="1"
else
official="0"
fi
# --
# import tarball
#
yell "\n######\n###### Building SDK container for version $version from '$tarball'"
create_versionfile "$version" "${os_version}"
docker_vernum="$(vernum_to_docker_image_version "${version}")"
import_tarball="flatcar-sdk-tarball:${docker_vernum}"
image_present="$(docker image ls "$import_tarball" --format '{{.Repository}}:{{.Tag}}')"
if [ "${image_present}" = "${import_tarball}" ] ; then
yell "Using existing SDK tarball image '${import_tarball}'"
else
yell "Importing SDK tarball"
if [ -n "$cleanup" ] ; then
echo "docker image rm -f '${import_tarball}'" >> "$cleanup"
fi
docker import "${tarball}" "${import_tarball}"
fi
# --
# build plain SDK container w/o board support
#
import_image="flatcar-sdk-import:${docker_vernum}"
image_present="$(docker image ls "${import_image}" --format '{{.Repository}}:{{.Tag}}')"
if [ "$image_present" = "${import_image}" ] ; then
yell "Using existing SDK import image '${import_image}'"
else
yell "Building plain SDK import image"
if [ -n "$cleanup" ] ; then
echo "docker image rm -f '${import_image}'" >> "$cleanup"
fi
docker build -t "$import_image" \
--build-arg VERSION="${docker_vernum}" \
-f sdk_lib/Dockerfile.sdk-import \
.
fi
# --
# build full SDK container w/ board support.
# This uses the SDK import container to first build toolchain binpkgs.
# Then, the import container and toolchain packages are used
# to build a full SDK container w/ amd64 and arm64 board support.
#
sdk_build_image="flatcar-sdk-build:${docker_vernum}"
image_present="$(docker image ls "${sdk_build_image}" --format '{{.Repository}}:{{.Tag}}')"
if [ "$image_present" = "${sdk_build_image}" ] ; then
yell "Using existing SDK build image '${sdk_build_image}'"
else
# --- Toolchains build ---
yell "Building toolchains in a temporary container."
# We need to use run_sdk_container instead of building from a Dockerfile
# since toolchains build uses catalyst which requires privileged access.
tarball_copied=""
if [ "$(basename "${tarball}")" != "${tarball}" ] ; then
cp --reflink=auto "${tarball}" ./
tarball="$(basename "${tarball}")"
tarball_copied="${tarball}"
fi
toolchains_container="flatcar-sdk-toolchains-build-${docker_vernum}"
if [ -n "$cleanup" ] ; then
echo "docker container rm -f '${toolchains_container}'" >> "$cleanup"
fi
./run_sdk_container -C "${import_image}" -n "${toolchains_container}" \
sudo ./build_toolchains --seed_tarball="./${tarball}"
# remove sdk tarball from scripts root so it's not part of the SDK container build context
if [ -f "${tarball_copied}" ] ; then
rm "${tarball_copied}"
fi
docker container rm "${toolchains_container}"
host_ip="$(ip addr show docker0 | grep -Po 'inet \K[\d.]+')"
binhost_port="$((1000 + (RANDOM % 55000) ))"
binhost="${host_ip}:${binhost_port}"
binhost_container="${toolchains_container}-binhost-${binhost_port}"
yell "Building SDK container + board support, toolchain packages served at http://${binhost} by ${binhost_container}"
# Spin up temporary toolchains package binhost
if [ -n "$cleanup" ] ; then
echo "docker container rm -f '${binhost_container}'" >> "$cleanup"
fi
docker run --rm -d -p "${binhost}":80 \
--name ${binhost_container} \
-v "$(pwd)/__build__/images/catalyst/packages/coreos-toolchains/target":/usr/share/caddy \
caddy caddy file-server \
--root /usr/share/caddy --browse
# --- Full SDK container build ---
yell "Initialising the SDK container and building board packages"
if [ -n "$cleanup" ] ; then
echo "docker image rm -f '${sdk_build_image}'" >> "$cleanup"
fi
docker build -t "${sdk_build_image}" \
--build-arg VERSION="${docker_vernum}" \
--build-arg BINHOST="http://${binhost}" \
--build-arg OFFICIAL="${official}" \
-f sdk_lib/Dockerfile.sdk-build \
.
docker stop "${binhost_container}"
fi
# --
# Derive "lean" SDK containers from full build. Main purpose
# of this step is to remove "white-outs", i.e. files which have been
# deleted in the full image but are still present in an intermediate layer.
#
for a in all arm64 amd64; do
yell "Creating '$a' arch SDK image"
rmarch=""; rmcross=""
case $a in
arm64) rmarch="amd64-usr"; rmcross="x86_64-cros-linux-gnu";;
amd64) rmarch="arm64-usr"; rmcross="aarch64-cros-linux-gnu";;
esac
docker build -t "$sdk_container_common_registry/flatcar-sdk-${a}:${docker_vernum}" \
--build-arg VERSION="${docker_vernum}" \
--build-arg RMARCH="${rmarch}" \
--build-arg RMCROSS="${rmcross}" \
-f sdk_lib/Dockerfile.lean-arch \
.
done
# --
# Cleanup
#
if ! $keep; then
yell "Cleaning up intermediate containers"
docker rmi flatcar-sdk-build:"${docker_vernum}"
docker rmi flatcar-sdk-import:"${docker_vernum}"
docker rmi flatcar-sdk-tarball:"${docker_vernum}"
fi

77
ci-automation/README.md Normal file
View File

@ -0,0 +1,77 @@
# CI automation glue logic scripts
Scripts in this directory aim to ease automation of Flatcar builds in continuous integration systems.
Each script represents a distinct build step; each step ingests the container image of the previous step and produces a new container image for the next step.
Notable exceptions are "SDK Bootstrap" (`sdk.sh`) which only creates an SDK tarball, and "VMs build" which does not output a container but only VM (vendor) images.
Please refer to the individual scripts for prerequisites, input parameters, and outputs.
All steps make use of a "build cache" server for pulling (https) and pushing (rsync) build inputs and artifacts.
## Build steps
The build pipeline can be used to build everything from scratch, including the SDK (starting from 1. below) or to build a new OS image (starting from 3.).
### SDK bootstrap build
1. SDK Bootstrap (`sdk.sh`): Use a seed SDK tarball and seed SDK container image to build a new SDK tarball.
The resulting SDK tarball will use packages and versions pinned in the coreos-overlay and portage-stable submodules.
This step updates the versionfile, recording the SDK container version just built.
It will generate and push a new version tag to the scripts repo.
2. SDK container build (`sdk_container.sh`) : use SDK tarball to build an SDK container image.
The resulting image will come in "amd64", "arm64", and "all" flavours, with support for respective OS target architectures. This step builds the Flatcar SDK container images published at ghcr.io/flatcar-linux.
```
.---------. .------------.
| scripts | | CI |
| repo | | automation |
`---------´ `------------´
| |
| "alpha-3449.0.0-dev23"
| |
| _______v_______
+------- clone -----> ( SDK bootstrap )
| `-------------´
|<- tag: sdk-3499.0.0-dev23 -´|
| |
| _______v_______
+-- clone -> ( SDK container )
| sdk-3499.0.0-dev23 `-------------´
| |
v v
continue to OS
image build
|
v
```
### OS image build
3. Packages build (`packages.sh`): using the SDK container version recorded in the versionfile, build OS image packages and generate a new container image (containing both SDK and packages).
This step updates the versionfile, recording the Flatcar OS image version just built.
It will generate and push a new version tag to the scripts repo.
4. Image build (`image.sh`): Using the container from 3., build an OS image and torcx store, and generate a new container image with everything in it.
5. VMs build (`vms.sh`). Using the packages+torcx+image container from 4., build vendor images. Results are vendor-specific OS images.
```
.---------. .------------.
| scripts | | CI |
| repo | | automation |
`---------´ `------------´
| |
| "alpha-3449.0.0-dev23"
| |
| ____v_____
+---------- clone --------> ( packages )
| `--------´
|<-- tag: alpha-3499.0.0-dev23 --´|
| |
| ___v___
+----- clone ---> ( image )
| alpha-3499.0.0-dev23 `-----´
| |
| __v__
+----- clone ---> ( vms )
alpha-3499.0.0-dev23 `---´
```

24
ci-automation/ci-config.env Normal file
View File

@ -0,0 +1,24 @@
# Copyright (c) 2021 The Flatcar Maintainers.
# Use of this source code is governed by a BSD-style license that can be
# found in the LICENSE file.
# Flatcar CI static configuration
BUILDCACHE_SERVER="bincache.flatcar-linux.net"
BUILDCACHE_PATH_PREFIX="/srv/bincache"
BUILDCACHE_USER="bincache"
RELEASES_SERVER="mirror.release.flatcar-linux.net"
CONTAINER_REGISTRY="ghcr.io/flatcar-linux"
GC_BUCKET="flatcar-linux"
# No PIGZ on Flatcar
PIGZ="docker run --rm -i ghcr.io/flatcar-linux/pigz --fast"
CI_GIT_AUTHOR="flatcar-ci"
CI_GIT_EMAIL="infra+ci@flatcar-linux.org"
# build artifacts go here (in container)
CONTAINER_TORCX_ROOT="/home/sdk/build/torcx"
CONTAINER_IMAGE_ROOT="/home/sdk/build/images"

208
ci-automation/ci_automation_common.sh Normal file
View File

@ -0,0 +1,208 @@
#!/bin/bash
#
# Copyright (c) 2021 The Flatcar Maintainers.
# Use of this source code is governed by a BSD-style license that can be
# found in the LICENSE file.
# CI automation common functions.
source ci-automation/ci-config.env
: ${PIGZ:=pigz}
# set up author and email so git does not complain when tagging
git -C . config user.name "${CI_GIT_AUTHOR}"
git -C . config user.email "${CI_GIT_EMAIL}"
function init_submodules() {
git submodule init
git submodule update
}
# --
function update_submodule() {
local submodule="$1"
local commit_ish="$2"
cd "sdk_container/src/third_party/${submodule}"
git fetch --all --tags
git checkout "${commit_ish}"
cd -
}
# --
function check_version_string() {
local version="$1"
if ! echo "${version}" | grep -qE '^(main-|alpha-|beta-|stable-|lts-)' ; then
echo "ERROR: invalid version '${version}', must start with 'main-', 'alpha-', 'beta-', 'stable-', or 'lts-'"
exit 1
fi
}
# --
function update_submodules() {
local coreos_git="$1"
local portage_git="$2"
init_submodules
update_submodule "coreos-overlay" "${coreos_git}"
update_submodule "portage-stable" "${portage_git}"
}
# --
function update_and_push_version() {
local version="$1"
# Add and commit local changes
git add "sdk_container/src/third_party/coreos-overlay"
git add "sdk_container/src/third_party/portage-stable"
git add "sdk_container/.repo/manifests/version.txt"
git commit --allow-empty -m "New version: ${version}"
git tag -f "${version}"
if git push origin "${version}" ; then
return
fi
# Push (above) may fail because a tag already exists.
# We check for tag presence, and for the difference
# between local and remote, and bail
# only if the remote / local contents differ.
# Remove local tag, (re-)fetch remote tags
git tag -d "${version}"
# refresh tree, let origin overwrite local tags
git fetch --all --tags --force
# This will return != 0 if
# - the remote tag does not exist (rc: 127)
# - the remote tag has changes compared to the local tree (rc: 1)
git diff --exit-code "${version}"
}
# --
function copy_from_buildcache() {
local what="$1"
local where_to="$2"
mkdir -p "$where_to"
curl --verbose --fail --silent --show-error --location --retry-delay 1 --retry 60 \
--retry-connrefused --retry-max-time 60 --connect-timeout 20 \
--remote-name --output-dir "${where_to}" "https://${BUILDCACHE_SERVER}/${what}"
}
# --
function gen_sshcmd() {
echo -n "ssh -o BatchMode=yes"
echo -n " -o StrictHostKeyChecking=no"
echo -n " -o UserKnownHostsFile=/dev/null"
echo " -o NumberOfPasswordPrompts=0"
}
# --
function copy_to_buildcache() {
local remote_path="${BUILDCACHE_PATH_PREFIX}/$1"
shift
local sshcmd="$(gen_sshcmd)"
$sshcmd "${BUILDCACHE_USER}@${BUILDCACHE_SERVER}" \
"mkdir -p ${remote_path}"
rsync -Pav -e "${sshcmd}" "$@" \
"${BUILDCACHE_USER}@${BUILDCACHE_SERVER}:${remote_path}"
}
# --
function image_exists_locally() {
local name="$1"
local version="$2"
local image="${name}:${version}"
local image_exists="$(docker images "${image}" \
--no-trunc --format '{{.Repository}}:{{.Tag}}')"
[ "${image}" = "${image_exists}" ]
}
# --
# Derive docker-safe image version string from vernum.
#
function vernum_to_docker_image_version() {
local vernum="$1"
echo "$vernum" | sed 's/[+]/-/g'
}
# --
# Return the full name (repo+name+tag) of an image. Useful for SDK images
# pulled from the registry (which have the registry pre-pended)
function docker_image_fullname() {
local image="$1"
local version="$2"
docker images --no-trunc --format '{{.Repository}}:{{.Tag}}' \
| grep -E "^(${CONTAINER_REGISTRY}/)*${image}:${version}$"
}
# --
function docker_image_to_buildcache() {
local image="$1"
local version="$2"
# strip potential container registry prefix
local tarball="$(basename "$image")-${version}.tar.gz"
docker save "${image}":"${version}" | $PIGZ -c > "${tarball}"
copy_to_buildcache "containers/${version}" "${tarball}"
}
# --
function docker_commit_to_buildcache() {
local container="$1"
local image_name="$2"
local image_version="$3"
docker commit "${container}" "${image_name}:${image_version}"
docker_image_to_buildcache "${image_name}" "${image_version}"
}
# --
function docker_image_from_buildcache() {
local name="$1"
local version="$2"
local tgz="${name}-${version}.tar.gz"
if image_exists_locally "${name}" "${version}" ; then
return
fi
local url="https://${BUILDCACHE_SERVER}/containers/${version}/${tgz}"
curl --verbose --fail --silent --show-error --location --retry-delay 1 --retry 60 \
--retry-connrefused --retry-max-time 60 --connect-timeout 20 \
--remote-name "${url}"
cat "${tgz}" | $PIGZ -d -c | docker load
rm "${tgz}"
}
# --
function docker_image_from_registry_or_buildcache() {
local image="$1"
local version="$2"
if image_exists_locally "${CONTAINER_REGISTRY}/${image}" "${version}" ; then
return
fi
if docker pull "${CONTAINER_REGISTRY}/${image}:${version}" ; then
return
fi
docker_image_from_buildcache "${image}" "${version}"
}
# --

143
ci-automation/garbage_collect.sh Normal file
View File

@ -0,0 +1,143 @@
#!/bin/bash
#
# Copyright (c) 2021 The Flatcar Maintainers.
# Use of this source code is governed by a BSD-style license that can be
# found in the LICENSE file.
# >>> This file is supposed to be SOURCED from the repository ROOT. <<<
#
# garbage_collect() should be called after sourcing.
#
# OPTIONAL INPUT
# - Number of (recent) versions to keep. Defaults to 50.
# - PURGE_VERSIONS (Env variable). Space-separated list of versions to purge
# instead of all but the 50 most recent ones.
# - DRY_RUN (Env variable). Set to "y" to just list what would be done but not
# actually purge anything.
# Flatcar CI automation garbage collector.
# This script removes development (non-official) build artifacts:
# - SDK tarballs, build step containers, and vendor images on buildcache
# - tags from the scripts repository
#
# Garbage collection is based on development (non-official) version tags
# in the scripts repo. The newest 50 builds will be retained,
# all older builds will be purged (50 is the default, see OPTIONAL INPUT above).
set -eu
function garbage_collect() {
local keep="${1:-50}"
local dry_run="${DRY_RUN:-}"
local purge_versions="${PURGE_VERSIONS:-}"
local versions_detected="$(git tag -l --sort=-committerdate \
| grep -E '(main|alpha|beta|stable|lts|sdk)-[0-9]+\.[0-9]+\.[0-9]+\-.*' \
| grep -vE '(-pro)$')"
echo "######## Full list of version(s) found ########"
echo "${versions_detected}" | awk '{printf "%5d %s\n", NR, $0}'
if [ -z "${purge_versions}" ] ; then
keep="$((keep + 1))" # for tail -n+...
purge_versions="$(echo "${versions_detected}" \
| tail -n+"${keep}")"
else
# make sure we only accept dev versions
purge_versions="$(echo "${purge_versions}" | sed 's/ /\n/g' \
| grep -E '(main|alpha|beta|stable|lts|sdk)-[0-9]+\.[0-9]+\.[0-9]+\-.*' \
| grep -vE '(-pro)$')"
fi
source ci-automation/ci_automation_common.sh
local sshcmd="$(gen_sshcmd)"
echo
echo "######## The following version(s) will be purged ########"
if [ "$dry_run" = "y" ] ; then
echo
echo "(NOTE this is just a dry run since DRY_RUN=y)"
echo
fi
echo "${purge_versions}" | awk -v keep="${keep}" '{if ($0 == "") next; printf "%5d %s\n", NR + keep - 1, $0}'
echo
echo
local version
for version in ${purge_versions}; do
echo "--------------------------------------------"
echo
echo "#### Processing version '${version}' ####"
echo
git checkout "${version}" -- sdk_container/.repo/manifests/version.txt
source sdk_container/.repo/manifests/version.txt
local os_vernum="${FLATCAR_VERSION}"
local os_docker_vernum="$(vernum_to_docker_image_version "${FLATCAR_VERSION}")"
local sdk_vernum="${FLATCAR_SDK_VERSION}"
local sdk_docker_vernum="$(vernum_to_docker_image_version "${FLATCAR_SDK_VERSION}")"
# Remove container image tarballs and SDK tarball (if applicable)
#
local rmpat=""
if [[ "${version}" = 'sdk-'* ]] ; then
echo "## ${version} is an SDK version. ##"
rmpat="${BUILDCACHE_PATH_PREFIX}/sdk/*/${sdk_vernum}/"
rmpat="${rmpat} ${BUILDCACHE_PATH_PREFIX}/containers/${sdk_docker_vernum}/flatcar-sdk-*"
else
echo "## ${version} is an OS image version. ##"
rmpat="${BUILDCACHE_PATH_PREFIX}/containers/${os_docker_vernum}/flatcar-packages-*"
rmpat="${rmpat} ${BUILDCACHE_PATH_PREFIX}/containers/${os_docker_vernum}/flatcar-images-*"
rmpat="${rmpat} ${BUILDCACHE_PATH_PREFIX}/images/*/${os_vernum}/"
fi
echo "## The following files will be removed ##"
$sshcmd "${BUILDCACHE_USER}@${BUILDCACHE_SERVER}" \
"ls -la ${rmpat} || true"
if [ "$dry_run" != "y" ] ; then
set -x
$sshcmd "${BUILDCACHE_USER}@${BUILDCACHE_SERVER}" \
"rm -rf ${rmpat}"
set +x
else
echo "## (DRY_RUN=y so not doing anything) ##"
fi
# Remove container image directory if empty
#
if [[ "${version}" = 'sdk-'* ]] ; then
rmpat="${BUILDCACHE_PATH_PREFIX}/containers/${sdk_docker_vernum}/"
else
rmpat="${BUILDCACHE_PATH_PREFIX}/containers/${os_docker_vernum}/"
fi
echo "## Checking if container directory is empty and can be removed (it's OK if this fails) ##"
echo "## The following directory will be removed if below output is empty: '${rmpat}' ##"
$sshcmd "${BUILDCACHE_USER}@${BUILDCACHE_SERVER}" \
"ls -la ${rmpat} || true"
if [ "$dry_run" != "y" ] ; then
set -x
$sshcmd "${BUILDCACHE_USER}@${BUILDCACHE_SERVER}" \
"rmdir ${rmpat} || true"
set +x
else
echo "## (DRY_RUN=y so not doing anything) ##"
fi
# Remove git tag (local and remote)
#
echo "## The following TAG will be deleted: '${version}' ##"
if [ "$dry_run" != "y" ] ; then
set -x
git tag -d "${version}"
git push --delete origin "${version}"
set +x
else
echo "## (DRY_RUN=y so not doing anything) ##"
fi
done
}

68
ci-automation/image.sh Normal file
View File

@ -0,0 +1,68 @@
#!/bin/bash
#
# Copyright (c) 2021 The Flatcar Maintainers.
# Use of this source code is governed by a BSD-style license that can be
# found in the LICENSE file.
# >>> This file is supposed to be SOURCED from the repository ROOT. <<<
#
# image_build() should be called w/ the positional INPUT parameters below.
# Binary OS image build automation stub.
# This script will build the OS image from a pre-built packages container.
#
# PREREQUISITES:
#
# 1. SDK version and OS image version are recorded in sdk_container/.repo/manifests/version.txt
# 2. Scripts repo version tag of OS image version to be built is available and checked out.
# 3. Flatcar packages container is available via build cache server
# from "/containers/[VERSION]/flatcar-packages-[ARCH]-[FLATCAR_VERSION].tar.gz"
# or present locally. Container must contain binary packages and torcx artefacts.
#
# INPUT:
#
# 1. Architecture (ARCH) of the TARGET OS image ("arm64", "amd64").
#
# OUTPUT:
#
# 1. Exported container image with OS image, dev container, and related artifacts at
# /home/sdk/image/[ARCH], torcx packages at /home/sdk/torcx
# named "flatcar-images-[ARCH]-[FLATCAR_VERSION].tar.gz"
# pushed to buildcache.
# 2. "./ci-cleanup.sh" with commands to clean up temporary build resources,
# to be run after this step finishes / when this step is aborted.
set -eu
function image_build() {
local arch="$1"
source ci-automation/ci_automation_common.sh
init_submodules
source sdk_container/.repo/manifests/version.txt
local vernum="${FLATCAR_VERSION}"
local docker_vernum="$(vernum_to_docker_image_version "${vernum}")"
local packages="flatcar-packages-${arch}"
local packages_image="${packages}:${docker_vernum}"
docker_image_from_buildcache "${packages}" "${docker_vernum}"
local image="flatcar-images-${arch}"
local image_container="${image}-${docker_vernum}"
# build image and store it in the container
./run_sdk_container -x ./ci-cleanup.sh -n "${image_container}" -C "${packages_image}" \
-v "${vernum}" \
mkdir -p "${CONTAINER_IMAGE_ROOT}"
./run_sdk_container -n "${image_container}" -C "${packages_image}" \
-v "${vernum}" \
./build_image --board="${arch}-usr" \
--output_root="${CONTAINER_IMAGE_ROOT}" \
--torcx_root="${CONTAINER_TORCX_ROOT}" prodtar container
# rename container and push to build cache
docker_commit_to_buildcache "${image_container}" "${image}" "${docker_vernum}"
}
# --

116
ci-automation/packages.sh Normal file
View File

@ -0,0 +1,116 @@
#!/bin/bash
#
# Copyright (c) 2021 The Flatcar Maintainers.
# Use of this source code is governed by a BSD-style license that can be
# found in the LICENSE file.
# >>> This file is supposed to be SOURCED from the repository ROOT. <<<
#
# packages_build() should be called w/ the positional INPUT parameters below.
# OS image binary packages build automation stub.
# This script will use an SDK container to build packages for an OS image.
# It will update the versionfile with the OS packages version built,
# and will add a version tag (see INPUT) to the scripts repo.
#
# PREREQUISITES:
#
# 1. SDK version is recorded in sdk_container/.repo/manifests/version.txt
# 2. SDK container is either
# - available via ghcr.io/flatcar-linux/flatcar-sdk-[ARCH]:[VERSION] (official SDK release)
# OR
# - available via build cache server "/containers/[VERSION]/flatcar-sdk-[ARCH]-[VERSION].tar.gz"
# (dev SDK)
#
# INPUT:
#
# 1. Version of the TARGET OS image to build (string).
# The version pattern '(alpha|beta|stable|lts)-MMMM.m.p' (e.g. 'alpha-3051.0.0')
# denotes a "official" build, i.e. a release build to be published.
# Use any version diverging from the pattern (e.g. 'alpha-3051.0.0-nightly-4302') for development / CI builds.
# A tag of this version will be created in the scripts repo and pushed upstream.
#
# 2. Architecture (ARCH) of the TARGET OS image ("arm64", "amd64").
#
#
# OPTIONAL INPUT:
#
# 3. coreos-overlay repository tag to use (commit-ish).
# Optional - use scripts repo sub-modules as-is if not set.
# This version will be checked out / pulled from remote in the coreos-overlay git submodule.
# The submodule config will be updated to point to this version before the TARGET SDK tag is created and pushed.
#
# 4. portage-stable repository tag to use (commit-ish).
# Optional - use scripts repo sub-modules as-is if not set.
# This version will be checked out / pulled from remote in the portage-stable git submodule.
# The submodule config will be updated to point to this version before the TARGET SDK tag is created and pushed.
#
# OUTPUT:
#
# 1. Exported container image "flatcar-packages-[ARCH]-[VERSION].tar.gz" with binary packages
# pushed to buildcache.
# 2. Updated scripts repository
# - version tag w/ submodules
# - sdk_container/.repo/manifests/version.txt denotes new FLATCAR OS version
# 3. "./ci-cleanup.sh" with commands to clean up temporary build resources,
# to be run after this step finishes / when this step is aborted.
set -eu
function packages_build() {
local version="$1"
local arch="$2"
local coreos_git="${3:-}"
local portage_git="${4:-}"
source ci-automation/ci_automation_common.sh
init_submodules
check_version_string "${version}"
source sdk_container/.repo/manifests/version.txt
local sdk_version="${FLATCAR_SDK_VERSION}"
if [ -n "${coreos_git}" ] ; then
update_submodule "coreos-overlay" "${coreos_git}"
fi
if [ -n "${portage_git}" ] ; then
update_submodule "portage-stable" "${portage_git}"
fi
# Get SDK from either the registry or import from build cache
# This is a NOP if the image is present locally.
local sdk_name="flatcar-sdk-${arch}"
local docker_sdk_vernum="$(vernum_to_docker_image_version "${sdk_version}")"
docker_image_from_registry_or_buildcache "${sdk_name}" "${docker_sdk_vernum}"
local sdk_image="$(docker_image_fullname "${sdk_name}" "${docker_sdk_vernum}")"
echo "docker image rm -f '${sdk_image}'" >> ./ci-cleanup.sh
# Set name of the packages container for later rename / export
local vernum="${version#*-}" # remove main-,alpha-,beta-,stable-,lts- version tag
local docker_vernum="$(vernum_to_docker_image_version "${vernum}")"
local packages_container="flatcar-packages-${arch}-${docker_vernum}"
# Build packages; store packages and torcx output in container
./run_sdk_container -x ./ci-cleanup.sh -n "${packages_container}" -v "${version}" \
-C "${sdk_image}" \
mkdir -p "${CONTAINER_TORCX_ROOT}"
./run_sdk_container -n "${packages_container}" -v "${version}" \
-C "${sdk_image}" \
./build_packages --board="${arch}-usr" \
--torcx_output_root="${CONTAINER_TORCX_ROOT}"
# run_sdk_container updates the version file, use that version from here on
source sdk_container/.repo/manifests/version.txt
local vernum="${FLATCAR_VERSION}"
local docker_vernum="$(vernum_to_docker_image_version "${vernum}")"
local packages_image="flatcar-packages-${arch}"
# generate image + push to build cache
docker_commit_to_buildcache "${packages_container}" "${packages_image}" "${docker_vernum}"
update_and_push_version "${version}"
}
# --

90
ci-automation/sdk_bootstrap.sh Normal file
View File

@ -0,0 +1,90 @@
#!/bin/bash
#
# Copyright (c) 2021 The Flatcar Maintainers.
# Use of this source code is governed by a BSD-style license that can be
# found in the LICENSE file.
# >>> This file is supposed to be SOURCED from the repository ROOT. <<<
#
# sdk_bootstrap() should be called w/ the positional INPUT parameters below.
# Bootstrap SDK build automation stub.
# This script will use a seed SDK container + tarball to bootstrap a
# new SDK tarball.
#
# INPUT:
#
# 1. Version of the SEED SDK to use (string).
# The seed SDK tarball must be available on https://mirror.release.flatcar-linux.net/sdk/ ...
# The seed SDK container must be available from https://github.com/orgs/flatcar-linux/packages
# (via ghcr.io/flatcar-linux/flatcar-sdk-all:[VERSION]).
#
# 2. Version of the TARGET SDK to build (string).
# The version pattern 'MMMM.m.p' (e.g. '3051.0.0') denotes a "official" build, i.e. a release build to be published.
# Use any version diverging from the pattern (e.g. '3051.0.0-nightly-4302') for development / CI builds.
# A tag "sdk-[VERSION]" will be created in the scripts repo and pushed upstream.
#
# OPTIONAL INPUT:
#
# 3. coreos-overlay repository tag to use (commit-ish).
# This version will be checked out / pulled from remote in the coreos-overlay git submodule.
# The submodule config will be updated to point to this version before the TARGET SDK tag is created and pushed.
# Leave empty to use coreos-overlay as-is.
#
# 4. portage-stable repository tag to use (commit-ish).
# This version will be checked out / pulled from remote in the portage-stable git submodule.
# The submodule config will be updated to point to this version before the TARGET SDK tag is created and pushed.
# Leave empty to use portage-stable as-is.
#
# 5. ARCH. Environment variable. Target architecture for the SDK to run on.
# Either "amd64" or "arm64"; defaults to "amd64" if not set.
#
# OUTPUT:
#
# 1. SDK tarball (gentoo catalyst output) of the new SDK, pushed to buildcache.
# 2. Updated scripts repository
# - version tag w/ submodules
# - sdk_container/.repo/manifests/version.txt denotes new SDK version
# 3. "./ci-cleanup.sh" with commands to clean up temporary build resources,
# to be run after this step finishes / when this step is aborted.
set -eu
function sdk_bootstrap() {
local seed_version="$1"
local version="$2"
local coreos_git="$3"
local portage_git="$4"
: ${ARCH:="amd64"}
source ci-automation/ci_automation_common.sh
init_submodules
check_version_string "${version}"
if [ -n "${coreos_git}" ] ; then
update_submodule "coreos-overlay" "${coreos_git}"
fi
if [ -n "${portage_git}" ] ; then
update_submodule "portage-stable" "${portage_git}"
fi
local vernum="${version#*-}" # remove alpha-,beta-,stable-,lts- version tag
local git_vernum="${vernum}"
# This will update FLATCAR_VERSION[_ID] and BUILD_ID in versionfile
./bootstrap_sdk_container -x ./ci-cleanup.sh "${seed_version}" "${vernum}"
# push SDK tarball to buildcache
source sdk_container/.repo/manifests/version.txt
local vernum="${FLATCAR_SDK_VERSION}"
local dest_tarball="flatcar-sdk-${ARCH}-${vernum}.tar.bz2"
cd "__build__/images/catalyst/builds/flatcar-sdk"
copy_to_buildcache "sdk/${ARCH}/${vernum}" "${dest_tarball}"*
cd -
# Create new tag in scripts repo w/ updated versionfile + submodules
update_and_push_version "sdk-${git_vernum}"
}
# --

59
ci-automation/sdk_container.sh Normal file
View File

@ -0,0 +1,59 @@
#!/bin/bash
#
# Copyright (c) 2021 The Flatcar Maintainers.
# Use of this source code is governed by a BSD-style license that can be
# found in the LICENSE file.
# >>> This file is supposed to be SOURCED from the repository ROOT. <<<
#
# sdk_container_build() should be called w/ the positional INPUT parameters below.
# SDK container builder automation stub.
# This script will build an SDK container w/ board support from an SDK tarball.
# NOTE that SDK tarball and scripts repo must have the same version or building
# the SDK container will fail.
#
# PREREQUISITES:
#
# 1. SDK version is recorded in sdk_container/.repo/manifests/version.txt and a matching
# SDK tarball is available on BUILDCACHE/sdk/[ARCH]/[VERSION]/flatcar-sdk-[ARCH]-[VERSION].tar.bz2
#
# OPTIONAL INPUT:
# 2. ARCH. Environment variable. Target architecture for the SDK to run on.
# Either "amd64" or "arm64"; defaults to "amd64" if not set.
#
# OUTPUT:
#
# 1. SDK container image of the new SDK, published to buildcache.
# 2. "./ci-cleanup.sh" with commands to clean up temporary build resources,
# to be run after this step finishes / when this step is aborted.
set -eu
function sdk_container_build() {
: ${ARCH:="amd64"}
source ci-automation/ci_automation_common.sh
init_submodules
source sdk_container/.repo/manifests/version.txt
local vernum="${FLATCAR_SDK_VERSION}"
local sdk_tarball="flatcar-sdk-${ARCH}-${vernum}.tar.bz2"
# __build__ is in .dockerignore, so the tarball is excluded from build context
mkdir -p __build__
copy_from_buildcache "sdk/${ARCH}/${vernum}/${sdk_tarball}" "./__build__"
# This will update the SDK_VERSION in versionfile
./build_sdk_container_image -x ./ci-cleanup.sh ./__build__/"${sdk_tarball}"
# push artifacts to build cache
local docker_vernum="$(vernum_to_docker_image_version "${vernum}")"
docker_image_to_buildcache "${CONTAINER_REGISTRY}/flatcar-sdk-all" "${docker_vernum}"
docker_image_to_buildcache "${CONTAINER_REGISTRY}/flatcar-sdk-amd64" "${docker_vernum}"
docker_image_to_buildcache "${CONTAINER_REGISTRY}/flatcar-sdk-arm64" "${docker_vernum}"
}
# --

4
ci-automation/util/Dockerfile.pigz Normal file
View File

@ -0,0 +1,4 @@
FROM alpine
RUN apk add pigz
ENTRYPOINT [ "/usr/bin/pigz" ]
CMD [ "-h" ]

78
ci-automation/vms.sh Normal file
View File

@ -0,0 +1,78 @@
#!/bin/bash
#
# Copyright (c) 2021 The Flatcar Maintainers.
# Use of this source code is governed by a BSD-style license that can be
# found in the LICENSE file.
# >>> This file is supposed to be SOURCED from the repository ROOT. <<<
#
# vm_build() should be called w/ the positional INPUT parameters below.
# Vendor images build automation stub.
# This script will build one or more vendor images ("vm") using a pre-built image container.
#
# PREREQUISITES:
#
# 1. SDK version and OS image version are recorded in sdk_container/.repo/manifests/version.txt
# 2. Scripts repo version tag of OS image version to be built is available and checked out.
# 3. Flatcar image container is available via build cache server
# from "/containers/[VERSION]/flatcar-images-[ARCH]-[FLATCAR_VERSION].tar.gz"
# or present locally. Must contain packages and image.
#
# INPUT:
#
# 1. Architecture (ARCH) of the TARGET vm images ("arm64", "amd64").
# 2. Image formats to be built. Can be multiple, separated by spaces.
# Run ./image_to_vm.sh -h in the SDK to get a list of supported images.
#
# OUTPUT:
#
# 1. Exported VM image(s), pushed to buildcache ( images/[ARCH]/[FLATCAR_VERSION]/ )
# 2. "./ci-cleanup.sh" with commands to clean up temporary build resources,
# to be run after this step finishes / when this step is aborted.
set -eu
function vm_build() {
local arch="$1"
shift
# $@ now contains image formats to build
source ci-automation/ci_automation_common.sh
init_submodules
source sdk_container/.repo/manifests/version.txt
local vernum="${FLATCAR_VERSION}"
local docker_vernum="$(vernum_to_docker_image_version "${vernum}")"
local image="flatcar-images-${arch}"
local image_image="${image}:${docker_vernum}"
local vms_container="flatcar-vms-${docker_vernum}"
docker_image_from_buildcache "${image}" "${docker_vernum}"
# clean up dangling containers from previous builds
docker container rm -f "${vms_container}" || true
local images_out="images/"
rm -rf "${images_out}"
echo "docker container rm -f '${vms_container}'" >> ci-cleanup.sh
for format; do
echo " ################### VENDOR '${format}' ################### "
./run_sdk_container -n "${vms_container}" -C "${image_image}" \
-v "${vernum}" \
./image_to_vm.sh --format "${format}" --board="${arch}-usr" \
--from "${CONTAINER_IMAGE_ROOT}/${arch}-usr/latest"
done
# copy resulting images + push to buildcache
./run_sdk_container -n "${vms_container}" \
-v "${vernum}" \
cp --reflink=auto -R "${CONTAINER_IMAGE_ROOT}/${arch}-usr/" "./${images_out}/"
cd "images/latest"
copy_to_buildcache "images/${arch}/${vernum}/" *
}
# --

127
run_sdk_container Executable file
View File

@ -0,0 +1,127 @@
#!/bin/bash
#
# Copyright (c) 2021 The Flatcar Maintainers.
# Use of this source code is governed by a BSD-style license that can be
# found in the LICENSE file.
set -eu
cd $(dirname "$0")
source sdk_lib/sdk_container_common.sh
arch="all"
name=""
os_version="$(get_git_version)"
sdk_version="$(get_sdk_version_from_versionfile)"
custom_image=""
tty=""
cleanup=""
usage() {
echo " Usage:"
echo " $0 [-t] [-v <version>] [-V sdk version] [-a arch] [-n <name> ] [-x <script>] [-C custom-container] [-F] [container-command]"
echo " Start an SDK container of a given SDK release version."
echo " This will create the container if it does not exist, otherwise start the existing container."
echo " If the container is already running then it will exec into the container."
echo
echo " container-command - command to be run in the container instead of"
echo " an interactive shell."
echo " -t Attach docker to a TTY (docker -t)"
echo " -v <version> - Sourcetree (OS image) version to use."
echo " Defaults to '$os_version' (current git commit)."
echo " FLATCAR_VERSION[_ID] in '$sdk_container_common_versionfile'"
echo " will be updated accordingly (also see -F)."
echo " -V <SDK ver> - SDK version to use. Defaults to '${sdk_version}'"
echo " (FLATCAR_SDK_VERSION from '$sdk_container_common_versionfile')."
echo " -a <amd64|arm64|all> - Target architecture (board support) of the SDK."
echo " 'all' (the default) contains support for both amd64 and arm64."
echo " -n <name> - Custom name to use for the container."
echo " -x <script> - For each resource generated during build (container etc.)"
echo " add a cleanup line to <script> which, when run, will free"
echo " the resource. Useful for CI."
echo " -C - Use an entirely custom container image instead of the SDK's"
echo " $sdk_container_common_registry/flatcar-sdk-[ARCH]:[SDK VERSION]"
echo " Useful for CI."
echo " -h Print this help."
echo
}
# --
while [ 0 -lt $# ] ; do
case "$1" in
-h) usage; exit 0;;
-t) tty="-t"; shift;;
-v) os_version="$2"; shift; shift;;
-V) sdk_version="$2"; shift; shift;;
-a) arch="$2"; shift; shift;;
-n) name="$2"; shift; shift;;
-x) cleanup="$2"; shift; shift;;
-C) custom_image="$2"; shift; shift;;
*) break;;
esac
done
if [ -n "$custom_image" ] ; then
container_image_name="${custom_image}"
else
docker_sdk_vernum="$(vernum_to_docker_image_version "${sdk_version}")"
container_image_name="$sdk_container_common_registry/flatcar-sdk-${arch}:${docker_sdk_vernum}"
fi
create_versionfile "$sdk_version" "$os_version"
if [ -z "$name" ] ; then
docker_sdk_vernum="$(vernum_to_docker_image_version "${sdk_version}")"
docker_os_vernum="$(vernum_to_docker_image_version "${os_version}")"
name="flatcar-sdk-${arch}-${docker_sdk_vernum}_os-${docker_os_vernum}"
fi
stat="$(docker ps --all --no-trunc --filter name="$name" --format '{{.Status}}'\
| cut -f1 -d' ')"
# pass SDK related environment variables and gcloud auth
# into container
setup_sdk_env
setup_gsutil
mkdir -p "__build__/images"
mkdir -p "sdk_container/.cache/sdks"
hostname="${name:0:63}"
hostname="${hostname//./_}"
if [ -n "$cleanup" ] ; then
echo "docker container rm -f '${name}'" >> "$cleanup"
fi
if [ -z "$stat" ] ; then
yell "Creating a new container '$name'"
gpg_volumes=$(gnupg_ssh_gcloud_mount_opts)
docker create $tty -i \
-v /dev:/dev \
-v "$(pwd)/sdk_container:/mnt/host/source/" \
-v "$(pwd)/sdk_container/git-override/.git-coreos-overlay:/mnt/host/source/src/third_party/coreos-overlay/.git" \
-v "$(pwd)/sdk_container/git-override/.git-portage-stable:/mnt/host/source/src/third_party/portage-stable/.git" \
-v "$(pwd)/__build__/images:/mnt/host/source/src/build" \
-v "$(pwd):/mnt/host/source/src/scripts" \
$gpg_volumes \
--privileged \
--network host \
-e SDK_USER_ID="$(id -u)" \
-e SDK_GROUP_ID="$(id -g)" \
--name="$name" \
--hostname="$hostname" \
"${container_image_name}"
fi
if [ "$stat" != "Up" ] ; then
yell "Starting stopped container '$name'"
trap "docker stop $name" EXIT
docker start "$name"
fi
docker exec $tty -i "$name" /home/sdk/sdk_entry.sh "$@"

6
sdk_container/.repo/manifests/os-release Normal file
View File

@ -0,0 +1,6 @@
NAME="Flatcar Container Linux by Kinvolk"
ID=flatcar
ID_LIKE=coreos
ANSI_COLOR="38;5;75"
HOME_URL="https://flatcar-linux.org/"
BUG_REPORT_URL="https://issues.flatcar-linux.org"

4
sdk_container/.repo/manifests/version.txt Normal file
View File

@ -0,0 +1,4 @@
FLATCAR_VERSION=3066.0.0
FLATCAR_VERSION_ID=3066.0.0
FLATCAR_BUILD_ID=""
FLATCAR_SDK_VERSION=3066.0.0

1
sdk_container/git-override/.git-coreos-overlay Normal file
View File

@ -0,0 +1 @@
gitdir: ../../scripts/.git/modules/sdk_container/src/third_party/coreos-overlay

1
sdk_container/git-override/.git-portage-stable Normal file
View File

@ -0,0 +1 @@
gitdir: ../../scripts/.git/modules/sdk_container/src/third_party/portage-stable

5
sdk_container/git-override/README.md Normal file
View File

@ -0,0 +1,5 @@
# GIT overrides for submodules
In the SDK container, the scripts root is at a different relative path to the submodules.
The overrides in this directory are into `coreos-overlay/.git` and `portage-stable/.git` so the submodules can find their parents.

1
sdk_container/src/third_party/coreos-overlay vendored Submodule

@ -0,0 +1 @@
Subproject commit 551a2843c5463c27ccb5cc3c606edf6dfb15302b

1
sdk_container/src/third_party/portage-stable vendored Submodule

@ -0,0 +1 @@
Subproject commit 2cba555cfb7eedadff14bf52cad41e81d51d5664

8
sdk_lib/90_env_keep Normal file
View File

@ -0,0 +1,8 @@
Defaults env_keep += "FLATCAR_BUILD_ID COREOS_OFFICIAL \
EMAIL GIT_AUTHOR_EMAIL GIT_AUTHOR_NAME \
GIT_COMMITTER_EMAIL GIT_COMMITTER_NAME \
GIT_PROXY_COMMAND GIT_SSH RSYNC_PROXY \
GNUPGHOME GPG_AGENT_INFO SSH_AUTH_SOCK \
BOTO_PATH GOOGLE_APPLICATION_CREDENTIALS \
USE FEATURES PORTAGE_USERNAME FORCE_STAGES \
all_proxy ftp_proxy http_proxy https_proxy no_proxy"

29
sdk_lib/Dockerfile.lean-arch Normal file
View File

@ -0,0 +1,29 @@
ARG VERSION
FROM flatcar-sdk-build:${VERSION} as meta
ARG RMARCH
ARG RMCROSS
RUN if [ -n "$RMCROSS" ]; then \
sudo crossdev --clean --force "$RMCROSS"; \
fi
RUN if [ -n "$RMARCH" ]; then \
sudo rm -rf /build/$RMARCH; \
sudo rm -f /usr/local/bin/*-$RMARCH; \
fi
# Note: .repo/manifests/version.txt will survive this. That's intended.
RUN sudo rm -rf /mnt/host/source/*
FROM scratch
COPY --from=meta / /
COPY --from=meta --chown=sdk:sdk /home/sdk /home/sdk
RUN chown -R sdk:sdk /mnt/host/source
# This is not used when starting the container via ./run_sdk_container
# but it's useful for standalone container use.
RUN mkdir -p /mnt/host/source/src/scripts
COPY --chown=sdk:sdk sdk_lib/sdk_init_selfcontained.sh /mnt/host/source/src/
ENTRYPOINT /home/sdk/sdk_entry.sh

17
sdk_lib/Dockerfile.sdk-build Normal file
View File

@ -0,0 +1,17 @@
ARG VERSION
FROM flatcar-sdk-import:${VERSION}
ARG BINHOST
ARG OFFICIAL=0
# mark build as official where appropriate
RUN echo "export COREOS_OFFICIAL=$OFFICIAL" > /mnt/host/source/.env
RUN /home/sdk/sdk_entry.sh ./setup_board --board="arm64-usr" --binhost="${BINHOST}/arm64-usr"
RUN /home/sdk/sdk_entry.sh ./setup_board --board="arm64-usr" --regen_configs
RUN /home/sdk/sdk_entry.sh ./setup_board --board="amd64-usr" --binhost="${BINHOST}/amd64-usr"
RUN /home/sdk/sdk_entry.sh ./setup_board --board="amd64-usr" --regen_configs
RUN rm /mnt/host/source/.env
RUN rm -rf /home/sdk/toolchain-pkgs

53
sdk_lib/Dockerfile.sdk-import Normal file
View File

@ -0,0 +1,53 @@
ARG VERSION
FROM flatcar-sdk-tarball:${VERSION}
# Make build scripts believe we're in the SDK chroot (which technically, we are)
RUN touch /etc/debian_chroot
RUN chmod 644 /etc/passwd
RUN chmod 644 /etc/group
RUN if ! grep -q portage /etc/group ; then \
echo "portage::250:portage" >>/etc/group; \
fi
RUN if ! grep -q portage /etc/passwd; then \
echo "portage:x:250:250:portage:/var/tmp/portage:/bin/false" >>/etc/passwd; \
fi
# fix "Unable to unshare: EPERM ..." in containers
# (see https://github.com/gentoo/gentoo-docker-images/issues/81)
RUN echo 'export FEATURES="-ipc-sandbox -network-sandbox -pid-sandbox"' \
>> /etc/skel/.bashrc
RUN groupadd sdk
RUN useradd -g sdk -G portage sdk
RUN echo "sdk ALL=(ALL) NOPASSWD: ALL" > /etc/sudoers.d/sdk-user
RUN rm -rf /mnt/host/source
COPY --chown=sdk:sdk sdk_container/ /mnt/host/source
COPY --chown=sdk:sdk . /mnt/host/source/src/scripts
RUN chown sdk:sdk /mnt/host/source
COPY sdk_lib/90_env_keep /etc/sudoers.d/90_env_keep
RUN chmod 0440 /etc/sudoers.d/90_env_keep
USER sdk:sdk
RUN mkdir -p /mnt/host/source/src/scripts /mnt/host/source/src/build
RUN ln -s /mnt/host/source /home/sdk/trunk
RUN rm /home/sdk/.bashrc
RUN cp /etc/skel/.bashrc /home/sdk
RUN echo "cd /home/sdk/trunk/src/scripts" >> /home/sdk/.bashrc
RUN echo 'export PATH="$PATH:/usr/local/bin:/usr/local/sbin"' >> /home/sdk/.bashrc
# user and SDK environment variables pass-through into container
RUN echo "if [ -f /mnt/host/source/.env ]; then source /mnt/host/source/.env; fi" >> /home/sdk/.bashrc
RUN echo "if [ -f /mnt/host/source/.sdkenv ]; then source /mnt/host/source/.sdkenv; fi" >> /home/sdk/.bashrc
COPY --chown=sdk:sdk sdk_lib/sdk_entry.sh /home/sdk
RUN chmod 755 /home/sdk/sdk_entry.sh
# This should be a NOP; if you see packages being rebuilt
# it's likely that submodules and SDK tarball are out of sync
RUN /home/sdk/sdk_entry.sh ./update_chroot --toolchain_boards="amd64-usr arm64-usr"
ENTRYPOINT /home/sdk/sdk_entry.sh

240
sdk_lib/sdk_container_common.sh Normal file
View File

@ -0,0 +1,240 @@
# Copyright (c) 2021 The Flatcar Maintainers.
# Use of this source code is governed by a BSD-style license that can be
# found in the LICENSE file.
#
# This file contains common functions used across SDK container scripts.
#
# globals
#
sdk_container_common_versionfile="sdk_container/.repo/manifests/version.txt"
sdk_container_common_registry="ghcr.io/flatcar-linux"
sdk_container_common_env_file="sdk_container/.sdkenv"
# Common "echo" function
function yell() {
echo -e "\n###### $@ ######"
}
# --
# Guess the SDK version from the current git commit.
#
function get_git_version() {
local tag="$(git tag --points-at HEAD)"
if [ -z "$tag" ] ; then
git describe --tags
else
echo "$tag"
fi
}
# --
function get_sdk_version_from_versionfile() {
( source "$sdk_container_common_versionfile"; echo "$FLATCAR_SDK_VERSION"; )
}
# --
# return true if a given version number is an official build
#
function is_official() {
local vernum="$1"
local official="$(echo "$vernum" | sed -n 's/^[0-9]\+\.[0-9]\+\.[0-9]\+$/true/p')"
test -n "$official"
}
# --
# extract the build ID suffix from a version string ("alpha-3244.0.1-nightly2" => "nightly2")
#
function build_id_from_version() {
local version="$1"
# support vernums and versions ("alpha-"... is optional)
echo "${version}" | sed -n 's/^\([a-z]\+-\)\?[0-9.]\+[-+]\(.*\)$/\2/p'
}
# --
# extract the version number (w/o build ID) from a version string ("alpha-3244.0.1-nightly2" => "3244.0.1")
#
function vernum_from_version() {
local version="$1"
# support vernums and versions ("alpha-"... is optional)
echo "${version}" | sed -n 's/^\([a-z]\+-\)\?\([0-9.]\+\).*/\2/p'
}
# --
# Strip prefix from version string if present ("alpha-3233.0.0[-...]" => "3233.0.0[-...]")
# and add a "+[build suffix]" if this is a non-official build. The "+" matches the version
# string generation in the build scripts.
function strip_version_prefix() {
local version="$1"
local build_id="$(build_id_from_version "${version}")"
local version_id="$(vernum_from_version "${version}")"
if [ -n "${build_id}" ] ; then
echo "${version_id}+${build_id}"
else
echo "${version_id}"
fi
}
# --
# Derive docker-safe image version string from vernum.
#
function vernum_to_docker_image_version() {
local vernum="$1"
echo "$vernum" | sed 's/[+]/-/g'
}
# --
# Creates the Flatcar build / SDK version file.
# Must be called from the script root.
#
# In the versionfile, FLATCAR_VERSION is the OS image version _number_ plus a build ID if this is no
# official build. The FLATCAR_VERSION_ID is the plain vernum w/o build ID - it's the same as FLATCAR_VERSION
# for official builds. The FLATCAR_BUILD_ID is the build ID suffix for non-official builds.
# Lastly, the FLATCAR_SDK_VERSION is the full version number (including build ID if no official SDK release)
# the OS image is to be built with.
#
function create_versionfile() {
local sdk_version="$1"
local os_version="${2:-$sdk_version}"
local build_id="$(build_id_from_version "${os_version}")"
local version_id="$(vernum_from_version "${os_version}")"
sdk_version="$(strip_version_prefix "${sdk_version}")"
os_version="$(strip_version_prefix "${os_version}")"
yell "Writing versionfile '$sdk_container_common_versionfile' to SDK '$sdk_version', OS '$os_version'."
cat >"$sdk_container_common_versionfile" <<EOF
FLATCAR_VERSION=${os_version}
FLATCAR_VERSION_ID=${version_id}
FLATCAR_BUILD_ID="${build_id}"
FLATCAR_SDK_VERSION=${sdk_version}
EOF
}
# --
#
# Set up SDK environment variables.
# Environment vars are put in a file that is sourced by the container's
# .bashrc (if present). GNUPGHOME and SSH_AUTH_SOCK are set
# to container-specific paths if applicable.
function setup_sdk_env() {
local var
rm -f "$sdk_container_common_env_file"
# conditionally set up gnupg, ssh socket, and gcloud auth / boto
# depending on availability on the host
GNUPGHOME="${GNUPGHOME:-$HOME/.gnupg}"
if [ -d "${GNUPGHOME}" ] ; then
echo "GNUPGHOME=\"/home/sdk/.gnupg\"" >> "$sdk_container_common_env_file"
echo "export GNUPGHOME" >> "$sdk_container_common_env_file"
export GNUPGHOME
fi
if [ -e "${SSH_AUTH_SOCK:-}" ] ; then
local sockname="$(basename "${SSH_AUTH_SOCK}")"
echo "SSH_AUTH_SOCK=\"/run/sdk/ssh/$sockname\"" >> "$sdk_container_common_env_file"
echo "export SSH_AUTH_SOCK" >> "$sdk_container_common_env_file"
fi
# keep in sync with 90_env_keep
for var in FLATCAR_BUILD_ID COREOS_OFFICIAL \
EMAIL GIT_AUTHOR_EMAIL GIT_AUTHOR_NAME \
GIT_COMMITTER_EMAIL GIT_COMMITTER_NAME \
GIT_PROXY_COMMAND GIT_SSH RSYNC_PROXY \
GPG_AGENT_INFO FORCE_STAGES \
all_proxy ftp_proxy http_proxy https_proxy no_proxy; do
if [ -n "${!var:-}" ] ; then
echo "${var}=\"${!var}\"" >> "$sdk_container_common_env_file"
echo "export ${var}" >> "$sdk_container_common_env_file"
fi
done
}
# --
# Set up gcloud legacy creds (via GOOGLE_APPLICATION_CREDENTIALS)
# for the SDK container.
# This will also create a boto config right next to the
# GOOGLE_APPLICATION_CREDENTIALS json file.
function setup_gsutil() {
local creds="${GOOGLE_APPLICATION_CREDENTIALS:-$HOME/.config/gcloud/application_default_credentials.json}"
if [ ! -e "$creds" ]; then
return
fi
local creds_dir="$(dirname "$creds")"
local botofile="$creds_dir/boto-flatcar-sdk"
# TODO t-lo: move generation of boto file to sdk_entry so
# it's only created inside the container.
# read creds file and create boto file for gsutil
local tmp="$(mktemp)"
trap "rm -f '$tmp'" EXIT
local oauth_refresh="$(jq -r '.refresh_token' "$creds")"
local client_id="$(jq -r '.client_id' "$creds")"
local client_secret="$(jq -r '.client_secret' "$creds")"
cat >>"$tmp" <<EOF
[Credentials]
gs_oauth2_refresh_token = $oauth_refresh
[OAuth2]
client_id = $client_id
client_secret = $client_secret
EOF
mv "$tmp" "$botofile"
echo "BOTO_PATH=\"$botofile\"" >> "$sdk_container_common_env_file"
echo "export BOTO_PATH" >> "$sdk_container_common_env_file"
echo "GOOGLE_APPLICATION_CREDENTIALS=\"$creds\"" >> "$sdk_container_common_env_file"
echo "export GOOGLE_APPLICATION_CREDENTIALS" >> "$sdk_container_common_env_file"
BOTO_PATH="$botofile"
GOOGLE_APPLICATION_CREDENTIALS="$creds"
export BOTO_PATH
export GOOGLE_APPLICATION_CREDENTIALS
}
# --
# Generate volume mount command line options for docker
# to pass gpg, ssh, and gcloud auth host directories
# into the SDK container.
function gnupg_ssh_gcloud_mount_opts() {
local sdk_gnupg_home="/home/sdk/.gnupg"
local gpgagent_dir="/run/user/$(id -u)/gnupg"
# pass host GPG home and Agent directories to container
if [ -d "$GNUPGHOME" ] ; then
echo "-v $GNUPGHOME:$sdk_gnupg_home"
fi
if [ -d "$gpgagent_dir" ] ; then
echo "-v $gpgagent_dir:$gpgagent_dir"
fi
if [ -e "${SSH_AUTH_SOCK:-}" ] ; then
local sshsockdir="$(dirname "$SSH_AUTH_SOCK")"
echo "-v $sshsockdir:/run/sdk/ssh"
fi
if [ -e "${GOOGLE_APPLICATION_CREDENTIALS:-}" ] ; then
local creds_dir="$(dirname "${GOOGLE_APPLICATION_CREDENTIALS}")"
if [ -d "$creds_dir" ] ; then
echo "-v $creds_dir:$creds_dir"
fi
fi
}

37
sdk_lib/sdk_entry.sh Executable file
View File

@ -0,0 +1,37 @@
#!/bin/bash
if [ -n "${SDK_USER_ID:-}" ] ; then
usermod -u $SDK_USER_ID sdk
fi
if [ -n "${SDK_GROUP_ID:-}" ] ; then
groupmod -g $SDK_GROUP_ID sdk
fi
chown -R sdk:sdk /home/sdk
# This is ugly.
# We need to sudo su - sdk -c so the SDK user gets a fresh login.
# 'sdk' is member of multiple groups, and plain docker USER only
# allows specifying membership of a single group.
# When a command is passed to the container, we run, respectively:
# sudo su - sdk -c "<command>".
# Then, we need to preserve whitespaces in arguments of commands
# passed to the container, e.g.
# ./update_chroot --toolchain_boards="amd64-usr arm64-usr".
# This is done via a separate ".cmd" file since we have used up
# our quotes for su -c "<cmd>" already.
if [ $# -gt 0 ] ; then
cmd="/home/sdk/.cmd"
echo -n "exec bash -i -c '" >"$cmd"
for arg in "$@"; do
echo -n "\"$arg\" " >>"$cmd"
done
echo "'" >>"$cmd"
chmod 755 "$cmd"
sudo su sdk -c "$cmd"
rc=$?
rm -f "$cmd"
exit $rc
else
exec sudo su sdk
fi

36
sdk_lib/sdk_init_selfcontained.sh Executable file
View File

@ -0,0 +1,36 @@
#!/bin/bash
echo "This script will initialise your Flatcar SDK container as a self-contained SDK."
echo "Please note that the preferred way of using the Flatcar SDK container is by cloning"
echo " https://github.com/flatcar-linux/scripts"
echo "and using the ./run_sdk_container script."
echo
echo "Press [RETURN] to continue, CTRL+C to abort"
echo
read junk
unset junk
# --
function clone_version() {
local repo="$1"
local dest="$2"
local version="$3"
git clone https://github.com/flatcar-linux/$repo "$dest"
git -C "${dest}" fetch --all
local tag=$(git -C "${dest}" tag -l | grep "${version}")
git -C "${dest}" checkout "$tag"
}
# --
version="$(source /mnt/host/source/.repo/manifests/version.txt; echo $FLATCAR_VERSION)"
mkdir -p /home/sdk/trunk/src/third_party/
clone_version scripts /home/sdk/trunk/src/scripts "$version"
clone_version portage-stable /home/sdk/trunk/src/third_party/portage-stable "$version"
clone_version coreos-overlay /home/sdk/trunk/src/third_party/coreos-overlay "$version"

5
setup_board
View File

@ -27,6 +27,8 @@ DEFINE_boolean getbinpkg "${FLAGS_TRUE}" \
"Download binary packages from remote repository." "Download binary packages from remote repository."
DEFINE_string getbinpkgver "" \ DEFINE_string getbinpkgver "" \
"Use binary packages from a specific version." "Use binary packages from a specific version."
DEFINE_string pkgdir "" \
"Use binary packages from a custom directory instead of /build/[ARCH]/var/lib/portage/pkgs/."
DEFINE_string binhost "" \ DEFINE_string binhost "" \
"Use binary packages from a specific location instead of $FLATCAR_DEV_BUILDS/... " "Use binary packages from a specific location instead of $FLATCAR_DEV_BUILDS/... "
DEFINE_boolean toolchainpkgonly "${FLAGS_FALSE}" \ DEFINE_boolean toolchainpkgonly "${FLAGS_FALSE}" \
@ -160,6 +162,7 @@ BOARD_ARCH=$(get_board_arch "$BOARD")
BOARD_CHOST=$(get_board_chost ${BOARD}) BOARD_CHOST=$(get_board_chost ${BOARD})
PORTAGE_PROFILE=$(get_board_profile "$BOARD") PORTAGE_PROFILE=$(get_board_profile "$BOARD")
BOARD_BINHOST="$FLAGS_binhost $(generate_binhost_list)" BOARD_BINHOST="$FLAGS_binhost $(generate_binhost_list)"
BOARD_PKGDIR="${FLAGS_pkgdir:-${BOARD_ROOT}/var/lib/portage/pkgs}"
if [[ ${FLAGS_regen_configs_only} -eq ${FLAGS_TRUE} ]]; then if [[ ${FLAGS_regen_configs_only} -eq ${FLAGS_TRUE} ]]; then
FLAGS_regen_configs=${FLAGS_TRUE} FLAGS_regen_configs=${FLAGS_TRUE}
@ -257,7 +260,7 @@ PORTAGE_USERNAME="$(portageq envvar PORTAGE_USERNAME)"
# Board specific settings # Board specific settings
CHOST="${BOARD_CHOST}" CHOST="${BOARD_CHOST}"
ROOT="${BOARD_ROOT}/" ROOT="${BOARD_ROOT}/"
PKGDIR="${BOARD_ROOT}/var/lib/portage/pkgs" PKGDIR="${BOARD_PKGDIR}"
PORT_LOGDIR="${BOARD_ROOT}/var/log/portage" PORT_LOGDIR="${BOARD_ROOT}/var/log/portage"
PORTAGE_TMPDIR="${BOARD_ROOT}/var/tmp" PORTAGE_TMPDIR="${BOARD_ROOT}/var/tmp"
PORTAGE_BINHOST="${BOARD_BINHOST}" PORTAGE_BINHOST="${BOARD_BINHOST}"