mirrors/flatcar-scripts
1
0
Fork 0
mirror of https://github.com/flatcar/scripts.git synced 2025-10-31 08:11:03 +01:00
Code Issues Packages Projects Releases Wiki Activity

sdk-container: add @krnowak's suggestions from code review

Browse Source 
Co-authored-by: Krzesimir Nowak <knowak@microsoft.com>
This commit is contained in:
Thilo Fromm 2021-11-05 09:22:25 +01:00 committed by flatcar-ci
parent 7f874e491e
commit 1d1c6048d4
14 changed files with 27 additions and 29 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

19
README.md
View File

@ -3,8 +3,8 @@
Welcome to the scripts repo, your starting place for most things here in the Flatcar Container Linux SDK. To get started you can find our documentation on [the Flatcar docs website][flatcar-docs]. Welcome to the scripts repo, your starting place for most things here in the Flatcar Container Linux SDK. To get started you can find our documentation on [the Flatcar docs website][flatcar-docs].
The SDK can be used to The SDK can be used to
* patch or update applications or libraries included in the Flatcar OS image * Patch or update applications or libraries included in the Flatcar OS image
* add or remove applications and / or libraries * Add or remove applications and / or libraries
* Modify the kernel configuration and add or remove kernel modules included with Flatcar * Modify the kernel configuration and add or remove kernel modules included with Flatcar
* Build OS images for a variety of targets (qemu, bare metal, AWS, Azure, VMWare, etc.) * Build OS images for a variety of targets (qemu, bare metal, AWS, Azure, VMWare, etc.)
* And lastly, the SDK can be used to upgrade SDK packages and to build new SDKs * And lastly, the SDK can be used to upgrade SDK packages and to build new SDKs
@ -14,7 +14,7 @@ The SDK can be used to
# Using the scripts repository: submodules and tags # Using the scripts repository: submodules and tags
The repository is meant to be the entry point for Flatcar builds and development. The repository is meant to be the entry point for Flatcar builds and development.
For building pakages, there are 2 addigional repositories, [coreos-overlay](https://github.com/flatcar-linux/) and [portage-stable](https://github.com/flatcar-linux/portage-stable), which contain all packages' `ebuild` (build configuration) files. For building packages, there are 2 additional repositories, [coreos-overlay](https://github.com/flatcar-linux/) and [portage-stable](https://github.com/flatcar-linux/portage-stable), which contain all packages' `ebuild` (build configuration) files.
These repositories are included in `scripts` via git submodules and are used by the SDK container wrapper scripts detailed on further below. These repositories are included in `scripts` via git submodules and are used by the SDK container wrapper scripts detailed on further below.
The submodules reside in: The submodules reside in:
``` ```
@ -28,10 +28,7 @@ scripts
When working with the scripts repo always make sure to initialise and to update these submodules; otherwise builds will break because build configuration is missing: When working with the scripts repo always make sure to initialise and to update these submodules; otherwise builds will break because build configuration is missing:
```bash ```bash
$ git clone https://github.com/flatcar-linux/scripts.git $ git clone --recurse-submodules https://github.com/flatcar-linux/scripts.git
$ cd scripts
$ git submodule init
$ git submodule update
``` ```
The `scripts` repository makes ample use of tags to mark releases. The `scripts` repository makes ample use of tags to mark releases.
@ -55,9 +52,9 @@ We provide a containerised SDK via https://github.com/orgs/flatcar-linux/package
* ARM64 SDK initialised for building ARM64 OS images on AMD64 hosts. Also about 6GB in size. * ARM64 SDK initialised for building ARM64 OS images on AMD64 hosts. Also about 6GB in size.
While work on a native ARM64 native SDK is ongoing, it's unfortunately not ready yet. If you want to help, patches are welcome! While work on a native ARM64 native SDK is ongoing, it's unfortunately not ready yet. If you want to help, patches are welcome!
The container can be run in one of two ways - "standalone", or integrated with the [Scripts](https://github.com/flatcar-linux/scripts) repo: The container can be run in one of two ways - "standalone", or integrated with the [scripts](https://github.com/flatcar-linux/scripts) repo:
* Standalone mode will use no host volumes and will allow you to play with the SDK in a sandboxed throw-away environment. In standalone mode, you interface with Docker directly to use the SDK container. * Standalone mode will use no host volumes and will allow you to play with the SDK in a sandboxed throw-away environment. In standalone mode, you interface with Docker directly to use the SDK container.
* Integrated mode will closely integrate with the Scripts directory and bind-mount it as well as the portage-stable and coreos-overlay gitmodules into the container. Integrated mode uses wrapper scripts to interact with the SDK container. This is the recommended way for developing patches for Flatcar. * Integrated mode will closely integrate with the scripts repo directory and bind-mount it as well as the portage-stable and coreos-overlay gitmodules into the container. Integrated mode uses wrapper scripts to interact with the SDK container. This is the recommended way for developing patches for Flatcar.
## Standalone mode ## Standalone mode
@ -124,7 +121,7 @@ Check out `./run_sdk_container -h` for more information on command line options.
Building an SDK container is done using `./build_sdk_container_image <tarball>`. Building an SDK container is done using `./build_sdk_container_image <tarball>`.
The tarball input is the result of an SDK bootstrap (see below). Version information for both OS as well as for the SDK will be extracted from the tarball name. The tarball input is the result of an SDK bootstrap (see below). Version information for both OS as well as for the SDK will be extracted from the tarball name.
The version file will be updated accordingly before the SDK container is built. The version file will be updated accordingly before the SDK container is built.
During the build, toolchain packages will be built and installed into the SDK container image. Both supported boards (`amd64-usr` and `arm64-usr`) will be initialisedin the container image. During the build, toolchain packages will be built and installed into the SDK container image. Both supported boards (`amd64-usr` and `arm64-usr`) will be initialised in the container image.
# Bootstrapping a new SDK tarball using the SDK container # Bootstrapping a new SDK tarball using the SDK container
@ -132,4 +129,4 @@ The script `./bootstrap_sdk_container` bootstraps a new SDK tarball using an exi
# Automation stubs for continuous integration # Automation stubs for continuous integration
Script stubs for various build stages can be found in the [ci-automation](ci-automation) folder. THese are helpful for glueing Flatcar Container Linux builds to a continuous integration system. Script stubs for various build stages can be found in the [ci-automation](ci-automation) folder. These are helpful for gluing Flatcar Container Linux builds to a continuous integration system.

2
bootstrap_sdk_container
View File

@ -36,7 +36,7 @@ usage() {
echo " -x <cleanup-script> - For each resource generated during build (container etc.)" echo " -x <cleanup-script> - For each resource generated during build (container etc.)"
echo " add a cleanup line to <script> which, when run, will free" echo " add a cleanup line to <script> which, when run, will free"
echo " the resource. Useful for CI." echo " the resource. Useful for CI."
echo " -h Print this help." echo " -h - Print this help."
echo echo
} }
# -- # --

4
build_sdk_container_image
View File

@ -43,7 +43,7 @@ usage() {
echo " This script will set up a new SDK container from a tarball." echo " This script will set up a new SDK container from a tarball."
echo " The resulting container comes in 3 flavours:" echo " The resulting container comes in 3 flavours:"
echo " 1. flatcar-sdk-all - includes both ARM64 and AMD64 support" echo " 1. flatcar-sdk-all - includes both ARM64 and AMD64 support"
echo " 2.+3. flatcar-sdk-(amd64|arm64) - only includes suport for one target." echo " 2.+3. flatcar-sdk-(amd64|arm64) - only includes support for one target."
echo " Usage:" echo " Usage:"
echo " $0 <tarball>] [-k] [-v <os-version>] [-x <script>]" echo " $0 <tarball>] [-k] [-v <os-version>] [-x <script>]"
echo echo
@ -78,7 +78,7 @@ while [ 0 -lt $# ] ; do
esac esac
done done
if [ -z "$tarball" -o ! -f "$tarball" ] ; then if [ -z "$tarball" -o ! -s "$tarball" ] ; then
echo "ERROR: missing / invalid SDK tarball argument" echo "ERROR: missing / invalid SDK tarball argument"
exit 1 exit 1
fi fi

8
ci-automation/README.md
View File

@ -7,11 +7,11 @@ Notable exceptions are "SDK Bootstrap" (`sdk.sh`) which only creates an SDK tarb
Please refer to the individual scripts for prerequisites, input parameters, and outputs. Please refer to the individual scripts for prerequisites, input parameters, and outputs.
All steps make use of a "build cache" server for pulling (https) and pushing (rsync) build inputs and artefacts. All steps make use of a "build cache" server for pulling (https) and pushing (rsync) build inputs and artifacts.
## Build steps ## Build steps
The build pipeline can be used to build everything from scratch, including the SDK (starting from 1. below) orto build a new OS image (starting from 3.). The build pipeline can be used to build everything from scratch, including the SDK (starting from 1. below) or to build a new OS image (starting from 3.).
### SDK bootstrap build ### SDK bootstrap build
@ -19,7 +19,7 @@ The build pipeline can be used to build everything from scratch, including the S
The resulting SDK tarball will use packages and versions pinned in the coreos-overlay and portage-stable submodules. The resulting SDK tarball will use packages and versions pinned in the coreos-overlay and portage-stable submodules.
This step updates the versionfile, recording the SDK container version just built. This step updates the versionfile, recording the SDK container version just built.
It will generate and push a new version tag to the scripts repo. It will generate and push a new version tag to the scripts repo.
2. SDK container build (`sdk_container.sh`) : use SDK tarball and to build an SDK container image. 2. SDK container build (`sdk_container.sh`) : use SDK tarball to build an SDK container image.
The resulting image will come in "amd64", "arm64", and "all" flavours, with support for respective OS target architectures. This step builds the Flatcar SDK container images published at ghcr.io/flatcar-linux. The resulting image will come in "amd64", "arm64", and "all" flavours, with support for respective OS target architectures. This step builds the Flatcar SDK container images published at ghcr.io/flatcar-linux.
``` ```
@ -52,7 +52,7 @@ The build pipeline can be used to build everything from scratch, including the S
This step updates the versionfile, recording the Flatcar OS image version just built. This step updates the versionfile, recording the Flatcar OS image version just built.
It will generate and push a new version tag to the scripts repo. It will generate and push a new version tag to the scripts repo.
4. Image build (`image.sh`): Using the container from 3., build an OS image and torcx store, and generate a new container image with everything in it. 4. Image build (`image.sh`): Using the container from 3., build an OS image and torcx store, and generate a new container image with everything in it.
5. VMs build (`vms.sh`). Using the packages+torcx+image container from 4., build vendor images. Result are vendor-specific OS images. 5. VMs build (`vms.sh`). Using the packages+torcx+image container from 4., build vendor images. Results are vendor-specific OS images.
``` ```
.---------. .------------. .---------. .------------.

2
ci-automation/ci-config.env
View File

@ -19,6 +19,6 @@ PIGZ="docker run --rm -i ghcr.io/flatcar-linux/pigz --fast"
CI_GIT_AUTHOR="flatcar-ci" CI_GIT_AUTHOR="flatcar-ci"
CI_GIT_EMAIL="infra+ci@flatcar-linux.org" CI_GIT_EMAIL="infra+ci@flatcar-linux.org"
# build artefacts go here (in container) # build artifacts go here (in container)
CONTAINER_TORCX_ROOT="/home/sdk/build/torcx" CONTAINER_TORCX_ROOT="/home/sdk/build/torcx"
CONTAINER_IMAGE_ROOT="/home/sdk/build/images" CONTAINER_IMAGE_ROOT="/home/sdk/build/images"

2
ci-automation/garbage_collect.sh
View File

@ -16,7 +16,7 @@
# actually purge anything. # actually purge anything.
# Flatcar CI automation garbage collector. # Flatcar CI automation garbage collector.
# This script removes development (non-official) build artefacts: # This script removes development (non-official) build artifacts:
# - SDK tarballs, build step containers, and vendor images on buildcache # - SDK tarballs, build step containers, and vendor images on buildcache
# - tags from the scripts repository # - tags from the scripts repository
# #

4
ci-automation/image.sh
View File

@ -6,7 +6,7 @@
# >>> This file is supposed to be SOURCED from the repository ROOT. <<< # >>> This file is supposed to be SOURCED from the repository ROOT. <<<
# #
# packages_build() should be called w/ the positional INPUT parameters below. # image_build() should be called w/ the positional INPUT parameters below.
# Binary OS image build automation stub. # Binary OS image build automation stub.
# This script will build the OS image from a pre-built packages container. # This script will build the OS image from a pre-built packages container.
@ -25,7 +25,7 @@
# #
# OUTPUT: # OUTPUT:
# #
# 1. Exported container image with OS image, dev container, and related artefacts at # 1. Exported container image with OS image, dev container, and related artifacts at
# /home/sdk/image/[ARCH], torcx packages at /home/sdk/torcx # /home/sdk/image/[ARCH], torcx packages at /home/sdk/torcx
# named "flatcar-images-[ARCH]-[FLATCAR_VERSION].tar.gz" # named "flatcar-images-[ARCH]-[FLATCAR_VERSION].tar.gz"
# pushed to buildcache. # pushed to buildcache.

2
ci-automation/sdk_bootstrap.sh
View File

@ -8,7 +8,7 @@
# #
# sdk_bootstrap() should be called w/ the positional INPUT parameters below. # sdk_bootstrap() should be called w/ the positional INPUT parameters below.
# Boostrap SDK build automation stub. # Bootstrap SDK build automation stub.
# This script will use a seed SDK container + tarball to bootstrap a # This script will use a seed SDK container + tarball to bootstrap a
# new SDK tarball. # new SDK tarball.
# #

2
ci-automation/sdk_container.sh
View File

@ -50,7 +50,7 @@ function sdk_container_build() {
# This will update the SDK_VERSION in versionfile # This will update the SDK_VERSION in versionfile
./build_sdk_container_image -x ./ci-cleanup.sh ./__build__/"${sdk_tarball}" ./build_sdk_container_image -x ./ci-cleanup.sh ./__build__/"${sdk_tarball}"
# push artefacts to build cache # push artifacts to build cache
local docker_vernum="$(vernum_to_docker_image_version "${vernum}")" local docker_vernum="$(vernum_to_docker_image_version "${vernum}")"
docker_image_to_buildcache "${CONTAINER_REGISTRY}/flatcar-sdk-all" "${docker_vernum}" docker_image_to_buildcache "${CONTAINER_REGISTRY}/flatcar-sdk-all" "${docker_vernum}"
docker_image_to_buildcache "${CONTAINER_REGISTRY}/flatcar-sdk-amd64" "${docker_vernum}" docker_image_to_buildcache "${CONTAINER_REGISTRY}/flatcar-sdk-amd64" "${docker_vernum}"

4
ci-automation/vms.sh
View File

@ -6,7 +6,7 @@
# >>> This file is supposed to be SOURCED from the repository ROOT. <<< # >>> This file is supposed to be SOURCED from the repository ROOT. <<<
# #
# packages_build() should be called w/ the positional INPUT parameters below. # vm_build() should be called w/ the positional INPUT parameters below.
# Vendor images build automation stub. # Vendor images build automation stub.
# This script will build one or more vendor images ("vm") using a pre-built image container. # This script will build one or more vendor images ("vm") using a pre-built image container.
@ -59,7 +59,7 @@ function vm_build() {
echo "docker container rm -f '${vms_container}'" >> ci-cleanup.sh echo "docker container rm -f '${vms_container}'" >> ci-cleanup.sh
for format in "${@}"; do for format; do
echo " ################### VENDOR '${format}' ################### " echo " ################### VENDOR '${format}' ################### "
./run_sdk_container -n "${vms_container}" -C "${image_image}" \ ./run_sdk_container -n "${vms_container}" -C "${image_image}" \
-v "${vernum}" \ -v "${vernum}" \

2
sdk_container/git-override/README.md
View File

@ -1,5 +1,5 @@
# GIT overrides for submodules # GIT overrides for submodules
In the SDK container, the scripts root is at a different relative path to the submodules. In the SDK container, the scripts root is at a different relative path to the submodules.
The overrides in this directory are into coreos-overlay/.git and portage-stable.gitso the submodules find itheir parents. The overrides in this directory are into `coreos-overlay/.git` and `portage-stable/.git` so the submodules can find their parents.

2
sdk_lib/Dockerfile.lean-arch
View File

@ -23,7 +23,7 @@ COPY --from=meta --chown=sdk:sdk /home/sdk /home/sdk
RUN chown -R sdk:sdk /mnt/host/source RUN chown -R sdk:sdk /mnt/host/source
# This is not used when starting the container via ./run_sdk_container # This is not used when starting the container via ./run_sdk_container
# but it's useful fir standalone container use. # but it's useful for standalone container use.
RUN mkdir -p /mnt/host/source/src/scripts RUN mkdir -p /mnt/host/source/src/scripts
COPY --chown=sdk:sdk sdk_lib/sdk_init_selfcontained.sh /mnt/host/source/src/ COPY --chown=sdk:sdk sdk_lib/sdk_init_selfcontained.sh /mnt/host/source/src/
ENTRYPOINT /home/sdk/sdk_entry.sh ENTRYPOINT /home/sdk/sdk_entry.sh

2
sdk_lib/sdk_container_common.sh
View File

@ -176,7 +176,7 @@ function setup_gsutil() {
local botofile="$creds_dir/boto-flatcar-sdk" local botofile="$creds_dir/boto-flatcar-sdk"
# TODO t-lo: move generation of boto file to sdk_entry so # TODO t-lo: move generation of boto file to sdk_entry so
# it's onyl created inside the container. # it's only created inside the container.
# read creds file and create boto file for gsutil # read creds file and create boto file for gsutil
local tmp="$(mktemp)" local tmp="$(mktemp)"

1
sdk_lib/sdk_init_selfcontained.sh
View File

@ -10,6 +10,7 @@ echo "Press [RETURN] to continue, CTRL+C to abort"
echo echo
read junk read junk
unset junk
# -- # --