From 10dc85b099d39e82738a6b2c7a6bc4c40972802e Mon Sep 17 00:00:00 2001 From: Thilo Fromm Date: Thu, 28 Oct 2021 16:34:15 +0200 Subject: [PATCH] README: add documentation on submodules and on tags Signed-off-by: Thilo Fromm --- README.md | 77 ++++++++++++++++++++++++++++++++++++++++++------------- 1 file changed, 59 insertions(+), 18 deletions(-) diff --git a/README.md b/README.md index 6fed1acb47..4c2e6df021 100644 --- a/README.md +++ b/README.md @@ -11,17 +11,53 @@ The SDK can be used to [flatcar-docs]: https://docs.flatcar-linux.org/os/sdk-modifying-flatcar/ +# Using the scripts repository: submodules and tags + +The repository is meant to be the entry point for Flatcar builds and development. +For building pakages, there are 2 addigional repositories, [coreos-overlay](https://github.com/flatcar-linux/) and [portage-stable](https://github.com/flatcar-linux/portage-stable), which contain all packages' `ebuild` (build configuration) files. +These repositories are included in `scripts` via git submodules and are used by the SDK container wrapper scripts detailed on further below. +The submodules reside in: +``` +scripts + +--sdk_container + +---------src + +--third_party + +------coreos-overlay + +------portage-stable +``` + +When working with the scripts repo always make sure to initialise and to update these submodules; otherwise builds will break because build configuration is missing: +```bash +$ git clone https://github.com/flatcar-linux/scripts.git +$ cd scripts +$ git submodule init +$ git submodule update +``` + +The `scripts` repository makes ample use of tags to mark releases. +Sometimes, local and origin tags can diverge (e.g. when re-tagging something locally to test a build). +Also, `git pull` and `git fetch` do not automatically pull new tags, so long-standing local sourcetrees may lack newer versions. +To fetch and update all tags and to remove tags locally which have been deleted upstream, do +``` +$ git pull --all --tags --prune --prune-tags +``` +If upstream retagged (of if a tag was changed locally) the corresponding upstream tag will not be pulled so the local tag remains. +In order to override local tags with upstream, run +``` +$ git pull --all --tags --prune --prune-tags --force +``` # Using the SDK container We provide a containerised SDK via https://github.com/orgs/flatcar-linux/packages. The container comes in 3 flavours: -* Full SDK initialised with both architectures supported by Flatcar (amd64 and arm64). This is the largest container, it's about 7GB in size. -* AMD64 SDK initialised for building AMD64 OS images. About 5.5GB in size. -* ARM64 SDK initialised for building ARM64 OS images on AMD64 hosts. Also about 5.5GB in size. (While work on a ARM64 native SDK is ongoing, it's unfortunately not ready yet). +* Full SDK initialised with both architectures supported by Flatcar (amd64 and arm64). This is the largest container, it's about 8GB in size (~3 GB compressed). +* AMD64 SDK initialised for building AMD64 OS images. About 6GB in size (2GB compressed). +* ARM64 SDK initialised for building ARM64 OS images on AMD64 hosts. Also about 6GB in size. +While work on a native ARM64 native SDK is ongoing, it's unfortunately not ready yet. If you want to help, patches are welcome! The container can be run in one of two ways - "standalone", or integrated with the [Scripts](https://github.com/flatcar-linux/scripts) repo: * Standalone mode will use no host volumes and will allow you to play with the SDK in a sandboxed throw-away environment. In standalone mode, you interface with Docker directly to use the SDK container. -* Integrated mode will closely integrate with the Scripts directory and bind-mount it as well as the portage-stable and coreos-overlay gitmodules into the container. Integrated mode uses wrapper scripts to interact with the SDK container. +* Integrated mode will closely integrate with the Scripts directory and bind-mount it as well as the portage-stable and coreos-overlay gitmodules into the container. Integrated mode uses wrapper scripts to interact with the SDK container. This is the recommended way for developing patches for Flatcar. ## Standalone mode @@ -29,10 +65,10 @@ In standalone mode, the SDK is just another Docker container. Interaction with t * Check the list of available versions and pick a version to use. The SDK Major versions correspond to Flatcar Major release versions. List of images: `https://github.com/orgs/flatcar-linux/packages/container/package/flatcar-sdk-all` - For the purpose of this example we'll use version `3005.0.0`. -* Fetch the container image: `docker pull ghcr.io/flatcar-linux/flatcar-sdk-all:3005.0.0` -* Start the image in interactive (tty) mode: `docker run -ti ghcr.io/flatcar-linux/flatcar-sdk-all:3005.0.0` - You are now inside the SDK container: + For the purpose of this example we'll use version `3033.0.0`. +* Fetch the container image: `docker pull ghcr.io/flatcar-linux/flatcar-sdk-all:3033.0.0` +* Start the image in interactive (tty) mode: `docker run -ti ghcr.io/flatcar-linux/flatcar-sdk-all:3033.0.0` + You are now inside the SDK container (the hostname will likely differ): `sdk@f236fda982a4 ~/trunk/src/scripts $` * Initialise the SDK in self-contained mode. This needs to be done once per container and will check out the scripts, coreos-overlay, and portage-stable repositories into the container. `sdk@f236fda982a4 ../sdk_init_selfcontained.sh` @@ -45,7 +81,7 @@ In order to build OS images (via `./build_image` and `./image_to_vm`) the SDK to This is necessary because the SDK currently employs loop devices to create and to partition OS images. To start a container in privileged mode with `/dev` available use: -* `docker run -ti --privileged -v /dev:/dev ghcr.io/flatcar-linux/flatcar-sdk-all:3005.0.0` +* `docker run -ti --privileged -v /dev:/dev ghcr.io/flatcar-linux/flatcar-sdk-all:3033.0.0` ## Integrated mode @@ -56,10 +92,11 @@ The wrapper scripts will re-use existing containers instead of creating new ones To clone the scripts repo and pick a version: * Clone the scripts repo: `git clone https://github.com/flatcar-linux/scripts.git` + * Make sure to initialise and fetch git submodules - Flatcar's ebuilds are in 2 separate repositories, connected to `scripts` via submodules. + * `git submodule init; git submodule update` * Optionally, check out a release tag to base your work on * list releases (e.g. all Alpha releases): `git tag -l alpha-*` - * check out the release version, e.g. `3005.0.0`: `git checkout 3005.0.0` -* Update the overlay submodules: `git submodules update` + * check out the release version, e.g. `3033.0.0`: `git checkout 3033.0.0` To use the SDK container: * Fetch image and start the SDK container: `./run_sdk_container -t` @@ -73,10 +110,10 @@ Subsequent calls to `./run_sdk_container` will re-use the container (as long as Check out `docker container ls --all` and you'll see something like ``` CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES -19ea3b6d00ad ghcr.io/flatcar-linux/flatcar-sdk-all:3005.0.0 "/bin/sh -c /home/sd…" 4 hours ago Exited (0) About an hour ago flatcar-sdk-all-3005.0.0_os-3005.0.0 +19ea3b6d00ad ghcr.io/flatcar-linux/flatcar-sdk-all:3033.0.0 "/bin/sh -c /home/sd…" 4 hours ago Exited (0) About an hour ago flatcar-sdk-all-3033.0.0_os-3033.0.0 ``` -Re-use of containers happens on a per-name basis. The above example's container name `flatcar-sdk-all-3005.0.0_os-3005.0.0` is generated automatically. Using `docker container rm` the container can be discarded - a subsequent call to `./run_sdk_container` will create a new one. Custom containers can be created by use of the `-n ` command line option; these will be re-used in subsequent calls to `./run_sdk_container` when using the same ``. +Re-use of containers happens on a per-name basis. The above example's container name `flatcar-sdk-all-3033.0.0_os-3033.0.0` is generated automatically. Using `docker container rm` the container can be discarded - a subsequent call to `./run_sdk_container` will create a new one. Custom containers can be created by use of the `-n ` command line option; these will be re-used in subsequent calls to `./run_sdk_container` when using the same ``. The local sourcetree can also be used with an entirely custom SDK container image. Users must ensure that the image is either fetch-able or present locally. The custom image can be specified using `-C `. This option is useful e.g. for building the local sourcetree with different SDK versions. @@ -84,11 +121,15 @@ Check out `./run_sdk_container -h` for more information on command line options. # Building a new SDK container -Building an SDK container is done using `./build_sdk_container_image`. -The SDK container is based on an SDK tarball which the script will fetch. -By default, the current git tree's release version will be built; this can be changed with the `-v` flag. -When using `-v`, the corresponding release version of the Scripts repository is checked out (unless suppressed by `-c`) before the container is generated. +Building an SDK container is done using `./build_sdk_container_image `. +The tarball input is the result of an SDK bootstrap (see below). Version information for both OS as well as for the SDK will be extracted from the tarball name. +The version file will be updated accordingly before the SDK container is built. +During the build, toolchain packages will be built and installed into the SDK container image. Both supported boards (`amd64-usr` and `arm64-usr`) will be initialisedin the container image. # Bootstrapping a new SDK tarball using the SDK container -The script `./bootstrap_sdk_container` bootstraps a new SDK tarball using an existing SDK container and seed tarball. Specifying the seed version is required for this script. +The script `./bootstrap_sdk_container` bootstraps a new SDK tarball using an existing SDK container and seed tarball. Specifying the seed version as well as the designated new SDK version is required for this script. + +# Automation stubs for continuous integration + +Script stubs for various build stages can be found in the [ci-automation](ci-automation) folder. THese are helpful for glueing Flatcar Container Linux builds to a continuous integration system.