docs: document the process of building custom kernel packages

Fixes #7612

Drop the customizing rootfs docs, and point towards system extensions
documentation, as it is the right way.

Document building custom Talos Linux kernel.

Signed-off-by: Andrey Smirnov <andrey.smirnov@siderolabs.com>
This commit is contained in:
Andrey Smirnov 2024-01-30 15:49:31 +04:00
parent 7af48bd559
commit aa03204b86
No known key found for this signature in database
GPG Key ID: FE042E3D4085A811
10 changed files with 199 additions and 198 deletions

View File

@ -4,8 +4,8 @@ no_list: true
linkTitle: "Documentation"
cascade:
type: docs
lastRelease: v1.6.2
kubernetesRelease: "1.29.0"
lastRelease: v1.6.3
kubernetesRelease: "1.29.1"
prevKubernetesRelease: "1.28.3"
nvidiaContainerToolkitRelease: "v1.13.5"
nvidiaDriverRelease: "535.129.03"

View File

@ -33,6 +33,14 @@ make <target> PLATFORM=linux/arm64 # build for arm64 only
make <target> PLATFORM=linux/arm64,linux/amd64 # build for arm64 and amd64, container images will be multi-arch
```
## Custom `PKGS`
When [customizing Linux kernel]({{< relref "customizing-the-kernel" >}}), the source for the [`siderolabs/pkgs`](https://github.com/siderolabs/pkgs) repository can
be overridden with:
* if you built and pushed only a custom `kernel` package, the reference can be overridden with `PKG_KERNEL` variable: `make <target> PKG_KERNEL=<registry>/<username>/kernel:<tag>`
* otherwise, patch the `Dockerfile` to account for the new `pkgs` repository reference
## Customizations
Some of the build parameters can be customized by passing environment variables to `make`, e.g. `GOAMD64=v1` can be used to build
@ -67,12 +75,24 @@ Building and pushing the image can be done with:
```bash
make installer PUSH=true IMAGE_REGISTRY=docker.io USERNAME=<username> # ghcr.io/siderolabs/installer
make imager PUSH=true IMAGE_REGISTRY=docker.io USERNAME=<username> # ghcr.io/siderolabs/installer
make imager PUSH=true IMAGE_REGISTRY=docker.io USERNAME=<username> # ghcr.io/siderolabs/imager
```
The [local registry]({{< relref "developing-talos" >}}) running on `127.0.0.1:5005` can be used as well to avoid pushing/pulling over the network:
```bash
make installer PUSH=true REGISTRY=127.0.0.1:5005
```
When building `imager` container, by default Talos will include the boot assets for both `amd64` and `arm64` architectures, if building only for single architecture, specify `INSTALLER_ARCH` variable:
```bash
make imager INSTALLER_ARCH=targetarch PLATFORM=linux/amd64
```
## Building ISO
The ISO image is built with the help of `imager` container image, by default `ghcr.io/siderolabs/imager` will be used with the matching tag:
The [ISO image]({{< relref "../talos-guides/install/boot-assets" >}}) is built with the help of `imager` container image, by default `ghcr.io/siderolabs/imager` will be used with the matching tag:
```bash
make iso

View File

@ -5,50 +5,91 @@ aliases:
- ../guides/customizing-the-kernel
---
The installer image contains [`ONBUILD`](https://docs.docker.com/engine/reference/builder/#onbuild) instructions that handle the following:
Talos Linux configures the kernel to allow loading only cryptographically signed modules.
The signing key is generated during the build process, it is unique to each build, and it is not available to the user.
The public key is embedded in the kernel, and it is used to verify the signature of the modules.
So if you want to use a custom kernel module, you will need to build your own kernel, and all required kernel modules in order to get the signature in sync with the kernel.
- the decompression, and unpacking of the `initramfs.xz`
- the unsquashing of the rootfs
- the copying of new rootfs files
- the squashing of the new rootfs
- and the packing, and compression of the new `initramfs.xz`
## Overview
When used as a base image, the installer will perform the above steps automatically with the requirement that a `customization` stage be defined in the `Dockerfile`.
In order to build a custom kernel (or a custom kernel module), the following steps are required:
Build and push your own kernel:
- build a new Linux kernel and modules, push the artifacts to a registry
- build a new Talos base artifacts: kernel and initramfs image
- produce a new Talos boot artifact (ISO, installer image, disk image, etc.)
```sh
git clone https://github.com/talos-systems/pkgs.git
cd pkgs
make kernel-menuconfig USERNAME=_your_github_user_name_
We will go through each step in detail.
docker login ghcr.io --username _your_github_user_name_
make kernel USERNAME=_your_github_user_name_ PUSH=true
```
## Building a Custom Kernel
Using a multi-stage `Dockerfile` we can define the `customization` stage and build `FROM` the installer image:
First, you might need to prepare the build environment, follow the [Building Custom Images]({{< relref "building-images" >}}) guide.
```docker
FROM scratch AS customization
# this is needed so that Talos copies base kernel modules info and default modules shipped with Talos
COPY --from=<custom kernel image> /lib/modules /kernel/lib/modules
# this copies over the custom modules
COPY --from=<custom kernel image> /lib/modules /lib/modules
Checkout the [`siderolabs/pkgs`](https://github.com/siderolabs/pkgs) repository:
FROM ghcr.io/siderolabs/installer:latest
COPY --from=<custom kernel image> /boot/vmlinuz /usr/install/${TARGETARCH}/vmlinuz
```shell
git clone https://github.com/siderolabs/pkgs.git
cd pkgs
git checkout {{< release_branch >}}
```
When building the image, the `customization` stage will automatically be copied into the rootfs.
The `customization` stage is not limited to a single `COPY` instruction.
In fact, you can do whatever you would like in this stage, but keep in mind that everything in `/` will be copied into the rootfs.
The kernel configuration is located in the files `kernel/build/config-ARCH` files.
It can be modified using the text editor, or by using the Linux kernel `menuconfig` tool:
To build the image, run:
```bash
DOCKER_BUILDKIT=0 docker build --build-arg RM="/lib/modules" -t installer:kernel .
```shell
make kernel-menuconfig
```
> Note: buildkit has a bug [#816](https://github.com/moby/buildkit/issues/816), to disable it use `DOCKER_BUILDKIT=0`
The kernel configuration can be cleaned up by running:
Now that we have a custom installer we can build Talos for the specific platform we wish to deploy to.
```shell
make kernel-olddefconfig
```
Both commands will output the new configuration to the `kernel/build/config-ARCH` files.
Once ready, build the kernel any out-of-tree modules (if required, e.g. `zfs`) and push the artifacts to a registry:
```shell
make kernel REGISTRY=127.0.0.1:5005 PUSH=true
```
By default, this command will compile and push the kernel both for `amd64` and `arm64` architectures, but you can specify a single architecture by overriding
a variable `PLATFORM`:
```shell
make kernel REGISTRY=127.0.0.1:5005 PUSH=true PLATFORM=linux/amd64
```
This will create a container image `127.0.0.1:5005/siderolabs/kernel:$TAG` with the kernel and modules.
## Building Talos Base Artifacts
Follow the [Building Custom Images]({{< relref "building-images" >}}) guide to set up the Talos source code checkout.
If some new kernel modules were introduced, adjust the list of the default modules compiled into the Talos `initramfs` by
editing the file `hack/modules-ARCH.txt`.
Try building base Talos artifacts:
```shell
make kernel initramfs PKG_KERNEL=127.0.0.1:5005/siderolabs/kernel:$TAG PLATFORM=linux/amd64
```
This should create a new image of the kernel and initramfs in `_out/vmlinuz-amd64` and `_out/initramfs-amd64.xz` respectively.
> Note: if building for `arm64`, replace `amd64` with `arm64` in the commands above.
As a final step, produce the new `imager` container image which can generate Talos boot assets:
```shell
make imager PKG_KERNEL=127.0.0.1:5005/siderolabs/kernel:$TAG PLATFORM=linux/amd64 INSTALLER_ARCH=targetarch
```
> Note: if you built the kernel for both `amd64` and `arm64`, a multi-arch `imager` container can be built as well by specifying `INSTALLER_ARCH=all` and `PLATFORM=linux/amd64,linux/arm64`.
## Building Talos Boot Assets
Follow the [Boot Assets]({{< relref "../talos-guides/install/boot-assets" >}}) guide to build Talos boot assets you might need to boot Talos: ISO, `installer` image, etc.
Replace the reference to the `imager` in guide with the reference to the `imager` container built above.
> Note: if you update the `imager` container, don't forget to `docker pull` it, as `docker` caches pulled images and won't pull the updated image automatically.

View File

@ -1,63 +0,0 @@
---
title: "Customizing the Root Filesystem"
description: "How to add your own content to the immutable root file system of Talos Linux."
aliases:
- ../guides/customizing-the-root-filesystem
---
The installer image contains [`ONBUILD`](https://docs.docker.com/engine/reference/builder/#onbuild) instructions that handle the following:
- the decompression, and unpacking of the `initramfs.xz`
- the unsquashing of the rootfs
- the copying of new rootfs files
- the squashing of the new rootfs
- and the packing, and compression of the new `initramfs.xz`
When used as a base image, the installer will perform the above steps automatically with the requirement that a `customization` stage be defined in the `Dockerfile`.
For example, say we have an image that contains the contents of a library we wish to add to the Talos rootfs.
We need to define a stage with the name `customization`:
```docker
FROM scratch AS customization
COPY --from=<name|index> <src> <dest>
```
Using a multi-stage `Dockerfile` we can define the `customization` stage and build `FROM` the installer image:
```docker
FROM scratch AS customization
COPY --from=<name|index> <src> <dest>
FROM ghcr.io/siderolabs/installer:latest
```
When building the image, the `customization` stage will automatically be copied into the rootfs.
The `customization` stage is not limited to a single `COPY` instruction.
In fact, you can do whatever you would like in this stage, but keep in mind that everything in `/` will be copied into the rootfs.
> Note: `<dest>` is the path relative to the rootfs that you wish to place the contents of `<src>`.
To build the image, run:
```bash
docker build --squash -t <organization>/installer:latest .
```
In the case that you need to perform some cleanup _before_ adding additional files to the rootfs, you can specify the `RM` [build-time variable](https://docs.docker.com/engine/reference/commandline/build/#set-build-time-variables---build-arg):
```bash
docker build --squash --build-arg RM="[<path> ...]" -t <organization>/installer:latest .
```
This will perform a `rm -rf` on the specified paths relative to the rootfs.
> Note: `RM` must be a whitespace delimited list.
The resulting image can be used to:
- generate an image for any of the supported providers
- perform bare-metall installs
- perform upgrades
We will step through common customizations in the remainder of this section.

View File

@ -3,6 +3,7 @@ title: "System Extensions"
description: "Customizing the Talos Linux immutable root file system."
aliases:
- ../../guides/system-extensions
- ../../advanced/customizing-the-root-filesystem
---
System extensions allow extending the Talos root filesystem, which enables a variety of features, such as including custom

View File

@ -33,6 +33,14 @@ make <target> PLATFORM=linux/arm64 # build for arm64 only
make <target> PLATFORM=linux/arm64,linux/amd64 # build for arm64 and amd64, container images will be multi-arch
```
## Custom `PKGS`
When [customizing Linux kernel]({{< relref "customizing-the-kernel" >}}), the source for the [`siderolabs/pkgs`](https://github.com/siderolabs/pkgs) repository can
be overridden with:
* if you built and pushed only a custom `kernel` package, the reference can be overridden with `PKG_KERNEL` variable: `make <target> PKG_KERNEL=<registry>/<username>/kernel:<tag>`
* if the full `pkgs` repository was built and pushed, the references can be overridden with `PKGS_PREFIX` and `PKGS` variables: `make <target> PKG_PREFIX=<registry>/<username> PKGS=<tag>`
## Customizations
Some of the build parameters can be customized by passing environment variables to `make`, e.g. `GOAMD64=v1` can be used to build
@ -67,12 +75,24 @@ Building and pushing the image can be done with:
```bash
make installer PUSH=true IMAGE_REGISTRY=docker.io USERNAME=<username> # ghcr.io/siderolabs/installer
make imager PUSH=true IMAGE_REGISTRY=docker.io USERNAME=<username> # ghcr.io/siderolabs/installer
make imager PUSH=true IMAGE_REGISTRY=docker.io USERNAME=<username> # ghcr.io/siderolabs/imager
```
The [local registry]({{< relref "developing-talos" >}}) running on `127.0.0.1:5005` can be used as well to avoid pushing/pulling over the network:
```bash
make installer PUSH=true REGISTRY=127.0.0.1:5005
```
When building `imager` container, by default Talos will include the boot assets for both `amd64` and `arm64` architectures, if building only for single architecture, specify `INSTALLER_ARCH` variable:
```bash
make imager INSTALLER_ARCH=targetarch PLATFORM=linux/amd64
```
## Building ISO
The ISO image is built with the help of `imager` container image, by default `ghcr.io/siderolabs/imager` will be used with the matching tag:
The [ISO image]({{< relref "../talos-guides/install/boot-assets" >}}) is built with the help of `imager` container image, by default `ghcr.io/siderolabs/imager` will be used with the matching tag:
```bash
make iso

View File

@ -5,50 +5,91 @@ aliases:
- ../guides/customizing-the-kernel
---
The installer image contains [`ONBUILD`](https://docs.docker.com/engine/reference/builder/#onbuild) instructions that handle the following:
Talos Linux configures the kernel to allow loading only cryptographically signed modules.
The signing key is generated during the build process, it is unique to each build, and it is not available to the user.
The public key is embedded in the kernel, and it is used to verify the signature of the modules.
So if you want to use a custom kernel module, you will need to build your own kernel, and all required kernel modules in order to get the signature in sync with the kernel.
- the decompression, and unpacking of the `initramfs.xz`
- the unsquashing of the rootfs
- the copying of new rootfs files
- the squashing of the new rootfs
- and the packing, and compression of the new `initramfs.xz`
## Overview
When used as a base image, the installer will perform the above steps automatically with the requirement that a `customization` stage be defined in the `Dockerfile`.
In order to build a custom kernel (or a custom kernel module), the following steps are required:
Build and push your own kernel:
- build a new Linux kernel and modules, push the artifacts to a registry
- build a new Talos base artifacts: kernel and initramfs image
- produce a new Talos boot artifact (ISO, installer image, disk image, etc.)
```sh
git clone https://github.com/talos-systems/pkgs.git
cd pkgs
make kernel-menuconfig USERNAME=_your_github_user_name_
We will go through each step in detail.
docker login ghcr.io --username _your_github_user_name_
make kernel USERNAME=_your_github_user_name_ PUSH=true
```
## Building a Custom Kernel
Using a multi-stage `Dockerfile` we can define the `customization` stage and build `FROM` the installer image:
First, you might need to prepare the build environment, follow the [Building Custom Images]({{< relref "building-images" >}}) guide.
```docker
FROM scratch AS customization
# this is needed so that Talos copies base kernel modules info and default modules shipped with Talos
COPY --from=<custom kernel image> /lib/modules /kernel/lib/modules
# this copies over the custom modules
COPY --from=<custom kernel image> /lib/modules /lib/modules
Checkout the [`siderolabs/pkgs`](https://github.com/siderolabs/pkgs) repository:
FROM ghcr.io/siderolabs/installer:latest
COPY --from=<custom kernel image> /boot/vmlinuz /usr/install/${TARGETARCH}/vmlinuz
```shell
git clone https://github.com/siderolabs/pkgs.git
cd pkgs
git checkout {{< release_branch >}}
```
When building the image, the `customization` stage will automatically be copied into the rootfs.
The `customization` stage is not limited to a single `COPY` instruction.
In fact, you can do whatever you would like in this stage, but keep in mind that everything in `/` will be copied into the rootfs.
The kernel configuration is located in the files `kernel/build/config-ARCH` files.
It can be modified using the text editor, or by using the Linux kernel `menuconfig` tool:
To build the image, run:
```bash
DOCKER_BUILDKIT=0 docker build --build-arg RM="/lib/modules" -t installer:kernel .
```shell
make kernel-menuconfig
```
> Note: buildkit has a bug [#816](https://github.com/moby/buildkit/issues/816), to disable it use `DOCKER_BUILDKIT=0`
The kernel configuration can be cleaned up by running:
Now that we have a custom installer we can build Talos for the specific platform we wish to deploy to.
```shell
make kernel-olddefconfig
```
Both commands will output the new configuration to the `kernel/build/config-ARCH` files.
Once ready, build the kernel any out-of-tree modules (if required, e.g. `zfs`) and push the artifacts to a registry:
```shell
make kernel REGISTRY=127.0.0.1:5005 PUSH=true
```
By default, this command will compile and push the kernel both for `amd64` and `arm64` architectures, but you can specify a single architecture by overriding
a variable `PLATFORM`:
```shell
make kernel REGISTRY=127.0.0.1:5005 PUSH=true PLATFORM=linux/amd64
```
This will create a container image `127.0.0.1:5005/siderolabs/kernel:$TAG` with the kernel and modules.
## Building Talos Base Artifacts
Follow the [Building Custom Images]({{< relref "building-images" >}}) guide to set up the Talos source code checkout.
If some new kernel modules were introduced, adjust the list of the default modules compiled into the Talos `initramfs` by
editing the file `hack/modules-ARCH.txt`.
Try building base Talos artifacts:
```shell
make kernel initramfs PKG_KERNEL=127.0.0.1:5005/siderolabs/kernel:$TAG PLATFORM=linux/amd64
```
This should create a new image of the kernel and initramfs in `_out/vmlinuz-amd64` and `_out/initramfs-amd64.xz` respectively.
> Note: if building for `arm64`, replace `amd64` with `arm64` in the commands above.
As a final step, produce the new `imager` container image which can generate Talos boot assets:
```shell
make imager PKG_KERNEL=127.0.0.1:5005/siderolabs/kernel:$TAG PLATFORM=linux/amd64 INSTALLER_ARCH=targetarch
```
> Note: if you built the kernel for both `amd64` and `arm64`, a multi-arch `imager` container can be built as well by specifying `INSTALLER_ARCH=all` and `PLATFORM=linux/amd64,linux/arm64`.
## Building Talos Boot Assets
Follow the [Boot Assets]({{< relref "../talos-guides/install/boot-assets" >}}) guide to build Talos boot assets you might need to boot Talos: ISO, `installer` image, etc.
Replace the reference to the `imager` in guide with the reference to the `imager` container built above.
> Note: if you update the `imager` container, don't forget to `docker pull` it, as `docker` caches pulled images and won't pull the updated image automatically.

View File

@ -1,63 +0,0 @@
---
title: "Customizing the Root Filesystem"
description: "How to add your own content to the immutable root file system of Talos Linux."
aliases:
- ../guides/customizing-the-root-filesystem
---
The installer image contains [`ONBUILD`](https://docs.docker.com/engine/reference/builder/#onbuild) instructions that handle the following:
- the decompression, and unpacking of the `initramfs.xz`
- the unsquashing of the rootfs
- the copying of new rootfs files
- the squashing of the new rootfs
- and the packing, and compression of the new `initramfs.xz`
When used as a base image, the installer will perform the above steps automatically with the requirement that a `customization` stage be defined in the `Dockerfile`.
For example, say we have an image that contains the contents of a library we wish to add to the Talos rootfs.
We need to define a stage with the name `customization`:
```docker
FROM scratch AS customization
COPY --from=<name|index> <src> <dest>
```
Using a multi-stage `Dockerfile` we can define the `customization` stage and build `FROM` the installer image:
```docker
FROM scratch AS customization
COPY --from=<name|index> <src> <dest>
FROM ghcr.io/siderolabs/installer:latest
```
When building the image, the `customization` stage will automatically be copied into the rootfs.
The `customization` stage is not limited to a single `COPY` instruction.
In fact, you can do whatever you would like in this stage, but keep in mind that everything in `/` will be copied into the rootfs.
> Note: `<dest>` is the path relative to the rootfs that you wish to place the contents of `<src>`.
To build the image, run:
```bash
docker build --squash -t <organization>/installer:latest .
```
In the case that you need to perform some cleanup _before_ adding additional files to the rootfs, you can specify the `RM` [build-time variable](https://docs.docker.com/engine/reference/commandline/build/#set-build-time-variables---build-arg):
```bash
docker build --squash --build-arg RM="[<path> ...]" -t <organization>/installer:latest .
```
This will perform a `rm -rf` on the specified paths relative to the rootfs.
> Note: `RM` must be a whitespace delimited list.
The resulting image can be used to:
- generate an image for any of the supported providers
- perform bare-metall installs
- perform upgrades
We will step through common customizations in the remainder of this section.

View File

@ -3,6 +3,7 @@ title: "System Extensions"
description: "Customizing the Talos Linux immutable root file system."
aliases:
- ../../guides/system-extensions
- ../../advanced/customizing-the-root-filesystem
---
System extensions allow extending the Talos root filesystem, which enables a variety of features, such as including custom

View File

@ -0,0 +1,3 @@
{{ $raw_version := replace .Page.FirstSection.Params.lastRelease "v" "" -}}
{{- $major_minor := split $raw_version "." | first 2 -}}
release-{{- delimit $major_minor "." -}}