From 1af32f45a5b0f43e8a73a2187eca4d3dfbb24913 Mon Sep 17 00:00:00 2001
From: iwilltry42 <iwilltry42@gmail.com>
Date: Mon, 11 Oct 2021 14:36:26 +0100
Subject: [PATCH] Deployed 33507fd1 to v5.0.0 with MkDocs 1.2.2 and mike 1.1.0

---
 v5.0.0/faq/faq/index.html               |  20 +++---
 v5.0.0/search/search_index.json         |   2 +-
 v5.0.0/sitemap.xml                      |  84 ++++++++++++------------
 v5.0.0/sitemap.xml.gz                   | Bin 477 -> 477 bytes
 v5.0.0/usage/advanced/calico/index.html |   6 +-
 v5.0.0/usage/commands/index.html        |   5 +-
 v5.0.0/usage/registries/index.html      |   4 +-
 7 files changed, 60 insertions(+), 61 deletions(-)

diff --git a/v5.0.0/faq/faq/index.html b/v5.0.0/faq/faq/index.html
index 40dd84f6..6e58a4cd 100644
--- a/v5.0.0/faq/faq/index.html
+++ b/v5.0.0/faq/faq/index.html
@@ -1302,8 +1302,8 @@
 <li>
 <p>change the kubelet&rsquo;s eviction thresholds upon cluster creation:</p>
 <div class="highlight"><pre><span></span><code>k3d cluster create <span class="se">\</span>
-  --k3s-agent-arg <span class="s1">&#39;--kubelet-arg=eviction-hard=imagefs.available&lt;1%,nodefs.available&lt;1%&#39;</span> <span class="se">\</span>
-  --k3s-agent-arg <span class="s1">&#39;--kubelet-arg=eviction-minimum-reclaim=imagefs.available=1%,nodefs.available=1%&#39;</span>
+  --k3s-arg <span class="s1">&#39;--kubelet-arg=eviction-hard=imagefs.available&lt;1%,nodefs.available&lt;1%@agent:*&#39;</span> <span class="se">\</span>
+  --k3s-arg <span class="s1">&#39;--kubelet-arg=eviction-minimum-reclaim=imagefs.available=1%,nodefs.available=1%@agent:*&#39;</span>
 </code></pre></div>
 </li>
 </ul>
@@ -1323,7 +1323,7 @@
 <ul>
 <li>The Problem: Passing a feature flag to the Kubernetes API Server running inside k3s.</li>
 <li>Example: you want to enable the EphemeralContainers feature flag in Kubernetes</li>
-<li>Solution: <code class="highlight">k3d cluster create --k3s-server-arg <span class="s1">&#39;--kube-apiserver-arg=feature-gates=EphemeralContainers=true&#39;</span></code><ul>
+<li>Solution: <code class="highlight">k3d cluster create --k3s-arg <span class="s1">&#39;--kube-apiserver-arg=feature-gates=EphemeralContainers=true@server:*&#39;</span></code><ul>
 <li><strong>Note</strong>: Be aware of where the flags require dashes (<code>--</code>) and where not.<ul>
 <li>the k3s flag (<code>--kube-apiserver-arg</code>) has the dashes</li>
 <li>the kube-apiserver flag <code>feature-gates</code> doesn&rsquo;t have them (k3s adds them internally)</li>
@@ -1336,10 +1336,10 @@
 <li>
 <p>Second example:</p>
 <div class="highlight"><pre><span></span><code>k3d cluster create k3d-one <span class="se">\</span>
-  --k3s-server-arg --cluster-cidr<span class="o">=</span><span class="s2">&quot;10.118.0.0/17&quot;</span> <span class="se">\</span>
-  --k3s-server-arg --service-cidr<span class="o">=</span><span class="s2">&quot;10.118.128.0/17&quot;</span> <span class="se">\</span>
-  --k3s-server-arg --disable<span class="o">=</span>servicelb <span class="se">\</span>
-  --k3s-server-arg --disable<span class="o">=</span>traefik <span class="se">\</span>
+  --k3s-arg <span class="s2">&quot;--cluster-cidr=10.118.0.0/17@server:*&quot;</span> <span class="se">\</span>
+  --k3s-arg <span class="s2">&quot;--service-cidr=10.118.128.0/17@server:*&quot;</span> <span class="se">\</span>
+  --k3s-arg <span class="s2">&quot;--disable=servicelb@server:*&quot;</span> <span class="se">\</span>
+  --k3s-arg <span class="s2">&quot;--disable=traefik@server:*&quot;</span> <span class="se">\</span>
   --verbose
 </code></pre></div>
 <ul>
@@ -1389,8 +1389,8 @@ Some can be fixed by passing the <code>HTTP_PROXY</code> environment variables t
 <li>
 <p>Workaround: as a workaround, we can tell <code>kube-proxy</code> to not even try to set this value:</p>
 <div class="highlight"><pre><span></span><code>k3d cluster create <span class="se">\</span>
-  --k3s-server-arg <span class="s2">&quot;--kube-proxy-arg=conntrack-max-per-core=0&quot;</span> <span class="se">\</span>
-  --k3s-agent-arg <span class="s2">&quot;--kube-proxy-arg=conntrack-max-per-core=0&quot;</span> <span class="se">\</span>
+  --k3s-arg <span class="s2">&quot;--kube-proxy-arg=conntrack-max-per-core=0@server:*&quot;</span> <span class="se">\</span>
+  --k3s-arg <span class="s2">&quot;--kube-proxy-arg=conntrack-max-per-core=0@agent:*&quot;</span> <span class="se">\</span>
   --image rancher/k3s:v1.20.6-k3s
 </code></pre></div>
 </li>
@@ -1448,7 +1448,7 @@ c) authenticate containerd inside k3s/k3d to use your DockerHub user  </p>
 <div class="md-source-date">
   <small>
     
-      Last update: <span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">September 17, 2021</span>
+      Last update: <span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">October 11, 2021</span>
       
     
   </small>
diff --git a/v5.0.0/search/search_index.json b/v5.0.0/search/search_index.json
index 8b344d41..2ef8635b 100644
--- a/v5.0.0/search/search_index.json
+++ b/v5.0.0/search/search_index.json
@@ -1 +1 @@
-{"config":{"indexing":"full","lang":["en"],"min_search_length":3,"prebuild_index":false,"separator":"[\\s\\-]+"},"docs":[{"location":"","text":"Overview \u00b6 What is k3d? \u00b6 k3d is a lightweight wrapper to run k3s (Rancher Lab\u2019s minimal Kubernetes distribution) in docker. k3d makes it very easy to create single- and multi-node k3s clusters in docker, e.g. for local development on Kubernetes. Note: k3d is a community-driven project , that is supported by Rancher (SUSE) and it\u2019s not an official Rancher (SUSE) project. View a quick demo Learning \u00b6 k3d demo repository: iwilltry42/k3d-demo Featured use-cases include: hot-reloading of code when developing on k3d (Python Flask App) build-deploy-test cycle using Tilt full cluster lifecycle for simple and multi-server clusters Proof of Concept of using k3d as a service in Drone CI Rancher Meetup - May 2020 - Simplifying Your Cloud-Native Development Workflow With K3s, K3c and K3d (YouTube) Requirements \u00b6 docker to be able to use k3d at all kubectl to interact with the Kubernetes cluster Releases \u00b6 Platform Stage Version Release Date GitHub Releases stable GitHub Releases latest Homebrew - - Chocolatey stable - Installation \u00b6 You have several options there: Install Script \u00b6 Install current latest release \u00b6 wget: wget -q -O - https://raw.githubusercontent.com/rancher/k3d/main/install.sh | bash curl: curl -s https://raw.githubusercontent.com/rancher/k3d/main/install.sh | bash Install specific release \u00b6 Use the install script to grab a specific release (via TAG environment variable): wget: wget -q -O - https://raw.githubusercontent.com/rancher/k3d/main/install.sh | TAG = v5.0.0 bash curl: curl -s https://raw.githubusercontent.com/rancher/k3d/main/install.sh | TAG = v5.0.0 bash Other Installers \u00b6 Other Installation Methods Homebrew (MacOS/Linux) : brew install k3d Note : The formula can be found in homebrew/homebrew-core and is mirrored to homebrew/linuxbrew-core AUR (Arch Linux User Repository) : yay -S rancher-k3d-bin Package rancher-k3d-bin Download GitHub Release Grab a release binary from the release tab and install it yourself Chocolatey (Windows) : choco install k3d Note : package source can be found in erwinkersten/chocolatey-packages arkade : arkade get k3d asdf : asdf plugin-add k3d && asdf install k3d latest Note : asdf plugin-add k3d , then asdf install k3d <tag> with <tag> = latest or 5.x.x for a specific version (maintained by spencergilbert/asdf-k3d ) Others install via go: go install github.com/rancher/k3d@latest ( Note : this will give you unreleased/bleeding-edge changes) Quick Start \u00b6 Create a cluster named mycluster with just a single server node: k3d cluster create mycluster Use the new cluster with kubectl , e.g.: kubectl get nodes Getting the cluster\u2019s kubeconfig (included in k3d cluster create ) Get the new cluster\u2019s connection details merged into your default kubeconfig (usually specified using the KUBECONFIG environment variable or the default path $HOME /.kube/config ) and directly switch to the new context: k3d kubeconfig merge mycluster --kubeconfig-switch-context Related Projects \u00b6 vscode-k3d : VSCode Extension to handle k3d clusters from within VSCode k3x : a graphics interface (for Linux) to k3d. AbsaOSS/k3d-action : fully customizable GitHub Action to run lightweight Kubernetes clusters. AutoK3s : a lightweight tool to help run K3s everywhere including k3d provider. nolar/setup-k3d-k3s : setup K3d/K3s for GitHub Actions.","title":"Overview"},{"location":"#overview","text":"","title":"Overview"},{"location":"#what-is-k3d","text":"k3d is a lightweight wrapper to run k3s (Rancher Lab\u2019s minimal Kubernetes distribution) in docker. k3d makes it very easy to create single- and multi-node k3s clusters in docker, e.g. for local development on Kubernetes. Note: k3d is a community-driven project , that is supported by Rancher (SUSE) and it\u2019s not an official Rancher (SUSE) project. View a quick demo","title":"What is k3d?"},{"location":"#learning","text":"k3d demo repository: iwilltry42/k3d-demo Featured use-cases include: hot-reloading of code when developing on k3d (Python Flask App) build-deploy-test cycle using Tilt full cluster lifecycle for simple and multi-server clusters Proof of Concept of using k3d as a service in Drone CI Rancher Meetup - May 2020 - Simplifying Your Cloud-Native Development Workflow With K3s, K3c and K3d (YouTube)","title":"Learning"},{"location":"#requirements","text":"docker to be able to use k3d at all kubectl to interact with the Kubernetes cluster","title":"Requirements"},{"location":"#releases","text":"Platform Stage Version Release Date GitHub Releases stable GitHub Releases latest Homebrew - - Chocolatey stable -","title":"Releases"},{"location":"#installation","text":"You have several options there:","title":"Installation"},{"location":"#install-script","text":"","title":" Install Script"},{"location":"#install-current-latest-release","text":"wget: wget -q -O - https://raw.githubusercontent.com/rancher/k3d/main/install.sh | bash curl: curl -s https://raw.githubusercontent.com/rancher/k3d/main/install.sh | bash","title":"Install current latest release"},{"location":"#install-specific-release","text":"Use the install script to grab a specific release (via TAG environment variable): wget: wget -q -O - https://raw.githubusercontent.com/rancher/k3d/main/install.sh | TAG = v5.0.0 bash curl: curl -s https://raw.githubusercontent.com/rancher/k3d/main/install.sh | TAG = v5.0.0 bash","title":"Install specific release"},{"location":"#other-installers","text":"Other Installation Methods Homebrew (MacOS/Linux) : brew install k3d Note : The formula can be found in homebrew/homebrew-core and is mirrored to homebrew/linuxbrew-core AUR (Arch Linux User Repository) : yay -S rancher-k3d-bin Package rancher-k3d-bin Download GitHub Release Grab a release binary from the release tab and install it yourself Chocolatey (Windows) : choco install k3d Note : package source can be found in erwinkersten/chocolatey-packages arkade : arkade get k3d asdf : asdf plugin-add k3d && asdf install k3d latest Note : asdf plugin-add k3d , then asdf install k3d <tag> with <tag> = latest or 5.x.x for a specific version (maintained by spencergilbert/asdf-k3d ) Others install via go: go install github.com/rancher/k3d@latest ( Note : this will give you unreleased/bleeding-edge changes)","title":"Other Installers"},{"location":"#quick-start","text":"Create a cluster named mycluster with just a single server node: k3d cluster create mycluster Use the new cluster with kubectl , e.g.: kubectl get nodes Getting the cluster\u2019s kubeconfig (included in k3d cluster create ) Get the new cluster\u2019s connection details merged into your default kubeconfig (usually specified using the KUBECONFIG environment variable or the default path $HOME /.kube/config ) and directly switch to the new context: k3d kubeconfig merge mycluster --kubeconfig-switch-context","title":"Quick Start"},{"location":"#related-projects","text":"vscode-k3d : VSCode Extension to handle k3d clusters from within VSCode k3x : a graphics interface (for Linux) to k3d. AbsaOSS/k3d-action : fully customizable GitHub Action to run lightweight Kubernetes clusters. AutoK3s : a lightweight tool to help run K3s everywhere including k3d provider. nolar/setup-k3d-k3s : setup K3d/K3s for GitHub Actions.","title":"Related Projects"},{"location":"design/defaults/","text":"Defaults \u00b6 k3d reserved settings \u00b6 When you create a K3s cluster in Docker using k3d, we make use of some K3s configuration options, making them \u201creserved\u201d for k3d. This means, that overriding those options with your own may break the cluster setup. Environment Variables \u00b6 The following K3s environment variables are used to configure the cluster: Variable K3d Default Configurable? K3S_URL https://$CLUSTERNAME-server-0:6443 no K3S_TOKEN random yes ( --token ) K3S_KUBECONFIG_OUTPUT /output/kubeconfig.yaml no k3d Loadbalancer \u00b6 By default, k3d creates an Nginx loadbalancer alongside the clusters it creates to handle the port-forwarding. The loadbalancer can partly be configured using k3d-defined settings. Nginx setting k3d default k3d setting proxy_timeout (default for all server stanzas) 600 (s) settings.defaultProxyTimeout worker_connections 1024 settings.workerConnections Overrides \u00b6 Example via CLI: k3d cluster create --lb-config-override settings.defaultProxyTimeout=900 Example via Config File: # ... truncated ... k3d : loadbalancer : configOverrides : - settings.workerConnections=2048 Multiple server nodes \u00b6 by default, when --server > 1 and no --datastore-x option is set, the first server node (server-0) will be the initializing server node the initializing server node will have the --cluster-init flag appended all other server nodes will refer to the initializing server node via --server https://<init-node>:6443 API-Ports \u00b6 by default, we expose the API-Port ( 6443 ) by forwarding traffic from the default server loadbalancer (nginx container) to the server node(s) port 6443 of the loadbalancer is then mapped to a specific ( --api-port flag) or a random (default) port on the host system Kubeconfig \u00b6 if --kubeconfig-update-default is set, we use the default loading rules to get the default kubeconfig: First: kubeconfig specified via the KUBECONFIG environment variable (error out if multiple are specified) Second: default kubeconfig in home directory (e.g. $HOME/.kube/config ) Networking \u00b6 by default, k3d creates a new (docker) network for every cluster","title":"Defaults"},{"location":"design/defaults/#defaults","text":"","title":"Defaults"},{"location":"design/defaults/#k3d-reserved-settings","text":"When you create a K3s cluster in Docker using k3d, we make use of some K3s configuration options, making them \u201creserved\u201d for k3d. This means, that overriding those options with your own may break the cluster setup.","title":"k3d reserved settings"},{"location":"design/defaults/#environment-variables","text":"The following K3s environment variables are used to configure the cluster: Variable K3d Default Configurable? K3S_URL https://$CLUSTERNAME-server-0:6443 no K3S_TOKEN random yes ( --token ) K3S_KUBECONFIG_OUTPUT /output/kubeconfig.yaml no","title":"Environment Variables"},{"location":"design/defaults/#k3d-loadbalancer","text":"By default, k3d creates an Nginx loadbalancer alongside the clusters it creates to handle the port-forwarding. The loadbalancer can partly be configured using k3d-defined settings. Nginx setting k3d default k3d setting proxy_timeout (default for all server stanzas) 600 (s) settings.defaultProxyTimeout worker_connections 1024 settings.workerConnections","title":"k3d Loadbalancer"},{"location":"design/defaults/#overrides","text":"Example via CLI: k3d cluster create --lb-config-override settings.defaultProxyTimeout=900 Example via Config File: # ... truncated ... k3d : loadbalancer : configOverrides : - settings.workerConnections=2048","title":"Overrides"},{"location":"design/defaults/#multiple-server-nodes","text":"by default, when --server > 1 and no --datastore-x option is set, the first server node (server-0) will be the initializing server node the initializing server node will have the --cluster-init flag appended all other server nodes will refer to the initializing server node via --server https://<init-node>:6443","title":"Multiple server nodes"},{"location":"design/defaults/#api-ports","text":"by default, we expose the API-Port ( 6443 ) by forwarding traffic from the default server loadbalancer (nginx container) to the server node(s) port 6443 of the loadbalancer is then mapped to a specific ( --api-port flag) or a random (default) port on the host system","title":"API-Ports"},{"location":"design/defaults/#kubeconfig","text":"if --kubeconfig-update-default is set, we use the default loading rules to get the default kubeconfig: First: kubeconfig specified via the KUBECONFIG environment variable (error out if multiple are specified) Second: default kubeconfig in home directory (e.g. $HOME/.kube/config )","title":"Kubeconfig"},{"location":"design/defaults/#networking","text":"by default, k3d creates a new (docker) network for every cluster","title":"Networking"},{"location":"design/networking/","text":"Networking \u00b6 Related issues: rancher/k3d #220 Introduction \u00b6 By default, k3d creates a new (docker) network for every new cluster. Using the --network STRING flag upon creation to connect to an existing network. Existing networks won\u2019t be managed by k3d together with the cluster lifecycle. Connecting to docker \u201cinternal\u201d/pre-defined networks \u00b6 host network \u00b6 When using the --network flag to connect to the host network (i.e. k3d cluster create --network host ), you won\u2019t be able to create more than one server node . An edge case would be one server node (with agent disabled) and one agent node. bridge network \u00b6 By default, every network that k3d creates is working in bridge mode. But when you try to use --network bridge to connect to docker\u2019s internal bridge network, you may run into issues with grabbing certificates from the API-Server. Single-Node clusters should work though. none \u201cnetwork\u201d \u00b6 Well.. this doesn\u2019t really make sense for k3d anyway \u00af\\_(\u30c4)_/\u00af","title":"Networking"},{"location":"design/networking/#networking","text":"Related issues: rancher/k3d #220","title":"Networking"},{"location":"design/networking/#introduction","text":"By default, k3d creates a new (docker) network for every new cluster. Using the --network STRING flag upon creation to connect to an existing network. Existing networks won\u2019t be managed by k3d together with the cluster lifecycle.","title":"Introduction"},{"location":"design/networking/#connecting-to-docker-internalpre-defined-networks","text":"","title":"Connecting to docker \"internal\"/pre-defined networks"},{"location":"design/networking/#host-network","text":"When using the --network flag to connect to the host network (i.e. k3d cluster create --network host ), you won\u2019t be able to create more than one server node . An edge case would be one server node (with agent disabled) and one agent node.","title":"host network"},{"location":"design/networking/#bridge-network","text":"By default, every network that k3d creates is working in bridge mode. But when you try to use --network bridge to connect to docker\u2019s internal bridge network, you may run into issues with grabbing certificates from the API-Server. Single-Node clusters should work though.","title":"bridge network"},{"location":"design/networking/#none-network","text":"Well.. this doesn\u2019t really make sense for k3d anyway \u00af\\_(\u30c4)_/\u00af","title":"none \"network\""},{"location":"design/project/","text":"Project Overview \u00b6 About This Page \u00b6 On this page we\u2019ll try to give an overview of all the moving bits and pieces in k3d to ease contributions to the project. Directory Overview \u00b6 .github/ templates for issues and pull requests GitHub Action workflow definitions cmd/ everything related to the actual k3d CLI, like the whole command tree, config initialization, argument parsing, etc. docgen/ sub-module used to auto-generate the documentation for the CLI commands, which ends up in docs/usage/commands/ docs/ all the resources used to build k3d.io using mkdocs pkg/ the place where the magic happens.. here you find all the main logic of k3d all function calls within cmd/ that do non-trivial things are imported from here this (or rather sub-packages) is what other projects would import as a module to work with k3d without using the CLI proxy/ configuration to build the rancher/k3d-proxy container image which is used as a loadbalancer/proxy in front of (almost) every k3d cluster this is basically just a combination of NGINX with confd and some k3d-specific configuration details tests/ a set of bash scripts used for end-to-end (E2E) tests of k3d mostly used for all the functionality of the k3d CLI which cannot be tested using Go unit tests tools/ sub-module used to build the rancher/k3d-tools container image which supports some k3d functionality like k3d image import vendor/ result of go mod vendor , which contains all dependencies of k3d version/ package used to code k3d/k3s versions into releases this is where go build injects the version tags when building k3d that\u2019s the output you see when issuing k3d version Packages Overview \u00b6 pkg/ actions/ hook actions describing actions (commands, etc.) that run at specific stages of the node/cluster lifecycle e.g. writing configuration files to the container filesystem just before the node (container) starts client/ all the top level functionality to work with k3d primitives create/retrieve/update/delete/start/stop clusters, nodes, registries, etc. managed by k3d config/ everything related to the k3d configuration (files), like SimpleConfig and ClusterConfig runtimes/ interface and implementations of runtimes that power k3d (currently, that\u2019s only Docker) functions in client/ eventually call runtime functions to \u201cmaterialize\u201d nodes and clusters tools/ functions eventually calling the k3d-tools container (see tools/ in the repo root) types/ definition of all k3d primitives and many other details and defaults e.g. contains the definition of a Node or a Cluster in k3d util/ some helper functions e.g. for string manipulation/generation, regexp or other re-usable usages Anatomy of a Cluster \u00b6 By default, every k3d cluster consists of at least 2 containers (nodes): (optional, but default and strongly recommended) loadbalancer image: rancher/k3d-proxy , built from proxy/ purpose: proxy and load balance requests from the outside (i.e. most of the times your local host) to the cluster by default, it e.g. proxies all the traffic for the Kubernetes API to port 6443 (default listening port of K3s) to all the server nodes in the cluster can be used for multiple port-mappings to one or more nodes in your cluster that way, port-mappings can also easily be added/removed after the cluster creation, as we can simply re-create the proxy without affecting cluster state (required, always present) primary server node image: rancher/k3s , built from github.com/k3s-io/k3s purpose: (initializing) server (formerly: master) node of the cluster runs the K3s executable (which runs containerd, the Kubernetes API Server, etcd/sqlite, etc.): k3s server in a multi-server setup, it initializes the cluster with an embedded etcd database (using the K3s --cluster-init flag) (optional) secondary server node(s) image: rancher/k3s , built from github.com/k3s-io/k3s (optional) agent node(s) image: rancher/k3s , built from github.com/k3s-io/k3s purpose: running the K3s agent process (kubelet, etc.): k3s agent Automation (CI) \u00b6 The k3d repository mainly leverages the following two CI systems: GitHub Actions 2 workflows in https://github.com/rancher/k3d/tree/main/.github/workflows to push the artifact to AUR (Arch Linux User Repository) logs/history can be seen in the Actions tab: https://github.com/rancher/k3d/actions DroneCI a set of pipelines in a single file: https://github.com/rancher/k3d/blob/main/.drone.yml static code analysis build tests docker builds + pushes render + push docs (pre-) release to GitHub push events end up here (also does the releases, when a tag is pushed): https://drone-publish.rancher.io/rancher/k3d pr s end up here: https://drone-pr.rancher.io/rancher/k3d Documentation \u00b6 The website k3d.io containing all the documentation for k3d is built using mkdocs , configured via the mkdocs.yml config file with all the content residing in the docs/ directory (Markdown). Use mkdocs serve in the repository root to build and serve the webpage locally. Some parts of the documentation are being auto-generated, like docs/usage/commands/ is auto-generated using Cobra\u2019s command docs generation functionality in docgen/ .","title":"Project Overview"},{"location":"design/project/#project-overview","text":"","title":"Project Overview"},{"location":"design/project/#about-this-page","text":"On this page we\u2019ll try to give an overview of all the moving bits and pieces in k3d to ease contributions to the project.","title":"About This Page"},{"location":"design/project/#directory-overview","text":".github/ templates for issues and pull requests GitHub Action workflow definitions cmd/ everything related to the actual k3d CLI, like the whole command tree, config initialization, argument parsing, etc. docgen/ sub-module used to auto-generate the documentation for the CLI commands, which ends up in docs/usage/commands/ docs/ all the resources used to build k3d.io using mkdocs pkg/ the place where the magic happens.. here you find all the main logic of k3d all function calls within cmd/ that do non-trivial things are imported from here this (or rather sub-packages) is what other projects would import as a module to work with k3d without using the CLI proxy/ configuration to build the rancher/k3d-proxy container image which is used as a loadbalancer/proxy in front of (almost) every k3d cluster this is basically just a combination of NGINX with confd and some k3d-specific configuration details tests/ a set of bash scripts used for end-to-end (E2E) tests of k3d mostly used for all the functionality of the k3d CLI which cannot be tested using Go unit tests tools/ sub-module used to build the rancher/k3d-tools container image which supports some k3d functionality like k3d image import vendor/ result of go mod vendor , which contains all dependencies of k3d version/ package used to code k3d/k3s versions into releases this is where go build injects the version tags when building k3d that\u2019s the output you see when issuing k3d version","title":"Directory Overview"},{"location":"design/project/#packages-overview","text":"pkg/ actions/ hook actions describing actions (commands, etc.) that run at specific stages of the node/cluster lifecycle e.g. writing configuration files to the container filesystem just before the node (container) starts client/ all the top level functionality to work with k3d primitives create/retrieve/update/delete/start/stop clusters, nodes, registries, etc. managed by k3d config/ everything related to the k3d configuration (files), like SimpleConfig and ClusterConfig runtimes/ interface and implementations of runtimes that power k3d (currently, that\u2019s only Docker) functions in client/ eventually call runtime functions to \u201cmaterialize\u201d nodes and clusters tools/ functions eventually calling the k3d-tools container (see tools/ in the repo root) types/ definition of all k3d primitives and many other details and defaults e.g. contains the definition of a Node or a Cluster in k3d util/ some helper functions e.g. for string manipulation/generation, regexp or other re-usable usages","title":"Packages Overview"},{"location":"design/project/#anatomy-of-a-cluster","text":"By default, every k3d cluster consists of at least 2 containers (nodes): (optional, but default and strongly recommended) loadbalancer image: rancher/k3d-proxy , built from proxy/ purpose: proxy and load balance requests from the outside (i.e. most of the times your local host) to the cluster by default, it e.g. proxies all the traffic for the Kubernetes API to port 6443 (default listening port of K3s) to all the server nodes in the cluster can be used for multiple port-mappings to one or more nodes in your cluster that way, port-mappings can also easily be added/removed after the cluster creation, as we can simply re-create the proxy without affecting cluster state (required, always present) primary server node image: rancher/k3s , built from github.com/k3s-io/k3s purpose: (initializing) server (formerly: master) node of the cluster runs the K3s executable (which runs containerd, the Kubernetes API Server, etcd/sqlite, etc.): k3s server in a multi-server setup, it initializes the cluster with an embedded etcd database (using the K3s --cluster-init flag) (optional) secondary server node(s) image: rancher/k3s , built from github.com/k3s-io/k3s (optional) agent node(s) image: rancher/k3s , built from github.com/k3s-io/k3s purpose: running the K3s agent process (kubelet, etc.): k3s agent","title":"Anatomy of a Cluster"},{"location":"design/project/#automation-ci","text":"The k3d repository mainly leverages the following two CI systems: GitHub Actions 2 workflows in https://github.com/rancher/k3d/tree/main/.github/workflows to push the artifact to AUR (Arch Linux User Repository) logs/history can be seen in the Actions tab: https://github.com/rancher/k3d/actions DroneCI a set of pipelines in a single file: https://github.com/rancher/k3d/blob/main/.drone.yml static code analysis build tests docker builds + pushes render + push docs (pre-) release to GitHub push events end up here (also does the releases, when a tag is pushed): https://drone-publish.rancher.io/rancher/k3d pr s end up here: https://drone-pr.rancher.io/rancher/k3d","title":"Automation (CI)"},{"location":"design/project/#documentation","text":"The website k3d.io containing all the documentation for k3d is built using mkdocs , configured via the mkdocs.yml config file with all the content residing in the docs/ directory (Markdown). Use mkdocs serve in the repository root to build and serve the webpage locally. Some parts of the documentation are being auto-generated, like docs/usage/commands/ is auto-generated using Cobra\u2019s command docs generation functionality in docgen/ .","title":"Documentation"},{"location":"faq/faq/","text":"FAQ \u00b6 Issues with BTRFS \u00b6 As @jaredallard pointed out , people running k3d on a system with btrfs , may need to mount /dev/mapper into the nodes for the setup to work. This will do: k3d cluster create CLUSTER_NAME -v /dev/mapper:/dev/mapper Issues with ZFS \u00b6 k3s currently has no support for ZFS and thus, creating multi-server setups (e.g. k3d cluster create multiserver --servers 3 ) fails, because the initializing server node (server flag --cluster-init ) errors out with the following log: starting kubernetes: preparing server: start cluster and https: raft_init () : io: create I/O capabilities probe file: posix_allocate: operation not supported on socket This issue can be worked around by providing docker with a different filesystem (that\u2019s also better for docker-in-docker stuff). A possible solution can be found here: https://github.com/rancher/k3s/issues/1688#issuecomment-619570374 Pods evicted due to lack of disk space \u00b6 Pods go to evicted state after doing X Related issues: #133 - Pods evicted due to NodeHasDiskPressure (collection of #119 and #130) Background: somehow docker runs out of space for the k3d node containers, which triggers a hard eviction in the kubelet Possible fix/workaround by @zer0def : use a docker storage driver which cleans up properly (e.g. overlay2) clean up or expand docker root filesystem change the kubelet\u2019s eviction thresholds upon cluster creation: k3d cluster create \\ --k3s-agent-arg '--kubelet-arg=eviction-hard=imagefs.available<1%,nodefs.available<1%' \\ --k3s-agent-arg '--kubelet-arg=eviction-minimum-reclaim=imagefs.available=1%,nodefs.available=1%' Restarting a multi-server cluster or the initializing server node fails \u00b6 What you do: You create a cluster with more than one server node and later, you either stop server-0 or stop/start the whole cluster What fails: After the restart, you cannot connect to the cluster anymore and kubectl will give you a lot of errors What causes this issue: it\u2019s a known issue with dqlite in k3s which doesn\u2019t allow the initializing server node to go down What\u2019s the solution: Hopefully, this will be solved by the planned replacement of dqlite with embedded etcd in k3s Related issues: #262 Passing additional arguments/flags to k3s (and on to e.g. the kube-apiserver) \u00b6 The Problem: Passing a feature flag to the Kubernetes API Server running inside k3s. Example: you want to enable the EphemeralContainers feature flag in Kubernetes Solution: k3d cluster create --k3s-server-arg '--kube-apiserver-arg=feature-gates=EphemeralContainers=true' Note : Be aware of where the flags require dashes ( -- ) and where not. the k3s flag ( --kube-apiserver-arg ) has the dashes the kube-apiserver flag feature-gates doesn\u2019t have them (k3s adds them internally) Second example: k3d cluster create k3d-one \\ --k3s-server-arg --cluster-cidr = \"10.118.0.0/17\" \\ --k3s-server-arg --service-cidr = \"10.118.128.0/17\" \\ --k3s-server-arg --disable = servicelb \\ --k3s-server-arg --disable = traefik \\ --verbose Note : There are many ways to use the \" and ' quotes, just be aware, that sometimes shells also try to interpret/interpolate parts of the commands How to access services (like a database) running on my Docker Host Machine \u00b6 As of version v3.1.0, we\u2019re injecting the host.k3d.internal entry into the k3d containers (k3s nodes) and into the CoreDNS ConfigMap, enabling you to access your host system by referring to it as host.k3d.internal Running behind a corporate proxy \u00b6 Running k3d behind a corporate proxy can lead to some issues with k3d that have already been reported in more than one issue. Some can be fixed by passing the HTTP_PROXY environment variables to k3d, some have to be fixed in docker\u2019s daemon.json file and some are as easy as adding a volume mount. Pods fail to start: x509: certificate signed by unknown authority \u00b6 Example Error Message: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to get sandbox image \"docker.io/rancher/pause:3.1\" : failed to pull image \"docker.io/rancher/pause:3.1\" : failed to pull and unpack image \"docker.io/rancher/pause:3.1\" : failed to resolve reference \"docker.io/rancher/pause:3.1\" : failed to do request: Head https://registry-1.docker.io/v2/rancher/pause/manifests/3.1: x509: certificate signed by unknown authority Problem: inside the container, the certificate of the corporate proxy cannot be validated Possible Solution: Mounting the CA Certificate from your host into the node containers at start time via k3d cluster create --volume /path/to/your/certs.crt:/etc/ssl/certs/yourcert.crt Issue: rancher/k3d#535 Spurious PID entries in /proc after deleting k3d cluster with shared mounts \u00b6 When you perform cluster create and deletion operations multiple times with same cluster name and shared volume mounts , it was observed that grep k3d /proc/*/mountinfo shows many spurious entries Problem: Due to above, at times you\u2019ll see no space left on device: unknown when a pod is scheduled to the nodes If you observe anything of above sort you can check for inaccessible file systems and unmount them by using below command (note: please remove xargs umount -l and check for the diff o/p first) diff <(df -ha | grep pods | awk '{print $NF}') <(df -h | grep pods | awk '{print $NF}') | awk '{print $2}' | xargs umount -l As per the conversation on rancher/k3d#594 above issue wasn\u2019t reported/known earlier and so there are high chances that it\u2019s not universal. [SOLVED] Nodes fail to start or get stuck in NotReady state with log nf_conntrack_max: permission denied \u00b6 Problem \u00b6 When: This happens when running k3d on a Linux system with a kernel version >= 5.12.2 (and others like >= 5.11.19) when creating a new cluster the node(s) stop or get stuck with a log line like this: <TIMESTAMP> F0516 05:05:31.782902 7 server.go:495] open /proc/sys/net/netfilter/nf_conntrack_max: permission denied Why: The issue was introduced by a change in the Linux kernel ( Changelog 5.12.2 : Commit ), that changed the netfilter_conntrack behavior in a way that kube-proxy is not able to set the nf_conntrack_max value anymore Workaround \u00b6 Workaround: as a workaround, we can tell kube-proxy to not even try to set this value: k3d cluster create \\ --k3s-server-arg \"--kube-proxy-arg=conntrack-max-per-core=0\" \\ --k3s-agent-arg \"--kube-proxy-arg=conntrack-max-per-core=0\" \\ --image rancher/k3s:v1.20.6-k3s Fix \u00b6 Note : k3d v4.4.5 already uses rancher/k3s:v1.21.1-k3s1 as the new default k3s image, so no workarounds needed there! This is going to be fixed \u201cupstream\u201d in k3s itself in rancher/k3s#3337 and backported to k3s versions as low as v1.18. The fix was released and backported in k3s, so you don\u2019t need to use the workaround when using one of the following k3s versions (or later ones) v1.18.19-k3s1 ( rancher/k3s#3344 ) v1.19.11-k3s1 ( rancher/k3s#3343 ) v1.20.7-k3s1 ( rancher/k3s#3342 ) v1.21.1-k3s1 ( rancher/k3s#3341 )) Issue Reference: rancher/k3s#607 DockerHub Pull Rate Limit \u00b6 Problem \u00b6 You\u2019re deploying something to the cluster using an image from DockerHub and the image fails to be pulled, with a 429 response code and a message saying You have reached your pull rate limit. You may increase the limit by authenticating and upgrading . Cause \u00b6 This is caused by DockerHub\u2019s pull rate limit (see https://docs.docker.com/docker-hub/download-rate-limit/ ), which limits pulls from unauthenticated/anonymous users to 100 pulls per hour and for authenticated users (not paying customers) to 200 pulls per hour (as of the time of writing). Solution \u00b6 a) use images from a private registry, e.g. configured as a pull-through cache for DockerHub b) use a different public registry without such limitations, if the same image is stored there c) authenticate containerd inside k3s/k3d to use your DockerHub user (c) Authenticate Containerd against DockerHub \u00b6 Create a registry configuration file for containerd: # saved as e.g. $HOME/registries.yaml configs : \"docker.io\" : auth : username : \"$USERNAME\" password : \"$PASSWORD\" Create a k3d cluster using that config: k3d cluster create --registry-config $HOME /registries.yaml Profit. That\u2019s it. In the test for this, we pulled the same image 120 times in a row (confirmed, that pull numbers went up), without being rate limited (as a non-paying, normal user)","title":"FAQ"},{"location":"faq/faq/#faq","text":"","title":"FAQ"},{"location":"faq/faq/#issues-with-btrfs","text":"As @jaredallard pointed out , people running k3d on a system with btrfs , may need to mount /dev/mapper into the nodes for the setup to work. This will do: k3d cluster create CLUSTER_NAME -v /dev/mapper:/dev/mapper","title":"Issues with BTRFS"},{"location":"faq/faq/#issues-with-zfs","text":"k3s currently has no support for ZFS and thus, creating multi-server setups (e.g. k3d cluster create multiserver --servers 3 ) fails, because the initializing server node (server flag --cluster-init ) errors out with the following log: starting kubernetes: preparing server: start cluster and https: raft_init () : io: create I/O capabilities probe file: posix_allocate: operation not supported on socket This issue can be worked around by providing docker with a different filesystem (that\u2019s also better for docker-in-docker stuff). A possible solution can be found here: https://github.com/rancher/k3s/issues/1688#issuecomment-619570374","title":"Issues with ZFS"},{"location":"faq/faq/#pods-evicted-due-to-lack-of-disk-space","text":"Pods go to evicted state after doing X Related issues: #133 - Pods evicted due to NodeHasDiskPressure (collection of #119 and #130) Background: somehow docker runs out of space for the k3d node containers, which triggers a hard eviction in the kubelet Possible fix/workaround by @zer0def : use a docker storage driver which cleans up properly (e.g. overlay2) clean up or expand docker root filesystem change the kubelet\u2019s eviction thresholds upon cluster creation: k3d cluster create \\ --k3s-agent-arg '--kubelet-arg=eviction-hard=imagefs.available<1%,nodefs.available<1%' \\ --k3s-agent-arg '--kubelet-arg=eviction-minimum-reclaim=imagefs.available=1%,nodefs.available=1%'","title":"Pods evicted due to lack of disk space"},{"location":"faq/faq/#restarting-a-multi-server-cluster-or-the-initializing-server-node-fails","text":"What you do: You create a cluster with more than one server node and later, you either stop server-0 or stop/start the whole cluster What fails: After the restart, you cannot connect to the cluster anymore and kubectl will give you a lot of errors What causes this issue: it\u2019s a known issue with dqlite in k3s which doesn\u2019t allow the initializing server node to go down What\u2019s the solution: Hopefully, this will be solved by the planned replacement of dqlite with embedded etcd in k3s Related issues: #262","title":"Restarting a multi-server cluster or the initializing server node fails"},{"location":"faq/faq/#passing-additional-argumentsflags-to-k3s-and-on-to-eg-the-kube-apiserver","text":"The Problem: Passing a feature flag to the Kubernetes API Server running inside k3s. Example: you want to enable the EphemeralContainers feature flag in Kubernetes Solution: k3d cluster create --k3s-server-arg '--kube-apiserver-arg=feature-gates=EphemeralContainers=true' Note : Be aware of where the flags require dashes ( -- ) and where not. the k3s flag ( --kube-apiserver-arg ) has the dashes the kube-apiserver flag feature-gates doesn\u2019t have them (k3s adds them internally) Second example: k3d cluster create k3d-one \\ --k3s-server-arg --cluster-cidr = \"10.118.0.0/17\" \\ --k3s-server-arg --service-cidr = \"10.118.128.0/17\" \\ --k3s-server-arg --disable = servicelb \\ --k3s-server-arg --disable = traefik \\ --verbose Note : There are many ways to use the \" and ' quotes, just be aware, that sometimes shells also try to interpret/interpolate parts of the commands","title":"Passing additional arguments/flags to k3s (and on to e.g. the kube-apiserver)"},{"location":"faq/faq/#how-to-access-services-like-a-database-running-on-my-docker-host-machine","text":"As of version v3.1.0, we\u2019re injecting the host.k3d.internal entry into the k3d containers (k3s nodes) and into the CoreDNS ConfigMap, enabling you to access your host system by referring to it as host.k3d.internal","title":"How to access services (like a database) running on my Docker Host Machine"},{"location":"faq/faq/#running-behind-a-corporate-proxy","text":"Running k3d behind a corporate proxy can lead to some issues with k3d that have already been reported in more than one issue. Some can be fixed by passing the HTTP_PROXY environment variables to k3d, some have to be fixed in docker\u2019s daemon.json file and some are as easy as adding a volume mount.","title":"Running behind a corporate proxy"},{"location":"faq/faq/#pods-fail-to-start-x509-certificate-signed-by-unknown-authority","text":"Example Error Message: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to get sandbox image \"docker.io/rancher/pause:3.1\" : failed to pull image \"docker.io/rancher/pause:3.1\" : failed to pull and unpack image \"docker.io/rancher/pause:3.1\" : failed to resolve reference \"docker.io/rancher/pause:3.1\" : failed to do request: Head https://registry-1.docker.io/v2/rancher/pause/manifests/3.1: x509: certificate signed by unknown authority Problem: inside the container, the certificate of the corporate proxy cannot be validated Possible Solution: Mounting the CA Certificate from your host into the node containers at start time via k3d cluster create --volume /path/to/your/certs.crt:/etc/ssl/certs/yourcert.crt Issue: rancher/k3d#535","title":"Pods fail to start: x509: certificate signed by unknown authority"},{"location":"faq/faq/#spurious-pid-entries-in-proc-after-deleting-k3d-cluster-with-shared-mounts","text":"When you perform cluster create and deletion operations multiple times with same cluster name and shared volume mounts , it was observed that grep k3d /proc/*/mountinfo shows many spurious entries Problem: Due to above, at times you\u2019ll see no space left on device: unknown when a pod is scheduled to the nodes If you observe anything of above sort you can check for inaccessible file systems and unmount them by using below command (note: please remove xargs umount -l and check for the diff o/p first) diff <(df -ha | grep pods | awk '{print $NF}') <(df -h | grep pods | awk '{print $NF}') | awk '{print $2}' | xargs umount -l As per the conversation on rancher/k3d#594 above issue wasn\u2019t reported/known earlier and so there are high chances that it\u2019s not universal.","title":"Spurious PID entries in /proc after deleting k3d cluster with shared mounts"},{"location":"faq/faq/#solved-nodes-fail-to-start-or-get-stuck-in-notready-state-with-log-nf_conntrack_max-permission-denied","text":"","title":"[SOLVED] Nodes fail to start or get stuck in NotReady state with log nf_conntrack_max: permission denied"},{"location":"faq/faq/#problem","text":"When: This happens when running k3d on a Linux system with a kernel version >= 5.12.2 (and others like >= 5.11.19) when creating a new cluster the node(s) stop or get stuck with a log line like this: <TIMESTAMP> F0516 05:05:31.782902 7 server.go:495] open /proc/sys/net/netfilter/nf_conntrack_max: permission denied Why: The issue was introduced by a change in the Linux kernel ( Changelog 5.12.2 : Commit ), that changed the netfilter_conntrack behavior in a way that kube-proxy is not able to set the nf_conntrack_max value anymore","title":"Problem"},{"location":"faq/faq/#workaround","text":"Workaround: as a workaround, we can tell kube-proxy to not even try to set this value: k3d cluster create \\ --k3s-server-arg \"--kube-proxy-arg=conntrack-max-per-core=0\" \\ --k3s-agent-arg \"--kube-proxy-arg=conntrack-max-per-core=0\" \\ --image rancher/k3s:v1.20.6-k3s","title":"Workaround"},{"location":"faq/faq/#fix","text":"Note : k3d v4.4.5 already uses rancher/k3s:v1.21.1-k3s1 as the new default k3s image, so no workarounds needed there! This is going to be fixed \u201cupstream\u201d in k3s itself in rancher/k3s#3337 and backported to k3s versions as low as v1.18. The fix was released and backported in k3s, so you don\u2019t need to use the workaround when using one of the following k3s versions (or later ones) v1.18.19-k3s1 ( rancher/k3s#3344 ) v1.19.11-k3s1 ( rancher/k3s#3343 ) v1.20.7-k3s1 ( rancher/k3s#3342 ) v1.21.1-k3s1 ( rancher/k3s#3341 )) Issue Reference: rancher/k3s#607","title":"Fix"},{"location":"faq/faq/#dockerhub-pull-rate-limit","text":"","title":"DockerHub Pull Rate Limit"},{"location":"faq/faq/#problem_1","text":"You\u2019re deploying something to the cluster using an image from DockerHub and the image fails to be pulled, with a 429 response code and a message saying You have reached your pull rate limit. You may increase the limit by authenticating and upgrading .","title":"Problem"},{"location":"faq/faq/#cause","text":"This is caused by DockerHub\u2019s pull rate limit (see https://docs.docker.com/docker-hub/download-rate-limit/ ), which limits pulls from unauthenticated/anonymous users to 100 pulls per hour and for authenticated users (not paying customers) to 200 pulls per hour (as of the time of writing).","title":"Cause"},{"location":"faq/faq/#solution","text":"a) use images from a private registry, e.g. configured as a pull-through cache for DockerHub b) use a different public registry without such limitations, if the same image is stored there c) authenticate containerd inside k3s/k3d to use your DockerHub user","title":"Solution"},{"location":"faq/faq/#c-authenticate-containerd-against-dockerhub","text":"Create a registry configuration file for containerd: # saved as e.g. $HOME/registries.yaml configs : \"docker.io\" : auth : username : \"$USERNAME\" password : \"$PASSWORD\" Create a k3d cluster using that config: k3d cluster create --registry-config $HOME /registries.yaml Profit. That\u2019s it. In the test for this, we pulled the same image 120 times in a row (confirmed, that pull numbers went up), without being rate limited (as a non-paying, normal user)","title":"(c) Authenticate Containerd against DockerHub"},{"location":"usage/commands/","text":"Command Tree \u00b6 k3d --verbose # GLOBAL: enable verbose (debug) logging (default: false) --trace # GLOBAL: enable super verbose logging (trace logging) (default: false) --version # show k3d and k3s version -h, --help # GLOBAL: show help text cluster [ CLUSTERNAME ] # default cluster name is 'k3s-default' create -a, --agents # specify how many agent nodes you want to create (integer, default: 0) --agents-memory # specify memory limit for agent containers/nodes (unit, e.g. 1g) --api-port # specify the port on which the cluster will be accessible (format '[HOST:]HOSTPORT', default: random) -c, --config # use a config file (format 'PATH') -e, --env # add environment variables to the nodes (quoted string, format: 'KEY[=VALUE][@NODEFILTER[;NODEFILTER...]]', use flag multiple times) --gpus # [from docker CLI] add GPU devices to the node containers (string, e.g. 'all') -i, --image # specify which k3s image should be used for the nodes (string, default: 'docker.io/rancher/k3s:v1.20.0-k3s2', tag changes per build) --k3s-agent-arg # add additional arguments to the k3s agent (quoted string, use flag multiple times) (see https://rancher.com/docs/k3s/latest/en/installation/install-options/agent-config/#k3s-agent-cli-help) --k3s-server-arg # add additional arguments to the k3s server (quoted string, use flag multiple times) (see https://rancher.com/docs/k3s/latest/en/installation/install-options/server-config/#k3s-server-cli-help) --kubeconfig-switch-context # (implies --kubeconfig-update-default) automatically sets the current-context of your default kubeconfig to the new cluster's context (default: true) --kubeconfig-update-default # enable the automated update of the default kubeconfig with the details of the newly created cluster (also sets '--wait=true') (default: true) -l, --label # add (docker) labels to the node containers (format: 'KEY[=VALUE][@NODEFILTER[;NODEFILTER...]]', use flag multiple times) --network # specify an existing (docker) network you want to connect to (string) --no-hostip # disable the automatic injection of the Host IP as 'host.k3d.internal' into the containers and CoreDNS (default: false) --no-image-volume # disable the creation of a volume for storing images (used for the 'k3d image import' command) (default: false) --no-lb # disable the creation of a load balancer in front of the server nodes (default: false) --no-rollback # disable the automatic rollback actions, if anything goes wrong (default: false) -p, --port # add some more port mappings (format: '[HOST:][HOSTPORT:]CONTAINERPORT[/PROTOCOL][@NODEFILTER]', use flag multiple times) --registry-create # create a new (docker) registry dedicated for this cluster (default: false) --registry-use # use an existing local (docker) registry with this cluster (string, use multiple times) -s, --servers # specify how many server nodes you want to create (integer, default: 1) --servers-memory # specify memory limit for server containers/nodes (unit, e.g. 1g) --token # specify a cluster token (string, default: auto-generated) --timeout # specify a timeout, after which the cluster creation will be interrupted and changes rolled back (duration, e.g. '10s') -v, --volume # specify additional bind-mounts (format: '[SOURCE:]DEST[@NODEFILTER[;NODEFILTER...]]', use flag multiple times) --wait # enable waiting for all server nodes to be ready before returning (default: true) start CLUSTERNAME # start a (stopped) cluster -a, --all # start all clusters (default: false) --wait # wait for all servers and server-loadbalancer to be up before returning (default: true) --timeout # maximum waiting time for '--wait' before canceling/returning (duration, e.g. '10s') stop CLUSTERNAME # stop a cluster -a, --all # stop all clusters (default: false) delete CLUSTERNAME # delete an existing cluster -a, --all # delete all existing clusters (default: false) list [ CLUSTERNAME [ CLUSTERNAME ... ]] --no-headers # do not print headers (default: false) --token # show column with cluster tokens (default: false) -o, --output # format the output (format: 'json|yaml') completion [ bash | zsh | fish | ( psh | powershell )] # generate completion scripts for common shells config init # write a default k3d config (as a starting point) -f, --force # force overwrite target file (default: false) -o, --output # file to write to (string, default \"k3d-default.yaml\") help [ COMMAND ] # show help text for any command image import [ IMAGE | ARCHIVE [ IMAGE | ARCHIVE ... ]] # Load one or more images from the local runtime environment or tar-archives into k3d clusters -c, --cluster # clusters to load the image into (string, use flag multiple times, default: k3s-default) -k, --keep-tarball # do not delete the image tarball from the shared volume after completion (default: false) kubeconfig get ( CLUSTERNAME [ CLUSTERNAME ... ] | --all ) # get kubeconfig from cluster(s) and write it to stdout -a, --all # get kubeconfigs from all clusters (default: false) merge | write ( CLUSTERNAME [ CLUSTERNAME ... ] | --all ) # get kubeconfig from cluster(s) and merge it/them into a (kubeconfig-)file -a, --all # get kubeconfigs from all clusters (default: false) -s, --kubeconfig-switch-context # switch current-context in kubeconfig to the new context (default: true) -d, --kubeconfig-merge-default # update the default kubeconfig (usually $KUBECONFIG or $HOME/.kube/config) -o, --output # specify the output file where the kubeconfig should be written to (string) --overwrite # [Careful!] forcefully overwrite the output file, ignoring existing contents (default: false) -u, --update # update conflicting fields in existing kubeconfig (default: true) node create NODENAME # Create new nodes (and add them to existing clusters) -c, --cluster # specify the cluster that the node shall connect to (string, default: k3s-default) -i, --image # specify which k3s image should be used for the node(s) (string, default: 'docker.io/rancher/k3s:v1.20.0-k3s2', tag changes per build) --replicas # specify how many replicas you want to create with this spec (integer, default: 1) --role # specify the node role (string, format: 'agent|server', default: agent) --timeout # specify a timeout duration, after which the node creation will be interrupted, if not done yet (duration, e.g. '10s') --wait # wait for the node to be up and running before returning (default: true) start NODENAME # start a (stopped) node stop NODENAME # stop a node delete NODENAME # delete an existing node -a, --all # delete all existing nodes (default: false) -r, --registries # also delete registries, as a special type of node (default: false) list NODENAME --no-headers # do not print headers (default: false) registry create REGISTRYNAME -i, --image # specify image used for the registry (string, default: \"docker.io/library/registry:2\") -p, --port # select host port to map to (format: '[HOST:]HOSTPORT', default: 'random') delete REGISTRYNAME -a, --all # delete all existing registries (default: false) list [ NAME [ NAME... ]] --no-headers # disable table headers (default: false) version # show k3d and k3s version","title":"Command Tree"},{"location":"usage/commands/#command-tree","text":"k3d --verbose # GLOBAL: enable verbose (debug) logging (default: false) --trace # GLOBAL: enable super verbose logging (trace logging) (default: false) --version # show k3d and k3s version -h, --help # GLOBAL: show help text cluster [ CLUSTERNAME ] # default cluster name is 'k3s-default' create -a, --agents # specify how many agent nodes you want to create (integer, default: 0) --agents-memory # specify memory limit for agent containers/nodes (unit, e.g. 1g) --api-port # specify the port on which the cluster will be accessible (format '[HOST:]HOSTPORT', default: random) -c, --config # use a config file (format 'PATH') -e, --env # add environment variables to the nodes (quoted string, format: 'KEY[=VALUE][@NODEFILTER[;NODEFILTER...]]', use flag multiple times) --gpus # [from docker CLI] add GPU devices to the node containers (string, e.g. 'all') -i, --image # specify which k3s image should be used for the nodes (string, default: 'docker.io/rancher/k3s:v1.20.0-k3s2', tag changes per build) --k3s-agent-arg # add additional arguments to the k3s agent (quoted string, use flag multiple times) (see https://rancher.com/docs/k3s/latest/en/installation/install-options/agent-config/#k3s-agent-cli-help) --k3s-server-arg # add additional arguments to the k3s server (quoted string, use flag multiple times) (see https://rancher.com/docs/k3s/latest/en/installation/install-options/server-config/#k3s-server-cli-help) --kubeconfig-switch-context # (implies --kubeconfig-update-default) automatically sets the current-context of your default kubeconfig to the new cluster's context (default: true) --kubeconfig-update-default # enable the automated update of the default kubeconfig with the details of the newly created cluster (also sets '--wait=true') (default: true) -l, --label # add (docker) labels to the node containers (format: 'KEY[=VALUE][@NODEFILTER[;NODEFILTER...]]', use flag multiple times) --network # specify an existing (docker) network you want to connect to (string) --no-hostip # disable the automatic injection of the Host IP as 'host.k3d.internal' into the containers and CoreDNS (default: false) --no-image-volume # disable the creation of a volume for storing images (used for the 'k3d image import' command) (default: false) --no-lb # disable the creation of a load balancer in front of the server nodes (default: false) --no-rollback # disable the automatic rollback actions, if anything goes wrong (default: false) -p, --port # add some more port mappings (format: '[HOST:][HOSTPORT:]CONTAINERPORT[/PROTOCOL][@NODEFILTER]', use flag multiple times) --registry-create # create a new (docker) registry dedicated for this cluster (default: false) --registry-use # use an existing local (docker) registry with this cluster (string, use multiple times) -s, --servers # specify how many server nodes you want to create (integer, default: 1) --servers-memory # specify memory limit for server containers/nodes (unit, e.g. 1g) --token # specify a cluster token (string, default: auto-generated) --timeout # specify a timeout, after which the cluster creation will be interrupted and changes rolled back (duration, e.g. '10s') -v, --volume # specify additional bind-mounts (format: '[SOURCE:]DEST[@NODEFILTER[;NODEFILTER...]]', use flag multiple times) --wait # enable waiting for all server nodes to be ready before returning (default: true) start CLUSTERNAME # start a (stopped) cluster -a, --all # start all clusters (default: false) --wait # wait for all servers and server-loadbalancer to be up before returning (default: true) --timeout # maximum waiting time for '--wait' before canceling/returning (duration, e.g. '10s') stop CLUSTERNAME # stop a cluster -a, --all # stop all clusters (default: false) delete CLUSTERNAME # delete an existing cluster -a, --all # delete all existing clusters (default: false) list [ CLUSTERNAME [ CLUSTERNAME ... ]] --no-headers # do not print headers (default: false) --token # show column with cluster tokens (default: false) -o, --output # format the output (format: 'json|yaml') completion [ bash | zsh | fish | ( psh | powershell )] # generate completion scripts for common shells config init # write a default k3d config (as a starting point) -f, --force # force overwrite target file (default: false) -o, --output # file to write to (string, default \"k3d-default.yaml\") help [ COMMAND ] # show help text for any command image import [ IMAGE | ARCHIVE [ IMAGE | ARCHIVE ... ]] # Load one or more images from the local runtime environment or tar-archives into k3d clusters -c, --cluster # clusters to load the image into (string, use flag multiple times, default: k3s-default) -k, --keep-tarball # do not delete the image tarball from the shared volume after completion (default: false) kubeconfig get ( CLUSTERNAME [ CLUSTERNAME ... ] | --all ) # get kubeconfig from cluster(s) and write it to stdout -a, --all # get kubeconfigs from all clusters (default: false) merge | write ( CLUSTERNAME [ CLUSTERNAME ... ] | --all ) # get kubeconfig from cluster(s) and merge it/them into a (kubeconfig-)file -a, --all # get kubeconfigs from all clusters (default: false) -s, --kubeconfig-switch-context # switch current-context in kubeconfig to the new context (default: true) -d, --kubeconfig-merge-default # update the default kubeconfig (usually $KUBECONFIG or $HOME/.kube/config) -o, --output # specify the output file where the kubeconfig should be written to (string) --overwrite # [Careful!] forcefully overwrite the output file, ignoring existing contents (default: false) -u, --update # update conflicting fields in existing kubeconfig (default: true) node create NODENAME # Create new nodes (and add them to existing clusters) -c, --cluster # specify the cluster that the node shall connect to (string, default: k3s-default) -i, --image # specify which k3s image should be used for the node(s) (string, default: 'docker.io/rancher/k3s:v1.20.0-k3s2', tag changes per build) --replicas # specify how many replicas you want to create with this spec (integer, default: 1) --role # specify the node role (string, format: 'agent|server', default: agent) --timeout # specify a timeout duration, after which the node creation will be interrupted, if not done yet (duration, e.g. '10s') --wait # wait for the node to be up and running before returning (default: true) start NODENAME # start a (stopped) node stop NODENAME # stop a node delete NODENAME # delete an existing node -a, --all # delete all existing nodes (default: false) -r, --registries # also delete registries, as a special type of node (default: false) list NODENAME --no-headers # do not print headers (default: false) registry create REGISTRYNAME -i, --image # specify image used for the registry (string, default: \"docker.io/library/registry:2\") -p, --port # select host port to map to (format: '[HOST:]HOSTPORT', default: 'random') delete REGISTRYNAME -a, --all # delete all existing registries (default: false) list [ NAME [ NAME... ]] --no-headers # disable table headers (default: false) version # show k3d and k3s version","title":"Command Tree"},{"location":"usage/configfile/","text":"Using Config Files \u00b6 The config file feature is available as of k3d v4.0.0 Introduction \u00b6 Syntax & Semantics The options defined in the config file are not 100% the same as the CLI flags. This concerns naming and style/usage/structure, e.g. --api-port is split up into a field named kubeAPI that has 3 different \u201cchild fields\u201d ( host , hostIP and hostPort ) k3d options are bundled in a scope named options.k3d , where --no-rollback is defined as options.k3d.disableRollback repeatable flags (like --port ) are reflected as YAML lists Usage \u00b6 Using a config file is as easy as putting it in a well-known place in your file system and then referencing it via flag: All options in config file: k3d cluster create --config /home/me/my-awesome-config.yaml (must be .yaml / .yml ) With CLI override (name): k3d cluster create somename --config /home/me/my-awesome-config.yaml With CLI override (extra volume): k3d cluster create --config /home/me/my-awesome-config.yaml --volume '/some/path:/some:path@server:0' Required Fields \u00b6 As of the time of writing this documentation, the config file only requires you to define two fields: apiVersion to match the version of the config file that you want to use (at this time it would be apiVersion: k3d.io/v1alpha3 ) kind to define the kind of config file that you want to use (currently we only have the Simple config) So this would be the minimal config file, which configures absolutely nothing: apiVersion : k3d.io/v1alpha3 kind : Simple Config Options \u00b6 The configuration options for k3d are continuously evolving and so is the config file (syntax) itself. Currently, the config file is still in an Alpha-State, meaning, that it is subject to change anytime (though we try to keep breaking changes low). Validation via JSON-Schema k3d uses a JSON-Schema to describe the expected format and fields of the configuration file. This schema is also used to validate a user-given config file. This JSON-Schema can be found in the specific config version sub-directory in the repository (e.g. here for v1alpha3 ) and could be used to lookup supported fields or by linters to validate the config file, e.g. in your code editor. All Options: Example \u00b6 Since the config options and the config file are changing quite a bit, it\u2019s hard to keep track of all the supported config file settings, so here\u2019s an example showing all of them as of the time of writing: # k3d configuration file, saved as e.g. /home/me/myk3dcluster.yaml apiVersion : k3d.io/v1alpha3 # this will change in the future as we make everything more stable kind : Simple # internally, we also have a Cluster config, which is not yet available externally name : mycluster # name that you want to give to your cluster (will still be prefixed with `k3d-`) servers : 1 # same as `--servers 1` agents : 2 # same as `--agents 2` kubeAPI : # same as `--api-port myhost.my.domain:6445` (where the name would resolve to 127.0.0.1) host : \"myhost.my.domain\" # important for the `server` setting in the kubeconfig hostIP : \"127.0.0.1\" # where the Kubernetes API will be listening on hostPort : \"6445\" # where the Kubernetes API listening port will be mapped to on your host system image : rancher/k3s:v1.20.4-k3s1 # same as `--image rancher/k3s:v1.20.4-k3s1` network : my-custom-net # same as `--network my-custom-net` subnet : \"172.28.0.0/16\" # same as `--subnet 172.28.0.0/16` token : superSecretToken # same as `--token superSecretToken` volumes : # repeatable flags are represented as YAML lists - volume : /my/host/path:/path/in/node # same as `--volume '/my/host/path:/path/in/node@server:0;agent:*'` nodeFilters : - server:0 - agent:* ports : - port : 8080:80 # same as `--port '8080:80@loadbalancer'` nodeFilters : - loadbalancer env : - envVar : bar=baz # same as `--env 'bar=baz@server:0'` nodeFilters : - server:0 registries : # define how registries should be created or used create : # creates a default registry to be used with the cluster; same as `--registry-create registry.localhost` name : registry.localhost host : \"0.0.0.0\" hostPort : \"5000\" use : - k3d-myotherregistry:5000 # some other k3d-managed registry; same as `--registry-use 'k3d-myotherregistry:5000'` config : | # define contents of the `registries.yaml` file (or reference a file); same as `--registry-config /path/to/config.yaml` mirrors: \"my.company.registry\": endpoint: - http://my.company.registry:5000 options : k3d : # k3d runtime settings wait : true # wait for cluster to be usable before returining; same as `--wait` (default: true) timeout : \"60s\" # wait timeout before aborting; same as `--timeout 60s` disableLoadbalancer : false # same as `--no-lb` disableImageVolume : false # same as `--no-image-volume` disableRollback : false # same as `--no-Rollback` loadbalancer : configOverrides : - settings.workerConnections=2048 k3s : # options passed on to K3s itself extraArgs : # additional arguments passed to the `k3s server|agent` command; same as `--k3s-arg` - arg : --tls-san=my.host.domain nodeFilters : - server:* nodeLabels : - label : foo=bar # same as `--k3s-node-label 'foo=bar@agent:1'` -> this results in a Kubernetes node label nodeFilters : - agent:1 kubeconfig : updateDefaultKubeconfig : true # add new cluster to your default Kubeconfig; same as `--kubeconfig-update-default` (default: true) switchCurrentContext : true # also set current-context to the new cluster's context; same as `--kubeconfig-switch-context` (default: true) runtime : # runtime (docker) specific options gpuRequest : all # same as `--gpus all` labels : - label : bar=baz # same as `--runtime-label 'bar=baz@agent:1'` -> this results in a runtime (docker) container label nodeFilters : - agent:1 Config File vs. CLI Flags \u00b6 k3d uses Cobra and Viper for CLI and general config handling respectively. This automatically introduces a \u201cconfig option order of priority\u201d ( precedence order ): Config Precedence Order Source: spf13/viper#why-viper Internal Setting > CLI Flag > Environment Variable > Config File > (k/v store >) Defaults This means, that you can define e.g. a \u201cbase configuration file\u201d with settings that you share across different clusters and override only the fields that differ between those clusters in your CLI flags/arguments. For example, you use the same config file to create three clusters which only have different names and kubeAPI ( --api-port ) settings. References \u00b6 k3d demo repository: https://github.com/iwilltry42/k3d-demo/blob/main/README.md#config-file-support SUSE Blog: https://www.suse.com/c/introduction-k3d-run-k3s-docker-src/ (Search for The \u201cConfiguration as Code\u201d Way )","title":"Using Config Files"},{"location":"usage/configfile/#using-config-files","text":"The config file feature is available as of k3d v4.0.0","title":"Using Config Files"},{"location":"usage/configfile/#introduction","text":"Syntax & Semantics The options defined in the config file are not 100% the same as the CLI flags. This concerns naming and style/usage/structure, e.g. --api-port is split up into a field named kubeAPI that has 3 different \u201cchild fields\u201d ( host , hostIP and hostPort ) k3d options are bundled in a scope named options.k3d , where --no-rollback is defined as options.k3d.disableRollback repeatable flags (like --port ) are reflected as YAML lists","title":"Introduction"},{"location":"usage/configfile/#usage","text":"Using a config file is as easy as putting it in a well-known place in your file system and then referencing it via flag: All options in config file: k3d cluster create --config /home/me/my-awesome-config.yaml (must be .yaml / .yml ) With CLI override (name): k3d cluster create somename --config /home/me/my-awesome-config.yaml With CLI override (extra volume): k3d cluster create --config /home/me/my-awesome-config.yaml --volume '/some/path:/some:path@server:0'","title":"Usage"},{"location":"usage/configfile/#required-fields","text":"As of the time of writing this documentation, the config file only requires you to define two fields: apiVersion to match the version of the config file that you want to use (at this time it would be apiVersion: k3d.io/v1alpha3 ) kind to define the kind of config file that you want to use (currently we only have the Simple config) So this would be the minimal config file, which configures absolutely nothing: apiVersion : k3d.io/v1alpha3 kind : Simple","title":"Required Fields"},{"location":"usage/configfile/#config-options","text":"The configuration options for k3d are continuously evolving and so is the config file (syntax) itself. Currently, the config file is still in an Alpha-State, meaning, that it is subject to change anytime (though we try to keep breaking changes low). Validation via JSON-Schema k3d uses a JSON-Schema to describe the expected format and fields of the configuration file. This schema is also used to validate a user-given config file. This JSON-Schema can be found in the specific config version sub-directory in the repository (e.g. here for v1alpha3 ) and could be used to lookup supported fields or by linters to validate the config file, e.g. in your code editor.","title":"Config Options"},{"location":"usage/configfile/#all-options-example","text":"Since the config options and the config file are changing quite a bit, it\u2019s hard to keep track of all the supported config file settings, so here\u2019s an example showing all of them as of the time of writing: # k3d configuration file, saved as e.g. /home/me/myk3dcluster.yaml apiVersion : k3d.io/v1alpha3 # this will change in the future as we make everything more stable kind : Simple # internally, we also have a Cluster config, which is not yet available externally name : mycluster # name that you want to give to your cluster (will still be prefixed with `k3d-`) servers : 1 # same as `--servers 1` agents : 2 # same as `--agents 2` kubeAPI : # same as `--api-port myhost.my.domain:6445` (where the name would resolve to 127.0.0.1) host : \"myhost.my.domain\" # important for the `server` setting in the kubeconfig hostIP : \"127.0.0.1\" # where the Kubernetes API will be listening on hostPort : \"6445\" # where the Kubernetes API listening port will be mapped to on your host system image : rancher/k3s:v1.20.4-k3s1 # same as `--image rancher/k3s:v1.20.4-k3s1` network : my-custom-net # same as `--network my-custom-net` subnet : \"172.28.0.0/16\" # same as `--subnet 172.28.0.0/16` token : superSecretToken # same as `--token superSecretToken` volumes : # repeatable flags are represented as YAML lists - volume : /my/host/path:/path/in/node # same as `--volume '/my/host/path:/path/in/node@server:0;agent:*'` nodeFilters : - server:0 - agent:* ports : - port : 8080:80 # same as `--port '8080:80@loadbalancer'` nodeFilters : - loadbalancer env : - envVar : bar=baz # same as `--env 'bar=baz@server:0'` nodeFilters : - server:0 registries : # define how registries should be created or used create : # creates a default registry to be used with the cluster; same as `--registry-create registry.localhost` name : registry.localhost host : \"0.0.0.0\" hostPort : \"5000\" use : - k3d-myotherregistry:5000 # some other k3d-managed registry; same as `--registry-use 'k3d-myotherregistry:5000'` config : | # define contents of the `registries.yaml` file (or reference a file); same as `--registry-config /path/to/config.yaml` mirrors: \"my.company.registry\": endpoint: - http://my.company.registry:5000 options : k3d : # k3d runtime settings wait : true # wait for cluster to be usable before returining; same as `--wait` (default: true) timeout : \"60s\" # wait timeout before aborting; same as `--timeout 60s` disableLoadbalancer : false # same as `--no-lb` disableImageVolume : false # same as `--no-image-volume` disableRollback : false # same as `--no-Rollback` loadbalancer : configOverrides : - settings.workerConnections=2048 k3s : # options passed on to K3s itself extraArgs : # additional arguments passed to the `k3s server|agent` command; same as `--k3s-arg` - arg : --tls-san=my.host.domain nodeFilters : - server:* nodeLabels : - label : foo=bar # same as `--k3s-node-label 'foo=bar@agent:1'` -> this results in a Kubernetes node label nodeFilters : - agent:1 kubeconfig : updateDefaultKubeconfig : true # add new cluster to your default Kubeconfig; same as `--kubeconfig-update-default` (default: true) switchCurrentContext : true # also set current-context to the new cluster's context; same as `--kubeconfig-switch-context` (default: true) runtime : # runtime (docker) specific options gpuRequest : all # same as `--gpus all` labels : - label : bar=baz # same as `--runtime-label 'bar=baz@agent:1'` -> this results in a runtime (docker) container label nodeFilters : - agent:1","title":"All Options: Example"},{"location":"usage/configfile/#config-file-vs-cli-flags","text":"k3d uses Cobra and Viper for CLI and general config handling respectively. This automatically introduces a \u201cconfig option order of priority\u201d ( precedence order ): Config Precedence Order Source: spf13/viper#why-viper Internal Setting > CLI Flag > Environment Variable > Config File > (k/v store >) Defaults This means, that you can define e.g. a \u201cbase configuration file\u201d with settings that you share across different clusters and override only the fields that differ between those clusters in your CLI flags/arguments. For example, you use the same config file to create three clusters which only have different names and kubeAPI ( --api-port ) settings.","title":"Config File vs. CLI Flags"},{"location":"usage/configfile/#references","text":"k3d demo repository: https://github.com/iwilltry42/k3d-demo/blob/main/README.md#config-file-support SUSE Blog: https://www.suse.com/c/introduction-k3d-run-k3s-docker-src/ (Search for The \u201cConfiguration as Code\u201d Way )","title":"References"},{"location":"usage/exposing_services/","text":"Exposing Services \u00b6 1. via Ingress (recommended) \u00b6 In this example, we will deploy a simple nginx webserver deployment and make it accessible via ingress. Therefore, we have to create the cluster in a way, that the internal port 80 (where the traefik ingress controller is listening on) is exposed on the host system. Create a cluster, mapping the ingress port 80 to localhost:8081 k3d cluster create --api-port 6550 -p \"8081:80@loadbalancer\" --agents 2 Good to know --api-port 6550 is not required for the example to work. It\u2019s used to have k3s \u2018s API-Server listening on port 6550 with that port mapped to the host system. the port-mapping construct 8081:80@loadbalancer means: \u201cmap port 8081 from the host to port 80 on the container which matches the nodefilter loadbalancer \u201c the loadbalancer nodefilter matches only the serverlb that\u2019s deployed in front of a cluster\u2019s server nodes all ports exposed on the serverlb will be proxied to the same ports on all server nodes in the cluster Get the kubeconfig file (redundant, as k3d cluster create already merges it into your default kubeconfig file) export KUBECONFIG = \" $( k3d kubeconfig write k3s-default ) \" Create a nginx deployment kubectl create deployment nginx --image = nginx Create a ClusterIP service for it kubectl create service clusterip nginx --tcp = 80 :80 Create an ingress object for it by copying the following manifest to a file and applying with kubectl apply -f thatfile.yaml Note : k3s deploys traefik as the default ingress controller # apiVersion: networking.k8s.io/v1beta1 # for k3s < v1.19 apiVersion : networking.k8s.io/v1 kind : Ingress metadata : name : nginx annotations : ingress.kubernetes.io/ssl-redirect : \"false\" spec : rules : - http : paths : - path : / pathType : Prefix backend : service : name : nginx port : number : 80 Curl it via localhost curl localhost:8081/ 2. via NodePort \u00b6 Create a cluster, mapping the port 30080 from agent-0 to localhost:8082 k3d cluster create mycluster -p \"8082:30080@agent:0\" --agents 2 Note 1 : Kubernetes\u2019 default NodePort range is 30000-32767 Note 2 : You may as well expose the whole NodePort range from the very beginning, e.g. via k3d cluster create mycluster --agents 3 -p \"30000-32767:30000-32767@server:0\" (See this video from @portainer ) Warning : Docker creates iptable entries and a new proxy process per port-mapping, so this may take a very long time or even freeze your system! \u2026 (Steps 2 and 3 like above) \u2026 Create a NodePort service for it by copying the following manifest to a file and applying it with kubectl apply -f apiVersion : v1 kind : Service metadata : labels : app : nginx name : nginx spec : ports : - name : 80-80 nodePort : 30080 port : 80 protocol : TCP targetPort : 80 selector : app : nginx type : NodePort Curl it via localhost curl localhost:8082/","title":"Exposing Services"},{"location":"usage/exposing_services/#exposing-services","text":"","title":"Exposing Services"},{"location":"usage/exposing_services/#1-via-ingress-recommended","text":"In this example, we will deploy a simple nginx webserver deployment and make it accessible via ingress. Therefore, we have to create the cluster in a way, that the internal port 80 (where the traefik ingress controller is listening on) is exposed on the host system. Create a cluster, mapping the ingress port 80 to localhost:8081 k3d cluster create --api-port 6550 -p \"8081:80@loadbalancer\" --agents 2 Good to know --api-port 6550 is not required for the example to work. It\u2019s used to have k3s \u2018s API-Server listening on port 6550 with that port mapped to the host system. the port-mapping construct 8081:80@loadbalancer means: \u201cmap port 8081 from the host to port 80 on the container which matches the nodefilter loadbalancer \u201c the loadbalancer nodefilter matches only the serverlb that\u2019s deployed in front of a cluster\u2019s server nodes all ports exposed on the serverlb will be proxied to the same ports on all server nodes in the cluster Get the kubeconfig file (redundant, as k3d cluster create already merges it into your default kubeconfig file) export KUBECONFIG = \" $( k3d kubeconfig write k3s-default ) \" Create a nginx deployment kubectl create deployment nginx --image = nginx Create a ClusterIP service for it kubectl create service clusterip nginx --tcp = 80 :80 Create an ingress object for it by copying the following manifest to a file and applying with kubectl apply -f thatfile.yaml Note : k3s deploys traefik as the default ingress controller # apiVersion: networking.k8s.io/v1beta1 # for k3s < v1.19 apiVersion : networking.k8s.io/v1 kind : Ingress metadata : name : nginx annotations : ingress.kubernetes.io/ssl-redirect : \"false\" spec : rules : - http : paths : - path : / pathType : Prefix backend : service : name : nginx port : number : 80 Curl it via localhost curl localhost:8081/","title":"1. via Ingress (recommended)"},{"location":"usage/exposing_services/#2-via-nodeport","text":"Create a cluster, mapping the port 30080 from agent-0 to localhost:8082 k3d cluster create mycluster -p \"8082:30080@agent:0\" --agents 2 Note 1 : Kubernetes\u2019 default NodePort range is 30000-32767 Note 2 : You may as well expose the whole NodePort range from the very beginning, e.g. via k3d cluster create mycluster --agents 3 -p \"30000-32767:30000-32767@server:0\" (See this video from @portainer ) Warning : Docker creates iptable entries and a new proxy process per port-mapping, so this may take a very long time or even freeze your system! \u2026 (Steps 2 and 3 like above) \u2026 Create a NodePort service for it by copying the following manifest to a file and applying it with kubectl apply -f apiVersion : v1 kind : Service metadata : labels : app : nginx name : nginx spec : ports : - name : 80-80 nodePort : 30080 port : 80 protocol : TCP targetPort : 80 selector : app : nginx type : NodePort Curl it via localhost curl localhost:8082/","title":"2. via NodePort"},{"location":"usage/kubeconfig/","text":"Handling Kubeconfigs \u00b6 By default, k3d will update your default kubeconfig with your new cluster\u2019s details and set the current-context to it (can be disabled). To get a kubeconfig set up for you to connect to a k3d cluster without this automatism, you can go different ways. What is the default kubeconfig? We determine the path of the used or default kubeconfig in two ways: Using the KUBECONFIG environment variable, if it specifies exactly one file Using the default path (e.g. on Linux it\u2019s $HOME /.kube/config ) Getting the kubeconfig for a newly created cluster \u00b6 Create a new kubeconfig file after cluster creation k3d kubeconfig write mycluster Note: this will create (or update) the file $HOME/.k3d/kubeconfig-mycluster.yaml Tip: Use it: export KUBECONFIG = $( k3d kubeconfig write mycluster ) Note 2 : alternatively you can use k3d kubeconfig get mycluster > some-file.yaml Update your default kubeconfig upon cluster creation (DEFAULT) k3d cluster create mycluster --kubeconfig-update-default Note: this won\u2019t switch the current-context (append --kubeconfig-switch-context to do so) Update your default kubeconfig after cluster creation k3d kubeconfig merge mycluster --kubeconfig-merge-default Note: this won\u2019t switch the current-context (append --kubeconfig-switch-context to do so) Update a different kubeconfig after cluster creation k3d kubeconfig merge mycluster --output some/other/file.yaml Note: this won\u2019t switch the current-context The file will be created if it doesn\u2019t exist Switching the current context None of the above options switch the current-context by default. This is intended to be least intrusive, since the current-context has a global effect. You can switch the current-context directly with the kubeconfig merge command by adding the --kubeconfig-switch-context flag. Removing cluster details from the kubeconfig \u00b6 k3d cluster delete mycluster will always remove the details for mycluster from the default kubeconfig. It will also delete the respective kubeconfig file in $HOME/.k3d/ if it exists. Handling multiple clusters \u00b6 k3d kubeconfig merge let\u2019s you specify one or more clusters via arguments or all via --all . All kubeconfigs will then be merged into a single file if --kubeconfig-merge-default or --output is specified. If none of those two flags was specified, a new file will be created per cluster and the merged path (e.g. $HOME/.k3d/kubeconfig-cluster1.yaml:$HOME/.k3d/cluster2.yaml ) will be returned. Note, that with multiple cluster specified, the --kubeconfig-switch-context flag will change the current context to the cluster which was last in the list.","title":"Handling Kubeconfigs"},{"location":"usage/kubeconfig/#handling-kubeconfigs","text":"By default, k3d will update your default kubeconfig with your new cluster\u2019s details and set the current-context to it (can be disabled). To get a kubeconfig set up for you to connect to a k3d cluster without this automatism, you can go different ways. What is the default kubeconfig? We determine the path of the used or default kubeconfig in two ways: Using the KUBECONFIG environment variable, if it specifies exactly one file Using the default path (e.g. on Linux it\u2019s $HOME /.kube/config )","title":"Handling Kubeconfigs"},{"location":"usage/kubeconfig/#getting-the-kubeconfig-for-a-newly-created-cluster","text":"Create a new kubeconfig file after cluster creation k3d kubeconfig write mycluster Note: this will create (or update) the file $HOME/.k3d/kubeconfig-mycluster.yaml Tip: Use it: export KUBECONFIG = $( k3d kubeconfig write mycluster ) Note 2 : alternatively you can use k3d kubeconfig get mycluster > some-file.yaml Update your default kubeconfig upon cluster creation (DEFAULT) k3d cluster create mycluster --kubeconfig-update-default Note: this won\u2019t switch the current-context (append --kubeconfig-switch-context to do so) Update your default kubeconfig after cluster creation k3d kubeconfig merge mycluster --kubeconfig-merge-default Note: this won\u2019t switch the current-context (append --kubeconfig-switch-context to do so) Update a different kubeconfig after cluster creation k3d kubeconfig merge mycluster --output some/other/file.yaml Note: this won\u2019t switch the current-context The file will be created if it doesn\u2019t exist Switching the current context None of the above options switch the current-context by default. This is intended to be least intrusive, since the current-context has a global effect. You can switch the current-context directly with the kubeconfig merge command by adding the --kubeconfig-switch-context flag.","title":"Getting the kubeconfig for a newly created cluster"},{"location":"usage/kubeconfig/#removing-cluster-details-from-the-kubeconfig","text":"k3d cluster delete mycluster will always remove the details for mycluster from the default kubeconfig. It will also delete the respective kubeconfig file in $HOME/.k3d/ if it exists.","title":"Removing cluster details from the kubeconfig"},{"location":"usage/kubeconfig/#handling-multiple-clusters","text":"k3d kubeconfig merge let\u2019s you specify one or more clusters via arguments or all via --all . All kubeconfigs will then be merged into a single file if --kubeconfig-merge-default or --output is specified. If none of those two flags was specified, a new file will be created per cluster and the merged path (e.g. $HOME/.k3d/kubeconfig-cluster1.yaml:$HOME/.k3d/cluster2.yaml ) will be returned. Note, that with multiple cluster specified, the --kubeconfig-switch-context flag will change the current context to the cluster which was last in the list.","title":"Handling multiple clusters"},{"location":"usage/multiserver/","text":"Creating multi-server clusters \u00b6 Important note For the best results (and less unexpected issues), choose 1, 3, 5, \u2026 server nodes. (Read more on etcd quorum on etcd.io ) At least 2 cores and 4GiB of RAM are recommended. Embedded etcd \u00b6 Create a cluster with 3 server nodes using k3s\u2019 embedded etcd database. The first server to be created will use the --cluster-init flag and k3d will wait for it to be up and running before creating (and connecting) the other server nodes. k3d cluster create multiserver --servers 3 Adding server nodes to a running cluster \u00b6 In theory (and also in practice in most cases), this is as easy as executing the following command: k3d node create newserver --cluster multiserver --role server There\u2019s a trap! If your cluster was initially created with only a single server node, then this will fail. That\u2019s because the initial server node was not started with the --cluster-init flag and thus is not using the etcd backend.","title":"Creating multi-server clusters"},{"location":"usage/multiserver/#creating-multi-server-clusters","text":"Important note For the best results (and less unexpected issues), choose 1, 3, 5, \u2026 server nodes. (Read more on etcd quorum on etcd.io ) At least 2 cores and 4GiB of RAM are recommended.","title":"Creating multi-server clusters"},{"location":"usage/multiserver/#embedded-etcd","text":"Create a cluster with 3 server nodes using k3s\u2019 embedded etcd database. The first server to be created will use the --cluster-init flag and k3d will wait for it to be up and running before creating (and connecting) the other server nodes. k3d cluster create multiserver --servers 3","title":"Embedded etcd"},{"location":"usage/multiserver/#adding-server-nodes-to-a-running-cluster","text":"In theory (and also in practice in most cases), this is as easy as executing the following command: k3d node create newserver --cluster multiserver --role server There\u2019s a trap! If your cluster was initially created with only a single server node, then this will fail. That\u2019s because the initial server node was not started with the --cluster-init flag and thus is not using the etcd backend.","title":"Adding server nodes to a running cluster"},{"location":"usage/registries/","text":"Using Image Registries \u00b6 Registries configuration file \u00b6 You can add registries by specifying them in a registries.yaml and referencing it at creation time: k3d cluster create mycluster --registry-config \"/home/YOU/my-registries.yaml\" . This file is a regular k3s registries configuration file , and looks like this: mirrors : \"my.company.registry:5000\" : endpoint : - http://my.company.registry:5000 In this example, an image with a name like my.company.registry:5000/nginx:latest would be pulled from the registry running at http://my.company.registry:5000 . This file can also be used for providing additional information necessary for accessing some registries, like authentication and certificates . Registries Configuration File embedded in k3d\u2019s SimpleConfig \u00b6 If you\u2019re using a SimpleConfig file to configure your k3d cluster, you may as well embed the registries.yaml in there directly: apiVersion : k3d.io/v1alpha3 kind : Simple name : test servers : 1 agents : 2 registries : create : name : myregistry config : | mirrors: \"my.company.registry\": endpoint: - http://my.company.registry:5000 Here, the config for the k3d-managed registry, created by the create: {...} option will be merged with the config specified under config: | . Authenticated registries \u00b6 When using authenticated registries, we can add the username and password in a configs section in the registries.yaml , like this: mirrors : my.company.registry : endpoint : - http://my.company.registry configs : my.company.registry : auth : username : aladin password : abracadabra Secure registries \u00b6 When using secure registries, the registries.yaml file must include information about the certificates. For example, if you want to use images from the secure registry running at https://my.company.registry , you must first download a CA file valid for that server and store it in some well-known directory like ${HOME}/.k3d/my-company-root.pem . Then you have to mount the CA file in some directory in the nodes in the cluster and include that mounted file in a configs section in the registries.yaml file . For example, if we mount the CA file in /etc/ssl/certs/my-company-root.pem , the registries.yaml will look like: mirrors : my.company.registry : endpoint : - https://my.company.registry configs : my.company.registry : tls : # we will mount \"my-company-root.pem\" in the /etc/ssl/certs/ directory. ca_file : \"/etc/ssl/certs/my-company-root.pem\" Finally, we can create the cluster, mounting the CA file in the path we specified in ca_file : k3d cluster create \\ --volume \" ${ HOME } /.k3d/my-registries.yaml:/etc/rancher/k3s/registries.yaml\" \\ --volume \" ${ HOME } /.k3d/my-company-root.pem:/etc/ssl/certs/my-company-root.pem\" Using a local registry \u00b6 Using k3d-managed registries \u00b6 Create a dedicated registry together with your cluster \u00b6 k3d cluster create mycluster --registry-create mycluster-registry : This creates your cluster mycluster together with a registry container called mycluster-registry k3d sets everything up in the cluster for containerd to be able to pull images from that registry (using the registries.yaml file) the port, which the registry is listening on will be mapped to a random port on your host system Check the k3d command output or docker ps -f name = mycluster-registry to find the exposed port (let\u2019s use 12345 here) Pull some image (optional) docker pull alpine:latest , re-tag it to reference your newly created registry docker tag alpine:latest mycluster-registry:12345/testimage:local and push it docker push mycluster-registry:12345/testimage:local Use kubectl to create a new pod in your cluster using that image to see, if the cluster can pull from the new registry: kubectl run --image mycluster-registry:12345/testimage:local testimage --command -- tail -f /dev/null (creates a container that will not do anything but keep on running) Create a customized k3d-managed registry \u00b6 k3d registry create myregistry.localhost --port 12345 creates a new registry called k3d-myregistry.localhost (could be used with automatic resolution of *.localhost , see next section - also, note the k3d- prefix that k3d adds to all resources it creates) k3d cluster create newcluster --registry-use k3d-myregistry.localhost:12345 (make sure you use the k3d- prefix here) creates a new cluster set up to us that registry continue with step 3 and 4 from the last section for testing Using your own (not k3d-managed) local registry \u00b6 We recommend using a k3d-managed registry, as it plays nicely together with k3d clusters, but here\u2019s also a guide to create your own (not k3d-managed) registry, if you need features or customizations, that k3d does not provide: Using your own (not k3d-managed) local registry You can start your own local registry it with some docker commands, like: docker volume create local_registry docker container run -d --name registry.localhost -v local_registry:/var/lib/registry --restart always -p 5000 :5000 registry:2 These commands will start your registry container with name and port registry.localhost:5000 . In order to push to this registry, you will need to make it accessible as described in the next section. Once your registry is up and running, we will need to add it to your registries.yaml configuration file. Finally, you have to connect the registry network to the k3d cluster network: docker network connect k3d-k3s-default registry.localhost . And then you can test your local registry . Pushing to your local registry address \u00b6 As per the guide above, the registry will be available as registry.localhost:5000 . All the nodes in your k3d cluster can resolve this hostname (thanks to the DNS server provided by the Docker daemon) but, in order to be able to push to this registry, this hostname also has to be resolved by your host. nss-myhostname to resolve *.localhost Luckily (for Linux users), NSS-myhostname ships with many Linux distributions and should resolve *.localhost automatically to 127.0.0.1 . Otherwise, it\u2019s installable using sudo apt install libnss-myhostname . If your system does not provide/support tools that can auto-resolve specific names to 127.0.0.1 , you can manually add an entry in your /etc/hosts ( c:\\windows\\system32\\drivers\\etc\\hosts on Windows) file like this: 127 .0.0.1 k3d-registry.localhost Just use localhost Alternatively, if you don\u2019t care about pretty names, just push directly to localhost:5000 (or whatever port you used) and it will work. If you later pull the image from the registry, only the repository path (e.g. myrepo/myimage:mytag in registry.localhost:5000/myrepo/myimage:mytag ) matters to find your image in the targeted registry. Testing your registry \u00b6 You should test that you can push to your registry from your local development machine. use images from that registry in Deployments in your k3d cluster. We will verify these two things for a local registry (located at k3d-registry.localhost:12345 ) running in your development machine. Things would be basically the same for checking an external registry, but some additional configuration could be necessary in your local machine when using an authenticated or secure registry (please refer to Docker\u2019s documentation for this). First, we can download some image (like nginx ) and push it to our local registry with: docker pull nginx:latest docker tag nginx:latest k3d-registry.localhost:5000/nginx:latest docker push k3d-registry.localhost:5000/nginx:latest Then we can deploy a pod referencing this image to your cluster: cat <<EOF | kubectl apply -f - apiVersion: apps/v1 kind: Deployment metadata: name: nginx-test-registry labels: app: nginx-test-registry spec: replicas: 1 selector: matchLabels: app: nginx-test-registry template: metadata: labels: app: nginx-test-registry spec: containers: - name: nginx-test-registry image: k3d-registry.localhost:12345/nginx:latest ports: - containerPort: 80 EOF Then you should check that the pod is running with kubectl get pods -l \"app=nginx-test-registry\" .","title":"Using Image Registries"},{"location":"usage/registries/#using-image-registries","text":"","title":"Using Image Registries"},{"location":"usage/registries/#registries-configuration-file","text":"You can add registries by specifying them in a registries.yaml and referencing it at creation time: k3d cluster create mycluster --registry-config \"/home/YOU/my-registries.yaml\" . This file is a regular k3s registries configuration file , and looks like this: mirrors : \"my.company.registry:5000\" : endpoint : - http://my.company.registry:5000 In this example, an image with a name like my.company.registry:5000/nginx:latest would be pulled from the registry running at http://my.company.registry:5000 . This file can also be used for providing additional information necessary for accessing some registries, like authentication and certificates .","title":"Registries configuration file"},{"location":"usage/registries/#registries-configuration-file-embedded-in-k3ds-simpleconfig","text":"If you\u2019re using a SimpleConfig file to configure your k3d cluster, you may as well embed the registries.yaml in there directly: apiVersion : k3d.io/v1alpha3 kind : Simple name : test servers : 1 agents : 2 registries : create : name : myregistry config : | mirrors: \"my.company.registry\": endpoint: - http://my.company.registry:5000 Here, the config for the k3d-managed registry, created by the create: {...} option will be merged with the config specified under config: | .","title":"Registries Configuration File embedded in k3d's SimpleConfig"},{"location":"usage/registries/#authenticated-registries","text":"When using authenticated registries, we can add the username and password in a configs section in the registries.yaml , like this: mirrors : my.company.registry : endpoint : - http://my.company.registry configs : my.company.registry : auth : username : aladin password : abracadabra","title":"Authenticated registries"},{"location":"usage/registries/#secure-registries","text":"When using secure registries, the registries.yaml file must include information about the certificates. For example, if you want to use images from the secure registry running at https://my.company.registry , you must first download a CA file valid for that server and store it in some well-known directory like ${HOME}/.k3d/my-company-root.pem . Then you have to mount the CA file in some directory in the nodes in the cluster and include that mounted file in a configs section in the registries.yaml file . For example, if we mount the CA file in /etc/ssl/certs/my-company-root.pem , the registries.yaml will look like: mirrors : my.company.registry : endpoint : - https://my.company.registry configs : my.company.registry : tls : # we will mount \"my-company-root.pem\" in the /etc/ssl/certs/ directory. ca_file : \"/etc/ssl/certs/my-company-root.pem\" Finally, we can create the cluster, mounting the CA file in the path we specified in ca_file : k3d cluster create \\ --volume \" ${ HOME } /.k3d/my-registries.yaml:/etc/rancher/k3s/registries.yaml\" \\ --volume \" ${ HOME } /.k3d/my-company-root.pem:/etc/ssl/certs/my-company-root.pem\"","title":"Secure registries"},{"location":"usage/registries/#using-a-local-registry","text":"","title":"Using a local registry"},{"location":"usage/registries/#using-k3d-managed-registries","text":"","title":"Using k3d-managed registries"},{"location":"usage/registries/#create-a-dedicated-registry-together-with-your-cluster","text":"k3d cluster create mycluster --registry-create mycluster-registry : This creates your cluster mycluster together with a registry container called mycluster-registry k3d sets everything up in the cluster for containerd to be able to pull images from that registry (using the registries.yaml file) the port, which the registry is listening on will be mapped to a random port on your host system Check the k3d command output or docker ps -f name = mycluster-registry to find the exposed port (let\u2019s use 12345 here) Pull some image (optional) docker pull alpine:latest , re-tag it to reference your newly created registry docker tag alpine:latest mycluster-registry:12345/testimage:local and push it docker push mycluster-registry:12345/testimage:local Use kubectl to create a new pod in your cluster using that image to see, if the cluster can pull from the new registry: kubectl run --image mycluster-registry:12345/testimage:local testimage --command -- tail -f /dev/null (creates a container that will not do anything but keep on running)","title":"Create a dedicated registry together with your cluster"},{"location":"usage/registries/#create-a-customized-k3d-managed-registry","text":"k3d registry create myregistry.localhost --port 12345 creates a new registry called k3d-myregistry.localhost (could be used with automatic resolution of *.localhost , see next section - also, note the k3d- prefix that k3d adds to all resources it creates) k3d cluster create newcluster --registry-use k3d-myregistry.localhost:12345 (make sure you use the k3d- prefix here) creates a new cluster set up to us that registry continue with step 3 and 4 from the last section for testing","title":"Create a customized k3d-managed registry"},{"location":"usage/registries/#using-your-own-not-k3d-managed-local-registry","text":"We recommend using a k3d-managed registry, as it plays nicely together with k3d clusters, but here\u2019s also a guide to create your own (not k3d-managed) registry, if you need features or customizations, that k3d does not provide: Using your own (not k3d-managed) local registry You can start your own local registry it with some docker commands, like: docker volume create local_registry docker container run -d --name registry.localhost -v local_registry:/var/lib/registry --restart always -p 5000 :5000 registry:2 These commands will start your registry container with name and port registry.localhost:5000 . In order to push to this registry, you will need to make it accessible as described in the next section. Once your registry is up and running, we will need to add it to your registries.yaml configuration file. Finally, you have to connect the registry network to the k3d cluster network: docker network connect k3d-k3s-default registry.localhost . And then you can test your local registry .","title":"Using your own (not k3d-managed) local registry"},{"location":"usage/registries/#pushing-to-your-local-registry-address","text":"As per the guide above, the registry will be available as registry.localhost:5000 . All the nodes in your k3d cluster can resolve this hostname (thanks to the DNS server provided by the Docker daemon) but, in order to be able to push to this registry, this hostname also has to be resolved by your host. nss-myhostname to resolve *.localhost Luckily (for Linux users), NSS-myhostname ships with many Linux distributions and should resolve *.localhost automatically to 127.0.0.1 . Otherwise, it\u2019s installable using sudo apt install libnss-myhostname . If your system does not provide/support tools that can auto-resolve specific names to 127.0.0.1 , you can manually add an entry in your /etc/hosts ( c:\\windows\\system32\\drivers\\etc\\hosts on Windows) file like this: 127 .0.0.1 k3d-registry.localhost Just use localhost Alternatively, if you don\u2019t care about pretty names, just push directly to localhost:5000 (or whatever port you used) and it will work. If you later pull the image from the registry, only the repository path (e.g. myrepo/myimage:mytag in registry.localhost:5000/myrepo/myimage:mytag ) matters to find your image in the targeted registry.","title":"Pushing to your local registry address"},{"location":"usage/registries/#testing-your-registry","text":"You should test that you can push to your registry from your local development machine. use images from that registry in Deployments in your k3d cluster. We will verify these two things for a local registry (located at k3d-registry.localhost:12345 ) running in your development machine. Things would be basically the same for checking an external registry, but some additional configuration could be necessary in your local machine when using an authenticated or secure registry (please refer to Docker\u2019s documentation for this). First, we can download some image (like nginx ) and push it to our local registry with: docker pull nginx:latest docker tag nginx:latest k3d-registry.localhost:5000/nginx:latest docker push k3d-registry.localhost:5000/nginx:latest Then we can deploy a pod referencing this image to your cluster: cat <<EOF | kubectl apply -f - apiVersion: apps/v1 kind: Deployment metadata: name: nginx-test-registry labels: app: nginx-test-registry spec: replicas: 1 selector: matchLabels: app: nginx-test-registry template: metadata: labels: app: nginx-test-registry spec: containers: - name: nginx-test-registry image: k3d-registry.localhost:12345/nginx:latest ports: - containerPort: 80 EOF Then you should check that the pod is running with kubectl get pods -l \"app=nginx-test-registry\" .","title":"Testing your registry"},{"location":"usage/advanced/calico/","text":"Use Calico instead of Flannel \u00b6 If you want to use NetworkPolicy you can use Calico in k3s instead of Flannel. 1. Download and modify the Calico descriptor \u00b6 You can following the documentation And then you have to change the ConfigMap calico-config . On the cni_network_config add the entry for allowing IP forwarding \"container_settings\" : { \"allow_ip_forwarding\" : true } Or you can directly use this calico.yaml manifest 2. Create the cluster without flannel and with calico \u00b6 On the k3s cluster creation : add the flag --flannel-backend=none . For this, on k3d you need to forward this flag to k3s with the option --k3s-server-arg . mount ( --volume ) the calico descriptor in the auto deploy manifest directory of k3s /var/lib/rancher/k3s/server/manifests/ So the command of the cluster creation is (when you are at root of the k3d repository) k3d cluster create \" ${ clustername } \" \\ --k3s-server-arg '--flannel-backend=none' \\ --volume \" $( pwd ) /docs/usage/guides/calico.yaml:/var/lib/rancher/k3s/server/manifests/calico.yaml\" In this example : change \"${clustername}\" with the name of the cluster (or set a variable). $(pwd)/docs/usage/guides/calico.yaml is the absolute path of the calico manifest, you can adapt it. You can add other options, see . The cluster will start without flannel and with Calico as CNI Plugin. For watching for the pod(s) deployment watch \"kubectl get pods -n kube-system\" You will have something like this at beginning (with the command line kubectl get pods -n kube-system ) NAME READY STATUS RESTARTS AGE helm-install-traefik-pn84f 0 /1 Pending 0 3s calico-node-97rx8 0 /1 Init:0/3 0 3s metrics-server-7566d596c8-hwnqq 0 /1 Pending 0 2s calico-kube-controllers-58b656d69f-2z7cn 0 /1 Pending 0 2s local-path-provisioner-6d59f47c7-rmswg 0 /1 Pending 0 2s coredns-8655855d6-cxtnr 0 /1 Pending 0 2s And when it finish to start NAME READY STATUS RESTARTS AGE metrics-server-7566d596c8-hwnqq 1 /1 Running 0 56s calico-node-97rx8 1 /1 Running 0 57s helm-install-traefik-pn84f 0 /1 Completed 1 57s svclb-traefik-lmjr5 2 /2 Running 0 28s calico-kube-controllers-58b656d69f-2z7cn 1 /1 Running 0 56s local-path-provisioner-6d59f47c7-rmswg 1 /1 Running 0 56s traefik-758cd5fc85-x8p57 1 /1 Running 0 28s coredns-8655855d6-cxtnr 1 /1 Running 0 56s Note : you can use the auto deploy manifest or a kubectl apply depending on your needs Calico is not as quick as Flannel (but it provides more features) References \u00b6 https://rancher.com/docs/k3s/latest/en/installation/network-options/ https://docs.projectcalico.org/getting-started/kubernetes/k3s/","title":"Use Calico instead of Flannel"},{"location":"usage/advanced/calico/#use-calico-instead-of-flannel","text":"If you want to use NetworkPolicy you can use Calico in k3s instead of Flannel.","title":"Use Calico instead of Flannel"},{"location":"usage/advanced/calico/#1-download-and-modify-the-calico-descriptor","text":"You can following the documentation And then you have to change the ConfigMap calico-config . On the cni_network_config add the entry for allowing IP forwarding \"container_settings\" : { \"allow_ip_forwarding\" : true } Or you can directly use this calico.yaml manifest","title":"1. Download and modify the Calico descriptor"},{"location":"usage/advanced/calico/#2-create-the-cluster-without-flannel-and-with-calico","text":"On the k3s cluster creation : add the flag --flannel-backend=none . For this, on k3d you need to forward this flag to k3s with the option --k3s-server-arg . mount ( --volume ) the calico descriptor in the auto deploy manifest directory of k3s /var/lib/rancher/k3s/server/manifests/ So the command of the cluster creation is (when you are at root of the k3d repository) k3d cluster create \" ${ clustername } \" \\ --k3s-server-arg '--flannel-backend=none' \\ --volume \" $( pwd ) /docs/usage/guides/calico.yaml:/var/lib/rancher/k3s/server/manifests/calico.yaml\" In this example : change \"${clustername}\" with the name of the cluster (or set a variable). $(pwd)/docs/usage/guides/calico.yaml is the absolute path of the calico manifest, you can adapt it. You can add other options, see . The cluster will start without flannel and with Calico as CNI Plugin. For watching for the pod(s) deployment watch \"kubectl get pods -n kube-system\" You will have something like this at beginning (with the command line kubectl get pods -n kube-system ) NAME READY STATUS RESTARTS AGE helm-install-traefik-pn84f 0 /1 Pending 0 3s calico-node-97rx8 0 /1 Init:0/3 0 3s metrics-server-7566d596c8-hwnqq 0 /1 Pending 0 2s calico-kube-controllers-58b656d69f-2z7cn 0 /1 Pending 0 2s local-path-provisioner-6d59f47c7-rmswg 0 /1 Pending 0 2s coredns-8655855d6-cxtnr 0 /1 Pending 0 2s And when it finish to start NAME READY STATUS RESTARTS AGE metrics-server-7566d596c8-hwnqq 1 /1 Running 0 56s calico-node-97rx8 1 /1 Running 0 57s helm-install-traefik-pn84f 0 /1 Completed 1 57s svclb-traefik-lmjr5 2 /2 Running 0 28s calico-kube-controllers-58b656d69f-2z7cn 1 /1 Running 0 56s local-path-provisioner-6d59f47c7-rmswg 1 /1 Running 0 56s traefik-758cd5fc85-x8p57 1 /1 Running 0 28s coredns-8655855d6-cxtnr 1 /1 Running 0 56s Note : you can use the auto deploy manifest or a kubectl apply depending on your needs Calico is not as quick as Flannel (but it provides more features)","title":"2. Create the cluster without flannel and with calico"},{"location":"usage/advanced/calico/#references","text":"https://rancher.com/docs/k3s/latest/en/installation/network-options/ https://docs.projectcalico.org/getting-started/kubernetes/k3s/","title":"References"},{"location":"usage/advanced/cuda/","text":"Running CUDA workloads \u00b6 If you want to run CUDA workloads on the K3s container you need to customize the container. CUDA workloads require the NVIDIA Container Runtime, so containerd needs to be configured to use this runtime. The K3s container itself also needs to run with this runtime. If you are using Docker you can install the NVIDIA Container Toolkit . Building a customized K3s image \u00b6 To get the NVIDIA container runtime in the K3s image you need to build your own K3s image. The native K3s image is based on Alpine but the NVIDIA container runtime is not supported on Alpine yet. To get around this we need to build the image with a supported base image. Dockerfile \u00b6 Dockerfile : ARG K3S_TAG = \"v1.21.2-k3s1\" FROM rancher/k3s:$K3S_TAG as k3s FROM nvidia/cuda:11.2.0-base-ubuntu18.04 ARG NVIDIA_CONTAINER_RUNTIME_VERSION ENV NVIDIA_CONTAINER_RUNTIME_VERSION = $NVIDIA_CONTAINER_RUNTIME_VERSION RUN echo 'debconf debconf/frontend select Noninteractive' | debconf-set-selections RUN apt-get update && \\ apt-get -y install gnupg2 curl # Install NVIDIA Container Runtime RUN curl -s -L https://nvidia.github.io/nvidia-container-runtime/gpgkey | apt-key add - RUN curl -s -L https://nvidia.github.io/nvidia-container-runtime/ubuntu18.04/nvidia-container-runtime.list | tee /etc/apt/sources.list.d/nvidia-container-runtime.list RUN apt-get update && \\ apt-get -y install nvidia-container-runtime = ${ NVIDIA_CONTAINER_RUNTIME_VERSION } COPY --from = k3s / / RUN mkdir -p /etc && \\ echo 'hosts: files dns' > /etc/nsswitch.conf RUN chmod 1777 /tmp # Provide custom containerd configuration to configure the nvidia-container-runtime RUN mkdir -p /var/lib/rancher/k3s/agent/etc/containerd/ COPY config.toml.tmpl /var/lib/rancher/k3s/agent/etc/containerd/config.toml.tmpl # Deploy the nvidia driver plugin on startup RUN mkdir -p /var/lib/rancher/k3s/server/manifests COPY device-plugin-daemonset.yaml /var/lib/rancher/k3s/server/manifests/nvidia-device-plugin-daemonset.yaml VOLUME /var/lib/kubelet VOLUME /var/lib/rancher/k3s VOLUME /var/lib/cni VOLUME /var/log ENV PATH = \" $PATH :/bin/aux\" ENTRYPOINT [ \"/bin/k3s\" ] CMD [ \"agent\" ] This Dockerfile is based on the K3s Dockerfile The following changes are applied: Change the base images to nvidia/cuda:11.2.0-base-ubuntu18.04 so the NVIDIA Container Runtime can be installed. The version of cuda:xx.x.x must match the one you\u2019re planning to use. Add a custom containerd config.toml template to add the NVIDIA Container Runtime. This replaces the default runc runtime Add a manifest for the NVIDIA driver plugin for Kubernetes Configure containerd \u00b6 We need to configure containerd to use the NVIDIA Container Runtime. We need to customize the config.toml that is used at startup. K3s provides a way to do this using a config.toml.tmpl file. More information can be found on the K3s site . [ plugins . opt ] path = \"{{ .NodeConfig.Containerd.Opt }}\" [ plugins . cri ] stream_server_address = \"127.0.0.1\" stream_server_port = \"10010\" {{ - if . IsRunningInUserNS }} disable_cgroup = true disable_apparmor = true restrict_oom_score_adj = true {{ end }} {{ - if . NodeConfig . AgentConfig . PauseImage }} sandbox_image = \"{{ .NodeConfig.AgentConfig.PauseImage }}\" {{ end }} {{ - if not . NodeConfig . NoFlannel }} [ plugins . cri . cni ] bin_dir = \"{{ .NodeConfig.AgentConfig.CNIBinDir }}\" conf_dir = \"{{ .NodeConfig.AgentConfig.CNIConfDir }}\" {{ end }} [ plugins . cri . containerd . runtimes . runc ] # ---- changed from ' io . containerd . runc . v2 ' for GPU support runtime_type = \"io.containerd.runtime.v1.linux\" # ---- added for GPU support [ plugins . linux ] runtime = \"nvidia-container-runtime\" {{ if . PrivateRegistryConfig }} {{ if . PrivateRegistryConfig . Mirrors }} [ plugins . cri . registry . mirrors ]{{ end }} {{ range $ k , $ v := . PrivateRegistryConfig . Mirrors }} [ plugins . cri . registry . mirrors . \"{{$k}}\" ] endpoint = [{{ range $ i , $ j := $ v . Endpoints }}{{ if $ i }}, {{ end }}{{ printf \"%q\" .}}{{ end }}] {{ end }} {{ range $ k , $ v := . PrivateRegistryConfig . Configs }} {{ if $ v . Auth }} [ plugins . cri . registry . configs . \"{{$k}}\" . auth ] {{ if $ v . Auth . Username }} username = \"{{ $v.Auth.Username }}\" {{ end }} {{ if $ v . Auth . Password }} password = \"{{ $v.Auth.Password }}\" {{ end }} {{ if $ v . Auth . Auth }} auth = \"{{ $v.Auth.Auth }}\" {{ end }} {{ if $ v . Auth . IdentityToken }} identitytoken = \"{{ $v.Auth.IdentityToken }}\" {{ end }} {{ end }} {{ if $ v . TLS }} [ plugins . cri . registry . configs . \"{{$k}}\" . tls ] {{ if $ v . TLS . CAFile }} ca_file = \"{{ $v.TLS.CAFile }}\" {{ end }} {{ if $ v . TLS . CertFile }} cert_file = \"{{ $v.TLS.CertFile }}\" {{ end }} {{ if $ v . TLS . KeyFile }} key_file = \"{{ $v.TLS.KeyFile }}\" {{ end }} {{ end }} {{ end }} {{ end }} The NVIDIA device plugin \u00b6 To enable NVIDIA GPU support on Kubernetes you also need to install the NVIDIA device plugin . The device plugin is a deamonset and allows you to automatically: Expose the number of GPUs on each nodes of your cluster Keep track of the health of your GPUs Run GPU enabled containers in your Kubernetes cluster. apiVersion : apps/v1 kind : DaemonSet metadata : name : nvidia-device-plugin-daemonset namespace : kube-system spec : selector : matchLabels : name : nvidia-device-plugin-ds template : metadata : # Mark this pod as a critical add-on; when enabled, the critical add-on scheduler # reserves resources for critical add-on pods so that they can be rescheduled after # a failure. This annotation works in tandem with the toleration below. annotations : scheduler.alpha.kubernetes.io/critical-pod : \"\" labels : name : nvidia-device-plugin-ds spec : tolerations : # Allow this pod to be rescheduled while the node is in \"critical add-ons only\" mode. # This, along with the annotation above marks this pod as a critical add-on. - key : CriticalAddonsOnly operator : Exists containers : - env : - name : DP_DISABLE_HEALTHCHECKS value : xids image : nvidia/k8s-device-plugin:1.11 name : nvidia-device-plugin-ctr securityContext : allowPrivilegeEscalation : true capabilities : drop : [ \"ALL\" ] volumeMounts : - name : device-plugin mountPath : /var/lib/kubelet/device-plugins volumes : - name : device-plugin hostPath : path : /var/lib/kubelet/device-plugins Build the K3s image \u00b6 To build the custom image we need to build K3s because we need the generated output. Put the following files in a directory: Dockerfile config.toml.tmpl device-plugin-daemonset.yaml build.sh cuda-vector-add.yaml The build.sh script is configured using exports & defaults to v1.21.2+k3s1 . Please set at least the IMAGE_REGISTRY variable! The script performs the following steps builds the custom K3s image including the nvidia drivers. build.sh : #!/bin/bash set -euxo pipefail K3S_TAG = ${ K3S_TAG := \"v1.21.2-k3s1\" } # replace + with -, if needed IMAGE_REGISTRY = ${ IMAGE_REGISTRY := \"MY_REGISTRY\" } IMAGE_REPOSITORY = ${ IMAGE_REPOSITORY := \"rancher/k3s\" } IMAGE_TAG = \" $K3S_TAG -cuda\" IMAGE = ${ IMAGE := \" $IMAGE_REGISTRY / $IMAGE_REPOSITORY : $IMAGE_TAG \" } NVIDIA_CONTAINER_RUNTIME_VERSION = ${ NVIDIA_CONTAINER_RUNTIME_VERSION := \"3.5.0-1\" } echo \"IMAGE= $IMAGE \" # due to some unknown reason, copying symlinks fails with buildkit enabled DOCKER_BUILDKIT = 0 docker build \\ --build-arg K3S_TAG = $K3S_TAG \\ --build-arg NVIDIA_CONTAINER_RUNTIME_VERSION = $NVIDIA_CONTAINER_RUNTIME_VERSION \\ -t $IMAGE . docker push $IMAGE echo \"Done!\" Run and test the custom image with k3d \u00b6 You can use the image with k3d: k3d cluster create gputest --image = $IMAGE --gpus = 1 Deploy a test pod : kubectl apply -f cuda-vector-add.yaml kubectl logs cuda-vector-add This should output something like the following: $ kubectl logs cuda-vector-add [ Vector addition of 50000 elements ] Copy input data from the host memory to the CUDA device CUDA kernel launch with 196 blocks of 256 threads Copy output data from the CUDA device to the host memory Test PASSED Done If the cuda-vector-add pod is stuck in Pending state, probably the device-driver daemonset didn\u2019t get deployed correctly from the auto-deploy manifests. In that case, you can apply it manually via kubectl apply -f device-plugin-daemonset.yaml . Known issues \u00b6 This approach does not work on WSL2 yet. The NVIDIA driver plugin and container runtime rely on the NVIDIA Management Library (NVML) which is not yet supported. See the CUDA on WSL User Guide . Acknowledgements \u00b6 Most of the information in this article was obtained from various sources: Add NVIDIA GPU support to k3s with containerd microk8s K3s k3s-gpu Authors \u00b6 @markrexwinkel @vainkop @iwilltry42","title":"Running CUDA workloads"},{"location":"usage/advanced/cuda/#running-cuda-workloads","text":"If you want to run CUDA workloads on the K3s container you need to customize the container. CUDA workloads require the NVIDIA Container Runtime, so containerd needs to be configured to use this runtime. The K3s container itself also needs to run with this runtime. If you are using Docker you can install the NVIDIA Container Toolkit .","title":"Running CUDA workloads"},{"location":"usage/advanced/cuda/#building-a-customized-k3s-image","text":"To get the NVIDIA container runtime in the K3s image you need to build your own K3s image. The native K3s image is based on Alpine but the NVIDIA container runtime is not supported on Alpine yet. To get around this we need to build the image with a supported base image.","title":"Building a customized K3s image"},{"location":"usage/advanced/cuda/#dockerfile","text":"Dockerfile : ARG K3S_TAG = \"v1.21.2-k3s1\" FROM rancher/k3s:$K3S_TAG as k3s FROM nvidia/cuda:11.2.0-base-ubuntu18.04 ARG NVIDIA_CONTAINER_RUNTIME_VERSION ENV NVIDIA_CONTAINER_RUNTIME_VERSION = $NVIDIA_CONTAINER_RUNTIME_VERSION RUN echo 'debconf debconf/frontend select Noninteractive' | debconf-set-selections RUN apt-get update && \\ apt-get -y install gnupg2 curl # Install NVIDIA Container Runtime RUN curl -s -L https://nvidia.github.io/nvidia-container-runtime/gpgkey | apt-key add - RUN curl -s -L https://nvidia.github.io/nvidia-container-runtime/ubuntu18.04/nvidia-container-runtime.list | tee /etc/apt/sources.list.d/nvidia-container-runtime.list RUN apt-get update && \\ apt-get -y install nvidia-container-runtime = ${ NVIDIA_CONTAINER_RUNTIME_VERSION } COPY --from = k3s / / RUN mkdir -p /etc && \\ echo 'hosts: files dns' > /etc/nsswitch.conf RUN chmod 1777 /tmp # Provide custom containerd configuration to configure the nvidia-container-runtime RUN mkdir -p /var/lib/rancher/k3s/agent/etc/containerd/ COPY config.toml.tmpl /var/lib/rancher/k3s/agent/etc/containerd/config.toml.tmpl # Deploy the nvidia driver plugin on startup RUN mkdir -p /var/lib/rancher/k3s/server/manifests COPY device-plugin-daemonset.yaml /var/lib/rancher/k3s/server/manifests/nvidia-device-plugin-daemonset.yaml VOLUME /var/lib/kubelet VOLUME /var/lib/rancher/k3s VOLUME /var/lib/cni VOLUME /var/log ENV PATH = \" $PATH :/bin/aux\" ENTRYPOINT [ \"/bin/k3s\" ] CMD [ \"agent\" ] This Dockerfile is based on the K3s Dockerfile The following changes are applied: Change the base images to nvidia/cuda:11.2.0-base-ubuntu18.04 so the NVIDIA Container Runtime can be installed. The version of cuda:xx.x.x must match the one you\u2019re planning to use. Add a custom containerd config.toml template to add the NVIDIA Container Runtime. This replaces the default runc runtime Add a manifest for the NVIDIA driver plugin for Kubernetes","title":"Dockerfile"},{"location":"usage/advanced/cuda/#configure-containerd","text":"We need to configure containerd to use the NVIDIA Container Runtime. We need to customize the config.toml that is used at startup. K3s provides a way to do this using a config.toml.tmpl file. More information can be found on the K3s site . [ plugins . opt ] path = \"{{ .NodeConfig.Containerd.Opt }}\" [ plugins . cri ] stream_server_address = \"127.0.0.1\" stream_server_port = \"10010\" {{ - if . IsRunningInUserNS }} disable_cgroup = true disable_apparmor = true restrict_oom_score_adj = true {{ end }} {{ - if . NodeConfig . AgentConfig . PauseImage }} sandbox_image = \"{{ .NodeConfig.AgentConfig.PauseImage }}\" {{ end }} {{ - if not . NodeConfig . NoFlannel }} [ plugins . cri . cni ] bin_dir = \"{{ .NodeConfig.AgentConfig.CNIBinDir }}\" conf_dir = \"{{ .NodeConfig.AgentConfig.CNIConfDir }}\" {{ end }} [ plugins . cri . containerd . runtimes . runc ] # ---- changed from ' io . containerd . runc . v2 ' for GPU support runtime_type = \"io.containerd.runtime.v1.linux\" # ---- added for GPU support [ plugins . linux ] runtime = \"nvidia-container-runtime\" {{ if . PrivateRegistryConfig }} {{ if . PrivateRegistryConfig . Mirrors }} [ plugins . cri . registry . mirrors ]{{ end }} {{ range $ k , $ v := . PrivateRegistryConfig . Mirrors }} [ plugins . cri . registry . mirrors . \"{{$k}}\" ] endpoint = [{{ range $ i , $ j := $ v . Endpoints }}{{ if $ i }}, {{ end }}{{ printf \"%q\" .}}{{ end }}] {{ end }} {{ range $ k , $ v := . PrivateRegistryConfig . Configs }} {{ if $ v . Auth }} [ plugins . cri . registry . configs . \"{{$k}}\" . auth ] {{ if $ v . Auth . Username }} username = \"{{ $v.Auth.Username }}\" {{ end }} {{ if $ v . Auth . Password }} password = \"{{ $v.Auth.Password }}\" {{ end }} {{ if $ v . Auth . Auth }} auth = \"{{ $v.Auth.Auth }}\" {{ end }} {{ if $ v . Auth . IdentityToken }} identitytoken = \"{{ $v.Auth.IdentityToken }}\" {{ end }} {{ end }} {{ if $ v . TLS }} [ plugins . cri . registry . configs . \"{{$k}}\" . tls ] {{ if $ v . TLS . CAFile }} ca_file = \"{{ $v.TLS.CAFile }}\" {{ end }} {{ if $ v . TLS . CertFile }} cert_file = \"{{ $v.TLS.CertFile }}\" {{ end }} {{ if $ v . TLS . KeyFile }} key_file = \"{{ $v.TLS.KeyFile }}\" {{ end }} {{ end }} {{ end }} {{ end }}","title":"Configure containerd"},{"location":"usage/advanced/cuda/#the-nvidia-device-plugin","text":"To enable NVIDIA GPU support on Kubernetes you also need to install the NVIDIA device plugin . The device plugin is a deamonset and allows you to automatically: Expose the number of GPUs on each nodes of your cluster Keep track of the health of your GPUs Run GPU enabled containers in your Kubernetes cluster. apiVersion : apps/v1 kind : DaemonSet metadata : name : nvidia-device-plugin-daemonset namespace : kube-system spec : selector : matchLabels : name : nvidia-device-plugin-ds template : metadata : # Mark this pod as a critical add-on; when enabled, the critical add-on scheduler # reserves resources for critical add-on pods so that they can be rescheduled after # a failure. This annotation works in tandem with the toleration below. annotations : scheduler.alpha.kubernetes.io/critical-pod : \"\" labels : name : nvidia-device-plugin-ds spec : tolerations : # Allow this pod to be rescheduled while the node is in \"critical add-ons only\" mode. # This, along with the annotation above marks this pod as a critical add-on. - key : CriticalAddonsOnly operator : Exists containers : - env : - name : DP_DISABLE_HEALTHCHECKS value : xids image : nvidia/k8s-device-plugin:1.11 name : nvidia-device-plugin-ctr securityContext : allowPrivilegeEscalation : true capabilities : drop : [ \"ALL\" ] volumeMounts : - name : device-plugin mountPath : /var/lib/kubelet/device-plugins volumes : - name : device-plugin hostPath : path : /var/lib/kubelet/device-plugins","title":"The NVIDIA device plugin"},{"location":"usage/advanced/cuda/#build-the-k3s-image","text":"To build the custom image we need to build K3s because we need the generated output. Put the following files in a directory: Dockerfile config.toml.tmpl device-plugin-daemonset.yaml build.sh cuda-vector-add.yaml The build.sh script is configured using exports & defaults to v1.21.2+k3s1 . Please set at least the IMAGE_REGISTRY variable! The script performs the following steps builds the custom K3s image including the nvidia drivers. build.sh : #!/bin/bash set -euxo pipefail K3S_TAG = ${ K3S_TAG := \"v1.21.2-k3s1\" } # replace + with -, if needed IMAGE_REGISTRY = ${ IMAGE_REGISTRY := \"MY_REGISTRY\" } IMAGE_REPOSITORY = ${ IMAGE_REPOSITORY := \"rancher/k3s\" } IMAGE_TAG = \" $K3S_TAG -cuda\" IMAGE = ${ IMAGE := \" $IMAGE_REGISTRY / $IMAGE_REPOSITORY : $IMAGE_TAG \" } NVIDIA_CONTAINER_RUNTIME_VERSION = ${ NVIDIA_CONTAINER_RUNTIME_VERSION := \"3.5.0-1\" } echo \"IMAGE= $IMAGE \" # due to some unknown reason, copying symlinks fails with buildkit enabled DOCKER_BUILDKIT = 0 docker build \\ --build-arg K3S_TAG = $K3S_TAG \\ --build-arg NVIDIA_CONTAINER_RUNTIME_VERSION = $NVIDIA_CONTAINER_RUNTIME_VERSION \\ -t $IMAGE . docker push $IMAGE echo \"Done!\"","title":"Build the K3s image"},{"location":"usage/advanced/cuda/#run-and-test-the-custom-image-with-k3d","text":"You can use the image with k3d: k3d cluster create gputest --image = $IMAGE --gpus = 1 Deploy a test pod : kubectl apply -f cuda-vector-add.yaml kubectl logs cuda-vector-add This should output something like the following: $ kubectl logs cuda-vector-add [ Vector addition of 50000 elements ] Copy input data from the host memory to the CUDA device CUDA kernel launch with 196 blocks of 256 threads Copy output data from the CUDA device to the host memory Test PASSED Done If the cuda-vector-add pod is stuck in Pending state, probably the device-driver daemonset didn\u2019t get deployed correctly from the auto-deploy manifests. In that case, you can apply it manually via kubectl apply -f device-plugin-daemonset.yaml .","title":"Run and test the custom image with k3d"},{"location":"usage/advanced/cuda/#known-issues","text":"This approach does not work on WSL2 yet. The NVIDIA driver plugin and container runtime rely on the NVIDIA Management Library (NVML) which is not yet supported. See the CUDA on WSL User Guide .","title":"Known issues"},{"location":"usage/advanced/cuda/#acknowledgements","text":"Most of the information in this article was obtained from various sources: Add NVIDIA GPU support to k3s with containerd microk8s K3s k3s-gpu","title":"Acknowledgements"},{"location":"usage/advanced/cuda/#authors","text":"@markrexwinkel @vainkop @iwilltry42","title":"Authors"},{"location":"usage/commands/k3d/","text":"k3d \u00b6 https://k3d.io/ -> Run k3s in Docker! Synopsis \u00b6 https://k3d.io/ k3d is a wrapper CLI that helps you to easily create k3s clusters inside docker. Nodes of a k3d cluster are docker containers running a k3s image. All Nodes of a k3d cluster are part of the same docker network. k3d [flags] Options \u00b6 -h, --help help for k3d --timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging) --version Show k3d and default k3s version SEE ALSO \u00b6 k3d cluster - Manage cluster(s) k3d completion - Generate completion scripts for [bash, zsh, fish, powershell | psh] k3d config - Work with config file(s) k3d image - Handle container images. k3d kubeconfig - Manage kubeconfig(s) k3d node - Manage node(s) k3d registry - Manage registry/registries k3d version - Show k3d and default k3s version","title":"K3d"},{"location":"usage/commands/k3d/#k3d","text":"https://k3d.io/ -> Run k3s in Docker!","title":"k3d"},{"location":"usage/commands/k3d/#synopsis","text":"https://k3d.io/ k3d is a wrapper CLI that helps you to easily create k3s clusters inside docker. Nodes of a k3d cluster are docker containers running a k3s image. All Nodes of a k3d cluster are part of the same docker network. k3d [flags]","title":"Synopsis"},{"location":"usage/commands/k3d/#options","text":"-h, --help help for k3d --timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging) --version Show k3d and default k3s version","title":"Options"},{"location":"usage/commands/k3d/#see-also","text":"k3d cluster - Manage cluster(s) k3d completion - Generate completion scripts for [bash, zsh, fish, powershell | psh] k3d config - Work with config file(s) k3d image - Handle container images. k3d kubeconfig - Manage kubeconfig(s) k3d node - Manage node(s) k3d registry - Manage registry/registries k3d version - Show k3d and default k3s version","title":"SEE ALSO"},{"location":"usage/commands/k3d_cluster/","text":"k3d cluster \u00b6 Manage cluster(s) Synopsis \u00b6 Manage cluster(s) k3d cluster [flags] Options \u00b6 -h, --help help for cluster Options inherited from parent commands \u00b6 --timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging) SEE ALSO \u00b6 k3d - https://k3d.io/ -> Run k3s in Docker! k3d cluster create - Create a new cluster k3d cluster delete - Delete cluster(s). k3d cluster edit - [EXPERIMENTAL] Edit cluster(s). k3d cluster list - List cluster(s) k3d cluster start - Start existing k3d cluster(s) k3d cluster stop - Stop existing k3d cluster(s)","title":"K3d cluster"},{"location":"usage/commands/k3d_cluster/#k3d-cluster","text":"Manage cluster(s)","title":"k3d cluster"},{"location":"usage/commands/k3d_cluster/#synopsis","text":"Manage cluster(s) k3d cluster [flags]","title":"Synopsis"},{"location":"usage/commands/k3d_cluster/#options","text":"-h, --help help for cluster","title":"Options"},{"location":"usage/commands/k3d_cluster/#options-inherited-from-parent-commands","text":"--timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging)","title":"Options inherited from parent commands"},{"location":"usage/commands/k3d_cluster/#see-also","text":"k3d - https://k3d.io/ -> Run k3s in Docker! k3d cluster create - Create a new cluster k3d cluster delete - Delete cluster(s). k3d cluster edit - [EXPERIMENTAL] Edit cluster(s). k3d cluster list - List cluster(s) k3d cluster start - Start existing k3d cluster(s) k3d cluster stop - Stop existing k3d cluster(s)","title":"SEE ALSO"},{"location":"usage/commands/k3d_cluster_create/","text":"k3d cluster create \u00b6 Create a new cluster Synopsis \u00b6 Create a new k3s cluster with containerized nodes (k3s in docker). Every cluster will consist of one or more containers: - 1 (or more) server node container (k3s) - (optionally) 1 loadbalancer container as the entrypoint to the cluster (nginx) - (optionally) 1 (or more) agent node containers (k3s) k3d cluster create NAME [flags] Options \u00b6 -a, --agents int Specify how many agents you want to create --agents-memory string Memory limit imposed on the agents nodes [From docker] --api-port [HOST:]HOSTPORT Specify the Kubernetes API server port exposed on the LoadBalancer (Format: [HOST:]HOSTPORT) - Example: `k3d cluster create --servers 3 --api-port 0.0.0.0:6550` -c, --config string Path of a config file to use -e, --env KEY[=VALUE][@NODEFILTER[;NODEFILTER...]] Add environment variables to nodes (Format: KEY[=VALUE][@NODEFILTER[;NODEFILTER...]] - Example: `k3d cluster create --agents 2 -e \"HTTP_PROXY=my.proxy.com@server:0\" -e \"SOME_KEY=SOME_VAL@server:0\"` --gpus string GPU devices to add to the cluster node containers ('all' to pass all GPUs) [From docker] -h, --help help for create -i, --image string Specify k3s image that you want to use for the nodes --k3s-arg ARG@NODEFILTER[;@NODEFILTER] Additional args passed to k3s command (Format: ARG@NODEFILTER[;@NODEFILTER]) - Example: `k3d cluster create --k3s-arg \"--disable=traefik@server:0\" --k3s-node-label KEY[=VALUE][@NODEFILTER[;NODEFILTER...]] Add label to k3s node (Format: KEY[=VALUE][@NODEFILTER[;NODEFILTER...]] - Example: `k3d cluster create --agents 2 --k3s-node-label \"my.label@agent:0,1\" --k3s-node-label \"other.label=somevalue@server:0\"` --kubeconfig-switch-context Directly switch the default kubeconfig's current-context to the new cluster's context (requires --kubeconfig-update-default) (default true) --kubeconfig-update-default Directly update the default kubeconfig with the new cluster's context (default true) --lb-config-override strings Use dotted YAML path syntax to override nginx loadbalancer settings --network string Join an existing network --no-image-volume Disable the creation of a volume for importing images --no-lb Disable the creation of a LoadBalancer in front of the server nodes --no-rollback Disable the automatic rollback actions, if anything goes wrong -p, --port [HOST:][HOSTPORT:]CONTAINERPORT[/PROTOCOL][@NODEFILTER] Map ports from the node containers (via the serverlb) to the host (Format: [HOST:][HOSTPORT:]CONTAINERPORT[/PROTOCOL][@NODEFILTER]) - Example: `k3d cluster create --agents 2 -p 8080:80@agent:0 -p 8081@agent:1` --registry-config string Specify path to an extra registries.yaml file --registry-create NAME[:HOST][:HOSTPORT] Create a k3d-managed registry and connect it to the cluster (Format: NAME[:HOST][:HOSTPORT] - Example: `k3d cluster create --registry-create mycluster-registry:0.0.0.0:5432` --registry-use stringArray Connect to one or more k3d-managed registries running locally --runtime-label KEY[=VALUE][@NODEFILTER[;NODEFILTER...]] Add label to container runtime (Format: KEY[=VALUE][@NODEFILTER[;NODEFILTER...]] - Example: `k3d cluster create --agents 2 --runtime-label \"my.label@agent:0,1\" --runtime-label \"other.label=somevalue@server:0\"` -s, --servers int Specify how many servers you want to create --servers-memory string Memory limit imposed on the server nodes [From docker] --subnet 172.28.0.0/16 [Experimental: IPAM] Define a subnet for the newly created container network (Example: 172.28.0.0/16) --timeout duration Rollback changes if cluster couldn't be created in specified duration. --token string Specify a cluster token. By default, we generate one. -v, --volume [SOURCE:]DEST[@NODEFILTER[;NODEFILTER...]] Mount volumes into the nodes (Format: [SOURCE:]DEST[@NODEFILTER[;NODEFILTER...]] - Example: `k3d cluster create --agents 2 -v /my/path@agent:0,1 -v /tmp/test:/tmp/other@server:0` --wait Wait for the server(s) to be ready before returning. Use '--timeout DURATION' to not wait forever. (default true) Options inherited from parent commands \u00b6 --timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging) SEE ALSO \u00b6 k3d cluster - Manage cluster(s)","title":"K3d cluster create"},{"location":"usage/commands/k3d_cluster_create/#k3d-cluster-create","text":"Create a new cluster","title":"k3d cluster create"},{"location":"usage/commands/k3d_cluster_create/#synopsis","text":"Create a new k3s cluster with containerized nodes (k3s in docker). Every cluster will consist of one or more containers: - 1 (or more) server node container (k3s) - (optionally) 1 loadbalancer container as the entrypoint to the cluster (nginx) - (optionally) 1 (or more) agent node containers (k3s) k3d cluster create NAME [flags]","title":"Synopsis"},{"location":"usage/commands/k3d_cluster_create/#options","text":"-a, --agents int Specify how many agents you want to create --agents-memory string Memory limit imposed on the agents nodes [From docker] --api-port [HOST:]HOSTPORT Specify the Kubernetes API server port exposed on the LoadBalancer (Format: [HOST:]HOSTPORT) - Example: `k3d cluster create --servers 3 --api-port 0.0.0.0:6550` -c, --config string Path of a config file to use -e, --env KEY[=VALUE][@NODEFILTER[;NODEFILTER...]] Add environment variables to nodes (Format: KEY[=VALUE][@NODEFILTER[;NODEFILTER...]] - Example: `k3d cluster create --agents 2 -e \"HTTP_PROXY=my.proxy.com@server:0\" -e \"SOME_KEY=SOME_VAL@server:0\"` --gpus string GPU devices to add to the cluster node containers ('all' to pass all GPUs) [From docker] -h, --help help for create -i, --image string Specify k3s image that you want to use for the nodes --k3s-arg ARG@NODEFILTER[;@NODEFILTER] Additional args passed to k3s command (Format: ARG@NODEFILTER[;@NODEFILTER]) - Example: `k3d cluster create --k3s-arg \"--disable=traefik@server:0\" --k3s-node-label KEY[=VALUE][@NODEFILTER[;NODEFILTER...]] Add label to k3s node (Format: KEY[=VALUE][@NODEFILTER[;NODEFILTER...]] - Example: `k3d cluster create --agents 2 --k3s-node-label \"my.label@agent:0,1\" --k3s-node-label \"other.label=somevalue@server:0\"` --kubeconfig-switch-context Directly switch the default kubeconfig's current-context to the new cluster's context (requires --kubeconfig-update-default) (default true) --kubeconfig-update-default Directly update the default kubeconfig with the new cluster's context (default true) --lb-config-override strings Use dotted YAML path syntax to override nginx loadbalancer settings --network string Join an existing network --no-image-volume Disable the creation of a volume for importing images --no-lb Disable the creation of a LoadBalancer in front of the server nodes --no-rollback Disable the automatic rollback actions, if anything goes wrong -p, --port [HOST:][HOSTPORT:]CONTAINERPORT[/PROTOCOL][@NODEFILTER] Map ports from the node containers (via the serverlb) to the host (Format: [HOST:][HOSTPORT:]CONTAINERPORT[/PROTOCOL][@NODEFILTER]) - Example: `k3d cluster create --agents 2 -p 8080:80@agent:0 -p 8081@agent:1` --registry-config string Specify path to an extra registries.yaml file --registry-create NAME[:HOST][:HOSTPORT] Create a k3d-managed registry and connect it to the cluster (Format: NAME[:HOST][:HOSTPORT] - Example: `k3d cluster create --registry-create mycluster-registry:0.0.0.0:5432` --registry-use stringArray Connect to one or more k3d-managed registries running locally --runtime-label KEY[=VALUE][@NODEFILTER[;NODEFILTER...]] Add label to container runtime (Format: KEY[=VALUE][@NODEFILTER[;NODEFILTER...]] - Example: `k3d cluster create --agents 2 --runtime-label \"my.label@agent:0,1\" --runtime-label \"other.label=somevalue@server:0\"` -s, --servers int Specify how many servers you want to create --servers-memory string Memory limit imposed on the server nodes [From docker] --subnet 172.28.0.0/16 [Experimental: IPAM] Define a subnet for the newly created container network (Example: 172.28.0.0/16) --timeout duration Rollback changes if cluster couldn't be created in specified duration. --token string Specify a cluster token. By default, we generate one. -v, --volume [SOURCE:]DEST[@NODEFILTER[;NODEFILTER...]] Mount volumes into the nodes (Format: [SOURCE:]DEST[@NODEFILTER[;NODEFILTER...]] - Example: `k3d cluster create --agents 2 -v /my/path@agent:0,1 -v /tmp/test:/tmp/other@server:0` --wait Wait for the server(s) to be ready before returning. Use '--timeout DURATION' to not wait forever. (default true)","title":"Options"},{"location":"usage/commands/k3d_cluster_create/#options-inherited-from-parent-commands","text":"--timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging)","title":"Options inherited from parent commands"},{"location":"usage/commands/k3d_cluster_create/#see-also","text":"k3d cluster - Manage cluster(s)","title":"SEE ALSO"},{"location":"usage/commands/k3d_cluster_delete/","text":"k3d cluster delete \u00b6 Delete cluster(s). Synopsis \u00b6 Delete cluster(s). k3d cluster delete [NAME [NAME ...] | --all] [flags] Options \u00b6 -a, --all Delete all existing clusters -c, --config string Path of a config file to use -h, --help help for delete Options inherited from parent commands \u00b6 --timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging) SEE ALSO \u00b6 k3d cluster - Manage cluster(s)","title":"K3d cluster delete"},{"location":"usage/commands/k3d_cluster_delete/#k3d-cluster-delete","text":"Delete cluster(s).","title":"k3d cluster delete"},{"location":"usage/commands/k3d_cluster_delete/#synopsis","text":"Delete cluster(s). k3d cluster delete [NAME [NAME ...] | --all] [flags]","title":"Synopsis"},{"location":"usage/commands/k3d_cluster_delete/#options","text":"-a, --all Delete all existing clusters -c, --config string Path of a config file to use -h, --help help for delete","title":"Options"},{"location":"usage/commands/k3d_cluster_delete/#options-inherited-from-parent-commands","text":"--timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging)","title":"Options inherited from parent commands"},{"location":"usage/commands/k3d_cluster_delete/#see-also","text":"k3d cluster - Manage cluster(s)","title":"SEE ALSO"},{"location":"usage/commands/k3d_cluster_edit/","text":"k3d cluster edit \u00b6 [EXPERIMENTAL] Edit cluster(s). Synopsis \u00b6 [EXPERIMENTAL] Edit cluster(s). k3d cluster edit CLUSTER [flags] Options \u00b6 -h, --help help for edit --port-add [HOST:][HOSTPORT:]CONTAINERPORT[/PROTOCOL][@NODEFILTER] [EXPERIMENTAL] Map ports from the node containers (via the serverlb) to the host (Format: [HOST:][HOSTPORT:]CONTAINERPORT[/PROTOCOL][@NODEFILTER]) - Example: `k3d node edit k3d-mycluster-serverlb --port-add 8080:80` Options inherited from parent commands \u00b6 --timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging) SEE ALSO \u00b6 k3d cluster - Manage cluster(s)","title":"K3d cluster edit"},{"location":"usage/commands/k3d_cluster_edit/#k3d-cluster-edit","text":"[EXPERIMENTAL] Edit cluster(s).","title":"k3d cluster edit"},{"location":"usage/commands/k3d_cluster_edit/#synopsis","text":"[EXPERIMENTAL] Edit cluster(s). k3d cluster edit CLUSTER [flags]","title":"Synopsis"},{"location":"usage/commands/k3d_cluster_edit/#options","text":"-h, --help help for edit --port-add [HOST:][HOSTPORT:]CONTAINERPORT[/PROTOCOL][@NODEFILTER] [EXPERIMENTAL] Map ports from the node containers (via the serverlb) to the host (Format: [HOST:][HOSTPORT:]CONTAINERPORT[/PROTOCOL][@NODEFILTER]) - Example: `k3d node edit k3d-mycluster-serverlb --port-add 8080:80`","title":"Options"},{"location":"usage/commands/k3d_cluster_edit/#options-inherited-from-parent-commands","text":"--timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging)","title":"Options inherited from parent commands"},{"location":"usage/commands/k3d_cluster_edit/#see-also","text":"k3d cluster - Manage cluster(s)","title":"SEE ALSO"},{"location":"usage/commands/k3d_cluster_list/","text":"k3d cluster list \u00b6 List cluster(s) Synopsis \u00b6 List cluster(s). k3d cluster list [NAME [NAME...]] [flags] Options \u00b6 -h, --help help for list --no-headers Disable headers -o, --output string Output format. One of: json|yaml --token Print k3s cluster token Options inherited from parent commands \u00b6 --timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging) SEE ALSO \u00b6 k3d cluster - Manage cluster(s)","title":"K3d cluster list"},{"location":"usage/commands/k3d_cluster_list/#k3d-cluster-list","text":"List cluster(s)","title":"k3d cluster list"},{"location":"usage/commands/k3d_cluster_list/#synopsis","text":"List cluster(s). k3d cluster list [NAME [NAME...]] [flags]","title":"Synopsis"},{"location":"usage/commands/k3d_cluster_list/#options","text":"-h, --help help for list --no-headers Disable headers -o, --output string Output format. One of: json|yaml --token Print k3s cluster token","title":"Options"},{"location":"usage/commands/k3d_cluster_list/#options-inherited-from-parent-commands","text":"--timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging)","title":"Options inherited from parent commands"},{"location":"usage/commands/k3d_cluster_list/#see-also","text":"k3d cluster - Manage cluster(s)","title":"SEE ALSO"},{"location":"usage/commands/k3d_cluster_start/","text":"k3d cluster start \u00b6 Start existing k3d cluster(s) Synopsis \u00b6 Start existing k3d cluster(s) k3d cluster start [NAME [NAME...] | --all] [flags] Options \u00b6 -a, --all Start all existing clusters -h, --help help for start --timeout duration Maximum waiting time for '--wait' before canceling/returning. --wait Wait for the server(s) (and loadbalancer) to be ready before returning. (default true) Options inherited from parent commands \u00b6 --timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging) SEE ALSO \u00b6 k3d cluster - Manage cluster(s)","title":"K3d cluster start"},{"location":"usage/commands/k3d_cluster_start/#k3d-cluster-start","text":"Start existing k3d cluster(s)","title":"k3d cluster start"},{"location":"usage/commands/k3d_cluster_start/#synopsis","text":"Start existing k3d cluster(s) k3d cluster start [NAME [NAME...] | --all] [flags]","title":"Synopsis"},{"location":"usage/commands/k3d_cluster_start/#options","text":"-a, --all Start all existing clusters -h, --help help for start --timeout duration Maximum waiting time for '--wait' before canceling/returning. --wait Wait for the server(s) (and loadbalancer) to be ready before returning. (default true)","title":"Options"},{"location":"usage/commands/k3d_cluster_start/#options-inherited-from-parent-commands","text":"--timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging)","title":"Options inherited from parent commands"},{"location":"usage/commands/k3d_cluster_start/#see-also","text":"k3d cluster - Manage cluster(s)","title":"SEE ALSO"},{"location":"usage/commands/k3d_cluster_stop/","text":"k3d cluster stop \u00b6 Stop existing k3d cluster(s) Synopsis \u00b6 Stop existing k3d cluster(s). k3d cluster stop [NAME [NAME...] | --all] [flags] Options \u00b6 -a, --all Stop all existing clusters -h, --help help for stop Options inherited from parent commands \u00b6 --timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging) SEE ALSO \u00b6 k3d cluster - Manage cluster(s)","title":"K3d cluster stop"},{"location":"usage/commands/k3d_cluster_stop/#k3d-cluster-stop","text":"Stop existing k3d cluster(s)","title":"k3d cluster stop"},{"location":"usage/commands/k3d_cluster_stop/#synopsis","text":"Stop existing k3d cluster(s). k3d cluster stop [NAME [NAME...] | --all] [flags]","title":"Synopsis"},{"location":"usage/commands/k3d_cluster_stop/#options","text":"-a, --all Stop all existing clusters -h, --help help for stop","title":"Options"},{"location":"usage/commands/k3d_cluster_stop/#options-inherited-from-parent-commands","text":"--timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging)","title":"Options inherited from parent commands"},{"location":"usage/commands/k3d_cluster_stop/#see-also","text":"k3d cluster - Manage cluster(s)","title":"SEE ALSO"},{"location":"usage/commands/k3d_completion/","text":"k3d completion \u00b6 Generate completion scripts for [bash, zsh, fish, powershell | psh] Synopsis \u00b6 To load completions: Bash: $ source < ( k3d completion bash ) # To load completions for each session, execute once: # Linux: $ k3d completion bash > / etc / bash_completion . d / k3d # macOS: $ k3d completion bash > / usr / local / etc / bash_completion . d / k3d Zsh: # If shell completion is not already enabled in your environment, # you will need to enable it. You can execute the following once: $ echo \"autoload -U compinit; compinit\" >> ~/. zshrc # To load completions for each session, execute once: $ k3d completion zsh > \"${fpath[1]}/k3d\" # You will need to start a new shell for this setup to take effect. fish: $ k3d completion fish | source # To load completions for each session, execute once: $ k3d completion fish > ~/. config / fish / completions / k3d . fish PowerShell: PS > k3d completion powershell | Out - String | Invoke - Expression # To load completions for every new session, run: PS > k3d completion powershell > k3d . ps1 # and source this file from your PowerShell profile. k3d completion SHELL Options \u00b6 -h, --help help for completion Options inherited from parent commands \u00b6 --timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging) SEE ALSO \u00b6 k3d - https://k3d.io/ -> Run k3s in Docker!","title":"K3d completion"},{"location":"usage/commands/k3d_completion/#k3d-completion","text":"Generate completion scripts for [bash, zsh, fish, powershell | psh]","title":"k3d completion"},{"location":"usage/commands/k3d_completion/#synopsis","text":"To load completions: Bash: $ source < ( k3d completion bash ) # To load completions for each session, execute once: # Linux: $ k3d completion bash > / etc / bash_completion . d / k3d # macOS: $ k3d completion bash > / usr / local / etc / bash_completion . d / k3d Zsh: # If shell completion is not already enabled in your environment, # you will need to enable it. You can execute the following once: $ echo \"autoload -U compinit; compinit\" >> ~/. zshrc # To load completions for each session, execute once: $ k3d completion zsh > \"${fpath[1]}/k3d\" # You will need to start a new shell for this setup to take effect. fish: $ k3d completion fish | source # To load completions for each session, execute once: $ k3d completion fish > ~/. config / fish / completions / k3d . fish PowerShell: PS > k3d completion powershell | Out - String | Invoke - Expression # To load completions for every new session, run: PS > k3d completion powershell > k3d . ps1 # and source this file from your PowerShell profile. k3d completion SHELL","title":"Synopsis"},{"location":"usage/commands/k3d_completion/#options","text":"-h, --help help for completion","title":"Options"},{"location":"usage/commands/k3d_completion/#options-inherited-from-parent-commands","text":"--timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging)","title":"Options inherited from parent commands"},{"location":"usage/commands/k3d_completion/#see-also","text":"k3d - https://k3d.io/ -> Run k3s in Docker!","title":"SEE ALSO"},{"location":"usage/commands/k3d_config/","text":"k3d config \u00b6 Work with config file(s) Synopsis \u00b6 Work with config file(s) k3d config [flags] Options \u00b6 -h, --help help for config Options inherited from parent commands \u00b6 --timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging) SEE ALSO \u00b6 k3d - https://k3d.io/ -> Run k3s in Docker! k3d config init - k3d config migrate -","title":"K3d config"},{"location":"usage/commands/k3d_config/#k3d-config","text":"Work with config file(s)","title":"k3d config"},{"location":"usage/commands/k3d_config/#synopsis","text":"Work with config file(s) k3d config [flags]","title":"Synopsis"},{"location":"usage/commands/k3d_config/#options","text":"-h, --help help for config","title":"Options"},{"location":"usage/commands/k3d_config/#options-inherited-from-parent-commands","text":"--timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging)","title":"Options inherited from parent commands"},{"location":"usage/commands/k3d_config/#see-also","text":"k3d - https://k3d.io/ -> Run k3s in Docker! k3d config init - k3d config migrate -","title":"SEE ALSO"},{"location":"usage/commands/k3d_config_init/","text":"k3d config init \u00b6 k3d config init [flags] Options \u00b6 -f, --force Force overwrite of target file -h, --help help for init -o, --output string Write a default k3d config (default \"k3d-default.yaml\") Options inherited from parent commands \u00b6 --timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging) SEE ALSO \u00b6 k3d config - Work with config file(s)","title":"K3d config init"},{"location":"usage/commands/k3d_config_init/#k3d-config-init","text":"k3d config init [flags]","title":"k3d config init"},{"location":"usage/commands/k3d_config_init/#options","text":"-f, --force Force overwrite of target file -h, --help help for init -o, --output string Write a default k3d config (default \"k3d-default.yaml\")","title":"Options"},{"location":"usage/commands/k3d_config_init/#options-inherited-from-parent-commands","text":"--timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging)","title":"Options inherited from parent commands"},{"location":"usage/commands/k3d_config_init/#see-also","text":"k3d config - Work with config file(s)","title":"SEE ALSO"},{"location":"usage/commands/k3d_config_migrate/","text":"k3d config migrate \u00b6 k3d config migrate INPUT [OUTPUT] [flags] Options \u00b6 -h, --help help for migrate Options inherited from parent commands \u00b6 --timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging) SEE ALSO \u00b6 k3d config - Work with config file(s)","title":"K3d config migrate"},{"location":"usage/commands/k3d_config_migrate/#k3d-config-migrate","text":"k3d config migrate INPUT [OUTPUT] [flags]","title":"k3d config migrate"},{"location":"usage/commands/k3d_config_migrate/#options","text":"-h, --help help for migrate","title":"Options"},{"location":"usage/commands/k3d_config_migrate/#options-inherited-from-parent-commands","text":"--timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging)","title":"Options inherited from parent commands"},{"location":"usage/commands/k3d_config_migrate/#see-also","text":"k3d config - Work with config file(s)","title":"SEE ALSO"},{"location":"usage/commands/k3d_image/","text":"k3d image \u00b6 Handle container images. Synopsis \u00b6 Handle container images. k3d image [flags] Options \u00b6 -h, --help help for image Options inherited from parent commands \u00b6 --timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging) SEE ALSO \u00b6 k3d - https://k3d.io/ -> Run k3s in Docker! k3d image import - Import image(s) from docker into k3d cluster(s).","title":"K3d image"},{"location":"usage/commands/k3d_image/#k3d-image","text":"Handle container images.","title":"k3d image"},{"location":"usage/commands/k3d_image/#synopsis","text":"Handle container images. k3d image [flags]","title":"Synopsis"},{"location":"usage/commands/k3d_image/#options","text":"-h, --help help for image","title":"Options"},{"location":"usage/commands/k3d_image/#options-inherited-from-parent-commands","text":"--timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging)","title":"Options inherited from parent commands"},{"location":"usage/commands/k3d_image/#see-also","text":"k3d - https://k3d.io/ -> Run k3s in Docker! k3d image import - Import image(s) from docker into k3d cluster(s).","title":"SEE ALSO"},{"location":"usage/commands/k3d_image_import/","text":"k3d image import \u00b6 Import image(s) from docker into k3d cluster(s). Synopsis \u00b6 Import image(s) from docker into k3d cluster(s). If an IMAGE starts with the prefix \u2018docker.io/\u2019, then this prefix is stripped internally. That is, \u2018docker.io/rancher/k3d-tools:latest\u2019 is treated as \u2018rancher/k3d-tools:latest\u2019. If an IMAGE starts with the prefix \u2018library/\u2019 (or \u2018docker.io/library/\u2019), then this prefix is stripped internally. That is, \u2018library/busybox:latest\u2019 (or \u2018docker.io/library/busybox:latest\u2019) are treated as \u2018busybox:latest\u2019. If an IMAGE does not have a version tag, then \u2018:latest\u2019 is assumed. That is, \u2018rancher/k3d-tools\u2019 is treated as \u2018rancher/k3d-tools:latest\u2019. A file ARCHIVE always takes precedence. So if a file \u2018./rancher/k3d-tools\u2019 exists, k3d will try to import it instead of the IMAGE of the same name. k3d image import [IMAGE | ARCHIVE [IMAGE | ARCHIVE...]] [flags] Options \u00b6 -c, --cluster stringArray Select clusters to load the image to. (default [k3s-default]) -h, --help help for import -k, --keep-tarball Do not delete the tarball containing the saved images from the shared volume -t, --keep-tools Do not delete the tools node after import Options inherited from parent commands \u00b6 --timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging) SEE ALSO \u00b6 k3d image - Handle container images.","title":"K3d image import"},{"location":"usage/commands/k3d_image_import/#k3d-image-import","text":"Import image(s) from docker into k3d cluster(s).","title":"k3d image import"},{"location":"usage/commands/k3d_image_import/#synopsis","text":"Import image(s) from docker into k3d cluster(s). If an IMAGE starts with the prefix \u2018docker.io/\u2019, then this prefix is stripped internally. That is, \u2018docker.io/rancher/k3d-tools:latest\u2019 is treated as \u2018rancher/k3d-tools:latest\u2019. If an IMAGE starts with the prefix \u2018library/\u2019 (or \u2018docker.io/library/\u2019), then this prefix is stripped internally. That is, \u2018library/busybox:latest\u2019 (or \u2018docker.io/library/busybox:latest\u2019) are treated as \u2018busybox:latest\u2019. If an IMAGE does not have a version tag, then \u2018:latest\u2019 is assumed. That is, \u2018rancher/k3d-tools\u2019 is treated as \u2018rancher/k3d-tools:latest\u2019. A file ARCHIVE always takes precedence. So if a file \u2018./rancher/k3d-tools\u2019 exists, k3d will try to import it instead of the IMAGE of the same name. k3d image import [IMAGE | ARCHIVE [IMAGE | ARCHIVE...]] [flags]","title":"Synopsis"},{"location":"usage/commands/k3d_image_import/#options","text":"-c, --cluster stringArray Select clusters to load the image to. (default [k3s-default]) -h, --help help for import -k, --keep-tarball Do not delete the tarball containing the saved images from the shared volume -t, --keep-tools Do not delete the tools node after import","title":"Options"},{"location":"usage/commands/k3d_image_import/#options-inherited-from-parent-commands","text":"--timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging)","title":"Options inherited from parent commands"},{"location":"usage/commands/k3d_image_import/#see-also","text":"k3d image - Handle container images.","title":"SEE ALSO"},{"location":"usage/commands/k3d_kubeconfig/","text":"k3d kubeconfig \u00b6 Manage kubeconfig(s) Synopsis \u00b6 Manage kubeconfig(s) k3d kubeconfig [flags] Options \u00b6 -h, --help help for kubeconfig Options inherited from parent commands \u00b6 --timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging) SEE ALSO \u00b6 k3d - https://k3d.io/ -> Run k3s in Docker! k3d kubeconfig get - Print kubeconfig(s) from cluster(s). k3d kubeconfig merge - Write/Merge kubeconfig(s) from cluster(s) into new or existing kubeconfig/file.","title":"K3d kubeconfig"},{"location":"usage/commands/k3d_kubeconfig/#k3d-kubeconfig","text":"Manage kubeconfig(s)","title":"k3d kubeconfig"},{"location":"usage/commands/k3d_kubeconfig/#synopsis","text":"Manage kubeconfig(s) k3d kubeconfig [flags]","title":"Synopsis"},{"location":"usage/commands/k3d_kubeconfig/#options","text":"-h, --help help for kubeconfig","title":"Options"},{"location":"usage/commands/k3d_kubeconfig/#options-inherited-from-parent-commands","text":"--timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging)","title":"Options inherited from parent commands"},{"location":"usage/commands/k3d_kubeconfig/#see-also","text":"k3d - https://k3d.io/ -> Run k3s in Docker! k3d kubeconfig get - Print kubeconfig(s) from cluster(s). k3d kubeconfig merge - Write/Merge kubeconfig(s) from cluster(s) into new or existing kubeconfig/file.","title":"SEE ALSO"},{"location":"usage/commands/k3d_kubeconfig_get/","text":"k3d kubeconfig get \u00b6 Print kubeconfig(s) from cluster(s). Synopsis \u00b6 Print kubeconfig(s) from cluster(s). k3d kubeconfig get [CLUSTER [CLUSTER [...]] | --all] [flags] Options \u00b6 -a, --all Output kubeconfigs from all existing clusters -h, --help help for get Options inherited from parent commands \u00b6 --timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging) SEE ALSO \u00b6 k3d kubeconfig - Manage kubeconfig(s)","title":"K3d kubeconfig get"},{"location":"usage/commands/k3d_kubeconfig_get/#k3d-kubeconfig-get","text":"Print kubeconfig(s) from cluster(s).","title":"k3d kubeconfig get"},{"location":"usage/commands/k3d_kubeconfig_get/#synopsis","text":"Print kubeconfig(s) from cluster(s). k3d kubeconfig get [CLUSTER [CLUSTER [...]] | --all] [flags]","title":"Synopsis"},{"location":"usage/commands/k3d_kubeconfig_get/#options","text":"-a, --all Output kubeconfigs from all existing clusters -h, --help help for get","title":"Options"},{"location":"usage/commands/k3d_kubeconfig_get/#options-inherited-from-parent-commands","text":"--timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging)","title":"Options inherited from parent commands"},{"location":"usage/commands/k3d_kubeconfig_get/#see-also","text":"k3d kubeconfig - Manage kubeconfig(s)","title":"SEE ALSO"},{"location":"usage/commands/k3d_kubeconfig_merge/","text":"k3d kubeconfig merge \u00b6 Write/Merge kubeconfig(s) from cluster(s) into new or existing kubeconfig/file. Synopsis \u00b6 Write/Merge kubeconfig(s) from cluster(s) into new or existing kubeconfig/file. k3d kubeconfig merge [CLUSTER [CLUSTER [...]] | --all] [flags] Options \u00b6 -a, --all Get kubeconfigs from all existing clusters -h, --help help for merge -d, --kubeconfig-merge-default Merge into the default kubeconfig ($KUBECONFIG or /home/thklein/.kube/config) -s, --kubeconfig-switch-context Switch to new context (default true) -o, --output string Define output [ - | FILE ] (default from $KUBECONFIG or /home/thklein/.kube/config --overwrite [Careful!] Overwrite existing file, ignoring its contents -u, --update Update conflicting fields in existing kubeconfig (default true) Options inherited from parent commands \u00b6 --timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging) SEE ALSO \u00b6 k3d kubeconfig - Manage kubeconfig(s)","title":"K3d kubeconfig merge"},{"location":"usage/commands/k3d_kubeconfig_merge/#k3d-kubeconfig-merge","text":"Write/Merge kubeconfig(s) from cluster(s) into new or existing kubeconfig/file.","title":"k3d kubeconfig merge"},{"location":"usage/commands/k3d_kubeconfig_merge/#synopsis","text":"Write/Merge kubeconfig(s) from cluster(s) into new or existing kubeconfig/file. k3d kubeconfig merge [CLUSTER [CLUSTER [...]] | --all] [flags]","title":"Synopsis"},{"location":"usage/commands/k3d_kubeconfig_merge/#options","text":"-a, --all Get kubeconfigs from all existing clusters -h, --help help for merge -d, --kubeconfig-merge-default Merge into the default kubeconfig ($KUBECONFIG or /home/thklein/.kube/config) -s, --kubeconfig-switch-context Switch to new context (default true) -o, --output string Define output [ - | FILE ] (default from $KUBECONFIG or /home/thklein/.kube/config --overwrite [Careful!] Overwrite existing file, ignoring its contents -u, --update Update conflicting fields in existing kubeconfig (default true)","title":"Options"},{"location":"usage/commands/k3d_kubeconfig_merge/#options-inherited-from-parent-commands","text":"--timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging)","title":"Options inherited from parent commands"},{"location":"usage/commands/k3d_kubeconfig_merge/#see-also","text":"k3d kubeconfig - Manage kubeconfig(s)","title":"SEE ALSO"},{"location":"usage/commands/k3d_node/","text":"k3d node \u00b6 Manage node(s) Synopsis \u00b6 Manage node(s) k3d node [flags] Options \u00b6 -h, --help help for node Options inherited from parent commands \u00b6 --timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging) SEE ALSO \u00b6 k3d - https://k3d.io/ -> Run k3s in Docker! k3d node create - Create a new k3s node in docker k3d node delete - Delete node(s). k3d node edit - [EXPERIMENTAL] Edit node(s). k3d node list - List node(s) k3d node start - Start an existing k3d node k3d node stop - Stop an existing k3d node","title":"K3d node"},{"location":"usage/commands/k3d_node/#k3d-node","text":"Manage node(s)","title":"k3d node"},{"location":"usage/commands/k3d_node/#synopsis","text":"Manage node(s) k3d node [flags]","title":"Synopsis"},{"location":"usage/commands/k3d_node/#options","text":"-h, --help help for node","title":"Options"},{"location":"usage/commands/k3d_node/#options-inherited-from-parent-commands","text":"--timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging)","title":"Options inherited from parent commands"},{"location":"usage/commands/k3d_node/#see-also","text":"k3d - https://k3d.io/ -> Run k3s in Docker! k3d node create - Create a new k3s node in docker k3d node delete - Delete node(s). k3d node edit - [EXPERIMENTAL] Edit node(s). k3d node list - List node(s) k3d node start - Start an existing k3d node k3d node stop - Stop an existing k3d node","title":"SEE ALSO"},{"location":"usage/commands/k3d_node_create/","text":"k3d node create \u00b6 Create a new k3s node in docker Synopsis \u00b6 Create a new containerized k3s node (k3s in docker). k3d node create NAME [flags] Options \u00b6 -c, --cluster string Select the cluster that the node shall connect to. (default \"k3s-default\") -h, --help help for create -i, --image string Specify k3s image used for the node(s) (default \"docker.io/rancher/k3s:v1.21.4-k3s2\") --k3s-node-label strings Specify k3s node labels in format \"foo=bar\" --memory string Memory limit imposed on the node [From docker] -n, --network strings Add node to (another) runtime network --replicas int Number of replicas of this node specification. (default 1) --role string Specify node role [server, agent] (default \"agent\") --runtime-label strings Specify container runtime labels in format \"foo=bar\" --timeout duration Maximum waiting time for '--wait' before canceling/returning. -t, --token string Override cluster token (required when connecting to an external cluster) --wait Wait for the node(s) to be ready before returning. (default true) Options inherited from parent commands \u00b6 --timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging) SEE ALSO \u00b6 k3d node - Manage node(s)","title":"K3d node create"},{"location":"usage/commands/k3d_node_create/#k3d-node-create","text":"Create a new k3s node in docker","title":"k3d node create"},{"location":"usage/commands/k3d_node_create/#synopsis","text":"Create a new containerized k3s node (k3s in docker). k3d node create NAME [flags]","title":"Synopsis"},{"location":"usage/commands/k3d_node_create/#options","text":"-c, --cluster string Select the cluster that the node shall connect to. (default \"k3s-default\") -h, --help help for create -i, --image string Specify k3s image used for the node(s) (default \"docker.io/rancher/k3s:v1.21.4-k3s2\") --k3s-node-label strings Specify k3s node labels in format \"foo=bar\" --memory string Memory limit imposed on the node [From docker] -n, --network strings Add node to (another) runtime network --replicas int Number of replicas of this node specification. (default 1) --role string Specify node role [server, agent] (default \"agent\") --runtime-label strings Specify container runtime labels in format \"foo=bar\" --timeout duration Maximum waiting time for '--wait' before canceling/returning. -t, --token string Override cluster token (required when connecting to an external cluster) --wait Wait for the node(s) to be ready before returning. (default true)","title":"Options"},{"location":"usage/commands/k3d_node_create/#options-inherited-from-parent-commands","text":"--timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging)","title":"Options inherited from parent commands"},{"location":"usage/commands/k3d_node_create/#see-also","text":"k3d node - Manage node(s)","title":"SEE ALSO"},{"location":"usage/commands/k3d_node_delete/","text":"k3d node delete \u00b6 Delete node(s). Synopsis \u00b6 Delete node(s). k3d node delete (NAME | --all) [flags] Options \u00b6 -a, --all Delete all existing nodes -h, --help help for delete -r, --registries Also delete registries Options inherited from parent commands \u00b6 --timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging) SEE ALSO \u00b6 k3d node - Manage node(s)","title":"K3d node delete"},{"location":"usage/commands/k3d_node_delete/#k3d-node-delete","text":"Delete node(s).","title":"k3d node delete"},{"location":"usage/commands/k3d_node_delete/#synopsis","text":"Delete node(s). k3d node delete (NAME | --all) [flags]","title":"Synopsis"},{"location":"usage/commands/k3d_node_delete/#options","text":"-a, --all Delete all existing nodes -h, --help help for delete -r, --registries Also delete registries","title":"Options"},{"location":"usage/commands/k3d_node_delete/#options-inherited-from-parent-commands","text":"--timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging)","title":"Options inherited from parent commands"},{"location":"usage/commands/k3d_node_delete/#see-also","text":"k3d node - Manage node(s)","title":"SEE ALSO"},{"location":"usage/commands/k3d_node_edit/","text":"k3d node edit \u00b6 [EXPERIMENTAL] Edit node(s). Synopsis \u00b6 [EXPERIMENTAL] Edit node(s). k3d node edit NODE [flags] Options \u00b6 -h, --help help for edit --port-add [HOST:][HOSTPORT:]CONTAINERPORT[/PROTOCOL][@NODEFILTER] [EXPERIMENTAL] (serverlb only!) Map ports from the node container to the host (Format: [HOST:][HOSTPORT:]CONTAINERPORT[/PROTOCOL][@NODEFILTER]) - Example: `k3d node edit k3d-mycluster-serverlb --port-add 8080:80` Options inherited from parent commands \u00b6 --timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging) SEE ALSO \u00b6 k3d node - Manage node(s)","title":"K3d node edit"},{"location":"usage/commands/k3d_node_edit/#k3d-node-edit","text":"[EXPERIMENTAL] Edit node(s).","title":"k3d node edit"},{"location":"usage/commands/k3d_node_edit/#synopsis","text":"[EXPERIMENTAL] Edit node(s). k3d node edit NODE [flags]","title":"Synopsis"},{"location":"usage/commands/k3d_node_edit/#options","text":"-h, --help help for edit --port-add [HOST:][HOSTPORT:]CONTAINERPORT[/PROTOCOL][@NODEFILTER] [EXPERIMENTAL] (serverlb only!) Map ports from the node container to the host (Format: [HOST:][HOSTPORT:]CONTAINERPORT[/PROTOCOL][@NODEFILTER]) - Example: `k3d node edit k3d-mycluster-serverlb --port-add 8080:80`","title":"Options"},{"location":"usage/commands/k3d_node_edit/#options-inherited-from-parent-commands","text":"--timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging)","title":"Options inherited from parent commands"},{"location":"usage/commands/k3d_node_edit/#see-also","text":"k3d node - Manage node(s)","title":"SEE ALSO"},{"location":"usage/commands/k3d_node_list/","text":"k3d node list \u00b6 List node(s) Synopsis \u00b6 List node(s). k3d node list [NODE [NODE...]] [flags] Options \u00b6 -h, --help help for list --no-headers Disable headers -o, --output string Output format. One of: json|yaml Options inherited from parent commands \u00b6 --timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging) SEE ALSO \u00b6 k3d node - Manage node(s)","title":"K3d node list"},{"location":"usage/commands/k3d_node_list/#k3d-node-list","text":"List node(s)","title":"k3d node list"},{"location":"usage/commands/k3d_node_list/#synopsis","text":"List node(s). k3d node list [NODE [NODE...]] [flags]","title":"Synopsis"},{"location":"usage/commands/k3d_node_list/#options","text":"-h, --help help for list --no-headers Disable headers -o, --output string Output format. One of: json|yaml","title":"Options"},{"location":"usage/commands/k3d_node_list/#options-inherited-from-parent-commands","text":"--timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging)","title":"Options inherited from parent commands"},{"location":"usage/commands/k3d_node_list/#see-also","text":"k3d node - Manage node(s)","title":"SEE ALSO"},{"location":"usage/commands/k3d_node_start/","text":"k3d node start \u00b6 Start an existing k3d node Synopsis \u00b6 Start an existing k3d node. k3d node start NODE [flags] Options \u00b6 -h, --help help for start Options inherited from parent commands \u00b6 --timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging) SEE ALSO \u00b6 k3d node - Manage node(s)","title":"K3d node start"},{"location":"usage/commands/k3d_node_start/#k3d-node-start","text":"Start an existing k3d node","title":"k3d node start"},{"location":"usage/commands/k3d_node_start/#synopsis","text":"Start an existing k3d node. k3d node start NODE [flags]","title":"Synopsis"},{"location":"usage/commands/k3d_node_start/#options","text":"-h, --help help for start","title":"Options"},{"location":"usage/commands/k3d_node_start/#options-inherited-from-parent-commands","text":"--timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging)","title":"Options inherited from parent commands"},{"location":"usage/commands/k3d_node_start/#see-also","text":"k3d node - Manage node(s)","title":"SEE ALSO"},{"location":"usage/commands/k3d_node_stop/","text":"k3d node stop \u00b6 Stop an existing k3d node Synopsis \u00b6 Stop an existing k3d node. k3d node stop NAME [flags] Options \u00b6 -h, --help help for stop Options inherited from parent commands \u00b6 --timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging) SEE ALSO \u00b6 k3d node - Manage node(s)","title":"K3d node stop"},{"location":"usage/commands/k3d_node_stop/#k3d-node-stop","text":"Stop an existing k3d node","title":"k3d node stop"},{"location":"usage/commands/k3d_node_stop/#synopsis","text":"Stop an existing k3d node. k3d node stop NAME [flags]","title":"Synopsis"},{"location":"usage/commands/k3d_node_stop/#options","text":"-h, --help help for stop","title":"Options"},{"location":"usage/commands/k3d_node_stop/#options-inherited-from-parent-commands","text":"--timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging)","title":"Options inherited from parent commands"},{"location":"usage/commands/k3d_node_stop/#see-also","text":"k3d node - Manage node(s)","title":"SEE ALSO"},{"location":"usage/commands/k3d_registry/","text":"k3d registry \u00b6 Manage registry/registries Synopsis \u00b6 Manage registry/registries k3d registry [flags] Options \u00b6 -h, --help help for registry Options inherited from parent commands \u00b6 --timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging) SEE ALSO \u00b6 k3d - https://k3d.io/ -> Run k3s in Docker! k3d registry create - Create a new registry k3d registry delete - Delete registry/registries. k3d registry list - List registries","title":"K3d registry"},{"location":"usage/commands/k3d_registry/#k3d-registry","text":"Manage registry/registries","title":"k3d registry"},{"location":"usage/commands/k3d_registry/#synopsis","text":"Manage registry/registries k3d registry [flags]","title":"Synopsis"},{"location":"usage/commands/k3d_registry/#options","text":"-h, --help help for registry","title":"Options"},{"location":"usage/commands/k3d_registry/#options-inherited-from-parent-commands","text":"--timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging)","title":"Options inherited from parent commands"},{"location":"usage/commands/k3d_registry/#see-also","text":"k3d - https://k3d.io/ -> Run k3s in Docker! k3d registry create - Create a new registry k3d registry delete - Delete registry/registries. k3d registry list - List registries","title":"SEE ALSO"},{"location":"usage/commands/k3d_registry_create/","text":"k3d registry create \u00b6 Create a new registry Synopsis \u00b6 Create a new registry. k3d registry create NAME [flags] Options \u00b6 -h, --help help for create -i, --image string Specify image used for the registry (default \"docker.io/library/registry:2\") --no-help Disable the help text (How-To use the registry) -p, --port [HOST:]HOSTPORT Select which port the registry should be listening on on your machine (localhost) (Format: [HOST:]HOSTPORT) - Example: `k3d registry create --port 0.0.0.0:5111` (default \"random\") Options inherited from parent commands \u00b6 --timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging) SEE ALSO \u00b6 k3d registry - Manage registry/registries","title":"K3d registry create"},{"location":"usage/commands/k3d_registry_create/#k3d-registry-create","text":"Create a new registry","title":"k3d registry create"},{"location":"usage/commands/k3d_registry_create/#synopsis","text":"Create a new registry. k3d registry create NAME [flags]","title":"Synopsis"},{"location":"usage/commands/k3d_registry_create/#options","text":"-h, --help help for create -i, --image string Specify image used for the registry (default \"docker.io/library/registry:2\") --no-help Disable the help text (How-To use the registry) -p, --port [HOST:]HOSTPORT Select which port the registry should be listening on on your machine (localhost) (Format: [HOST:]HOSTPORT) - Example: `k3d registry create --port 0.0.0.0:5111` (default \"random\")","title":"Options"},{"location":"usage/commands/k3d_registry_create/#options-inherited-from-parent-commands","text":"--timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging)","title":"Options inherited from parent commands"},{"location":"usage/commands/k3d_registry_create/#see-also","text":"k3d registry - Manage registry/registries","title":"SEE ALSO"},{"location":"usage/commands/k3d_registry_delete/","text":"k3d registry delete \u00b6 Delete registry/registries. Synopsis \u00b6 Delete registry/registries. k3d registry delete (NAME | --all) [flags] Options \u00b6 -a, --all Delete all existing registries -h, --help help for delete Options inherited from parent commands \u00b6 --timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging) SEE ALSO \u00b6 k3d registry - Manage registry/registries","title":"K3d registry delete"},{"location":"usage/commands/k3d_registry_delete/#k3d-registry-delete","text":"Delete registry/registries.","title":"k3d registry delete"},{"location":"usage/commands/k3d_registry_delete/#synopsis","text":"Delete registry/registries. k3d registry delete (NAME | --all) [flags]","title":"Synopsis"},{"location":"usage/commands/k3d_registry_delete/#options","text":"-a, --all Delete all existing registries -h, --help help for delete","title":"Options"},{"location":"usage/commands/k3d_registry_delete/#options-inherited-from-parent-commands","text":"--timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging)","title":"Options inherited from parent commands"},{"location":"usage/commands/k3d_registry_delete/#see-also","text":"k3d registry - Manage registry/registries","title":"SEE ALSO"},{"location":"usage/commands/k3d_registry_list/","text":"k3d registry list \u00b6 List registries Synopsis \u00b6 List registries. k3d registry list [NAME [NAME...]] [flags] Options \u00b6 -h, --help help for list --no-headers Disable headers -o, --output string Output format. One of: json|yaml Options inherited from parent commands \u00b6 --timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging) SEE ALSO \u00b6 k3d registry - Manage registry/registries","title":"K3d registry list"},{"location":"usage/commands/k3d_registry_list/#k3d-registry-list","text":"List registries","title":"k3d registry list"},{"location":"usage/commands/k3d_registry_list/#synopsis","text":"List registries. k3d registry list [NAME [NAME...]] [flags]","title":"Synopsis"},{"location":"usage/commands/k3d_registry_list/#options","text":"-h, --help help for list --no-headers Disable headers -o, --output string Output format. One of: json|yaml","title":"Options"},{"location":"usage/commands/k3d_registry_list/#options-inherited-from-parent-commands","text":"--timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging)","title":"Options inherited from parent commands"},{"location":"usage/commands/k3d_registry_list/#see-also","text":"k3d registry - Manage registry/registries","title":"SEE ALSO"},{"location":"usage/commands/k3d_version/","text":"k3d version \u00b6 Show k3d and default k3s version Synopsis \u00b6 Show k3d and default k3s version k3d version [flags] Options \u00b6 -h, --help help for version Options inherited from parent commands \u00b6 --timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging) SEE ALSO \u00b6 k3d - https://k3d.io/ -> Run k3s in Docker!","title":"K3d version"},{"location":"usage/commands/k3d_version/#k3d-version","text":"Show k3d and default k3s version","title":"k3d version"},{"location":"usage/commands/k3d_version/#synopsis","text":"Show k3d and default k3s version k3d version [flags]","title":"Synopsis"},{"location":"usage/commands/k3d_version/#options","text":"-h, --help help for version","title":"Options"},{"location":"usage/commands/k3d_version/#options-inherited-from-parent-commands","text":"--timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging)","title":"Options inherited from parent commands"},{"location":"usage/commands/k3d_version/#see-also","text":"k3d - https://k3d.io/ -> Run k3s in Docker!","title":"SEE ALSO"}]}
\ No newline at end of file
+{"config":{"indexing":"full","lang":["en"],"min_search_length":3,"prebuild_index":false,"separator":"[\\s\\-]+"},"docs":[{"location":"","text":"Overview \u00b6 What is k3d? \u00b6 k3d is a lightweight wrapper to run k3s (Rancher Lab\u2019s minimal Kubernetes distribution) in docker. k3d makes it very easy to create single- and multi-node k3s clusters in docker, e.g. for local development on Kubernetes. Note: k3d is a community-driven project , that is supported by Rancher (SUSE) and it\u2019s not an official Rancher (SUSE) project. View a quick demo Learning \u00b6 k3d demo repository: iwilltry42/k3d-demo Featured use-cases include: hot-reloading of code when developing on k3d (Python Flask App) build-deploy-test cycle using Tilt full cluster lifecycle for simple and multi-server clusters Proof of Concept of using k3d as a service in Drone CI Rancher Meetup - May 2020 - Simplifying Your Cloud-Native Development Workflow With K3s, K3c and K3d (YouTube) Requirements \u00b6 docker to be able to use k3d at all kubectl to interact with the Kubernetes cluster Releases \u00b6 Platform Stage Version Release Date GitHub Releases stable GitHub Releases latest Homebrew - - Chocolatey stable - Installation \u00b6 You have several options there: Install Script \u00b6 Install current latest release \u00b6 wget: wget -q -O - https://raw.githubusercontent.com/rancher/k3d/main/install.sh | bash curl: curl -s https://raw.githubusercontent.com/rancher/k3d/main/install.sh | bash Install specific release \u00b6 Use the install script to grab a specific release (via TAG environment variable): wget: wget -q -O - https://raw.githubusercontent.com/rancher/k3d/main/install.sh | TAG = v5.0.0 bash curl: curl -s https://raw.githubusercontent.com/rancher/k3d/main/install.sh | TAG = v5.0.0 bash Other Installers \u00b6 Other Installation Methods Homebrew (MacOS/Linux) : brew install k3d Note : The formula can be found in homebrew/homebrew-core and is mirrored to homebrew/linuxbrew-core AUR (Arch Linux User Repository) : yay -S rancher-k3d-bin Package rancher-k3d-bin Download GitHub Release Grab a release binary from the release tab and install it yourself Chocolatey (Windows) : choco install k3d Note : package source can be found in erwinkersten/chocolatey-packages arkade : arkade get k3d asdf : asdf plugin-add k3d && asdf install k3d latest Note : asdf plugin-add k3d , then asdf install k3d <tag> with <tag> = latest or 5.x.x for a specific version (maintained by spencergilbert/asdf-k3d ) Others install via go: go install github.com/rancher/k3d@latest ( Note : this will give you unreleased/bleeding-edge changes) Quick Start \u00b6 Create a cluster named mycluster with just a single server node: k3d cluster create mycluster Use the new cluster with kubectl , e.g.: kubectl get nodes Getting the cluster\u2019s kubeconfig (included in k3d cluster create ) Get the new cluster\u2019s connection details merged into your default kubeconfig (usually specified using the KUBECONFIG environment variable or the default path $HOME /.kube/config ) and directly switch to the new context: k3d kubeconfig merge mycluster --kubeconfig-switch-context Related Projects \u00b6 vscode-k3d : VSCode Extension to handle k3d clusters from within VSCode k3x : a graphics interface (for Linux) to k3d. AbsaOSS/k3d-action : fully customizable GitHub Action to run lightweight Kubernetes clusters. AutoK3s : a lightweight tool to help run K3s everywhere including k3d provider. nolar/setup-k3d-k3s : setup K3d/K3s for GitHub Actions.","title":"Overview"},{"location":"#overview","text":"","title":"Overview"},{"location":"#what-is-k3d","text":"k3d is a lightweight wrapper to run k3s (Rancher Lab\u2019s minimal Kubernetes distribution) in docker. k3d makes it very easy to create single- and multi-node k3s clusters in docker, e.g. for local development on Kubernetes. Note: k3d is a community-driven project , that is supported by Rancher (SUSE) and it\u2019s not an official Rancher (SUSE) project. View a quick demo","title":"What is k3d?"},{"location":"#learning","text":"k3d demo repository: iwilltry42/k3d-demo Featured use-cases include: hot-reloading of code when developing on k3d (Python Flask App) build-deploy-test cycle using Tilt full cluster lifecycle for simple and multi-server clusters Proof of Concept of using k3d as a service in Drone CI Rancher Meetup - May 2020 - Simplifying Your Cloud-Native Development Workflow With K3s, K3c and K3d (YouTube)","title":"Learning"},{"location":"#requirements","text":"docker to be able to use k3d at all kubectl to interact with the Kubernetes cluster","title":"Requirements"},{"location":"#releases","text":"Platform Stage Version Release Date GitHub Releases stable GitHub Releases latest Homebrew - - Chocolatey stable -","title":"Releases"},{"location":"#installation","text":"You have several options there:","title":"Installation"},{"location":"#install-script","text":"","title":" Install Script"},{"location":"#install-current-latest-release","text":"wget: wget -q -O - https://raw.githubusercontent.com/rancher/k3d/main/install.sh | bash curl: curl -s https://raw.githubusercontent.com/rancher/k3d/main/install.sh | bash","title":"Install current latest release"},{"location":"#install-specific-release","text":"Use the install script to grab a specific release (via TAG environment variable): wget: wget -q -O - https://raw.githubusercontent.com/rancher/k3d/main/install.sh | TAG = v5.0.0 bash curl: curl -s https://raw.githubusercontent.com/rancher/k3d/main/install.sh | TAG = v5.0.0 bash","title":"Install specific release"},{"location":"#other-installers","text":"Other Installation Methods Homebrew (MacOS/Linux) : brew install k3d Note : The formula can be found in homebrew/homebrew-core and is mirrored to homebrew/linuxbrew-core AUR (Arch Linux User Repository) : yay -S rancher-k3d-bin Package rancher-k3d-bin Download GitHub Release Grab a release binary from the release tab and install it yourself Chocolatey (Windows) : choco install k3d Note : package source can be found in erwinkersten/chocolatey-packages arkade : arkade get k3d asdf : asdf plugin-add k3d && asdf install k3d latest Note : asdf plugin-add k3d , then asdf install k3d <tag> with <tag> = latest or 5.x.x for a specific version (maintained by spencergilbert/asdf-k3d ) Others install via go: go install github.com/rancher/k3d@latest ( Note : this will give you unreleased/bleeding-edge changes)","title":"Other Installers"},{"location":"#quick-start","text":"Create a cluster named mycluster with just a single server node: k3d cluster create mycluster Use the new cluster with kubectl , e.g.: kubectl get nodes Getting the cluster\u2019s kubeconfig (included in k3d cluster create ) Get the new cluster\u2019s connection details merged into your default kubeconfig (usually specified using the KUBECONFIG environment variable or the default path $HOME /.kube/config ) and directly switch to the new context: k3d kubeconfig merge mycluster --kubeconfig-switch-context","title":"Quick Start"},{"location":"#related-projects","text":"vscode-k3d : VSCode Extension to handle k3d clusters from within VSCode k3x : a graphics interface (for Linux) to k3d. AbsaOSS/k3d-action : fully customizable GitHub Action to run lightweight Kubernetes clusters. AutoK3s : a lightweight tool to help run K3s everywhere including k3d provider. nolar/setup-k3d-k3s : setup K3d/K3s for GitHub Actions.","title":"Related Projects"},{"location":"design/defaults/","text":"Defaults \u00b6 k3d reserved settings \u00b6 When you create a K3s cluster in Docker using k3d, we make use of some K3s configuration options, making them \u201creserved\u201d for k3d. This means, that overriding those options with your own may break the cluster setup. Environment Variables \u00b6 The following K3s environment variables are used to configure the cluster: Variable K3d Default Configurable? K3S_URL https://$CLUSTERNAME-server-0:6443 no K3S_TOKEN random yes ( --token ) K3S_KUBECONFIG_OUTPUT /output/kubeconfig.yaml no k3d Loadbalancer \u00b6 By default, k3d creates an Nginx loadbalancer alongside the clusters it creates to handle the port-forwarding. The loadbalancer can partly be configured using k3d-defined settings. Nginx setting k3d default k3d setting proxy_timeout (default for all server stanzas) 600 (s) settings.defaultProxyTimeout worker_connections 1024 settings.workerConnections Overrides \u00b6 Example via CLI: k3d cluster create --lb-config-override settings.defaultProxyTimeout=900 Example via Config File: # ... truncated ... k3d : loadbalancer : configOverrides : - settings.workerConnections=2048 Multiple server nodes \u00b6 by default, when --server > 1 and no --datastore-x option is set, the first server node (server-0) will be the initializing server node the initializing server node will have the --cluster-init flag appended all other server nodes will refer to the initializing server node via --server https://<init-node>:6443 API-Ports \u00b6 by default, we expose the API-Port ( 6443 ) by forwarding traffic from the default server loadbalancer (nginx container) to the server node(s) port 6443 of the loadbalancer is then mapped to a specific ( --api-port flag) or a random (default) port on the host system Kubeconfig \u00b6 if --kubeconfig-update-default is set, we use the default loading rules to get the default kubeconfig: First: kubeconfig specified via the KUBECONFIG environment variable (error out if multiple are specified) Second: default kubeconfig in home directory (e.g. $HOME/.kube/config ) Networking \u00b6 by default, k3d creates a new (docker) network for every cluster","title":"Defaults"},{"location":"design/defaults/#defaults","text":"","title":"Defaults"},{"location":"design/defaults/#k3d-reserved-settings","text":"When you create a K3s cluster in Docker using k3d, we make use of some K3s configuration options, making them \u201creserved\u201d for k3d. This means, that overriding those options with your own may break the cluster setup.","title":"k3d reserved settings"},{"location":"design/defaults/#environment-variables","text":"The following K3s environment variables are used to configure the cluster: Variable K3d Default Configurable? K3S_URL https://$CLUSTERNAME-server-0:6443 no K3S_TOKEN random yes ( --token ) K3S_KUBECONFIG_OUTPUT /output/kubeconfig.yaml no","title":"Environment Variables"},{"location":"design/defaults/#k3d-loadbalancer","text":"By default, k3d creates an Nginx loadbalancer alongside the clusters it creates to handle the port-forwarding. The loadbalancer can partly be configured using k3d-defined settings. Nginx setting k3d default k3d setting proxy_timeout (default for all server stanzas) 600 (s) settings.defaultProxyTimeout worker_connections 1024 settings.workerConnections","title":"k3d Loadbalancer"},{"location":"design/defaults/#overrides","text":"Example via CLI: k3d cluster create --lb-config-override settings.defaultProxyTimeout=900 Example via Config File: # ... truncated ... k3d : loadbalancer : configOverrides : - settings.workerConnections=2048","title":"Overrides"},{"location":"design/defaults/#multiple-server-nodes","text":"by default, when --server > 1 and no --datastore-x option is set, the first server node (server-0) will be the initializing server node the initializing server node will have the --cluster-init flag appended all other server nodes will refer to the initializing server node via --server https://<init-node>:6443","title":"Multiple server nodes"},{"location":"design/defaults/#api-ports","text":"by default, we expose the API-Port ( 6443 ) by forwarding traffic from the default server loadbalancer (nginx container) to the server node(s) port 6443 of the loadbalancer is then mapped to a specific ( --api-port flag) or a random (default) port on the host system","title":"API-Ports"},{"location":"design/defaults/#kubeconfig","text":"if --kubeconfig-update-default is set, we use the default loading rules to get the default kubeconfig: First: kubeconfig specified via the KUBECONFIG environment variable (error out if multiple are specified) Second: default kubeconfig in home directory (e.g. $HOME/.kube/config )","title":"Kubeconfig"},{"location":"design/defaults/#networking","text":"by default, k3d creates a new (docker) network for every cluster","title":"Networking"},{"location":"design/networking/","text":"Networking \u00b6 Related issues: rancher/k3d #220 Introduction \u00b6 By default, k3d creates a new (docker) network for every new cluster. Using the --network STRING flag upon creation to connect to an existing network. Existing networks won\u2019t be managed by k3d together with the cluster lifecycle. Connecting to docker \u201cinternal\u201d/pre-defined networks \u00b6 host network \u00b6 When using the --network flag to connect to the host network (i.e. k3d cluster create --network host ), you won\u2019t be able to create more than one server node . An edge case would be one server node (with agent disabled) and one agent node. bridge network \u00b6 By default, every network that k3d creates is working in bridge mode. But when you try to use --network bridge to connect to docker\u2019s internal bridge network, you may run into issues with grabbing certificates from the API-Server. Single-Node clusters should work though. none \u201cnetwork\u201d \u00b6 Well.. this doesn\u2019t really make sense for k3d anyway \u00af\\_(\u30c4)_/\u00af","title":"Networking"},{"location":"design/networking/#networking","text":"Related issues: rancher/k3d #220","title":"Networking"},{"location":"design/networking/#introduction","text":"By default, k3d creates a new (docker) network for every new cluster. Using the --network STRING flag upon creation to connect to an existing network. Existing networks won\u2019t be managed by k3d together with the cluster lifecycle.","title":"Introduction"},{"location":"design/networking/#connecting-to-docker-internalpre-defined-networks","text":"","title":"Connecting to docker \"internal\"/pre-defined networks"},{"location":"design/networking/#host-network","text":"When using the --network flag to connect to the host network (i.e. k3d cluster create --network host ), you won\u2019t be able to create more than one server node . An edge case would be one server node (with agent disabled) and one agent node.","title":"host network"},{"location":"design/networking/#bridge-network","text":"By default, every network that k3d creates is working in bridge mode. But when you try to use --network bridge to connect to docker\u2019s internal bridge network, you may run into issues with grabbing certificates from the API-Server. Single-Node clusters should work though.","title":"bridge network"},{"location":"design/networking/#none-network","text":"Well.. this doesn\u2019t really make sense for k3d anyway \u00af\\_(\u30c4)_/\u00af","title":"none \"network\""},{"location":"design/project/","text":"Project Overview \u00b6 About This Page \u00b6 On this page we\u2019ll try to give an overview of all the moving bits and pieces in k3d to ease contributions to the project. Directory Overview \u00b6 .github/ templates for issues and pull requests GitHub Action workflow definitions cmd/ everything related to the actual k3d CLI, like the whole command tree, config initialization, argument parsing, etc. docgen/ sub-module used to auto-generate the documentation for the CLI commands, which ends up in docs/usage/commands/ docs/ all the resources used to build k3d.io using mkdocs pkg/ the place where the magic happens.. here you find all the main logic of k3d all function calls within cmd/ that do non-trivial things are imported from here this (or rather sub-packages) is what other projects would import as a module to work with k3d without using the CLI proxy/ configuration to build the rancher/k3d-proxy container image which is used as a loadbalancer/proxy in front of (almost) every k3d cluster this is basically just a combination of NGINX with confd and some k3d-specific configuration details tests/ a set of bash scripts used for end-to-end (E2E) tests of k3d mostly used for all the functionality of the k3d CLI which cannot be tested using Go unit tests tools/ sub-module used to build the rancher/k3d-tools container image which supports some k3d functionality like k3d image import vendor/ result of go mod vendor , which contains all dependencies of k3d version/ package used to code k3d/k3s versions into releases this is where go build injects the version tags when building k3d that\u2019s the output you see when issuing k3d version Packages Overview \u00b6 pkg/ actions/ hook actions describing actions (commands, etc.) that run at specific stages of the node/cluster lifecycle e.g. writing configuration files to the container filesystem just before the node (container) starts client/ all the top level functionality to work with k3d primitives create/retrieve/update/delete/start/stop clusters, nodes, registries, etc. managed by k3d config/ everything related to the k3d configuration (files), like SimpleConfig and ClusterConfig runtimes/ interface and implementations of runtimes that power k3d (currently, that\u2019s only Docker) functions in client/ eventually call runtime functions to \u201cmaterialize\u201d nodes and clusters tools/ functions eventually calling the k3d-tools container (see tools/ in the repo root) types/ definition of all k3d primitives and many other details and defaults e.g. contains the definition of a Node or a Cluster in k3d util/ some helper functions e.g. for string manipulation/generation, regexp or other re-usable usages Anatomy of a Cluster \u00b6 By default, every k3d cluster consists of at least 2 containers (nodes): (optional, but default and strongly recommended) loadbalancer image: rancher/k3d-proxy , built from proxy/ purpose: proxy and load balance requests from the outside (i.e. most of the times your local host) to the cluster by default, it e.g. proxies all the traffic for the Kubernetes API to port 6443 (default listening port of K3s) to all the server nodes in the cluster can be used for multiple port-mappings to one or more nodes in your cluster that way, port-mappings can also easily be added/removed after the cluster creation, as we can simply re-create the proxy without affecting cluster state (required, always present) primary server node image: rancher/k3s , built from github.com/k3s-io/k3s purpose: (initializing) server (formerly: master) node of the cluster runs the K3s executable (which runs containerd, the Kubernetes API Server, etcd/sqlite, etc.): k3s server in a multi-server setup, it initializes the cluster with an embedded etcd database (using the K3s --cluster-init flag) (optional) secondary server node(s) image: rancher/k3s , built from github.com/k3s-io/k3s (optional) agent node(s) image: rancher/k3s , built from github.com/k3s-io/k3s purpose: running the K3s agent process (kubelet, etc.): k3s agent Automation (CI) \u00b6 The k3d repository mainly leverages the following two CI systems: GitHub Actions 2 workflows in https://github.com/rancher/k3d/tree/main/.github/workflows to push the artifact to AUR (Arch Linux User Repository) logs/history can be seen in the Actions tab: https://github.com/rancher/k3d/actions DroneCI a set of pipelines in a single file: https://github.com/rancher/k3d/blob/main/.drone.yml static code analysis build tests docker builds + pushes render + push docs (pre-) release to GitHub push events end up here (also does the releases, when a tag is pushed): https://drone-publish.rancher.io/rancher/k3d pr s end up here: https://drone-pr.rancher.io/rancher/k3d Documentation \u00b6 The website k3d.io containing all the documentation for k3d is built using mkdocs , configured via the mkdocs.yml config file with all the content residing in the docs/ directory (Markdown). Use mkdocs serve in the repository root to build and serve the webpage locally. Some parts of the documentation are being auto-generated, like docs/usage/commands/ is auto-generated using Cobra\u2019s command docs generation functionality in docgen/ .","title":"Project Overview"},{"location":"design/project/#project-overview","text":"","title":"Project Overview"},{"location":"design/project/#about-this-page","text":"On this page we\u2019ll try to give an overview of all the moving bits and pieces in k3d to ease contributions to the project.","title":"About This Page"},{"location":"design/project/#directory-overview","text":".github/ templates for issues and pull requests GitHub Action workflow definitions cmd/ everything related to the actual k3d CLI, like the whole command tree, config initialization, argument parsing, etc. docgen/ sub-module used to auto-generate the documentation for the CLI commands, which ends up in docs/usage/commands/ docs/ all the resources used to build k3d.io using mkdocs pkg/ the place where the magic happens.. here you find all the main logic of k3d all function calls within cmd/ that do non-trivial things are imported from here this (or rather sub-packages) is what other projects would import as a module to work with k3d without using the CLI proxy/ configuration to build the rancher/k3d-proxy container image which is used as a loadbalancer/proxy in front of (almost) every k3d cluster this is basically just a combination of NGINX with confd and some k3d-specific configuration details tests/ a set of bash scripts used for end-to-end (E2E) tests of k3d mostly used for all the functionality of the k3d CLI which cannot be tested using Go unit tests tools/ sub-module used to build the rancher/k3d-tools container image which supports some k3d functionality like k3d image import vendor/ result of go mod vendor , which contains all dependencies of k3d version/ package used to code k3d/k3s versions into releases this is where go build injects the version tags when building k3d that\u2019s the output you see when issuing k3d version","title":"Directory Overview"},{"location":"design/project/#packages-overview","text":"pkg/ actions/ hook actions describing actions (commands, etc.) that run at specific stages of the node/cluster lifecycle e.g. writing configuration files to the container filesystem just before the node (container) starts client/ all the top level functionality to work with k3d primitives create/retrieve/update/delete/start/stop clusters, nodes, registries, etc. managed by k3d config/ everything related to the k3d configuration (files), like SimpleConfig and ClusterConfig runtimes/ interface and implementations of runtimes that power k3d (currently, that\u2019s only Docker) functions in client/ eventually call runtime functions to \u201cmaterialize\u201d nodes and clusters tools/ functions eventually calling the k3d-tools container (see tools/ in the repo root) types/ definition of all k3d primitives and many other details and defaults e.g. contains the definition of a Node or a Cluster in k3d util/ some helper functions e.g. for string manipulation/generation, regexp or other re-usable usages","title":"Packages Overview"},{"location":"design/project/#anatomy-of-a-cluster","text":"By default, every k3d cluster consists of at least 2 containers (nodes): (optional, but default and strongly recommended) loadbalancer image: rancher/k3d-proxy , built from proxy/ purpose: proxy and load balance requests from the outside (i.e. most of the times your local host) to the cluster by default, it e.g. proxies all the traffic for the Kubernetes API to port 6443 (default listening port of K3s) to all the server nodes in the cluster can be used for multiple port-mappings to one or more nodes in your cluster that way, port-mappings can also easily be added/removed after the cluster creation, as we can simply re-create the proxy without affecting cluster state (required, always present) primary server node image: rancher/k3s , built from github.com/k3s-io/k3s purpose: (initializing) server (formerly: master) node of the cluster runs the K3s executable (which runs containerd, the Kubernetes API Server, etcd/sqlite, etc.): k3s server in a multi-server setup, it initializes the cluster with an embedded etcd database (using the K3s --cluster-init flag) (optional) secondary server node(s) image: rancher/k3s , built from github.com/k3s-io/k3s (optional) agent node(s) image: rancher/k3s , built from github.com/k3s-io/k3s purpose: running the K3s agent process (kubelet, etc.): k3s agent","title":"Anatomy of a Cluster"},{"location":"design/project/#automation-ci","text":"The k3d repository mainly leverages the following two CI systems: GitHub Actions 2 workflows in https://github.com/rancher/k3d/tree/main/.github/workflows to push the artifact to AUR (Arch Linux User Repository) logs/history can be seen in the Actions tab: https://github.com/rancher/k3d/actions DroneCI a set of pipelines in a single file: https://github.com/rancher/k3d/blob/main/.drone.yml static code analysis build tests docker builds + pushes render + push docs (pre-) release to GitHub push events end up here (also does the releases, when a tag is pushed): https://drone-publish.rancher.io/rancher/k3d pr s end up here: https://drone-pr.rancher.io/rancher/k3d","title":"Automation (CI)"},{"location":"design/project/#documentation","text":"The website k3d.io containing all the documentation for k3d is built using mkdocs , configured via the mkdocs.yml config file with all the content residing in the docs/ directory (Markdown). Use mkdocs serve in the repository root to build and serve the webpage locally. Some parts of the documentation are being auto-generated, like docs/usage/commands/ is auto-generated using Cobra\u2019s command docs generation functionality in docgen/ .","title":"Documentation"},{"location":"faq/faq/","text":"FAQ \u00b6 Issues with BTRFS \u00b6 As @jaredallard pointed out , people running k3d on a system with btrfs , may need to mount /dev/mapper into the nodes for the setup to work. This will do: k3d cluster create CLUSTER_NAME -v /dev/mapper:/dev/mapper Issues with ZFS \u00b6 k3s currently has no support for ZFS and thus, creating multi-server setups (e.g. k3d cluster create multiserver --servers 3 ) fails, because the initializing server node (server flag --cluster-init ) errors out with the following log: starting kubernetes: preparing server: start cluster and https: raft_init () : io: create I/O capabilities probe file: posix_allocate: operation not supported on socket This issue can be worked around by providing docker with a different filesystem (that\u2019s also better for docker-in-docker stuff). A possible solution can be found here: https://github.com/rancher/k3s/issues/1688#issuecomment-619570374 Pods evicted due to lack of disk space \u00b6 Pods go to evicted state after doing X Related issues: #133 - Pods evicted due to NodeHasDiskPressure (collection of #119 and #130) Background: somehow docker runs out of space for the k3d node containers, which triggers a hard eviction in the kubelet Possible fix/workaround by @zer0def : use a docker storage driver which cleans up properly (e.g. overlay2) clean up or expand docker root filesystem change the kubelet\u2019s eviction thresholds upon cluster creation: k3d cluster create \\ --k3s-arg '--kubelet-arg=eviction-hard=imagefs.available<1%,nodefs.available<1%@agent:*' \\ --k3s-arg '--kubelet-arg=eviction-minimum-reclaim=imagefs.available=1%,nodefs.available=1%@agent:*' Restarting a multi-server cluster or the initializing server node fails \u00b6 What you do: You create a cluster with more than one server node and later, you either stop server-0 or stop/start the whole cluster What fails: After the restart, you cannot connect to the cluster anymore and kubectl will give you a lot of errors What causes this issue: it\u2019s a known issue with dqlite in k3s which doesn\u2019t allow the initializing server node to go down What\u2019s the solution: Hopefully, this will be solved by the planned replacement of dqlite with embedded etcd in k3s Related issues: #262 Passing additional arguments/flags to k3s (and on to e.g. the kube-apiserver) \u00b6 The Problem: Passing a feature flag to the Kubernetes API Server running inside k3s. Example: you want to enable the EphemeralContainers feature flag in Kubernetes Solution: k3d cluster create --k3s-arg '--kube-apiserver-arg=feature-gates=EphemeralContainers=true@server:*' Note : Be aware of where the flags require dashes ( -- ) and where not. the k3s flag ( --kube-apiserver-arg ) has the dashes the kube-apiserver flag feature-gates doesn\u2019t have them (k3s adds them internally) Second example: k3d cluster create k3d-one \\ --k3s-arg \"--cluster-cidr=10.118.0.0/17@server:*\" \\ --k3s-arg \"--service-cidr=10.118.128.0/17@server:*\" \\ --k3s-arg \"--disable=servicelb@server:*\" \\ --k3s-arg \"--disable=traefik@server:*\" \\ --verbose Note : There are many ways to use the \" and ' quotes, just be aware, that sometimes shells also try to interpret/interpolate parts of the commands How to access services (like a database) running on my Docker Host Machine \u00b6 As of version v3.1.0, we\u2019re injecting the host.k3d.internal entry into the k3d containers (k3s nodes) and into the CoreDNS ConfigMap, enabling you to access your host system by referring to it as host.k3d.internal Running behind a corporate proxy \u00b6 Running k3d behind a corporate proxy can lead to some issues with k3d that have already been reported in more than one issue. Some can be fixed by passing the HTTP_PROXY environment variables to k3d, some have to be fixed in docker\u2019s daemon.json file and some are as easy as adding a volume mount. Pods fail to start: x509: certificate signed by unknown authority \u00b6 Example Error Message: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to get sandbox image \"docker.io/rancher/pause:3.1\" : failed to pull image \"docker.io/rancher/pause:3.1\" : failed to pull and unpack image \"docker.io/rancher/pause:3.1\" : failed to resolve reference \"docker.io/rancher/pause:3.1\" : failed to do request: Head https://registry-1.docker.io/v2/rancher/pause/manifests/3.1: x509: certificate signed by unknown authority Problem: inside the container, the certificate of the corporate proxy cannot be validated Possible Solution: Mounting the CA Certificate from your host into the node containers at start time via k3d cluster create --volume /path/to/your/certs.crt:/etc/ssl/certs/yourcert.crt Issue: rancher/k3d#535 Spurious PID entries in /proc after deleting k3d cluster with shared mounts \u00b6 When you perform cluster create and deletion operations multiple times with same cluster name and shared volume mounts , it was observed that grep k3d /proc/*/mountinfo shows many spurious entries Problem: Due to above, at times you\u2019ll see no space left on device: unknown when a pod is scheduled to the nodes If you observe anything of above sort you can check for inaccessible file systems and unmount them by using below command (note: please remove xargs umount -l and check for the diff o/p first) diff <(df -ha | grep pods | awk '{print $NF}') <(df -h | grep pods | awk '{print $NF}') | awk '{print $2}' | xargs umount -l As per the conversation on rancher/k3d#594 above issue wasn\u2019t reported/known earlier and so there are high chances that it\u2019s not universal. [SOLVED] Nodes fail to start or get stuck in NotReady state with log nf_conntrack_max: permission denied \u00b6 Problem \u00b6 When: This happens when running k3d on a Linux system with a kernel version >= 5.12.2 (and others like >= 5.11.19) when creating a new cluster the node(s) stop or get stuck with a log line like this: <TIMESTAMP> F0516 05:05:31.782902 7 server.go:495] open /proc/sys/net/netfilter/nf_conntrack_max: permission denied Why: The issue was introduced by a change in the Linux kernel ( Changelog 5.12.2 : Commit ), that changed the netfilter_conntrack behavior in a way that kube-proxy is not able to set the nf_conntrack_max value anymore Workaround \u00b6 Workaround: as a workaround, we can tell kube-proxy to not even try to set this value: k3d cluster create \\ --k3s-arg \"--kube-proxy-arg=conntrack-max-per-core=0@server:*\" \\ --k3s-arg \"--kube-proxy-arg=conntrack-max-per-core=0@agent:*\" \\ --image rancher/k3s:v1.20.6-k3s Fix \u00b6 Note : k3d v4.4.5 already uses rancher/k3s:v1.21.1-k3s1 as the new default k3s image, so no workarounds needed there! This is going to be fixed \u201cupstream\u201d in k3s itself in rancher/k3s#3337 and backported to k3s versions as low as v1.18. The fix was released and backported in k3s, so you don\u2019t need to use the workaround when using one of the following k3s versions (or later ones) v1.18.19-k3s1 ( rancher/k3s#3344 ) v1.19.11-k3s1 ( rancher/k3s#3343 ) v1.20.7-k3s1 ( rancher/k3s#3342 ) v1.21.1-k3s1 ( rancher/k3s#3341 )) Issue Reference: rancher/k3s#607 DockerHub Pull Rate Limit \u00b6 Problem \u00b6 You\u2019re deploying something to the cluster using an image from DockerHub and the image fails to be pulled, with a 429 response code and a message saying You have reached your pull rate limit. You may increase the limit by authenticating and upgrading . Cause \u00b6 This is caused by DockerHub\u2019s pull rate limit (see https://docs.docker.com/docker-hub/download-rate-limit/ ), which limits pulls from unauthenticated/anonymous users to 100 pulls per hour and for authenticated users (not paying customers) to 200 pulls per hour (as of the time of writing). Solution \u00b6 a) use images from a private registry, e.g. configured as a pull-through cache for DockerHub b) use a different public registry without such limitations, if the same image is stored there c) authenticate containerd inside k3s/k3d to use your DockerHub user (c) Authenticate Containerd against DockerHub \u00b6 Create a registry configuration file for containerd: # saved as e.g. $HOME/registries.yaml configs : \"docker.io\" : auth : username : \"$USERNAME\" password : \"$PASSWORD\" Create a k3d cluster using that config: k3d cluster create --registry-config $HOME /registries.yaml Profit. That\u2019s it. In the test for this, we pulled the same image 120 times in a row (confirmed, that pull numbers went up), without being rate limited (as a non-paying, normal user)","title":"FAQ"},{"location":"faq/faq/#faq","text":"","title":"FAQ"},{"location":"faq/faq/#issues-with-btrfs","text":"As @jaredallard pointed out , people running k3d on a system with btrfs , may need to mount /dev/mapper into the nodes for the setup to work. This will do: k3d cluster create CLUSTER_NAME -v /dev/mapper:/dev/mapper","title":"Issues with BTRFS"},{"location":"faq/faq/#issues-with-zfs","text":"k3s currently has no support for ZFS and thus, creating multi-server setups (e.g. k3d cluster create multiserver --servers 3 ) fails, because the initializing server node (server flag --cluster-init ) errors out with the following log: starting kubernetes: preparing server: start cluster and https: raft_init () : io: create I/O capabilities probe file: posix_allocate: operation not supported on socket This issue can be worked around by providing docker with a different filesystem (that\u2019s also better for docker-in-docker stuff). A possible solution can be found here: https://github.com/rancher/k3s/issues/1688#issuecomment-619570374","title":"Issues with ZFS"},{"location":"faq/faq/#pods-evicted-due-to-lack-of-disk-space","text":"Pods go to evicted state after doing X Related issues: #133 - Pods evicted due to NodeHasDiskPressure (collection of #119 and #130) Background: somehow docker runs out of space for the k3d node containers, which triggers a hard eviction in the kubelet Possible fix/workaround by @zer0def : use a docker storage driver which cleans up properly (e.g. overlay2) clean up or expand docker root filesystem change the kubelet\u2019s eviction thresholds upon cluster creation: k3d cluster create \\ --k3s-arg '--kubelet-arg=eviction-hard=imagefs.available<1%,nodefs.available<1%@agent:*' \\ --k3s-arg '--kubelet-arg=eviction-minimum-reclaim=imagefs.available=1%,nodefs.available=1%@agent:*'","title":"Pods evicted due to lack of disk space"},{"location":"faq/faq/#restarting-a-multi-server-cluster-or-the-initializing-server-node-fails","text":"What you do: You create a cluster with more than one server node and later, you either stop server-0 or stop/start the whole cluster What fails: After the restart, you cannot connect to the cluster anymore and kubectl will give you a lot of errors What causes this issue: it\u2019s a known issue with dqlite in k3s which doesn\u2019t allow the initializing server node to go down What\u2019s the solution: Hopefully, this will be solved by the planned replacement of dqlite with embedded etcd in k3s Related issues: #262","title":"Restarting a multi-server cluster or the initializing server node fails"},{"location":"faq/faq/#passing-additional-argumentsflags-to-k3s-and-on-to-eg-the-kube-apiserver","text":"The Problem: Passing a feature flag to the Kubernetes API Server running inside k3s. Example: you want to enable the EphemeralContainers feature flag in Kubernetes Solution: k3d cluster create --k3s-arg '--kube-apiserver-arg=feature-gates=EphemeralContainers=true@server:*' Note : Be aware of where the flags require dashes ( -- ) and where not. the k3s flag ( --kube-apiserver-arg ) has the dashes the kube-apiserver flag feature-gates doesn\u2019t have them (k3s adds them internally) Second example: k3d cluster create k3d-one \\ --k3s-arg \"--cluster-cidr=10.118.0.0/17@server:*\" \\ --k3s-arg \"--service-cidr=10.118.128.0/17@server:*\" \\ --k3s-arg \"--disable=servicelb@server:*\" \\ --k3s-arg \"--disable=traefik@server:*\" \\ --verbose Note : There are many ways to use the \" and ' quotes, just be aware, that sometimes shells also try to interpret/interpolate parts of the commands","title":"Passing additional arguments/flags to k3s (and on to e.g. the kube-apiserver)"},{"location":"faq/faq/#how-to-access-services-like-a-database-running-on-my-docker-host-machine","text":"As of version v3.1.0, we\u2019re injecting the host.k3d.internal entry into the k3d containers (k3s nodes) and into the CoreDNS ConfigMap, enabling you to access your host system by referring to it as host.k3d.internal","title":"How to access services (like a database) running on my Docker Host Machine"},{"location":"faq/faq/#running-behind-a-corporate-proxy","text":"Running k3d behind a corporate proxy can lead to some issues with k3d that have already been reported in more than one issue. Some can be fixed by passing the HTTP_PROXY environment variables to k3d, some have to be fixed in docker\u2019s daemon.json file and some are as easy as adding a volume mount.","title":"Running behind a corporate proxy"},{"location":"faq/faq/#pods-fail-to-start-x509-certificate-signed-by-unknown-authority","text":"Example Error Message: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to get sandbox image \"docker.io/rancher/pause:3.1\" : failed to pull image \"docker.io/rancher/pause:3.1\" : failed to pull and unpack image \"docker.io/rancher/pause:3.1\" : failed to resolve reference \"docker.io/rancher/pause:3.1\" : failed to do request: Head https://registry-1.docker.io/v2/rancher/pause/manifests/3.1: x509: certificate signed by unknown authority Problem: inside the container, the certificate of the corporate proxy cannot be validated Possible Solution: Mounting the CA Certificate from your host into the node containers at start time via k3d cluster create --volume /path/to/your/certs.crt:/etc/ssl/certs/yourcert.crt Issue: rancher/k3d#535","title":"Pods fail to start: x509: certificate signed by unknown authority"},{"location":"faq/faq/#spurious-pid-entries-in-proc-after-deleting-k3d-cluster-with-shared-mounts","text":"When you perform cluster create and deletion operations multiple times with same cluster name and shared volume mounts , it was observed that grep k3d /proc/*/mountinfo shows many spurious entries Problem: Due to above, at times you\u2019ll see no space left on device: unknown when a pod is scheduled to the nodes If you observe anything of above sort you can check for inaccessible file systems and unmount them by using below command (note: please remove xargs umount -l and check for the diff o/p first) diff <(df -ha | grep pods | awk '{print $NF}') <(df -h | grep pods | awk '{print $NF}') | awk '{print $2}' | xargs umount -l As per the conversation on rancher/k3d#594 above issue wasn\u2019t reported/known earlier and so there are high chances that it\u2019s not universal.","title":"Spurious PID entries in /proc after deleting k3d cluster with shared mounts"},{"location":"faq/faq/#solved-nodes-fail-to-start-or-get-stuck-in-notready-state-with-log-nf_conntrack_max-permission-denied","text":"","title":"[SOLVED] Nodes fail to start or get stuck in NotReady state with log nf_conntrack_max: permission denied"},{"location":"faq/faq/#problem","text":"When: This happens when running k3d on a Linux system with a kernel version >= 5.12.2 (and others like >= 5.11.19) when creating a new cluster the node(s) stop or get stuck with a log line like this: <TIMESTAMP> F0516 05:05:31.782902 7 server.go:495] open /proc/sys/net/netfilter/nf_conntrack_max: permission denied Why: The issue was introduced by a change in the Linux kernel ( Changelog 5.12.2 : Commit ), that changed the netfilter_conntrack behavior in a way that kube-proxy is not able to set the nf_conntrack_max value anymore","title":"Problem"},{"location":"faq/faq/#workaround","text":"Workaround: as a workaround, we can tell kube-proxy to not even try to set this value: k3d cluster create \\ --k3s-arg \"--kube-proxy-arg=conntrack-max-per-core=0@server:*\" \\ --k3s-arg \"--kube-proxy-arg=conntrack-max-per-core=0@agent:*\" \\ --image rancher/k3s:v1.20.6-k3s","title":"Workaround"},{"location":"faq/faq/#fix","text":"Note : k3d v4.4.5 already uses rancher/k3s:v1.21.1-k3s1 as the new default k3s image, so no workarounds needed there! This is going to be fixed \u201cupstream\u201d in k3s itself in rancher/k3s#3337 and backported to k3s versions as low as v1.18. The fix was released and backported in k3s, so you don\u2019t need to use the workaround when using one of the following k3s versions (or later ones) v1.18.19-k3s1 ( rancher/k3s#3344 ) v1.19.11-k3s1 ( rancher/k3s#3343 ) v1.20.7-k3s1 ( rancher/k3s#3342 ) v1.21.1-k3s1 ( rancher/k3s#3341 )) Issue Reference: rancher/k3s#607","title":"Fix"},{"location":"faq/faq/#dockerhub-pull-rate-limit","text":"","title":"DockerHub Pull Rate Limit"},{"location":"faq/faq/#problem_1","text":"You\u2019re deploying something to the cluster using an image from DockerHub and the image fails to be pulled, with a 429 response code and a message saying You have reached your pull rate limit. You may increase the limit by authenticating and upgrading .","title":"Problem"},{"location":"faq/faq/#cause","text":"This is caused by DockerHub\u2019s pull rate limit (see https://docs.docker.com/docker-hub/download-rate-limit/ ), which limits pulls from unauthenticated/anonymous users to 100 pulls per hour and for authenticated users (not paying customers) to 200 pulls per hour (as of the time of writing).","title":"Cause"},{"location":"faq/faq/#solution","text":"a) use images from a private registry, e.g. configured as a pull-through cache for DockerHub b) use a different public registry without such limitations, if the same image is stored there c) authenticate containerd inside k3s/k3d to use your DockerHub user","title":"Solution"},{"location":"faq/faq/#c-authenticate-containerd-against-dockerhub","text":"Create a registry configuration file for containerd: # saved as e.g. $HOME/registries.yaml configs : \"docker.io\" : auth : username : \"$USERNAME\" password : \"$PASSWORD\" Create a k3d cluster using that config: k3d cluster create --registry-config $HOME /registries.yaml Profit. That\u2019s it. In the test for this, we pulled the same image 120 times in a row (confirmed, that pull numbers went up), without being rate limited (as a non-paying, normal user)","title":"(c) Authenticate Containerd against DockerHub"},{"location":"usage/commands/","text":"Command Tree \u00b6 k3d --verbose # GLOBAL: enable verbose (debug) logging (default: false) --trace # GLOBAL: enable super verbose logging (trace logging) (default: false) --version # show k3d and k3s version -h, --help # GLOBAL: show help text cluster [ CLUSTERNAME ] # default cluster name is 'k3s-default' create -a, --agents # specify how many agent nodes you want to create (integer, default: 0) --agents-memory # specify memory limit for agent containers/nodes (unit, e.g. 1g) --api-port # specify the port on which the cluster will be accessible (format '[HOST:]HOSTPORT', default: random) -c, --config # use a config file (format 'PATH') -e, --env # add environment variables to the nodes (quoted string, format: 'KEY[=VALUE][@NODEFILTER[;NODEFILTER...]]', use flag multiple times) --gpus # [from docker CLI] add GPU devices to the node containers (string, e.g. 'all') -i, --image # specify which k3s image should be used for the nodes (string, default: 'docker.io/rancher/k3s:v1.20.0-k3s2', tag changes per build) --k3s-arg # add additional arguments to the k3s server/agent (quoted string, use flag multiple times) (see https://rancher.com/docs/k3s/latest/en/installation/install-options/server-config/#k3s-server-cli-help & https://rancher.com/docs/k3s/latest/en/installation/install-options/agent-config/#k3s-agent-cli-help) --kubeconfig-switch-context # (implies --kubeconfig-update-default) automatically sets the current-context of your default kubeconfig to the new cluster's context (default: true) --kubeconfig-update-default # enable the automated update of the default kubeconfig with the details of the newly created cluster (also sets '--wait=true') (default: true) -l, --label # add (docker) labels to the node containers (format: 'KEY[=VALUE][@NODEFILTER[;NODEFILTER...]]', use flag multiple times) --network # specify an existing (docker) network you want to connect to (string) --no-hostip # disable the automatic injection of the Host IP as 'host.k3d.internal' into the containers and CoreDNS (default: false) --no-image-volume # disable the creation of a volume for storing images (used for the 'k3d image import' command) (default: false) --no-lb # disable the creation of a load balancer in front of the server nodes (default: false) --no-rollback # disable the automatic rollback actions, if anything goes wrong (default: false) -p, --port # add some more port mappings (format: '[HOST:][HOSTPORT:]CONTAINERPORT[/PROTOCOL][@NODEFILTER]', use flag multiple times) --registry-create # create a new (docker) registry dedicated for this cluster (default: false) --registry-use # use an existing local (docker) registry with this cluster (string, use multiple times) -s, --servers # specify how many server nodes you want to create (integer, default: 1) --servers-memory # specify memory limit for server containers/nodes (unit, e.g. 1g) --token # specify a cluster token (string, default: auto-generated) --timeout # specify a timeout, after which the cluster creation will be interrupted and changes rolled back (duration, e.g. '10s') -v, --volume # specify additional bind-mounts (format: '[SOURCE:]DEST[@NODEFILTER[;NODEFILTER...]]', use flag multiple times) --wait # enable waiting for all server nodes to be ready before returning (default: true) start CLUSTERNAME # start a (stopped) cluster -a, --all # start all clusters (default: false) --wait # wait for all servers and server-loadbalancer to be up before returning (default: true) --timeout # maximum waiting time for '--wait' before canceling/returning (duration, e.g. '10s') stop CLUSTERNAME # stop a cluster -a, --all # stop all clusters (default: false) delete CLUSTERNAME # delete an existing cluster -a, --all # delete all existing clusters (default: false) list [ CLUSTERNAME [ CLUSTERNAME ... ]] --no-headers # do not print headers (default: false) --token # show column with cluster tokens (default: false) -o, --output # format the output (format: 'json|yaml') completion [ bash | zsh | fish | ( psh | powershell )] # generate completion scripts for common shells config init # write a default k3d config (as a starting point) -f, --force # force overwrite target file (default: false) -o, --output # file to write to (string, default \"k3d-default.yaml\") help [ COMMAND ] # show help text for any command image import [ IMAGE | ARCHIVE [ IMAGE | ARCHIVE ... ]] # Load one or more images from the local runtime environment or tar-archives into k3d clusters -c, --cluster # clusters to load the image into (string, use flag multiple times, default: k3s-default) -k, --keep-tarball # do not delete the image tarball from the shared volume after completion (default: false) kubeconfig get ( CLUSTERNAME [ CLUSTERNAME ... ] | --all ) # get kubeconfig from cluster(s) and write it to stdout -a, --all # get kubeconfigs from all clusters (default: false) merge | write ( CLUSTERNAME [ CLUSTERNAME ... ] | --all ) # get kubeconfig from cluster(s) and merge it/them into a (kubeconfig-)file -a, --all # get kubeconfigs from all clusters (default: false) -s, --kubeconfig-switch-context # switch current-context in kubeconfig to the new context (default: true) -d, --kubeconfig-merge-default # update the default kubeconfig (usually $KUBECONFIG or $HOME/.kube/config) -o, --output # specify the output file where the kubeconfig should be written to (string) --overwrite # [Careful!] forcefully overwrite the output file, ignoring existing contents (default: false) -u, --update # update conflicting fields in existing kubeconfig (default: true) node create NODENAME # Create new nodes (and add them to existing clusters) -c, --cluster # specify the cluster that the node shall connect to (string, default: k3s-default) -i, --image # specify which k3s image should be used for the node(s) (string, default: 'docker.io/rancher/k3s:v1.20.0-k3s2', tag changes per build) --replicas # specify how many replicas you want to create with this spec (integer, default: 1) --role # specify the node role (string, format: 'agent|server', default: agent) --timeout # specify a timeout duration, after which the node creation will be interrupted, if not done yet (duration, e.g. '10s') --wait # wait for the node to be up and running before returning (default: true) start NODENAME # start a (stopped) node stop NODENAME # stop a node delete NODENAME # delete an existing node -a, --all # delete all existing nodes (default: false) -r, --registries # also delete registries, as a special type of node (default: false) list NODENAME --no-headers # do not print headers (default: false) registry create REGISTRYNAME -i, --image # specify image used for the registry (string, default: \"docker.io/library/registry:2\") -p, --port # select host port to map to (format: '[HOST:]HOSTPORT', default: 'random') delete REGISTRYNAME -a, --all # delete all existing registries (default: false) list [ NAME [ NAME... ]] --no-headers # disable table headers (default: false) version # show k3d and k3s version","title":"Command Tree"},{"location":"usage/commands/#command-tree","text":"k3d --verbose # GLOBAL: enable verbose (debug) logging (default: false) --trace # GLOBAL: enable super verbose logging (trace logging) (default: false) --version # show k3d and k3s version -h, --help # GLOBAL: show help text cluster [ CLUSTERNAME ] # default cluster name is 'k3s-default' create -a, --agents # specify how many agent nodes you want to create (integer, default: 0) --agents-memory # specify memory limit for agent containers/nodes (unit, e.g. 1g) --api-port # specify the port on which the cluster will be accessible (format '[HOST:]HOSTPORT', default: random) -c, --config # use a config file (format 'PATH') -e, --env # add environment variables to the nodes (quoted string, format: 'KEY[=VALUE][@NODEFILTER[;NODEFILTER...]]', use flag multiple times) --gpus # [from docker CLI] add GPU devices to the node containers (string, e.g. 'all') -i, --image # specify which k3s image should be used for the nodes (string, default: 'docker.io/rancher/k3s:v1.20.0-k3s2', tag changes per build) --k3s-arg # add additional arguments to the k3s server/agent (quoted string, use flag multiple times) (see https://rancher.com/docs/k3s/latest/en/installation/install-options/server-config/#k3s-server-cli-help & https://rancher.com/docs/k3s/latest/en/installation/install-options/agent-config/#k3s-agent-cli-help) --kubeconfig-switch-context # (implies --kubeconfig-update-default) automatically sets the current-context of your default kubeconfig to the new cluster's context (default: true) --kubeconfig-update-default # enable the automated update of the default kubeconfig with the details of the newly created cluster (also sets '--wait=true') (default: true) -l, --label # add (docker) labels to the node containers (format: 'KEY[=VALUE][@NODEFILTER[;NODEFILTER...]]', use flag multiple times) --network # specify an existing (docker) network you want to connect to (string) --no-hostip # disable the automatic injection of the Host IP as 'host.k3d.internal' into the containers and CoreDNS (default: false) --no-image-volume # disable the creation of a volume for storing images (used for the 'k3d image import' command) (default: false) --no-lb # disable the creation of a load balancer in front of the server nodes (default: false) --no-rollback # disable the automatic rollback actions, if anything goes wrong (default: false) -p, --port # add some more port mappings (format: '[HOST:][HOSTPORT:]CONTAINERPORT[/PROTOCOL][@NODEFILTER]', use flag multiple times) --registry-create # create a new (docker) registry dedicated for this cluster (default: false) --registry-use # use an existing local (docker) registry with this cluster (string, use multiple times) -s, --servers # specify how many server nodes you want to create (integer, default: 1) --servers-memory # specify memory limit for server containers/nodes (unit, e.g. 1g) --token # specify a cluster token (string, default: auto-generated) --timeout # specify a timeout, after which the cluster creation will be interrupted and changes rolled back (duration, e.g. '10s') -v, --volume # specify additional bind-mounts (format: '[SOURCE:]DEST[@NODEFILTER[;NODEFILTER...]]', use flag multiple times) --wait # enable waiting for all server nodes to be ready before returning (default: true) start CLUSTERNAME # start a (stopped) cluster -a, --all # start all clusters (default: false) --wait # wait for all servers and server-loadbalancer to be up before returning (default: true) --timeout # maximum waiting time for '--wait' before canceling/returning (duration, e.g. '10s') stop CLUSTERNAME # stop a cluster -a, --all # stop all clusters (default: false) delete CLUSTERNAME # delete an existing cluster -a, --all # delete all existing clusters (default: false) list [ CLUSTERNAME [ CLUSTERNAME ... ]] --no-headers # do not print headers (default: false) --token # show column with cluster tokens (default: false) -o, --output # format the output (format: 'json|yaml') completion [ bash | zsh | fish | ( psh | powershell )] # generate completion scripts for common shells config init # write a default k3d config (as a starting point) -f, --force # force overwrite target file (default: false) -o, --output # file to write to (string, default \"k3d-default.yaml\") help [ COMMAND ] # show help text for any command image import [ IMAGE | ARCHIVE [ IMAGE | ARCHIVE ... ]] # Load one or more images from the local runtime environment or tar-archives into k3d clusters -c, --cluster # clusters to load the image into (string, use flag multiple times, default: k3s-default) -k, --keep-tarball # do not delete the image tarball from the shared volume after completion (default: false) kubeconfig get ( CLUSTERNAME [ CLUSTERNAME ... ] | --all ) # get kubeconfig from cluster(s) and write it to stdout -a, --all # get kubeconfigs from all clusters (default: false) merge | write ( CLUSTERNAME [ CLUSTERNAME ... ] | --all ) # get kubeconfig from cluster(s) and merge it/them into a (kubeconfig-)file -a, --all # get kubeconfigs from all clusters (default: false) -s, --kubeconfig-switch-context # switch current-context in kubeconfig to the new context (default: true) -d, --kubeconfig-merge-default # update the default kubeconfig (usually $KUBECONFIG or $HOME/.kube/config) -o, --output # specify the output file where the kubeconfig should be written to (string) --overwrite # [Careful!] forcefully overwrite the output file, ignoring existing contents (default: false) -u, --update # update conflicting fields in existing kubeconfig (default: true) node create NODENAME # Create new nodes (and add them to existing clusters) -c, --cluster # specify the cluster that the node shall connect to (string, default: k3s-default) -i, --image # specify which k3s image should be used for the node(s) (string, default: 'docker.io/rancher/k3s:v1.20.0-k3s2', tag changes per build) --replicas # specify how many replicas you want to create with this spec (integer, default: 1) --role # specify the node role (string, format: 'agent|server', default: agent) --timeout # specify a timeout duration, after which the node creation will be interrupted, if not done yet (duration, e.g. '10s') --wait # wait for the node to be up and running before returning (default: true) start NODENAME # start a (stopped) node stop NODENAME # stop a node delete NODENAME # delete an existing node -a, --all # delete all existing nodes (default: false) -r, --registries # also delete registries, as a special type of node (default: false) list NODENAME --no-headers # do not print headers (default: false) registry create REGISTRYNAME -i, --image # specify image used for the registry (string, default: \"docker.io/library/registry:2\") -p, --port # select host port to map to (format: '[HOST:]HOSTPORT', default: 'random') delete REGISTRYNAME -a, --all # delete all existing registries (default: false) list [ NAME [ NAME... ]] --no-headers # disable table headers (default: false) version # show k3d and k3s version","title":"Command Tree"},{"location":"usage/configfile/","text":"Using Config Files \u00b6 The config file feature is available as of k3d v4.0.0 Introduction \u00b6 Syntax & Semantics The options defined in the config file are not 100% the same as the CLI flags. This concerns naming and style/usage/structure, e.g. --api-port is split up into a field named kubeAPI that has 3 different \u201cchild fields\u201d ( host , hostIP and hostPort ) k3d options are bundled in a scope named options.k3d , where --no-rollback is defined as options.k3d.disableRollback repeatable flags (like --port ) are reflected as YAML lists Usage \u00b6 Using a config file is as easy as putting it in a well-known place in your file system and then referencing it via flag: All options in config file: k3d cluster create --config /home/me/my-awesome-config.yaml (must be .yaml / .yml ) With CLI override (name): k3d cluster create somename --config /home/me/my-awesome-config.yaml With CLI override (extra volume): k3d cluster create --config /home/me/my-awesome-config.yaml --volume '/some/path:/some:path@server:0' Required Fields \u00b6 As of the time of writing this documentation, the config file only requires you to define two fields: apiVersion to match the version of the config file that you want to use (at this time it would be apiVersion: k3d.io/v1alpha3 ) kind to define the kind of config file that you want to use (currently we only have the Simple config) So this would be the minimal config file, which configures absolutely nothing: apiVersion : k3d.io/v1alpha3 kind : Simple Config Options \u00b6 The configuration options for k3d are continuously evolving and so is the config file (syntax) itself. Currently, the config file is still in an Alpha-State, meaning, that it is subject to change anytime (though we try to keep breaking changes low). Validation via JSON-Schema k3d uses a JSON-Schema to describe the expected format and fields of the configuration file. This schema is also used to validate a user-given config file. This JSON-Schema can be found in the specific config version sub-directory in the repository (e.g. here for v1alpha3 ) and could be used to lookup supported fields or by linters to validate the config file, e.g. in your code editor. All Options: Example \u00b6 Since the config options and the config file are changing quite a bit, it\u2019s hard to keep track of all the supported config file settings, so here\u2019s an example showing all of them as of the time of writing: # k3d configuration file, saved as e.g. /home/me/myk3dcluster.yaml apiVersion : k3d.io/v1alpha3 # this will change in the future as we make everything more stable kind : Simple # internally, we also have a Cluster config, which is not yet available externally name : mycluster # name that you want to give to your cluster (will still be prefixed with `k3d-`) servers : 1 # same as `--servers 1` agents : 2 # same as `--agents 2` kubeAPI : # same as `--api-port myhost.my.domain:6445` (where the name would resolve to 127.0.0.1) host : \"myhost.my.domain\" # important for the `server` setting in the kubeconfig hostIP : \"127.0.0.1\" # where the Kubernetes API will be listening on hostPort : \"6445\" # where the Kubernetes API listening port will be mapped to on your host system image : rancher/k3s:v1.20.4-k3s1 # same as `--image rancher/k3s:v1.20.4-k3s1` network : my-custom-net # same as `--network my-custom-net` subnet : \"172.28.0.0/16\" # same as `--subnet 172.28.0.0/16` token : superSecretToken # same as `--token superSecretToken` volumes : # repeatable flags are represented as YAML lists - volume : /my/host/path:/path/in/node # same as `--volume '/my/host/path:/path/in/node@server:0;agent:*'` nodeFilters : - server:0 - agent:* ports : - port : 8080:80 # same as `--port '8080:80@loadbalancer'` nodeFilters : - loadbalancer env : - envVar : bar=baz # same as `--env 'bar=baz@server:0'` nodeFilters : - server:0 registries : # define how registries should be created or used create : # creates a default registry to be used with the cluster; same as `--registry-create registry.localhost` name : registry.localhost host : \"0.0.0.0\" hostPort : \"5000\" use : - k3d-myotherregistry:5000 # some other k3d-managed registry; same as `--registry-use 'k3d-myotherregistry:5000'` config : | # define contents of the `registries.yaml` file (or reference a file); same as `--registry-config /path/to/config.yaml` mirrors: \"my.company.registry\": endpoint: - http://my.company.registry:5000 options : k3d : # k3d runtime settings wait : true # wait for cluster to be usable before returining; same as `--wait` (default: true) timeout : \"60s\" # wait timeout before aborting; same as `--timeout 60s` disableLoadbalancer : false # same as `--no-lb` disableImageVolume : false # same as `--no-image-volume` disableRollback : false # same as `--no-Rollback` loadbalancer : configOverrides : - settings.workerConnections=2048 k3s : # options passed on to K3s itself extraArgs : # additional arguments passed to the `k3s server|agent` command; same as `--k3s-arg` - arg : --tls-san=my.host.domain nodeFilters : - server:* nodeLabels : - label : foo=bar # same as `--k3s-node-label 'foo=bar@agent:1'` -> this results in a Kubernetes node label nodeFilters : - agent:1 kubeconfig : updateDefaultKubeconfig : true # add new cluster to your default Kubeconfig; same as `--kubeconfig-update-default` (default: true) switchCurrentContext : true # also set current-context to the new cluster's context; same as `--kubeconfig-switch-context` (default: true) runtime : # runtime (docker) specific options gpuRequest : all # same as `--gpus all` labels : - label : bar=baz # same as `--runtime-label 'bar=baz@agent:1'` -> this results in a runtime (docker) container label nodeFilters : - agent:1 Config File vs. CLI Flags \u00b6 k3d uses Cobra and Viper for CLI and general config handling respectively. This automatically introduces a \u201cconfig option order of priority\u201d ( precedence order ): Config Precedence Order Source: spf13/viper#why-viper Internal Setting > CLI Flag > Environment Variable > Config File > (k/v store >) Defaults This means, that you can define e.g. a \u201cbase configuration file\u201d with settings that you share across different clusters and override only the fields that differ between those clusters in your CLI flags/arguments. For example, you use the same config file to create three clusters which only have different names and kubeAPI ( --api-port ) settings. References \u00b6 k3d demo repository: https://github.com/iwilltry42/k3d-demo/blob/main/README.md#config-file-support SUSE Blog: https://www.suse.com/c/introduction-k3d-run-k3s-docker-src/ (Search for The \u201cConfiguration as Code\u201d Way )","title":"Using Config Files"},{"location":"usage/configfile/#using-config-files","text":"The config file feature is available as of k3d v4.0.0","title":"Using Config Files"},{"location":"usage/configfile/#introduction","text":"Syntax & Semantics The options defined in the config file are not 100% the same as the CLI flags. This concerns naming and style/usage/structure, e.g. --api-port is split up into a field named kubeAPI that has 3 different \u201cchild fields\u201d ( host , hostIP and hostPort ) k3d options are bundled in a scope named options.k3d , where --no-rollback is defined as options.k3d.disableRollback repeatable flags (like --port ) are reflected as YAML lists","title":"Introduction"},{"location":"usage/configfile/#usage","text":"Using a config file is as easy as putting it in a well-known place in your file system and then referencing it via flag: All options in config file: k3d cluster create --config /home/me/my-awesome-config.yaml (must be .yaml / .yml ) With CLI override (name): k3d cluster create somename --config /home/me/my-awesome-config.yaml With CLI override (extra volume): k3d cluster create --config /home/me/my-awesome-config.yaml --volume '/some/path:/some:path@server:0'","title":"Usage"},{"location":"usage/configfile/#required-fields","text":"As of the time of writing this documentation, the config file only requires you to define two fields: apiVersion to match the version of the config file that you want to use (at this time it would be apiVersion: k3d.io/v1alpha3 ) kind to define the kind of config file that you want to use (currently we only have the Simple config) So this would be the minimal config file, which configures absolutely nothing: apiVersion : k3d.io/v1alpha3 kind : Simple","title":"Required Fields"},{"location":"usage/configfile/#config-options","text":"The configuration options for k3d are continuously evolving and so is the config file (syntax) itself. Currently, the config file is still in an Alpha-State, meaning, that it is subject to change anytime (though we try to keep breaking changes low). Validation via JSON-Schema k3d uses a JSON-Schema to describe the expected format and fields of the configuration file. This schema is also used to validate a user-given config file. This JSON-Schema can be found in the specific config version sub-directory in the repository (e.g. here for v1alpha3 ) and could be used to lookup supported fields or by linters to validate the config file, e.g. in your code editor.","title":"Config Options"},{"location":"usage/configfile/#all-options-example","text":"Since the config options and the config file are changing quite a bit, it\u2019s hard to keep track of all the supported config file settings, so here\u2019s an example showing all of them as of the time of writing: # k3d configuration file, saved as e.g. /home/me/myk3dcluster.yaml apiVersion : k3d.io/v1alpha3 # this will change in the future as we make everything more stable kind : Simple # internally, we also have a Cluster config, which is not yet available externally name : mycluster # name that you want to give to your cluster (will still be prefixed with `k3d-`) servers : 1 # same as `--servers 1` agents : 2 # same as `--agents 2` kubeAPI : # same as `--api-port myhost.my.domain:6445` (where the name would resolve to 127.0.0.1) host : \"myhost.my.domain\" # important for the `server` setting in the kubeconfig hostIP : \"127.0.0.1\" # where the Kubernetes API will be listening on hostPort : \"6445\" # where the Kubernetes API listening port will be mapped to on your host system image : rancher/k3s:v1.20.4-k3s1 # same as `--image rancher/k3s:v1.20.4-k3s1` network : my-custom-net # same as `--network my-custom-net` subnet : \"172.28.0.0/16\" # same as `--subnet 172.28.0.0/16` token : superSecretToken # same as `--token superSecretToken` volumes : # repeatable flags are represented as YAML lists - volume : /my/host/path:/path/in/node # same as `--volume '/my/host/path:/path/in/node@server:0;agent:*'` nodeFilters : - server:0 - agent:* ports : - port : 8080:80 # same as `--port '8080:80@loadbalancer'` nodeFilters : - loadbalancer env : - envVar : bar=baz # same as `--env 'bar=baz@server:0'` nodeFilters : - server:0 registries : # define how registries should be created or used create : # creates a default registry to be used with the cluster; same as `--registry-create registry.localhost` name : registry.localhost host : \"0.0.0.0\" hostPort : \"5000\" use : - k3d-myotherregistry:5000 # some other k3d-managed registry; same as `--registry-use 'k3d-myotherregistry:5000'` config : | # define contents of the `registries.yaml` file (or reference a file); same as `--registry-config /path/to/config.yaml` mirrors: \"my.company.registry\": endpoint: - http://my.company.registry:5000 options : k3d : # k3d runtime settings wait : true # wait for cluster to be usable before returining; same as `--wait` (default: true) timeout : \"60s\" # wait timeout before aborting; same as `--timeout 60s` disableLoadbalancer : false # same as `--no-lb` disableImageVolume : false # same as `--no-image-volume` disableRollback : false # same as `--no-Rollback` loadbalancer : configOverrides : - settings.workerConnections=2048 k3s : # options passed on to K3s itself extraArgs : # additional arguments passed to the `k3s server|agent` command; same as `--k3s-arg` - arg : --tls-san=my.host.domain nodeFilters : - server:* nodeLabels : - label : foo=bar # same as `--k3s-node-label 'foo=bar@agent:1'` -> this results in a Kubernetes node label nodeFilters : - agent:1 kubeconfig : updateDefaultKubeconfig : true # add new cluster to your default Kubeconfig; same as `--kubeconfig-update-default` (default: true) switchCurrentContext : true # also set current-context to the new cluster's context; same as `--kubeconfig-switch-context` (default: true) runtime : # runtime (docker) specific options gpuRequest : all # same as `--gpus all` labels : - label : bar=baz # same as `--runtime-label 'bar=baz@agent:1'` -> this results in a runtime (docker) container label nodeFilters : - agent:1","title":"All Options: Example"},{"location":"usage/configfile/#config-file-vs-cli-flags","text":"k3d uses Cobra and Viper for CLI and general config handling respectively. This automatically introduces a \u201cconfig option order of priority\u201d ( precedence order ): Config Precedence Order Source: spf13/viper#why-viper Internal Setting > CLI Flag > Environment Variable > Config File > (k/v store >) Defaults This means, that you can define e.g. a \u201cbase configuration file\u201d with settings that you share across different clusters and override only the fields that differ between those clusters in your CLI flags/arguments. For example, you use the same config file to create three clusters which only have different names and kubeAPI ( --api-port ) settings.","title":"Config File vs. CLI Flags"},{"location":"usage/configfile/#references","text":"k3d demo repository: https://github.com/iwilltry42/k3d-demo/blob/main/README.md#config-file-support SUSE Blog: https://www.suse.com/c/introduction-k3d-run-k3s-docker-src/ (Search for The \u201cConfiguration as Code\u201d Way )","title":"References"},{"location":"usage/exposing_services/","text":"Exposing Services \u00b6 1. via Ingress (recommended) \u00b6 In this example, we will deploy a simple nginx webserver deployment and make it accessible via ingress. Therefore, we have to create the cluster in a way, that the internal port 80 (where the traefik ingress controller is listening on) is exposed on the host system. Create a cluster, mapping the ingress port 80 to localhost:8081 k3d cluster create --api-port 6550 -p \"8081:80@loadbalancer\" --agents 2 Good to know --api-port 6550 is not required for the example to work. It\u2019s used to have k3s \u2018s API-Server listening on port 6550 with that port mapped to the host system. the port-mapping construct 8081:80@loadbalancer means: \u201cmap port 8081 from the host to port 80 on the container which matches the nodefilter loadbalancer \u201c the loadbalancer nodefilter matches only the serverlb that\u2019s deployed in front of a cluster\u2019s server nodes all ports exposed on the serverlb will be proxied to the same ports on all server nodes in the cluster Get the kubeconfig file (redundant, as k3d cluster create already merges it into your default kubeconfig file) export KUBECONFIG = \" $( k3d kubeconfig write k3s-default ) \" Create a nginx deployment kubectl create deployment nginx --image = nginx Create a ClusterIP service for it kubectl create service clusterip nginx --tcp = 80 :80 Create an ingress object for it by copying the following manifest to a file and applying with kubectl apply -f thatfile.yaml Note : k3s deploys traefik as the default ingress controller # apiVersion: networking.k8s.io/v1beta1 # for k3s < v1.19 apiVersion : networking.k8s.io/v1 kind : Ingress metadata : name : nginx annotations : ingress.kubernetes.io/ssl-redirect : \"false\" spec : rules : - http : paths : - path : / pathType : Prefix backend : service : name : nginx port : number : 80 Curl it via localhost curl localhost:8081/ 2. via NodePort \u00b6 Create a cluster, mapping the port 30080 from agent-0 to localhost:8082 k3d cluster create mycluster -p \"8082:30080@agent:0\" --agents 2 Note 1 : Kubernetes\u2019 default NodePort range is 30000-32767 Note 2 : You may as well expose the whole NodePort range from the very beginning, e.g. via k3d cluster create mycluster --agents 3 -p \"30000-32767:30000-32767@server:0\" (See this video from @portainer ) Warning : Docker creates iptable entries and a new proxy process per port-mapping, so this may take a very long time or even freeze your system! \u2026 (Steps 2 and 3 like above) \u2026 Create a NodePort service for it by copying the following manifest to a file and applying it with kubectl apply -f apiVersion : v1 kind : Service metadata : labels : app : nginx name : nginx spec : ports : - name : 80-80 nodePort : 30080 port : 80 protocol : TCP targetPort : 80 selector : app : nginx type : NodePort Curl it via localhost curl localhost:8082/","title":"Exposing Services"},{"location":"usage/exposing_services/#exposing-services","text":"","title":"Exposing Services"},{"location":"usage/exposing_services/#1-via-ingress-recommended","text":"In this example, we will deploy a simple nginx webserver deployment and make it accessible via ingress. Therefore, we have to create the cluster in a way, that the internal port 80 (where the traefik ingress controller is listening on) is exposed on the host system. Create a cluster, mapping the ingress port 80 to localhost:8081 k3d cluster create --api-port 6550 -p \"8081:80@loadbalancer\" --agents 2 Good to know --api-port 6550 is not required for the example to work. It\u2019s used to have k3s \u2018s API-Server listening on port 6550 with that port mapped to the host system. the port-mapping construct 8081:80@loadbalancer means: \u201cmap port 8081 from the host to port 80 on the container which matches the nodefilter loadbalancer \u201c the loadbalancer nodefilter matches only the serverlb that\u2019s deployed in front of a cluster\u2019s server nodes all ports exposed on the serverlb will be proxied to the same ports on all server nodes in the cluster Get the kubeconfig file (redundant, as k3d cluster create already merges it into your default kubeconfig file) export KUBECONFIG = \" $( k3d kubeconfig write k3s-default ) \" Create a nginx deployment kubectl create deployment nginx --image = nginx Create a ClusterIP service for it kubectl create service clusterip nginx --tcp = 80 :80 Create an ingress object for it by copying the following manifest to a file and applying with kubectl apply -f thatfile.yaml Note : k3s deploys traefik as the default ingress controller # apiVersion: networking.k8s.io/v1beta1 # for k3s < v1.19 apiVersion : networking.k8s.io/v1 kind : Ingress metadata : name : nginx annotations : ingress.kubernetes.io/ssl-redirect : \"false\" spec : rules : - http : paths : - path : / pathType : Prefix backend : service : name : nginx port : number : 80 Curl it via localhost curl localhost:8081/","title":"1. via Ingress (recommended)"},{"location":"usage/exposing_services/#2-via-nodeport","text":"Create a cluster, mapping the port 30080 from agent-0 to localhost:8082 k3d cluster create mycluster -p \"8082:30080@agent:0\" --agents 2 Note 1 : Kubernetes\u2019 default NodePort range is 30000-32767 Note 2 : You may as well expose the whole NodePort range from the very beginning, e.g. via k3d cluster create mycluster --agents 3 -p \"30000-32767:30000-32767@server:0\" (See this video from @portainer ) Warning : Docker creates iptable entries and a new proxy process per port-mapping, so this may take a very long time or even freeze your system! \u2026 (Steps 2 and 3 like above) \u2026 Create a NodePort service for it by copying the following manifest to a file and applying it with kubectl apply -f apiVersion : v1 kind : Service metadata : labels : app : nginx name : nginx spec : ports : - name : 80-80 nodePort : 30080 port : 80 protocol : TCP targetPort : 80 selector : app : nginx type : NodePort Curl it via localhost curl localhost:8082/","title":"2. via NodePort"},{"location":"usage/kubeconfig/","text":"Handling Kubeconfigs \u00b6 By default, k3d will update your default kubeconfig with your new cluster\u2019s details and set the current-context to it (can be disabled). To get a kubeconfig set up for you to connect to a k3d cluster without this automatism, you can go different ways. What is the default kubeconfig? We determine the path of the used or default kubeconfig in two ways: Using the KUBECONFIG environment variable, if it specifies exactly one file Using the default path (e.g. on Linux it\u2019s $HOME /.kube/config ) Getting the kubeconfig for a newly created cluster \u00b6 Create a new kubeconfig file after cluster creation k3d kubeconfig write mycluster Note: this will create (or update) the file $HOME/.k3d/kubeconfig-mycluster.yaml Tip: Use it: export KUBECONFIG = $( k3d kubeconfig write mycluster ) Note 2 : alternatively you can use k3d kubeconfig get mycluster > some-file.yaml Update your default kubeconfig upon cluster creation (DEFAULT) k3d cluster create mycluster --kubeconfig-update-default Note: this won\u2019t switch the current-context (append --kubeconfig-switch-context to do so) Update your default kubeconfig after cluster creation k3d kubeconfig merge mycluster --kubeconfig-merge-default Note: this won\u2019t switch the current-context (append --kubeconfig-switch-context to do so) Update a different kubeconfig after cluster creation k3d kubeconfig merge mycluster --output some/other/file.yaml Note: this won\u2019t switch the current-context The file will be created if it doesn\u2019t exist Switching the current context None of the above options switch the current-context by default. This is intended to be least intrusive, since the current-context has a global effect. You can switch the current-context directly with the kubeconfig merge command by adding the --kubeconfig-switch-context flag. Removing cluster details from the kubeconfig \u00b6 k3d cluster delete mycluster will always remove the details for mycluster from the default kubeconfig. It will also delete the respective kubeconfig file in $HOME/.k3d/ if it exists. Handling multiple clusters \u00b6 k3d kubeconfig merge let\u2019s you specify one or more clusters via arguments or all via --all . All kubeconfigs will then be merged into a single file if --kubeconfig-merge-default or --output is specified. If none of those two flags was specified, a new file will be created per cluster and the merged path (e.g. $HOME/.k3d/kubeconfig-cluster1.yaml:$HOME/.k3d/cluster2.yaml ) will be returned. Note, that with multiple cluster specified, the --kubeconfig-switch-context flag will change the current context to the cluster which was last in the list.","title":"Handling Kubeconfigs"},{"location":"usage/kubeconfig/#handling-kubeconfigs","text":"By default, k3d will update your default kubeconfig with your new cluster\u2019s details and set the current-context to it (can be disabled). To get a kubeconfig set up for you to connect to a k3d cluster without this automatism, you can go different ways. What is the default kubeconfig? We determine the path of the used or default kubeconfig in two ways: Using the KUBECONFIG environment variable, if it specifies exactly one file Using the default path (e.g. on Linux it\u2019s $HOME /.kube/config )","title":"Handling Kubeconfigs"},{"location":"usage/kubeconfig/#getting-the-kubeconfig-for-a-newly-created-cluster","text":"Create a new kubeconfig file after cluster creation k3d kubeconfig write mycluster Note: this will create (or update) the file $HOME/.k3d/kubeconfig-mycluster.yaml Tip: Use it: export KUBECONFIG = $( k3d kubeconfig write mycluster ) Note 2 : alternatively you can use k3d kubeconfig get mycluster > some-file.yaml Update your default kubeconfig upon cluster creation (DEFAULT) k3d cluster create mycluster --kubeconfig-update-default Note: this won\u2019t switch the current-context (append --kubeconfig-switch-context to do so) Update your default kubeconfig after cluster creation k3d kubeconfig merge mycluster --kubeconfig-merge-default Note: this won\u2019t switch the current-context (append --kubeconfig-switch-context to do so) Update a different kubeconfig after cluster creation k3d kubeconfig merge mycluster --output some/other/file.yaml Note: this won\u2019t switch the current-context The file will be created if it doesn\u2019t exist Switching the current context None of the above options switch the current-context by default. This is intended to be least intrusive, since the current-context has a global effect. You can switch the current-context directly with the kubeconfig merge command by adding the --kubeconfig-switch-context flag.","title":"Getting the kubeconfig for a newly created cluster"},{"location":"usage/kubeconfig/#removing-cluster-details-from-the-kubeconfig","text":"k3d cluster delete mycluster will always remove the details for mycluster from the default kubeconfig. It will also delete the respective kubeconfig file in $HOME/.k3d/ if it exists.","title":"Removing cluster details from the kubeconfig"},{"location":"usage/kubeconfig/#handling-multiple-clusters","text":"k3d kubeconfig merge let\u2019s you specify one or more clusters via arguments or all via --all . All kubeconfigs will then be merged into a single file if --kubeconfig-merge-default or --output is specified. If none of those two flags was specified, a new file will be created per cluster and the merged path (e.g. $HOME/.k3d/kubeconfig-cluster1.yaml:$HOME/.k3d/cluster2.yaml ) will be returned. Note, that with multiple cluster specified, the --kubeconfig-switch-context flag will change the current context to the cluster which was last in the list.","title":"Handling multiple clusters"},{"location":"usage/multiserver/","text":"Creating multi-server clusters \u00b6 Important note For the best results (and less unexpected issues), choose 1, 3, 5, \u2026 server nodes. (Read more on etcd quorum on etcd.io ) At least 2 cores and 4GiB of RAM are recommended. Embedded etcd \u00b6 Create a cluster with 3 server nodes using k3s\u2019 embedded etcd database. The first server to be created will use the --cluster-init flag and k3d will wait for it to be up and running before creating (and connecting) the other server nodes. k3d cluster create multiserver --servers 3 Adding server nodes to a running cluster \u00b6 In theory (and also in practice in most cases), this is as easy as executing the following command: k3d node create newserver --cluster multiserver --role server There\u2019s a trap! If your cluster was initially created with only a single server node, then this will fail. That\u2019s because the initial server node was not started with the --cluster-init flag and thus is not using the etcd backend.","title":"Creating multi-server clusters"},{"location":"usage/multiserver/#creating-multi-server-clusters","text":"Important note For the best results (and less unexpected issues), choose 1, 3, 5, \u2026 server nodes. (Read more on etcd quorum on etcd.io ) At least 2 cores and 4GiB of RAM are recommended.","title":"Creating multi-server clusters"},{"location":"usage/multiserver/#embedded-etcd","text":"Create a cluster with 3 server nodes using k3s\u2019 embedded etcd database. The first server to be created will use the --cluster-init flag and k3d will wait for it to be up and running before creating (and connecting) the other server nodes. k3d cluster create multiserver --servers 3","title":"Embedded etcd"},{"location":"usage/multiserver/#adding-server-nodes-to-a-running-cluster","text":"In theory (and also in practice in most cases), this is as easy as executing the following command: k3d node create newserver --cluster multiserver --role server There\u2019s a trap! If your cluster was initially created with only a single server node, then this will fail. That\u2019s because the initial server node was not started with the --cluster-init flag and thus is not using the etcd backend.","title":"Adding server nodes to a running cluster"},{"location":"usage/registries/","text":"Using Image Registries \u00b6 Registries configuration file \u00b6 You can add registries by specifying them in a registries.yaml and referencing it at creation time: k3d cluster create mycluster --registry-config \"/home/YOU/my-registries.yaml\" . This file is a regular k3s registries configuration file , and looks like this: mirrors : \"my.company.registry:5000\" : endpoint : - http://my.company.registry:5000 In this example, an image with a name like my.company.registry:5000/nginx:latest would be pulled from the registry running at http://my.company.registry:5000 . This file can also be used for providing additional information necessary for accessing some registries, like authentication and certificates . Registries Configuration File embedded in k3d\u2019s SimpleConfig \u00b6 If you\u2019re using a SimpleConfig file to configure your k3d cluster, you may as well embed the registries.yaml in there directly: apiVersion : k3d.io/v1alpha3 kind : Simple name : test servers : 1 agents : 2 registries : create : name : myregistry config : | mirrors: \"my.company.registry\": endpoint: - http://my.company.registry:5000 Here, the config for the k3d-managed registry, created by the create: {...} option will be merged with the config specified under config: | . Authenticated registries \u00b6 When using authenticated registries, we can add the username and password in a configs section in the registries.yaml , like this: mirrors : my.company.registry : endpoint : - http://my.company.registry configs : my.company.registry : auth : username : aladin password : abracadabra Secure registries \u00b6 When using secure registries, the registries.yaml file must include information about the certificates. For example, if you want to use images from the secure registry running at https://my.company.registry , you must first download a CA file valid for that server and store it in some well-known directory like ${HOME}/.k3d/my-company-root.pem . Then you have to mount the CA file in some directory in the nodes in the cluster and include that mounted file in a configs section in the registries.yaml file . For example, if we mount the CA file in /etc/ssl/certs/my-company-root.pem , the registries.yaml will look like: mirrors : my.company.registry : endpoint : - https://my.company.registry configs : my.company.registry : tls : # we will mount \"my-company-root.pem\" in the /etc/ssl/certs/ directory. ca_file : \"/etc/ssl/certs/my-company-root.pem\" Finally, we can create the cluster, mounting the CA file in the path we specified in ca_file : k3d cluster create \\ --volume \" ${ HOME } /.k3d/my-registries.yaml:/etc/rancher/k3s/registries.yaml\" \\ --volume \" ${ HOME } /.k3d/my-company-root.pem:/etc/ssl/certs/my-company-root.pem\" Using a local registry \u00b6 Using k3d-managed registries \u00b6 Create a dedicated registry together with your cluster \u00b6 k3d cluster create mycluster --registry-create mycluster-registry : This creates your cluster mycluster together with a registry container called mycluster-registry k3d sets everything up in the cluster for containerd to be able to pull images from that registry (using the registries.yaml file) the port, which the registry is listening on will be mapped to a random port on your host system Check the k3d command output or docker ps -f name = mycluster-registry to find the exposed port (let\u2019s use 12345 here) Pull some image (optional) docker pull alpine:latest , re-tag it to reference your newly created registry docker tag alpine:latest mycluster-registry:12345/testimage:local and push it docker push mycluster-registry:12345/testimage:local Use kubectl to create a new pod in your cluster using that image to see, if the cluster can pull from the new registry: kubectl run --image mycluster-registry:12345/testimage:local testimage --command -- tail -f /dev/null (creates a container that will not do anything but keep on running) Create a customized k3d-managed registry \u00b6 k3d registry create myregistry.localhost --port 12345 creates a new registry called k3d-myregistry.localhost (could be used with automatic resolution of *.localhost , see next section - also, note the k3d- prefix that k3d adds to all resources it creates) k3d cluster create newcluster --registry-use k3d-myregistry.localhost:12345 (make sure you use the k3d- prefix here) creates a new cluster set up to use that registry continue with step 3 and 4 from the last section for testing Using your own (not k3d-managed) local registry \u00b6 We recommend using a k3d-managed registry, as it plays nicely together with k3d clusters, but here\u2019s also a guide to create your own (not k3d-managed) registry, if you need features or customizations, that k3d does not provide: Using your own (not k3d-managed) local registry You can start your own local registry it with some docker commands, like: docker volume create local_registry docker container run -d --name registry.localhost -v local_registry:/var/lib/registry --restart always -p 5000 :5000 registry:2 These commands will start your registry container with name and port registry.localhost:5000 . In order to push to this registry, you will need to make it accessible as described in the next section. Once your registry is up and running, we will need to add it to your registries.yaml configuration file. Finally, you have to connect the registry network to the k3d cluster network: docker network connect k3d-k3s-default registry.localhost . And then you can test your local registry . Pushing to your local registry address \u00b6 As per the guide above, the registry will be available as registry.localhost:5000 . All the nodes in your k3d cluster can resolve this hostname (thanks to the DNS server provided by the Docker daemon) but, in order to be able to push to this registry, this hostname also has to be resolved by your host. nss-myhostname to resolve *.localhost Luckily (for Linux users), NSS-myhostname ships with many Linux distributions and should resolve *.localhost automatically to 127.0.0.1 . Otherwise, it\u2019s installable using sudo apt install libnss-myhostname . If your system does not provide/support tools that can auto-resolve specific names to 127.0.0.1 , you can manually add an entry in your /etc/hosts ( c:\\windows\\system32\\drivers\\etc\\hosts on Windows) file like this: 127 .0.0.1 k3d-registry.localhost Just use localhost Alternatively, if you don\u2019t care about pretty names, just push directly to localhost:5000 (or whatever port you used) and it will work. If you later pull the image from the registry, only the repository path (e.g. myrepo/myimage:mytag in registry.localhost:5000/myrepo/myimage:mytag ) matters to find your image in the targeted registry. Testing your registry \u00b6 You should test that you can push to your registry from your local development machine. use images from that registry in Deployments in your k3d cluster. We will verify these two things for a local registry (located at k3d-registry.localhost:12345 ) running in your development machine. Things would be basically the same for checking an external registry, but some additional configuration could be necessary in your local machine when using an authenticated or secure registry (please refer to Docker\u2019s documentation for this). First, we can download some image (like nginx ) and push it to our local registry with: docker pull nginx:latest docker tag nginx:latest k3d-registry.localhost:5000/nginx:latest docker push k3d-registry.localhost:5000/nginx:latest Then we can deploy a pod referencing this image to your cluster: cat <<EOF | kubectl apply -f - apiVersion: apps/v1 kind: Deployment metadata: name: nginx-test-registry labels: app: nginx-test-registry spec: replicas: 1 selector: matchLabels: app: nginx-test-registry template: metadata: labels: app: nginx-test-registry spec: containers: - name: nginx-test-registry image: k3d-registry.localhost:12345/nginx:latest ports: - containerPort: 80 EOF Then you should check that the pod is running with kubectl get pods -l \"app=nginx-test-registry\" .","title":"Using Image Registries"},{"location":"usage/registries/#using-image-registries","text":"","title":"Using Image Registries"},{"location":"usage/registries/#registries-configuration-file","text":"You can add registries by specifying them in a registries.yaml and referencing it at creation time: k3d cluster create mycluster --registry-config \"/home/YOU/my-registries.yaml\" . This file is a regular k3s registries configuration file , and looks like this: mirrors : \"my.company.registry:5000\" : endpoint : - http://my.company.registry:5000 In this example, an image with a name like my.company.registry:5000/nginx:latest would be pulled from the registry running at http://my.company.registry:5000 . This file can also be used for providing additional information necessary for accessing some registries, like authentication and certificates .","title":"Registries configuration file"},{"location":"usage/registries/#registries-configuration-file-embedded-in-k3ds-simpleconfig","text":"If you\u2019re using a SimpleConfig file to configure your k3d cluster, you may as well embed the registries.yaml in there directly: apiVersion : k3d.io/v1alpha3 kind : Simple name : test servers : 1 agents : 2 registries : create : name : myregistry config : | mirrors: \"my.company.registry\": endpoint: - http://my.company.registry:5000 Here, the config for the k3d-managed registry, created by the create: {...} option will be merged with the config specified under config: | .","title":"Registries Configuration File embedded in k3d's SimpleConfig"},{"location":"usage/registries/#authenticated-registries","text":"When using authenticated registries, we can add the username and password in a configs section in the registries.yaml , like this: mirrors : my.company.registry : endpoint : - http://my.company.registry configs : my.company.registry : auth : username : aladin password : abracadabra","title":"Authenticated registries"},{"location":"usage/registries/#secure-registries","text":"When using secure registries, the registries.yaml file must include information about the certificates. For example, if you want to use images from the secure registry running at https://my.company.registry , you must first download a CA file valid for that server and store it in some well-known directory like ${HOME}/.k3d/my-company-root.pem . Then you have to mount the CA file in some directory in the nodes in the cluster and include that mounted file in a configs section in the registries.yaml file . For example, if we mount the CA file in /etc/ssl/certs/my-company-root.pem , the registries.yaml will look like: mirrors : my.company.registry : endpoint : - https://my.company.registry configs : my.company.registry : tls : # we will mount \"my-company-root.pem\" in the /etc/ssl/certs/ directory. ca_file : \"/etc/ssl/certs/my-company-root.pem\" Finally, we can create the cluster, mounting the CA file in the path we specified in ca_file : k3d cluster create \\ --volume \" ${ HOME } /.k3d/my-registries.yaml:/etc/rancher/k3s/registries.yaml\" \\ --volume \" ${ HOME } /.k3d/my-company-root.pem:/etc/ssl/certs/my-company-root.pem\"","title":"Secure registries"},{"location":"usage/registries/#using-a-local-registry","text":"","title":"Using a local registry"},{"location":"usage/registries/#using-k3d-managed-registries","text":"","title":"Using k3d-managed registries"},{"location":"usage/registries/#create-a-dedicated-registry-together-with-your-cluster","text":"k3d cluster create mycluster --registry-create mycluster-registry : This creates your cluster mycluster together with a registry container called mycluster-registry k3d sets everything up in the cluster for containerd to be able to pull images from that registry (using the registries.yaml file) the port, which the registry is listening on will be mapped to a random port on your host system Check the k3d command output or docker ps -f name = mycluster-registry to find the exposed port (let\u2019s use 12345 here) Pull some image (optional) docker pull alpine:latest , re-tag it to reference your newly created registry docker tag alpine:latest mycluster-registry:12345/testimage:local and push it docker push mycluster-registry:12345/testimage:local Use kubectl to create a new pod in your cluster using that image to see, if the cluster can pull from the new registry: kubectl run --image mycluster-registry:12345/testimage:local testimage --command -- tail -f /dev/null (creates a container that will not do anything but keep on running)","title":"Create a dedicated registry together with your cluster"},{"location":"usage/registries/#create-a-customized-k3d-managed-registry","text":"k3d registry create myregistry.localhost --port 12345 creates a new registry called k3d-myregistry.localhost (could be used with automatic resolution of *.localhost , see next section - also, note the k3d- prefix that k3d adds to all resources it creates) k3d cluster create newcluster --registry-use k3d-myregistry.localhost:12345 (make sure you use the k3d- prefix here) creates a new cluster set up to use that registry continue with step 3 and 4 from the last section for testing","title":"Create a customized k3d-managed registry"},{"location":"usage/registries/#using-your-own-not-k3d-managed-local-registry","text":"We recommend using a k3d-managed registry, as it plays nicely together with k3d clusters, but here\u2019s also a guide to create your own (not k3d-managed) registry, if you need features or customizations, that k3d does not provide: Using your own (not k3d-managed) local registry You can start your own local registry it with some docker commands, like: docker volume create local_registry docker container run -d --name registry.localhost -v local_registry:/var/lib/registry --restart always -p 5000 :5000 registry:2 These commands will start your registry container with name and port registry.localhost:5000 . In order to push to this registry, you will need to make it accessible as described in the next section. Once your registry is up and running, we will need to add it to your registries.yaml configuration file. Finally, you have to connect the registry network to the k3d cluster network: docker network connect k3d-k3s-default registry.localhost . And then you can test your local registry .","title":"Using your own (not k3d-managed) local registry"},{"location":"usage/registries/#pushing-to-your-local-registry-address","text":"As per the guide above, the registry will be available as registry.localhost:5000 . All the nodes in your k3d cluster can resolve this hostname (thanks to the DNS server provided by the Docker daemon) but, in order to be able to push to this registry, this hostname also has to be resolved by your host. nss-myhostname to resolve *.localhost Luckily (for Linux users), NSS-myhostname ships with many Linux distributions and should resolve *.localhost automatically to 127.0.0.1 . Otherwise, it\u2019s installable using sudo apt install libnss-myhostname . If your system does not provide/support tools that can auto-resolve specific names to 127.0.0.1 , you can manually add an entry in your /etc/hosts ( c:\\windows\\system32\\drivers\\etc\\hosts on Windows) file like this: 127 .0.0.1 k3d-registry.localhost Just use localhost Alternatively, if you don\u2019t care about pretty names, just push directly to localhost:5000 (or whatever port you used) and it will work. If you later pull the image from the registry, only the repository path (e.g. myrepo/myimage:mytag in registry.localhost:5000/myrepo/myimage:mytag ) matters to find your image in the targeted registry.","title":"Pushing to your local registry address"},{"location":"usage/registries/#testing-your-registry","text":"You should test that you can push to your registry from your local development machine. use images from that registry in Deployments in your k3d cluster. We will verify these two things for a local registry (located at k3d-registry.localhost:12345 ) running in your development machine. Things would be basically the same for checking an external registry, but some additional configuration could be necessary in your local machine when using an authenticated or secure registry (please refer to Docker\u2019s documentation for this). First, we can download some image (like nginx ) and push it to our local registry with: docker pull nginx:latest docker tag nginx:latest k3d-registry.localhost:5000/nginx:latest docker push k3d-registry.localhost:5000/nginx:latest Then we can deploy a pod referencing this image to your cluster: cat <<EOF | kubectl apply -f - apiVersion: apps/v1 kind: Deployment metadata: name: nginx-test-registry labels: app: nginx-test-registry spec: replicas: 1 selector: matchLabels: app: nginx-test-registry template: metadata: labels: app: nginx-test-registry spec: containers: - name: nginx-test-registry image: k3d-registry.localhost:12345/nginx:latest ports: - containerPort: 80 EOF Then you should check that the pod is running with kubectl get pods -l \"app=nginx-test-registry\" .","title":"Testing your registry"},{"location":"usage/advanced/calico/","text":"Use Calico instead of Flannel \u00b6 If you want to use NetworkPolicy you can use Calico in k3s instead of Flannel. 1. Download and modify the Calico descriptor \u00b6 You can following the documentation And then you have to change the ConfigMap calico-config . On the cni_network_config add the entry for allowing IP forwarding \"container_settings\" : { \"allow_ip_forwarding\" : true } Or you can directly use this calico.yaml manifest 2. Create the cluster without flannel and with calico \u00b6 On the k3s cluster creation : add the flag --flannel-backend=none . For this, on k3d you need to forward this flag to k3s with the option --k3s-arg . mount ( --volume ) the calico descriptor in the auto deploy manifest directory of k3s /var/lib/rancher/k3s/server/manifests/ So the command of the cluster creation is (when you are at root of the k3d repository) k3d cluster create \" ${ clustername } \" \\ --k3s-arg '--flannel-backend=none@server:*' \\ --volume \" $( pwd ) /docs/usage/guides/calico.yaml:/var/lib/rancher/k3s/server/manifests/calico.yaml\" In this example : change \"${clustername}\" with the name of the cluster (or set a variable). $(pwd)/docs/usage/guides/calico.yaml is the absolute path of the calico manifest, you can adapt it. You can add other options, see . The cluster will start without flannel and with Calico as CNI Plugin. For watching for the pod(s) deployment watch \"kubectl get pods -n kube-system\" You will have something like this at beginning (with the command line kubectl get pods -n kube-system ) NAME READY STATUS RESTARTS AGE helm-install-traefik-pn84f 0 /1 Pending 0 3s calico-node-97rx8 0 /1 Init:0/3 0 3s metrics-server-7566d596c8-hwnqq 0 /1 Pending 0 2s calico-kube-controllers-58b656d69f-2z7cn 0 /1 Pending 0 2s local-path-provisioner-6d59f47c7-rmswg 0 /1 Pending 0 2s coredns-8655855d6-cxtnr 0 /1 Pending 0 2s And when it finish to start NAME READY STATUS RESTARTS AGE metrics-server-7566d596c8-hwnqq 1 /1 Running 0 56s calico-node-97rx8 1 /1 Running 0 57s helm-install-traefik-pn84f 0 /1 Completed 1 57s svclb-traefik-lmjr5 2 /2 Running 0 28s calico-kube-controllers-58b656d69f-2z7cn 1 /1 Running 0 56s local-path-provisioner-6d59f47c7-rmswg 1 /1 Running 0 56s traefik-758cd5fc85-x8p57 1 /1 Running 0 28s coredns-8655855d6-cxtnr 1 /1 Running 0 56s Note : you can use the auto deploy manifest or a kubectl apply depending on your needs Calico is not as quick as Flannel (but it provides more features) References \u00b6 https://rancher.com/docs/k3s/latest/en/installation/network-options/ https://docs.projectcalico.org/getting-started/kubernetes/k3s/","title":"Use Calico instead of Flannel"},{"location":"usage/advanced/calico/#use-calico-instead-of-flannel","text":"If you want to use NetworkPolicy you can use Calico in k3s instead of Flannel.","title":"Use Calico instead of Flannel"},{"location":"usage/advanced/calico/#1-download-and-modify-the-calico-descriptor","text":"You can following the documentation And then you have to change the ConfigMap calico-config . On the cni_network_config add the entry for allowing IP forwarding \"container_settings\" : { \"allow_ip_forwarding\" : true } Or you can directly use this calico.yaml manifest","title":"1. Download and modify the Calico descriptor"},{"location":"usage/advanced/calico/#2-create-the-cluster-without-flannel-and-with-calico","text":"On the k3s cluster creation : add the flag --flannel-backend=none . For this, on k3d you need to forward this flag to k3s with the option --k3s-arg . mount ( --volume ) the calico descriptor in the auto deploy manifest directory of k3s /var/lib/rancher/k3s/server/manifests/ So the command of the cluster creation is (when you are at root of the k3d repository) k3d cluster create \" ${ clustername } \" \\ --k3s-arg '--flannel-backend=none@server:*' \\ --volume \" $( pwd ) /docs/usage/guides/calico.yaml:/var/lib/rancher/k3s/server/manifests/calico.yaml\" In this example : change \"${clustername}\" with the name of the cluster (or set a variable). $(pwd)/docs/usage/guides/calico.yaml is the absolute path of the calico manifest, you can adapt it. You can add other options, see . The cluster will start without flannel and with Calico as CNI Plugin. For watching for the pod(s) deployment watch \"kubectl get pods -n kube-system\" You will have something like this at beginning (with the command line kubectl get pods -n kube-system ) NAME READY STATUS RESTARTS AGE helm-install-traefik-pn84f 0 /1 Pending 0 3s calico-node-97rx8 0 /1 Init:0/3 0 3s metrics-server-7566d596c8-hwnqq 0 /1 Pending 0 2s calico-kube-controllers-58b656d69f-2z7cn 0 /1 Pending 0 2s local-path-provisioner-6d59f47c7-rmswg 0 /1 Pending 0 2s coredns-8655855d6-cxtnr 0 /1 Pending 0 2s And when it finish to start NAME READY STATUS RESTARTS AGE metrics-server-7566d596c8-hwnqq 1 /1 Running 0 56s calico-node-97rx8 1 /1 Running 0 57s helm-install-traefik-pn84f 0 /1 Completed 1 57s svclb-traefik-lmjr5 2 /2 Running 0 28s calico-kube-controllers-58b656d69f-2z7cn 1 /1 Running 0 56s local-path-provisioner-6d59f47c7-rmswg 1 /1 Running 0 56s traefik-758cd5fc85-x8p57 1 /1 Running 0 28s coredns-8655855d6-cxtnr 1 /1 Running 0 56s Note : you can use the auto deploy manifest or a kubectl apply depending on your needs Calico is not as quick as Flannel (but it provides more features)","title":"2. Create the cluster without flannel and with calico"},{"location":"usage/advanced/calico/#references","text":"https://rancher.com/docs/k3s/latest/en/installation/network-options/ https://docs.projectcalico.org/getting-started/kubernetes/k3s/","title":"References"},{"location":"usage/advanced/cuda/","text":"Running CUDA workloads \u00b6 If you want to run CUDA workloads on the K3s container you need to customize the container. CUDA workloads require the NVIDIA Container Runtime, so containerd needs to be configured to use this runtime. The K3s container itself also needs to run with this runtime. If you are using Docker you can install the NVIDIA Container Toolkit . Building a customized K3s image \u00b6 To get the NVIDIA container runtime in the K3s image you need to build your own K3s image. The native K3s image is based on Alpine but the NVIDIA container runtime is not supported on Alpine yet. To get around this we need to build the image with a supported base image. Dockerfile \u00b6 Dockerfile : ARG K3S_TAG = \"v1.21.2-k3s1\" FROM rancher/k3s:$K3S_TAG as k3s FROM nvidia/cuda:11.2.0-base-ubuntu18.04 ARG NVIDIA_CONTAINER_RUNTIME_VERSION ENV NVIDIA_CONTAINER_RUNTIME_VERSION = $NVIDIA_CONTAINER_RUNTIME_VERSION RUN echo 'debconf debconf/frontend select Noninteractive' | debconf-set-selections RUN apt-get update && \\ apt-get -y install gnupg2 curl # Install NVIDIA Container Runtime RUN curl -s -L https://nvidia.github.io/nvidia-container-runtime/gpgkey | apt-key add - RUN curl -s -L https://nvidia.github.io/nvidia-container-runtime/ubuntu18.04/nvidia-container-runtime.list | tee /etc/apt/sources.list.d/nvidia-container-runtime.list RUN apt-get update && \\ apt-get -y install nvidia-container-runtime = ${ NVIDIA_CONTAINER_RUNTIME_VERSION } COPY --from = k3s / / RUN mkdir -p /etc && \\ echo 'hosts: files dns' > /etc/nsswitch.conf RUN chmod 1777 /tmp # Provide custom containerd configuration to configure the nvidia-container-runtime RUN mkdir -p /var/lib/rancher/k3s/agent/etc/containerd/ COPY config.toml.tmpl /var/lib/rancher/k3s/agent/etc/containerd/config.toml.tmpl # Deploy the nvidia driver plugin on startup RUN mkdir -p /var/lib/rancher/k3s/server/manifests COPY device-plugin-daemonset.yaml /var/lib/rancher/k3s/server/manifests/nvidia-device-plugin-daemonset.yaml VOLUME /var/lib/kubelet VOLUME /var/lib/rancher/k3s VOLUME /var/lib/cni VOLUME /var/log ENV PATH = \" $PATH :/bin/aux\" ENTRYPOINT [ \"/bin/k3s\" ] CMD [ \"agent\" ] This Dockerfile is based on the K3s Dockerfile The following changes are applied: Change the base images to nvidia/cuda:11.2.0-base-ubuntu18.04 so the NVIDIA Container Runtime can be installed. The version of cuda:xx.x.x must match the one you\u2019re planning to use. Add a custom containerd config.toml template to add the NVIDIA Container Runtime. This replaces the default runc runtime Add a manifest for the NVIDIA driver plugin for Kubernetes Configure containerd \u00b6 We need to configure containerd to use the NVIDIA Container Runtime. We need to customize the config.toml that is used at startup. K3s provides a way to do this using a config.toml.tmpl file. More information can be found on the K3s site . [ plugins . opt ] path = \"{{ .NodeConfig.Containerd.Opt }}\" [ plugins . cri ] stream_server_address = \"127.0.0.1\" stream_server_port = \"10010\" {{ - if . IsRunningInUserNS }} disable_cgroup = true disable_apparmor = true restrict_oom_score_adj = true {{ end }} {{ - if . NodeConfig . AgentConfig . PauseImage }} sandbox_image = \"{{ .NodeConfig.AgentConfig.PauseImage }}\" {{ end }} {{ - if not . NodeConfig . NoFlannel }} [ plugins . cri . cni ] bin_dir = \"{{ .NodeConfig.AgentConfig.CNIBinDir }}\" conf_dir = \"{{ .NodeConfig.AgentConfig.CNIConfDir }}\" {{ end }} [ plugins . cri . containerd . runtimes . runc ] # ---- changed from ' io . containerd . runc . v2 ' for GPU support runtime_type = \"io.containerd.runtime.v1.linux\" # ---- added for GPU support [ plugins . linux ] runtime = \"nvidia-container-runtime\" {{ if . PrivateRegistryConfig }} {{ if . PrivateRegistryConfig . Mirrors }} [ plugins . cri . registry . mirrors ]{{ end }} {{ range $ k , $ v := . PrivateRegistryConfig . Mirrors }} [ plugins . cri . registry . mirrors . \"{{$k}}\" ] endpoint = [{{ range $ i , $ j := $ v . Endpoints }}{{ if $ i }}, {{ end }}{{ printf \"%q\" .}}{{ end }}] {{ end }} {{ range $ k , $ v := . PrivateRegistryConfig . Configs }} {{ if $ v . Auth }} [ plugins . cri . registry . configs . \"{{$k}}\" . auth ] {{ if $ v . Auth . Username }} username = \"{{ $v.Auth.Username }}\" {{ end }} {{ if $ v . Auth . Password }} password = \"{{ $v.Auth.Password }}\" {{ end }} {{ if $ v . Auth . Auth }} auth = \"{{ $v.Auth.Auth }}\" {{ end }} {{ if $ v . Auth . IdentityToken }} identitytoken = \"{{ $v.Auth.IdentityToken }}\" {{ end }} {{ end }} {{ if $ v . TLS }} [ plugins . cri . registry . configs . \"{{$k}}\" . tls ] {{ if $ v . TLS . CAFile }} ca_file = \"{{ $v.TLS.CAFile }}\" {{ end }} {{ if $ v . TLS . CertFile }} cert_file = \"{{ $v.TLS.CertFile }}\" {{ end }} {{ if $ v . TLS . KeyFile }} key_file = \"{{ $v.TLS.KeyFile }}\" {{ end }} {{ end }} {{ end }} {{ end }} The NVIDIA device plugin \u00b6 To enable NVIDIA GPU support on Kubernetes you also need to install the NVIDIA device plugin . The device plugin is a deamonset and allows you to automatically: Expose the number of GPUs on each nodes of your cluster Keep track of the health of your GPUs Run GPU enabled containers in your Kubernetes cluster. apiVersion : apps/v1 kind : DaemonSet metadata : name : nvidia-device-plugin-daemonset namespace : kube-system spec : selector : matchLabels : name : nvidia-device-plugin-ds template : metadata : # Mark this pod as a critical add-on; when enabled, the critical add-on scheduler # reserves resources for critical add-on pods so that they can be rescheduled after # a failure. This annotation works in tandem with the toleration below. annotations : scheduler.alpha.kubernetes.io/critical-pod : \"\" labels : name : nvidia-device-plugin-ds spec : tolerations : # Allow this pod to be rescheduled while the node is in \"critical add-ons only\" mode. # This, along with the annotation above marks this pod as a critical add-on. - key : CriticalAddonsOnly operator : Exists containers : - env : - name : DP_DISABLE_HEALTHCHECKS value : xids image : nvidia/k8s-device-plugin:1.11 name : nvidia-device-plugin-ctr securityContext : allowPrivilegeEscalation : true capabilities : drop : [ \"ALL\" ] volumeMounts : - name : device-plugin mountPath : /var/lib/kubelet/device-plugins volumes : - name : device-plugin hostPath : path : /var/lib/kubelet/device-plugins Build the K3s image \u00b6 To build the custom image we need to build K3s because we need the generated output. Put the following files in a directory: Dockerfile config.toml.tmpl device-plugin-daemonset.yaml build.sh cuda-vector-add.yaml The build.sh script is configured using exports & defaults to v1.21.2+k3s1 . Please set at least the IMAGE_REGISTRY variable! The script performs the following steps builds the custom K3s image including the nvidia drivers. build.sh : #!/bin/bash set -euxo pipefail K3S_TAG = ${ K3S_TAG := \"v1.21.2-k3s1\" } # replace + with -, if needed IMAGE_REGISTRY = ${ IMAGE_REGISTRY := \"MY_REGISTRY\" } IMAGE_REPOSITORY = ${ IMAGE_REPOSITORY := \"rancher/k3s\" } IMAGE_TAG = \" $K3S_TAG -cuda\" IMAGE = ${ IMAGE := \" $IMAGE_REGISTRY / $IMAGE_REPOSITORY : $IMAGE_TAG \" } NVIDIA_CONTAINER_RUNTIME_VERSION = ${ NVIDIA_CONTAINER_RUNTIME_VERSION := \"3.5.0-1\" } echo \"IMAGE= $IMAGE \" # due to some unknown reason, copying symlinks fails with buildkit enabled DOCKER_BUILDKIT = 0 docker build \\ --build-arg K3S_TAG = $K3S_TAG \\ --build-arg NVIDIA_CONTAINER_RUNTIME_VERSION = $NVIDIA_CONTAINER_RUNTIME_VERSION \\ -t $IMAGE . docker push $IMAGE echo \"Done!\" Run and test the custom image with k3d \u00b6 You can use the image with k3d: k3d cluster create gputest --image = $IMAGE --gpus = 1 Deploy a test pod : kubectl apply -f cuda-vector-add.yaml kubectl logs cuda-vector-add This should output something like the following: $ kubectl logs cuda-vector-add [ Vector addition of 50000 elements ] Copy input data from the host memory to the CUDA device CUDA kernel launch with 196 blocks of 256 threads Copy output data from the CUDA device to the host memory Test PASSED Done If the cuda-vector-add pod is stuck in Pending state, probably the device-driver daemonset didn\u2019t get deployed correctly from the auto-deploy manifests. In that case, you can apply it manually via kubectl apply -f device-plugin-daemonset.yaml . Known issues \u00b6 This approach does not work on WSL2 yet. The NVIDIA driver plugin and container runtime rely on the NVIDIA Management Library (NVML) which is not yet supported. See the CUDA on WSL User Guide . Acknowledgements \u00b6 Most of the information in this article was obtained from various sources: Add NVIDIA GPU support to k3s with containerd microk8s K3s k3s-gpu Authors \u00b6 @markrexwinkel @vainkop @iwilltry42","title":"Running CUDA workloads"},{"location":"usage/advanced/cuda/#running-cuda-workloads","text":"If you want to run CUDA workloads on the K3s container you need to customize the container. CUDA workloads require the NVIDIA Container Runtime, so containerd needs to be configured to use this runtime. The K3s container itself also needs to run with this runtime. If you are using Docker you can install the NVIDIA Container Toolkit .","title":"Running CUDA workloads"},{"location":"usage/advanced/cuda/#building-a-customized-k3s-image","text":"To get the NVIDIA container runtime in the K3s image you need to build your own K3s image. The native K3s image is based on Alpine but the NVIDIA container runtime is not supported on Alpine yet. To get around this we need to build the image with a supported base image.","title":"Building a customized K3s image"},{"location":"usage/advanced/cuda/#dockerfile","text":"Dockerfile : ARG K3S_TAG = \"v1.21.2-k3s1\" FROM rancher/k3s:$K3S_TAG as k3s FROM nvidia/cuda:11.2.0-base-ubuntu18.04 ARG NVIDIA_CONTAINER_RUNTIME_VERSION ENV NVIDIA_CONTAINER_RUNTIME_VERSION = $NVIDIA_CONTAINER_RUNTIME_VERSION RUN echo 'debconf debconf/frontend select Noninteractive' | debconf-set-selections RUN apt-get update && \\ apt-get -y install gnupg2 curl # Install NVIDIA Container Runtime RUN curl -s -L https://nvidia.github.io/nvidia-container-runtime/gpgkey | apt-key add - RUN curl -s -L https://nvidia.github.io/nvidia-container-runtime/ubuntu18.04/nvidia-container-runtime.list | tee /etc/apt/sources.list.d/nvidia-container-runtime.list RUN apt-get update && \\ apt-get -y install nvidia-container-runtime = ${ NVIDIA_CONTAINER_RUNTIME_VERSION } COPY --from = k3s / / RUN mkdir -p /etc && \\ echo 'hosts: files dns' > /etc/nsswitch.conf RUN chmod 1777 /tmp # Provide custom containerd configuration to configure the nvidia-container-runtime RUN mkdir -p /var/lib/rancher/k3s/agent/etc/containerd/ COPY config.toml.tmpl /var/lib/rancher/k3s/agent/etc/containerd/config.toml.tmpl # Deploy the nvidia driver plugin on startup RUN mkdir -p /var/lib/rancher/k3s/server/manifests COPY device-plugin-daemonset.yaml /var/lib/rancher/k3s/server/manifests/nvidia-device-plugin-daemonset.yaml VOLUME /var/lib/kubelet VOLUME /var/lib/rancher/k3s VOLUME /var/lib/cni VOLUME /var/log ENV PATH = \" $PATH :/bin/aux\" ENTRYPOINT [ \"/bin/k3s\" ] CMD [ \"agent\" ] This Dockerfile is based on the K3s Dockerfile The following changes are applied: Change the base images to nvidia/cuda:11.2.0-base-ubuntu18.04 so the NVIDIA Container Runtime can be installed. The version of cuda:xx.x.x must match the one you\u2019re planning to use. Add a custom containerd config.toml template to add the NVIDIA Container Runtime. This replaces the default runc runtime Add a manifest for the NVIDIA driver plugin for Kubernetes","title":"Dockerfile"},{"location":"usage/advanced/cuda/#configure-containerd","text":"We need to configure containerd to use the NVIDIA Container Runtime. We need to customize the config.toml that is used at startup. K3s provides a way to do this using a config.toml.tmpl file. More information can be found on the K3s site . [ plugins . opt ] path = \"{{ .NodeConfig.Containerd.Opt }}\" [ plugins . cri ] stream_server_address = \"127.0.0.1\" stream_server_port = \"10010\" {{ - if . IsRunningInUserNS }} disable_cgroup = true disable_apparmor = true restrict_oom_score_adj = true {{ end }} {{ - if . NodeConfig . AgentConfig . PauseImage }} sandbox_image = \"{{ .NodeConfig.AgentConfig.PauseImage }}\" {{ end }} {{ - if not . NodeConfig . NoFlannel }} [ plugins . cri . cni ] bin_dir = \"{{ .NodeConfig.AgentConfig.CNIBinDir }}\" conf_dir = \"{{ .NodeConfig.AgentConfig.CNIConfDir }}\" {{ end }} [ plugins . cri . containerd . runtimes . runc ] # ---- changed from ' io . containerd . runc . v2 ' for GPU support runtime_type = \"io.containerd.runtime.v1.linux\" # ---- added for GPU support [ plugins . linux ] runtime = \"nvidia-container-runtime\" {{ if . PrivateRegistryConfig }} {{ if . PrivateRegistryConfig . Mirrors }} [ plugins . cri . registry . mirrors ]{{ end }} {{ range $ k , $ v := . PrivateRegistryConfig . Mirrors }} [ plugins . cri . registry . mirrors . \"{{$k}}\" ] endpoint = [{{ range $ i , $ j := $ v . Endpoints }}{{ if $ i }}, {{ end }}{{ printf \"%q\" .}}{{ end }}] {{ end }} {{ range $ k , $ v := . PrivateRegistryConfig . Configs }} {{ if $ v . Auth }} [ plugins . cri . registry . configs . \"{{$k}}\" . auth ] {{ if $ v . Auth . Username }} username = \"{{ $v.Auth.Username }}\" {{ end }} {{ if $ v . Auth . Password }} password = \"{{ $v.Auth.Password }}\" {{ end }} {{ if $ v . Auth . Auth }} auth = \"{{ $v.Auth.Auth }}\" {{ end }} {{ if $ v . Auth . IdentityToken }} identitytoken = \"{{ $v.Auth.IdentityToken }}\" {{ end }} {{ end }} {{ if $ v . TLS }} [ plugins . cri . registry . configs . \"{{$k}}\" . tls ] {{ if $ v . TLS . CAFile }} ca_file = \"{{ $v.TLS.CAFile }}\" {{ end }} {{ if $ v . TLS . CertFile }} cert_file = \"{{ $v.TLS.CertFile }}\" {{ end }} {{ if $ v . TLS . KeyFile }} key_file = \"{{ $v.TLS.KeyFile }}\" {{ end }} {{ end }} {{ end }} {{ end }}","title":"Configure containerd"},{"location":"usage/advanced/cuda/#the-nvidia-device-plugin","text":"To enable NVIDIA GPU support on Kubernetes you also need to install the NVIDIA device plugin . The device plugin is a deamonset and allows you to automatically: Expose the number of GPUs on each nodes of your cluster Keep track of the health of your GPUs Run GPU enabled containers in your Kubernetes cluster. apiVersion : apps/v1 kind : DaemonSet metadata : name : nvidia-device-plugin-daemonset namespace : kube-system spec : selector : matchLabels : name : nvidia-device-plugin-ds template : metadata : # Mark this pod as a critical add-on; when enabled, the critical add-on scheduler # reserves resources for critical add-on pods so that they can be rescheduled after # a failure. This annotation works in tandem with the toleration below. annotations : scheduler.alpha.kubernetes.io/critical-pod : \"\" labels : name : nvidia-device-plugin-ds spec : tolerations : # Allow this pod to be rescheduled while the node is in \"critical add-ons only\" mode. # This, along with the annotation above marks this pod as a critical add-on. - key : CriticalAddonsOnly operator : Exists containers : - env : - name : DP_DISABLE_HEALTHCHECKS value : xids image : nvidia/k8s-device-plugin:1.11 name : nvidia-device-plugin-ctr securityContext : allowPrivilegeEscalation : true capabilities : drop : [ \"ALL\" ] volumeMounts : - name : device-plugin mountPath : /var/lib/kubelet/device-plugins volumes : - name : device-plugin hostPath : path : /var/lib/kubelet/device-plugins","title":"The NVIDIA device plugin"},{"location":"usage/advanced/cuda/#build-the-k3s-image","text":"To build the custom image we need to build K3s because we need the generated output. Put the following files in a directory: Dockerfile config.toml.tmpl device-plugin-daemonset.yaml build.sh cuda-vector-add.yaml The build.sh script is configured using exports & defaults to v1.21.2+k3s1 . Please set at least the IMAGE_REGISTRY variable! The script performs the following steps builds the custom K3s image including the nvidia drivers. build.sh : #!/bin/bash set -euxo pipefail K3S_TAG = ${ K3S_TAG := \"v1.21.2-k3s1\" } # replace + with -, if needed IMAGE_REGISTRY = ${ IMAGE_REGISTRY := \"MY_REGISTRY\" } IMAGE_REPOSITORY = ${ IMAGE_REPOSITORY := \"rancher/k3s\" } IMAGE_TAG = \" $K3S_TAG -cuda\" IMAGE = ${ IMAGE := \" $IMAGE_REGISTRY / $IMAGE_REPOSITORY : $IMAGE_TAG \" } NVIDIA_CONTAINER_RUNTIME_VERSION = ${ NVIDIA_CONTAINER_RUNTIME_VERSION := \"3.5.0-1\" } echo \"IMAGE= $IMAGE \" # due to some unknown reason, copying symlinks fails with buildkit enabled DOCKER_BUILDKIT = 0 docker build \\ --build-arg K3S_TAG = $K3S_TAG \\ --build-arg NVIDIA_CONTAINER_RUNTIME_VERSION = $NVIDIA_CONTAINER_RUNTIME_VERSION \\ -t $IMAGE . docker push $IMAGE echo \"Done!\"","title":"Build the K3s image"},{"location":"usage/advanced/cuda/#run-and-test-the-custom-image-with-k3d","text":"You can use the image with k3d: k3d cluster create gputest --image = $IMAGE --gpus = 1 Deploy a test pod : kubectl apply -f cuda-vector-add.yaml kubectl logs cuda-vector-add This should output something like the following: $ kubectl logs cuda-vector-add [ Vector addition of 50000 elements ] Copy input data from the host memory to the CUDA device CUDA kernel launch with 196 blocks of 256 threads Copy output data from the CUDA device to the host memory Test PASSED Done If the cuda-vector-add pod is stuck in Pending state, probably the device-driver daemonset didn\u2019t get deployed correctly from the auto-deploy manifests. In that case, you can apply it manually via kubectl apply -f device-plugin-daemonset.yaml .","title":"Run and test the custom image with k3d"},{"location":"usage/advanced/cuda/#known-issues","text":"This approach does not work on WSL2 yet. The NVIDIA driver plugin and container runtime rely on the NVIDIA Management Library (NVML) which is not yet supported. See the CUDA on WSL User Guide .","title":"Known issues"},{"location":"usage/advanced/cuda/#acknowledgements","text":"Most of the information in this article was obtained from various sources: Add NVIDIA GPU support to k3s with containerd microk8s K3s k3s-gpu","title":"Acknowledgements"},{"location":"usage/advanced/cuda/#authors","text":"@markrexwinkel @vainkop @iwilltry42","title":"Authors"},{"location":"usage/commands/k3d/","text":"k3d \u00b6 https://k3d.io/ -> Run k3s in Docker! Synopsis \u00b6 https://k3d.io/ k3d is a wrapper CLI that helps you to easily create k3s clusters inside docker. Nodes of a k3d cluster are docker containers running a k3s image. All Nodes of a k3d cluster are part of the same docker network. k3d [flags] Options \u00b6 -h, --help help for k3d --timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging) --version Show k3d and default k3s version SEE ALSO \u00b6 k3d cluster - Manage cluster(s) k3d completion - Generate completion scripts for [bash, zsh, fish, powershell | psh] k3d config - Work with config file(s) k3d image - Handle container images. k3d kubeconfig - Manage kubeconfig(s) k3d node - Manage node(s) k3d registry - Manage registry/registries k3d version - Show k3d and default k3s version","title":"K3d"},{"location":"usage/commands/k3d/#k3d","text":"https://k3d.io/ -> Run k3s in Docker!","title":"k3d"},{"location":"usage/commands/k3d/#synopsis","text":"https://k3d.io/ k3d is a wrapper CLI that helps you to easily create k3s clusters inside docker. Nodes of a k3d cluster are docker containers running a k3s image. All Nodes of a k3d cluster are part of the same docker network. k3d [flags]","title":"Synopsis"},{"location":"usage/commands/k3d/#options","text":"-h, --help help for k3d --timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging) --version Show k3d and default k3s version","title":"Options"},{"location":"usage/commands/k3d/#see-also","text":"k3d cluster - Manage cluster(s) k3d completion - Generate completion scripts for [bash, zsh, fish, powershell | psh] k3d config - Work with config file(s) k3d image - Handle container images. k3d kubeconfig - Manage kubeconfig(s) k3d node - Manage node(s) k3d registry - Manage registry/registries k3d version - Show k3d and default k3s version","title":"SEE ALSO"},{"location":"usage/commands/k3d_cluster/","text":"k3d cluster \u00b6 Manage cluster(s) Synopsis \u00b6 Manage cluster(s) k3d cluster [flags] Options \u00b6 -h, --help help for cluster Options inherited from parent commands \u00b6 --timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging) SEE ALSO \u00b6 k3d - https://k3d.io/ -> Run k3s in Docker! k3d cluster create - Create a new cluster k3d cluster delete - Delete cluster(s). k3d cluster edit - [EXPERIMENTAL] Edit cluster(s). k3d cluster list - List cluster(s) k3d cluster start - Start existing k3d cluster(s) k3d cluster stop - Stop existing k3d cluster(s)","title":"K3d cluster"},{"location":"usage/commands/k3d_cluster/#k3d-cluster","text":"Manage cluster(s)","title":"k3d cluster"},{"location":"usage/commands/k3d_cluster/#synopsis","text":"Manage cluster(s) k3d cluster [flags]","title":"Synopsis"},{"location":"usage/commands/k3d_cluster/#options","text":"-h, --help help for cluster","title":"Options"},{"location":"usage/commands/k3d_cluster/#options-inherited-from-parent-commands","text":"--timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging)","title":"Options inherited from parent commands"},{"location":"usage/commands/k3d_cluster/#see-also","text":"k3d - https://k3d.io/ -> Run k3s in Docker! k3d cluster create - Create a new cluster k3d cluster delete - Delete cluster(s). k3d cluster edit - [EXPERIMENTAL] Edit cluster(s). k3d cluster list - List cluster(s) k3d cluster start - Start existing k3d cluster(s) k3d cluster stop - Stop existing k3d cluster(s)","title":"SEE ALSO"},{"location":"usage/commands/k3d_cluster_create/","text":"k3d cluster create \u00b6 Create a new cluster Synopsis \u00b6 Create a new k3s cluster with containerized nodes (k3s in docker). Every cluster will consist of one or more containers: - 1 (or more) server node container (k3s) - (optionally) 1 loadbalancer container as the entrypoint to the cluster (nginx) - (optionally) 1 (or more) agent node containers (k3s) k3d cluster create NAME [flags] Options \u00b6 -a, --agents int Specify how many agents you want to create --agents-memory string Memory limit imposed on the agents nodes [From docker] --api-port [HOST:]HOSTPORT Specify the Kubernetes API server port exposed on the LoadBalancer (Format: [HOST:]HOSTPORT) - Example: `k3d cluster create --servers 3 --api-port 0.0.0.0:6550` -c, --config string Path of a config file to use -e, --env KEY[=VALUE][@NODEFILTER[;NODEFILTER...]] Add environment variables to nodes (Format: KEY[=VALUE][@NODEFILTER[;NODEFILTER...]] - Example: `k3d cluster create --agents 2 -e \"HTTP_PROXY=my.proxy.com@server:0\" -e \"SOME_KEY=SOME_VAL@server:0\"` --gpus string GPU devices to add to the cluster node containers ('all' to pass all GPUs) [From docker] -h, --help help for create -i, --image string Specify k3s image that you want to use for the nodes --k3s-arg ARG@NODEFILTER[;@NODEFILTER] Additional args passed to k3s command (Format: ARG@NODEFILTER[;@NODEFILTER]) - Example: `k3d cluster create --k3s-arg \"--disable=traefik@server:0\" --k3s-node-label KEY[=VALUE][@NODEFILTER[;NODEFILTER...]] Add label to k3s node (Format: KEY[=VALUE][@NODEFILTER[;NODEFILTER...]] - Example: `k3d cluster create --agents 2 --k3s-node-label \"my.label@agent:0,1\" --k3s-node-label \"other.label=somevalue@server:0\"` --kubeconfig-switch-context Directly switch the default kubeconfig's current-context to the new cluster's context (requires --kubeconfig-update-default) (default true) --kubeconfig-update-default Directly update the default kubeconfig with the new cluster's context (default true) --lb-config-override strings Use dotted YAML path syntax to override nginx loadbalancer settings --network string Join an existing network --no-image-volume Disable the creation of a volume for importing images --no-lb Disable the creation of a LoadBalancer in front of the server nodes --no-rollback Disable the automatic rollback actions, if anything goes wrong -p, --port [HOST:][HOSTPORT:]CONTAINERPORT[/PROTOCOL][@NODEFILTER] Map ports from the node containers (via the serverlb) to the host (Format: [HOST:][HOSTPORT:]CONTAINERPORT[/PROTOCOL][@NODEFILTER]) - Example: `k3d cluster create --agents 2 -p 8080:80@agent:0 -p 8081@agent:1` --registry-config string Specify path to an extra registries.yaml file --registry-create NAME[:HOST][:HOSTPORT] Create a k3d-managed registry and connect it to the cluster (Format: NAME[:HOST][:HOSTPORT] - Example: `k3d cluster create --registry-create mycluster-registry:0.0.0.0:5432` --registry-use stringArray Connect to one or more k3d-managed registries running locally --runtime-label KEY[=VALUE][@NODEFILTER[;NODEFILTER...]] Add label to container runtime (Format: KEY[=VALUE][@NODEFILTER[;NODEFILTER...]] - Example: `k3d cluster create --agents 2 --runtime-label \"my.label@agent:0,1\" --runtime-label \"other.label=somevalue@server:0\"` -s, --servers int Specify how many servers you want to create --servers-memory string Memory limit imposed on the server nodes [From docker] --subnet 172.28.0.0/16 [Experimental: IPAM] Define a subnet for the newly created container network (Example: 172.28.0.0/16) --timeout duration Rollback changes if cluster couldn't be created in specified duration. --token string Specify a cluster token. By default, we generate one. -v, --volume [SOURCE:]DEST[@NODEFILTER[;NODEFILTER...]] Mount volumes into the nodes (Format: [SOURCE:]DEST[@NODEFILTER[;NODEFILTER...]] - Example: `k3d cluster create --agents 2 -v /my/path@agent:0,1 -v /tmp/test:/tmp/other@server:0` --wait Wait for the server(s) to be ready before returning. Use '--timeout DURATION' to not wait forever. (default true) Options inherited from parent commands \u00b6 --timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging) SEE ALSO \u00b6 k3d cluster - Manage cluster(s)","title":"K3d cluster create"},{"location":"usage/commands/k3d_cluster_create/#k3d-cluster-create","text":"Create a new cluster","title":"k3d cluster create"},{"location":"usage/commands/k3d_cluster_create/#synopsis","text":"Create a new k3s cluster with containerized nodes (k3s in docker). Every cluster will consist of one or more containers: - 1 (or more) server node container (k3s) - (optionally) 1 loadbalancer container as the entrypoint to the cluster (nginx) - (optionally) 1 (or more) agent node containers (k3s) k3d cluster create NAME [flags]","title":"Synopsis"},{"location":"usage/commands/k3d_cluster_create/#options","text":"-a, --agents int Specify how many agents you want to create --agents-memory string Memory limit imposed on the agents nodes [From docker] --api-port [HOST:]HOSTPORT Specify the Kubernetes API server port exposed on the LoadBalancer (Format: [HOST:]HOSTPORT) - Example: `k3d cluster create --servers 3 --api-port 0.0.0.0:6550` -c, --config string Path of a config file to use -e, --env KEY[=VALUE][@NODEFILTER[;NODEFILTER...]] Add environment variables to nodes (Format: KEY[=VALUE][@NODEFILTER[;NODEFILTER...]] - Example: `k3d cluster create --agents 2 -e \"HTTP_PROXY=my.proxy.com@server:0\" -e \"SOME_KEY=SOME_VAL@server:0\"` --gpus string GPU devices to add to the cluster node containers ('all' to pass all GPUs) [From docker] -h, --help help for create -i, --image string Specify k3s image that you want to use for the nodes --k3s-arg ARG@NODEFILTER[;@NODEFILTER] Additional args passed to k3s command (Format: ARG@NODEFILTER[;@NODEFILTER]) - Example: `k3d cluster create --k3s-arg \"--disable=traefik@server:0\" --k3s-node-label KEY[=VALUE][@NODEFILTER[;NODEFILTER...]] Add label to k3s node (Format: KEY[=VALUE][@NODEFILTER[;NODEFILTER...]] - Example: `k3d cluster create --agents 2 --k3s-node-label \"my.label@agent:0,1\" --k3s-node-label \"other.label=somevalue@server:0\"` --kubeconfig-switch-context Directly switch the default kubeconfig's current-context to the new cluster's context (requires --kubeconfig-update-default) (default true) --kubeconfig-update-default Directly update the default kubeconfig with the new cluster's context (default true) --lb-config-override strings Use dotted YAML path syntax to override nginx loadbalancer settings --network string Join an existing network --no-image-volume Disable the creation of a volume for importing images --no-lb Disable the creation of a LoadBalancer in front of the server nodes --no-rollback Disable the automatic rollback actions, if anything goes wrong -p, --port [HOST:][HOSTPORT:]CONTAINERPORT[/PROTOCOL][@NODEFILTER] Map ports from the node containers (via the serverlb) to the host (Format: [HOST:][HOSTPORT:]CONTAINERPORT[/PROTOCOL][@NODEFILTER]) - Example: `k3d cluster create --agents 2 -p 8080:80@agent:0 -p 8081@agent:1` --registry-config string Specify path to an extra registries.yaml file --registry-create NAME[:HOST][:HOSTPORT] Create a k3d-managed registry and connect it to the cluster (Format: NAME[:HOST][:HOSTPORT] - Example: `k3d cluster create --registry-create mycluster-registry:0.0.0.0:5432` --registry-use stringArray Connect to one or more k3d-managed registries running locally --runtime-label KEY[=VALUE][@NODEFILTER[;NODEFILTER...]] Add label to container runtime (Format: KEY[=VALUE][@NODEFILTER[;NODEFILTER...]] - Example: `k3d cluster create --agents 2 --runtime-label \"my.label@agent:0,1\" --runtime-label \"other.label=somevalue@server:0\"` -s, --servers int Specify how many servers you want to create --servers-memory string Memory limit imposed on the server nodes [From docker] --subnet 172.28.0.0/16 [Experimental: IPAM] Define a subnet for the newly created container network (Example: 172.28.0.0/16) --timeout duration Rollback changes if cluster couldn't be created in specified duration. --token string Specify a cluster token. By default, we generate one. -v, --volume [SOURCE:]DEST[@NODEFILTER[;NODEFILTER...]] Mount volumes into the nodes (Format: [SOURCE:]DEST[@NODEFILTER[;NODEFILTER...]] - Example: `k3d cluster create --agents 2 -v /my/path@agent:0,1 -v /tmp/test:/tmp/other@server:0` --wait Wait for the server(s) to be ready before returning. Use '--timeout DURATION' to not wait forever. (default true)","title":"Options"},{"location":"usage/commands/k3d_cluster_create/#options-inherited-from-parent-commands","text":"--timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging)","title":"Options inherited from parent commands"},{"location":"usage/commands/k3d_cluster_create/#see-also","text":"k3d cluster - Manage cluster(s)","title":"SEE ALSO"},{"location":"usage/commands/k3d_cluster_delete/","text":"k3d cluster delete \u00b6 Delete cluster(s). Synopsis \u00b6 Delete cluster(s). k3d cluster delete [NAME [NAME ...] | --all] [flags] Options \u00b6 -a, --all Delete all existing clusters -c, --config string Path of a config file to use -h, --help help for delete Options inherited from parent commands \u00b6 --timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging) SEE ALSO \u00b6 k3d cluster - Manage cluster(s)","title":"K3d cluster delete"},{"location":"usage/commands/k3d_cluster_delete/#k3d-cluster-delete","text":"Delete cluster(s).","title":"k3d cluster delete"},{"location":"usage/commands/k3d_cluster_delete/#synopsis","text":"Delete cluster(s). k3d cluster delete [NAME [NAME ...] | --all] [flags]","title":"Synopsis"},{"location":"usage/commands/k3d_cluster_delete/#options","text":"-a, --all Delete all existing clusters -c, --config string Path of a config file to use -h, --help help for delete","title":"Options"},{"location":"usage/commands/k3d_cluster_delete/#options-inherited-from-parent-commands","text":"--timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging)","title":"Options inherited from parent commands"},{"location":"usage/commands/k3d_cluster_delete/#see-also","text":"k3d cluster - Manage cluster(s)","title":"SEE ALSO"},{"location":"usage/commands/k3d_cluster_edit/","text":"k3d cluster edit \u00b6 [EXPERIMENTAL] Edit cluster(s). Synopsis \u00b6 [EXPERIMENTAL] Edit cluster(s). k3d cluster edit CLUSTER [flags] Options \u00b6 -h, --help help for edit --port-add [HOST:][HOSTPORT:]CONTAINERPORT[/PROTOCOL][@NODEFILTER] [EXPERIMENTAL] Map ports from the node containers (via the serverlb) to the host (Format: [HOST:][HOSTPORT:]CONTAINERPORT[/PROTOCOL][@NODEFILTER]) - Example: `k3d node edit k3d-mycluster-serverlb --port-add 8080:80` Options inherited from parent commands \u00b6 --timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging) SEE ALSO \u00b6 k3d cluster - Manage cluster(s)","title":"K3d cluster edit"},{"location":"usage/commands/k3d_cluster_edit/#k3d-cluster-edit","text":"[EXPERIMENTAL] Edit cluster(s).","title":"k3d cluster edit"},{"location":"usage/commands/k3d_cluster_edit/#synopsis","text":"[EXPERIMENTAL] Edit cluster(s). k3d cluster edit CLUSTER [flags]","title":"Synopsis"},{"location":"usage/commands/k3d_cluster_edit/#options","text":"-h, --help help for edit --port-add [HOST:][HOSTPORT:]CONTAINERPORT[/PROTOCOL][@NODEFILTER] [EXPERIMENTAL] Map ports from the node containers (via the serverlb) to the host (Format: [HOST:][HOSTPORT:]CONTAINERPORT[/PROTOCOL][@NODEFILTER]) - Example: `k3d node edit k3d-mycluster-serverlb --port-add 8080:80`","title":"Options"},{"location":"usage/commands/k3d_cluster_edit/#options-inherited-from-parent-commands","text":"--timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging)","title":"Options inherited from parent commands"},{"location":"usage/commands/k3d_cluster_edit/#see-also","text":"k3d cluster - Manage cluster(s)","title":"SEE ALSO"},{"location":"usage/commands/k3d_cluster_list/","text":"k3d cluster list \u00b6 List cluster(s) Synopsis \u00b6 List cluster(s). k3d cluster list [NAME [NAME...]] [flags] Options \u00b6 -h, --help help for list --no-headers Disable headers -o, --output string Output format. One of: json|yaml --token Print k3s cluster token Options inherited from parent commands \u00b6 --timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging) SEE ALSO \u00b6 k3d cluster - Manage cluster(s)","title":"K3d cluster list"},{"location":"usage/commands/k3d_cluster_list/#k3d-cluster-list","text":"List cluster(s)","title":"k3d cluster list"},{"location":"usage/commands/k3d_cluster_list/#synopsis","text":"List cluster(s). k3d cluster list [NAME [NAME...]] [flags]","title":"Synopsis"},{"location":"usage/commands/k3d_cluster_list/#options","text":"-h, --help help for list --no-headers Disable headers -o, --output string Output format. One of: json|yaml --token Print k3s cluster token","title":"Options"},{"location":"usage/commands/k3d_cluster_list/#options-inherited-from-parent-commands","text":"--timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging)","title":"Options inherited from parent commands"},{"location":"usage/commands/k3d_cluster_list/#see-also","text":"k3d cluster - Manage cluster(s)","title":"SEE ALSO"},{"location":"usage/commands/k3d_cluster_start/","text":"k3d cluster start \u00b6 Start existing k3d cluster(s) Synopsis \u00b6 Start existing k3d cluster(s) k3d cluster start [NAME [NAME...] | --all] [flags] Options \u00b6 -a, --all Start all existing clusters -h, --help help for start --timeout duration Maximum waiting time for '--wait' before canceling/returning. --wait Wait for the server(s) (and loadbalancer) to be ready before returning. (default true) Options inherited from parent commands \u00b6 --timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging) SEE ALSO \u00b6 k3d cluster - Manage cluster(s)","title":"K3d cluster start"},{"location":"usage/commands/k3d_cluster_start/#k3d-cluster-start","text":"Start existing k3d cluster(s)","title":"k3d cluster start"},{"location":"usage/commands/k3d_cluster_start/#synopsis","text":"Start existing k3d cluster(s) k3d cluster start [NAME [NAME...] | --all] [flags]","title":"Synopsis"},{"location":"usage/commands/k3d_cluster_start/#options","text":"-a, --all Start all existing clusters -h, --help help for start --timeout duration Maximum waiting time for '--wait' before canceling/returning. --wait Wait for the server(s) (and loadbalancer) to be ready before returning. (default true)","title":"Options"},{"location":"usage/commands/k3d_cluster_start/#options-inherited-from-parent-commands","text":"--timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging)","title":"Options inherited from parent commands"},{"location":"usage/commands/k3d_cluster_start/#see-also","text":"k3d cluster - Manage cluster(s)","title":"SEE ALSO"},{"location":"usage/commands/k3d_cluster_stop/","text":"k3d cluster stop \u00b6 Stop existing k3d cluster(s) Synopsis \u00b6 Stop existing k3d cluster(s). k3d cluster stop [NAME [NAME...] | --all] [flags] Options \u00b6 -a, --all Stop all existing clusters -h, --help help for stop Options inherited from parent commands \u00b6 --timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging) SEE ALSO \u00b6 k3d cluster - Manage cluster(s)","title":"K3d cluster stop"},{"location":"usage/commands/k3d_cluster_stop/#k3d-cluster-stop","text":"Stop existing k3d cluster(s)","title":"k3d cluster stop"},{"location":"usage/commands/k3d_cluster_stop/#synopsis","text":"Stop existing k3d cluster(s). k3d cluster stop [NAME [NAME...] | --all] [flags]","title":"Synopsis"},{"location":"usage/commands/k3d_cluster_stop/#options","text":"-a, --all Stop all existing clusters -h, --help help for stop","title":"Options"},{"location":"usage/commands/k3d_cluster_stop/#options-inherited-from-parent-commands","text":"--timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging)","title":"Options inherited from parent commands"},{"location":"usage/commands/k3d_cluster_stop/#see-also","text":"k3d cluster - Manage cluster(s)","title":"SEE ALSO"},{"location":"usage/commands/k3d_completion/","text":"k3d completion \u00b6 Generate completion scripts for [bash, zsh, fish, powershell | psh] Synopsis \u00b6 To load completions: Bash: $ source < ( k3d completion bash ) # To load completions for each session, execute once: # Linux: $ k3d completion bash > / etc / bash_completion . d / k3d # macOS: $ k3d completion bash > / usr / local / etc / bash_completion . d / k3d Zsh: # If shell completion is not already enabled in your environment, # you will need to enable it. You can execute the following once: $ echo \"autoload -U compinit; compinit\" >> ~/. zshrc # To load completions for each session, execute once: $ k3d completion zsh > \"${fpath[1]}/k3d\" # You will need to start a new shell for this setup to take effect. fish: $ k3d completion fish | source # To load completions for each session, execute once: $ k3d completion fish > ~/. config / fish / completions / k3d . fish PowerShell: PS > k3d completion powershell | Out - String | Invoke - Expression # To load completions for every new session, run: PS > k3d completion powershell > k3d . ps1 # and source this file from your PowerShell profile. k3d completion SHELL Options \u00b6 -h, --help help for completion Options inherited from parent commands \u00b6 --timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging) SEE ALSO \u00b6 k3d - https://k3d.io/ -> Run k3s in Docker!","title":"K3d completion"},{"location":"usage/commands/k3d_completion/#k3d-completion","text":"Generate completion scripts for [bash, zsh, fish, powershell | psh]","title":"k3d completion"},{"location":"usage/commands/k3d_completion/#synopsis","text":"To load completions: Bash: $ source < ( k3d completion bash ) # To load completions for each session, execute once: # Linux: $ k3d completion bash > / etc / bash_completion . d / k3d # macOS: $ k3d completion bash > / usr / local / etc / bash_completion . d / k3d Zsh: # If shell completion is not already enabled in your environment, # you will need to enable it. You can execute the following once: $ echo \"autoload -U compinit; compinit\" >> ~/. zshrc # To load completions for each session, execute once: $ k3d completion zsh > \"${fpath[1]}/k3d\" # You will need to start a new shell for this setup to take effect. fish: $ k3d completion fish | source # To load completions for each session, execute once: $ k3d completion fish > ~/. config / fish / completions / k3d . fish PowerShell: PS > k3d completion powershell | Out - String | Invoke - Expression # To load completions for every new session, run: PS > k3d completion powershell > k3d . ps1 # and source this file from your PowerShell profile. k3d completion SHELL","title":"Synopsis"},{"location":"usage/commands/k3d_completion/#options","text":"-h, --help help for completion","title":"Options"},{"location":"usage/commands/k3d_completion/#options-inherited-from-parent-commands","text":"--timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging)","title":"Options inherited from parent commands"},{"location":"usage/commands/k3d_completion/#see-also","text":"k3d - https://k3d.io/ -> Run k3s in Docker!","title":"SEE ALSO"},{"location":"usage/commands/k3d_config/","text":"k3d config \u00b6 Work with config file(s) Synopsis \u00b6 Work with config file(s) k3d config [flags] Options \u00b6 -h, --help help for config Options inherited from parent commands \u00b6 --timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging) SEE ALSO \u00b6 k3d - https://k3d.io/ -> Run k3s in Docker! k3d config init - k3d config migrate -","title":"K3d config"},{"location":"usage/commands/k3d_config/#k3d-config","text":"Work with config file(s)","title":"k3d config"},{"location":"usage/commands/k3d_config/#synopsis","text":"Work with config file(s) k3d config [flags]","title":"Synopsis"},{"location":"usage/commands/k3d_config/#options","text":"-h, --help help for config","title":"Options"},{"location":"usage/commands/k3d_config/#options-inherited-from-parent-commands","text":"--timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging)","title":"Options inherited from parent commands"},{"location":"usage/commands/k3d_config/#see-also","text":"k3d - https://k3d.io/ -> Run k3s in Docker! k3d config init - k3d config migrate -","title":"SEE ALSO"},{"location":"usage/commands/k3d_config_init/","text":"k3d config init \u00b6 k3d config init [flags] Options \u00b6 -f, --force Force overwrite of target file -h, --help help for init -o, --output string Write a default k3d config (default \"k3d-default.yaml\") Options inherited from parent commands \u00b6 --timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging) SEE ALSO \u00b6 k3d config - Work with config file(s)","title":"K3d config init"},{"location":"usage/commands/k3d_config_init/#k3d-config-init","text":"k3d config init [flags]","title":"k3d config init"},{"location":"usage/commands/k3d_config_init/#options","text":"-f, --force Force overwrite of target file -h, --help help for init -o, --output string Write a default k3d config (default \"k3d-default.yaml\")","title":"Options"},{"location":"usage/commands/k3d_config_init/#options-inherited-from-parent-commands","text":"--timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging)","title":"Options inherited from parent commands"},{"location":"usage/commands/k3d_config_init/#see-also","text":"k3d config - Work with config file(s)","title":"SEE ALSO"},{"location":"usage/commands/k3d_config_migrate/","text":"k3d config migrate \u00b6 k3d config migrate INPUT [OUTPUT] [flags] Options \u00b6 -h, --help help for migrate Options inherited from parent commands \u00b6 --timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging) SEE ALSO \u00b6 k3d config - Work with config file(s)","title":"K3d config migrate"},{"location":"usage/commands/k3d_config_migrate/#k3d-config-migrate","text":"k3d config migrate INPUT [OUTPUT] [flags]","title":"k3d config migrate"},{"location":"usage/commands/k3d_config_migrate/#options","text":"-h, --help help for migrate","title":"Options"},{"location":"usage/commands/k3d_config_migrate/#options-inherited-from-parent-commands","text":"--timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging)","title":"Options inherited from parent commands"},{"location":"usage/commands/k3d_config_migrate/#see-also","text":"k3d config - Work with config file(s)","title":"SEE ALSO"},{"location":"usage/commands/k3d_image/","text":"k3d image \u00b6 Handle container images. Synopsis \u00b6 Handle container images. k3d image [flags] Options \u00b6 -h, --help help for image Options inherited from parent commands \u00b6 --timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging) SEE ALSO \u00b6 k3d - https://k3d.io/ -> Run k3s in Docker! k3d image import - Import image(s) from docker into k3d cluster(s).","title":"K3d image"},{"location":"usage/commands/k3d_image/#k3d-image","text":"Handle container images.","title":"k3d image"},{"location":"usage/commands/k3d_image/#synopsis","text":"Handle container images. k3d image [flags]","title":"Synopsis"},{"location":"usage/commands/k3d_image/#options","text":"-h, --help help for image","title":"Options"},{"location":"usage/commands/k3d_image/#options-inherited-from-parent-commands","text":"--timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging)","title":"Options inherited from parent commands"},{"location":"usage/commands/k3d_image/#see-also","text":"k3d - https://k3d.io/ -> Run k3s in Docker! k3d image import - Import image(s) from docker into k3d cluster(s).","title":"SEE ALSO"},{"location":"usage/commands/k3d_image_import/","text":"k3d image import \u00b6 Import image(s) from docker into k3d cluster(s). Synopsis \u00b6 Import image(s) from docker into k3d cluster(s). If an IMAGE starts with the prefix \u2018docker.io/\u2019, then this prefix is stripped internally. That is, \u2018docker.io/rancher/k3d-tools:latest\u2019 is treated as \u2018rancher/k3d-tools:latest\u2019. If an IMAGE starts with the prefix \u2018library/\u2019 (or \u2018docker.io/library/\u2019), then this prefix is stripped internally. That is, \u2018library/busybox:latest\u2019 (or \u2018docker.io/library/busybox:latest\u2019) are treated as \u2018busybox:latest\u2019. If an IMAGE does not have a version tag, then \u2018:latest\u2019 is assumed. That is, \u2018rancher/k3d-tools\u2019 is treated as \u2018rancher/k3d-tools:latest\u2019. A file ARCHIVE always takes precedence. So if a file \u2018./rancher/k3d-tools\u2019 exists, k3d will try to import it instead of the IMAGE of the same name. k3d image import [IMAGE | ARCHIVE [IMAGE | ARCHIVE...]] [flags] Options \u00b6 -c, --cluster stringArray Select clusters to load the image to. (default [k3s-default]) -h, --help help for import -k, --keep-tarball Do not delete the tarball containing the saved images from the shared volume -t, --keep-tools Do not delete the tools node after import Options inherited from parent commands \u00b6 --timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging) SEE ALSO \u00b6 k3d image - Handle container images.","title":"K3d image import"},{"location":"usage/commands/k3d_image_import/#k3d-image-import","text":"Import image(s) from docker into k3d cluster(s).","title":"k3d image import"},{"location":"usage/commands/k3d_image_import/#synopsis","text":"Import image(s) from docker into k3d cluster(s). If an IMAGE starts with the prefix \u2018docker.io/\u2019, then this prefix is stripped internally. That is, \u2018docker.io/rancher/k3d-tools:latest\u2019 is treated as \u2018rancher/k3d-tools:latest\u2019. If an IMAGE starts with the prefix \u2018library/\u2019 (or \u2018docker.io/library/\u2019), then this prefix is stripped internally. That is, \u2018library/busybox:latest\u2019 (or \u2018docker.io/library/busybox:latest\u2019) are treated as \u2018busybox:latest\u2019. If an IMAGE does not have a version tag, then \u2018:latest\u2019 is assumed. That is, \u2018rancher/k3d-tools\u2019 is treated as \u2018rancher/k3d-tools:latest\u2019. A file ARCHIVE always takes precedence. So if a file \u2018./rancher/k3d-tools\u2019 exists, k3d will try to import it instead of the IMAGE of the same name. k3d image import [IMAGE | ARCHIVE [IMAGE | ARCHIVE...]] [flags]","title":"Synopsis"},{"location":"usage/commands/k3d_image_import/#options","text":"-c, --cluster stringArray Select clusters to load the image to. (default [k3s-default]) -h, --help help for import -k, --keep-tarball Do not delete the tarball containing the saved images from the shared volume -t, --keep-tools Do not delete the tools node after import","title":"Options"},{"location":"usage/commands/k3d_image_import/#options-inherited-from-parent-commands","text":"--timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging)","title":"Options inherited from parent commands"},{"location":"usage/commands/k3d_image_import/#see-also","text":"k3d image - Handle container images.","title":"SEE ALSO"},{"location":"usage/commands/k3d_kubeconfig/","text":"k3d kubeconfig \u00b6 Manage kubeconfig(s) Synopsis \u00b6 Manage kubeconfig(s) k3d kubeconfig [flags] Options \u00b6 -h, --help help for kubeconfig Options inherited from parent commands \u00b6 --timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging) SEE ALSO \u00b6 k3d - https://k3d.io/ -> Run k3s in Docker! k3d kubeconfig get - Print kubeconfig(s) from cluster(s). k3d kubeconfig merge - Write/Merge kubeconfig(s) from cluster(s) into new or existing kubeconfig/file.","title":"K3d kubeconfig"},{"location":"usage/commands/k3d_kubeconfig/#k3d-kubeconfig","text":"Manage kubeconfig(s)","title":"k3d kubeconfig"},{"location":"usage/commands/k3d_kubeconfig/#synopsis","text":"Manage kubeconfig(s) k3d kubeconfig [flags]","title":"Synopsis"},{"location":"usage/commands/k3d_kubeconfig/#options","text":"-h, --help help for kubeconfig","title":"Options"},{"location":"usage/commands/k3d_kubeconfig/#options-inherited-from-parent-commands","text":"--timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging)","title":"Options inherited from parent commands"},{"location":"usage/commands/k3d_kubeconfig/#see-also","text":"k3d - https://k3d.io/ -> Run k3s in Docker! k3d kubeconfig get - Print kubeconfig(s) from cluster(s). k3d kubeconfig merge - Write/Merge kubeconfig(s) from cluster(s) into new or existing kubeconfig/file.","title":"SEE ALSO"},{"location":"usage/commands/k3d_kubeconfig_get/","text":"k3d kubeconfig get \u00b6 Print kubeconfig(s) from cluster(s). Synopsis \u00b6 Print kubeconfig(s) from cluster(s). k3d kubeconfig get [CLUSTER [CLUSTER [...]] | --all] [flags] Options \u00b6 -a, --all Output kubeconfigs from all existing clusters -h, --help help for get Options inherited from parent commands \u00b6 --timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging) SEE ALSO \u00b6 k3d kubeconfig - Manage kubeconfig(s)","title":"K3d kubeconfig get"},{"location":"usage/commands/k3d_kubeconfig_get/#k3d-kubeconfig-get","text":"Print kubeconfig(s) from cluster(s).","title":"k3d kubeconfig get"},{"location":"usage/commands/k3d_kubeconfig_get/#synopsis","text":"Print kubeconfig(s) from cluster(s). k3d kubeconfig get [CLUSTER [CLUSTER [...]] | --all] [flags]","title":"Synopsis"},{"location":"usage/commands/k3d_kubeconfig_get/#options","text":"-a, --all Output kubeconfigs from all existing clusters -h, --help help for get","title":"Options"},{"location":"usage/commands/k3d_kubeconfig_get/#options-inherited-from-parent-commands","text":"--timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging)","title":"Options inherited from parent commands"},{"location":"usage/commands/k3d_kubeconfig_get/#see-also","text":"k3d kubeconfig - Manage kubeconfig(s)","title":"SEE ALSO"},{"location":"usage/commands/k3d_kubeconfig_merge/","text":"k3d kubeconfig merge \u00b6 Write/Merge kubeconfig(s) from cluster(s) into new or existing kubeconfig/file. Synopsis \u00b6 Write/Merge kubeconfig(s) from cluster(s) into new or existing kubeconfig/file. k3d kubeconfig merge [CLUSTER [CLUSTER [...]] | --all] [flags] Options \u00b6 -a, --all Get kubeconfigs from all existing clusters -h, --help help for merge -d, --kubeconfig-merge-default Merge into the default kubeconfig ($KUBECONFIG or /home/thklein/.kube/config) -s, --kubeconfig-switch-context Switch to new context (default true) -o, --output string Define output [ - | FILE ] (default from $KUBECONFIG or /home/thklein/.kube/config --overwrite [Careful!] Overwrite existing file, ignoring its contents -u, --update Update conflicting fields in existing kubeconfig (default true) Options inherited from parent commands \u00b6 --timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging) SEE ALSO \u00b6 k3d kubeconfig - Manage kubeconfig(s)","title":"K3d kubeconfig merge"},{"location":"usage/commands/k3d_kubeconfig_merge/#k3d-kubeconfig-merge","text":"Write/Merge kubeconfig(s) from cluster(s) into new or existing kubeconfig/file.","title":"k3d kubeconfig merge"},{"location":"usage/commands/k3d_kubeconfig_merge/#synopsis","text":"Write/Merge kubeconfig(s) from cluster(s) into new or existing kubeconfig/file. k3d kubeconfig merge [CLUSTER [CLUSTER [...]] | --all] [flags]","title":"Synopsis"},{"location":"usage/commands/k3d_kubeconfig_merge/#options","text":"-a, --all Get kubeconfigs from all existing clusters -h, --help help for merge -d, --kubeconfig-merge-default Merge into the default kubeconfig ($KUBECONFIG or /home/thklein/.kube/config) -s, --kubeconfig-switch-context Switch to new context (default true) -o, --output string Define output [ - | FILE ] (default from $KUBECONFIG or /home/thklein/.kube/config --overwrite [Careful!] Overwrite existing file, ignoring its contents -u, --update Update conflicting fields in existing kubeconfig (default true)","title":"Options"},{"location":"usage/commands/k3d_kubeconfig_merge/#options-inherited-from-parent-commands","text":"--timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging)","title":"Options inherited from parent commands"},{"location":"usage/commands/k3d_kubeconfig_merge/#see-also","text":"k3d kubeconfig - Manage kubeconfig(s)","title":"SEE ALSO"},{"location":"usage/commands/k3d_node/","text":"k3d node \u00b6 Manage node(s) Synopsis \u00b6 Manage node(s) k3d node [flags] Options \u00b6 -h, --help help for node Options inherited from parent commands \u00b6 --timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging) SEE ALSO \u00b6 k3d - https://k3d.io/ -> Run k3s in Docker! k3d node create - Create a new k3s node in docker k3d node delete - Delete node(s). k3d node edit - [EXPERIMENTAL] Edit node(s). k3d node list - List node(s) k3d node start - Start an existing k3d node k3d node stop - Stop an existing k3d node","title":"K3d node"},{"location":"usage/commands/k3d_node/#k3d-node","text":"Manage node(s)","title":"k3d node"},{"location":"usage/commands/k3d_node/#synopsis","text":"Manage node(s) k3d node [flags]","title":"Synopsis"},{"location":"usage/commands/k3d_node/#options","text":"-h, --help help for node","title":"Options"},{"location":"usage/commands/k3d_node/#options-inherited-from-parent-commands","text":"--timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging)","title":"Options inherited from parent commands"},{"location":"usage/commands/k3d_node/#see-also","text":"k3d - https://k3d.io/ -> Run k3s in Docker! k3d node create - Create a new k3s node in docker k3d node delete - Delete node(s). k3d node edit - [EXPERIMENTAL] Edit node(s). k3d node list - List node(s) k3d node start - Start an existing k3d node k3d node stop - Stop an existing k3d node","title":"SEE ALSO"},{"location":"usage/commands/k3d_node_create/","text":"k3d node create \u00b6 Create a new k3s node in docker Synopsis \u00b6 Create a new containerized k3s node (k3s in docker). k3d node create NAME [flags] Options \u00b6 -c, --cluster string Select the cluster that the node shall connect to. (default \"k3s-default\") -h, --help help for create -i, --image string Specify k3s image used for the node(s) (default \"docker.io/rancher/k3s:v1.21.4-k3s2\") --k3s-node-label strings Specify k3s node labels in format \"foo=bar\" --memory string Memory limit imposed on the node [From docker] -n, --network strings Add node to (another) runtime network --replicas int Number of replicas of this node specification. (default 1) --role string Specify node role [server, agent] (default \"agent\") --runtime-label strings Specify container runtime labels in format \"foo=bar\" --timeout duration Maximum waiting time for '--wait' before canceling/returning. -t, --token string Override cluster token (required when connecting to an external cluster) --wait Wait for the node(s) to be ready before returning. (default true) Options inherited from parent commands \u00b6 --timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging) SEE ALSO \u00b6 k3d node - Manage node(s)","title":"K3d node create"},{"location":"usage/commands/k3d_node_create/#k3d-node-create","text":"Create a new k3s node in docker","title":"k3d node create"},{"location":"usage/commands/k3d_node_create/#synopsis","text":"Create a new containerized k3s node (k3s in docker). k3d node create NAME [flags]","title":"Synopsis"},{"location":"usage/commands/k3d_node_create/#options","text":"-c, --cluster string Select the cluster that the node shall connect to. (default \"k3s-default\") -h, --help help for create -i, --image string Specify k3s image used for the node(s) (default \"docker.io/rancher/k3s:v1.21.4-k3s2\") --k3s-node-label strings Specify k3s node labels in format \"foo=bar\" --memory string Memory limit imposed on the node [From docker] -n, --network strings Add node to (another) runtime network --replicas int Number of replicas of this node specification. (default 1) --role string Specify node role [server, agent] (default \"agent\") --runtime-label strings Specify container runtime labels in format \"foo=bar\" --timeout duration Maximum waiting time for '--wait' before canceling/returning. -t, --token string Override cluster token (required when connecting to an external cluster) --wait Wait for the node(s) to be ready before returning. (default true)","title":"Options"},{"location":"usage/commands/k3d_node_create/#options-inherited-from-parent-commands","text":"--timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging)","title":"Options inherited from parent commands"},{"location":"usage/commands/k3d_node_create/#see-also","text":"k3d node - Manage node(s)","title":"SEE ALSO"},{"location":"usage/commands/k3d_node_delete/","text":"k3d node delete \u00b6 Delete node(s). Synopsis \u00b6 Delete node(s). k3d node delete (NAME | --all) [flags] Options \u00b6 -a, --all Delete all existing nodes -h, --help help for delete -r, --registries Also delete registries Options inherited from parent commands \u00b6 --timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging) SEE ALSO \u00b6 k3d node - Manage node(s)","title":"K3d node delete"},{"location":"usage/commands/k3d_node_delete/#k3d-node-delete","text":"Delete node(s).","title":"k3d node delete"},{"location":"usage/commands/k3d_node_delete/#synopsis","text":"Delete node(s). k3d node delete (NAME | --all) [flags]","title":"Synopsis"},{"location":"usage/commands/k3d_node_delete/#options","text":"-a, --all Delete all existing nodes -h, --help help for delete -r, --registries Also delete registries","title":"Options"},{"location":"usage/commands/k3d_node_delete/#options-inherited-from-parent-commands","text":"--timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging)","title":"Options inherited from parent commands"},{"location":"usage/commands/k3d_node_delete/#see-also","text":"k3d node - Manage node(s)","title":"SEE ALSO"},{"location":"usage/commands/k3d_node_edit/","text":"k3d node edit \u00b6 [EXPERIMENTAL] Edit node(s). Synopsis \u00b6 [EXPERIMENTAL] Edit node(s). k3d node edit NODE [flags] Options \u00b6 -h, --help help for edit --port-add [HOST:][HOSTPORT:]CONTAINERPORT[/PROTOCOL][@NODEFILTER] [EXPERIMENTAL] (serverlb only!) Map ports from the node container to the host (Format: [HOST:][HOSTPORT:]CONTAINERPORT[/PROTOCOL][@NODEFILTER]) - Example: `k3d node edit k3d-mycluster-serverlb --port-add 8080:80` Options inherited from parent commands \u00b6 --timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging) SEE ALSO \u00b6 k3d node - Manage node(s)","title":"K3d node edit"},{"location":"usage/commands/k3d_node_edit/#k3d-node-edit","text":"[EXPERIMENTAL] Edit node(s).","title":"k3d node edit"},{"location":"usage/commands/k3d_node_edit/#synopsis","text":"[EXPERIMENTAL] Edit node(s). k3d node edit NODE [flags]","title":"Synopsis"},{"location":"usage/commands/k3d_node_edit/#options","text":"-h, --help help for edit --port-add [HOST:][HOSTPORT:]CONTAINERPORT[/PROTOCOL][@NODEFILTER] [EXPERIMENTAL] (serverlb only!) Map ports from the node container to the host (Format: [HOST:][HOSTPORT:]CONTAINERPORT[/PROTOCOL][@NODEFILTER]) - Example: `k3d node edit k3d-mycluster-serverlb --port-add 8080:80`","title":"Options"},{"location":"usage/commands/k3d_node_edit/#options-inherited-from-parent-commands","text":"--timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging)","title":"Options inherited from parent commands"},{"location":"usage/commands/k3d_node_edit/#see-also","text":"k3d node - Manage node(s)","title":"SEE ALSO"},{"location":"usage/commands/k3d_node_list/","text":"k3d node list \u00b6 List node(s) Synopsis \u00b6 List node(s). k3d node list [NODE [NODE...]] [flags] Options \u00b6 -h, --help help for list --no-headers Disable headers -o, --output string Output format. One of: json|yaml Options inherited from parent commands \u00b6 --timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging) SEE ALSO \u00b6 k3d node - Manage node(s)","title":"K3d node list"},{"location":"usage/commands/k3d_node_list/#k3d-node-list","text":"List node(s)","title":"k3d node list"},{"location":"usage/commands/k3d_node_list/#synopsis","text":"List node(s). k3d node list [NODE [NODE...]] [flags]","title":"Synopsis"},{"location":"usage/commands/k3d_node_list/#options","text":"-h, --help help for list --no-headers Disable headers -o, --output string Output format. One of: json|yaml","title":"Options"},{"location":"usage/commands/k3d_node_list/#options-inherited-from-parent-commands","text":"--timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging)","title":"Options inherited from parent commands"},{"location":"usage/commands/k3d_node_list/#see-also","text":"k3d node - Manage node(s)","title":"SEE ALSO"},{"location":"usage/commands/k3d_node_start/","text":"k3d node start \u00b6 Start an existing k3d node Synopsis \u00b6 Start an existing k3d node. k3d node start NODE [flags] Options \u00b6 -h, --help help for start Options inherited from parent commands \u00b6 --timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging) SEE ALSO \u00b6 k3d node - Manage node(s)","title":"K3d node start"},{"location":"usage/commands/k3d_node_start/#k3d-node-start","text":"Start an existing k3d node","title":"k3d node start"},{"location":"usage/commands/k3d_node_start/#synopsis","text":"Start an existing k3d node. k3d node start NODE [flags]","title":"Synopsis"},{"location":"usage/commands/k3d_node_start/#options","text":"-h, --help help for start","title":"Options"},{"location":"usage/commands/k3d_node_start/#options-inherited-from-parent-commands","text":"--timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging)","title":"Options inherited from parent commands"},{"location":"usage/commands/k3d_node_start/#see-also","text":"k3d node - Manage node(s)","title":"SEE ALSO"},{"location":"usage/commands/k3d_node_stop/","text":"k3d node stop \u00b6 Stop an existing k3d node Synopsis \u00b6 Stop an existing k3d node. k3d node stop NAME [flags] Options \u00b6 -h, --help help for stop Options inherited from parent commands \u00b6 --timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging) SEE ALSO \u00b6 k3d node - Manage node(s)","title":"K3d node stop"},{"location":"usage/commands/k3d_node_stop/#k3d-node-stop","text":"Stop an existing k3d node","title":"k3d node stop"},{"location":"usage/commands/k3d_node_stop/#synopsis","text":"Stop an existing k3d node. k3d node stop NAME [flags]","title":"Synopsis"},{"location":"usage/commands/k3d_node_stop/#options","text":"-h, --help help for stop","title":"Options"},{"location":"usage/commands/k3d_node_stop/#options-inherited-from-parent-commands","text":"--timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging)","title":"Options inherited from parent commands"},{"location":"usage/commands/k3d_node_stop/#see-also","text":"k3d node - Manage node(s)","title":"SEE ALSO"},{"location":"usage/commands/k3d_registry/","text":"k3d registry \u00b6 Manage registry/registries Synopsis \u00b6 Manage registry/registries k3d registry [flags] Options \u00b6 -h, --help help for registry Options inherited from parent commands \u00b6 --timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging) SEE ALSO \u00b6 k3d - https://k3d.io/ -> Run k3s in Docker! k3d registry create - Create a new registry k3d registry delete - Delete registry/registries. k3d registry list - List registries","title":"K3d registry"},{"location":"usage/commands/k3d_registry/#k3d-registry","text":"Manage registry/registries","title":"k3d registry"},{"location":"usage/commands/k3d_registry/#synopsis","text":"Manage registry/registries k3d registry [flags]","title":"Synopsis"},{"location":"usage/commands/k3d_registry/#options","text":"-h, --help help for registry","title":"Options"},{"location":"usage/commands/k3d_registry/#options-inherited-from-parent-commands","text":"--timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging)","title":"Options inherited from parent commands"},{"location":"usage/commands/k3d_registry/#see-also","text":"k3d - https://k3d.io/ -> Run k3s in Docker! k3d registry create - Create a new registry k3d registry delete - Delete registry/registries. k3d registry list - List registries","title":"SEE ALSO"},{"location":"usage/commands/k3d_registry_create/","text":"k3d registry create \u00b6 Create a new registry Synopsis \u00b6 Create a new registry. k3d registry create NAME [flags] Options \u00b6 -h, --help help for create -i, --image string Specify image used for the registry (default \"docker.io/library/registry:2\") --no-help Disable the help text (How-To use the registry) -p, --port [HOST:]HOSTPORT Select which port the registry should be listening on on your machine (localhost) (Format: [HOST:]HOSTPORT) - Example: `k3d registry create --port 0.0.0.0:5111` (default \"random\") Options inherited from parent commands \u00b6 --timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging) SEE ALSO \u00b6 k3d registry - Manage registry/registries","title":"K3d registry create"},{"location":"usage/commands/k3d_registry_create/#k3d-registry-create","text":"Create a new registry","title":"k3d registry create"},{"location":"usage/commands/k3d_registry_create/#synopsis","text":"Create a new registry. k3d registry create NAME [flags]","title":"Synopsis"},{"location":"usage/commands/k3d_registry_create/#options","text":"-h, --help help for create -i, --image string Specify image used for the registry (default \"docker.io/library/registry:2\") --no-help Disable the help text (How-To use the registry) -p, --port [HOST:]HOSTPORT Select which port the registry should be listening on on your machine (localhost) (Format: [HOST:]HOSTPORT) - Example: `k3d registry create --port 0.0.0.0:5111` (default \"random\")","title":"Options"},{"location":"usage/commands/k3d_registry_create/#options-inherited-from-parent-commands","text":"--timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging)","title":"Options inherited from parent commands"},{"location":"usage/commands/k3d_registry_create/#see-also","text":"k3d registry - Manage registry/registries","title":"SEE ALSO"},{"location":"usage/commands/k3d_registry_delete/","text":"k3d registry delete \u00b6 Delete registry/registries. Synopsis \u00b6 Delete registry/registries. k3d registry delete (NAME | --all) [flags] Options \u00b6 -a, --all Delete all existing registries -h, --help help for delete Options inherited from parent commands \u00b6 --timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging) SEE ALSO \u00b6 k3d registry - Manage registry/registries","title":"K3d registry delete"},{"location":"usage/commands/k3d_registry_delete/#k3d-registry-delete","text":"Delete registry/registries.","title":"k3d registry delete"},{"location":"usage/commands/k3d_registry_delete/#synopsis","text":"Delete registry/registries. k3d registry delete (NAME | --all) [flags]","title":"Synopsis"},{"location":"usage/commands/k3d_registry_delete/#options","text":"-a, --all Delete all existing registries -h, --help help for delete","title":"Options"},{"location":"usage/commands/k3d_registry_delete/#options-inherited-from-parent-commands","text":"--timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging)","title":"Options inherited from parent commands"},{"location":"usage/commands/k3d_registry_delete/#see-also","text":"k3d registry - Manage registry/registries","title":"SEE ALSO"},{"location":"usage/commands/k3d_registry_list/","text":"k3d registry list \u00b6 List registries Synopsis \u00b6 List registries. k3d registry list [NAME [NAME...]] [flags] Options \u00b6 -h, --help help for list --no-headers Disable headers -o, --output string Output format. One of: json|yaml Options inherited from parent commands \u00b6 --timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging) SEE ALSO \u00b6 k3d registry - Manage registry/registries","title":"K3d registry list"},{"location":"usage/commands/k3d_registry_list/#k3d-registry-list","text":"List registries","title":"k3d registry list"},{"location":"usage/commands/k3d_registry_list/#synopsis","text":"List registries. k3d registry list [NAME [NAME...]] [flags]","title":"Synopsis"},{"location":"usage/commands/k3d_registry_list/#options","text":"-h, --help help for list --no-headers Disable headers -o, --output string Output format. One of: json|yaml","title":"Options"},{"location":"usage/commands/k3d_registry_list/#options-inherited-from-parent-commands","text":"--timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging)","title":"Options inherited from parent commands"},{"location":"usage/commands/k3d_registry_list/#see-also","text":"k3d registry - Manage registry/registries","title":"SEE ALSO"},{"location":"usage/commands/k3d_version/","text":"k3d version \u00b6 Show k3d and default k3s version Synopsis \u00b6 Show k3d and default k3s version k3d version [flags] Options \u00b6 -h, --help help for version Options inherited from parent commands \u00b6 --timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging) SEE ALSO \u00b6 k3d - https://k3d.io/ -> Run k3s in Docker!","title":"K3d version"},{"location":"usage/commands/k3d_version/#k3d-version","text":"Show k3d and default k3s version","title":"k3d version"},{"location":"usage/commands/k3d_version/#synopsis","text":"Show k3d and default k3s version k3d version [flags]","title":"Synopsis"},{"location":"usage/commands/k3d_version/#options","text":"-h, --help help for version","title":"Options"},{"location":"usage/commands/k3d_version/#options-inherited-from-parent-commands","text":"--timestamps Enable Log timestamps --trace Enable super verbose output (trace logging) --verbose Enable verbose output (debug logging)","title":"Options inherited from parent commands"},{"location":"usage/commands/k3d_version/#see-also","text":"k3d - https://k3d.io/ -> Run k3s in Docker!","title":"SEE ALSO"}]}
\ No newline at end of file
diff --git a/v5.0.0/sitemap.xml b/v5.0.0/sitemap.xml
index ed0c5590..8d3e72cf 100644
--- a/v5.0.0/sitemap.xml
+++ b/v5.0.0/sitemap.xml
@@ -2,212 +2,212 @@
 <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
     <url>
          <loc>https://k3d.io/v5.0.0/</loc>
-         <lastmod>2021-10-05</lastmod>
+         <lastmod>2021-10-11</lastmod>
          <changefreq>daily</changefreq>
     </url>
     <url>
          <loc>https://k3d.io/v5.0.0/design/defaults/</loc>
-         <lastmod>2021-10-05</lastmod>
+         <lastmod>2021-10-11</lastmod>
          <changefreq>daily</changefreq>
     </url>
     <url>
          <loc>https://k3d.io/v5.0.0/design/networking/</loc>
-         <lastmod>2021-10-05</lastmod>
+         <lastmod>2021-10-11</lastmod>
          <changefreq>daily</changefreq>
     </url>
     <url>
          <loc>https://k3d.io/v5.0.0/design/project/</loc>
-         <lastmod>2021-10-05</lastmod>
+         <lastmod>2021-10-11</lastmod>
          <changefreq>daily</changefreq>
     </url>
     <url>
          <loc>https://k3d.io/v5.0.0/faq/faq/</loc>
-         <lastmod>2021-10-05</lastmod>
+         <lastmod>2021-10-11</lastmod>
          <changefreq>daily</changefreq>
     </url>
     <url>
          <loc>https://k3d.io/v5.0.0/usage/commands/</loc>
-         <lastmod>2021-10-05</lastmod>
+         <lastmod>2021-10-11</lastmod>
          <changefreq>daily</changefreq>
     </url>
     <url>
          <loc>https://k3d.io/v5.0.0/usage/configfile/</loc>
-         <lastmod>2021-10-05</lastmod>
+         <lastmod>2021-10-11</lastmod>
          <changefreq>daily</changefreq>
     </url>
     <url>
          <loc>https://k3d.io/v5.0.0/usage/exposing_services/</loc>
-         <lastmod>2021-10-05</lastmod>
+         <lastmod>2021-10-11</lastmod>
          <changefreq>daily</changefreq>
     </url>
     <url>
          <loc>https://k3d.io/v5.0.0/usage/kubeconfig/</loc>
-         <lastmod>2021-10-05</lastmod>
+         <lastmod>2021-10-11</lastmod>
          <changefreq>daily</changefreq>
     </url>
     <url>
          <loc>https://k3d.io/v5.0.0/usage/multiserver/</loc>
-         <lastmod>2021-10-05</lastmod>
+         <lastmod>2021-10-11</lastmod>
          <changefreq>daily</changefreq>
     </url>
     <url>
          <loc>https://k3d.io/v5.0.0/usage/registries/</loc>
-         <lastmod>2021-10-05</lastmod>
+         <lastmod>2021-10-11</lastmod>
          <changefreq>daily</changefreq>
     </url>
     <url>
          <loc>https://k3d.io/v5.0.0/usage/advanced/calico/</loc>
-         <lastmod>2021-10-05</lastmod>
+         <lastmod>2021-10-11</lastmod>
          <changefreq>daily</changefreq>
     </url>
     <url>
          <loc>https://k3d.io/v5.0.0/usage/advanced/cuda/</loc>
-         <lastmod>2021-10-05</lastmod>
+         <lastmod>2021-10-11</lastmod>
          <changefreq>daily</changefreq>
     </url>
     <url>
          <loc>https://k3d.io/v5.0.0/usage/commands/k3d/</loc>
-         <lastmod>2021-10-05</lastmod>
+         <lastmod>2021-10-11</lastmod>
          <changefreq>daily</changefreq>
     </url>
     <url>
          <loc>https://k3d.io/v5.0.0/usage/commands/k3d_cluster/</loc>
-         <lastmod>2021-10-05</lastmod>
+         <lastmod>2021-10-11</lastmod>
          <changefreq>daily</changefreq>
     </url>
     <url>
          <loc>https://k3d.io/v5.0.0/usage/commands/k3d_cluster_create/</loc>
-         <lastmod>2021-10-05</lastmod>
+         <lastmod>2021-10-11</lastmod>
          <changefreq>daily</changefreq>
     </url>
     <url>
          <loc>https://k3d.io/v5.0.0/usage/commands/k3d_cluster_delete/</loc>
-         <lastmod>2021-10-05</lastmod>
+         <lastmod>2021-10-11</lastmod>
          <changefreq>daily</changefreq>
     </url>
     <url>
          <loc>https://k3d.io/v5.0.0/usage/commands/k3d_cluster_edit/</loc>
-         <lastmod>2021-10-05</lastmod>
+         <lastmod>2021-10-11</lastmod>
          <changefreq>daily</changefreq>
     </url>
     <url>
          <loc>https://k3d.io/v5.0.0/usage/commands/k3d_cluster_list/</loc>
-         <lastmod>2021-10-05</lastmod>
+         <lastmod>2021-10-11</lastmod>
          <changefreq>daily</changefreq>
     </url>
     <url>
          <loc>https://k3d.io/v5.0.0/usage/commands/k3d_cluster_start/</loc>
-         <lastmod>2021-10-05</lastmod>
+         <lastmod>2021-10-11</lastmod>
          <changefreq>daily</changefreq>
     </url>
     <url>
          <loc>https://k3d.io/v5.0.0/usage/commands/k3d_cluster_stop/</loc>
-         <lastmod>2021-10-05</lastmod>
+         <lastmod>2021-10-11</lastmod>
          <changefreq>daily</changefreq>
     </url>
     <url>
          <loc>https://k3d.io/v5.0.0/usage/commands/k3d_completion/</loc>
-         <lastmod>2021-10-05</lastmod>
+         <lastmod>2021-10-11</lastmod>
          <changefreq>daily</changefreq>
     </url>
     <url>
          <loc>https://k3d.io/v5.0.0/usage/commands/k3d_config/</loc>
-         <lastmod>2021-10-05</lastmod>
+         <lastmod>2021-10-11</lastmod>
          <changefreq>daily</changefreq>
     </url>
     <url>
          <loc>https://k3d.io/v5.0.0/usage/commands/k3d_config_init/</loc>
-         <lastmod>2021-10-05</lastmod>
+         <lastmod>2021-10-11</lastmod>
          <changefreq>daily</changefreq>
     </url>
     <url>
          <loc>https://k3d.io/v5.0.0/usage/commands/k3d_config_migrate/</loc>
-         <lastmod>2021-10-05</lastmod>
+         <lastmod>2021-10-11</lastmod>
          <changefreq>daily</changefreq>
     </url>
     <url>
          <loc>https://k3d.io/v5.0.0/usage/commands/k3d_image/</loc>
-         <lastmod>2021-10-05</lastmod>
+         <lastmod>2021-10-11</lastmod>
          <changefreq>daily</changefreq>
     </url>
     <url>
          <loc>https://k3d.io/v5.0.0/usage/commands/k3d_image_import/</loc>
-         <lastmod>2021-10-05</lastmod>
+         <lastmod>2021-10-11</lastmod>
          <changefreq>daily</changefreq>
     </url>
     <url>
          <loc>https://k3d.io/v5.0.0/usage/commands/k3d_kubeconfig/</loc>
-         <lastmod>2021-10-05</lastmod>
+         <lastmod>2021-10-11</lastmod>
          <changefreq>daily</changefreq>
     </url>
     <url>
          <loc>https://k3d.io/v5.0.0/usage/commands/k3d_kubeconfig_get/</loc>
-         <lastmod>2021-10-05</lastmod>
+         <lastmod>2021-10-11</lastmod>
          <changefreq>daily</changefreq>
     </url>
     <url>
          <loc>https://k3d.io/v5.0.0/usage/commands/k3d_kubeconfig_merge/</loc>
-         <lastmod>2021-10-05</lastmod>
+         <lastmod>2021-10-11</lastmod>
          <changefreq>daily</changefreq>
     </url>
     <url>
          <loc>https://k3d.io/v5.0.0/usage/commands/k3d_node/</loc>
-         <lastmod>2021-10-05</lastmod>
+         <lastmod>2021-10-11</lastmod>
          <changefreq>daily</changefreq>
     </url>
     <url>
          <loc>https://k3d.io/v5.0.0/usage/commands/k3d_node_create/</loc>
-         <lastmod>2021-10-05</lastmod>
+         <lastmod>2021-10-11</lastmod>
          <changefreq>daily</changefreq>
     </url>
     <url>
          <loc>https://k3d.io/v5.0.0/usage/commands/k3d_node_delete/</loc>
-         <lastmod>2021-10-05</lastmod>
+         <lastmod>2021-10-11</lastmod>
          <changefreq>daily</changefreq>
     </url>
     <url>
          <loc>https://k3d.io/v5.0.0/usage/commands/k3d_node_edit/</loc>
-         <lastmod>2021-10-05</lastmod>
+         <lastmod>2021-10-11</lastmod>
          <changefreq>daily</changefreq>
     </url>
     <url>
          <loc>https://k3d.io/v5.0.0/usage/commands/k3d_node_list/</loc>
-         <lastmod>2021-10-05</lastmod>
+         <lastmod>2021-10-11</lastmod>
          <changefreq>daily</changefreq>
     </url>
     <url>
          <loc>https://k3d.io/v5.0.0/usage/commands/k3d_node_start/</loc>
-         <lastmod>2021-10-05</lastmod>
+         <lastmod>2021-10-11</lastmod>
          <changefreq>daily</changefreq>
     </url>
     <url>
          <loc>https://k3d.io/v5.0.0/usage/commands/k3d_node_stop/</loc>
-         <lastmod>2021-10-05</lastmod>
+         <lastmod>2021-10-11</lastmod>
          <changefreq>daily</changefreq>
     </url>
     <url>
          <loc>https://k3d.io/v5.0.0/usage/commands/k3d_registry/</loc>
-         <lastmod>2021-10-05</lastmod>
+         <lastmod>2021-10-11</lastmod>
          <changefreq>daily</changefreq>
     </url>
     <url>
          <loc>https://k3d.io/v5.0.0/usage/commands/k3d_registry_create/</loc>
-         <lastmod>2021-10-05</lastmod>
+         <lastmod>2021-10-11</lastmod>
          <changefreq>daily</changefreq>
     </url>
     <url>
          <loc>https://k3d.io/v5.0.0/usage/commands/k3d_registry_delete/</loc>
-         <lastmod>2021-10-05</lastmod>
+         <lastmod>2021-10-11</lastmod>
          <changefreq>daily</changefreq>
     </url>
     <url>
          <loc>https://k3d.io/v5.0.0/usage/commands/k3d_registry_list/</loc>
-         <lastmod>2021-10-05</lastmod>
+         <lastmod>2021-10-11</lastmod>
          <changefreq>daily</changefreq>
     </url>
     <url>
          <loc>https://k3d.io/v5.0.0/usage/commands/k3d_version/</loc>
-         <lastmod>2021-10-05</lastmod>
+         <lastmod>2021-10-11</lastmod>
          <changefreq>daily</changefreq>
     </url>
 </urlset>
\ No newline at end of file
diff --git a/v5.0.0/sitemap.xml.gz b/v5.0.0/sitemap.xml.gz
index e153818275f64a44c0cec921696288c30be3a028..7f6c220d9e0540c1a222da542e3dc88bd85f2adb 100644
GIT binary patch
literal 477
zcmV<30V4h%iwFqXJ!D}5|8r?{Wo=<_E_iKh0M*&eZk#X>0N{I`V#R$iB#P8lEV=aw
z+B0(P!8=|55RchS-oA!Kt34!AE|ok;{MnZE$>3pt?&Zzd-5DZb@Q13cn`#GM1&!Vu
zs^7nU?!Q$p{X;iLOOSTCv8O|IOlkTd#N~3S2@^OuQ5~WYq>lN7#8xPp`g_$s?DDd7
za6MX=&IZ*ly-@Dv^rUMH;{04Ubt5{l)ZI6glw8pLWAoVV+h*Ukxye>`*GwIyH!wzc
z?X^Vv-if=Ct3lilU*~8bG(PV}Iom{q@;szVh^Ktf3I$Fv`~{Ut(~a_VT?=E)Bn=1^
zoReNtsX9NR8PP(SK6sk~<q0tm#4{==bQn+bAGi)>nLd}J9xg)&u}okDgOnncdV!@r
zORqo+B`vB@I&#*s@MxGT%NGvH&Xh`>GapF@6+tFo1lC}I6&N%oMqHcYDkCl>8F_ib
z#DN>!l;>G~J!F9`H<S7RI-utf5w2Q?Ch~Yhpj(QVIP-Q2EJ9c5FDsv52LrrDcW@9{
z+`m3(E=)f8?W$#y+^$+CneA$2lGd(PE;;RnWs%TsAB^4^QTfEKo0v`fx`~;@h3#M6
T|Cg>ig4F*5%mX9|Bpd($AWY!l

literal 477
zcmV<30V4h%iwFoG6<lEg|8r?{Wo=<_E_iKh0M*&eZlf>|0N{I{BH|uQT2<Oz5qj$r
zv}fengENhPsK+Gj+t&~j<<OP(vXTdhKijfC89WToK7Bg7IYA^0eqZhCrrLm4L8CYO
z>h<M$`>T5D?%F9@g0#tvJ?*PQO5-0P&gXMYn83-A>JW_}b;u_q)<V(LzpL(Ulb5-J
z%h9s5HmGjyg>pB?2VG+jr^mXf8_|ln?zX9<<bv+*oBQ2%*KC`|++;1gX{HX+8yF&d
zbXubQYQ;^-#UQSTf9Gf*G(PVJIoU*o@;sz-h{t@<3I&cayn{-m=?3|+tc9^Ak_Lnd
z&PlJSRGlBt3}~TDAAF91@`UIK;t3TLI*iBZ4K71jrqAW5hw~6ZEE5>PAf<?<USO$D
z(kswHNsB6!j-0eCJR0W8@`b&!6Qxq;%umu@MUV*?fi+lQ1qO|Y5!dFp%7{xzMqb`9
za^MCx=6RN14_RQ#&7^*S_UL&;gp1api98+==;k6O&b%E1i_lg2%gSG{y#ZdM+dGIX
z?q44?7bc(lcG0p)ZWk?+%yzLdNoyA?mz;LRvPfvx4@O^%sC;7ARm>)SUByh|!uqf7
T|4Wx0LF&E%lYM!IBpd($^_=Z&

diff --git a/v5.0.0/usage/advanced/calico/index.html b/v5.0.0/usage/advanced/calico/index.html
index b5271572..721ed1ca 100644
--- a/v5.0.0/usage/advanced/calico/index.html
+++ b/v5.0.0/usage/advanced/calico/index.html
@@ -1042,12 +1042,12 @@
 <h2 id="2-create-the-cluster-without-flannel-and-with-calico">2. Create the cluster without flannel and with calico<a class="headerlink" href="#2-create-the-cluster-without-flannel-and-with-calico" title="Permanent link">&para;</a></h2>
 <p>On the k3s cluster creation :</p>
 <ul>
-<li>add the flag <code>--flannel-backend=none</code>. For this, on k3d you need to forward this flag to k3s with the option <code>--k3s-server-arg</code>.</li>
+<li>add the flag <code>--flannel-backend=none</code>. For this, on k3d you need to forward this flag to k3s with the option <code>--k3s-arg</code>.</li>
 <li>mount (<code>--volume</code>) the calico descriptor in the auto deploy manifest directory of k3s <code>/var/lib/rancher/k3s/server/manifests/</code></li>
 </ul>
 <p>So the command of the cluster creation is (when you are at root of the k3d repository)</p>
 <div class="highlight"><pre><span></span><code>k3d cluster create <span class="s2">&quot;</span><span class="si">${</span><span class="nv">clustername</span><span class="si">}</span><span class="s2">&quot;</span> <span class="se">\</span>
-  --k3s-server-arg <span class="s1">&#39;--flannel-backend=none&#39;</span> <span class="se">\</span>
+  --k3s-arg <span class="s1">&#39;--flannel-backend=none@server:*&#39;</span> <span class="se">\</span>
   --volume <span class="s2">&quot;</span><span class="k">$(</span><span class="nb">pwd</span><span class="k">)</span><span class="s2">/docs/usage/guides/calico.yaml:/var/lib/rancher/k3s/server/manifests/calico.yaml&quot;</span>
 </code></pre></div>
 <p>In this example :</p>
@@ -1098,7 +1098,7 @@ coredns-8655855d6-cxtnr                    <span class="m">1</span>/1     Runnin
 <div class="md-source-date">
   <small>
     
-      Last update: <span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">September 17, 2021</span>
+      Last update: <span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">October 11, 2021</span>
       
     
   </small>
diff --git a/v5.0.0/usage/commands/index.html b/v5.0.0/usage/commands/index.html
index 1facc235..5cfb4c74 100644
--- a/v5.0.0/usage/commands/index.html
+++ b/v5.0.0/usage/commands/index.html
@@ -960,8 +960,7 @@
       -e, --env  <span class="c1"># add environment variables to the nodes (quoted string, format: &#39;KEY[=VALUE][@NODEFILTER[;NODEFILTER...]]&#39;, use flag multiple times)</span>
       --gpus  <span class="c1"># [from docker CLI] add GPU devices to the node containers (string, e.g. &#39;all&#39;)</span>
       -i, --image  <span class="c1"># specify which k3s image should be used for the nodes (string, default: &#39;docker.io/rancher/k3s:v1.20.0-k3s2&#39;, tag changes per build)</span>
-      --k3s-agent-arg  <span class="c1"># add additional arguments to the k3s agent (quoted string, use flag multiple times) (see https://rancher.com/docs/k3s/latest/en/installation/install-options/agent-config/#k3s-agent-cli-help)</span>
-      --k3s-server-arg  <span class="c1"># add additional arguments to the k3s server (quoted string, use flag multiple times) (see https://rancher.com/docs/k3s/latest/en/installation/install-options/server-config/#k3s-server-cli-help)</span>
+      --k3s-arg  <span class="c1"># add additional arguments to the k3s server/agent (quoted string, use flag multiple times) (see https://rancher.com/docs/k3s/latest/en/installation/install-options/server-config/#k3s-server-cli-help &amp; https://rancher.com/docs/k3s/latest/en/installation/install-options/agent-config/#k3s-agent-cli-help)</span>
       --kubeconfig-switch-context  <span class="c1"># (implies --kubeconfig-update-default) automatically sets the current-context of your default kubeconfig to the new cluster&#39;s context (default: true)</span>
       --kubeconfig-update-default  <span class="c1"># enable the automated update of the default kubeconfig with the details of the newly created cluster (also sets &#39;--wait=true&#39;) (default: true)</span>
       -l, --label  <span class="c1"># add (docker) labels to the node containers (format: &#39;KEY[=VALUE][@NODEFILTER[;NODEFILTER...]]&#39;, use flag multiple times)</span>
@@ -1044,7 +1043,7 @@
 <div class="md-source-date">
   <small>
     
-      Last update: <span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">March 30, 2021</span>
+      Last update: <span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">October 11, 2021</span>
       
     
   </small>
diff --git a/v5.0.0/usage/registries/index.html b/v5.0.0/usage/registries/index.html
index 10449bc8..0ab10826 100644
--- a/v5.0.0/usage/registries/index.html
+++ b/v5.0.0/usage/registries/index.html
@@ -1258,7 +1258,7 @@ For example, if we mount the CA file in <code>/etc/ssl/certs/my-company-root.pem
 <h4 id="create-a-customized-k3d-managed-registry">Create a customized k3d-managed registry<a class="headerlink" href="#create-a-customized-k3d-managed-registry" title="Permanent link">&para;</a></h4>
 <ol>
 <li><code class="highlight">k3d registry create myregistry.localhost --port <span class="m">12345</span></code> creates a new registry called <code>k3d-myregistry.localhost</code> (could be used with automatic resolution of <code>*.localhost</code>, see next section - also, <strong>note the <code>k3d-</code> prefix</strong> that k3d adds to all resources it creates)</li>
-<li><code class="highlight">k3d cluster create newcluster --registry-use k3d-myregistry.localhost:12345</code> (make sure you use the <strong><code>k3d-</code> prefix</strong> here) creates a new cluster set up to us that registry</li>
+<li><code class="highlight">k3d cluster create newcluster --registry-use k3d-myregistry.localhost:12345</code> (make sure you use the <strong><code>k3d-</code> prefix</strong> here) creates a new cluster set up to use that registry</li>
 <li>continue with step 3 and 4 from the last section for testing</li>
 </ol>
 <!-- Admonition to describe usage of a non-k3d-managed registry -->
@@ -1336,7 +1336,7 @@ docker push k3d-registry.localhost:5000/nginx:latest
 <div class="md-source-date">
   <small>
     
-      Last update: <span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">September 17, 2021</span>
+      Last update: <span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">October 11, 2021</span>
       
     
   </small>