k3d/docs/registries.md

Using registries with k3d

Registries configuration file

You can add registries by specifying them in a registries.yaml in your $HOME/.k3d directory. This file will be loaded automatically by k3d if present and will be shared between all your k3d clusters, but you can also use a specific file for a new cluster with the --registries-file argument.

This file is a regular k3s registries configuration file, and looks like this:

mirrors:
  "my.company.registry:5000":
    endpoint:
      - http://my.company.registry:5000

In this example, an image with a name like my.company.registry:5000/nginx:latest would be pulled from the registry running at http://my.company.registry:5000.

Note well there is an important limitation: this configuration file will only work with k3s >= v0.10.0. It will fail silently with previous versions of k3s, but you find in the section below an alternative solution.

This file can also be used for providing additional information necessary for accessing some registries, like authentication and certificates.

Authenticated registries

When using authenticated registries, we can add the username and password in a configs section in the registries.yaml, like this:

mirrors:
  my.company.registry:
    endpoint:
      - http://my.company.registry

configs:
  my.company.registry:
    auth:
      username: aladin
      password: abracadabra

Secure registries

When using secure registries, the registries.yaml file must include information about the certificates. For example, if you want to use images from the secure registry running at https://my.company.registry, you must first download a CA file valid for that server and store it in some well-known directory like ${HOME}/.k3d/my-company-root.pem.

Then you have to mount the CA file in some directory in the nodes in the cluster and include that mounted file in a configs section in the registries.yaml file. For example, if we mount the CA file in /etc/ssl/certs/my-company-root.pem, the registries.yaml will look like:

mirrors:
  my.company.registry:
    endpoint:
      - https://my.company.registry

configs:
  my.company.registry:
    tls:
      # we will mount "my-company-root.pem" in the /etc/ssl/certs/ directory.
      ca_file: "/etc/ssl/certs/my-company-root.pem"

Finally, we can create the cluster, mounting the CA file in the path we specified in ca_file:

k3d create --volume ${HOME}/.k3d/my-company-root.pem:/etc/ssl/certs/my-company-root.pem ...

Using a local registry

Using the k3d registry

k3d can manage a local registry that you can use for pushing your images to, and your k3d nodes will be able to use those images automatically. k3d will create the registry for you and connect it to your k3d cluster. It is important to note that this registry will be shared between all your k3d clusters, and it will be released when the last of the k3d clusters that was using it is deleted.

In order to enable the k3d registry when creating a new cluster, you must run k3d with the --enable-registry argument

k3d create --enable-registry ...

Then you must add an entry in /etc/hosts as described in the next section. And then you should check you local registry.

Using your own local registry

If you don't want k3d to manage your registry, you can start it with some docker commands, like:

docker volume create local_registry
docker container run -d --name registry.local -v local_registry:/var/lib/registry --restart always -p 5000:5000 registry:2

These commands will start you registry in registry.local:5000. In order to push to this registry, you will need to add the line at /etc/hosts as we described in the previous section . Once your registry is up and running, we will need to add it to your registries.yaml configuration file. Finally, you must connect the registry network to the k3d cluster network: docker network connect k3d-k3s-default registry.local. And then you can check you local registry.

Pushing to your local registry address

The registry will be located, by default, at registry.local:5000 (customizable with the --registry-name and --registry-port parameters). All the nodes in your k3d cluster can resolve this hostname (thanks to the DNS server provided by the Docker daemon) but, in order to be able to push to this registry, this hostname but also be resolved from your host.

The easiest solution for this is to add an entry in your /etc/hosts file like this:

127.0.0.1 registry.local

Once again, this will only work with k3s >= v0.10.0 (see the section below when using k3s <= v0.9.1)

Local registry volume

The local k3d registry uses a volume for storying the images. This volume will be destroyed when the k3d registry is released. In order to persist this volume and make these images survive the removal of the registry, you can specify a volume with the --registry-volume and use the --keep-registry-volume flag when deleting the cluster. This will create a volume with the given name the first time the registry is used, while successive invocations will just mount this existing volume in the k3d registry container.

Docker Hub cache

The local k3d registry can also be used for caching images from the Docker Hub. You can start the registry as a pull-through cache when the cluster is created with --enable-registry-cache. Used in conjuction with --registry-volume/--keep-registry-volume can speed up all the downloads from the Hub by keeping a persistent cache of images in your local machine.

Testing your registry

You should test that you can

• push to your registry from your local development machine.
• use images from that registry in Deployments in your k3d cluster.

We will verify these two things for a local registry (located at registry.local:5000) running in your development machine. Things would be basically the same for checking an external registry, but some additional configuration could be necessary in your local machine when using an authenticated or secure registry (please refer to Docker's documentation for this).

Firstly, we can download some image (like nginx) and push it to our local registry with:

docker pull nginx:latest
docker tag nginx:latest registry.local:5000/nginx:latest
docker push registry.local:5000/nginx:latest

Then we can deploy a pod referencing this image to your cluster:

cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-test-registry
  labels:
    app: nginx-test-registry
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx-test-registry
  template:
    metadata:
      labels:
        app: nginx-test-registry
    spec:
      containers:
      - name: nginx-test-registry
        image: registry.local:5000/nginx:latest
        ports:
        - containerPort: 80
EOF

Then you should check that the pod is running with kubectl get pods -l "app=nginx-test-registry".

Configuring registries for k3s <= v0.9.1

k3s servers below v0.9.1 do not recognize the registries.yaml file as we described in the previous section, so you will need to embed the contents of that file in a containerd configuration file. You will have to create your own containerd configuration file at some well-known path like ${HOME}/.k3d/config.toml.tmpl, like this:

# Original section: no changes
[plugins.opt]
path = "{{ .NodeConfig.Containerd.Opt }}"
[plugins.cri]
stream_server_address = "{{ .NodeConfig.AgentConfig.NodeName }}"
stream_server_port = "10010"
{{- if .IsRunningInUserNS }}
disable_cgroup = true
disable_apparmor = true
restrict_oom_score_adj = true
{{ end -}}
{{- if .NodeConfig.AgentConfig.PauseImage }}
sandbox_image = "{{ .NodeConfig.AgentConfig.PauseImage }}"
{{ end -}}
{{- if not .NodeConfig.NoFlannel }}
  [plugins.cri.cni]
    bin_dir = "{{ .NodeConfig.AgentConfig.CNIBinDir }}"
    conf_dir = "{{ .NodeConfig.AgentConfig.CNIConfDir }}"
{{ end -}}

# Added section: additional registries and the endpoints
[plugins.cri.registry.mirrors]
  [plugins.cri.registry.mirrors."registry.local:5000"]
    endpoint = ["http://registry.local:5000"]

and then mount it at /var/lib/rancher/k3s/agent/etc/containerd/config.toml.tmpl (where the containerd in your k3d nodes will load it) when creating the k3d cluster:

k3d create \
    --volume ${HOME}/.k3d/config.toml.tmpl:/var/lib/rancher/k3s/agent/etc/containerd/config.toml.tmpl