--- title: "Vagrant & Libvirt" aliases: - ../../../virtualized-platforms/vagrant-libvirt --- ## Pre-requisities 1. Linux OS 2. [Vagrant](https://www.vagrantup.com) installed 3. [vagrant-libvirt](https://github.com/vagrant-libvirt/vagrant-libvirt) plugin installed 4. [talosctl](https://www.talos.dev/{{< version >}}/introduction/getting-started/#talosctl) installed 5. [kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl) installed ## Overview We will use Vagrant and its libvirt plugin to create a KVM-based cluster with 3 control plane nodes and 1 worker node. For this, we will mount Talos ISO into the VMs using a virtual CD-ROM, and configure the VMs to attempt to boot from the disk first with the fallback to the CD-ROM. We will also configure a virtual IP address on Talos to achieve high-availability on kube-apiserver. ## Preparing the environment First, we download the latest `talos-amd64.iso` ISO from GitHub releases into the `/tmp` directory. ```bash wget --timestamping https://github.com/siderolabs/talos/releases/download/{{< release >}}/talos-amd64.iso -O /tmp/talos-amd64.iso ``` Create a `Vagrantfile` with the following contents: ```ruby Vagrant.configure("2") do |config| config.vm.define "control-plane-node-1" do |vm| vm.vm.provider :libvirt do |domain| domain.cpus = 2 domain.memory = 2048 domain.serial :type => "file", :source => {:path => "/tmp/control-plane-node-1.log"} domain.storage :file, :device => :cdrom, :path => "/tmp/talos-amd64.iso" domain.storage :file, :size => '4G', :type => 'raw' domain.boot 'hd' domain.boot 'cdrom' end end config.vm.define "control-plane-node-2" do |vm| vm.vm.provider :libvirt do |domain| domain.cpus = 2 domain.memory = 2048 domain.serial :type => "file", :source => {:path => "/tmp/control-plane-node-2.log"} domain.storage :file, :device => :cdrom, :path => "/tmp/talos-amd64.iso" domain.storage :file, :size => '4G', :type => 'raw' domain.boot 'hd' domain.boot 'cdrom' end end config.vm.define "control-plane-node-3" do |vm| vm.vm.provider :libvirt do |domain| domain.cpus = 2 domain.memory = 2048 domain.serial :type => "file", :source => {:path => "/tmp/control-plane-node-3.log"} domain.storage :file, :device => :cdrom, :path => "/tmp/talos-amd64.iso" domain.storage :file, :size => '4G', :type => 'raw' domain.boot 'hd' domain.boot 'cdrom' end end config.vm.define "worker-node-1" do |vm| vm.vm.provider :libvirt do |domain| domain.cpus = 1 domain.memory = 1024 domain.serial :type => "file", :source => {:path => "/tmp/worker-node-1.log"} domain.storage :file, :device => :cdrom, :path => "/tmp/talos-amd64.iso" domain.storage :file, :size => '4G', :type => 'raw' domain.boot 'hd' domain.boot 'cdrom' end end end ``` ## Bring up the nodes Check the status of vagrant VMs: ```bash vagrant status ``` You should see the VMs in "not created" state: ```text Current machine states: control-plane-node-1 not created (libvirt) control-plane-node-2 not created (libvirt) control-plane-node-3 not created (libvirt) worker-node-1 not created (libvirt) ``` Bring up the vagrant environment: ```bash vagrant up --provider=libvirt ``` Check the status again: ```bash vagrant status ``` Now you should see the VMs in "running" state: ```text Current machine states: control-plane-node-1 running (libvirt) control-plane-node-2 running (libvirt) control-plane-node-3 running (libvirt) worker-node-1 running (libvirt) ``` Find out the IP addresses assigned by the libvirt DHCP by running: ```bash virsh list | grep vagrant | awk '{print $2}' | xargs -t -L1 virsh domifaddr ``` Output will look like the following: ```text virsh domifaddr vagrant_control-plane-node-2 Name MAC address Protocol Address ------------------------------------------------------------------------------- vnet0 52:54:00:f9:10:e5 ipv4 192.168.121.119/24 virsh domifaddr vagrant_control-plane-node-1 Name MAC address Protocol Address ------------------------------------------------------------------------------- vnet1 52:54:00:0f:ae:59 ipv4 192.168.121.203/24 virsh domifaddr vagrant_worker-node-1 Name MAC address Protocol Address ------------------------------------------------------------------------------- vnet2 52:54:00:6f:28:95 ipv4 192.168.121.69/24 virsh domifaddr vagrant_control-plane-node-3 Name MAC address Protocol Address ------------------------------------------------------------------------------- vnet3 52:54:00:03:45:10 ipv4 192.168.121.125/24 ``` Our control plane nodes have the IPs: `192.168.121.203`, `192.168.121.119`, `192.168.121.125` and the worker node has the IP `192.168.121.69`. Now you should be able to interact with Talos nodes that are in maintenance mode: ```bash talosctl -n 192.168.121.203 disks --insecure ``` Sample output: ```text DEV MODEL SERIAL TYPE UUID WWID MODALIAS NAME SIZE BUS_PATH /dev/vda - - HDD - - virtio:d00000002v00001AF4 - 8.6 GB /pci0000:00/0000:00:03.0/virtio0/ ``` ## Installing Talos Pick an endpoint IP in the `vagrant-libvirt` subnet but not used by any nodes, for example `192.168.121.100`. Generate a machine configuration: ```bash talosctl gen config my-cluster https://192.168.121.100:6443 --install-disk /dev/vda ``` Edit `controlplane.yaml` to add the virtual IP you picked to a network interface under `.spec.machine.network.interfaces`, for example: ```yaml spec: machine: network: interfaces: - interface: eth0 dhcp: true vip: ip: 192.168.121.100 ``` Apply the configuration to the initial control plane node: ```bash talosctl -n 192.168.121.203 apply-config --insecure --file controlplane.yaml ``` You can tail the logs of the node: ```bash sudo tail -f /tmp/control-plane-node-1.log ``` Set up your shell to use the generated talosconfig and configure its endpoints (use the IPs of the control plane nodes): ```bash export TALOSCONFIG=$(realpath ./talosconfig) talosctl config endpoint 192.168.121.203 192.168.121.119 192.168.121.125 ``` Bootstrap the Kubernetes cluster from the initial control plane node: ```bash talosctl -n 192.168.121.203 bootstrap ``` Finally, apply the machine configurations to the remaining nodes: ```bash talosctl -n 192.168.121.119 apply-config --insecure --file controlplane.yaml talosctl -n 192.168.121.125 apply-config --insecure --file controlplane.yaml talosctl -n 192.168.121.69 apply-config --insecure --file worker.yaml ``` After a while, you should see that all the members have joined: ```bash talosctl -n 192.168.121.203 get members ``` The output will be like the following: ```text NODE NAMESPACE TYPE ID VERSION HOSTNAME MACHINE TYPE OS ADDRESSES 192.168.121.203 cluster Member talos-192-168-121-119 1 talos-192-168-121-119 controlplane Talos (v1.1.0) ["192.168.121.119"] 192.168.121.203 cluster Member talos-192-168-121-69 1 talos-192-168-121-69 worker Talos (v1.1.0) ["192.168.121.69"] 192.168.121.203 cluster Member talos-192-168-121-203 6 talos-192-168-121-203 controlplane Talos (v1.1.0) ["192.168.121.100","192.168.121.203"] 192.168.121.203 cluster Member talos-192-168-121-125 1 talos-192-168-121-125 controlplane Talos (v1.1.0) ["192.168.121.125"] ``` ## Interacting with Kubernetes cluster Retrieve the kubeconfig from the cluster: ```bash talosctl -n 192.168.121.203 kubeconfig ./kubeconfig ``` List the nodes in the cluster: ```bash kubectl --kubeconfig ./kubeconfig get node -owide ``` You will see an output similar to: ```text NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME talos-192-168-121-203 Ready control-plane,master 3m10s v1.24.2 192.168.121.203 Talos (v1.1.0) 5.15.48-talos containerd://1.6.6 talos-192-168-121-69 Ready 2m25s v1.24.2 192.168.121.69 Talos (v1.1.0) 5.15.48-talos containerd://1.6.6 talos-192-168-121-119 Ready control-plane,master 8m46s v1.24.2 192.168.121.119 Talos (v1.1.0) 5.15.48-talos containerd://1.6.6 talos-192-168-121-125 Ready control-plane,master 3m11s v1.24.2 192.168.121.125 Talos (v1.1.0) 5.15.48-talos containerd://1.6.6 ``` Congratulations, you have a highly-available Talos cluster running! ## Cleanup You can destroy the vagrant environment by running: ```bash vagrant destroy -f ``` And remove the ISO image you downloaded: ```bash sudo rm -f /tmp/talos-amd64.iso ```