In the integration tests, we were accessing the API of the Talos machines which are in maintenance mode by directly hitting their SideroLink mgmt endpoint.
This worked only because the test was running on the same host as Omni itself (as we spawned Omni as process). This approach breaks when we install Omni via its helm chart on a Kubernetes cluster.
Fix this by going to them through Omni as well.
Additionally, centralize the talos client creation in the tests.
Additionally: bump Talos machinery, and pass the service account key explicitly to the Talos client when creating it, instead of relying on it to pick it from env vars.
Signed-off-by: Utku Ozdemir <utku.ozdemir@siderolabs.com>
Picker can retry if it couldn't pick enough machines, but we set the picked flags only once and never reset them.
Change the approach to take it into account.
Signed-off-by: Utku Ozdemir <utku.ozdemir@siderolabs.com>
META section updates are no-op for non-UKI machines, but still, the recent changes in the kernel args PR started clearing them (since now we compute schematic ID always), causing the schematic ID to be updated, which caused cluster machines to be upgraded and restarted.
Remove the UKI check and keep meta valus always as-is.
Update the integration tests to:
- Also include META values.
- Make Omni upgrade test pick both UKI and non-UKI machines.
Signed-off-by: Utku Ozdemir <utku.ozdemir@siderolabs.com>
(Re)implement the kernel args support functionality in the following way:
- Only support UKI or UKI-like (>=1.12 with GrubUseUKICmdline) systems.
- In `MachineStatusController`:
- When we see a machine for the first time, do a one-time operation of extracting of the extra kernel args from it and store them in the newly introduced `KernelArgs` resource. This resource is user-owned from that point on.
- Mark the `MachineStatus` with an annotation as "its kernel args are initialized".
- Start storing the the raw schematic.
- Take a one-time snapshot of the extensions on the machine and set them as "initial extensions". They might not be the "actual initial", i.e., the set of extensions when we actually seen the machine for the first time, but we do this in a best-effort basis. We need this, since now we cannot simply go back to the initial schematic ID when all extensions are removed - kernel args are also included in the schematic.
- Start collecting the kernel cmdline from Talos machines as well.
- Adapt the `SchematicConfiguration` controller to not revert to the initial schematic ID ever - it now always computes the needed schematic - when it wants to revert to the initial set of extensions, it uses the new field on the `MachineStatus`.
- Introduce the resource `MachineUpgradeStatus` and its controller `MachineUpgradeStatusController`, which handles the maintenance mode upgrades when kernel args are updated. The controller is named this way, since our long-term plan is to centralize all upgrade calls to be done from this controller. Currently, it does not change Talos version or the set of extensions. It works only in maintenance mode, only for kernel args changes (when supported).
- Introduce the resource `KernelArgsStatus` and its controller `KernelArgsStatusController`, which provides information about the kernel args updates. Its status is reliable in both maintenance and non-maintenance modes.
- Build a UI to update these args (with @Unix4ever's help).
Co-authored-by: Artem Chernyshev <artem.chernyshev@talos-systems.com>
Signed-off-by: Utku Ozdemir <utku.ozdemir@siderolabs.com>
