When Talos runs in a container, `ethtool` availability depends on host
kernel support, and we don't strictly need `ethtool` to make networking
work, so make it optional instead of hard failure.
Example: https://gist.github.com/rgl/392d6e16d176f28430230b06ec80496c
Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
This is going to be useful in the third party code which is using
upgrade modules, to collect output logs instead of printing them to the
stdout.
Signed-off-by: Artem Chernyshev <artem.0xD2@gmail.com>
Route scope doesn't depend on destination IP type being link-local, e.g.
in Azure route to link local address is create with gateway, and that
should be global (universe) scope route.
Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
Fixes#3847Fixes#3919
1. Looks like `::1/128` is assigned to `lo` interface by the kernel
without our help, and kernel does it properly whether IPv6 is enabled
for not (including particular interface).
2. If IPv6 is disabled completely with command line, we should ignore
failures to write ipv6 sysctls (as these are not security-related,
skipping them isn't a risk).
Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
We run etcd health check every 30s, and create/destroy client every 30s.
This puts a lot of pressure on etcd itself and machined.
There's protobuf overhead, TLS connection overhead, etc.
As we don't support changing etcd PKI (yet), client created once is good
enough for the lifetime of the node.
Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
Resources code extensively uses DeepCopy to prevent in-memory copy of
the resource to be mutated outside of the resource model.
Previous implementation relied on YAML serialization to copy the
machine configuration which was slow, potentially might lead to panics
and it generates pressure on garbage collection.
This implementation uses k8s code generator to generate DeepCopy methods
with some manual helpers when code generator can't handle it.
Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
This allows us to use tags for Go submodules `pkg/machinery/v0.11.0` and
still keeps Talos tag follow semantic version `v0.11.0`.
Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
This simply uses new protobuf package instead of old one.
Old protobuf package is still in use by Talos dependencies.
Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
Plus, convert a few absolute URLs with a version number to relative URLs without versions.
Signed-off-by: Alexey Palazhchenko <alexey.palazhchenko@gmail.com>
This extends network device machine configuration validation to make
sure that bond slaves don't have any addressing methods set, as this
might run into a conflict with the bond setup.
Also makes sure no interface is part of two bonds.
Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
The problem was that gRPC method `status.Code(err)` doesn't unwrap
errors, while Talos client returns errors wrapped with
`multierror.Error` and `fmt.Errrorf`, so `status.Code` doesn't return
error code correctly.
Fix that by introducing our own client method which correctly goes over
the chain of wrapped errors.
Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
I believe `clientv3.SetEndpoints()` calls doesn't make etcd client
connect to the endpoints mentioned immediately, it might stil reuse old
connection (?).
At the same time `MaintenanceClient` which implements `MoveLeader` calls
doesn't support explicit endpoint setting (as other similar calls do),
so we have to manually force the connection to the leader node we need.
Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
The problem is that there's no official way to close Kuberentes client
underlying TCP/HTTP connections. So each time Talos initializes
connection to the control plane endpoint, new client is built, but this
client is never closed, so the connection stays active on the load
balancers, on the API server level, etc. It also eats some resources out
of Talos itself.
We add a way to close underlying connections by using helper from the
Kubernetes client libraries to force close all TCP connections which
should shut down all HTTP/2 connections as well.
Alternative approach might be to cache a client for some time, but many
of the clients are created with temporary PKI, so even cached client
still needs to be closed once it gets stale, and it's not clear how to
recreate a client in case existing one is broken for one reason or
another (and we need to force a re-connection).
Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
When forfeiting etcd leadership, it might be that the node still reports
leadership status while not being a leader once the actual API call is
used. We should ignore such an error as the node is not a leader.
Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
This should fix an error like:
```
failed to create etcd client: error getting kubernetes endpoints: Unauthorized
```
The problem is that the generated cert was used immediately, so even
slight time sync issue across nodes might render the cert not (yet)
usable. Cert is generated on one node, but might be used on any other
node (as it goes via the LB).
Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
This PR bumps pkgs to v0.7.0-alpha.0, so that we gain a fix for
hotplugging of nvme drives.
Signed-off-by: Spencer Smith <robertspencersmith@gmail.com>
This removes `retrying error` messages while waiting for the API server
pod state to reflect changes from the updated static pod definition.
Log more lines to notify about the progress.
Skip `kube-proxy` if not found (as we allow it to be disabled).
```
$ talosctl upgrade-k8s -n 172.20.0.2 --from 1.21.0 --to 1.21.2
discovered master nodes ["172.20.0.2" "172.20.0.3" "172.20.0.4"]
updating "kube-apiserver" to version "1.21.2"
> "172.20.0.2": starting update
> "172.20.0.2": machine configuration patched
> "172.20.0.2": waiting for API server state pod update
< "172.20.0.2": successfully updated
> "172.20.0.3": starting update
> "172.20.0.3": machine configuration patched
> "172.20.0.3": waiting for API server state pod update
< "172.20.0.3": successfully updated
> "172.20.0.4": starting update
> "172.20.0.4": machine configuration patched
> "172.20.0.4": waiting for API server state pod update
< "172.20.0.4": successfully updated
updating "kube-controller-manager" to version "1.21.2"
> "172.20.0.2": starting update
> "172.20.0.2": machine configuration patched
> "172.20.0.2": waiting for API server state pod update
< "172.20.0.2": successfully updated
> "172.20.0.3": starting update
> "172.20.0.3": machine configuration patched
> "172.20.0.3": waiting for API server state pod update
< "172.20.0.3": successfully updated
> "172.20.0.4": starting update
> "172.20.0.4": machine configuration patched
> "172.20.0.4": waiting for API server state pod update
< "172.20.0.4": successfully updated
updating "kube-scheduler" to version "1.21.2"
> "172.20.0.2": starting update
> "172.20.0.2": machine configuration patched
> "172.20.0.2": waiting for API server state pod update
< "172.20.0.2": successfully updated
> "172.20.0.3": starting update
> "172.20.0.3": machine configuration patched
> "172.20.0.3": waiting for API server state pod update
< "172.20.0.3": successfully updated
> "172.20.0.4": starting update
> "172.20.0.4": machine configuration patched
> "172.20.0.4": waiting for API server state pod update
< "172.20.0.4": successfully updated
updating daemonset "kube-proxy" to version "1.21.2"
kube-proxy skipped as DaemonSet was not found
```
Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
Fixes#3861
What this change effectively does is that it changes immediate reconcile
request to an error return, so that controller will be restarted with a
backoff.
More details:
* root cause of the update/teardown conflict is that the finalizer is
still pending on the tearing down resource
* finalizer might not be removed immediately, e.g. if the controller
which put the finalizer is itself in the crash loop
* if the merge controller queues reconcile immediately, it restarts
itself, but the finalizer is still there, so it once again goes into
reconcile loop and that goes forever until the finalizer is removed, so
instead if the controller fails, it will be restarted with exponential
backoff lowering the load on the system
Change is validated with the unit-tests reproducing the conflict.
Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
With the recent changes, bootstrap API might wait for the time to be in
sync (as the apid is launched before time is sync). We set timeout to
500ms for the bootstrap API call, so there's a chance that a call might
time out, and we should ignore it.
Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
This fixes endless block on RemoteGenerator.Close method rewriting the
RemoteGenerator using the retry package.
Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
This commit also introduces a hidden `--json` flag for `talosctl version` command
that is not supported and should be re-worked at #907.
Refs #3852.
Signed-off-by: Alexey Palazhchenko <alexey.palazhchenko@gmail.com>
Basically all delay options are interlocked with `miimon`: if `miimon`
is zero, all delays are set to zero, and kernel complains even if zero
delay attribute is sent while miimon is zero.
Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
The sequence of events to reproduce the problem:
* some resource was merged as final representation with ID `x`
* underlying source resource gets destroyed
* merge controller marks final resource `x` for teardown and waits
for the finalizers to be empty
* another source resource appears which gets merged to same final `x`
* as `x` is in the teardown phase, spec controller will ignore it
* merge controller doesn't see the problem as well, as `x` spec is
correct, but the phase is wrong (which merge controller ignores)
This pulls in COSI fix to return an error if a resource in teardown
phase is modified. This way merge controller knows that the resource `x`
is in the teardown phase, so it should be first fully torn down, and
then new representation should be re-created as new resource with same ID
`x`.
Regression unit-tests included (they don't reproduce the sequence of
events always reliably, but they do with 10% probability).
Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
This PR makes sure we pin to a known CAPI version because with the new
v0.4.x released, we'll fail until we support the v1alpha4 APIs.
Signed-off-by: Spencer Smith <robertspencersmith@gmail.com>
