This PR will update k8s to the latest 1.18 release and bump sonobuoy to
help resolve some e2e flakes. Also adds some retry logic around the
sonobuoy run.
Signed-off-by: Spencer Smith <robertspencersmith@gmail.com>
Since the `--once` option of `extlinux` seems to only work with BIOS, we
needed to change to remove any reliance on this option. Instead of
booting the upgraded version once, and then making it the default after
a successful boot, we now make it the default, and then revert on any
boot error.
Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
This PR will pull in the latest release of k8s 1.18 so we can start
validating it through our test suite.
Signed-off-by: Spencer Smith <robertspencersmith@gmail.com>
This PR introduces a new strategy for upgrades. Instead of attempting to
zap the partition table, create a new one, and then format the
partitions, this change will only update the `vmlinuz`, and
`initramfs.xz` being used to boot. It introduces an A/B style upgrade
process, which will allow for easy rollbacks. One deviation from our
original intention with upgrades is that this change does not completely
reset a node. It falls just short of that and does not reset the
partition table. This forces us to keep the current partition scheme in
mind as we make changes in the future, because an upgrade assumes a
specific partition scheme. We can improve upgrades further in the
future, but this will at least make them more dependable. Finally, one
more feature in this PR is the ability to keep state. This enables
single node clusters to upgrade since we keep the etcd data around.
Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
BREAKING CHANGE: This PR fixes a bug where we were only passing `cluster.local` to the
kubelet configuration. It will also pull in a new version of the
bootkube fork to ensure that custom domains got propogated down to the
API Server certs, as well as the CoreDNS configuration for a cluster.
Existing users should be aware that, if they were previously trying to
use this option in machine configs, that an upgrade will may break
their cluster. It will update a kubelet flag with the new domain, but
CoreDNS and API Server certs will not change since bootkube has already
run. One option may be to change these values manually inside the
Kubernetes cluster. However, it may prove easier to rebuild the cluster
if necessary.
Additionally, this PR also exposes a flag to `osctl config generate`
to allow tweaking this domain value as well.
Signed-off-by: Spencer Smith <robertspencersmith@gmail.com>
This PR allows Talos to respect the panic=0 flag if users pass that in
their kernel args. Doing this makes it easier to catch kernel panics in
debug scenarios and allows the user to manually trigger a restart with
ctrl+alt+del when they're ready.
Signed-off-by: Spencer Smith <robertspencersmith@gmail.com>
This PR will clean up bootkube assets regardless of whether bootkube
succeeds. This will allow for a failed bootkube deployment to retry on
reboot.
Signed-off-by: Spencer Smith <robertspencersmith@gmail.com>
This PR will allow users to set the `persist: true` value in their
config data to tell talos not to re-pull the config data at each reboot.
The default will still remain as a "pull every time" methodolgy in order
to encourage immutability by default.
Signed-off-by: Spencer Smith <robertspencersmith@gmail.com>
New service `routerd` performs exactly single task: based on incoming
API call service name, it routes the requests to the appropriate Talos
service (`networkd`, `osd`, etc.) Service `routerd` listens of file
socket and routes requests to file sockets.
Service `apid` now does single task as well:
* it either fans out request to other `apid` services running on other
nodes and aggregates responses
* or it forwards requests to local `routerd` (when request destination
is local node)
Cons:
* one more proxying layer on request path
Pros:
* more clear service roles
* `routerd` is part of core Talos, services should register with it to
expose their API; no auth in the service (not exposed to the world)
* `apid` might be replaced with other implementation, it depends on TLS infra,
auth, etc.
* `apid` is better segregated from other Talos services (can only access
`routerd`, can't talk to other Talos services directly, so less exposure
in case of a bug)
This change is no-op to the end users.
Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
This extracts admin kubeconfig generation out of bootkube, now based on
Talos x509 library. On each API request for `kubeconfig`, config is
generated on the fly and sent back on the wire.
This fixes two issues:
* any master node can now generate `kubeconfig` (worker nodes can do
that too, but that should probably change in the future)
* after upgrade-and-wipe the disk scenario, `osctl kubeconfig` still
works
Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
This class of tests is included/excluded by build tags, but as it is
pretty different from other integration tests, we build it as separate
executable. Provision tests provision cluster for the test run, perform
some actions and verify results (could be upgrade, reset, scale up/down,
etc.)
There's now framework to implement upgrade tests, first of the tests
tests upgrade from latest 0.3 (0.3.2 at the moment) to current version
of Talos (being built in CI). Tests starts by booting with 0.3
kernel/initramfs, runs 0.3 installer to install 0.3.2 cluster, wait for
bootstrap, followed by upgrade to 0.4 in rolling fashion. As Firecracker
supports bootloader, this boots 0.4 system from boot disk (as installed
by installer).
Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
This implements `osctl cluster destroy` for Firecracker, adds
new utility command `osctl cluser show`.
Firecracker mode now has control process for firecracker VMs, allowing
clean reboots and background operations.
Lots of small fixes to Firecracker mode, clean CNI shutdown, cleaning up
netns, etc.
Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
This PR allows for pod checkpointer and coredns images to be customized
for bootkube. We can already customize the hyperkube image and all other
images used by bootkube are CNI-related and can be customized with the
"custom" CNI setup.
Signed-off-by: Spencer Smith <robertspencersmith@gmail.com>
This is initial PR to push the initial code, it has several known
problems which are going to be addressed in follow-up PRs:
1. there's no "cluster destroy", so the only way to stop the VMs is to
`pkill firecracker`
2. provisioner creates state in `/tmp` and never deletes it, that is
required to keep cluster running when `osctl cluster create` finishes
3. doesn't run any controller process around firecracker to support
reboots/CNI cleanup (vethxyz interfaces are lingering on the host as
they're never cleaned up)
The plan is to create some structure in `~/.talos` to manage cluster
state, e.g. `~/.talos/clusters/<name>` which will contain all the
required files (disk images, file sockets, VM logs, etc.). This
directory structure will also work as a way to detect running clusters
and clean them up.
For point number 3, `osctl cluster create` is going to exec lightweight
process to control the firecracker VM process and to simulate VM reboots
if firecracker finishes cleanly (when VM reboots).
Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
This adds a help menu to the Makefile. It documents all build
dependencies, and how to get started.
Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
Primarily doc/constant changes.
Added additionnal bits to `docs` target in makefile to generate osctl
docs as well as config files. Explicitly define a HOME variable so we
get consistent home directories for talosconfig variables in our docs.
Signed-off-by: Brad Beam <brad.beam@talos-systems.com>
When we upgrade a node, we kill off all pods before performing a fresh
install. The issue with this is that we run the risk of killing the CNI
pod before we finish killing all other pods, leaving the CRI unable to
teardown the pod's networking. This works around that by first killing
any pods running without host networking so that the CNI can do its'
job, and then removing the remaining pods.
Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
This PR will allow users to specify one or many URLs for CNI so that
they can bypass bootkube deploying flannel and bring their own. Will
close#1593
Signed-off-by: Spencer Smith <robertspencersmith@gmail.com>
This replaces codegen version of apid proxying with
talos-systems/grpc-proxy based version. Proxying is transparent, it
doesn't require exact information about methods and response types. It
requires some common layout response to enhance it properly with node
metadata or errors.
There should be no signifcant changes to the API with the previous
version, but it's worth mentioning a few changes:
1. grpc.ClientConn is established just once per upstream (either local
service or remote apid instance).
2. When called without `-t` (`targets`), apid proxies immediately down
to local service skipping proxying to itself (as before), which results
in empty node metadata in response (before it had local node IP). Might
revert this later to proxy to itself (?).
3. Streaming APIs are now fully supported with multiple targets, but
message definition doesn't contain `ResponseMetadata`, so streaming APIs
are broken now with targets (needs a fix).
4. Errors are now returned as responses with `Error` field set in
`ResponseMetadata`, this requires client library update and `osctl` to
handle it properly.
Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
Push versions to constants, introduce 'platform' to version API to
discover node mode. Check kernel version for non-containers.
A bit of refactoring on version package to expose something closer to a
single response.
Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
This introduces the notion of metadata for a node. In this initial pass
there are only two fields. A timestamp to indicate when the install was
performed, and a field to indicate if the install was performed as part
of an upgrade.
Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
Since bootkube should only be ran once, we need a way to determine if it
has already been ran. This makes use of etcd to store a key-value pair
indicating that the cluster has been initialized.
Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
This PR introduces APId. This service replaces the frontend functionality
previously provided by OSD. The main driver for this is two fold:
1. Create a single purpose application to expose the talos api
2. Make use of code generation to DRY api changes
Signed-off-by: Brad Beam <brad.beam@talos-systems.com>
This brings in a few minor improvements to the metal platform. The first
is to use talos.config=metal-iso to indicate that the machine's config
can be found in an ISO image. The second is a fix to ensure that /mnt
exists.
This adds support for creating more than one node using the qemu-boot.sh
script.
Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
We have decided that proxyd is not the best architectue for HA
Kubernetes. Our recommendation to users will be to create a load
balancer instead.
Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
Memory usage reduced around 8-10x: now it stays stable at 1GB.
I disabled some of the new linters, and one rule which is violated a
lot.
I might make sense to go back and enable `wsl` fixing all the issues
(leaving that for another PR).
Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
This adds more methods to the Cluster interface that allows for more
granular control of the cluster network settings.
Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
