318 Commits

Author SHA1 Message Date
Andrey Smirnov
c8ae00937e chore: bump dependencies via dependabot
See #3226, #3227, #3228

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-03-01 06:16:53 -08:00
Andrey Smirnov
d7cdc8cc15 feat: implement simple layer 2 shared IP for CP
This adds a VIP (virtual IP) option to the network configuration of an
interface, which will allow a set of nodes to share a floating IP
address among them.  For now, this is restricted to control plane use
and only a single shared IP is supported.

Fixes #3111

Signed-off-by: Seán C McCord <ulexus@gmail.com>
Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-02-26 14:14:34 -08:00
Andrey Smirnov
4e19b597ab test: add integration test with Canal CNI and reset API
Canal CNI is known to be trying to reach out to k8s control plane on pod
teardown.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-02-24 11:34:02 -08:00
Andrey Smirnov
85d1669fb0 chore: bump dependencies via dependabot
PRs: #3202, #3201, #3200

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-02-24 07:05:55 -08:00
Andrey Smirnov
84ad6cbb1a chore: switch CI to stop embedding local registry into the builds
This adds new `IMAGE_REGISTRY` variable (similar to `IMAGE_TAG`) which
affects only the registry image gets pushed to, but it's not built into
the binaries and images as a default registry.

This fixes a problem when release builds reference our CI local
registry.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-02-24 18:05:37 +03:00
Andrey Smirnov
e9fc54f6e3 feat: update Kubernetes to 1.20.3
https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.20.md#changelog-since-v1202

Also updater pkgs for:

* talos-systems/pkgs#238 (raspberrypi-firmware update)
* talos-systems/pkgs#242 (Linux 5.10.17 + init_on_free=0)

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-02-19 05:22:34 -08:00
Artem Chernyshev
54d6a45217 feat: add state encryption support
State partition encryption support adds a new section to the machine config.
And a new step to the sequencer flow which saves encryption
configuration object as json serialized value in the META partition.

Everything else is the same as is for the ephemeral partition.
Additionally enabled state partition encryption in the disk encryption
integration tests.

Signed-off-by: Artem Chernyshev <artem.0xD2@gmail.com>
2021-02-18 06:55:22 -08:00
Artem Chernyshev
58ff2c9808 feat: implement ephemeral partition encryption
This PR introduces the first part of disk encryption support.
New config section `systemDiskEncryption` was added into MachineConfig.
For now it contains only Ephemeral partition encryption.

Encryption itself supports two kinds of keys for now:
- node id deterministic key.
- static key which is hardcoded in the config and mainly used for test
purposes.

Talosctl cluster create can now be told to encrypt ephemeral partition
by using `--encrypt-ephemeral` flag.

Additionally:
- updated pkgs library version.
- changed Dockefile to copy cryptsetup deps from pkgs.

Signed-off-by: Artem Chernyshev <artem.0xD2@gmail.com>
2021-02-17 13:39:04 -08:00
Spencer Smith
6207fa517b test: update aws templates
This PR moves to using our new cluster-api-templates for AWS.

Signed-off-by: Spencer Smith <robertspencersmith@gmail.com>
2021-02-17 16:08:48 -05:00
Spencer Smith
f1d1f72b58 chore(ci): update gcp templates
This PR moves to using the official CAPG provider release, as well as
making use of our snazzy new capi templates.

Signed-off-by: Spencer Smith <robertspencersmith@gmail.com>
2021-02-17 10:33:41 -05:00
Andrey Smirnov
d29a56b0c0 chore: update dependencies via dependabot
See #3116, #3117

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-02-08 07:20:51 -08:00
Andrey Smirnov
8974b529af chore: bump dependencies (via dependabot)
See #3072, #3073, #3074, #3076, #3077, #3078

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-02-01 05:48:34 -08:00
Andrey Smirnov
d71ac4c4ff feat: update Kubernetes to 1.20.2
Minor point release, official changelog:

https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.20.md

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2021-01-15 09:06:18 -08:00
Andrey Smirnov
a8dd2ff30d fix: checkpoint controller-manager and scheduler
Default manifests created by bootkube so far were only enabling
pod-checkpointer for kube-apiserver. This seems to have issues with
single-node control plane scenario, when without scheduler and
controller-manager node might fall into `NodeAffinity` state.

See https://github.com/talos-systems/bootkube-plugin/pull/23

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2020-12-28 11:53:17 -08:00
Artem Chernyshev
73c81c501e fix: pass disk image flags to e2e-qemu cluster create command
Forgot to add it in the original PR.

Signed-off-by: Artem Chernyshev <artem.0xD2@gmail.com>
2020-12-22 23:57:31 -08:00
Artem Chernyshev
6540e9bf70 feat: support disk image in talosctl cluster create
Fixes: https://github.com/talos-systems/talos/issues/2973

Can now supply disk image using `--disk-image-path` flag.
May need to enable `--with-apply-config` if it's necessary to bootstrap
nodes properly.

Signed-off-by: Artem Chernyshev <artem.0xD2@gmail.com>
2020-12-22 17:06:00 +03:00
Andrey Smirnov
b1d4814308 feat: update Kubernetes to 1.20.1
See https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.20.md

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2020-12-21 23:52:29 +03:00
Artem Chernyshev
f5b55f41fa docs: add fallback to default page description if none is set on current page
Also filled in some of the descriptions specific for particular pages.

Signed-off-by: Artem Chernyshev <artem.0xD2@gmail.com>
2020-12-17 13:03:27 -08:00
Andrey Smirnov
9d1ac81be5 chore: lower MTU to 1450 for the tests in the CI
This should help with the CNI encapsulation in the cluster.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2020-12-17 17:14:07 +03:00
Andrey Smirnov
803d566b85 chore: add drone pipeline to upload cloud images
At the moment only AMIs for AWS, and upload results should be pushed
back to the docs as separate PR.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2020-12-16 20:50:35 +03:00
Andrew Rynhard
6f979d463c test: add ISO test
Adds a simple test for the ISO. Boots the ISO, and then uses the `apply-config` command
in `talosctl` to create a cluster.

Signed-off-by: Andrew Rynhard <andrew@rynhard.io>
2020-12-10 13:04:29 -08:00
Andrey Smirnov
80184393bc feat: update kernel to 5.9.13, new KSPP requirements
Pulls in following changes:

* https://github.com/talos-systems/toolchain/pull/20
* https://github.com/talos-systems/tools/pull/116
* https://github.com/talos-systems/pkgs/pull/214
* https://github.com/talos-systems/pkgs/pull/215
* https://github.com/talos-systems/pkgs/pull/216
* https://github.com/talos-systems/pkgs/pull/217
* https://github.com/talos-systems/go-procfs/pull/4

New empty amd64 images for u-boot & rpi-firmware reduce the size of
amd64 installer image.

For backwards compatibility QEMU provisioner still injects "legacy" KSPP
kernel args into initial boot environment.

Installer correctly upgrades KSPP options when moving from one version
of Talos to another.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2020-12-10 12:41:58 -08:00
Andrey Smirnov
872e792dbc feat: update Kubernetes to 1.20.0
Official K8s release matching Talos 0.8.0.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2020-12-09 06:11:48 -08:00
Andrey Smirnov
5d5697e398 chore: limit unit-test run concurrency
As we run unit-tests concurrently, it makes sense to limit each run
concurrency.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2020-12-07 10:46:02 -08:00
Andrey Smirnov
11c2b8f80c test: bump defaults for provision tests resources
Our defaults are too low now.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2020-12-07 07:01:41 -08:00
Andrey Smirnov
621968977e feat: update kubernetes to 1.20.0-rc.0
Talos 0.8 is going to ship with K8s 1.20.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2020-12-02 10:50:58 -08:00
Andrey Smirnov
61cacb3956 docs: provide list of AMIs on AWS documentation page
I wasn't able to find a more elegant way to get Vue content into
Markdown, so a bit of a hack with the template. Looks like frontmatter
parsing is also incomplete, so I could rely only on the page title.

Also changed the format of the image export to make it much easier to
work on that data in Vue.

Fixes #2834

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2020-11-25 10:26:50 -08:00
Andrey Smirnov
28ba6e416e feat: update Kubernetes to v1.20.0-beta.2
Talos 0.8 is going to ship with K8s 1.20.x.

Changes to support new `control-plane` label,
upgrade-k8s supports automated fixups for 1.20.

See also: https://github.com/talos-systems/bootkube-plugin/pull/22

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2020-11-25 06:39:14 -08:00
Andrey Smirnov
b1c0f99c04 chore: add cloud image uploader (AWS AMIs for now)
It generates JSON file in the end with the upload results:

```
{"aws":{"regions":{"eu-central-1":{"arch":{"amd64":{"ami_id":"ami-0f559e06baf488ee1"},"arm64":{"ami_id":"ami-01edd1830a3c5d95c"}}},"eu-west-3":{"arch":{"amd64":{"ami_id":"ami-020f95a280c4c1c55"},"arm64":{"ami_id":"ami-0edcc7d694931a52c"}}}}}}
```

Regions, architectures can be modified as well.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2020-11-20 08:42:01 -08:00
Andrey Smirnov
1add26b42a chore: bump K8s to 1.19.4 in e2e scripts with CABPT version
This should fix the problem with the kubelet image.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2020-11-16 07:18:44 -08:00
Andrey Smirnov
61facf700a chore: build arm64 images in CI
This changes installer image/iso output to be tar via stdout
(optionally), so that we can copy back artifacts back from remote docker
daemon.

Fixes #2776

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2020-11-13 12:34:48 -08:00
Andrey Smirnov
16c5fa64f6 docs: improve configuration reference documentation
`docgen` now correctly extracts documentation for the structure itself,
and supports mix of Go-style and yaml-style docblocks, so that we can
keep linter happy while embedding examples.

Fixes for the encoder to keep things marshaled same way even with the
presence of the struct docs.

Add real examples replacing fake ones.

Add top-level hacked examples for `Config`, `MachineConfig` and
`ClusterConfig` to show the overall structure without pulling in all the
deep structure of those types.

Fixes #2768

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2020-11-13 11:50:26 -08:00
Andrey Smirnov
df6ad3fa80 feat: upgrade Kubernetes default version to 1.19.4
k8s.io modules don't have 1.19.4 tag yet :(

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2020-11-12 08:51:04 -08:00
Andrew Rynhard
5979e3399d chore: remove unused binaries
Removes socat (no dependency in containerd since 1.4), and static binaries.

Signed-off-by: Andrew Rynhard <andrew@rynhard.io>
2020-10-31 17:18:53 -07:00
Andrey Smirnov
350d75eb46 feat: build talosctl-cni-bundle, use it in talosctl for QEMU
This builds a bundle with CNI plugins for talosctl which is
automatically downloaded by `talosctl` if CNI plugins are missing.

CNI directories are moved by default to the `~/.talos/cni` path.

Also add a bunch of pre-flight checks to the QEMU provisioner to make it
easier to bootstrap the Talos QEMU cluster.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2020-10-30 16:30:37 -07:00
Artem Chernyshev
061b296530 feat: allow specifying user-disks in talosctl cluster create
User-disks are supported by QEMU and Firecracker providers.
Can be defined by using the following parameters:
```
--user-disk /mount/path:1GB
```

Can get more than 1 user disk.
Same set of user disks will be created for all master and worker nodes.

Additionally enable user-disks in qemu e2e test.

Signed-off-by: Artem Chernyshev <artem.0xD2@gmail.com>
2020-10-30 08:44:08 -07:00
Andrey Smirnov
a32c0a78f7 docs: improve the config reference documentation
Lots of small changes, changing layout, adding back references,
propagating examples, etc.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2020-10-29 18:41:46 -07:00
Andrew Rynhard
1b0ed13231 docs: move to gridsome
Brings in a new theme, improved content, and restructured layout.

Signed-off-by: Andrew Rynhard <andrew@rynhard.io>
2020-10-26 21:14:14 -07:00
Artem Chernyshev
d0ed6d7cc6 feat: encode comments as part of talosctl generated configs
Comments encoding works, defaults encoding works.
Docgen was revamped: now it generates go files.
While markdown files are all handled by
`pkg/machinery/config/encoder/markdown.go`.

Changed scheme for docs. Now it no longer relies on a single `doc.go` in
the root of a package. Instead it can generate separate `*_doc.go` files
for each file in the package. `docgen` now expects to get 3 params
instead of 2. 3rd parameter is used to define a unique method name for
getting the list of structs in the file.

Backward compatibility is supported if we define package name as the 3rd
parameter.

1st parameter no longer scans whole current directory, instead it points
to the particular file that should be processed by docgen.

`talosctl docs` command now supports two flags: `--config` and `--cli`.
They allow generating only docs for v1alpha1 configs or for talosctl. If
no flags are defined, all docs are generated.

Additionally made field types clickable in the output markdown file.

Signed-off-by: Artem Chernyshev <artem.0xD2@gmail.com>
2020-10-26 13:05:23 -07:00
Spencer Smith
7b4633b35d chore: update CI scripts
This PR pulls in the latest version of our CAPI providers, as well as
makes some minor tweaks to our bash scripts to disable terminal output
of commands during certain actions.

Signed-off-by: Spencer Smith <robertspencersmith@gmail.com>
2020-10-22 09:00:41 -07:00
Andrey Smirnov
7474f96d2d chore: publish list of images to release notes
This uses `talosctl images`. This way it's easy to find installer image
or talos image for any release.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2020-10-07 09:59:35 -07:00
Andrew Rynhard
7017327059 chore: update qemu hack script to use ISO
This can serve as an example of providing the config via an ISO, and
simplify local setups a bit.

Signed-off-by: Andrew Rynhard <andrew@rynhard.io>
2020-09-30 10:58:42 -07:00
Andrey Smirnov
ff0d4b305a feat: build Talos images/artifacts for amd64/arm64
By default, build outside of Drone works the same and builds only amd64
version, loads images back into dockerd, etc.

If multiple platforms are used, multi-arch images are built which can't
be exported to docker or to `.tar` image, they're always pushed to the
registry (even for PR builds to our internal CI registry).

Artifacts as files (initramfs, kernel) now have `-arch` suffix:
`vmlinuz-amd64`, `initramfs-amd64.xz`. "Magic" script normalizes output
paths depending on whether single platform or multiple platforms were
given.

VM provisioners accept magic `${ARCH}` in initramfs/kernel paths which
gets replaced by cluster architecture.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2020-09-27 10:32:07 -07:00
Andrew Rynhard
7d2741fc4b chore: migrate to ghcr.io
Move to GHCR.

Signed-off-by: Andrew Rynhard <andrew@rynhard.io>
2020-09-23 15:06:30 -07:00
Andrey Smirnov
788cd15c29 test: add e2e test to the provision (upgrade) tests
Add sonobuoy runner code with log fetching on failure. Use hand-picked
set of e2e tests to run: verify basic pod functionality, verify service
connectivity.

Add option `--run-e2e` to the `talosctl health` to run quick e2e test to
verify cluster health.

Add option to run provision tests with custom CNI, run one track of
provision tests with Cilium.

Bump Cilium to 1.8.2.

Talos 0.6 won't uncordon node automatically after upgrade from 0.5, as
0.5 doesn't put annotation. Workaround that in upgrade tests.

Bump upgrade test version to 0.6.0 release.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2020-09-08 13:26:31 -07:00
Andrew Rynhard
1a4059a553 feat: add grub bootloader
This moves to using grub instead of syslinux.

BREAKING CHANGE: Single node upgrades will fail in this change. This
will also break the A/B fallback setup since this version introduces
an entirely new partition scheme, that any fallback will not know about.
We plan on addressing these issues in a follow up change.

Signed-off-by: Andrew Rynhard <andrew@rynhard.io>
2020-09-01 12:06:43 -07:00
Andrey Smirnov
59adf7315d feat: provide option to run Talos under UEFI in QEMU
This also adds integration pipeline tests for UEFI.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2020-08-28 12:51:10 -07:00
Spencer Smith
c07ce17b7a fix: update e2e scripts to work with python3
This PR replaces python with python3

Signed-off-by: Spencer Smith <robertspencersmith@gmail.com>
2020-08-11 12:38:52 -07:00
Spencer Smith
303c477051 chore: update capi CI manifests to use control planes
This PR will update the CI testing to make use of our control plane
provider, as well as the other CAPI components.

Signed-off-by: Spencer Smith <robertspencersmith@gmail.com>
2020-08-11 11:14:44 -04:00
Andrey Smirnov
55f3249783 test: use registry mirrors in CI
This relies on registry caching mirrors running in the CI.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2020-07-31 16:30:41 +03:00