233 Commits

Author SHA1 Message Date
Spencer Smith
aed8c06730 chore: rename v1 node configs to v1alpha1
This PR moves to using v1alpha1 as the inital node config version, so
we can graduate these configs a little more cleanly later on.

Signed-off-by: Spencer Smith <robertspencersmith@gmail.com>
2019-09-09 13:03:49 -04:00
Seán C McCord
a99637cc0a fix: use ntp client constructor
Uses NTP client constructor so that defaults are appropriately used.

Fixes #1126

Signed-off-by: Seán C McCord <ulexus@gmail.com>
2019-09-08 19:18:37 -07:00
Seán C McCord
f7ad24ec4f feat: allow network interface to be ignored
Added a property to userdata to allow a network interface to be ignored,
such that Talos will perform no operations on it (including DHCP).

Also added kernel commandline parameter (talos.network.interface.ignore)
to specify a network interface should be ignored.

Also allows chaining of kernel cmdline parameter Contains() where the
parameter in question does not exist.

Fixes #1124

Signed-off-by: Seán C McCord <ulexus@gmail.com>
2019-09-07 16:33:52 -07:00
Brad Beam
2fadd4da6f chore(machined): Increase pid_max to 262k
Minor improvement for busy systems

Signed-off-by: Brad Beam <brad.beam@talos-systems.com>
2019-09-06 19:47:24 -07:00
Andrew Rynhard
37a8ce78ae fix: prevent EBUSY when unmounting system disk
Reading /proc/mounts while simultaneously unmounting mountpoints
prevents unmounting all submounts under /var. This is due to the fact
that /proc/mounts will change as we perform unmounts, and that causes a
read of the file to become inaccurate. We now read /proc/mounts into
memory to get a snapshot of all submounts under /var, and then we
proceed with unmounting them.

This also adds some additional logging that I found to be useful while
debugging this. It also adds logic to skip of DaemonSet managed pods.

Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
2019-09-06 05:05:59 -07:00
Brad Beam
a0ace6881b refactor(ntpd): Improvements to the robustness of ntp
- Use the Validate method to ensure we get an appropriate time back
- Hard set the clock initially, adjust clock by offsets afterwards
- Introduce functional opts to configure ntp client
- Add additional test coverage

Signed-off-by: Brad Beam <brad.beam@talos-systems.com>
2019-09-05 21:52:29 -05:00
Andrew Rynhard
9337dcdfcd feat: configure interfaces concurrently
This uses a wait group to configure interfaces concurrently.

Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
2019-09-05 14:45:42 -07:00
Andrey Smirnov
c0698c1815 chore(machined): implement process reaper for PID 1 machined process
In UNIX, any zombies without parent process get re-parented to process
with PID 1 (usually running init), and PID 1 process should take care of
them (usually simply clean them up). Cleaning up zombies is important,
as they still take kerner resources, and having enormous amount of
zombie processes signifcantly degrades system performance.

For Talos, PID 1 process is machined, and machined itself forks to run
other processes in process runner and `pkg/cmd` one-time commands. Naive
solution of running `wait()` loop doesn't work as it might race with
`Process.Wait()` and clean up zombie which wasn't re-parented which
leads to process execution false failure.

After considering other solutions, we decided to go with the simple
approach: machined runs global zombie process reaper which publishes
information about reaped zombies. Any call to `Process.Wait()` (or
`Command.Wait()` which calls it) should be replaced with listening to
reaper's channel for notifications to catch info about the process which
was created in this call.

There are several changes in this PR:

1. Reaper implementation itself, started from machined.

2. Process runner and `pkg/cmd` can either use regular `Command.Wait()`
or use reaper notifications depending on reaper status (running/not
running). This allows using this code outside of machined.

3. Small bug fixes with process log which was affecting the tests.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2019-09-05 10:01:02 -07:00
Andrew Rynhard
db78ed93ec fix: set default install image
This sets the default install image just before installation. It was
erroneously placed in the boot verification.

Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
2019-09-04 11:48:23 -07:00
Seán C McCord
845cd92e5d fix: increase retries for DHCP
Increased retry count to 6 for DHCP.  In my testing, this worked
reliably in my setup, where the default (3) did not.

Ultimately, this should probably be configurable from the userdata.
Instead, this just makes it work for me.

Fixes #1099

Signed-off-by: Seán C McCord <ulexus@gmail.com>
2019-09-02 19:02:53 -07:00
Andrey Smirnov
662ef94026 chore: make TestContainerdSuite/TestRunTwice more robust
Fixes #1010

Wait for containerd shim socket to be removed before running container
second time.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2019-09-02 19:02:05 -07:00
Andrey Smirnov
d49c4baf62 chore: make health tests more robust
Fixes #1018 #1020

Add more wait loops to address cases when unit-tests are running
extremely slow under high load on the build machine.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2019-09-02 19:01:33 -07:00
Andrey Smirnov
3012851208 fix(machined): limit max stderr output, use pkg/cmd consistently
Use circular buffer instead of (unlimited) `bytes.Buffer` to limit
amount of stderr output captured. If command being run produces too much
output on stderr, this might consume too much RAM.

Use `pkg/cmd` to run command in `udevd` service. This should allow
easier udevd integration.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2019-09-02 19:01:15 -07:00
Brad Beam
1373806165 fix(init): Enable containerd subreaper
Should take care of our issue with Zombies

Signed-off-by: Brad Beam <brad.beam@talos-systems.com>
2019-08-30 14:32:13 -07:00
Andrew Rynhard
ef2154745d fix: leave etcd when upgrading control plane node
We need to remove the current node from etcd when upgrading.

Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
2019-08-30 07:16:56 -07:00
Brad Beam
a6ba81bf4e fix(networkd): Fix hostname retrieval
If multiple interfaces exist on a node, but the first interface was unsuccessful
in getting a dhcp response, we would seg fault when trying to retrieve the hostname
for that interface. This was due to d.Ack being nil and us having no guard around it

Signed-off-by: Brad Beam <brad.beam@talos-systems.com>
2019-08-28 21:25:15 -05:00
Andrew Rynhard
d89b199825 chore: change upgrade request "url" to "image"
This aligns the nomenclature used throughout the codebase.

Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
2019-08-27 21:43:20 -07:00
Andrew Rynhard
2e8f393fc5 chore: remove unused init token
This removes a token that we never used. Right now its just noise, so
let's remove it.

Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
2019-08-27 21:36:52 -07:00
Andrew Rynhard
d098785a17 chore: remove local upgrade functionality
We have no need for this anymore since installs and upgrades are now
completely handled in a container.

Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
2019-08-27 18:44:18 -07:00
Andrew Rynhard
bf8fc1dcbd chore: lint protobuf definitions
This adds linting to our protobuf definitions via prototool.

Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
2019-08-27 18:12:36 -07:00
Andrew Rynhard
d4770d41ad feat: run installs via container
This moves to performing installs via a container.

Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
2019-08-27 15:01:20 -05:00
Spencer Smith
739e232896 feat: upgrade kubernetes to v1.16.0-beta.1
This PR will upgrade to the latest beta of v1.16 in order to get us
closer to catching the v1.16.0 release as soon as it drops.

Signed-off-by: Spencer Smith <robertspencersmith@gmail.com>
2019-08-27 13:25:33 -04:00
Brad Beam
f028d29d31 chore: Increase timers for healthchecks
We've seen some instances where the initial delay is not long enough (containerd)
as well as a period of every second increases the log size for services like
proxyd which log incoming connections.

Signed-off-by: Brad Beam <brad.beam@talos-systems.com>
2019-08-27 09:54:05 -07:00
Andrew Rynhard
0bdaff1a90 feat: perform upgrades via container
This moves to performing upgrades via a container.

Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
2019-08-27 09:44:50 -07:00
Spencer Smith
f85750cdca feat: generate and use v1 machine configs
This PR will implement the v1 machine config proposal. This will allow
for a streamlined config for talos nodes.

Signed-off-by: Spencer Smith <robertspencersmith@gmail.com>
2019-08-26 19:36:14 -04:00
Andrew Rynhard
43e20217e8 feat: add ability to pass data on event bus
We need to support eventing with associated data. This moves the event
bus to an observer design pattern that allows observers to register for
specific events, and to receive the associated data.

Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
2019-08-26 13:27:02 -07:00
Spencer Smith
6f8e089271 chore: use kubeadm v1beta2 structs everywhere
This PR will move to using the external kubeadm v1beta2 structs for our
code base. This will hopefully allow for more stable integrations with
kubeadm in the long term, as well as solve some needs we have in the
machine config rewrite.

Signed-off-by: Spencer Smith <robertspencersmith@gmail.com>
2019-08-26 12:07:36 -04:00
Brad Beam
692571bdec feat(networkd): Add grpc endpoint
Allows us to list routes and interface details

Signed-off-by: Brad Beam <brad.beam@talos-systems.com>
2019-08-25 19:48:08 -07:00
Brad Beam
d36007fb29 feat(osd): Add ntpd client
Allows us to access ntp api

Signed-off-by: Brad Beam <brad.beam@talos-systems.com>
2019-08-25 13:38:34 -07:00
Andrew Rynhard
9eaa2d8140 feat: add sequencer interface
This adds an interface that can be used to descibe boot, shutdown, and
upgrade events in a set of phases.

Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
2019-08-25 12:59:42 -07:00
Andrew Rynhard
be8f58c15d feat: add overlay task
This adds a well defined task for handling all overlay mount points that
are required by the system.

Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
2019-08-25 10:47:54 -07:00
Brad Beam
cdc989ddda refactor(networkd): Switch from rtnetlink to rtnl
Gives a better abstraction on rtnetlink interaction

Signed-off-by: Brad Beam <brad.beam@talos-systems.com>
2019-08-21 13:24:51 -05:00
Brad Beam
313c118ad0 refactor(networkd): Replace networkd with a standalone app
This is a major rewrite of our network subsystem.

- This changes networkd to run as a standalone app versus internal goroutine
- This changes out the netlink package with the more idiomatic netlink/rtnetlink
  packages
- This changes the initial network bootstrap/discovery from using a single
  interface to attempting to bring up all interfaces
- This moves us back on to the upstream dhcp library

Signed-off-by: Brad Beam <brad.beam@talos-systems.com>
2019-08-21 13:24:51 -05:00
Andrew Rynhard
0af1eba159 refactor: add more runtime modes
In order to DRY up all installation methods and mount methods, this PR
introduces a few more runtime modes. The modes are then used to
determine the strategy for creating and or mounting the paritions.

Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
2019-08-19 20:23:45 -07:00
Andrew Rynhard
794c7231f5 feat: run dedicated instance of containerd for system services
In order to facilitate upgrades and resets that are capable of
manipulating the system block device, we need to run an instance of
containerd that has zero dependencies on the disk. We run containerd
purely in memory for running system services.

Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
2019-08-19 12:32:59 -07:00
Andrew Rynhard
2e65cff3ce feat: mount /sys/fs/bpf
The BPF filesystem is required to pin BPF objects.

Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
2019-08-18 07:37:08 -07:00
Brad Beam
ec0f188309 fix(machined): Remove host mounts for specific CNI providers
We shouldnt need these anymore

Signed-off-by: Brad Beam <brad.beam@talos-systems.com>
2019-08-17 20:20:45 -07:00
Brad Beam
cf64847772 refactor(proxyd): Update multilisteners to use error chan.
This cleans up the multiple listener implementation.

Signed-off-by: Brad Beam <brad.beam@talos-systems.com>
2019-08-16 12:21:02 -05:00
Andrew Rynhard
6940aaf233 fix: verify installation definition
This fixes the possibility of panicing on a nil pointer by running the
verification steps earlier.

Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
2019-08-16 09:58:12 -07:00
Brad Beam
76a9c15044 feat: Add gRPC server for ntp
Part of the API refactor; this introduces a gRPC server for ntp.
This allows the ability to query node time and check time against
specific ntp servers.

This refactor also moves the ntp functionality into a sub package for
better project organization.

Signed-off-by: Brad Beam <brad.beam@talos-systems.com>
2019-08-16 09:46:43 -07:00
Brad Beam
70a478895f feat(proxyd): Add gRPC server
Part of the API refactor; this introduces a gRPC server for proxyd
to expose some of the internal state.

Signed-off-by: Brad Beam <brad.beam@talos-systems.com>
2019-08-15 16:35:03 -05:00
Andrew Rynhard
a116145c1b feat: rename DATA partition to EPHEMERAL
This changes the data partition name to something more appropriate. We
chose ephemeral to make it very clear that the disk should not be used
for application data.

Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
2019-08-15 08:00:22 -07:00
Brad Beam
249acda74a feat: Allow hostname to be specified in userdata
This sets up the ability to define hostname via userdata. I dont expect
this will get used publicly much, but provides a mechanism to convey
the hostname from various sources internally.

Signed-off-by: Brad Beam <brad.beam@talos-systems.com>
2019-08-14 22:41:27 -05:00
Andrew Rynhard
09693a26c9 chore: update go modules to use Kubernetes v1.16.0-alpha.3
This is not ideal, but it works. We essentially need to start using
replace statements in order to pull in the modules we need.

Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
2019-08-14 15:34:09 -07:00
Andrew Rynhard
142500ce3e fix(proxyd): print bootstrap backend dial errors
This prints any error that occurs when dialing the bootstrap backend.

Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
2019-08-11 15:12:09 -07:00
Seán C McCord
fd76d90028 fix(proxyd): do not pre-bracket IPv6 backend addrs
Fixes #996

Signed-off-by: Seán C McCord <ulexus@gmail.com>
2019-08-11 15:00:22 -07:00
Andrew Rynhard
ad79e8dfcf feat: remove the machine config on reset
This wil remove the machine config on a reset so that a new machine
configwill be downloaded and used on a reboot.

Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
2019-08-11 12:51:55 -07:00
Seán C McCord
63cfd8a405 fix(proxyd): wrap Dial addresses
Handle IPv6 addresses in proxyd frontend.

Fixes #988

Signed-off-by: Seán C McCord <ulexus@gmail.com>
2019-08-10 23:00:28 -07:00
Seán C McCord
7691bb060c fix: enable IPv6 forwarding
Fixes #985

Signed-off-by: Seán C McCord <ulexus@gmail.com>
2019-08-10 22:39:56 -07:00
Andrey Smirnov
ae54f7e40d fix: stalls in local Docker cluster boot
Problem was triggered by udevd trigger, root cause is not clear, but
workaround is to disable it for container mode.

Implement CPU/mem limits for `osctl cluster create`, apply defaults,
bump defaults for cicd.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2019-08-10 13:31:47 +03:00