132 Commits

Author SHA1 Message Date
Seán C McCord
bcb6a2d3a5 fix: prepend custom options for kernel commandline
Added a decomposition option to the kernel.NewDefaultCmdline() so that
the Defaults can be added _after_ constructing a custom commandline.
This is then implemented for `osctl install`.

Fixes #1128

Signed-off-by: Seán C McCord <ulexus@gmail.com>
2019-09-08 16:58:49 -07:00
Seán C McCord
f7ad24ec4f feat: allow network interface to be ignored
Added a property to userdata to allow a network interface to be ignored,
such that Talos will perform no operations on it (including DHCP).

Also added kernel commandline parameter (talos.network.interface.ignore)
to specify a network interface should be ignored.

Also allows chaining of kernel cmdline parameter Contains() where the
parameter in question does not exist.

Fixes #1124

Signed-off-by: Seán C McCord <ulexus@gmail.com>
2019-09-07 16:33:52 -07:00
Andrew Rynhard
db78ed93ec fix: set default install image
This sets the default install image just before installation. It was
erroneously placed in the boot verification.

Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
2019-09-04 11:48:23 -07:00
Andrew Rynhard
d098785a17 chore: remove local upgrade functionality
We have no need for this anymore since installs and upgrades are now
completely handled in a container.

Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
2019-08-27 18:44:18 -07:00
Andrew Rynhard
d4770d41ad feat: run installs via container
This moves to performing installs via a container.

Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
2019-08-27 15:01:20 -05:00
Andrew Rynhard
0bdaff1a90 feat: perform upgrades via container
This moves to performing upgrades via a container.

Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
2019-08-27 09:44:50 -07:00
Spencer Smith
6f8e089271 chore: use kubeadm v1beta2 structs everywhere
This PR will move to using the external kubeadm v1beta2 structs for our
code base. This will hopefully allow for more stable integrations with
kubeadm in the long term, as well as solve some needs we have in the
machine config rewrite.

Signed-off-by: Spencer Smith <robertspencersmith@gmail.com>
2019-08-26 12:07:36 -04:00
Andrew Rynhard
be8f58c15d feat: add overlay task
This adds a well defined task for handling all overlay mount points that
are required by the system.

Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
2019-08-25 10:47:54 -07:00
Andrew Rynhard
1eb02875c2 feat: use BLKPG ioctl for partition events
This moves to using BLKPG ioctl instead of BLKRRPART. BLKRRPART is older
and more sensitive to EBUSY errors. BLKPG has the potential to minimize
the changes of encountering an EBUSY error when manipulating partition
tables.

In looking at a comparison between BLKPG and BLKRRPART, it seems that
both have their pros and cons. Eventually a combination of the two may
serve us better, but for now I think BLKPG will get us further.

Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
2019-08-25 07:55:24 -07:00
Brad Beam
313c118ad0 refactor(networkd): Replace networkd with a standalone app
This is a major rewrite of our network subsystem.

- This changes networkd to run as a standalone app versus internal goroutine
- This changes out the netlink package with the more idiomatic netlink/rtnetlink
  packages
- This changes the initial network bootstrap/discovery from using a single
  interface to attempting to bring up all interfaces
- This moves us back on to the upstream dhcp library

Signed-off-by: Brad Beam <brad.beam@talos-systems.com>
2019-08-21 13:24:51 -05:00
Andrew Rynhard
0af1eba159 refactor: add more runtime modes
In order to DRY up all installation methods and mount methods, this PR
introduces a few more runtime modes. The modes are then used to
determine the strategy for creating and or mounting the paritions.

Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
2019-08-19 20:23:45 -07:00
Andrew Rynhard
2e65cff3ce feat: mount /sys/fs/bpf
The BPF filesystem is required to pin BPF objects.

Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
2019-08-18 07:37:08 -07:00
Andrew Rynhard
e305acac20 feat: add standardized command runner
This adds a command runner function that can be used everywhere we need
to exec a binary. It adds addtional logic around error handling that
will allow for viewing errors in the case of a failed command.

Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
2019-08-17 03:38:36 -07:00
Andrew Rynhard
6940aaf233 fix: verify installation definition
This fixes the possibility of panicing on a nil pointer by running the
verification steps earlier.

Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
2019-08-16 09:58:12 -07:00
Brad Beam
46c283b6c9 chore: Disable rate limited kmessage
This should allow for better troubleshooting during early boot/startup

Signed-off-by: Brad Beam <brad.beam@talos-systems.com>
2019-08-16 09:36:52 -07:00
Andrew Rynhard
a116145c1b feat: rename DATA partition to EPHEMERAL
This changes the data partition name to something more appropriate. We
chose ephemeral to make it very clear that the disk should not be used
for application data.

Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
2019-08-15 08:00:22 -07:00
Andrew Rynhard
09693a26c9 chore: update go modules to use Kubernetes v1.16.0-alpha.3
This is not ideal, but it works. We essentially need to start using
replace statements in order to pull in the modules we need.

Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
2019-08-14 15:34:09 -07:00
Seán C McCord
6d22744eca fix: store PartitionName when on NVMe disk
Fixes #978

Signed-off-by: Seán C McCord <ulexus@gmail.com>
2019-08-10 17:10:01 -07:00
Brad Beam
da1f73249f fix(machined): Clean up installation process
This also includes a fix for #955 which had the unintended side effect
of breaking image creation ( since it would attempt to grow the filesystem
always ).

The refactor standardizes around looking for the DATA and ESP labels to
discover any existing installations/filesystems. If none are found, an
installation will proceed -- for both image creation and bare metal.
During bootup, the DATA partition will always attempt to expand/grow.

This also introduces a new phase to verify the installation through the
existance of /boot/installed ( migrated from install stage ).

Signed-off-by: Brad Beam <brad.beam@talos-systems.com>
2019-08-08 22:10:14 -05:00
Brad Beam
53b1330c44 fix(initramfs): Allow data partition to grow
This fix ensures that we always grow the data partition during an installation.

Signed-off-by: Brad Beam <brad.beam@talos-systems.com>
2019-08-07 09:11:02 -05:00
Andrew Rynhard
90c91807bd refactor: restructure the project layout
This change moves packages into more appropriate places.

Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
2019-08-01 22:19:42 -07:00
Andrew Rynhard
a9c4a95a4b fix: mount the owned partitions in cloud platforms
This adds the logic for mounting the owned block device and resizing the
ephemeral partition for cloud platforms.

Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
2019-08-01 21:48:23 -07:00
Andrew Rynhard
ca35b85300 refactor: improve installation reliability
This change aims to make installations more unified and reliable. It
introduces the concept of a mountpoint manager that is capable of
mounting, unmounting, and moving a set of mountpoints in the correct
order.

Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
2019-08-01 11:44:40 -07:00
Spencer Smith
bc5fe085bd fix: set mtu value regardless of interface state
This PR will fix a bug we encountered in GCE, where the interface was
already up and the MTU value wasn't getting set.

Signed-off-by: Spencer Smith <robertspencersmith@gmail.com>
2019-07-31 15:02:02 -04:00
Andrew Rynhard
e63c882b89 refactor: split machined into phases
This change aims to standardize the boot process. It introduces the
concept of a phase, which is comprised of tasks. Phases are ran in serial and
the tasks that make up a phase are ran concurrently.

Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
2019-07-29 12:40:03 -07:00
Andrey Smirnov
f56a9d5b96 chore: implement first version of CRI runner
It runs containers via CRI interface in a pod sandbox. This is the very
first version:  I tried not to introduce any changes to common runner
interface.

There should be some CRI-speficic options for the runner (like polling
interval, as it doesn't have nice `Wait()` API), plus my plan so far is
to use OCI as the common layer for container options, so that we can
analyze OCI and translate to CRI (when possible, return errors when
option is not implemented).

CRI interface doesn't have a concept of 'unpacking' an image, so we
probably need to unpack via containerd API (or any other
runtime-specific API) by targeting CRI namespace.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2019-07-26 21:07:46 +03:00
Andrew Rynhard
b7a9acbe88 refactor: move setup logic into machined
The responsibility of init should only be to mount the rootfs. This
change moves Talos specific logic into machined. This will allow us to
define a version of Talos in a single binary instead of split across
two. This will enable cleaner upgrades and helps make the codebase
easier to reason about.

Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
2019-07-26 07:48:49 -07:00
Andrew Rynhard
6852fa969f chore: create raw image as sparse file
This change reduces the size of raw disk significantly by creating it as
a sparse file.

Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
2019-07-25 11:28:07 -07:00
Andrew Rynhard
0ec17e4169 feat: run rootfs from squashfs
This change moves the rootfs to a squashfs image.

Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
2019-07-25 08:38:31 -07:00
Andrew Rynhard
0b8778d772 feat: enable missing KSPP sysctls
These were disabled in previous versions of Talos since BPF was
completely disabled. With this change, we now implement all recommended
sysctls.

Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
2019-07-24 22:41:43 -07:00
Andrew Rynhard
5a68b8b371 fix: mount cgroups properly
This change mounts cgroups properly.

Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
2019-07-24 22:10:15 -07:00
Spencer Smith
2208eb5924 fix: check proper value of parseip in dhcp
This PR fixes a small bug where we weren't properly checking the value
of a net.ParseIP() call and setting the hostname to the first octet of
an IP.

Signed-off-by: Spencer Smith <robertspencersmith@gmail.com>
2019-07-24 15:11:27 -04:00
Spencer Smith
45def0a242 feat: attempt to connect to all trustd endpoints when downloading PKI
This PR will connect to each trustd endpoint specified, returning once
successful. Closes #891.

Signed-off-by: Spencer Smith <robertspencersmith@gmail.com>
2019-07-22 19:40:12 -04:00
Andrew Rynhard
88bdedf3e6 fix: make /etc/resolv.conf writable
We need /etc/resolv.conf to be writable so that networkd can update it.
This change achieves this by creating a symlink at /etc/resolv.conf that
points to /var/resolv.conf.

Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
2019-07-19 20:37:00 -07:00
Spencer Smith
a15499d25a fix: Only generate pki from trustd if not control plane
This PR will fix a bug where the non-init nodes were not generating
their certs locally and relying on trustd instead. This broke down
because we aren't saving the CA key when we're generating with the
trustd identity function (because we don't need it for workers).

Signed-off-by: Spencer Smith <robertspencersmith@gmail.com>
2019-07-18 20:20:38 -04:00
Spencer Smith
4a31b66850 feat: allow mtu specification for network devices
This PR is needed so we can specify an MTU of 1460 for GCE VMs

Signed-off-by: Spencer Smith <robertspencersmith@gmail.com>
2019-07-17 13:51:23 -07:00
Brad Beam
f650e32833 fix: Truncate hostname if necessary
We should only set the hostname to the actual host name instead of FQDN.
This hasnt been much of an issue, but GCE does return the FQDN for the
hostname field in dhcp.

Signed-off-by: Brad Beam <brad.beam@talos-systems.com>
2019-07-17 09:17:26 -07:00
Andrew Rynhard
75ea51633c fix: prefix file stat with rootfs prefix
Without this, the check for the existence of the symlinks created in the
rootfs preparation step will always fail. On a reboot init will fail
because it tries to create a symlink that already exists.

Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
2019-07-16 22:09:30 -07:00
Andrew Rynhard
fe2b81f4b4 fix: create symlinks to /etc/ssl/certs
In order to accomodate the various ways that SSL certs are managed by
the different Linux distros, kubeadm creates control plane compoents
with volume mounts of the type DirectoryOrCreate to all well known SSL
cert locations. This change creates symlinks to /etc/ss/certs at all the
well known paths to account for the fact that the rootfs is read-only.

Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
2019-07-16 16:35:59 -07:00
Andrew Rynhard
8e8aae98dd feat: add machined
This commit splits our current init into init and machined.

Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
2019-07-16 13:12:21 -07:00
Brad Beam
e9482a4041 fix: Fix integration of extra kernel args
Switch from `StringSliceVar` to `StringArrayVar` to maintain commas
in kernel args.

Update entrypoint script to allow specifying extra kernel args.

Remove default console settings in kernel config.

Signed-off-by: Brad Beam <brad.beam@talos-systems.com>
2019-07-16 14:38:55 -05:00
Andrew Rynhard
1e9548d149 feat: use new pkgs for initramfs and rootfs
This brings in the newly compiled libraries and binaries from our new
pkg builds.

Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
2019-07-15 10:32:29 -07:00
Andrew Rynhard
d4a59b7c14 fix(init) mount root partition as read-only
This uses the correct mount flag for read-only.
We mistakenly had the flag for opening a file as read-only.

Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
2019-07-12 21:34:59 -07:00
Andrew Rynhard
d197d5c6cd feat: add install flag for extra kernel args
In addition to adding a flag, this adds a field to the user data that allows
for extra kernel arguments to be specified.

Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
2019-07-12 13:27:44 -07:00
Andrey Smirnov
c10ef0f15a chore: extract CRI client as separate package
This is preparation for implementing CRI runner.

CRI client moved into its own package, I split it into multiple files
and added rudimentary tests for it.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2019-07-11 01:52:19 +03:00
Andrey Smirnov
82fe5b55e5 chore: make unit-tests use isolated instances of containerd
This makes test launch their own isolated instance of containerd with
its own root/state directories and listening socket address. Each test
brings this instance up/down on its own.

Add options to override containerd address in the code (used only in the
tests).

Enable parallel go test runs once again.

P.S. I wish I could share that 'SetupSuite' phase across the tests, but
afaik there's no way in Go to share `_test.go` code across packages. If
we put it as normal package, this might pull in test dependencies (like
`testify`) into production code, which I don't like.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2019-07-10 19:46:32 +03:00
Brad Beam
c194621e56 feat(initramfs): Add kernel arg for default interface
Should allow us to handle edge cases where eth0 is not the primary interface

Signed-off-by: Brad Beam <brad.beam@talos-systems.com>
2019-07-05 12:17:18 -07:00
Andrew Rynhard
2c6bf9b1fd fix: append probed block devices
This change fixes a bug that caused installations to fail.
On each iteration, the previously discovered block devices were dropped because the `probed` variable was reassigned.
We now append the discovered block devices to a slice declared outside the for loop.

Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
2019-07-04 21:39:34 -07:00
Andrey Smirnov
5d91d762ce feat(osd): implement container metrics for CRI inspector (#824)
This refactors metrics interface to remove containerd-specific stuff and
make it common for CRI & containerd.

Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
2019-07-04 11:25:15 -07:00
Andrew Rynhard
cca60ed121
fix: probe specified install device (#818)
Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
2019-07-02 20:46:29 -07:00