In order to facilitate upgrades and resets that are capable of
manipulating the system block device, we need to run an instance of
containerd that has zero dependencies on the disk. We run containerd
purely in memory for running system services.
Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
It is now possible to `start`/`stop`/`restart` any service via `osctl`
commands.
There are some changes in `ServiceRunner` to support re-use (re-entering
running state). `Services` singleton now tracks service running state to
avoid calling `Start()` on already running `ServiceRunner` instance.
Method `Start()` was renamed to `LoadAndStart()` to break up service
loading (adding to the list of service) and actual service start.
Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
This change allows for more accurate mount reporting as /proc/mounts is
a symlink to /proc/self/mounts and contains mounts that are relative to
the running process. In our case this was osd. This caused inaccurate
reporting of mounts since they were relative to osd when we really
wanted mounts relative to machined.
Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
Temporary workaround while we get more information on the
specifics for what is failing.
Ref: #795
Signed-off-by: Brad Beam <brad.beam@talos-systems.com>
This PR moves the reset API to the init API definition.
It leverages the same code we use for upgrades.
Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
This refactors metrics interface to remove containerd-specific stuff and
make it common for CRI & containerd.
Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
Service `osd` doesn't have access to rootfs, as it is running in a
container, so move API to `init` which has unconstrained access to
rootfs. (This is in line with another API, `osctl cp`).
Fixes: #752
Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
Actual API is implemented in the `init`, as it has access to root
filesystem. `osd` proxies API back to `init` with some tricks to support
grpc streaming.
Given some absolute path, `init` produces and streams back .tar.gz
archive with filesystem contents.
`osctl cp` works in two modes. First mode streams data to stdout, so
that we can do e.g.: `osctl cp /etc - | tar tz`. Second mode extracts
archive to specified location, dropping ownership info and adjusting
permissions a bit. Timestamps are not preserved.
If full dump with owner/permisisons is required, it's better to stream
data to `tar xz`, for quick and dirty look into filesystem contents
under unprivileged user it's easier to use in-place extraction.
Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
Instead of pulling a full list of containers, implement inspector query
for a single container following the spec to build display name.
Also adds many more tests for the container inspector.
Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
I couldn't find any use for the `timeout` flag nor the value passed in
the API, but it block much more useful and present in other commands
flag 'target'.
Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
This reverts commit f200eb7a8a0b7c2d29710f695000eb7680ce8b7d.
grpc can't send back both response and an error.
Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
Fixes: #689, #690
Refactor container inspection code into a package of its own with some
rudimentary tests. Use this package consistently in osd commands dealing
with containers.
Improvements for the next PRs:
* implement API to fetch info about container by ID (to avoid fetching
full list)
* handle and display errors on client side, not to the log of the
server
* more tests, including k8s containers (how can we do that?)
Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
This returns list of all the services registered, with their current
status, past events, health state, etc.
New CLI is `osctl service [<id>]`: without `<id>` it prints list of all
the services, with specific `<id>` it provides details for a service.
I decided to create "parallel" data structures in protobuf as Go
structures don't map nicely onto what protoc generates: pointers vs.
values, additional fields like mutexes, etc. Probably there's a better
approach, I'm open for it.
For CLI, I tried to keep CLI stuff in `cmd/` package, and I also created
simple wrapper to remove duplicated code which sets up client for each
command.
Examples:
```
$ osctl service
SERVICE STATE HEALTH LAST CHANGE LAST EVENT
containerd Running OK 21s ago Health check successful
kubeadm Running ? 2s ago Started task kubeadm (PID 280) for container kubeadm
kubelet Running ? 0s ago Started task kubelet (PID 383) for container kubelet
ntpd Running ? 14s ago Started task ntpd (PID 129) for container ntpd
osd Running ? 14s ago Started task osd (PID 126) for container osd
proxyd Waiting ? 14s ago Waiting for conditions
trustd Running ? 14s ago Started task trustd (PID 125) for container trustd
udevd Running ? 14s ago Started task udevd (PID 130) for container udevd
```
```
$ osctl service proxyd
ID proxyd
STATE Running
HEALTH ?
EVENTS [Preparing]: Running pre state (22s ago)
[Waiting]: Waiting for conditions (22s ago)
[Preparing]: Creating service runner (6s ago)
[Running]: Started task proxyd (PID 461) for container proxyd (6s ago)
```
Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
Without this we never set the namespace for the context which prevents it from functioning at all
Signed-off-by: Brad Beam <brad.beam@talos-systems.com>
This implements insecure over-file-socket gRPC API for init with two
first simplest APIs: reboot and shutdown (poweroff).
File socket is mounted only to `osd` service, so it is the only service
which can access init API. Osd forwards reboot/shutdown already
implemented APIs to init which actually executes these.
This enables graceful shutdown/reboot with service shutdown, sync, etc.
Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
Most crucial changes in `init/main.go`: on shutdown now Talos tries
to stop gracefully all the services. All the shutdown paths are unified,
including poweroff, reboot and panic handling on startup.
While I was at it, I also fixed bug with containers failing to start
when old snapshot is still around.
Service lifecycle is wrapped with `ServiceRunner` object now which
handles state transitions and captures events related to state changes.
Every change goes to the log as well.
There's no way to capture service state yet, but that is planned to be
implemented as RPC API for `init` which is exposed via `osd` to `osctl`.
Future steps:
1. Implement service dependencies for correct startup order and
shutdown order.
2. Implement service health, so that we can say "start trustd when
containerd is up and healthy".
3. Implement gRPC API for init, expose via osd (service status, restart,
poweroff, ...)
4. Impement 'String()' for conditions, so that we can see what service
is waiting on right now.
Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
This changes `runner.Runner` API to support more methods to allow for
containerd runner to create container object only once, and start/stop
tasks to implement restarts.
New API: `Open()` (initialize), `Run()` (run once until exits), `Stop()`
(stop running instance), `Close()` (free resource, no longer available
for new `Run()`).
So the sequence might be: `Open`, `Run`, `Stop`, `Run`, `Stop`, `Close`.
Process and containerd runners were updated for the new API, and
'restart' part was removed, now both runners only run the task once.
Restart piece was implemented in an abstract way for any wrapped
`runner.Runner` in the `runner/restart` package. Restart supports three
restart policies: `Once`, `UntilSuccess` and `Forever`.
Service API was changed slightly to return the `runner.Runner`
interface, and `system.Services` now handles running the service.
For all the services, code was adjusted to either return runner (run
once), or was wrapped with `restart` runner to provide restart policy.
Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
