Use a global instance, handle loading/saving META in global context.
Deprecate legacy syslinux ADV, provide an easier interface for
consumers.
Expose META as resources.
Fix the bootloader revert process (it was completely broken for quite a
while :sad:).
This is a first step which mostly does preparation work, real changes
will come in the next PRs:
* add APIs to write to META
* consume META keys for platform network config for `metal`
* custom key for URL `${code}`
Signed-off-by: Andrey Smirnov <andrey.smirnov@talos-systems.com>
Move dashboard package into a common location where both Talos and talosctl can use it.
Add support for overriding stdin, stdout, stderr and ctt in process runner.
Create a dashboard service which runs the dashboard on /dev/tty2.
Redirect kernel messages to tty1 and switch to tty2 after starting the dashboard on it.
Related to siderolabs/talos#6841, siderolabs/talos#4791.
Signed-off-by: Utku Ozdemir <utku.ozdemir@siderolabs.com>
Load & start machined earlier and in initialize sequence, so that it is possible to use its API over its unix socket in maintenance mode.
Additionally, do not return features from Version API if a config is not yet available.
Related to siderolabs/talos#4791.
Signed-off-by: Utku Ozdemir <utku.ozdemir@siderolabs.com>
Use a wrapper for starting processes which can setup proper cgroups,
OOMscore, and also drop capabilities for the process, then it calls
`execve`.
The containerd tests is also fixed to support cgroups when
running tests in buildkit. It used to pass previously as we did not
error if cgroup setup failed.
Signed-off-by: Noel Georgi <git@frezbo.dev>
There's a cyclic dependency on siderolink library which imports talos
machinery back. We will fix that after we get talos pushed under a new
name.
Signed-off-by: Andrey Smirnov <andrey.smirnov@talos-systems.com>
This the first step towards replacing all import paths to be based on
`siderolabs/` instead of `talos-systems/`.
All updates contain no functional changes, just refactorings to adapt to
the new path structure.
Signed-off-by: Andrey Smirnov <andrey.smirnov@talos-systems.com>
I had to do several things:
- contextcheck now supports Go 1.18 generics, but I had to disable it because of this https://github.com/kkHAIKE/contextcheck/issues/9
- dupword produces to many false positives, so it's also disabled
- revive found all packages which didn't have a documentation comment before. And tehre is A LOT of them. I updated some of them, but gave up at some point and just added them to exclude rules for now.
- change lint-vulncheck to use `base` stage as base
Signed-off-by: Dmitriy Matrenichev <dmitry.matrenichev@siderolabs.com>
This implements a simple way to upgrade Talos node running in
maintenance mode (only if Talos is installed, i.e. if `STATE` and
`EPHEMERAL` partitions are wiped).
Upgrade is only available over SideroLink for security reasons.
Upgrade in maintenance mode doesn't support any options, and it works
without machine configuration, so proxy environment variables are not
available, registry mirrors can't be used, and extensions are not
installed.
Fixes#6224
Signed-off-by: Andrey Smirnov <andrey.smirnov@talos-systems.com>
Fixes#5652
This reworks and unifies HTTP client/transport management in Talos:
* cleanhttp is used everywhere consistently
* DefaultClient is using pooled client, other clients use regular
transport
* like before, Proxy vars are inspected on each request (but now
consistently)
* manifest download functions now recreate the client on each run to
pick up latest changes
* system CA list is picked up from a fixed locations, and supports
reloading on changes
Signed-off-by: Andrey Smirnov <andrey.smirnov@talos-systems.com>
Fix `Talos` sequencer to run only a single sequence at the same time.
Sequences priority was updated. To match the table:
| what is running (columns) what is requested (rows) | boot | reboot | reset | upgrade |
|----------------------------------------------------|------|--------|-------|---------|
| reboot | Y | Y | Y | N |
| reset | Y | N | N | N |
| upgrade | Y | N | N | N |
With a small addition that `WithTakeover` is still there.
If set, priority is ignored.
This is mainly used for `Shutdown` sequence invokation.
And if doing apply config with reboot enabled.
Signed-off-by: Artem Chernyshev <artem.chernyshev@talos-systems.com>
Previously Talos would not shutdown gracefully if hyper-v issued the
'perform_shutdown' call. Said call would execute '/sbin/poweroff' which
did not exist in Talos. We hardlink machined to '/sbin/poweroff' and
make it send a shutdown API call to PID 1 machined.
Fixes#5641
Signed-off-by: Philipp Sauter <philipp.sauter@siderolabs.com>
Some failures can be fixed by updating the machine configuration.
Now `userDisks` and `userFiles` do not make Talos to enter into reboot
loop but pause for 35 minutes.
Additionally, `apid` and `machined` are now started right after
containerd is up and running.
That makes it possible for the operator to connect to the node using
talosctl and fix the config.
Fixes: https://github.com/talos-systems/talos/issues/4669
Signed-off-by: Artem Chernyshev <artem.chernyshev@talos-systems.com>
Because of the bug Drainer never worked properly, as `shutdown` channel
wasn't initialied.
Also add unit-tests and add some small clean-ups which don't affect
functionality.
Signed-off-by: Andrey Smirnov <andrey.smirnov@talos-systems.com>
Report Talos events to any gRPC server.
Destination address is specified by using kernel parameters.
Fixes: https://github.com/talos-systems/talos/issues/4458
Signed-off-by: Artem Chernyshev <artem.chernyshev@talos-systems.com>
This is a complete rewrite of time sync process.
Now the time sync process starts early at boot time, and it adapts to
configuration changes:
* before config is available, `pool.ntp.org` is used
* once config is available, configured time servers are used
Controller updates same time sync resource as other controllers had
dependency on, so they have a chance to wait for the time sync event.
Talos services which depend on time now wait on same resource instead of
waiting on timed health.
New features:
* time sync now sticks to the particular time server unless there's an
error from that server, and server is changed in that case, this
improves time sync accuracy
* time sync acts on config changes immediately, so it's possible to
reconfigure time sync at any time
* there's a new 'epoch' field in time sync resources which allows
time-dependent controllers to regenerate certs when there's a big enough
jump in time
Features to implement later:
* apid shouldn't depend on timed, it should be started early and it
should regenerate certs on time jump
* trustd should be updated in same way
Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
This moves implementation of the user-facing APIs to the machined, and
as now all the APIs are implemented by machined, remove routerd and
adjust apid to proxy to machined.
Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
This removes container images for the aforementioned services, they are
now built into `machined` executable which launches one or another
service based on `argv[0]`.
Containers are started with rootfs directory which contains only a
single executable file for the service.
This creates rootfs on squashfs for each container in
`/opt/<container>`.
Service `networkd` is not touched as it's handled in #3350.
This removes all the image imports, snapshots and other things which
were associated with the existing way to run containers.
Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
Fixes: https://github.com/talos-systems/talos/issues/2997
Listen for restart events in parallel with the boot sequence and cancel
the context if got `RestartEvent`.
Signed-off-by: Artem Chernyshev <artem.0xD2@gmail.com>
This is required to correctly handle ACPI reboot or forceful reboots
during sequence that locks the controller.
Additionally fix `NoSchedule` untaint when the configuration is changed.
Signed-off-by: Artem Chernyshev <artem.0xD2@gmail.com>
As of now, we're not using Go profiling, so it's safe to disable it to
save some memory and CPU costs. Once we start using it, we can re-enable
it conditionally.
Each process allocates around 1.4MiB on amd64 for memory profiling
buckets.
Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
There are several ways Talos node might be restarted or shut down:
* error in sequence (initiated from machined)
* panic in main goroutine (machined recovers panics)
* error in sequence (initiated via API, event caught by machined)
* reboot/shutdown via Talos API
Before this change, paths (1) and (2) were handled in machined, and no
disks were unmounted and processes killed, so technically all the
processes are running and potentially writing to the filesystems.
Paths (3) and (4) try to stop services (but not pods) and unmount
explicitly mounted filesystems, followed by reboot directly from
sequencer (bypassing machined handler).
There was a bug that user disks were never explicitly unmounted (but
they might have been unmounted if mounted on top `/var`).
This refactors all the reboot/shutdown paths to flow through machined's
main function: on paths (4) event is sent via event API from the
sequencer back to the machined and machined initiates proper shutdown
sequence.
Refactoring in machined leads to all the paths (1)-(4) flowing through
the same function `handle(error)`.
Added two additional checks before flushing buffers:
* kill all non-system processes, this also kills all mount namespaces
* unmount any filesystem backed by `/dev/*`
This ensures all filesystems are unmounted before buffers are flushed.
Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
This brings in `os-runtime` package and exposes resources with first
iteration of read-only API.
Two Talos resources (and one controller) are implemented:
* legacy.Service resource tracks Talos 'service' `RUNNING` state
* config.V1Alpha1 stores current runtime config
Glue point between existing runtime and new os-runtime based runtime is
in `v1alpha2` implementation and `V1Alpha2()` sub-interfaces of existing
`Runtime`, `State`, `Controller` interfaces.
Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
Fixes were applied automatically.
Import ordering might be questionable, but it's strict:
* stdlib
* other packages
* same package imports
Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
If bootloader meta failed to be found/to be reverted, don't abort the
whole sequence of actions leading to reboot, otherwise control returns
back and machined tries to run next sequence in failed state.
Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
This moves to using grub instead of syslinux.
BREAKING CHANGE: Single node upgrades will fail in this change. This
will also break the A/B fallback setup since this version introduces
an entirely new partition scheme, that any fallback will not know about.
We plan on addressing these issues in a follow up change.
Signed-off-by: Andrew Rynhard <andrew@rynhard.io>
This moves `pkg/config`, `pkg/client` and `pkg/constants`
under `pkg/machinery` umbrella.
And `pkg/machinery` is published as Go module inside Talos repository.
Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
This prevents reboots when some actions triggers sequence while another
sequence is still running.
Fixes#2209
Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
1. Add [xid-based](https://github.com/rs/xid) event IDs. Xids
are sortable and unique enough. Xids also encode event publishing
time with a second precision.
2. Add three ways to look back into event history: based on number of
events, on time and ID. Lookup via ID might be used to restart event
polling in case of broken API connection from the same moment.
3. Reimplement core event buffer with positions which are always
incremented instead of generation+index, this implementation is much
more simple (idea from circular buffer).
4. By default, Events API works the same - it shows no history and
starts streaming new events only.
Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
The idea here is to use single slice of events for all the consumers.
Each consumer keeps its own position within the stream, and stream is
structured as circular buffer to avoid using too much memory.
This implementation allows for one more future: looking "back" into the
event history and returning past event starting with some offset (e.g.
timestamp, event ID, etc.). This feature is not implemented yet.
Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
In recent refactoring the machined API service was changed to run outside
of the service framework. This brings it back as a service.
Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
This ensures that the machined RPC logs are written to disk so that users
can retrieve them via the log API.
Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
This is a rewrite of machined. It addresses some of the limitations and
complexity in the implementation. This introduces the idea of a
controller. A controller is responsible for managing the runtime, the
sequencer, and a new state type introduced in this PR.
A few highlights are:
- no more event bus
- functional approach to tasks (no more types defined for each task)
- the task function definition now offers a lot more context, like
access to raw API requests, the current sequence, a logger, the new
state interface, and the runtime interface.
- no more panics to handle reboots
- additional initialize and reboot sequences
- graceful gRPC server shutdown on critical errors
- config is now stored at install time to avoid having to download it at
install time and at boot time
- upgrades now use the local config instead of downloading it
- the upgrade API's preserve option takes precedence over the config's
install force option
Additionally, this pulls various packes in under machined to make the
code easier to navigate.
Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
This PR does the following:
- updates the conform config
- cleans up conform scopes
- moves slash commands to the talos-bot
- adds a check list to the pull request template
- disables codecov comments
- uses `BOT_TOKEN` so all actions are performed as the talos-bot user
- adds a `make conformance` target to make it easy for contributors to
check their commit before creating a PR
- bumps golangci-lint to v1.24.0
Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
We need to ensure that we delete the upgrade tag from the ADV even if
the tag value is an empty string.
Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
We should always set the fallback tag on an upgrade, and only revert if
the tag value is not an empty string.
Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
Since the `--once` option of `extlinux` seems to only work with BIOS, we
needed to change to remove any reliance on this option. Instead of
booting the upgraded version once, and then making it the default after
a successful boot, we now make it the default, and then revert on any
boot error.
Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
This PR introduces a new strategy for upgrades. Instead of attempting to
zap the partition table, create a new one, and then format the
partitions, this change will only update the `vmlinuz`, and
`initramfs.xz` being used to boot. It introduces an A/B style upgrade
process, which will allow for easy rollbacks. One deviation from our
original intention with upgrades is that this change does not completely
reset a node. It falls just short of that and does not reset the
partition table. This forces us to keep the current partition scheme in
mind as we make changes in the future, because an upgrade assumes a
specific partition scheme. We can improve upgrades further in the
future, but this will at least make them more dependable. Finally, one
more feature in this PR is the ability to keep state. This enables
single node clusters to upgrade since we keep the etcd data around.
Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
machined's main.go waits for boot sequence to finish, while metal
platform initializer tries to send a message to the event bus without
any listeners, so this is pure deadlock.
Resolve that by panicking from initializer, this aborts phase and
sequence, and leads to reboot on panic. Not really clean as it leaves
scary stacktraces in the dmesg, but it works. Cleanup might be done by
introducing error value for reboot, and ignoring it when printing the
errors.
Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
If we don't block, there is the potential for multiple shutdown,
reboot, and upgrade requests to be processed.
Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
Followup from #1680.
This also moves the setting from phases to machine.init to set it earlier in
the boot sequence to ensure that we get the defaults set properly from the
start and set it only once.
Signed-off-by: Brad Beam <brad.beam@talos-systems.com>
