Fixes: https://github.com/talos-systems/talos/issues/2997
Listen for restart events in parallel with the boot sequence and cancel
the context if got `RestartEvent`.
Signed-off-by: Artem Chernyshev <artem.0xD2@gmail.com>
This is required to correctly handle ACPI reboot or forceful reboots
during sequence that locks the controller.
Additionally fix `NoSchedule` untaint when the configuration is changed.
Signed-off-by: Artem Chernyshev <artem.0xD2@gmail.com>
As of now, we're not using Go profiling, so it's safe to disable it to
save some memory and CPU costs. Once we start using it, we can re-enable
it conditionally.
Each process allocates around 1.4MiB on amd64 for memory profiling
buckets.
Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
There are several ways Talos node might be restarted or shut down:
* error in sequence (initiated from machined)
* panic in main goroutine (machined recovers panics)
* error in sequence (initiated via API, event caught by machined)
* reboot/shutdown via Talos API
Before this change, paths (1) and (2) were handled in machined, and no
disks were unmounted and processes killed, so technically all the
processes are running and potentially writing to the filesystems.
Paths (3) and (4) try to stop services (but not pods) and unmount
explicitly mounted filesystems, followed by reboot directly from
sequencer (bypassing machined handler).
There was a bug that user disks were never explicitly unmounted (but
they might have been unmounted if mounted on top `/var`).
This refactors all the reboot/shutdown paths to flow through machined's
main function: on paths (4) event is sent via event API from the
sequencer back to the machined and machined initiates proper shutdown
sequence.
Refactoring in machined leads to all the paths (1)-(4) flowing through
the same function `handle(error)`.
Added two additional checks before flushing buffers:
* kill all non-system processes, this also kills all mount namespaces
* unmount any filesystem backed by `/dev/*`
This ensures all filesystems are unmounted before buffers are flushed.
Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
This brings in `os-runtime` package and exposes resources with first
iteration of read-only API.
Two Talos resources (and one controller) are implemented:
* legacy.Service resource tracks Talos 'service' `RUNNING` state
* config.V1Alpha1 stores current runtime config
Glue point between existing runtime and new os-runtime based runtime is
in `v1alpha2` implementation and `V1Alpha2()` sub-interfaces of existing
`Runtime`, `State`, `Controller` interfaces.
Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
Fixes were applied automatically.
Import ordering might be questionable, but it's strict:
* stdlib
* other packages
* same package imports
Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
If bootloader meta failed to be found/to be reverted, don't abort the
whole sequence of actions leading to reboot, otherwise control returns
back and machined tries to run next sequence in failed state.
Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
This moves to using grub instead of syslinux.
BREAKING CHANGE: Single node upgrades will fail in this change. This
will also break the A/B fallback setup since this version introduces
an entirely new partition scheme, that any fallback will not know about.
We plan on addressing these issues in a follow up change.
Signed-off-by: Andrew Rynhard <andrew@rynhard.io>
This moves `pkg/config`, `pkg/client` and `pkg/constants`
under `pkg/machinery` umbrella.
And `pkg/machinery` is published as Go module inside Talos repository.
Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
This prevents reboots when some actions triggers sequence while another
sequence is still running.
Fixes#2209
Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
1. Add [xid-based](https://github.com/rs/xid) event IDs. Xids
are sortable and unique enough. Xids also encode event publishing
time with a second precision.
2. Add three ways to look back into event history: based on number of
events, on time and ID. Lookup via ID might be used to restart event
polling in case of broken API connection from the same moment.
3. Reimplement core event buffer with positions which are always
incremented instead of generation+index, this implementation is much
more simple (idea from circular buffer).
4. By default, Events API works the same - it shows no history and
starts streaming new events only.
Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
The idea here is to use single slice of events for all the consumers.
Each consumer keeps its own position within the stream, and stream is
structured as circular buffer to avoid using too much memory.
This implementation allows for one more future: looking "back" into the
event history and returning past event starting with some offset (e.g.
timestamp, event ID, etc.). This feature is not implemented yet.
Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
In recent refactoring the machined API service was changed to run outside
of the service framework. This brings it back as a service.
Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
This ensures that the machined RPC logs are written to disk so that users
can retrieve them via the log API.
Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
This is a rewrite of machined. It addresses some of the limitations and
complexity in the implementation. This introduces the idea of a
controller. A controller is responsible for managing the runtime, the
sequencer, and a new state type introduced in this PR.
A few highlights are:
- no more event bus
- functional approach to tasks (no more types defined for each task)
- the task function definition now offers a lot more context, like
access to raw API requests, the current sequence, a logger, the new
state interface, and the runtime interface.
- no more panics to handle reboots
- additional initialize and reboot sequences
- graceful gRPC server shutdown on critical errors
- config is now stored at install time to avoid having to download it at
install time and at boot time
- upgrades now use the local config instead of downloading it
- the upgrade API's preserve option takes precedence over the config's
install force option
Additionally, this pulls various packes in under machined to make the
code easier to navigate.
Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
This PR does the following:
- updates the conform config
- cleans up conform scopes
- moves slash commands to the talos-bot
- adds a check list to the pull request template
- disables codecov comments
- uses `BOT_TOKEN` so all actions are performed as the talos-bot user
- adds a `make conformance` target to make it easy for contributors to
check their commit before creating a PR
- bumps golangci-lint to v1.24.0
Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
We need to ensure that we delete the upgrade tag from the ADV even if
the tag value is an empty string.
Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
We should always set the fallback tag on an upgrade, and only revert if
the tag value is not an empty string.
Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
Since the `--once` option of `extlinux` seems to only work with BIOS, we
needed to change to remove any reliance on this option. Instead of
booting the upgraded version once, and then making it the default after
a successful boot, we now make it the default, and then revert on any
boot error.
Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
This PR introduces a new strategy for upgrades. Instead of attempting to
zap the partition table, create a new one, and then format the
partitions, this change will only update the `vmlinuz`, and
`initramfs.xz` being used to boot. It introduces an A/B style upgrade
process, which will allow for easy rollbacks. One deviation from our
original intention with upgrades is that this change does not completely
reset a node. It falls just short of that and does not reset the
partition table. This forces us to keep the current partition scheme in
mind as we make changes in the future, because an upgrade assumes a
specific partition scheme. We can improve upgrades further in the
future, but this will at least make them more dependable. Finally, one
more feature in this PR is the ability to keep state. This enables
single node clusters to upgrade since we keep the etcd data around.
Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
machined's main.go waits for boot sequence to finish, while metal
platform initializer tries to send a message to the event bus without
any listeners, so this is pure deadlock.
Resolve that by panicking from initializer, this aborts phase and
sequence, and leads to reboot on panic. Not really clean as it leaves
scary stacktraces in the dmesg, but it works. Cleanup might be done by
introducing error value for reboot, and ignoring it when printing the
errors.
Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
If we don't block, there is the potential for multiple shutdown,
reboot, and upgrade requests to be processed.
Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
Followup from #1680.
This also moves the setting from phases to machine.init to set it earlier in
the boot sequence to ensure that we get the defaults set properly from the
start and set it only once.
Signed-off-by: Brad Beam <brad.beam@talos-systems.com>
This creates an IMA policy at boot. It uses the default TCB policy with
a dont_measure rule for XFS.
Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
This removes the github.com/pkg/errors package in favor of the official
error wrapping in go 1.13.
Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
This should provide a better UX around misconfigured Talos nodes. It is
just the start of something we can expand on.
Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
The builtin recover func is scoped to the current go routine, and since
our boot sequence is kicked off in its' own go routine, we were failing
to recover from panics.
Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
This change ensures that the installer has access to the machine config
so that it can set the extra kernel arguments when installing.
Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
In order for other projects to make use of our APIs, they must not
reside underneath the internal directory. This moves the protobuf
definitions to a top-level "api" directory and scopes them according to
their domain. This change also removes generated code from the gitignore
file so that users don't have to generate the code themseleves.
Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
The gofumpt linter is a stricter drop-in replacement for gofmt. The
rules are ones that I strongly agree with and I think it would be better
if we added this linter instead of nit picking every PR.
Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
In UNIX, any zombies without parent process get re-parented to process
with PID 1 (usually running init), and PID 1 process should take care of
them (usually simply clean them up). Cleaning up zombies is important,
as they still take kerner resources, and having enormous amount of
zombie processes signifcantly degrades system performance.
For Talos, PID 1 process is machined, and machined itself forks to run
other processes in process runner and `pkg/cmd` one-time commands. Naive
solution of running `wait()` loop doesn't work as it might race with
`Process.Wait()` and clean up zombie which wasn't re-parented which
leads to process execution false failure.
After considering other solutions, we decided to go with the simple
approach: machined runs global zombie process reaper which publishes
information about reaped zombies. Any call to `Process.Wait()` (or
`Command.Wait()` which calls it) should be replaced with listening to
reaper's channel for notifications to catch info about the process which
was created in this call.
There are several changes in this PR:
1. Reaper implementation itself, started from machined.
2. Process runner and `pkg/cmd` can either use regular `Command.Wait()`
or use reaper notifications depending on reaper status (running/not
running). This allows using this code outside of machined.
3. Small bug fixes with process log which was affecting the tests.
Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
We need to support eventing with associated data. This moves the event
bus to an observer design pattern that allows observers to register for
specific events, and to receive the associated data.
Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
This adds an interface that can be used to descibe boot, shutdown, and
upgrade events in a set of phases.
Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
This adds a well defined task for handling all overlay mount points that
are required by the system.
Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
This is a major rewrite of our network subsystem.
- This changes networkd to run as a standalone app versus internal goroutine
- This changes out the netlink package with the more idiomatic netlink/rtnetlink
packages
- This changes the initial network bootstrap/discovery from using a single
interface to attempting to bring up all interfaces
- This moves us back on to the upstream dhcp library
Signed-off-by: Brad Beam <brad.beam@talos-systems.com>
This also includes a fix for #955 which had the unintended side effect
of breaking image creation ( since it would attempt to grow the filesystem
always ).
The refactor standardizes around looking for the DATA and ESP labels to
discover any existing installations/filesystems. If none are found, an
installation will proceed -- for both image creation and bare metal.
During bootup, the DATA partition will always attempt to expand/grow.
This also introduces a new phase to verify the installation through the
existance of /boot/installed ( migrated from install stage ).
Signed-off-by: Brad Beam <brad.beam@talos-systems.com>
This re-arranges phases a bit so that shutdown actions are pushed back
to the top-level main.go of machined.
Small rudimentary event.Bus is introduce to facilitate event passing
(shutdown/restart) between various machined components and main.go. This
might be not the best implementation, just something to allow this
message passing without global variables or such.
Machined API was refactored to run as goroutine service.
ACPI & signal handlers re-built as phase tasks, and activated for
non-container, container modes respectively.
As part of the fix, now `docker stop` triggers correct shutdown of Talos
(not a big deal, but good for testing).
Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
This change aims to make installations more unified and reliable. It
introduces the concept of a mountpoint manager that is capable of
mounting, unmounting, and moving a set of mountpoints in the correct
order.
Signed-off-by: Andrew Rynhard <andrew@andrewrynhard.com>
