From 030bba4e68751ddffaf9734853da3dfb395d2f28 Mon Sep 17 00:00:00 2001 From: Tom Proctor Date: Tue, 28 Nov 2023 14:07:07 +0000 Subject: [PATCH] Support rootless plugin containers (#24236) * Pulls in github.com/go-secure-stdlib/plugincontainer@v0.3.0 which exposes a new `Config.Rootless` option to opt in to extra container configuration options that allow establishing communication with a non-root plugin within a rootless container runtime. * Adds a new "rootless" option for plugin runtimes, so Vault needs to be explicitly told whether the container runtime on the machine is rootless or not. It defaults to false as rootless installs are not the default. * Updates `run_config.go` to use the new option when the plugin runtime is rootless. * Adds new `-rootless` flag to `vault plugin runtime register`, and `rootless` API option to the register API. * Adds rootless Docker installation to CI to support tests for the new functionality. * Minor test refactor to minimise the number of test Vault cores that need to be made for the external plugin container tests. * Documentation for the new rootless configuration and the new (reduced) set of restrictions for plugin containers. * As well as adding rootless support, we've decided to drop explicit support for podman for now, but there's no barrier other than support burden to adding it back again in future so it will depend on demand. --- .github/workflows/test-go.yml | 32 +++- api/sys_plugins_runtimes.go | 1 + changelog/24236.txt | 3 + command/plugin_runtime_register.go | 11 ++ command/plugin_runtime_register_test.go | 8 +- go.mod | 5 +- go.sum | 16 +- sdk/go.mod | 5 +- sdk/go.sum | 11 +- sdk/helper/pluginruntimeutil/config.go | 1 + sdk/helper/pluginutil/run_config.go | 6 +- vault/external_plugin_container_test.go | 144 +++++++++--------- vault/external_plugin_test.go | 8 +- .../plugin/external_plugin_test.go | 24 +-- vault/logical_system.go | 8 + vault/logical_system_paths.go | 9 ++ vault/logical_system_test.go | 3 + vault/testdata/Dockerfile | 4 + vault/testing.go | 42 ++--- .../system/plugins-runtimes-catalog.mdx | 4 + .../docs/commands/plugin/runtime/register.mdx | 4 + .../docs/plugins/containerized-plugins.mdx | 42 ++--- 22 files changed, 242 insertions(+), 149 deletions(-) create mode 100644 changelog/24236.txt diff --git a/.github/workflows/test-go.yml b/.github/workflows/test-go.yml index 2b498be4d0..24825b5272 100644 --- a/.github/workflows/test-go.yml +++ b/.github/workflows/test-go.yml @@ -286,14 +286,37 @@ jobs: "runsc": { "path": "/usr/local/bin/runsc", "runtimeArgs": [ - "--host-uds=all", - "--host-fifo=open" + "--host-uds=create" ] } } } EOF sudo systemctl reload docker + - name: Install rootless Docker + # Enterprise repo runners do not allow sudo, so can't system packages there yet. + if: ${{ !inputs.enterprise }} + run: | + sudo apt-get install -y uidmap dbus-user-session + export FORCE_ROOTLESS_INSTALL=1 + curl -fsSL https://get.docker.com/rootless | sh + mkdir -p ~/.config/docker/ + tee ~/.config/docker/daemon.json <)` – Part of the request URL. Specifies the plugin runtime name. Use the runtime name to look up plugin runtimes in the catalog. +- `rootless` `(bool: false)` - Whether the container runtime is running as a + non-privileged user. Must be set if plugin container images are also configured + to run as a non-root user. + - `oci_runtime` `(string: )` – Specifies OCI-compliant container runtime to use. Default is "runsc", gVisor's OCI runtime. diff --git a/website/content/docs/commands/plugin/runtime/register.mdx b/website/content/docs/commands/plugin/runtime/register.mdx index 698ee318db..75119d4e72 100644 --- a/website/content/docs/commands/plugin/runtime/register.mdx +++ b/website/content/docs/commands/plugin/runtime/register.mdx @@ -45,6 +45,10 @@ flags](/vault/docs/commands) included on all commands. - `-type` `(string: )` - Plugin runtime type. Vault currently only supports `container` as a runtime type. +- `-rootless` `(bool: false)` - Whether the container runtime is running as a + non-privileged user. Must be set if plugin container images are also configured + to run as a non-root user. + - `-cgroup_parent` `(string: "")` - Parent cgroup to set for each container. Use `cgroup_parent` to control the total resource usage for a group of plugins. diff --git a/website/content/docs/plugins/containerized-plugins.mdx b/website/content/docs/plugins/containerized-plugins.mdx index 1bd4cb2de7..748d6d6809 100644 --- a/website/content/docs/plugins/containerized-plugins.mdx +++ b/website/content/docs/plugins/containerized-plugins.mdx @@ -39,7 +39,7 @@ increases the isolation between plugins, and between plugins and Vault. All plugins have the following basic requirements to be containerized: -- **Your plugin must be built with at least v1.5.0 of the HashiCorp +- **Your plugin must be built with at least v1.6.0 of the HashiCorp [`go-plugin`](https://github.com/hashicorp/go-plugin) library**. - **The image entrypoint should run the plugin binary**. @@ -52,39 +52,39 @@ in [supported configurations](#supported-configurations). Vault's containerized plugins are compatible with a variety of configurations. In particular, it has been tested with the following: -- Docker and Podman. -- Default and rootless container engine. -- OCI runtimes runsc and runc. -- Plugin container images with root and non-root users. +- Default and [rootless](https://docs.docker.com/engine/security/rootless/) Docker. +- OCI-compatible runtimes `runsc` and `runc`. +- Plugin container images running as root and non-root users. - [Mlock](/vault/docs/configuration#disable_mlock) disabled or enabled. Not all combinations work and some have additional requirements, listed below. If you use a configuration that matches multiple headings, you should combine the requirements from each matching heading. -### Rootless installation with non-root container user +### `runsc` runtime -Not currently supported. We are hoping to provide support in future. +- You must pass an additional `--host-uds=create` flag to the `runsc` runtime. -### runsc runtime +### Rootless Docker with `runsc` runtime -- You must pass an additional `--host-uds=all` flag to the `runsc` runtime. - -### Rootless installation with `runsc` - -- Does not currently support cgroup limits. - You must pass an additional `--ignore-cgroups` flag to the `runsc` runtime. + - Cgroup limits are not currently supported for this configuration. + +### Rootless Docker with non-root container user + +- You must use a container plugin runtime with + [`rootless`](/vault/docs/commands/plugin/runtime/register#rootless) enabled. +- Your filesystem must have Posix 1e ACL support, available by default in most + modern Linux file systems. +- Only supported for gVisor's `runsc` runtime. + +### Rootless Docker with mlock enabled + +- Only supported for gVisor's `runsc` runtime. ### Non-root container user with mlock enabled -- You must set the IPC_LOCK capability on the plugin binary. - -### Rootless container engine with mlock enabled - -- You must set the IPC_LOCK capability on the container engine's binary. -- You do not need to set the IPC_LOCK capability if running with Docker and runsc. - The `runsc` runtime supports mlock syscalls in rootless Docker without needing - IPC_LOCK itself. +- You must set the `IPC_LOCK` capability on the plugin binary. ## Container lifecycle and metadata