vault/website/pages/docs/platform/k8s/helm/run.mdx

---
layout: docs
page_title: Running Vault - Kubernetes
sidebar_title: Running Vault
description: >-
  Vault can run directly on Kubernetes in various configurations.  For
  pure-Kubernetes worloads, this enables Vault to also exist purely within
  Kubernetes.
---

# Running Vault on Kubernetes

Vault can run directly on Kubernetes in various modes: `dev,` `standalone` and `ha`.
For pure-Kubernetes workloads, this enables Vault to also exist purely
within Kubernetes.

This page starts with a large how-to section for various specific tasks.

!> **IMPORTANT NOTE:** Vault Enterprise is currently not supported. We are actively
working a version for Vault Enterprise and it will be available in the future.

~> **Important Note:** This chart is not compatible with Helm 3. Please use Helm 2 with this chart.

## Helm Chart

The [Vault Helm chart](https://github.com/hashicorp/vault-helm)
is the recommended way to install and configure Vault on Kubernetes.
In addition to running Vault itself, the Helm chart is the primary
method for installing and configuring Vault to integrate with other
services such as Consul for High Availability deployments.

While the Helm chart exposes dozens of useful configurations and automatically
sets up complex resources, it **does not automatically operate Vault.**
You are still responsible for learning how to monitor, backup,
upgrade, etc. the Vault cluster.

The Helm chart has no required configuration and will install a Vault
cluster with sane defaults out of the box. Prior to going to production,
it is highly recommended that you
[learn about the configuration options](/docs/platform/k8s/helm/configuration).

~> **Security Warning:** By default, the chart will install an insecure configuration
of Vault. This provides a less complicated out-of-box experience for new users,
but is not appropriate for a production setup. It is highly recommended to use
a [properly secured Kubernetes cluster](https://kubernetes.io/docs/tasks/administer-cluster/securing-a-cluster/).
See the [architecture reference](/docs/platform/k8s/run#architecture)
for a production deployment checklist.

## How-To

### Installing Vault

To install Vault, clone the vault-helm repository, checkout the latest release, and install
Vault. You can run `helm install` with the `--dry-run` flag to see the
resources it would configure. In a production environment, you should always
use the `--dry-run` flag prior to making any changes to the Vault cluster
via Helm. See the [chart configuration values](/docs/platform/k8s/helm/configuration)
for additional configuration options.

```sh
# Clone the chart repo
$ git clone https://github.com/hashicorp/vault-helm.git
$ cd vault-helm

# Checkout a tagged version
$ git checkout v0.3.3

# Run Helm
$ helm install --name vault ./
...
```

!> **IMPORTANT NOTE:** Vault Helm will not initialize and unseal Vault automatically.
Initialization is required after installation followed by unsealing. Vault can be
configured to auto-unseal using KMS providers such as
[Google Cloud Platform](/docs/platform/k8s/helm/run#google-kms-auto-unseal). This
allows the pods to auto unseal if they're rescheduled in Kubernetes.

If standalone is used, the Vault pod must be initialized and unsealed.
For HA deployments, only one of the Vault pods needs to be initialized and, all Vault pods need to be unsealed.

```sh
$ kubectl exec -ti vault-0 -- vault operator init
$ kubectl exec -ti vault-0 -- vault operator unseal
```

For HA deployments, unseal the remaining pods:

```sh
$ kubectl exec -ti <NAME OF POD> -- vault operator unseal
```

### Viewing the Vault UI

The Vault UI is enabled by default when using the Helm chart.
For security reasons, it isn't exposed via a Service by default so you must
use `kubectl port-forward` to visit the UI. Once the port is forwarded as
shown below, navigate your browser to `http://localhost:8200`.

```
$ kubectl port-forward vault-0 8200:8200
...
```

The UI can also be exposed via a Kubernetes Service. To do this, configure
the [`ui.service` chart values](/docs/platform/k8s/helm#v-ui).

### Upgrading Vault on Kubernetes

To upgrade Vault on Kubernetes, we follow the same pattern as
[generally upgrading Vault](/docs/upgrading), except we can use
the Helm chart to update the Vault server Statefulset. It is important to understand
how to [generally upgrade Vault](/docs/upgrading) before reading this
section.

The Vault Statefulset uses `OnDelete` update strategy. It is critical to use `OnDelete` instead
of `RollingUpdate` because standbys must be updated before the active primary. A
failover to an older version of Vault must always be avoided.

!> **IMPORTANT NOTE:** Always back up your data before upgrading! Vault does not
make backward-compatibility guarantees for its data store. Simply replacing the
newly-installed Vault binary with the previous version will not cleanly
downgrade Vault, as upgrades may perform changes to the underlying data
structure that make the data incompatible with a downgrade. If you need to roll
back to a previous version of Vault, you should roll back your data store as
well.

#### Upgrading Vault Servers

To initiate the upgrade, change the `server.image` values to the
desired Vault version. For illustrative purposes, the example below will
use `vault:123.456`.

```yaml
server:
  image:
    repository: 'vault'
    tag: '123.456'
```

Next, run the upgrade. You should run this with `--dry-run` first to verify
the changes that will be sent to the Kubernetes cluster.

```shell
$ helm upgrade vault ./
...
```

This should cause no changes (although the resource will be updated). If
everything is stable, `helm upgrade` can be run.

The `helm upgrade` command should have updated the Statefulset template for
the Vault servers, however, no pods have been deleted. The pods must be manually
deleted to upgrade. Deleting the pods will not delete any persisted data.

If Vault is not deployed using `ha` mode, the single Vault server may be deleted by
running:

```bash
$ kubectl delete pod <name of Vault pod>
```

If Vault is deployed using `ha` mode, the standby pods must be upgraded first.
To identify which pod is currently the active primary, run the following commad
on each Vault pod:

```bash
$ kubectl exec -ti <name of pod> -- vault status | grep "HA Mode"
```

Next, delete every pod that is not the active primary:

```bash
$ kubectl delete pod <name of Vault pods>
```

If auto-unseal is not being used, the newly scheduled Vault standby pods will need
to be unsealed:

```bash
$ kubectl exec -ti <name of pod> -- vault operator unseal
```

Finally, once the standby nodes have been updated and unsealed, delete the active
primary:

```bash
$ kubectl delete pod <name of Vault primary>
```

Similar to the standby nodes, the former primary will also need to be unsealed:

```bash
$ kubectl exec -ti <name of pod> -- vault operator unseal
```

After a few moments the Vault cluster should elect a new active primary. The Vault
cluster is now upgraded!

### Protecting Sensitive Vault Configurations

Vault Helm renders a Vault configuration file during installation and stores the
file in a Kubernetes configmap. Some configurations require sensitive data to be
included in the configuration file and would not be encrypted at rest once created
in Kubernetes.

The following example shows how to add extra configuration files to Vault Helm
to protect sensitive configurations from being in plaintext at rest using Kubernetes
secrets.

First, create a partial Vault configuration with the sensitive settings Vault will
load during startup:

```yaml
cat <<EOF >>config.hcl
storage "mysql" {
username = "user1234"
password = "secret123!"
database = "vault"
}
EOF
```

Next, create a Kubernetes secret containing this partial configuration:

```bash
kubectl create secret generic vault-storage-config \
    --from-file=config.hcl
```

Finally, mount this secret as an extra volume and add an additional `-config` flag
to the Vault startup command:

```bash
helm install --name=vault \
  --set='server.extraVolumes[0].type=secret' \
  --set='server.extraVolumes[0].name=vault-storage-config' \
  --set='server.extraArgs=-config=/vault/userconfig/vault-storage-config/config.hcl' .
```

#### Google KMS Auto Unseal

The following example demonstrates configuring Vault Helm to use
[Google KMS for Auto Unseal](/docs/configuration/seal/gcpckms).

In order to authenticate and use KMS in Google Cloud, Vault Helm needs credentials. The `credentials.json`
file will need to be mounted as a secret to the Vault container.

##### Create the Secret

First, create the secret in Kubernetes:

```bash
kubectl create secret generic kms-creds --from-file=credentials.json
```

Vault Helm will mount this to `/vault/userconfig/kms-creds/credentials.json`.

##### Config Example

The following is an example of how to configure Vault Helm to use Google KMS:

```yaml
global:
  enabled: true

server:
  image:
    repository: 'vault'
    tag: '1.3.1'

  extraEnvironmentVars:
    GOOGLE_REGION: global
    GOOGLE_PROJECT: <PROJECT NAME>
    GOOGLE_APPLICATION_CREDENTIALS: /vault/userconfig/kms-creds/credentials.json

  extraVolumes:
    - type: 'secret'
      name: 'kms-creds'

  ha:
    enabled: true
    replicas: 3

    config: |
      ui = true

      listener "tcp" {
        tls_disable = 1
        address = "[::]:8200"
        cluster_address = "[::]:8201"
      }

      seal "gcpckms" {
        project     = "<NAME OF PROJECT>"
        region      = "global"
        key_ring    = "<NAME OF KEYRING>"
        crypto_key  = "<NAME OF KEY>"
      }

      storage "consul" {
        path = "vault"
        address = "HOST_IP:8500"
      }
```

#### Amazon EKS Auto Unseal

The following example demonstrates configuring Vault Helm to use
[AWS EKS for Auto Unseal](/docs/configuration/seal/awskms).

In order to authenticate and use EKS in AWS, Vault Helm needs credentials. The AWS access key
ID and key will be mounted as secret environment variables in the Vault pods.

##### Create the Secret

First, create a secret with your EKS access key/secret:

```bash
kubectl create secret generic eks-creds\
  --from-literal=AWS_ACCESS_KEY_ID="${AWS_ACCESS_KEY_ID?}" \
  --from-literal=AWS_SECRET_ACCESS_KEY="${AWS_SECRET_ACCESS_KEY?}"
```

##### Config Example

The following is an example of how to configure Vault Helm to use AWS EKS:

```yaml
global:
  enabled: true

server:
  image:
    repository: 'vault'
    tag: '1.3.1'

  extraSecretEnvironmentVars:
    - envName: AWS_ACCESS_KEY_ID
      secretName: eks-creds
      secretKey: AWS_ACCESS_KEY_ID
    - envName: AWS_SECRET_ACCESS_KEY
      secretName: eks-creds
      secretKey: AWS_SECRET_ACCESS_KEY

  ha:
    enabled: true
    config: |
      ui = true

      listener "tcp" {
        tls_disable = 1
        address = "[::]:8200"
        cluster_address = "[::]:8201"
      }

      seal "awskms" {
        region     = "KMS_REGION_HERE"
        kms_key_id = "KMS_KEY_ID_HERE"
      }

      storage "consul" {
        path = "vault"
        address = "HOST_IP:8500"
      }
```

## Architecture

We recommend running Vault on Kubernetes with the same
[general architecture](/docs/internals/architecture)
as running it anywhere else. There are some benefits Kubernetes can provide
that eases operating a Vault cluster and we document those below. The standard
[production deployment guide](https://learn.hashicorp.com/vault/day-one/production-hardening) is still an
important read even if running Vault within Kubernetes.

### Production Deployment Checklist

_End-to-End TLS._ Vault should always be used with TLS in production. If
intermediate load balancers or reverse proxies are used to front Vault,
they should not terminate TLS. This way traffic is always encrypted in transit
to Vault and minimizes risks introduced by intermediate layers. See the
[official documentation](/docs/platform/k8s/helm#standalone-server-with-tls)
for example on configuring Vault Helm to use TLS.

_Single Tenancy._ Vault should be the only main process running on a machine.
This reduces the risk that another process running on the same machine is
compromised and can interact with Vault. This can be accomplished by using Vault
Helm's `affinity` configurable. See the
[official documentation](/docs/platform/k8s/helm#highly-available-vault-cluster-with-consul)
for example on configuring Vault Helm to use affinity rules.

_Enable Auditing._ Vault supports several auditing backends. Enabling auditing
provides a history of all operations performed by Vault and provides a forensics
trail in the case of misuse or compromise. Audit logs securely hash any sensitive
data, but access should still be restricted to prevent any unintended disclosures.
Vault Helm includes a configurable `auditStorage` option that will provision a persistent
volume to store audit logs. See the
[official documentation](/docs/platform/k8s/helm#standalone-server-with-audit-storage)
for an example on configuring Vault Helm to use auditing.

_Immutable Upgrades._ Vault relies on an external storage backend for persistence,
and this decoupling allows the servers running Vault to be managed immutably.
When upgrading to new versions, new servers with the upgraded version of Vault
are brought online. They are attached to the same shared storage backend and
unsealed. Then the old servers are destroyed. This reduces the need for remote
access and upgrade orchestration which may introduce security gaps. See the
[upgrade section](/docs/platform/k8s/run#how-to) for instructions
on upgrading Vault on Kubernetes.

_Upgrade Frequently._ Vault is actively developed, and updating frequently is
important to incorporate security fixes and any changes in default settings such
as key lengths or cipher suites. Subscribe to the Vault mailing list and
GitHub CHANGELOG for updates.

_Restrict Storage Access._ Vault encrypts all data at rest, regardless of which
storage backend is used. Although the data is encrypted, an attacker with arbitrary
control can cause data corruption or loss by modifying or deleting keys. Access
to the storage backend should be restricted to only Vault to avoid unauthorized
access or operations.