### Performance Standbys revert to Standby mode on unseal

#### Affected versions

- 1.14.12
- 1.15.8
- 1.16.2

#### Issue

If you previously set a value for `retention_months` via the
[sys/internal/counters/config](/vault/api-docs/system/internal-counters#update-the-client-count-configuration)
endpoint, upgrading to Vault Enterprise versions 1.14.12, 1.15.8, and 1.16.2
will cause [Performance Standby](/vault/docs/enterprise/performance-standby)
nodes to revert to Standby mode.

Adding nodes with Vault Enterprise versions 1.14.12, 1.15.8, or 1.16.2 to a
cluster with an older versioned leader will see any previously set
`retention_months` value and attempt to write the new minimum value of `48`. The
storage write will result in a read-only error:

```
[ERROR] core: performance standby post-unseal setup failed: error="cannot write to readonly storage"
```

You can verify the status of your nodes by checking the
[/sys/health](/vault/api-docs/system/health) endpoint.

Deployments that rely on scaling across Performance Standbys will now forward
all requests to the active node, increasing the utilization of the active node.

<Note title="Post-upgrade cluster membership">
During the last step of a full upgrade, the old leader steps down, causing one
of the Standby nodes to become leader.
</Note>

A fix for the read-only storage error has been prioritized and escalated. The
fix will be in releases 1.14.13, 1.15.9 and 1.16.3.

<Warning title="Important">
If you have already upgraded to versions 1.14.12, 1.15.8, or 1.16.2, please
refer to the workaround section for options.
</Warning>

#### Workaround

Once the leader of the cluster has been updgraded to version 1.14.12, 1.15.8, or
1.16.2, the workaround is to update the `retention_months` value on the active
node via the
[sys/internal/counters/config](/vault/api-docs/system/internal-counters#update-the-client-count-configuration)
endpoint:

```shell
$ vault write sys/internal/counters/config retention_months=48
```

This storage entry will be written to all nodes in the cluster, allowing them to
immediately unseal as Performance Standbys.

After the new `retention_months` value is written to storage on the active node,
adding new nodes to the cluster will not cause the read-only error.
