mirror of
https://github.com/hashicorp/vault.git
synced 2025-08-07 15:17:03 +02:00
Suggested edits for identity doc updates (#29339)
* Identity dupe resolution guide first draft * initial edits * save progress * save changes * add script to find template policies * save progress * save work * push latest updates * missed one * Update website/content/docs/upgrading/deduplication/entity-group.mdx Co-authored-by: Paul Banks <pbanks@hashicorp.com> * apply additional feedback * apply feedback --------- Co-authored-by: Paul Banks <pbanks@hashicorp.com>
This commit is contained in:
Sarah Chavis
GitHub
parent
d127c4de93
commit
1b5260d696
@ -0,0 +1,364 @@
|
|||||||
|
---
|
||||||
|
layout: docs
|
||||||
|
page_title: Resolve ACL policy templates
|
||||||
|
description: >-
|
||||||
|
Resolve templated ACL behavior for deduplicated entities and groups.
|
||||||
|
---
|
||||||
|
|
||||||
|
# Resolve deduplication impact on ACL policy templates
|
||||||
|
|
||||||
|
Fix templated ACL policy behavior for entities and groups renamed during
|
||||||
|
identity deduplication.
|
||||||
|
|
||||||
|
<Tip title="Assumptions">
|
||||||
|
|
||||||
|
- You are running Vault 1.19 or later.
|
||||||
|
- You have [deduplication **renaming** targets in your system logs](/vault/docs/upgrading/identity-deduplication).
|
||||||
|
- You have admin permission on the relevant Vault server or cluster.
|
||||||
|
|
||||||
|
</Tip>
|
||||||
|
|
||||||
|
|
||||||
|
## How renaming affect ACL policies
|
||||||
|
|
||||||
|
Operators use
|
||||||
|
[policy templating](/vault/docs/concepts/policies#templated-policies) to specify
|
||||||
|
permissions that dynamically resolve based on specific properties of an
|
||||||
|
authenticated entity. During deduplication, Vault renames entities or groups
|
||||||
|
with duplicate identities by appending a universally unique identifier (UUID) to
|
||||||
|
all but one of the entities to avoid future duplication.
|
||||||
|
|
||||||
|
If a templated ACL policy relies on an entity or group name, renaming duplicate
|
||||||
|
identities could inadvertently change the meaning of the policy. In practice,
|
||||||
|
identity deduplication is unlikely to grant unintended access to an existing
|
||||||
|
resource since the renamed identity contains a UUID that should
|
||||||
|
not collide with other resource paths. But the rename might **remove** access to
|
||||||
|
existing resources after deduplication if the associated resource policy
|
||||||
|
references the old entity or group name.
|
||||||
|
|
||||||
|
For example, assume you have the following templated ACL policy that grants
|
||||||
|
users access to a `kv` plugin:
|
||||||
|
|
||||||
|
```hcl
|
||||||
|
path "kv/users/{{identity.entity.name}}/*" {
|
||||||
|
capabilities = ["read", "create", "update"]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
The policy grants a user entity named `janine` permission to read and write
|
||||||
|
key-value pairs to the `kv` plugin on the path `kv/users/janine/`. If there are
|
||||||
|
duplicate entities named `janine`, Vault renames all but one of entities
|
||||||
|
to `janine-<UUID>` during deduplication.
|
||||||
|
|
||||||
|
The ACL policy now gives the renamed entity permission to read and write
|
||||||
|
While the renamed entity can still access the `kv` plugin, the change in
|
||||||
|
no longer grants access to path `kv/users/janine/`. As a result, the user can no
|
||||||
|
longer access their data stored under `kv/users/janine/`.
|
||||||
|
|
||||||
|
|
||||||
|
## Find affected ACL policies
|
||||||
|
|
||||||
|
Deduplication has the potential to break templated ACL policies that rely on
|
||||||
|
entity or group name keys:
|
||||||
|
|
||||||
|
Template key | Description
|
||||||
|
------------------------------------------------------------- | ----------------
|
||||||
|
`identity.entity.name` | Entity name
|
||||||
|
`identity.entity.aliases.<mount accessor>.name` | Entity alias name for the given mount
|
||||||
|
`identity.groups.ids.<group id>.name` | Group name for the given group ID
|
||||||
|
`identity.groups.names.<group name>.id` | Group ID for the given group name
|
||||||
|
`identity.groups.names.<group name>.metadata.<metadata key>` | Metadata for the given group name and key
|
||||||
|
|
||||||
|
You can use the following `bash` script to check for templated policies using
|
||||||
|
entity or group names:
|
||||||
|
|
||||||
|
<Tabs>
|
||||||
|
|
||||||
|
<Tab heading="CLI" group="cli">
|
||||||
|
|
||||||
|
```bash
|
||||||
|
policy_list=$(vault policy list)
|
||||||
|
|
||||||
|
for policy in ${policy_list} ; do
|
||||||
|
|
||||||
|
# Get the policy details
|
||||||
|
policy_details=$(vault policy read ${policy})
|
||||||
|
|
||||||
|
# Check for a name based template key
|
||||||
|
if echo "${policy_details}" | grep -q "\.name" ; then
|
||||||
|
echo ""
|
||||||
|
echo "Policy name: ${policy}"
|
||||||
|
fi
|
||||||
|
echo "${policy_details}" |
|
||||||
|
tr -s ' ' '\n' |
|
||||||
|
grep "\.name" --label=" template key" -H
|
||||||
|
done
|
||||||
|
```
|
||||||
|
|
||||||
|
</Tab>
|
||||||
|
|
||||||
|
<Tab heading="API" group="api">
|
||||||
|
|
||||||
|
```bash
|
||||||
|
policy_list=$(
|
||||||
|
curl \
|
||||||
|
--silent \
|
||||||
|
--request GET \
|
||||||
|
--header "X-Vault-Token: ${VAULT_TOKEN}" \
|
||||||
|
${VAULT_ADDR}/v1/${VAULT_NAMESPACE}/sys/policy \
|
||||||
|
| jq .data.policies | jq -c '.[]' | tr -d '"'
|
||||||
|
)
|
||||||
|
|
||||||
|
for policy in ${policy_list} ; do
|
||||||
|
|
||||||
|
# Get the policy details
|
||||||
|
policy_details=$(
|
||||||
|
curl \
|
||||||
|
--silent \
|
||||||
|
--request GET \
|
||||||
|
--header "X-Vault-Token: ${VAULT_TOKEN}" \
|
||||||
|
${VAULT_ADDR}/v1/${VAULT_NAMESPACE}/sys/policy/${policy} \
|
||||||
|
| jq '.data.rules' | sed 's/\\"/"/g' | sed 's/\\n/ /g'
|
||||||
|
)
|
||||||
|
|
||||||
|
# Check for a name based template key
|
||||||
|
if echo "${policy_details}" | grep -q "\.name" ; then
|
||||||
|
echo ""
|
||||||
|
echo "Policy name: ${policy}"
|
||||||
|
fi
|
||||||
|
echo "${policy_details}" |
|
||||||
|
tr -s ' ' '\n' |
|
||||||
|
grep "\.name" --label=" template key" -H
|
||||||
|
done
|
||||||
|
```
|
||||||
|
|
||||||
|
</Tab>
|
||||||
|
|
||||||
|
</Tabs>
|
||||||
|
|
||||||
|
Compare the returned list of policies and template keys against the list of
|
||||||
|
deduplication targets identified in your system logs to determine if the policy
|
||||||
|
will break after duplication.
|
||||||
|
|
||||||
|
If you identify policies that may break during deduplication, consult with
|
||||||
|
the relevant teams to determine if the policies are still used and if you should
|
||||||
|
address the renaming problem before or after deduplication.
|
||||||
|
|
||||||
|
If you find policies that need remediation, there are two ways to fix user
|
||||||
|
access:
|
||||||
|
|
||||||
|
1. Update relevant templated ACL policies.
|
||||||
|
1. Move the affected resource.
|
||||||
|
|
||||||
|
|
||||||
|
## Solution 1: Update relevant ACL policies
|
||||||
|
|
||||||
|
The easiest way to deal with broken ACL policy templates due to deduplication is
|
||||||
|
to define new policies or add rules to existing policies that explicitly grants
|
||||||
|
access to previous resources for the affected entities.
|
||||||
|
|
||||||
|
If you have a large number of duplicate identities, it may be easier to
|
||||||
|
pre-populate a custom metadata field (e.g., `prev_name`) for the renamed
|
||||||
|
entities. Then create a single policy that uses the custom metadata field
|
||||||
|
instead of the entity name and attach that policy to the renamed entities.
|
||||||
|
|
||||||
|
<Tabs>
|
||||||
|
|
||||||
|
<Tab heading="CLI" group="cli">
|
||||||
|
|
||||||
|
Assume you have a file called `rename-targets.txt` with a list of entity names
|
||||||
|
currently attached to a templated ACL policy, which will no longer work after
|
||||||
|
deduplication.
|
||||||
|
|
||||||
|
1. Use `vault write` and the
|
||||||
|
[`{namespace}/identity/entity/name/{name}`](/vault/api-docs/secret/identity/entity#create-update-entity-by-name)
|
||||||
|
endpoint to update your target entries with a custom metadata field called
|
||||||
|
`old_name` set to the **current** entity name:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
while read entity_name; do
|
||||||
|
|
||||||
|
if [[ "" = "${entity_name}" ]] ; then continue ; fi
|
||||||
|
|
||||||
|
# Create a payload file with the new metadata field
|
||||||
|
echo -n '{"metadata": { "old_name": "'${entity_name}'"}}' > ./metadata.json
|
||||||
|
|
||||||
|
# Save the metadata to the entity
|
||||||
|
vault write /identity/entity/name/${entity_name} @metadata.json
|
||||||
|
|
||||||
|
done < rename-targets.txt
|
||||||
|
```
|
||||||
|
|
||||||
|
1. Create a new templatized ACL policy file that replaces the entity name
|
||||||
|
reference with a reference to the metadata field. For example:
|
||||||
|
|
||||||
|
```hcl
|
||||||
|
path "kv/users/{{identity.entity.metadata.old_name}}/*" {
|
||||||
|
capabilities = ["read", "create", "update"]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
1. Use `vault policy write` to create a new ACL policy with the policy
|
||||||
|
definition file:
|
||||||
|
|
||||||
|
```shell-session
|
||||||
|
$ vault policy write <policy_name> <path_to_policy_file>
|
||||||
|
```
|
||||||
|
|
||||||
|
For example:
|
||||||
|
|
||||||
|
<CodeBlockConfig hideClipboard="true">
|
||||||
|
|
||||||
|
```shell-session
|
||||||
|
$ vault policy write "kv-access-preservation" ./dedupe-policy.hcl
|
||||||
|
```
|
||||||
|
|
||||||
|
</CodeBlockConfig>
|
||||||
|
|
||||||
|
1. Use `vault read` with the
|
||||||
|
[`{namespace}/identity/entity/name/{name}`](/vault/api-docs/secret/identity/entity#create-update-entity-by-name)
|
||||||
|
path to read in the existing policy assignments and add the new policy to the
|
||||||
|
target entities. For example:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
policy_name="<policy_name>"
|
||||||
|
while read entity_name; do
|
||||||
|
|
||||||
|
if [[ "" = "${entity_name}" ]] ; then continue ; fi
|
||||||
|
|
||||||
|
# Create a payload file with new policy added to any existing policy
|
||||||
|
# assignments
|
||||||
|
vault read \
|
||||||
|
-field policies \
|
||||||
|
-format json \
|
||||||
|
/identity/entity/name/${entity_name} \
|
||||||
|
| jq ". + [\"${policy_name}\"] | {policies: .}" > policy_update.json
|
||||||
|
|
||||||
|
# Update the policy assignment for the entity
|
||||||
|
vault write /identity/entity/name/${entity_name} @policy_update.json
|
||||||
|
|
||||||
|
done < rename-targets.txt
|
||||||
|
```
|
||||||
|
|
||||||
|
</Tab>
|
||||||
|
|
||||||
|
<Tab heading="API" group="api">
|
||||||
|
|
||||||
|
Assume you have a file called `rename-targets.txt` with a list of entity names
|
||||||
|
currently attached to a templated ACL policy, which will no longer work after
|
||||||
|
deduplication.
|
||||||
|
|
||||||
|
1. Use the
|
||||||
|
[`{namespace}/identity/entity/name/{name}`](/vault/api-docs/secret/identity/entity#create-update-entity-by-name)
|
||||||
|
endpoint to update your target entries with a custom metadata field called
|
||||||
|
`old_name` set to the **current** entity name:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
while read entity_name; do
|
||||||
|
|
||||||
|
if [[ "" = "${entity_name}" ]] ; then continue ; fi
|
||||||
|
|
||||||
|
# Create a payload file with the new metadata
|
||||||
|
echo -n '{"metadata": { "old_name": "'${entity_name}'"}}' > ./metadata.json
|
||||||
|
|
||||||
|
# Save the metadata to the entity
|
||||||
|
curl \
|
||||||
|
--request POST \
|
||||||
|
--header "X-Vault-Token: ${VAULT_TOKEN}" \
|
||||||
|
--data @./metadata.json \
|
||||||
|
${VAULT_ADDR}/v1/${VAULT_NAMESPACE}/identity/entity/name/${entity_name}
|
||||||
|
|
||||||
|
done < rename-targets.txt
|
||||||
|
```
|
||||||
|
|
||||||
|
1. Create a new templatized ACL policy file that replaces the entity name
|
||||||
|
reference with a reference to the metadata field. For example:
|
||||||
|
|
||||||
|
```hcl
|
||||||
|
path "kv/users/{{identity.entity.metadata.old_name}}/*" {
|
||||||
|
capabilities = ["read", "create", "update"]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
1. Escape your policy file and make a `POST` call to the
|
||||||
|
[`{namespace}/sys/policy/{policy_name}`](/vault/api-docs/system/policy#create-update-policy)
|
||||||
|
with your policy details:
|
||||||
|
|
||||||
|
```shell-session
|
||||||
|
$ jq -Rs '{ "policy": . | gsub("[\\r\\n\\t]"; "") }' <path_to_policy_file> |
|
||||||
|
curl \
|
||||||
|
--request POST \
|
||||||
|
--header "X-Vault-Token: ${VAULT_TOKEN}" \
|
||||||
|
"$(</dev/stdin)" \
|
||||||
|
${VAULT_ADDR}/v1/${VAULT_NAMESPACE}/sys/policy/<policy_name>
|
||||||
|
```
|
||||||
|
|
||||||
|
For example:
|
||||||
|
|
||||||
|
<CodeBlockConfig hideClipboard="true">
|
||||||
|
|
||||||
|
```shell-session
|
||||||
|
$ jq -Rs '{ "policy": . | gsub("[\\r\\n\\t]"; "") }' ./dedupe-policy.hcl |
|
||||||
|
curl \
|
||||||
|
--request POST \
|
||||||
|
--header "X-Vault-Token: ${VAULT_TOKEN}" \
|
||||||
|
--data "$(</dev/stdin)" \
|
||||||
|
${VAULT_ADDR}/v1/${VAULT_NAMESPACE}/sys/policy/kv-access-preservation
|
||||||
|
```
|
||||||
|
</CodeBlockConfig>
|
||||||
|
|
||||||
|
`/sys/mounts/{plugin_mount_path}` does not return data on success.
|
||||||
|
|
||||||
|
|
||||||
|
1. Make a `GET` call to the
|
||||||
|
[`{namespace}/identity/entity/name/{name}`](/vault/api-docs/secret/identity/entity#create-update-entity-by-name)
|
||||||
|
endpoint to read in the existing policy assignments and add the new policy to the
|
||||||
|
target entities. For example:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
policy_name="<policy_name>"
|
||||||
|
while read entity_name; do
|
||||||
|
|
||||||
|
if [[ "" = "${entity_name}" ]] ; then continue ; fi
|
||||||
|
|
||||||
|
# Create a payload file with new policy added to any existing policy assignments
|
||||||
|
curl \
|
||||||
|
--request GET \
|
||||||
|
--header "X-Vault-Token: ${VAULT_TOKEN}" \
|
||||||
|
${VAULT_ADDR}/v1/${VAULT_NAMESPACE}/identity/entity/name/${entity_name} \
|
||||||
|
| jq ".data.policies + [\"${policy_name}\"] | {policies: .}" > policy_update.json
|
||||||
|
|
||||||
|
# Update the policy assignment for the entity
|
||||||
|
curl \
|
||||||
|
--request POST \
|
||||||
|
--header "X-Vault-Token: ${VAULT_TOKEN}" \
|
||||||
|
--data @policy_update.json \
|
||||||
|
${VAULT_ADDR}/v1/${VAULT_NAMESPACE}/identity/entity/name/${entity_name}
|
||||||
|
|
||||||
|
done < rename-targets.txt
|
||||||
|
```
|
||||||
|
|
||||||
|
</Tab>
|
||||||
|
|
||||||
|
</Tabs>
|
||||||
|
|
||||||
|
|
||||||
|
## Solution 2: Move the affected resource
|
||||||
|
|
||||||
|
Some resources are immovable and others are difficult to move. If you decide you
|
||||||
|
can, and want, to move resources affected by the renaming:
|
||||||
|
|
||||||
|
1. Inventory the relevant resources and the associated mount paths.
|
||||||
|
1. Determine which resources you will migrate before activating the
|
||||||
|
deduplication flag.
|
||||||
|
- Resources migrating **before** deduplication will be unavailable on the
|
||||||
|
old and new paths until **deduplication** completes.
|
||||||
|
- Resources migrating **after** deduplication will be unavailable until
|
||||||
|
**migration** completes.
|
||||||
|
1. Move any resources flagged for migration before deduplication and alert the
|
||||||
|
relevant users that the paths will remain unavailable until deduplication
|
||||||
|
completes.
|
||||||
|
1. [Enable the deduplication option for Vault](/vault/docs/upgrading/deduplication#dedupe-flag)
|
||||||
|
1. Move any resources flagged for migration after deduplication and alert the
|
||||||
|
relevant users that the paths will remain unavailable until migration
|
||||||
|
completes and the changes fully propagate.
|
||||||
221
website/content/docs/upgrading/deduplication/different-case.mdx
Normal file
221
website/content/docs/upgrading/deduplication/different-case.mdx
Normal file
@ -0,0 +1,221 @@
|
|||||||
|
---
|
||||||
|
layout: docs
|
||||||
|
page_title: Resolve different-case entity alias duplicates
|
||||||
|
description: >-
|
||||||
|
Fix duplicate identities for entity aliases due to case differences.
|
||||||
|
---
|
||||||
|
|
||||||
|
# Fix different-case entity alias duplicates
|
||||||
|
|
||||||
|
Fix duplicate identities for entity aliases due to case differences.
|
||||||
|
|
||||||
|
**You must review different-case entity alias duplicates before enabling forced
|
||||||
|
identity deduplication due to the potential security risk of incorrectly merging
|
||||||
|
distinct aliases**.
|
||||||
|
|
||||||
|
|
||||||
|
<Tip title="Assumptions">
|
||||||
|
|
||||||
|
- You are running Vault 1.19 or later.
|
||||||
|
- You have [deduplication **merge** targets in your system logs](/vault/docs/upgrading/identity-deduplication).
|
||||||
|
- You have admin permission on the relevant Vault server or cluster.
|
||||||
|
|
||||||
|
</Tip>
|
||||||
|
|
||||||
|
|
||||||
|
## Why duplicates happen
|
||||||
|
|
||||||
|
Historical bugs in Vault allowed authN mounts to create duplicate entries of the
|
||||||
|
same entity alias if the aliases used difference cases. For example, if an
|
||||||
|
external LDAP system returns unnormalized usernames or changes the normalization
|
||||||
|
scheme over time. By default, Vault uses case-insensitive matching for the
|
||||||
|
usernames and considers `username` and `uSeRnAmE` to be the same alias. But,
|
||||||
|
past bugs have left some users with different-case duplicates in storage.
|
||||||
|
|
||||||
|
To avoid the security impact of incorrectly merging identities that do not
|
||||||
|
represent the same user, Vault **tries** to honor past behavior by switching the
|
||||||
|
identity engine to use case-sensitive matching mode. But case-sensitive matching
|
||||||
|
is poorly supported, deviates from the foundational security assumptions Vault
|
||||||
|
makes, and can lead to unexpected behavior.
|
||||||
|
|
||||||
|
## Example server log
|
||||||
|
|
||||||
|
The Vault system log provides warnings about duplicate aliases, including:
|
||||||
|
|
||||||
|
- the alias string
|
||||||
|
- the associated mount path (`mount accessor`)
|
||||||
|
- the alias ID (`id`)
|
||||||
|
- the ID for the entity linked to the alias (`canonical_id`)
|
||||||
|
|
||||||
|
<CodeBlockConfig hideClipboard>
|
||||||
|
|
||||||
|
```text
|
||||||
|
[WARN] identity: 2 different-case entity alias duplicates found (potential security risk)
|
||||||
|
[WARN] identity: entity-alias "alias-case" with mount accessor "auth_userpass_34aca7ec" duplicates 1 others: id=df3568a4-3b65-4104-9481-1129ecbed72f canonical_id=5f013d99-a6c7-9a00-6ad5-4ad724b14f60 force_deduplication="would merge into entity 7da76b0d-fe9b-a125-3362-2a8ff055dcf8"
|
||||||
|
[WARN] identity: entity-alias "alias-cAsE" with mount accessor "auth_userpass_34aca7ec" duplicates 1 others: id=2992253b-1e99-4e47-b6f9-afb0c7cedf7a canonical_id=7da76b0d-fe9b-a125-3362-2a8ff055dcf8 force_deduplication="would merge others into this entity"
|
||||||
|
[WARN] identity: entity-alias "alias-case" with mount accessor "auth_userpass_a555989a" duplicates 1 others: id=0e4bd46e-a868-4dd4-a34a-cb73f097e3a5 canonical_id=1ce06951-e2fd-0923-8ae0-a0e2c6c2378b force_deduplication="would merge into entity 37f41fcf-5e15-f13f-248d-9fa8405b1ecd"
|
||||||
|
[WARN] identity: entity-alias "alias-cAsE" with mount accessor "auth_userpass_a555989a" duplicates 1 others: id=6cafe546-665c-4bd0-a4fd-5e699ba413c1 canonical_id=37f41fcf-5e15-f13f-248d-9fa8405b1ecd force_deduplication="would merge others into this entity"
|
||||||
|
[WARN] identity: end of different-case entity-alias duplicates
|
||||||
|
```
|
||||||
|
|
||||||
|
</CodeBlockConfig>
|
||||||
|
|
||||||
|
When reviewing system logs for duplicate resolution, always consider the authN
|
||||||
|
mount and namespace when grouping potential duplicates. For example, despite
|
||||||
|
having identical, case-insensitive names in the example logs, the four aliases
|
||||||
|
represent two distinct identities because they come from different authN mounts:
|
||||||
|
|
||||||
|
- `alias-case` and `alias-cAsE` are duplicates in the mount `auth_userpass_34aca7ec`
|
||||||
|
and will merge under the alias with canonical ID `7da76b0d-fe9b-a125-3362-2a8ff055dcf8`
|
||||||
|
during deduplication.
|
||||||
|
- `alias-case` and `alias-cAsE` are duplicates in the mount `auth_userpass_a555989a`
|
||||||
|
and will merge with the alias with canonical ID `37f41fcf-5e15-f13f-248d-9fa8405b1ecd`
|
||||||
|
during deduplication.
|
||||||
|
|
||||||
|
|
||||||
|
To resolve different-case duplicates, you must determine if the case difference
|
||||||
|
is a **mergeable duplicate** or **unmergeable duplicate**:
|
||||||
|
|
||||||
|
- For mergeable duplicates, the case difference is **unintentional** and the
|
||||||
|
duplicate entries represent the same logical entity. You can ignore mergeable
|
||||||
|
duplicates and let Vault force-merge them for you to establish and enforce
|
||||||
|
default identity matching behavior.
|
||||||
|
|
||||||
|
- For unmergeable duplicates, the case difference is **deliberate** and the
|
||||||
|
source identity provider **intended** to distinguish between different
|
||||||
|
entities using case. For example, `Alice` and `alice` are different entities
|
||||||
|
that correspond to different humans with distinct permission sets. **You must
|
||||||
|
resolve unmergeable duplicates before activating the deduplication flag.**
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
## Case 1: Resolving mergeable duplicates
|
||||||
|
|
||||||
|
Vault will force-merge duplicates once you enable forced identity deduplication
|
||||||
|
with the `force-identity-deduplication` flag. Vault also merges any policies
|
||||||
|
attached to the duplicates so the final entry has the union of all permissions
|
||||||
|
granted to the duplicates.
|
||||||
|
|
||||||
|
We recommend confirming the expected merge behavior **before** enabling
|
||||||
|
deduplication by reviewing the `force_deduplication` label in the log line to:
|
||||||
|
|
||||||
|
1. confirm the identified duplicates are all mergeable.
|
||||||
|
1. confirm permissions granted by the unified policy are appropriate for the
|
||||||
|
identity.
|
||||||
|
|
||||||
|
After deduplication, users can continue to log in using any case combination of
|
||||||
|
their username, but those logins will map to a single entity gated by the same
|
||||||
|
policies per the expected, default Vault behavior.
|
||||||
|
|
||||||
|
|
||||||
|
## Case 2: Resolving unmergeable duplicates
|
||||||
|
|
||||||
|
<Warning>
|
||||||
|
|
||||||
|
We strongly recommend resolving duplicates and moving away from case-sensitive
|
||||||
|
usernames as soon as possible to restore the security model supported by Vault.
|
||||||
|
|
||||||
|
Vault does not officially support case-sensitive names because it carries
|
||||||
|
significant security risks. And unresolved duplicates could merge at any time
|
||||||
|
due to unrelated changes in storage such as when deleting related entities or
|
||||||
|
groups.
|
||||||
|
|
||||||
|
</Warning>
|
||||||
|
|
||||||
|
When different-case variations of an alias name actually represent different
|
||||||
|
logical users or entities in an external system, investigate each alias before
|
||||||
|
resolving duplicates. You **must** resolve the duplicates on each mount before
|
||||||
|
enabling forced identity deduplication.
|
||||||
|
|
||||||
|
**If you confirm you need to keep the duplicated aliases**, you will need to
|
||||||
|
reconfigure the authN mount or external service to stop creating case-sensitive
|
||||||
|
aliases. For example, if the authN plugin allows it, you could update the plugin
|
||||||
|
configuration to append unique IDs to alias names to differentiate between
|
||||||
|
similar usernames.
|
||||||
|
|
||||||
|
Regardless of how you choose to address unmergeable duplicates, be mindful of
|
||||||
|
how the modified behavior may disrupt users as you roll out the change.
|
||||||
|
|
||||||
|
<Note>
|
||||||
|
|
||||||
|
We are not aware of specific integrations where changing mount or external
|
||||||
|
service behaviors is necessary and expect those situations to be rare. If you
|
||||||
|
determine your deployment requires changes to authN mounts or external services,
|
||||||
|
we encourage Vault Enterprise customers to work with HashiCorp support staff to
|
||||||
|
determine the best strategy and implementation.
|
||||||
|
|
||||||
|
</Note>
|
||||||
|
|
||||||
|
**If you can confirm you do not need the duplicated alias, or the entire auth
|
||||||
|
mount**, delete all but one of the entity aliases on each mount using the ID
|
||||||
|
noted in the log line with `id=`.
|
||||||
|
|
||||||
|
<Tabs>
|
||||||
|
|
||||||
|
<Tab heading="CLI" group="cli">
|
||||||
|
|
||||||
|
Use `vault delete` with the `/identity/entity-alias/id/{id}` path to delete the
|
||||||
|
duplicate identity:
|
||||||
|
|
||||||
|
```shell-session
|
||||||
|
$ vault delete /identity/entity-alias/id/<id>
|
||||||
|
```
|
||||||
|
|
||||||
|
For example, if one of the duplicate entity log entries includes
|
||||||
|
`id=df3568a4-3b65-4104-9481-1129ecbed72f`:
|
||||||
|
|
||||||
|
<CodeBlockConfig hideClipboard="true">
|
||||||
|
|
||||||
|
```shell-session
|
||||||
|
$ vault delete /identity/entity-alias/id/df3568a4-3b65-4104-9481-1129ecbed72f
|
||||||
|
|
||||||
|
Success! Data deleted (if it existed) at: identity/entity-alias/id/df3568a4-3b65-4104-9481-1129ecbed72f
|
||||||
|
```
|
||||||
|
|
||||||
|
</CodeBlockConfig>
|
||||||
|
|
||||||
|
</Tab>
|
||||||
|
|
||||||
|
<Tab heading="API" group="api">
|
||||||
|
|
||||||
|
Call the [`/identity/entity-alias/id/{id}`](/vault/api-docs/secret/identity/entity-alias#delete-entity-alias-by-id)
|
||||||
|
endpoint to delete the duplicate entry:
|
||||||
|
|
||||||
|
```shell-session
|
||||||
|
$ curl \
|
||||||
|
--request DELETE \
|
||||||
|
--header "X-Vault-Token: ${VAULT_TOKEN}" \
|
||||||
|
${VAULT_ADDR}/v1/identity/entity-alias/id/<id>
|
||||||
|
```
|
||||||
|
|
||||||
|
For example, if one of the duplicate entity log entries includes
|
||||||
|
`id=df3568a4-3b65-4104-9481-1129ecbed72f`:
|
||||||
|
|
||||||
|
<CodeBlockConfig hideClipboard="true">
|
||||||
|
|
||||||
|
```shell-session
|
||||||
|
$ curl \
|
||||||
|
--request DELETE \
|
||||||
|
--header "X-Vault-Token: ${VAULT_TOKEN}" \
|
||||||
|
${VAULT_ADDR}/v1/identity/entity-alias/id/df3568a4-3b65-4104-9481-1129ecbed72f | jq
|
||||||
|
```
|
||||||
|
|
||||||
|
</CodeBlockConfig>
|
||||||
|
|
||||||
|
</Tab>
|
||||||
|
|
||||||
|
</Tabs>
|
||||||
|
|
||||||
|
After you resolve the duplicates by deleting them or reconfiguring the relevant
|
||||||
|
services:
|
||||||
|
|
||||||
|
1. If the node is a standby, restart the node manually.
|
||||||
|
|
||||||
|
1. Recheck the system logs for the current node to confirm the duplicates no
|
||||||
|
longer appear in the log during unseal.
|
||||||
|
|
||||||
|
|
||||||
|
## Next steps
|
||||||
|
|
||||||
|
Once you confirm any remaining duplicates can be force-merged safely, you can
|
||||||
|
[enable forced identity deduplication](/vault/docs/upgrading/deduplication#dedupe-flag).
|
||||||
136
website/content/docs/upgrading/deduplication/entity-group.mdx
Normal file
136
website/content/docs/upgrading/deduplication/entity-group.mdx
Normal file
@ -0,0 +1,136 @@
|
|||||||
|
---
|
||||||
|
layout: docs
|
||||||
|
page_title: Fix entity and group duplicates
|
||||||
|
description: >-
|
||||||
|
Fix duplicate identities for Vault entities and groups
|
||||||
|
---
|
||||||
|
|
||||||
|
# Fix entity and group duplicates
|
||||||
|
|
||||||
|
Fix duplicate identities for entities and groups targeted for renaming.
|
||||||
|
|
||||||
|
<Tip title="Assumptions">
|
||||||
|
|
||||||
|
- You are running Vault 1.19 or later.
|
||||||
|
- You have [deduplication **renaming** targets in your system logs](/vault/docs/upgrading/identity-deduplication).
|
||||||
|
- You have admin permission on the relevant Vault server or cluster.
|
||||||
|
|
||||||
|
</Tip>
|
||||||
|
|
||||||
|
|
||||||
|
## Why duplicates happen
|
||||||
|
|
||||||
|
Historical bugs in Vault allowed authN mounts to create duplicate entries of the
|
||||||
|
same entity or group name if the names used difference cases. By default, Vault
|
||||||
|
uses case-insensitive matching for names and considers `bob` and `BOB` to be the
|
||||||
|
same name. But, past bugs have allowed for identical or different-case
|
||||||
|
duplicates in entity and group names.
|
||||||
|
|
||||||
|
Duplicate entity and group names can cause unexpected behavior and make working
|
||||||
|
with Vault more difficult:
|
||||||
|
|
||||||
|
- Fetching entities/groups by name returns the first identity found. Depending
|
||||||
|
on your Vault version, the identity returned can vary depending on the server
|
||||||
|
handling the request and when the last seal/unseal event occurred.
|
||||||
|
- Deleting entities/groups by name is not guaranteed to delete all duplicates.
|
||||||
|
- All lookup requests **must** use ID to find the right identity.
|
||||||
|
individual identity.
|
||||||
|
|
||||||
|
## Example server log
|
||||||
|
|
||||||
|
The Vault system log provides warnings about duplicate entities and groups,
|
||||||
|
including:
|
||||||
|
|
||||||
|
- the entity/group name
|
||||||
|
- the associated namespace (`namespace ID`)
|
||||||
|
- the entity/group ID (`id`)
|
||||||
|
- the expected deduplication action (`force_deduplication`)
|
||||||
|
|
||||||
|
<CodeBlockConfig hideClipboard>
|
||||||
|
|
||||||
|
```text
|
||||||
|
2025-01-28T13:15:13.641-0800 [WARN] identity: 2 entity duplicates found
|
||||||
|
2025-01-28T13:15:13.641-0800 [WARN] identity: entity "entity-cAsE" with namespace ID "root" duplicates 2 others: id=2562b42d-f603-ac6b-2591-8a20dd050897 force_deduplication="would not rename"
|
||||||
|
2025-01-28T13:15:13.641-0800 [WARN] identity: entity "entity-case" with namespace ID "root" duplicates 2 others: id=290a643d-6043-da5e-943f-3a3d09e4ecbd force_deduplication="would rename to entity-case-290a643d-6043-da5e-943f-3a3d09e4ecbd"
|
||||||
|
2025-01-28T13:15:13.641-0800 [WARN] identity: entity "entity-case" with namespace ID "root" duplicates 2 others: id=b0141be5-3f03-a1c7-a57b-02f045d04426 force_deduplication="would rename to entity-case-b0141be5-3f03-a1c7-a57b-02f045d04426"
|
||||||
|
2025-01-28T13:15:13.641-0800 [WARN] identity: entity "entity-cAsE" with namespace ID "sYMXY" duplicates 2 others: id=95f0743b-a1d5-26da-b4ef-a50490da0787 force_deduplication="would not rename"
|
||||||
|
2025-01-28T13:15:13.641-0800 [WARN] identity: entity "entity-case" with namespace ID "sYMXY" duplicates 2 others: id=9d3be96f-490a-9625-3118-896bd2b3a5f3 force_deduplication="would rename to entity-case-9d3be96f-490a-9625-3118-896bd2b3a5f3"
|
||||||
|
2025-01-28T13:15:13.641-0800 [WARN] identity: entity "entity-case" with namespace ID "sYMXY" duplicates 2 others: id=d82231d2-3716-6b9c-e80d-7d09c0409739 force_deduplication="would rename to entity-case-d82231d2-3716-6b9c-e80d-7d09c0409739"
|
||||||
|
2025-01-28T13:15:13.641-0800 [WARN] identity: end of entity duplicates
|
||||||
|
2025-01-28T13:15:13.641-0800 [WARN] identity: 2 group duplicates found
|
||||||
|
2025-01-28T13:15:13.641-0800 [WARN] identity: group "group-case" with namespace ID "root" duplicates 2 others: id=8ad26e0c-8cf6-5b67-7c77-6571fa374f34 force_deduplication="would not rename"
|
||||||
|
2025-01-28T13:15:13.641-0800 [WARN] identity: group "group-cAsE" with namespace ID "root" duplicates 2 others: id=9fe86ea0-f80c-1131-5be1-1d6e3b70237f force_deduplication="would rename to group-cAsE-9fe86ea0-f80c-1131-5be1-1d6e3b70237f"
|
||||||
|
2025-01-28T13:15:13.641-0800 [WARN] identity: group "group-case" with namespace ID "root" duplicates 2 others: id=32dd070c-c1f8-c796-9a71-15887014b813 force_deduplication="would rename to group-case-32dd070c-c1f8-c796-9a71-15887014b813"
|
||||||
|
2025-01-28T13:15:13.641-0800 [WARN] identity: group "group-case" with namespace ID "sYMXY" duplicates 2 others: id=8aaeff7e-7343-c883-1e0c-c5c9968f75a5 force_deduplication="would not rename"
|
||||||
|
2025-01-28T13:15:13.641-0800 [WARN] identity: group "group-case" with namespace ID "sYMXY" duplicates 2 others: id=f11277b3-d985-4d72-d2e9-9c8c6c0db02c force_deduplication="would rename to group-case-f11277b3-d985-4d72-d2e9-9c8c6c0db02c"
|
||||||
|
2025-01-28T13:15:13.641-0800 [WARN] identity: group "group-cAsE" with namespace ID "sYMXY" duplicates 2 others: id=7c753d07-b0d9-e13b-6184-48247b8f7504 force_deduplication="would rename to group-cAsE-07c753d07-b0d9-e13b-6184-48247b8f7504"
|
||||||
|
2025-01-28T13:15:13.641-0800 [WARN] identity: end of group duplicates
|
||||||
|
```
|
||||||
|
|
||||||
|
</CodeBlockConfig>
|
||||||
|
|
||||||
|
Duplicate entities and groups might be exact matches or differ in case (for
|
||||||
|
example, `Admin` and `admin`). But when reviewing system logs for duplicate
|
||||||
|
resolution, always consider the associated namespace when grouping potential
|
||||||
|
duplicates. For example, despite having identical, case-insensitive names in the
|
||||||
|
example logs, Vault only targets 8 for renaming because the names belong to
|
||||||
|
different namespaces:
|
||||||
|
|
||||||
|
Identity type | Old name | New name
|
||||||
|
------------- | ------------------- | --------
|
||||||
|
Entity | `root/entity-cAsE` | Unchanged
|
||||||
|
Entity | `root/entity-case` | `root/entity-case-290a643d-6043-da5e-943f-3a3d09e4ecbd`
|
||||||
|
Entity | `root/entity-case` | `root/entity-case-b0141be5-3f03-a1c7-a57b-02f045d04426`
|
||||||
|
Entity | `sYMXY/entity-cAsE` | Unchanged
|
||||||
|
Entity | `sYMXY/entity-case` | `sYMXY/entity-case-9d3be96f-490a-9625-3118-896bd2b3a5f3`
|
||||||
|
Entity | `sYMXY/entity-case` | `sYMXY/entity-case-d82231d2-3716-6b9c-e80d-7d09c0409739`
|
||||||
|
Group | `root/group-case` | Unchanged
|
||||||
|
Group | `root/group-cAsE` | `group-cAsE-9fe86ea0-f80c-1131-5be1-1d6e3b70237f`
|
||||||
|
Group | `root/group-case` | `group-case-32dd070c-c1f8-c796-9a71-15887014b813`
|
||||||
|
Group | `sYMXY/group-case` | Unchanged
|
||||||
|
Group | `sYMXY/group-case` | `group-case-f11277b3-d985-4d72-d2e9-9c8c6c0db02c`
|
||||||
|
Group | `sYMXY/group-cAsE` | `group-cAsE-07c753d07-b0d9-e13b-6184-48247b8f7504`
|
||||||
|
|
||||||
|
|
||||||
|
The automatic renaming process is security safe and preserves existing data.
|
||||||
|
Renaming **does not** grant new permissions to existing tokens and **does not**
|
||||||
|
delete data, so you can manually roll back the change if necessary.
|
||||||
|
|
||||||
|
|
||||||
|
## Resolving entity and group duplicates
|
||||||
|
|
||||||
|
In most cases, renaming preserves resource access such that logins and
|
||||||
|
permissions remain unaffected. But, there are two edge cases you may need to
|
||||||
|
address before deduplication: **templated policies** and **external references**.
|
||||||
|
|
||||||
|
- If you use templated ACL policies that reference an entity or group name as
|
||||||
|
part of the resource path, deduplication may affect access to those resources.
|
||||||
|
|
||||||
|
- If you use Terraform to manage Vault resources or have other systems outside
|
||||||
|
Vault that reference entity or group names, those references will break when the
|
||||||
|
deduplication process renames those entities/groups.
|
||||||
|
|
||||||
|
|
||||||
|
For each duplicate identified by the logs, you have 3 options:
|
||||||
|
|
||||||
|
1. **If you know you do not need or use the renaming target**, you can use
|
||||||
|
the
|
||||||
|
[`/identity/entity/id/{id}`](/vault/api-docs/secret/identity/entity#delete-entity-by-id) and
|
||||||
|
[`/identity/group/id/{id}`](/vault/api-docs/secret/identity/group#delete-group-by-id)
|
||||||
|
endpoints to delete the identities by ID.
|
||||||
|
|
||||||
|
1. **If you know the risk to templated policies or external references is low or
|
||||||
|
nonexistent**, you can opt to ignore the duplicates and address issues
|
||||||
|
if/when they occur.
|
||||||
|
|
||||||
|
|
||||||
|
1. **If you cannot confirm the risk to templated policies or external references**,
|
||||||
|
review the guidance in the following troubleshooting guide:
|
||||||
|
- [Resolve deduplication impact on ACL policy templates](/vault/docs/upgrading/deduplication/acl-policy-templates)
|
||||||
|
- [Resolve deduplication impact on Terraform resource references](/vault/docs/upgrading/deduplication/external-refs)
|
||||||
|
|
||||||
|
## Next steps
|
||||||
|
|
||||||
|
Once you are comfortable that all the entity and group duplicates are properly
|
||||||
|
addressed, you can
|
||||||
|
[enable forced identity deduplication](/vault/docs/upgrading/deduplication#dedupe-flag).
|
||||||
225
website/content/docs/upgrading/deduplication/index.mdx
Normal file
225
website/content/docs/upgrading/deduplication/index.mdx
Normal file
@ -0,0 +1,225 @@
|
|||||||
|
---
|
||||||
|
layout: docs
|
||||||
|
page_title: Resolve duplicate identities
|
||||||
|
description: >-
|
||||||
|
Find duplicate identities in your Vault cluster and safely resolve them.
|
||||||
|
---
|
||||||
|
|
||||||
|
# Find and resolve duplicate Vault identities
|
||||||
|
|
||||||
|
Bugs in Vault versions before 1.19 might lead to duplicate
|
||||||
|
[identities](/vault/docs/concepts/identity) for entities, aliases, and groups.
|
||||||
|
Duplicate identities in the persistent storage of a Vault cluster can cause
|
||||||
|
unexpected behavior as duplicate identities are outside typical expectations and
|
||||||
|
test scenarios for Vault.
|
||||||
|
|
||||||
|
Vault server logs include information to help you identify when duplicate
|
||||||
|
identities exist in a given cluster. We strongly recommend identifying
|
||||||
|
and resolving duplicates as soon as possible to return your Vault clusters to
|
||||||
|
normal, supported behavior.
|
||||||
|
|
||||||
|
## Before you start
|
||||||
|
|
||||||
|
- **You must have Vault 1.19 or later running on all clusters**. Vault only logs
|
||||||
|
deduplication details once you upgrade to 1.19+.
|
||||||
|
- **You must have admin permissions for the Vault cluster**.
|
||||||
|
|
||||||
|
## Step 1: Look for duplicates
|
||||||
|
|
||||||
|
To identify duplicates in server logs:
|
||||||
|
|
||||||
|
1. Look for `core: post-unseal setup starting` to find the last unseal operation
|
||||||
|
in the system logs of the active node. For example:
|
||||||
|
|
||||||
|
```text
|
||||||
|
[INFO] core: post-unseal setup starting
|
||||||
|
...<check here> ...
|
||||||
|
[INFO] core: post-unseal setup complete
|
||||||
|
```
|
||||||
|
|
||||||
|
1. Check for `DUPLICATES DETECTED` between the `setup starting` and
|
||||||
|
`setup complete` entries in the system log. For example:
|
||||||
|
|
||||||
|
```text
|
||||||
|
[WARN] identity: DUPLICATES DETECTED, see following logs for details [...]
|
||||||
|
```
|
||||||
|
|
||||||
|
If you do not see entries for `DUPLICATES DETECTED`, the current cluster is
|
||||||
|
clean and you can move to the next cluster. If you confirm none of the logs
|
||||||
|
have duplicates, you can jump to [Step #4](#dedupe-flag).
|
||||||
|
|
||||||
|
<Note>
|
||||||
|
|
||||||
|
If you use replication, repeat the duplication check on all primary clusters and
|
||||||
|
all **performance replication** (PR) secondary clusters. PR secondary clusters
|
||||||
|
handle client requests and may have additional, **local** duplicates.
|
||||||
|
|
||||||
|
Disaster recovery (DR) secondaries do not handle client requests, so you do not
|
||||||
|
need to check DR secondary clusters separately.
|
||||||
|
|
||||||
|
</Note>
|
||||||
|
|
||||||
|
## Step 2: Find deduplication targets
|
||||||
|
|
||||||
|
If you find `DUPLICATES DETECTED` in the system logs, create a list of same-case
|
||||||
|
aliases and renaming targets from the system log to make working through
|
||||||
|
duplicate resolution easier. If you have a large number of deduplication targets,
|
||||||
|
you can use the following script to pull out the relevant items and create
|
||||||
|
four files:
|
||||||
|
|
||||||
|
- **`unseal-process.log`** - section of the system log specific to the unseal
|
||||||
|
process.
|
||||||
|
- **`merge-details.txt`** - same-case identities Vault will auto-merge during
|
||||||
|
deduplication.
|
||||||
|
- **`rename-targets.txt`** - entities and group names in the form
|
||||||
|
`namespace/name` Vault will rename during deduplication.
|
||||||
|
- **`rename-details.txt`** - full rename details for rename targets.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
|
||||||
|
declare -A identity_key=(
|
||||||
|
['entity_alias']="identity: entity-alias"
|
||||||
|
['entity_rename']="identity: entity"
|
||||||
|
['group_rename']="identity: group"
|
||||||
|
)
|
||||||
|
|
||||||
|
declare -A match_str=(
|
||||||
|
['has_rename']="would rename to"
|
||||||
|
['has_merge']="would merge"
|
||||||
|
['rename']="s/.*would rename to \(.*\)\".*/\1/"
|
||||||
|
['group_name']="s/.*identity: group \"\(.*\)\" with.*/\1/"
|
||||||
|
['entity_name']="s/.*identity: entity \"\(.*\)\" with.*/\1/"
|
||||||
|
['namespace']="s/.*namespace ID \"\(.*\)\" duplicates.*/\1/"
|
||||||
|
)
|
||||||
|
|
||||||
|
declare -A rename_targets
|
||||||
|
|
||||||
|
# Grab the unseal portion of the log
|
||||||
|
log_start="core: post-unseal setup starting"
|
||||||
|
log_stop="core: post-unseal setup complete"
|
||||||
|
sed "/${log_start}/,/${log_stop}/!d;/${log_stop}/q" ../short.log > unseal-process.log
|
||||||
|
|
||||||
|
while read line;
|
||||||
|
do
|
||||||
|
|
||||||
|
# The log line relates to a renamed identity
|
||||||
|
if [[ "${line}" == *"${match_str['has_rename']}"* ]] ; then
|
||||||
|
|
||||||
|
if [[ "${line}" == *"${identity_key['entity_rename']}"* ]] ; then
|
||||||
|
type="${identity_key['entity_rename']}"
|
||||||
|
name_match="${match_str['entity_name']}"
|
||||||
|
fi
|
||||||
|
if [[ "${line}" == *"${identity_key['group_rename']}"* ]] ; then
|
||||||
|
type="${identity_key['group_rename']}"
|
||||||
|
name_match="${match_str['group_name']}"
|
||||||
|
fi
|
||||||
|
|
||||||
|
new=$(echo $line | sed -e "${match_str['rename']}")
|
||||||
|
old=$(echo $line | sed -e "${name_match}")
|
||||||
|
space=$(echo $line | sed -e "${match_str['namespace']}")
|
||||||
|
rename_targets["${space}/${old}"]="[${type}] ${space}/${old} --> ${space}/${new}"
|
||||||
|
|
||||||
|
fi
|
||||||
|
|
||||||
|
# The line indicates a same-case merge operation
|
||||||
|
if [[ "${line}" == *"${match_str['has_merge']}"* ]] ; then
|
||||||
|
merge_details=$(echo $line | sed -e "${identity_key['entity_alias']}")
|
||||||
|
echo ${merge_details} >> merge-details.txt
|
||||||
|
fi
|
||||||
|
|
||||||
|
done < unseal-process.log
|
||||||
|
|
||||||
|
# Save the rename target information to files for further processing
|
||||||
|
root_ns="root/"
|
||||||
|
for name in "${!rename_targets[@]}"
|
||||||
|
do
|
||||||
|
echo ${name//"${root_ns}"/""} >> rename-targets.txt
|
||||||
|
echo ${rename_targets["${name}"]} >> rename-details.txt
|
||||||
|
done
|
||||||
|
```
|
||||||
|
|
||||||
|
|
||||||
|
## Step 3: Resolve any duplicates
|
||||||
|
|
||||||
|
If you **did not** find duplicates in the log, you can jump to
|
||||||
|
[Step #5](#dedupe-flag).
|
||||||
|
|
||||||
|
If you **did** find duplicates in the logs:
|
||||||
|
|
||||||
|
1. Identify the duplication type for each entry in your server logs:
|
||||||
|
- **In PR/DR deployments**: identify duplication types on your primary
|
||||||
|
clusters.
|
||||||
|
- **In PR deployments**: identify duplication types for **local aliases** on
|
||||||
|
your performance secondaries. Local duplicates exist independently on the
|
||||||
|
secondary cluster and require explicit resolution but follow the same
|
||||||
|
process as resolving non-local different-case entity alias duplicates.
|
||||||
|
|
||||||
|
1. Follow the appropriate de-duplication steps based on the duplication type:
|
||||||
|
- [Fix different-case entity alias duplicates](/vault/docs/upgrading/deduplication/different-case)
|
||||||
|
- [Fix entity and group duplicates](/vault/docs/upgrading/deduplication/entity-group)
|
||||||
|
|
||||||
|
|
||||||
|
## Step 4: Prepare for (possible) latency impacts
|
||||||
|
|
||||||
|
When you activate deduplication for a cluster, the cluster reloads the in-memory
|
||||||
|
cache of all entities, aliases and groups. When deduplication replicates to
|
||||||
|
performance replication secondaries and performance standby nodes, those nodes
|
||||||
|
also pause and reload their identity system caches.
|
||||||
|
|
||||||
|
Vault remains unsealed during deduplication, but nodes may pause processing
|
||||||
|
other requests until deduplication completes. On **moderately large** clusters,
|
||||||
|
deduplication may take long enough to impact request latencies. On **large**
|
||||||
|
clusters, deduplication may take longer than 30 seconds, which could cause some
|
||||||
|
requests to timeout.
|
||||||
|
|
||||||
|
In general, deduplication should take less time than sealing and unsealing Vault
|
||||||
|
and be less disruptive than a regular failover. And clusters that had duplicates
|
||||||
|
should see their future `unseal` times reduced.
|
||||||
|
|
||||||
|
|
||||||
|
## Step 5: Enforce identity de-duplication ((#dedupe-flag))
|
||||||
|
|
||||||
|
Once you resolve existing duplicates, you can enable the
|
||||||
|
`force-identity-deduplication` activation flag on the primary cluster using the
|
||||||
|
[activation flag API path](/vault/api-docs/system/activation-flags).
|
||||||
|
|
||||||
|
Activating feature flags is a one-time, one-way action. Once you activate a
|
||||||
|
feature gated with a feature flag, you cannot un-activate the feature.
|
||||||
|
|
||||||
|
<Tabs>
|
||||||
|
|
||||||
|
<Tab heading="CLI" group="cli">
|
||||||
|
|
||||||
|
```shell-session
|
||||||
|
vault write -f sys/activation-flags/force-identity-deduplication/activate
|
||||||
|
```
|
||||||
|
|
||||||
|
</Tab>
|
||||||
|
|
||||||
|
<Tab heading="API" group="api">
|
||||||
|
|
||||||
|
```shell-session
|
||||||
|
$ curl \
|
||||||
|
--request PUT \
|
||||||
|
--header "X-Vault-Token: ${VAULT_TOKEN}" \
|
||||||
|
${VAULT_ADDR}/v1/sys/activation-flags/force-identity-deduplication/activate
|
||||||
|
```
|
||||||
|
|
||||||
|
</Tab>
|
||||||
|
|
||||||
|
</Tabs>
|
||||||
|
|
||||||
|
You can track the start and end of deduplication on each node in the system
|
||||||
|
by looking for `force-identity-deduplication activated` entries in the logs:
|
||||||
|
|
||||||
|
```
|
||||||
|
INFO core: force-identity-deduplication activated, reloading identity store
|
||||||
|
...
|
||||||
|
INFO core: force-identity-deduplication activated, reloading identity store complete
|
||||||
|
```
|
||||||
|
|
||||||
|
Going forward, Vault requires unique identities and re-runs the deduplication
|
||||||
|
check as a part of the unseal process. You can review the impact on unseal time
|
||||||
|
by reviewing system log timestamps. The difference between the log lines
|
||||||
|
`core: post-unseal setup starting` and `core: post-unseal setup complete`
|
||||||
|
indicates the total unseal time required for the cluster.
|
||||||
160
website/content/docs/upgrading/deduplication/terraform-refs.mdx
Normal file
160
website/content/docs/upgrading/deduplication/terraform-refs.mdx
Normal file
@ -0,0 +1,160 @@
|
|||||||
|
---
|
||||||
|
layout: docs
|
||||||
|
page_title: Resolve Terraform config
|
||||||
|
description: >-
|
||||||
|
Fix external reference behavior for deduplicated entities and groups in
|
||||||
|
Terraform config files.
|
||||||
|
---
|
||||||
|
|
||||||
|
# Resolve deduplication impact on Terraform resource references
|
||||||
|
|
||||||
|
Fix external reference behavior in Terraform configuration files for entities
|
||||||
|
and groups renamed during identity deduplication.
|
||||||
|
|
||||||
|
<Tip title="Assumptions">
|
||||||
|
|
||||||
|
- You are running Vault 1.19 or later.
|
||||||
|
- You have [deduplication **renaming** targets in your system logs](/vault/docs/upgrading/identity-deduplication).
|
||||||
|
- You have admin permission on the relevant Vault server or cluster.
|
||||||
|
|
||||||
|
</Tip>
|
||||||
|
|
||||||
|
|
||||||
|
## How renaming affects external references
|
||||||
|
|
||||||
|
Renaming entities and groups can break references in Terraform (and other
|
||||||
|
external services) when those reference refer directly to the entity or group by
|
||||||
|
name. For example, assume you have a Terraform configuration file with named
|
||||||
|
identity resources like the following:
|
||||||
|
|
||||||
|
<CodeBlockConfig highlight="11,16" hideClipboard="true">
|
||||||
|
|
||||||
|
```hcl
|
||||||
|
terraform {
|
||||||
|
required_providers {
|
||||||
|
vault = {
|
||||||
|
source = "hashicorp/vault"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
provider "vault" {}
|
||||||
|
|
||||||
|
resource "vault_identity_entity" "BOB" {
|
||||||
|
name = "BOB"
|
||||||
|
policies = ["TEST"]
|
||||||
|
}
|
||||||
|
|
||||||
|
resource "vault_identity_entity" "bob" {
|
||||||
|
name = "bob"
|
||||||
|
policies = ["test"]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
</CodeBlockConfig>
|
||||||
|
|
||||||
|
By default, Vault ignore case when matching identities, treats `BOB` and `bob`
|
||||||
|
as the same name, and rejects the second resource as a duplicate. However, if
|
||||||
|
your Vault cluster is running in a mode that allows both resource names due to
|
||||||
|
historical issues, the resources might exist as separate entities.
|
||||||
|
|
||||||
|
If Vault identifies `bob` and `BOB` as duplicates during deduplication, it
|
||||||
|
renames one of the identities `<name>-<uuid>`. After deduplication, Terraform
|
||||||
|
tries to reapply the previous name for the related resource, but the in-place
|
||||||
|
update fails because the existing resource now violates the case-insensitive
|
||||||
|
name constraint on the Vault side.
|
||||||
|
|
||||||
|
For example:
|
||||||
|
|
||||||
|
<CodeBlockConfig highlight="2,3,11,21,28" hideClipboard="true">
|
||||||
|
|
||||||
|
```
|
||||||
|
➜ tf_dupe_testing terraform apply
|
||||||
|
vault_identity_entity.bob: Refreshing state... [id=e8c5e633-fe37-5a49-4a29-32e2643d03bd]
|
||||||
|
vault_identity_entity.BOB: Refreshing state... [id=2577bc3f-67ab-dab7-93dc-e86f78194ff0]
|
||||||
|
|
||||||
|
Terraform used the selected providers to generate the following execution plan. Resource
|
||||||
|
actions are indicated with the following symbols:
|
||||||
|
~ update in-place
|
||||||
|
|
||||||
|
Terraform will perform the following actions:
|
||||||
|
|
||||||
|
# vault_identity_entity.bob will be updated in-place
|
||||||
|
~ resource "vault_identity_entity" "bob" {
|
||||||
|
+ external_policies = false
|
||||||
|
id = "e8c5e633-fe37-5a49-4a29-32e2643d03bd"
|
||||||
|
~ name = "bob-e8c5e633-fe37-5a49-4a29-32e2643d03bd" -> "bob"
|
||||||
|
# (3 unchanged attributes hidden)
|
||||||
|
}
|
||||||
|
|
||||||
|
...
|
||||||
|
|
||||||
|
vault_identity_entity.bob: Modifying... [id=e8c5e633-fe37-5a49-4a29-32e2643d03bd]
|
||||||
|
╷
|
||||||
|
│ Error: error updating IdentityEntity "e8c5e633-fe37-5a49-4a29-32e2643d03bd": Error making API request.
|
||||||
|
│
|
||||||
|
│ URL: PUT https://127.0.0.1:8200/v1/identity/entity/id/e8c5e633-fe37-5a49-4a29-32e2643d03bd
|
||||||
|
│ Code: 400. Errors:
|
||||||
|
│
|
||||||
|
│ * entity name is already in use
|
||||||
|
│
|
||||||
|
│ with vault_identity_entity.bob,
|
||||||
|
│ on main.tf line 17, in resource "vault_identity_entity" "bob":
|
||||||
|
│ 17: resource "vault_identity_entity" "bob" {
|
||||||
|
```
|
||||||
|
|
||||||
|
</CodeBlockConfig>
|
||||||
|
|
||||||
|
|
||||||
|
## Solution
|
||||||
|
|
||||||
|
The easiest way to deal with renamed entities and groups is to manually update the
|
||||||
|
the associated resource in your Terraform configuration with the updated name
|
||||||
|
before forcing deduplication.
|
||||||
|
|
||||||
|
<Tip>
|
||||||
|
|
||||||
|
Use the same process to identify and update target names for other external
|
||||||
|
systems that reference an entity or group by name.
|
||||||
|
|
||||||
|
</Tip>
|
||||||
|
|
||||||
|
For example, if your system logs include lines like the following:
|
||||||
|
|
||||||
|
<CodeBlockConfig hideClipboard="true">
|
||||||
|
|
||||||
|
```text
|
||||||
|
2025-01-28T13:15:13.641-0800 [WARN] identity: entity "bob" with namespace ID "admin" duplicates 1 others: id=8ad26e0c-8cf6-5b67-7c77-6571fa374881 force_deduplication="would not rename"
|
||||||
|
2025-01-28T13:15:13.641-0800 [WARN] identity: entity "BOB" with namespace ID "admin" duplicates 1 others: id=9fe86ea0-f80c-1199-5ad1-1d01ab70237f force_deduplication="would rename to BOB-9fe86ea0-f80c-1199-5ad1-1d01ab70237f"
|
||||||
|
```
|
||||||
|
|
||||||
|
</CodeBlockConfig>
|
||||||
|
|
||||||
|
You would update any resources associated with `BOB` in your Terraform
|
||||||
|
configuration files. For example:
|
||||||
|
|
||||||
|
<CodeBlockConfig highlight="11,16" hideClipboard="true">
|
||||||
|
|
||||||
|
```hcl
|
||||||
|
terraform {
|
||||||
|
required_providers {
|
||||||
|
vault = {
|
||||||
|
source = "hashicorp/vault"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
provider "vault" {}
|
||||||
|
|
||||||
|
resource "vault_identity_entity" "BOB-9fe86ea0-f80c-1199-5ad1-1d01ab70237f" {
|
||||||
|
name = "BOB-9fe86ea0-f80c-1199-5ad1-1d01ab70237f"
|
||||||
|
policies = ["TEST"]
|
||||||
|
}
|
||||||
|
|
||||||
|
resource "vault_identity_entity" "bob" {
|
||||||
|
name = "bob"
|
||||||
|
policies = ["test"]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
</CodeBlockConfig>
|
||||||
@ -42,6 +42,24 @@ based on the table below.
|
|||||||
| CE | true | any value other than sha2-512 | An error is returned | Pure Ed25519 |
|
| CE | true | any value other than sha2-512 | An error is returned | Pure Ed25519 |
|
||||||
| CE | true | sha2-512 | An error is returned (not supported on CE) | Pure Ed25519 |
|
| CE | true | sha2-512 | An error is returned (not supported on CE) | Pure Ed25519 |
|
||||||
|
|
||||||
|
### Identity system duplicate cleanup
|
||||||
|
|
||||||
|
**Users should review their server logs after upgrading to see if identity
|
||||||
|
duplicates are reported.**
|
||||||
|
|
||||||
|
Vault 1.19.0 enables users impacted by historical identity duplicate bugs to
|
||||||
|
manually trigger a one-time de-duplication process. This process restores the
|
||||||
|
cluster to supported default behavior by resolving the duplicates.
|
||||||
|
|
||||||
|
Vault 1.19.0 also includes improved reporting in server logs to help diagnose
|
||||||
|
whether this de-duplication is required. No behavior will change until the
|
||||||
|
operator activates the relevant flag via the API.
|
||||||
|
|
||||||
|
To understand if you need to take action, consult
|
||||||
|
[Resolve duplicate identities](/vault/docs/upgrading/deduplication) which
|
||||||
|
demonstrates the log lines to watch for and how to ensure your resolve
|
||||||
|
duplicates safely.
|
||||||
|
|
||||||
### LDAP user DN search with `upndomain`
|
### LDAP user DN search with `upndomain`
|
||||||
|
|
||||||
The github.com/hashicorp/cap/ldap dependency has been upgraded to include a security improvement
|
The github.com/hashicorp/cap/ldap dependency has been upgraded to include a security improvement
|
||||||
|
|||||||
@ -2551,6 +2551,31 @@
|
|||||||
"title": "Upgrade to Raft WAL",
|
"title": "Upgrade to Raft WAL",
|
||||||
"path": "upgrading/raft-wal"
|
"path": "upgrading/raft-wal"
|
||||||
},
|
},
|
||||||
|
{
|
||||||
|
"title": "Resolve duplicate identities",
|
||||||
|
"routes": [
|
||||||
|
{
|
||||||
|
"title": "Process overview",
|
||||||
|
"path": "upgrading/deduplication"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"title": "Fix different-case entity alias duplicates",
|
||||||
|
"path": "upgrading/deduplication/different-case"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"title": "Fix entity and group duplicates",
|
||||||
|
"path": "upgrading/deduplication/entity-group"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"title": "Resolve ACL policy templates",
|
||||||
|
"path": "upgrading/deduplication/acl-policy-templates"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"title": "Resolve Terraform config",
|
||||||
|
"path": "upgrading/deduplication/terraform-refs"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
{
|
{
|
||||||
"title": "Upgrade to 1.19.x",
|
"title": "Upgrade to 1.19.x",
|
||||||
"path": "upgrading/upgrade-to-1.19.x"
|
"path": "upgrading/upgrade-to-1.19.x"
|
||||||
|
|||||||
Loading…
Reference in New Issue
Block a user