mirror of
https://github.com/hashicorp/vault.git
synced 2025-08-22 23:21:08 +02:00
34a1796d03
* Create 'Troubleshoot' section * Remove extra spaces * Update redirects.js * Remove extra comma * Change the title * Update website/content/docs/troubleshoot/generate-root-token.mdx Co-authored-by: Brian Shumate <brianshumate@users.noreply.github.com> * Update website/content/docs/troubleshoot/generate-root-token.mdx Co-authored-by: Brian Shumate <brianshumate@users.noreply.github.com> * Update website/content/docs/troubleshoot/generate-root-token.mdx Co-authored-by: Brian Shumate <brianshumate@users.noreply.github.com> * Update website/content/docs/troubleshoot/generate-root-token.mdx Co-authored-by: Brian Shumate <brianshumate@users.noreply.github.com> * Update website/content/docs/troubleshoot/generate-root-token.mdx Co-authored-by: Brian Shumate <brianshumate@users.noreply.github.com> * Update website/content/docs/troubleshoot/generate-root-token.mdx Co-authored-by: Brian Shumate <brianshumate@users.noreply.github.com> * Update website/content/docs/troubleshoot/generate-root-token.mdx Co-authored-by: Brian Shumate <brianshumate@users.noreply.github.com> * Update website/content/docs/troubleshoot/generate-root-token.mdx Co-authored-by: Brian Shumate <brianshumate@users.noreply.github.com> * Update website/content/docs/troubleshoot/generate-root-token.mdx Co-authored-by: Brian Shumate <brianshumate@users.noreply.github.com> * Update website/content/docs/troubleshoot/generate-root-token.mdx Co-authored-by: Brian Shumate <brianshumate@users.noreply.github.com> * edit suggestions (#28047) * Fix the relative path - add missing '/' * Fix a typo --------- Co-authored-by: Brian Shumate <brianshumate@users.noreply.github.com> Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
106 lines
5.0 KiB
Plaintext
106 lines
5.0 KiB
Plaintext
---
|
|
layout: docs
|
|
page_title: Prevent lease explosions
|
|
description: >-
|
|
Learn how to prevent lease explosions in Vault.
|
|
---
|
|
|
|
# Prevent lease explosions
|
|
|
|
As your Vault environment scales to meet deployment needs, you run the risk of
|
|
lease explosions. Lease explosions can occur when a Vault cluster is
|
|
over-subscribed and clients overwhelm system resources with consistent,
|
|
high-volume API requests
|
|
|
|
Unchecked lease explosions create a memory drain on the active node, which can
|
|
cascade to other nodes and result in denial-of-service issues for the entire
|
|
cluster.
|
|
|
|
## Look for early warning signs
|
|
|
|
Cleaning up after a lease explosion is time consuming and resource intensive, so
|
|
we strongly recommend monitoring your Vault instance for signals that your
|
|
Vault deployment has matured and requires tuning:
|
|
|
|
Issue | Possible cause
|
|
-------------------------------------------------------------------------------- | --------------
|
|
Unused leases consume storage space for extended periods while waiting to expire | The TTL values for dynamic secret leases or authentication tokens may be too high
|
|
Lease revocation fails frequently | Failures in an external service (e.g., for dynamic secrets)
|
|
Build up of leases associated with unused credentials | Clients are not reusing valid, existing leases
|
|
Lease revocation is slow | Insufficient IOPS for the storage backend
|
|
Rapid lease count growth disproportionate to the number of clients | Misconfiguration or anti-patterns in client usage
|
|
|
|
|
|
## Enforce client best practices
|
|
|
|
High lease counts can degrade system performance:
|
|
|
|
- Use the smallest default time-to-live (TTL) possible for tokens and leases to
|
|
avoid excessive unexpired lease backlogs and high-volume, simultaneous
|
|
expirations.
|
|
- Review telemetry for aberrant client behavior that might lead to rapid
|
|
over-subscription.
|
|
- Limit the number of simultaneous dynamic secret requests and service token
|
|
authentication requests.
|
|
- Ensure that machine clients adhere to [recommended AppRole patterns](/vault/tutorials/recommended-patterns/pattern-approle).
|
|
- Review [AppRole best practices](https://www.hashicorp.com/blog/how-and-why-to-use-approle-correctly-in-hashicorp-vault).
|
|
|
|
## Set reasonable TTL guardrails
|
|
|
|
Choose appropriate defaults for your situation and use resource quotas as
|
|
guardrails against lease explosion. You can set default and maximum TTLs
|
|
globally, in the mount configuration for a specific authN or secrets plugin, and
|
|
at the role-level (e.g., database credential roles).
|
|
|
|
Vault prioritizes TTL values by granularity:
|
|
|
|
- Global values act as the default.
|
|
- Plugin TTL values override global values.
|
|
- Role, group, and user level TTL values override plugin and global values.
|
|
|
|
<Note title="TTL changes are not retroactive">
|
|
|
|
Leases and tokens keep the TTL value in affect during their creation. When you
|
|
adjust TTL values, the new limits only apply to leases and tokens issued after
|
|
you deploy the changes.
|
|
|
|
</Note>
|
|
|
|
## Monitor key metrics and logs
|
|
|
|
Proactive monitoring is key to finding problematic behavior and usage patterns
|
|
before they escalate:
|
|
|
|
- Review [key Vault metrics](/well-architected-framework/reliability/reliability-vault-monitoring-key-metrics)
|
|
- Understand [metric anti-patterns](/well-architected-framework/operational-excellence/security-vault-anti-patterns#poor-metrics-or-no-telemetry-data)
|
|
- Monitor [Vault audit device logs](/vault/tutorials/monitoring/monitor-telemetry-audit-splunk) for quota-related failures.
|
|
|
|
## Control resource usage with quotas
|
|
|
|
Use API rate limiting quotas and
|
|
[lease count quotas](/vault/tutorials/operations/resource-quotas#lease-count-quotas)
|
|
to limit the number of leases generated on a per-mount basis and control
|
|
resource consumption for your Vault instance where hard limits makes sense.
|
|
|
|
## Consider batch tokens
|
|
|
|
If your environment inherently leads to a large number of lease requests,
|
|
consider using batch tokens over service tokens.
|
|
|
|
The following resources can help you decide if batch tokens are reasonable for
|
|
your situation:
|
|
|
|
- [Vault service tokens vs batch tokens](/vault/tutorials/tokens/batch-tokens#service-tokens-vs-batch-tokens)
|
|
- [Service vs batch token lease handling](/vault/docs/concepts/tokens#service-vs-batch-token-lease-handling)
|
|
|
|
## Next steps
|
|
|
|
Proactive monitoring and periodic usage analysis can help you identify potential
|
|
problems before they escalate.
|
|
|
|
- Brush up on [general Vault resource quotas](/vault/docs/concepts/resource-quotas) in general.
|
|
- Learn about [lease count quotas for Vault Enterprise](/vault/docs/enterprise/lease-count-quotas).
|
|
- Learn how to [query audit device logs](/vault/tutorials/monitoring/query-audit-device-logs).
|
|
- Review [recommended Vault lease limits](/vault/docs/internals/limits#lease-limits).
|
|
- Review [lease anti-patterns](/well-architected-framework/operational-excellence/security-vault-anti-patterns#not-adjusting-the-default-lease-time) for a clear explanation of the issue and solution.
|