* auth/ldap: fix login errors
This fixes 2 ldap auth login errors
* Missing entity alias attribute value
* Vault relies on case insensitive user attribute keys for mapping user
attributes to entity alias metadata. This sets the appropriate
configs in the cap library.
* ldap group search anonymous bind regression
* Anonymous group searches can be rejected by some LDAP servers if
they contain a userDN. This sets the configs in the cap library to
specify unauthenticated binds for anonymous group searches should
exclude a DN.
Closes https://github.com/hashicorp/vault/issues/26171
Closes https://github.com/hashicorp/vault/issues/26183
* changelog
* go mod tidy
* go get cap/ldap@latest and go mod tidy
* Validate OCSP response is signed by expected issuer and serial number matches request
- There was a bug in the OCSP response signature logic, it properly
verified but kept around the ocspRes object around so we ignored
the errors found and passed the response object back up the stack.
- Now extract the verification logic into a dedicated function, if
it returns an error, blank the ocspRes response as we can't trust it.
- Address an issue that the OCSP requests from multiple servers were
clobbering each others responses as the index loop variable was not
properly captured.
- Add a missing validation that the response was for the serial number
we requested
* Add cl
* CE parts for mount-namespace entry limit
* Remove redundant code from refactor
* Add doc comment note about ent-only use of interface
* Add CHANGELOG
* Support OCSP responses without a NextUpdate value set
- Validate that the ThisUpdate value is
properly prior to our current time and
if NextUpdate is set that, ThisUpdate is
before NextUpdate.
- If we don't have a value for NextUpdate just compare against ThisUpdate.
* Add ocsp_this_update_max_ttl support to cert auth
- Allow configuring a maximum TTL of the OCSP response based on the
ThisUpdate time like OpenSSL does
- Add test to validate that we don't cache OCSP responses with no NextUpdate
* Add cl
* Add missing ` in docs
* Rename ocsp_this_update_max_ttl to ocsp_this_update_max_age
* Missed a few TTL references
* Fix error message
* Address OCSP client caching issue
- The OCSP cache built into the client that is used by cert-auth
would cache the responses but when pulling out a cached value the
response wasn't validating properly and was then thrown away.
- The issue was around a confusion of the client's internal status
vs the Go SDK OCSP status integer values.
- Add a test that validates the cache is now used
* Add cl
* Fix PKI test failing now due to the OCSP cache working
- Remove the previous lookup before revocation as now the OCSP
cache works so we don't see the new revocation as we are actually
leveraging the cache
* Bump github.com/jackc/pgx/v4 from 4.18.1 to 4.18.2 in /sdk
Bumps [github.com/jackc/pgx/v4](https://github.com/jackc/pgx) from 4.18.1 to 4.18.2.
- [Changelog](https://github.com/jackc/pgx/blob/v4.18.2/CHANGELOG.md)
- [Commits](https://github.com/jackc/pgx/compare/v4.18.1...v4.18.2)
---
updated-dependencies:
- dependency-name: github.com/jackc/pgx/v4
dependency-type: indirect
...
Signed-off-by: dependabot[bot] <support@github.com>
* go mod tidy
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Peter Wilson <peter.wilson@hashicorp.com>
* Bump github.com/jackc/pgproto3/v2 from 2.3.2 to 2.3.3 in /sdk
Bumps [github.com/jackc/pgproto3/v2](https://github.com/jackc/pgproto3) from 2.3.2 to 2.3.3.
- [Commits](https://github.com/jackc/pgproto3/compare/v2.3.2...v2.3.3)
---
updated-dependencies:
- dependency-name: github.com/jackc/pgproto3/v2
dependency-type: indirect
...
Signed-off-by: dependabot[bot] <support@github.com>
* go mod tidy
* go mod tidy
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Peter Wilson <peter.wilson@hashicorp.com>
* fix VAULT-24372
* use redaction settings in context to redact values in sys/leader
* add tests to check redaction in GetLeaderStatus and GetSealStatus
* add ENT badge to sys/config/ui/custom-messages api-docs page in ToC
* remove unrelated change to website ToC
* add gosimport to make fmt and run it
* move installation to tools.sh
* correct weird spacing issue
* Update Makefile
Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com>
* fix a weird issue
---------
Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com>
Add WaitForMatchingMerkleRootsClients and Clients to sdk testcluster. Fix internal TestCluster.SetRootToken, which wasn't updating the builtin clients' token.
Initial version of an internal plugin interface for event subscription plugins,
and an AWS SQS plugin as an example.
Co-authored-by: Tom Proctor <tomhjp@users.noreply.github.com>
* Make plugin-specific env take precedence over sys env
* Expand the existing plugin env integration test
---------
Co-authored-by: Austin Gebauer <34121980+austingebauer@users.noreply.github.com>
* Base Binary Cert and CSR Parse functions.
* Add otherSANS parsing.
* Notate what doesn't exist on a CSR.
* Fix otherSans call err-checking and add basic-constriants to CSR
* Move BasicConstraint parsing to be optionally set.
* Refactored to use existing ParseBasicConstraintsExtension.
* Add handling for the ChangeSubjectName ext on CSR that is needed for EST
* Remove ChangeSubjectName - it's an attribute, not an extension, and there is no clean way to parse it, so pair down for now.
* Make these public methods, so they can be used in vault.
* Add unit tests for certutil.ParseCertificateToCreationParameters.
Also add unit tests for certutil.ParseCertificateToFields.
* Cleanup TestParseCertificate.
* Add unit tests for certutil.ParseCsrToCreationParameters and ParseCsrToFields.
* Fix return values for "add_basic_constraints" in certutil.ParseCsrToFields.
Add a test for parsing CSRs where "add_basic_constraints" is false.
* Clear up some todos.
* Add a test for certutil.ParseCertificateToCreationParameters for non-CA cert.
* Tweak TestParseCertificate/full_non_CA_cert.
* Basics of three remaining fields - keyUsage; extKeyUsage; PolicyIdentifiers
* Fix tests and err handling
* Add unit tests for policy_identifiers; ext_key_usage_oids; key_usage
* Add test on ext_key_usage_oids
* Remove duplicate usages elsewhere.
* Add error handling to csr-checks.
* Remove extranames on returned types.
* Remove useless function.
---------
Co-authored-by: Victor Rodriguez <vrizo@hashicorp.com>
This PR introduces a new testonly endpoint for introspecting the
RequestLimiter state. It makes use of the endpoint to verify that changes to
the request_limiter config are honored across reload.
In the future, we may choose to make the sys/internal/request-limiter/status
endpoint available in normal binaries, but this is an expedient way to expose
the status for testing without having to rush the design.
In order to re-use as much of the existing command package utility funcionality
as possible without introducing sprawling code changes, I introduced a new
server_util.go and exported some fields via accessors.
The tests shook out a couple of bugs (including a deadlock and lack of
locking around the core limiterRegistry state).
* add new plugin wif fields to AWS Secrets Engine
* add changelog
* go get awsutil v0.3.0
* fix up changelog
* fix test and field parsing helper
* godoc on new test
* require role arn when audience set
* make fmt
---------
Co-authored-by: Austin Gebauer <agebauer@hashicorp.com>
Co-authored-by: Austin Gebauer <34121980+austingebauer@users.noreply.github.com>
This commit introduces two new adaptive concurrency limiters in Vault,
which should handle overloading of the server during periods of
untenable request rate. The limiter adjusts the number of allowable
in-flight requests based on latency measurements performed across the
request duration. This approach allows us to reject entire requests
prior to doing any work and prevents clients from exceeding server
capacity.
The limiters intentionally target two separate vectors that have been
proven to lead to server over-utilization.
- Back pressure from the storage backend, resulting in bufferbloat in
the WAL system. (enterprise)
- Back pressure from CPU over-utilization via PKI issue requests
(specifically for RSA keys), resulting in failed heartbeats.
Storage constraints can be accounted for by limiting logical requests
according to their http.Method. We only limit requests with write-based
methods, since these will result in storage Puts and exhibit the
aforementioned bufferbloat.
CPU constraints are accounted for using the same underlying library and
technique; however, they require special treatment. The maximum number
of concurrent pki/issue requests found in testing (again, specifically
for RSA keys) is far lower than the minimum tolerable write request
rate. Without separate limiting, we would artificially impose limits on
tolerable request rates for non-PKI requests. To specifically target PKI
issue requests, we add a new PathsSpecial field, called limited,
allowing backends to specify a list of paths which should get
special-case request limiting.
For the sake of code cleanliness and future extensibility, we introduce
the concept of a LimiterRegistry. The registry proposed in this PR has
two entries, corresponding with the two vectors above. Each Limiter
entry has its own corresponding maximum and minimum concurrency,
allowing them to react to latency deviation independently and handle
high volumes of requests to targeted bottlenecks (CPU and storage).
In both cases, utilization will be effectively throttled before Vault
reaches any degraded state. The resulting 503 - Service Unavailable is a
retryable HTTP response code, which can be handled to gracefully retry
and eventually succeed. Clients should handle this by retrying with
jitter and exponential backoff. This is done within Vault's API, using
the go-retryablehttp library.
Limiter testing was performed via benchmarks of mixed workloads and
across a deployment of agent pods with great success.
Adds the ability to pin a version for a specific plugin type + name to enable an easier plugin upgrade UX. After pinning and reloading, that version should be the only version in use.
No HTTP API implementation yet for managing pins, so no user-facing effects yet.
* Implement raft-wal
* go mod tidy
* add metrics, fix a panic
* fix the panic for real this time
* PR feedback
* refactor tests to use a helper and reduce duplication
* add a test to verify we don't use raft-wal if raft.db exists
* add config to enable the verifier
* add tests for parsing verification intervals
* run the verifier in the background
* wire up the verifier
* go mod tidy
* refactor config parsing
* remove unused function
* trying to get the verifier working
* wire up some more verifier bits
* sorted out an error, added a new test, lots of debug logging that needs to come out
* fix a bug and remove all the debugging statements
* make sure we close raft-wal stablestore too
* run verifier tests for both boltdb and raft-wal
* PR feedback
* Vault 20270 docker test raft wal (#24463)
* adding a migration test from boltdb to raftwal and back
adding a migration test using snapshot restore
* feedback
* Update physical/raft/raft.go
Co-authored-by: Paul Banks <pbanks@hashicorp.com>
* PR feedback
* change verifier function
* make this shorter
* add changelog
* Fix Close behavior
* make supporting empty logs more explicit
* add some godocs
---------
Co-authored-by: hamid ghaf <hamid@hashicorp.com>
Co-authored-by: Hamid Ghaf <83242695+hghaf099@users.noreply.github.com>
Co-authored-by: Paul Banks <pbanks@hashicorp.com>
* Migration of OtherSANs Parsing Call to SDK helper from pki-issuer
* Based on PR feedback from Steve, remove internal variable, reference certutil directly.
- As TestInteg_KMIP_Audit showed, the x509.Certificate's
big.Int SerialNumber is mangled when we do a deep clone
of the LogInput's Request TLSConnection object.
- As the tls.ConnectionState does not have a Clone itself and
we don't modify this field, it should be safe to just grab
the existing reference into the cloned version.
* audit: entry_formatter update to ensure no race detection issues
* in progress with looking at a clone method for LogInput
* Tidy up LogInput Clone method
* less memory allocation
* fix hmac key clone
