This PR introduces the CE plumbing for a new HTTP header, called
X-Vault-AOP-Force-Reject, which will force any associated request to
reject storage writes as if Vault were overloaded.
This flag is intended to test end-to-end functionality of write
rejection in Vault. This is specifically useful for testing 503 -
Service Unavailable HTTP response codes during load shedding.
* Add support for x_forwarded_for_client_cert_header
* add changelog entry
* add tests for a badly and properly formatted certs
* both conditions should be true
* handle case where r.TLS is nil
* prepend client_certs to PeerCertificates list
* Add support for x_forwarded_for_client_cert_header
* add changelog entry
* add tests for a badly and properly formatted certs
* both conditions should be true
* handle case where r.TLS is nil
* prepend client_certs to PeerCertificates list
* add option for decoders to handle different proxies
* Add support for x_forwarded_for_client_cert_header
* add changelog entry
* add tests for a badly and properly formatted certs
* both conditions should be true
* handle case where r.TLS is nil
* prepend client_certs to PeerCertificates list
* add option for decoders to handle different proxies
* fix tests
* fix typo
---------
Co-authored-by: Alexander Scheel <alex.scheel@hashicorp.com>
Co-authored-by: Scott Miller <smiller@hashicorp.com>
Co-authored-by: Violet Hynes <violet.hynes@hashicorp.com>
* fix VAULT-24372
* use redaction settings in context to redact values in sys/leader
* add tests to check redaction in GetLeaderStatus and GetSealStatus
* add ENT badge to sys/config/ui/custom-messages api-docs page in ToC
* remove unrelated change to website ToC
This PR introduces a new testonly endpoint for introspecting the
RequestLimiter state. It makes use of the endpoint to verify that changes to
the request_limiter config are honored across reload.
In the future, we may choose to make the sys/internal/request-limiter/status
endpoint available in normal binaries, but this is an expedient way to expose
the status for testing without having to rush the design.
In order to re-use as much of the existing command package utility funcionality
as possible without introducing sprawling code changes, I introduced a new
server_util.go and exported some fields via accessors.
The tests shook out a couple of bugs (including a deadlock and lack of
locking around the core limiterRegistry state).
This commit introduces two new adaptive concurrency limiters in Vault,
which should handle overloading of the server during periods of
untenable request rate. The limiter adjusts the number of allowable
in-flight requests based on latency measurements performed across the
request duration. This approach allows us to reject entire requests
prior to doing any work and prevents clients from exceeding server
capacity.
The limiters intentionally target two separate vectors that have been
proven to lead to server over-utilization.
- Back pressure from the storage backend, resulting in bufferbloat in
the WAL system. (enterprise)
- Back pressure from CPU over-utilization via PKI issue requests
(specifically for RSA keys), resulting in failed heartbeats.
Storage constraints can be accounted for by limiting logical requests
according to their http.Method. We only limit requests with write-based
methods, since these will result in storage Puts and exhibit the
aforementioned bufferbloat.
CPU constraints are accounted for using the same underlying library and
technique; however, they require special treatment. The maximum number
of concurrent pki/issue requests found in testing (again, specifically
for RSA keys) is far lower than the minimum tolerable write request
rate. Without separate limiting, we would artificially impose limits on
tolerable request rates for non-PKI requests. To specifically target PKI
issue requests, we add a new PathsSpecial field, called limited,
allowing backends to specify a list of paths which should get
special-case request limiting.
For the sake of code cleanliness and future extensibility, we introduce
the concept of a LimiterRegistry. The registry proposed in this PR has
two entries, corresponding with the two vectors above. Each Limiter
entry has its own corresponding maximum and minimum concurrency,
allowing them to react to latency deviation independently and handle
high volumes of requests to targeted bottlenecks (CPU and storage).
In both cases, utilization will be effectively throttled before Vault
reaches any degraded state. The resulting 503 - Service Unavailable is a
retryable HTTP response code, which can be handled to gracefully retry
and eventually succeed. Clients should handle this by retrying with
jitter and exponential backoff. This is done within Vault's API, using
the go-retryablehttp library.
Limiter testing was performed via benchmarks of mixed workloads and
across a deployment of agent pods with great success.
* Re-process .well-known redirects with a recursive handler call rather than a 302 redirect
* Track when the RequestURI mismatches path (in a redirect) and add it to the audit log
* call cancelFunc
* Re-implementation of API redirects with more deterministic matching
* add missing file
* Handle query params properly
* licensing
* Add single src deregister
* Implement specifically RFC 5785 (.well-known) redirects.
Also implement a unit test for HA setups, making sure the standby node redirects to the active (as usual), and that then the active redirects the .well-known request to a backend, and that that is subsequently satisfied.
* Remove test code
* Rename well known redirect logic
* comments/cleanup
* PR feedback
* Remove wip typo
* Update http/handler.go
Co-authored-by: Steven Clark <steven.clark@hashicorp.com>
* Fix registrations with trailing slashes
---------
Co-authored-by: Steven Clark <steven.clark@hashicorp.com>
* wip
* more pruning
* Integrate OCSP into binary paths PoC
- Simplify some of the changes to the router
- Remove the binary test PKI endpoint
- Switch OCSP to use the new binary paths backend variable
* Fix proto generation and test compilation
* Add unit test for binary request handling
---------
Co-authored-by: Scott G. Miller <smiller@hashicorp.com>
* create custom type for disable-replication-status-endpoints context key
make use of custom context key type in middleware function
* clean up code to remove various compiler warnings
unnecessary return statement
if condition that is always true
fix use of deprecated ioutil.NopCloser
empty if block
* remove unused unexported function
* clean up code
remove unnecessary nil check around a range expression
* clean up code
removed redundant return statement
* use http.StatusTemporaryRedirect constant instead of literal integer
* create custom type for context key for max_request_size parameter
* create custom type for context key for original request path
* CI: Pre-emptively delete logs dir after cache restore in test-collect-reports (#23600)
* Fix OktaNumberChallenge (#23565)
* remove arg
* changelog
* exclude changelog in verifying doc/ui PRs (#23601)
* Audit: eventlogger sink node reopen on SIGHUP (#23598)
* ensure nodes are asked to reload audit files on SIGHUP
* added changelog
* Capture errors emitted from all nodes during proccessing of audit pipelines (#23582)
* Update security-scan.yml
* Listeners: Redaction only for TCP (#23592)
* redaction should only work for TCP listeners, also fix bug that allowed custom response headers for unix listeners
* fix failing test
* updates from PR feedback
* fix panic when unlocking unlocked user (#23611)
* VAULT-18307: update rotation period for aws static roles on update (#23528)
* add disable_replication_status_endpoints tcp listener config parameter
* add wrapping handler for disabled replication status endpoints setting
* adapt disable_replication_status_endpoints configuration parsing code to refactored parsing code
* refactor configuration parsing code to facilitate testing
* fix a panic when parsing configuration
* update refactored configuration parsing code
* fix merge corruption
* add changelog file
* document new TCP listener configuration parameter
* make sure disable_replication_status_endpoints only has effect on TCP listeners
* use active voice for explanation of disable_replication_status_endpoints
* fix minor merge issue
---------
Co-authored-by: Kuba Wieczorek <kuba.wieczorek@hashicorp.com>
Co-authored-by: Angel Garbarino <Monkeychip@users.noreply.github.com>
Co-authored-by: Hamid Ghaf <83242695+hghaf099@users.noreply.github.com>
Co-authored-by: Peter Wilson <peter.wilson@hashicorp.com>
Co-authored-by: Mark Collao <106274486+mcollao-hc@users.noreply.github.com>
Co-authored-by: davidadeleon <56207066+davidadeleon@users.noreply.github.com>
Co-authored-by: kpcraig <3031348+kpcraig@users.noreply.github.com>
* add redaction config settings to listener
* sys seal redaction + test modification for default handler properties
* build date should be redacted by 'redact_version' too
* sys-health redaction + test fiddling
* sys-leader redaction
* added changelog
* Lots of places need ListenerConfig
* Renamed options to something more specific for now
* tests for listener config options
* changelog updated
* updates based on PR comments
* updates based on PR comments - removed unrequired test case field
* fixes for docker tests and potentially server dev mode related flags
* Seal HA: Use new SealWrappedValue type to abstract seal wrapped values
Introduce SealWrappedValue to abstract seal wrapped values.
Make SealWrappedValue capable of marshalling into a BlobInfo, when there is
plaintext or a single encryption, or to a custom serialization consisting of a
header, length and a marshalled MultiWrapValue protobuf.
* Vault-13769: Support configuring and using multiple seals for unsealing
* Make sealWrapBackend start using multiple seals
* Make seal.Access no longer implement wrapping.Wrapper.
Instead, add the Encrypt and Decrypt methods to the Access interface.
* Make raft snapshot system use funcs SealWrapValue + UnsealWrapValue.
Move the snapshot.Sealer implementation to the vault package to
avoid circular imports.
* Update sealWrapBackend to use multiple seals for encryption.
Use all the encryption wrappers when storing seal wrapped values.
Try do decrypt using the highest priority wrapper, but try all
combinations of encrypted values and wrappers if necessary.
* Allow the use of multiple seals for entropy augmentation
Add seal_name variable in entropy stanza
Add new MultiSourcer to accommodate the new entropy augmentation behavior.
* Individually health check each wrapper, and add a sys/seal-backend-status endpoint.
* Address a race, and also a failed test mock that I didn't catch
* Track partial wrapping failures...
... where one or more but not all access.Encrypts fail for a given write.
Note these failures by adding a time ordered UUID storage entry containing
the path in a special subdirectory of root storage. Adds a callback
pattern to accomplish this, with certain high value writes like initial
barrier key storage not allowing a partial failure. The followup work
would be to detect return to health and iterate through these storage
entries, rewrapping.
* Add new data structure to track seal config generation (#4492)
* Add new data structure to track seal config generation
* Remove import cycle
* Fix undefined variable errors
* update comment
* Update setSeal response
* Fix setSealResponse in operator_diagnose
* Scope the wrapper health check locks individually (#4491)
* Refactor setSeal function in server.go. (#4505)
Refactor setSeal function in server.go.
* Decouple CreateSecureRandomReaderFunc from seal package.
Instead of using a list of seal.SealInfo structs, make
CreateSecureRandomReaderFunc use a list of new EntropySourcerInfo structs. This
brakes the denpency of package configutil on the seal package.
* Move SealGenerationInfo tracking to the seal Access.
* Move SealGenerationInfo tracking to the seal Access.
The SealGenerationInfo is now kept track by a Seal's Access instead of by the
Config object. The access implementation now records the correct generation
number on seal wrapped values.
* Only store and read SealGenerationInfo if VAULT_ENABLE_SEAL_HA_BETA is true.
* Add MultiWrapValue protobuf message
MultiWrapValue can be used to keep track of different encryptions of a value.
---------
Co-authored-by: Victor Rodriguez <vrizo@hashicorp.com>
* Use generation to determine if a seal wrapped value is up-to-date. (#4542)
* Add logging to seal Access implementation.
* Seal HA buf format run (#4561)
* Run buf format.
* Add buf.lock to ensure go-kms-wrapping module is imported.
* Vault-18958: Add unit tests for config checks
* Add safety logic for seal configuration changes
* Revert "Add safety logic for seal configuration changes"
This reverts commit 7fec48035a5cf274e5a4d98901716d08d766ce90.
* changes and tests for checking seal config
* add ent tests
* remove check for empty name and add type into test cases
* add error message for empty name
* fix no seals test
---------
Co-authored-by: divyapola5 <divya@hashicorp.com>
* Handle migrations between single-wrapper and multi-wrapper autoSeals
* Extract method SetPhysicalSealConfig.
* Extract function physicalSealConfig.
The extracted function is the only code now reading SealConfig entries from
storage.
* Extract function setPhysicalSealConfig.
The extracted function is the only code now writing SealConfig entries from
storage (except for migration from the old recovery config path).
* Move SealConfig to new file vault/seal_config.go.
* Add SealConfigType quasy-enumeration.
SealConfigType is to serve as the typed values for field SealConfig.Type.
* Rename Seal.RecoveryType to RecoverySealConfigType.
Make RecoverySealConfigType return a SealConfigType instead of a string.
* Rename Seal.BarrierType to BarrierSealConfigType.
Make BarrierSealConfigType return a SealConfigType.
Remove seal.SealType (really a two-step rename to SealConfigType).
* Add Seal methods ClearBarrierConfig and ClearRecoveryConfig.
* Handle autoseal <-> multiseal migrations.
While going between single-wrapper and multiple-wrapper autoseals are not
migrations that require an unwrap seal (such as going from shamir to autoseal),
the stored "barrier" SealConfig needs to be updated in these cases.
Specifically, the value of SealConfg.Type is "multiseal" for autoSeals that have
more than one wrapper; on the other hand, for autoseals with a single wrapper,
SealConfig.Type is the type of the wrapper.
* Remove error return value from NewAutoSeal constructor.
* Automatically rewrap partially seal wrapped values on an interval
* Add in rewrapping of partially wrapped values on an interval, regardless of seal health/status.
* Don't set SealGenerationInfo Rewrapped flag in the partial rewrap call.
* Unexport the SealGenerationInfo's Rewrapped field, add a mutex to it for thread safe access, and add accessor methods for it.
* Add a success callback to the manual seal rewrap process that updates the SealGenerationInfo's rewrapped field. This is done via a callback to avoid an import cycle in the SealRewrap code.
* Fix a failing seal wrap backend test which was broken by the unexporting of SealGenerationInfo's Rewrapped field.
* Nil check the seal rewrap success callback before calling it.
* Change SealGenerationInfo rewrapped parameter to an atomic.Bool rather than a sync.RWMutex for simplicity and performance.
* Add nil check for SealAccess before updating SealGenerationInfo rewrapped status during seal rewrap call.
* Update partial rewrap check interval from 10 seconds to 1 minute.
* Update a reference to SealGenerationInfo Rewrapped field to use new getter method.
* Fix up some data raciness in partial rewrapping.
* Account for possibly nil storage entry when retrieving partially wrapped value.
* Allow multi-wrapper autoSeals to include disabled seal wrappers.
* Restore propagation of wrapper configuration errors by setSeal.
Function setSeal is meant to propagate non KeyNotFound errors returned by calls
to configutil.ConfigureWrapper.
* Remove unused Access methods SetConfig and Type.
* Allow multi-wrapper autoSeals to include disabled seal wrappers.
Make it possible for an autoSeal that uses multiple wrappers to include disabled
wrappers that can be used to decrypt entries, but are skipped for encryption.
e an unwrapSeal when there are disabled seals.
* Fix bug with not providing name (#4580)
* add suffix to name defaults
* add comment
* only change name for disabled seal
* Only attempt to rewrap partial values when all seals are healthy.
* Only attempt to rewrap partial values when all seals are healthy.
* Change logging level from info to debug for notice about rewrap skipping based on seal health.
* Remove stale TODOs and commented out code.
---------
Co-authored-by: rculpepper <rculpepper@hashicorp.com>
Co-authored-by: Larroyo <95649169+DeLuci@users.noreply.github.com>
Co-authored-by: Scott G. Miller <smiller@hashicorp.com>
Co-authored-by: Divya Pola <87338962+divyapola5@users.noreply.github.com>
Co-authored-by: Matt Schultz <matt.schultz@hashicorp.com>
Co-authored-by: divyapola5 <divya@hashicorp.com>
Co-authored-by: Rachel Culpepper <84159930+rculpepper@users.noreply.github.com>
For now, only the leader of a cluster can handle subscription requests,
so we forward the connection request otherwise.
We forward using a 307 temporary redirect (the fallback way).
Forwarding a request over gRPC currently only supports a single request
and response, but a websocket connection is long-lived with potentially
many messages back and forth.
We modified the `vault events subscribe` command to honor those
redirects. `wscat` supports them with the `-L` flag.
In the future, we may add a gRPC method to handle forwarding WebSocket
requests, but doing so adds quite a bit of complexity (even over
normal request forwarding) due to the intricate nature of the `http` /
`vault.Core` interactions required. (I initially went down this path.)
I added tests for the forwarding header, and also tested manually.
(Testing with `-dev-three-node` is a little clumsy since it does not
properly support experiments, for some reason.)
Co-authored-by: Tom Proctor <tomhjp@users.noreply.github.com>
* Initial oss-patch apply
* Added changelog
* Renamed changelog txt
* Added the imports to the handler file
* Added a check that no two ports are the same, and modified changelog
* Edited go sum entry
* Tidy up using go mod
* Use strutil instead
* Revert go sum and go mod
* Revert sdk go sum
* Edited go.sum to before
* Edited go.sum again to initial
* Revert changes
* Adding explicit MPL license for sub-package.
This directory and its subdirectories (packages) contain files licensed with the MPLv2 `LICENSE` file in this directory and are intentionally licensed separately from the BSL `LICENSE` file at the root of this repository.
* Adding explicit MPL license for sub-package.
This directory and its subdirectories (packages) contain files licensed with the MPLv2 `LICENSE` file in this directory and are intentionally licensed separately from the BSL `LICENSE` file at the root of this repository.
* Updating the license from MPL to Business Source License.
Going forward, this project will be licensed under the Business Source License v1.1. Please see our blog post for more details at https://hashi.co/bsl-blog, FAQ at www.hashicorp.com/licensing-faq, and details of the license at www.hashicorp.com/bsl.
* add missing license headers
* Update copyright file headers to BUS-1.1
* Fix test that expected exact offset on hcl file
---------
Co-authored-by: hashicorp-copywrite[bot] <110428419+hashicorp-copywrite[bot]@users.noreply.github.com>
Co-authored-by: Sarah Thompson <sthompson@hashicorp.com>
Co-authored-by: Brian Kassouf <bkassouf@hashicorp.com>
Also updates the event receieved to include a timestamp.
Websockets support both JSON and protobuf binary formats.
This can be used by either `wscat` or the new
`vault events subscribe`:
e.g.,
```sh
$ wscat -H "X-Vault-Token: $(vault print token)" --connect ws://127.0.0.1:8200/v1/sys/events/subscribe/abc?json=true
{"event":{"id":"5c5c8c83-bf43-7da5-fe88-fc3cac814b2e", "note":"testing"}, "eventType":"abc", "timestamp":"2023-02-07T18:40:50.598408Z"}
...
```
and
```sh
$ vault events subscribe abc
{"event":{"id":"5c5c8c83-bf43-7da5-fe88-fc3cac814b2e", "note":"testing"}, "eventType":"abc", "timestamp":"2023-02-07T18:40:50.598408Z"}
...
```
Co-authored-by: Tom Proctor <tomhjp@users.noreply.github.com>
Create global quotas of each type in every NewTestCluster. Also switch some key locks to use DeadlockMutex to make it easier to discover deadlocks in testing.
NewTestCluster also now starts the cluster, and the Start method becomes a no-op. Unless SkipInit is provided, we also wait for a node to become active, eliminating the need for WaitForActiveNode. This was needed because otherwise we can't safely make the quota api call. We can't do it in Start because Start doesn't return an error, and I didn't want to begin storing the testing object T instead TestCluster just so we could call t.Fatal inside Start.
The last change here was to address the problem of how to skip setting up quotas when creating a cluster with a nonstandard handler that might not even implement the quotas endpoint. The challenge is that because we were taking a func pointer to generate the real handler func, we didn't have any way to compare that func pointer to the standard handler-generating func http.Handler without creating a circular dependency between packages vault and http. The solution was to pass a method instead of an anonymous func pointer so that we can do reflection on it.
* VAULT-8719 Support data array for alias clash error response so UI can understand error
* VAULT-8719 Changelog
* VAULT-8719 Update alias mount update logic
* VAULT-8719 Further restrict IsError()
* Login MFA
* ENT OSS segragation (#14088)
* Delete method id if not used in an MFA enforcement config (#14063)
* Delete an MFA methodID only if it is not used by an MFA enforcement config
* Fixing a bug: mfa/validate is an unauthenticated path, and goes through the handleLoginRequest path
* adding use_passcode field to DUO config (#14059)
* add changelog
* preventing replay attack on MFA passcodes (#14056)
* preventing replay attack on MFA passcodes
* using %w instead of %s for error
* Improve CLI command for login mfa (#14106)
CLI prints a warning message indicating the login request needs to get validated
* adding the validity period of a passcode to error messages (#14115)
* PR feedback
* duo to handle preventing passcode reuse
Co-authored-by: hghaf099 <83242695+hghaf099@users.noreply.github.com>
Co-authored-by: hamid ghaf <hamid@hashicorp.com>
* VAULT-1564 report in-flight requests
* adding a changelog
* Changing some variable names and fixing comments
* minor style change
* adding unauthenticated support for in-flight-req
* adding documentation for the listener.profiling stanza
* adding an atomic counter for the inflight requests
addressing comments
* addressing comments
* logging completed requests
* fixing a test
* providing log_requests_info as a config option to determine at which level requests should be logged
* removing a member and a method from the StatusHeaderResponseWriter struct
* adding api docks
* revert changes in NewHTTPResponseWriter
* Fix logging invalid log_requests_info value
* Addressing comments
* Fixing a test
* use an tomic value for logRequestsInfo, and moving the CreateClientID function to Core
* fixing go.sum
* minor refactoring
* protecting InFlightRequests from data race
* another try on fixing a data race
* another try to fix a data race
* addressing comments
* fixing couple of tests
* changing log_requests_info to log_requests_level
* minor style change
* fixing a test
* removing the lock in InFlightRequests
* use single-argument form for interface assertion
* adding doc for the new configuration paramter
* adding the new doc to the nav data file
* minor fix
* Unify NewHTTPResponseWriter ant NewStatusHeaderResponseWriter to fix ResponseWriter issues
* adding changelog
* removing unnecessary function from the WrappingResponseWriter interface
* changing logical requests responseWriter type
* reverting change to HTTPResponseWriter
* Customizing HTTP headers in the config file
* Add changelog, fix bad imports
* fixing some bugs
* fixing interaction of custom headers and /ui
* Defining a member in core to set custom response headers
* missing additional file
* Some refactoring
* Adding automated tests for the feature
* Changing some error messages based on some recommendations
* Incorporating custom response headers struct into the request context
* removing some unused references
* fixing a test
* changing some error messages, removing a default header value from /ui
* fixing a test
* wrapping ResponseWriter to set the custom headers
* adding a new test
* some cleanup
* removing some extra lines
* Addressing comments
* fixing some agent tests
* skipping custom headers from agent listener config,
removing two of the default headers as they cause issues with Vault in UI mode
Adding X-Content-Type-Options to the ui default headers
Let Content-Type be set as before
* Removing default custom headers, and renaming some function varibles
* some refacotring
* Refactoring and addressing comments
* removing a function and fixing comments
* copy over the webui
move web_ui to http
remove web ui files, add .gitkeep
updates, messing with gitkeep and ignoring web_ui
update ui scripts
gitkeep
ignore http/web_ui
Remove debugging
remove the jwt reference, that was from something else
restore old jwt plugin
move things around
Revert "move things around"
This reverts commit 2a35121850f5b6b82064ecf78ebee5246601c04f.
Update ui path handling to not need the web_ui name part
add desc
move the http.FS conversion internal to assetFS
update gitignore
remove bindata dep
clean up some comments
remove asset check script that's no longer needed
Update readme
remove more bindata things
restore asset check
update packagespec
update stub
stub the assetFS method and set uiBuiltIn to false for non-ui builds
update packagespec to build ui
* fail if assets aren't found
* tidy up vendor
* go mod tidy
* updating .circleci
* restore tools.go
* re-re-re-run make packages
* re-enable arm64
* Adding change log
* Removing a file
Co-authored-by: hamid ghaf <hamid@hashicorp.com>
* VAULT-1303 when a request to vault fails, show namespace if set
* Adding changelog
* Fix Changelog file name
* Set namespace in ResponseWriter headers if it is set
* Using consts.NamespaceHeaderName instead of the literal string
