* VAULT-22481: Audit filter node (#24465)
* Initial commit on adding filter nodes for audit
* tests for audit filter
* test: longer filter - more conditions
* copywrite headers
* Check interface for the right type
* Add audit filtering feature (#24554)
* Support filter nodes in backend factories and add some tests
* More tests and cleanup
* Attempt to move control of registration for nodes and pipelines to the audit broker (#24505)
* invert control of the pipelines/nodes to the audit broker vs. within each backend
* update noop audit test code to implement the pipeliner interface
* noop mount path has trailing slash
* attempting to make NoopAudit more friendly
* NoopAudit uses known salt
* Refactor audit.ProcessManual to support filter nodes
* HasFiltering
* rename the pipeliner
* use exported AuditEvent in Filter
* Add tests for registering and deregistering backends on the audit broker
* Add missing licence header to one file, fix a typo in two tests
---------
Co-authored-by: Peter Wilson <peter.wilson@hashicorp.com>
* Add changelog file
* update bexpr datum to use a strong type
* go docs updates
* test path
* PR review comments
* handle scenarios/outcomes from broker.send
* don't need to re-check the complete sinks
* add extra check to deregister to ensure that re-registering non-filtered device sets sink threshold
* Ensure that the multierror is appended before attempting to return it
---------
Co-authored-by: Peter Wilson <peter.wilson@hashicorp.com>
* Implement custom-message management endpoints in a namespace aware manner
* completion of non-enterprise version of custom-messages
* clean up of error handling and fixing a nil pointer error
* rename UICustomMessagesEntry to UICustomMessageEntry
* add unit tests to cover new functions in UIConfig related to custom messages
* unit tests for all custom message handling
* add missing header comments for new files
* add changelog file
* fix test setup error that led to unexpected failure
* change return type from slice of pointers to struct to slice of struct and add godocs to every function
* add Internal suffix to internal methods for the UIConfig struct
* add validation for start and end times of custom messages
* improvements based on review feedback
* explore new approach for custom messages
* introduce new error to force HTTP 404 when referencing non-existant UI custom message
* remove changelog entry until feature is complete
* implement CRUD endpoints using single storage entry per namespace
* add mutex to protect operations that read the storage entry and write it back
* add copyright header comment to new files
* fix failing tests due to change in target function behaviour in order to return 404 error when mandated
* feedback from review plus some improvements on my own as well
* define constants for recognized message types and replace hardcoded strings occurrences with new constants
* incorporate feedback comment
* beef up testing with non-root namespaces in putEntry and getEntryForNamespace
* renaming CreateMessage to AddMessage in uicustommessages.Manager and uicustommessages.Entry
* adding missing copyright header comments
* wip
* Work on the tuneable allowance and some bugs
* Call handleCancellableRequest instead, which gets the audit order more correct and includes the preauth response
* Get rid of no longer needed operation
* Phew, this wasn't necessary
* Add auth error handling by the backend, and fix a bug with handleInvalidCredentials
* Cleanup req/resp naming
* Use the new form, and data
* Discovered that tokens werent really being checked because isLoginRequest returns true for the re-request into the backend, when it shouldnt
* Add a few more checks in the delegated request handler for bad inputs
- Protect the delegated handler from bad inputs from the backend such
as an empty accessor, a path that isn't registered as a login request
- Add similar protections for bad auth results as we do in the normal
login request paths. Technically not 100% needed but if somehow the
handleCancelableRequest doesn't use the handleLoginRequest code path
we could get into trouble in the future
- Add delegated-auth-accessors flag to the secrets tune command and
api-docs
* Unit tests and some small fixes
* Remove transit preauth test, rely on unit tests
* Cleanup and add a little more commentary in tests
* Fix typos, add another failure use-case which we reference a disabled auth mount
* PR Feedback
- Use router to lookup mount instead of defining a new lookup method
- Enforce auth table types and namespace when mount is found
- Define a type alias for the handleInvalidCreds
- Fix typos/grammar
- Clean up globals in test
* Additional PR feedback
- Add test for delegated auth handler
- Force batch token usage
- Add a test to validate failures if a non-batch token is used
- Check for Data member being nil in test cases
* Update failure error message around requiring batch tokens
* Trap MFA requests
* Reword some error messages
* Add test and fixes for delegated response wrapping
* Move MFA test to dedicated mount
- If the delegated auth tests were running in parallel, the MFA test
case might influence the other tests, so move the MFA to a dedicated
mount
* PR feedback: use textproto.CanonicalMIMEHeaderKey
- Change the X-Vault-Wrap-Ttl constant to X-Vault-Wrap-TTL
and use textproto.CanonicalMIMEHeaderKey to format it
within the delete call.
- This protects the code around changes of the constant typing
* PR feedback
- Append Error to RequestDelegatedAuth
- Force error interface impl through explicit nil var assignment on
RequestDelegatedAuthError
- Clean up test factory and leverage NewTestSoloCluster
- Leverage newer maps.Clone as this is 1.16 only
---------
Co-authored-by: Scott G. Miller <smiller@hashicorp.com>
* Re-implementation of API redirects with more deterministic matching
* add missing file
* Handle query params properly
* licensing
* Add single src deregister
* Implement specifically RFC 5785 (.well-known) redirects.
Also implement a unit test for HA setups, making sure the standby node redirects to the active (as usual), and that then the active redirects the .well-known request to a backend, and that that is subsequently satisfied.
* Remove test code
* Rename well known redirect logic
* comments/cleanup
* PR feedback
* Remove wip typo
* Update http/handler.go
Co-authored-by: Steven Clark <steven.clark@hashicorp.com>
* Fix registrations with trailing slashes
---------
Co-authored-by: Steven Clark <steven.clark@hashicorp.com>
* wip
* more pruning
* Integrate OCSP into binary paths PoC
- Simplify some of the changes to the router
- Remove the binary test PKI endpoint
- Switch OCSP to use the new binary paths backend variable
* Fix proto generation and test compilation
* Add unit test for binary request handling
---------
Co-authored-by: Scott G. Miller <smiller@hashicorp.com>
* create custom type for disable-replication-status-endpoints context key
make use of custom context key type in middleware function
* clean up code to remove various compiler warnings
unnecessary return statement
if condition that is always true
fix use of deprecated ioutil.NopCloser
empty if block
* remove unused unexported function
* clean up code
remove unnecessary nil check around a range expression
* clean up code
removed redundant return statement
* use http.StatusTemporaryRedirect constant instead of literal integer
* create custom type for context key for max_request_size parameter
* create custom type for context key for original request path
Subscribing to events through a WebSocket now support boolean
expressions to filter only the events wanted based on the fields
* `event_type`
* `operation`
* `source_plugin_mount`
* `data_path`
* `namespace`
Example expressions:
These can be passed to `vault events subscribe`, e.g.,:
* `event_type == abc`
* `source_plugin_mount == secret/`
* `event_type != def and operation != write`
```sh
vault events subscribe -filter='source_plugin_mount == secret/' 'kv*'
```
The docs for the `vault events subscribe` command and API endpoint
will be coming shortly in a different PR, and will include a better
specification for these expressions, similar to (or linking to)
https://developer.hashicorp.com/boundary/docs/concepts/filtering
* reduce calls to DetermineRoleFromLoginRequest from 3 to 1 for aws auth method
* change ordering of LoginCreateToken args
* replace another determineRoleFromLoginRequest function with role from context
* add changelog
* Check for role in context if not there make call to DeteremineRoleFromLoginRequest
* move context role check below nanmespace check
* Update changelog/22583.txt
Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com>
* revert signature to same order
* make sure resp is last argument
* retrieve role from context closer to where role variable is needed
* remove failsafe for role in mfa login
* Update changelog/22583.txt
---------
Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com>
Biggest change: we rename `Send` to `SendEvent` in `logical.EventSender`..
Initially we picked `Send` to match the underlying go-eventlogger
broker's `Send` method, and to avoid the stuttering of `events.SendEvent`.
However, I think it is more useful for the `logical.EventSender`
interface to use the method `SendEvent` so that, for example,
`framework.Backend` can implement it.
This is a relatively change now that should not affect anything
except the KV plugin, which is being fixed in another PR.
Another change: if the `secret_path` metadata is present, then
the plugin-aware `EventBus` will prepend it with the plugin mount.
This allows the `secret_path` to be the full path to any referenced
secret.
This change is also backwards compatible, since this field was not
present in the KV plugin. (It did use the slightly different `path`
field, which we can keep for now.)
* Migrate protobuf generation to Buf
Buf simplifies the generation story and allows us to lean
into other features in the Buf ecosystem, such as dependency
management, linting, breaking change detection, formatting
and remote plugins.
* Format all protobuf files with buf
Also add a CI job to ensure formatting remains consistent
* Add CI job to warn on proto generate diffs
Some files were not regenerated with the latest version
of the protobuf binary. This CI job will ensure we are always
detect if the protobuf files need regenerating.
* Add CI job for linting protobuf files
* add APILockShouldBlockRequest to backend proto
* make proto
* add APILockShouldBlockRequest to system view
* Revert "make proto"
This reverts commit 7f33733f185e8a7419590d82150e85abdcc5e707.
* Revert "add APILockShouldBlockRequest to backend proto"
This reverts commit a3bf41f7f2a0811dd323fbff4da45a582c942f2b.
* move APILockShouldBlockRequest to extended sys view
* add changelog entry
* Add stub ACME billing interfaces
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add initial implementation of client count
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Correctly attribute to mount, namespace
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Refactor adding entities of custom types
This begins to add custom types of events; presently these are counted
as non-entity tokens, but prefixed with a custom ClientID prefix.
In the future, this will be the basis for counting these events
separately (into separate buckets and separate storage segments).
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Refactor creation of ACME mounts
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add test case for billing
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Better support managed key system view casting
Without an additional parameter, SystemView could be of a different
internal implementation type that cannot be directly casted to in OSS.
Use a separate parameter for the managed key system view to use instead.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Refactor creation of mounts for enterprise
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Validate mounts in ACME billing tests
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Use a hopefully unique separator for encoded identifiers
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Use mount accesor, not path
Co-authored-by: miagilepner <mia.epner@hashicorp.com>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Rename AddEventToFragment->AddActivityToFragment
Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
---------
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Co-authored-by: miagilepner <mia.epner@hashicorp.com>
Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com>
* Add header operation to sdk/logical
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add support for routing HEAD operations
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add changelog entry
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
---------
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
The [WebSockets spec](https://www.rfc-editor.org/rfc/rfc6455) states
that text messages must be valid UTF-8 encoded strings, which protobuf
messages virtually never are. This now correctly sends the protobuf events
as binary messages.
We change the format to correspond to CloudEvents, as originally intended,
and remove a redundant timestamp and newline.
We also bump the eventlogger to fix a race condition that this code triggers.
Also updates the event receieved to include a timestamp.
Websockets support both JSON and protobuf binary formats.
This can be used by either `wscat` or the new
`vault events subscribe`:
e.g.,
```sh
$ wscat -H "X-Vault-Token: $(vault print token)" --connect ws://127.0.0.1:8200/v1/sys/events/subscribe/abc?json=true
{"event":{"id":"5c5c8c83-bf43-7da5-fe88-fc3cac814b2e", "note":"testing"}, "eventType":"abc", "timestamp":"2023-02-07T18:40:50.598408Z"}
...
```
and
```sh
$ vault events subscribe abc
{"event":{"id":"5c5c8c83-bf43-7da5-fe88-fc3cac814b2e", "note":"testing"}, "eventType":"abc", "timestamp":"2023-02-07T18:40:50.598408Z"}
...
```
Co-authored-by: Tom Proctor <tomhjp@users.noreply.github.com>
This isn't perfect for sure, but it's solidifying and becoming a useful
base to work off.
This routes events sent from auth and secrets plugins to the main
`EventBus` in the Vault Core. Events sent from plugins are automatically
tagged with the namespace and plugin information associated with them.
* Adds managed key usages for MAC generate/verify and RNG.
* Remove MAC-related key usages from managed key in favor of sign/verify.
* Remove context from random source managed key interface.
* named MFA method configurations
* fix a test
* CL
* fix an issue with same config name different ID and add a test
* feedback
* feedback on test
* consistent use of passcode for all MFA methods (#18611)
* make use of passcode factor consistent for all MFA types
* improved type for MFA factors
* add method name to login CLI
* minor refactoring
* only accept MFA method name with its namespace path in the login request MFA header
* fix a bug
* fixing an ErrorOrNil return value
* more informative error message
* Apply suggestions from code review
Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com>
* feedback
* test refactor a bit
* adding godoc for a test
* feedback
* remove sanitize method name
* guard a possbile nil ref
Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com>
* Add WriteForwardedStorage to sdk's plugin, logical in OSS
This should allow backends to specify paths to forward write
(storage.Put(...) and storage.Delete(...)) operations for.
Notably, these semantics are subject to change and shouldn't yet be
relied on.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Collect paths for write forwarding in OSS
This adds a path manager to Core, allowing tracking across all Vault
versions of paths which could use write forwarding if available. In
particular, even on OSS offerings, we'll need to template {{clusterId}}
into the paths, in the event of later upgrading to Enterprise. If we
didn't, we'd end up writing paths which will no longer be accessible
post-migration, due to write forwarding now replacing the sentinel with
the actual cluster identifier.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add forwarded writer implementation to OSS
Here, for paths given to us, we determine if we need to do cluster
translation and perform local writing. This is the OSS variant.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Wire up mount-specific request forwarding in OSS
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Clarify that state lock needs to be held to call HAState in OSS
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Move cluster sentinel constant to sdk/logical
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Expose ClusterID to Plugins via SystemView
This will let plugins learn what the Cluster's ID is, without having to
resort to hacks like writing a random string to its cluster-prefixed
namespace and then reading it once it has replicated.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add GRPC ClusterID implementation
For any external plugins which wish to use it.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Creates a new `eventbus` package under `vault` with
an implementation of the `go-eventlogger` broker.
Also creates a stub of a common broker that will be accessible
in the core, and creates a simple event sending interface.
* add Link config, init, and capabilities
* add node status proto
* bump protoc version to 3.21.9
* make proto
* adding link tests
* remove wrapped link
* add changelog entry
* update changelog entry
Move version out of SDK. For now it's a copy rather than move: the part not addressed by this change is sdk/helper/useragent.String, which we'll want to remove in favour of PluginString. That will have to wait until we've removed uses of useragent.String from all builtins.
* VAULT-8719 Support data array for alias clash error response so UI can understand error
* VAULT-8719 Changelog
* VAULT-8719 Update alias mount update logic
* VAULT-8719 Further restrict IsError()
Add plugin version to GRPC interface
Added a version interface in the sdk/logical so that it can be shared between all plugin types, and then wired it up to RunningVersion in the mounts, auth list, and database systems.
I've tested that this works with auth, database, and secrets plugin types, with the following logic to populate RunningVersion:
If a plugin has a PluginVersion() method implemented, then that is used
If not, and the plugin is built into the Vault binary, then the go.mod version is used
Otherwise, the it will be the empty string.
My apologies for the length of this PR.
* Placeholder backend should be external
We use a placeholder backend (previously a framework.Backend) before a
GRPC plugin is lazy-loaded. This makes us later think the plugin is a
builtin plugin.
So we added a `placeholderBackend` type that overrides the
`IsExternal()` method so that later we know that the plugin is external,
and don't give it a default builtin version.
