* add rotation_schedule field to db backend
* add cron schedule field
* use priority queue with scheduled rotation types
* allow marshalling of cron schedule type
* return warning on use of mutually exclusive fields
* handle mutual exclusion of rotation fields (#22306)
* handle mutual exclusion of rotation fields
* fix import
* adv ttl mgmt: add rotation_window field (#22303)
* adv ttl mgmt: add rotation_window field
* do some rotation_window validation and add unit tests
* adv ttl mgmt: Ensure initialization sets appropriate rotation schedule (#22341)
* general cleanup and refactor rotation type checks
* make NextRotationTime account for the rotation type
* add comments
* add unit tests to handle mutual exclusion (#22352)
* add unit tests to handle mutual exclusion
* revert rotation_test.go and add missing test case to path_roles_test.go
* adv ttl mgmt: add tests for init queue (#22376)
* Vault 18908/handle manual rotation (#22389)
* support manual rotation for schedule based roles
* update description and naming
* adv ttl mgmt: consider rotation window (#22448)
* consider rotation window
ensure rotations only occur within a rotation window for schedule-based
rotations
* use helper method to set priority in rotateCredential
* fix bug with priority check
* remove test for now
* add and remove comments
* add unit tests for manual rotation (#22453)
* adv ttl mgmt: add tests for rotation_window
* adv ttl mgmt: refactor window tests (#22472)
* Handle GET static-creds endpoint (#22476)
* update read static-creds endpoint to include correct resp data
* return rotation_window if set
* update
* add changelog
* add unit test for static-creds read endpoint (#22505)
* Add the ability to set seconds in cron schedule for testing purposes
* update test so we don't use global var
* update with suggestions
---------
Co-authored-by: JM Faircloth <jmfaircloth@hashicorp.com>
Co-authored-by: John-Michael Faircloth <fairclothjm@users.noreply.github.com>
* add rotation_schedule field to db backend
* add cron schedule field
* use priority queue with scheduled rotation types
* allow marshalling of cron schedule type
* return warning on use of mutually exclusive fields
* handle mutual exclusion of rotation fields (#22306)
* handle mutual exclusion of rotation fields
* fix import
* adv ttl mgmt: add rotation_window field (#22303)
* adv ttl mgmt: add rotation_window field
* do some rotation_window validation and add unit tests
* adv ttl mgmt: Ensure initialization sets appropriate rotation schedule (#22341)
* general cleanup and refactor rotation type checks
* make NextRotationTime account for the rotation type
* add comments
* add unit tests to handle mutual exclusion (#22352)
* add unit tests to handle mutual exclusion
* revert rotation_test.go and add missing test case to path_roles_test.go
* adv ttl mgmt: add tests for init queue (#22376)
* Vault 18908/handle manual rotation (#22389)
* support manual rotation for schedule based roles
* update description and naming
* adv ttl mgmt: consider rotation window (#22448)
* consider rotation window
ensure rotations only occur within a rotation window for schedule-based
rotations
* use helper method to set priority in rotateCredential
* fix bug with priority check
* remove test for now
* add and remove comments
* add unit tests for manual rotation (#22453)
* adv ttl mgmt: add tests for rotation_window
* adv ttl mgmt: refactor window tests (#22472)
* Handle GET static-creds endpoint (#22476)
* update read static-creds endpoint to include correct resp data
* return rotation_window if set
* update
* add changelog
* add unit test for static-creds read endpoint (#22505)
---------
Co-authored-by: Milena Zlaticanin <60530402+Zlaticanin@users.noreply.github.com>
* Adding explicit MPL license for sub-package.
This directory and its subdirectories (packages) contain files licensed with the MPLv2 `LICENSE` file in this directory and are intentionally licensed separately from the BSL `LICENSE` file at the root of this repository.
* Adding explicit MPL license for sub-package.
This directory and its subdirectories (packages) contain files licensed with the MPLv2 `LICENSE` file in this directory and are intentionally licensed separately from the BSL `LICENSE` file at the root of this repository.
* Updating the license from MPL to Business Source License.
Going forward, this project will be licensed under the Business Source License v1.1. Please see our blog post for more details at https://hashi.co/bsl-blog, FAQ at www.hashicorp.com/licensing-faq, and details of the license at www.hashicorp.com/bsl.
* add missing license headers
* Update copyright file headers to BUS-1.1
* Fix test that expected exact offset on hcl file
---------
Co-authored-by: hashicorp-copywrite[bot] <110428419+hashicorp-copywrite[bot]@users.noreply.github.com>
Co-authored-by: Sarah Thompson <sthompson@hashicorp.com>
Co-authored-by: Brian Kassouf <bkassouf@hashicorp.com>
The upcoming event main plugin will use a very similar pattern
as the database plugin map, so it makes sense to refactor this and move
this map out. It also cleans up the database plugin backend so that
it does not have to keep track of the lock.
Co-authored-by: Tom Proctor <tomhjp@users.noreply.github.com>
* poc: client cert credential type
* go mod tidy
* fix typo
* fix newUserReqToProto
* add changelog
* add newline
* fix changelog
* add test for the client cert generator
* Fix formatting
* unset signing bundle URLs
* set BasicConstraintsValidForNonCA to false
* backdate cert by 30s
* remove empty creation params URLs
* check cert BasicConstraintsValid
* set default key bits in newClientCertificateGenerator
* fix client cert gen test with default values
* Add default for key_type
* fix default key_type
* update test with default key type
* update test
* Update changelog/20425.txt
Co-authored-by: Austin Gebauer <34121980+austingebauer@users.noreply.github.com>
* set default key bits and sig bits
* remove the default for key type ad fix the test
* make fmt + add comments for each exported field
* restart test
---------
Co-authored-by: Austin Gebauer <agebauer@hashicorp.com>
Co-authored-by: Austin Gebauer <34121980+austingebauer@users.noreply.github.com>
* Removes _builtin_ versions from mount storage where it already exists
* Stops new builtin versions being put into storage on mount creation/tuning
* Stops the plugin catalog from returning a builtin plugin that has been overridden, so it more accurately reflects the plugins that are available to actually run
Add plugin version to GRPC interface
Added a version interface in the sdk/logical so that it can be shared between all plugin types, and then wired it up to RunningVersion in the mounts, auth list, and database systems.
I've tested that this works with auth, database, and secrets plugin types, with the following logic to populate RunningVersion:
If a plugin has a PluginVersion() method implemented, then that is used
If not, and the plugin is built into the Vault binary, then the go.mod version is used
Otherwise, the it will be the empty string.
My apologies for the length of this PR.
* Placeholder backend should be external
We use a placeholder backend (previously a framework.Backend) before a
GRPC plugin is lazy-loaded. This makes us later think the plugin is a
builtin plugin.
So we added a `placeholderBackend` type that overrides the
`IsExternal()` method so that later we know that the plugin is external,
and don't give it a default builtin version.
* Support version selection for database plugins
* Don't consider unversioned plugins for version selection algorithm
* Added version to 'plugin not found' error
* Add PluginFactoryVersion function to avoid changing sdk/ API
v6 was released in the last 24h, and our tests fail to connect to the db when v6 is used.
Using v6 needs investigating, but for now I'm pinning to the last known good version.
strings.ReplaceAll(s, old, new) is a wrapper function for
strings.Replace(s, old, new, -1). But strings.ReplaceAll is more
readable and removes the hardcoded -1.
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
Add database plugin metrics around connections
This is a replacement for #15923 that takes into account recent lock
cleanup.
I went ahead and added back in the hanging plugin test, which I meant to
add in #15944 but forgot.
I tested this by spinning up a statsd sink in the tests and verifying I
got a stream of metrics:
```
$ nc -u -l 8125 | grep backend
test.swenson-Q9Q0L72D39.secrets.database.backend.connections.count.pgx.5.:1.000000|g
test.swenson-Q9Q0L72D39.secrets.database.backend.connections.count.pgx.5.:0.000000|g
test.swenson-Q9Q0L72D39.secrets.database.backend.connections.count.pgx.5.:1.000000|g
test.swenson-Q9Q0L72D39.secrets.database.backend.connections.count.pgx.5.:0.000000|g
```
We have to rework the shared gauge code to work without a full
`ClusterMetricSink`, since we don't have access to the core metrics from
within a plugin.
This only reports metrics every 10 minutes by default, but it solves
some problems we would have had with the gauge values becoming stale and
needing to be re-sent.
Co-authored-by: Tom Proctor <tomhjp@users.noreply.github.com>
Cleanup and simplify lock usage in database plugin
Following up from discussions in #15923 and #15933, I wanted to split
out a separate PR that drastically reduced the complexity of the use of
the databaseBackend lock. We no longer need it at all for the
`credRotationQueue`, and we can move it to be solely used in a few,
small connections map management functions.
Co-authored-by: Calvin Leung Huang <1883212+calvn@users.noreply.github.com>
* WIP replacing lib/pq
* change timezome param to be URI format
* add changelog
* add changelog for redshift
* update changelog
* add test for DSN style connection string
* more parseurl and quoteidentify to sdk; include copyright and license
* call dbutil.ParseURL instead, fix import ordering
Co-authored-by: Calvin Leung Huang <1883212+calvn@users.noreply.github.com>
VAULT-5827 Don't prepare SQL queries before executing them
We don't support proper prepared statements, i.e., preparing once and
executing many times since we do our own templating. So preparing our
queries does not really accomplish anything, and can have severe
performance impacts (see
https://github.com/hashicorp/vault-plugin-database-snowflake/issues/13
for example).
This behavior seems to have been copy-pasted for many years but not for
any particular reason that we have been able to find. First use was in
https://github.com/hashicorp/vault/pull/15
So here we switch to new methods suffixed with `Direct` to indicate
that they don't `Prepare` before running `Exec`, and switch everything
here to use those. We maintain the older methods with the existing
behavior (with `Prepare`) for backwards compatibility.
VAULT-5827 Update mongodb, brotli
Closes https://github.com/hashicorp/vault-plugin-secrets-mongodbatlas/issues/11
* `brotli` 1.0.1 was withdrawn
* `go-client-mongodb-atlas` has an old dependency on a renamed repo, and
has been renamed twice. This caused issues in
https://github.com/hashicorp/vault-plugin-secrets-mongodbatlas/issues/11
for example.
* VAULT-5827 Set unwrap token during database tests
The unwrap token is necessary for the plugins to start correctly when
running when running acceptance tests locally, e.g.,
```
$ VAULT_MONGODBATLAS_PROJECT_ID=... VAULT_MONGODBATLAS_PRIVATE_KEY=... VAULT_MONGODBATLAS_PUBLIC_KEY=... TEST='-run TestBackend_StaticRole_Rotations_MongoDBAtlas github.com/hashicorp/vault/builtin/logical/database' make test
--- FAIL: TestBackend_StaticRole_Rotations_MongoDBAtlas (5.33s)
rotation_test.go:818: err:%!s(<nil>) resp:&logical.Response{Secret:<nil>, Auth:<nil>, Data:map[string]interface {}{"error":"error creating database object: invalid database version: 2 errors occurred:\n\t* Unrecognized remote plugin message: PASS\n\nThis usually means that the plugin is either invalid or simply\nneeds to be recompiled to support the latest protocol.\n\t* Incompatible API version with plugin. Plugin version: 5, Client versions: [3 4]\n\n"}, Redirect:"", Warnings:[]string(nil), WrapInfo:(*wrapping.ResponseWrapInfo)(nil), Headers:map[string][]string(nil)}
```
Note the `PASS` message there, which indicates that the plugin exited
before starting the RPC server.
* Warnings indicating ignored and replaced parameters
* Avoid additional var creation
* Add warnings only if the response is non-nil
* Return the response even when error is non-nil
* Fix tests
* Rearrange comments
* Print warning in the log
* Fix another test
* Add CL
The URL password redaction operation did not handle the case where the
database connection URL was provided as a percent-encoded string, and
its password component contained reserved characters. It attempted to
redact the password by replacing the unescaped password in the
percent-encoded URL. This resulted in the password being revealed when
reading the configuration from Vault.
* plugin/catalog: support plugin registration whe type is explicitly provided
* don't use database type on plugin backend test; mock doesn't satisfy the DB interface
* check multiplexing support from plugin directly on newPluginClient
* do not return mutiplexed bool on catalog helper funcs
* feat: DB plugin multiplexing (#13734)
* WIP: start from main and get a plugin runner from core
* move MultiplexedClient map to plugin catalog
- call sys.NewPluginClient from PluginFactory
- updates to getPluginClient
- thread through isMetadataMode
* use go-plugin ClientProtocol interface
- call sys.NewPluginClient from dbplugin.NewPluginClient
* move PluginSets to dbplugin package
- export dbplugin HandshakeConfig
- small refactor of PluginCatalog.getPluginClient
* add removeMultiplexedClient; clean up on Close()
- call client.Kill from plugin catalog
- set rpcClient when muxed client exists
* add ID to dbplugin.DatabasePluginClient struct
* only create one plugin process per plugin type
* update NewPluginClient to return connection ID to sdk
- wrap grpc.ClientConn so we can inject the ID into context
- get ID from context on grpc server
* add v6 multiplexing protocol version
* WIP: backwards compat for db plugins
* Ensure locking on plugin catalog access
- Create public GetPluginClient method for plugin catalog
- rename postgres db plugin
* use the New constructor for db plugins
* grpc server: use write lock for Close and rlock for CRUD
* cleanup MultiplexedClients on Close
* remove TODO
* fix multiplexing regression with grpc server connection
* cleanup grpc server instances on close
* embed ClientProtocol in Multiplexer interface
* use PluginClientConfig arg to make NewPluginClient plugin type agnostic
* create a new plugin process for non-muxed plugins
* feat: plugin multiplexing: handle plugin client cleanup (#13896)
* use closure for plugin client cleanup
* log and return errors; add comments
* move rpcClient wrapping to core for ID injection
* refactor core plugin client and sdk
* remove unused ID method
* refactor and only wrap clientConn on multiplexed plugins
* rename structs and do not export types
* Slight refactor of system view interface
* Revert "Slight refactor of system view interface"
This reverts commit 73d420e5cd.
* Revert "Revert "Slight refactor of system view interface""
This reverts commit f75527008a.
* only provide pluginRunner arg to the internal newPluginClient method
* embed ClientProtocol in pluginClient and name logger
* Add back MLock support
* remove enableMlock arg from setupPluginCatalog
* rename plugin util interface to PluginClient
Co-authored-by: Brian Kassouf <bkassouf@hashicorp.com>
* feature: multiplexing: fix unit tests (#14007)
* fix grpc_server tests and add coverage
* update run_config tests
* add happy path test case for grpc_server ID from context
* update test helpers
* feat: multiplexing: handle v5 plugin compiled with new sdk
* add mux supported flag and increase test coverage
* set multiplexingSupport field in plugin server
* remove multiplexingSupport field in sdk
* revert postgres to non-multiplexed
* add comments on grpc server fields
* use pointer receiver on grpc server methods
* add changelog
* use pointer for grpcserver instance
* Use a gRPC server to determine if a plugin should be multiplexed
* Apply suggestions from code review
Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
* add lock to removePluginClient
* add multiplexingSupport field to externalPlugin struct
* do not send nil to grpc MultiplexingSupport
* check err before logging
* handle locking scenario for cleanupFunc
* allow ServeConfigMultiplex to dispense v5 plugin
* reposition structs, add err check and comments
* add comment on locking for cleanupExternalPlugin
Co-authored-by: Brian Kassouf <bkassouf@hashicorp.com>
Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
* core: set namespace from the sysview's mount entry on GeneratePasswordFromPolicy
* test: update TestDynamicSystemView to be ns-aware, update tests
* add changelog entry
* port of ldap fix for early cred rotation
* some more porting
* another couple lines to port
* final commits before report
* remove deadlock
* needs testing
* updates
* Sync with OpenLDAP PR
* Update the update error handling for items not found in the queue
* WIP unit tests
* Need to configure DB mount correctly, with db type mockv5
* Need to find a way to inject errors into that mock db
* throw error on role creation failure
* do not swallow error on role creation
* comment out wip tests and add in a test for disallowed role
* Use newly generated password in WAL
Co-authored-by: Michael Golowka <72365+pcman312@users.noreply.github.com>
* return err on popFromRotationQueueByKey error; cleanup on setStaticAccount
* test: fix TestPlugin_lifecycle
* Uncomment and fix unit tests
* Use mock database plugin to inject errors
* Tidy test code to rely less on code internals where possible
* Some stronger test assertions
* Undo logging updates
* Add changelog
* Remove ticker and background threads from WAL tests
* Keep pre-existing API behaviour of allowing update static role to act as a create
* Switch test back to update operation
* Revert my revert, and fix some test bugs
* Fix TestBackend_StaticRole_LockRegression
* clean up defer on TestPlugin_lifecycle
* unwrap reqs on cleanup
* setStaticAccount: don't hold a write lock
* TestStoredWALsCorrectlyProcessed: set replication state to unknown
Co-authored-by: Tom Proctor <tomhjp@users.noreply.github.com>
Co-authored-by: Michael Golowka <72365+pcman312@users.noreply.github.com>
Co-authored-by: Calvin Leung Huang <1883212+calvn@users.noreply.github.com>
