* Use DRBG based RSA key generation everywhere
* switch to the conditional generator
* Use DRBG based RSA key generation everywhere
* switch to the conditional generator
* Add an ENV var to disable the DRBG in a pinch
* update go.mod
* Use DRBG based RSA key generation everywhere
* switch to the conditional generator
* Add an ENV var to disable the DRBG in a pinch
* Use DRBG based RSA key generation everywhere
* update go.mod
* fix import
* Remove rsa2 alias, remove test code
* move cryptoutil/rsa.go to sdk
* move imports too
* remove makefile change
* rsa2->rsa
* more rsa2->rsa, remove test code
* fix some overzelous search/replace
* Update to a real tag
* changelog
* copyright
* work around copyright check
* work around copyright check pt2
* bunch of dupe imports
* missing import
* wrong license
* fix go.mod conflict
* missed a spot
* dupe import
* skip connection verification on config read
* ensure appropriate default on config update call that results in a creation
* changelog
* leave verify_connection in config read response
* update test to handle output of verify_connection parameter
* fix remaining tests
* add inline cert auth to postres db plugin
* handle both sslinline and new TLS plugin fields
* refactor PrepareTestContainerWithSSL
* add tests for postgres inline TLS fields
* changelog
* revert back to errwrap since the middleware sanitizing depends on it
* enable only setting sslrootcert
* adds sslinline option to postgres conn string
* for database secrets type postgres, inspects the connection string for sslinline and generates a tlsconfig from the connection string.
* support fallback hosts
* remove broken multihost test
* bootstrap container with cert material
* overwrite pg config and set key file perms
* add feature flag check
* add tests
* add license and comments
* test all ssl modes
* add test cases for dsn (key/value) connection strings
* add fallback test cases
* fix error formatting
* add test for multi-host when using pgx native conn url parsing
---------
Co-authored-by: Branden Horiuchi <Branden.Horiuchi@blackline.com>
When creating database connections, there is a race
condition when multiple goroutines try to create the
connection at the same time. This happens, for
example, on leadership changes in a cluster.
Normally, the extra database connections are cleaned
up when this is detected. However, some database
implementations, notably Postgres, do not seem to
clean up in a timely manner, and can leak in these
scenarios.
To fix this, we create a global lock when creating
database connections to prevent multiple connections
from being created at the same time.
We also clean up the logic at the end so that
if (somehow) we ended up creating an additional
connection, we use the existing one rather than
the new one. This by itself would solve our
problem long-term, however, would still involve
many transient database connections being created
and immediately killed on leadership changes.
It's not ideal to have a single global lock for
database connection creation. Some potential
alternatives:
* a map of locks from the connection name to the lock.
The biggest downside is the we probably will want to
garbage collect this map so that we don't have an
unbounded number of locks.
* a small pool of locks, where we hash the connection
names to pick the lock. Using such a pool generally
is a good way to introduce deadlock, but since we
will only use it in a specific case, and the purpose
is to improve performance for concurrent connection
creation, this is probably acceptable.
Co-authored-by: Jason O'Donnell <2160810+jasonodonnell@users.noreply.github.com>
* add gosimport to make fmt and run it
* move installation to tools.sh
* correct weird spacing issue
* Update Makefile
Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com>
* fix a weird issue
---------
Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com>
Adds the ability to pin a version for a specific plugin type + name to enable an easier plugin upgrade UX. After pinning and reloading, that version should be the only version in use.
No HTTP API implementation yet for managing pins, so no user-facing effects yet.
* Support reloading database plugins across multiple mounts
* Add clarifying comment to MountEntry.Path field
* Tests: Replace non-parallelisable t.Setenv with plugin env settings
* Refactor plugin catalog into its own package
* Fix some unnecessarily slow tests due to accidentally running multiple plugin processes
* Clean up MakeTestPluginDir helper
* Move getBackendVersion tests to plugin catalog package
* Use corehelpers.MakeTestPlugin consistently
* Fix semgrep failure: check for nil value from logical.Storage
* secrets/db: add rotation error path test
We add a test to verify that failed rotations can successfully recover
and that they do not occur outside of a rotation window. Additionally,
we remove registering some external plugins in getCluster() that shaves
off about 5 minutes the database package tests.
* remove dead code and add test comment
* revert to original container helper after refactor
* add rotation_schedule field to db backend
* add cron schedule field
* use priority queue with scheduled rotation types
* allow marshalling of cron schedule type
* return warning on use of mutually exclusive fields
* handle mutual exclusion of rotation fields (#22306)
* handle mutual exclusion of rotation fields
* fix import
* adv ttl mgmt: add rotation_window field (#22303)
* adv ttl mgmt: add rotation_window field
* do some rotation_window validation and add unit tests
* adv ttl mgmt: Ensure initialization sets appropriate rotation schedule (#22341)
* general cleanup and refactor rotation type checks
* make NextRotationTime account for the rotation type
* add comments
* add unit tests to handle mutual exclusion (#22352)
* add unit tests to handle mutual exclusion
* revert rotation_test.go and add missing test case to path_roles_test.go
* adv ttl mgmt: add tests for init queue (#22376)
* Vault 18908/handle manual rotation (#22389)
* support manual rotation for schedule based roles
* update description and naming
* adv ttl mgmt: consider rotation window (#22448)
* consider rotation window
ensure rotations only occur within a rotation window for schedule-based
rotations
* use helper method to set priority in rotateCredential
* fix bug with priority check
* remove test for now
* add and remove comments
* add unit tests for manual rotation (#22453)
* adv ttl mgmt: add tests for rotation_window
* adv ttl mgmt: refactor window tests (#22472)
* Handle GET static-creds endpoint (#22476)
* update read static-creds endpoint to include correct resp data
* return rotation_window if set
* update
* add changelog
* add unit test for static-creds read endpoint (#22505)
* Add the ability to set seconds in cron schedule for testing purposes
* update test so we don't use global var
* update with suggestions
---------
Co-authored-by: JM Faircloth <jmfaircloth@hashicorp.com>
Co-authored-by: John-Michael Faircloth <fairclothjm@users.noreply.github.com>
* add rotation_schedule field to db backend
* add cron schedule field
* use priority queue with scheduled rotation types
* allow marshalling of cron schedule type
* return warning on use of mutually exclusive fields
* handle mutual exclusion of rotation fields (#22306)
* handle mutual exclusion of rotation fields
* fix import
* adv ttl mgmt: add rotation_window field (#22303)
* adv ttl mgmt: add rotation_window field
* do some rotation_window validation and add unit tests
* adv ttl mgmt: Ensure initialization sets appropriate rotation schedule (#22341)
* general cleanup and refactor rotation type checks
* make NextRotationTime account for the rotation type
* add comments
* add unit tests to handle mutual exclusion (#22352)
* add unit tests to handle mutual exclusion
* revert rotation_test.go and add missing test case to path_roles_test.go
* adv ttl mgmt: add tests for init queue (#22376)
* Vault 18908/handle manual rotation (#22389)
* support manual rotation for schedule based roles
* update description and naming
* adv ttl mgmt: consider rotation window (#22448)
* consider rotation window
ensure rotations only occur within a rotation window for schedule-based
rotations
* use helper method to set priority in rotateCredential
* fix bug with priority check
* remove test for now
* add and remove comments
* add unit tests for manual rotation (#22453)
* adv ttl mgmt: add tests for rotation_window
* adv ttl mgmt: refactor window tests (#22472)
* Handle GET static-creds endpoint (#22476)
* update read static-creds endpoint to include correct resp data
* return rotation_window if set
* update
* add changelog
* add unit test for static-creds read endpoint (#22505)
---------
Co-authored-by: Milena Zlaticanin <60530402+Zlaticanin@users.noreply.github.com>
* Adding explicit MPL license for sub-package.
This directory and its subdirectories (packages) contain files licensed with the MPLv2 `LICENSE` file in this directory and are intentionally licensed separately from the BSL `LICENSE` file at the root of this repository.
* Adding explicit MPL license for sub-package.
This directory and its subdirectories (packages) contain files licensed with the MPLv2 `LICENSE` file in this directory and are intentionally licensed separately from the BSL `LICENSE` file at the root of this repository.
* Updating the license from MPL to Business Source License.
Going forward, this project will be licensed under the Business Source License v1.1. Please see our blog post for more details at https://hashi.co/bsl-blog, FAQ at www.hashicorp.com/licensing-faq, and details of the license at www.hashicorp.com/bsl.
* add missing license headers
* Update copyright file headers to BUS-1.1
* Fix test that expected exact offset on hcl file
---------
Co-authored-by: hashicorp-copywrite[bot] <110428419+hashicorp-copywrite[bot]@users.noreply.github.com>
Co-authored-by: Sarah Thompson <sthompson@hashicorp.com>
Co-authored-by: Brian Kassouf <bkassouf@hashicorp.com>
The upcoming event main plugin will use a very similar pattern
as the database plugin map, so it makes sense to refactor this and move
this map out. It also cleans up the database plugin backend so that
it does not have to keep track of the lock.
Co-authored-by: Tom Proctor <tomhjp@users.noreply.github.com>
* poc: client cert credential type
* go mod tidy
* fix typo
* fix newUserReqToProto
* add changelog
* add newline
* fix changelog
* add test for the client cert generator
* Fix formatting
* unset signing bundle URLs
* set BasicConstraintsValidForNonCA to false
* backdate cert by 30s
* remove empty creation params URLs
* check cert BasicConstraintsValid
* set default key bits in newClientCertificateGenerator
* fix client cert gen test with default values
* Add default for key_type
* fix default key_type
* update test with default key type
* update test
* Update changelog/20425.txt
Co-authored-by: Austin Gebauer <34121980+austingebauer@users.noreply.github.com>
* set default key bits and sig bits
* remove the default for key type ad fix the test
* make fmt + add comments for each exported field
* restart test
---------
Co-authored-by: Austin Gebauer <agebauer@hashicorp.com>
Co-authored-by: Austin Gebauer <34121980+austingebauer@users.noreply.github.com>
* Removes _builtin_ versions from mount storage where it already exists
* Stops new builtin versions being put into storage on mount creation/tuning
* Stops the plugin catalog from returning a builtin plugin that has been overridden, so it more accurately reflects the plugins that are available to actually run
Add plugin version to GRPC interface
Added a version interface in the sdk/logical so that it can be shared between all plugin types, and then wired it up to RunningVersion in the mounts, auth list, and database systems.
I've tested that this works with auth, database, and secrets plugin types, with the following logic to populate RunningVersion:
If a plugin has a PluginVersion() method implemented, then that is used
If not, and the plugin is built into the Vault binary, then the go.mod version is used
Otherwise, the it will be the empty string.
My apologies for the length of this PR.
* Placeholder backend should be external
We use a placeholder backend (previously a framework.Backend) before a
GRPC plugin is lazy-loaded. This makes us later think the plugin is a
builtin plugin.
So we added a `placeholderBackend` type that overrides the
`IsExternal()` method so that later we know that the plugin is external,
and don't give it a default builtin version.
* Support version selection for database plugins
* Don't consider unversioned plugins for version selection algorithm
* Added version to 'plugin not found' error
* Add PluginFactoryVersion function to avoid changing sdk/ API
v6 was released in the last 24h, and our tests fail to connect to the db when v6 is used.
Using v6 needs investigating, but for now I'm pinning to the last known good version.
strings.ReplaceAll(s, old, new) is a wrapper function for
strings.Replace(s, old, new, -1). But strings.ReplaceAll is more
readable and removes the hardcoded -1.
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
Add database plugin metrics around connections
This is a replacement for #15923 that takes into account recent lock
cleanup.
I went ahead and added back in the hanging plugin test, which I meant to
add in #15944 but forgot.
I tested this by spinning up a statsd sink in the tests and verifying I
got a stream of metrics:
```
$ nc -u -l 8125 | grep backend
test.swenson-Q9Q0L72D39.secrets.database.backend.connections.count.pgx.5.:1.000000|g
test.swenson-Q9Q0L72D39.secrets.database.backend.connections.count.pgx.5.:0.000000|g
test.swenson-Q9Q0L72D39.secrets.database.backend.connections.count.pgx.5.:1.000000|g
test.swenson-Q9Q0L72D39.secrets.database.backend.connections.count.pgx.5.:0.000000|g
```
We have to rework the shared gauge code to work without a full
`ClusterMetricSink`, since we don't have access to the core metrics from
within a plugin.
This only reports metrics every 10 minutes by default, but it solves
some problems we would have had with the gauge values becoming stale and
needing to be re-sent.
Co-authored-by: Tom Proctor <tomhjp@users.noreply.github.com>
Cleanup and simplify lock usage in database plugin
Following up from discussions in #15923 and #15933, I wanted to split
out a separate PR that drastically reduced the complexity of the use of
the databaseBackend lock. We no longer need it at all for the
`credRotationQueue`, and we can move it to be solely used in a few,
small connections map management functions.
Co-authored-by: Calvin Leung Huang <1883212+calvn@users.noreply.github.com>
* WIP replacing lib/pq
* change timezome param to be URI format
* add changelog
* add changelog for redshift
* update changelog
* add test for DSN style connection string
* more parseurl and quoteidentify to sdk; include copyright and license
* call dbutil.ParseURL instead, fix import ordering
Co-authored-by: Calvin Leung Huang <1883212+calvn@users.noreply.github.com>
