This also temporarily disables couchbase, elasticsearch, and
mongodbatlas because the `Serve` function needs to change signatures
and those plugins are vendored in from external repos, causing problems
when building.
This is part 1 of 4 for renaming the `newdbplugin` package. This copies the existing package to the new location but keeps the current one in place so we can migrate the existing references over more easily.
* Lower the interval for rotation during tests, to make it more likely
that our five second grace period is sufficient.
* Rewrite to make the rotateCredentials ticker a configurable value.
* 'go mod vendor' for SDK changes.
* Refactor PG container creation.
* Rework rotation tests to use shorter sleeps.
* Refactor rotation tests.
* Add a static role rotation test for MongoDB Atlas.
* fix: rotate root credentials for database plugins using WAL
* test: adds a test for WAL rollback logic
* fix: progress on wal rollback
* docs: updates some comments
* docs: updates some comments
* test: adds additional test coverage for WAL rollback
* chore: remove unneeded log
* style: error handling, imports, signature line wraps
* fix: always close db plugin connection
* Switch mongodb driver to mongo-driver
* Tidy mod
* Make writeConcern private
* Implement review feedback
* Add retry functionality
* Added backoff time
* go mod vendor
* Fix failing test
* goimport
* Implement SetCredentials for MongoDB, adding support for static accounts
* rework SetCredentials to split from CreateUser, and to parse the url for database
* Add integration test for mongodb static account rotation
* check the length of the password results to avoid out-of-bounds
* remove unused method
* use the pre-existing test helper for this. Add parse method to helper
* remove unused command
* remove create/update database user for static accounts
* update tests after create/delete removed
* small cleanups
* update postgresql setcredentials test
* Add priority queue to sdk
* fix issue of storing pointers and now copy
* update to use copy structure
* Remove file, put Item struct def. into other file
* add link
* clean up docs
* refactor internal data structure to hide heap method implementations. Other cleanup after feedback
* rename PushItem and PopItem to just Push/Pop, after encapsulating the heap methods
* updates after feedback
* refactoring/renaming
* guard against pushing a nil item
* minor updates after feedback
* Add SetCredentials, GenerateCredentials gRPC methods to combined database backend gPRC
* Initial Combined database backend implementation of static accounts and automatic rotation
* vendor updates
* initial implementation of static accounts with Combined database backend, starting with PostgreSQL implementation
* add lock and setup of rotation queue
* vendor the queue
* rebase on new method signature of queue
* remove mongo tests for now
* update default role sql
* gofmt after rebase
* cleanup after rebasing to remove checks for ErrNotFound error
* rebase cdcr-priority-queue
* vendor dependencies with 'go mod vendor'
* website database docs for Static Role support
* document the rotate-role API endpoint
* postgres specific static role docs
* use constants for paths
* updates from review
* remove dead code
* combine and clarify error message for older plugins
* Update builtin/logical/database/backend.go
Co-Authored-By: Jim Kalafut <jim@kalafut.net>
* cleanups from feedback
* code and comment cleanups
* move db.RLock higher to protect db.GenerateCredentials call
* Return output with WALID if we failed to delete the WAL
* Update builtin/logical/database/path_creds_create.go
Co-Authored-By: Jim Kalafut <jim@kalafut.net>
* updates after running 'make fmt'
* update after running 'make proto'
* Update builtin/logical/database/path_roles.go
Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>
* Update builtin/logical/database/path_roles.go
Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>
* update comment and remove and rearrange some dead code
* Update website/source/api/secret/databases/index.html.md
Co-Authored-By: Jim Kalafut <jim@kalafut.net>
* cleanups after review
* Update sdk/database/dbplugin/grpc_transport.go
Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>
* code cleanup after feedback
* remove PasswordLastSet; it's not used
* document GenerateCredentials and SetCredentials
* Update builtin/logical/database/path_rotate_credentials.go
Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>
* wrap pop and popbykey in backend methods to protect against nil cred rotation queue
* use strings.HasPrefix instead of direct equality check for path
* Forgot to commit this
* updates after feedback
* re-purpose an outdated test to now check that static and dynamic roles cannot share a name
* check for unique name across dynamic and static roles
* refactor loadStaticWALs to return a map of name/setCredentialsWAL struct to consolidate where we're calling set credentials
* remove commented out code
* refactor to have loadstaticwals filter out wals for roles that no longer exist
* return error if nil input given
* add nil check for input into setStaticAccount
* Update builtin/logical/database/path_roles.go
Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>
* add constant for queue tick time in seconds, used for comparrison in updates
* Update builtin/logical/database/path_roles.go
Co-Authored-By: Jim Kalafut <jim@kalafut.net>
* code cleanup after review
* remove misplaced code comment
* remove commented out code
* create a queue in the Factory method, even if it's never used
* update path_roles to use a common set of fields, with specific overrides for dynamic/static roles by type
* document new method
* move rotation things into a specific file
* rename test file and consolidate some static account tests
* Update builtin/logical/database/path_roles.go
Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>
* Update builtin/logical/database/rotation.go
Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>
* Update builtin/logical/database/rotation.go
Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>
* Update builtin/logical/database/rotation.go
Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>
* Update builtin/logical/database/rotation.go
Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>
* Update builtin/logical/database/rotation.go
Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>
* update code comments, method names, and move more methods into rotation.go
* update comments to be capitalized
* remove the item from the queue before we try to destroy it
* findStaticWAL returns an error
* use lowercase keys when encoding WAL entries
* small cleanups
* remove vestigial static account check
* remove redundant DeleteWAL call in populate queue
* if we error on loading role, push back to queue with 10 second backoff
* poll in initqueue to make sure the backend is setup and can write/delete data
* add revoke_user_on_delete flag to allow users to opt-in to revoking the static database user on delete of the Vault role. Default false
* add code comments on read-only loop
* code comment updates
* re-push if error returned from find static wal
* add locksutil and acquire locks when pop'ing from the queue
* grab exclusive locks for updating static roles
* Add SetCredentials and GenerateCredentials stubs to mockPlugin
* add a switch in initQueue to listen for cancelation
* remove guard on zero time, it should have no affect
* create a new context in Factory to pass on and use for closing the backend queue
* restore master copy of vendor dir
Merge both functions for creating mongodb containers into one.
Add retries to docker container cleanups.
Require $VAULT_ACC be set to enable AWS tests.
* add explanation to the database interface
* wordsmithing
* wrap comments, add comments for Type and Close methods
* will be stored, not with be stored
* update comment on the Type method
The result will still pass gofmtcheck and won't trigger additional
changes if someone isn't using goimports, but it will avoid the
piecemeal imports changes we've been seeing.
* Strip empty strings from database revocation stmts
It's technically valid to give empty strings as statements to run on
most databases. However, in the case of revocation statements, it's not
only generally inadvisable but can lead to lack of revocations when you
expect them. This strips empty strings from the array of revocation
statements.
It also makes two other changes:
* Return statements on read as empty but valid arrays rather than nulls,
so that typing information is inferred (this is more in line with the
rest of Vault these days)
* Changes field data for TypeStringSlice and TypeCommaStringSlice such
that a client-supplied value of `""` doesn't turn into `[]string{""}`
but rather `[]string{}`.
The latter and the explicit revocation statement changes are related,
and defense in depth.
The backwards compatibility logic was preventing updates to role
statements from taking effect. This change removes persistence of
deprecated statement fields.
* Database updates
* Add create/update distinction for connection config
* Add create/update distinction for role config
* Add db name and revocation statements to leases to give revocation a
shot at working if the role has been deleted
Fixes#3544Fixes#4782
* Add create/update info to docs
* prevent returning password in reads of connection config info
* fixing a test
* masking password in connection url on reads
* addressing feedback
* removing extra check
* govet cleanup in token store
* adding general ttl handling to login requests
* consolidating TTL calculation to system view
* deprecate LeaseExtend
* deprecate LeaseExtend
* set the increment to the correct value
* move calculateTTL out of SystemView
* remove unused value
* add back clearing of lease id
* implement core ttl in some backends
* removing increment and issue time from lease options
* adding ttl tests, fixing some compile issue
* adding ttl tests
* fixing some explicit max TTL logic
* fixing up some tests
* removing unneeded test
* off by one errors...
* adding back some logic for bc
* adding period to return on renewal
* tweaking max ttl capping slightly
* use the appropriate precision for ttl calculation
* deprecate proto fields instead of delete
* addressing feedback
* moving TTL handling for backends to core
* mongo is a secret backend not auth
* adding estimated ttl for backends that also manage the expiration time
* set the estimate values before calling the renew request
* moving calculate TTL to framework, revert removal of increment and issue time from logical
* minor edits
* addressing feedback
* address more feedback
* logbridge with hclog and identical output
* Initial search & replace
This compiles, but there is a fair amount of TODO
and commented out code, especially around the
plugin logclient/logserver code.
* strip logbridge
* fix majority of tests
* update logxi aliases
* WIP fixing tests
* more test fixes
* Update test to hclog
* Fix format
* Rename hclog -> log
* WIP making hclog and logxi love each other
* update logger_test.go
* clean up merged comments
* Replace RawLogger interface with a Logger
* Add some logger names
* Replace Trace with Debug
* update builtin logical logging patterns
* Fix build errors
* More log updates
* update log approach in command and builtin
* More log updates
* update helper, http, and logical directories
* Update loggers
* Log updates
* Update logging
* Update logging
* Update logging
* Update logging
* update logging in physical
* prefixing and lowercase
* Update logging
* Move phyisical logging name to server command
* Fix som tests
* address jims feedback so far
* incorporate brians feedback so far
* strip comments
* move vault.go to logging package
* update Debug to Trace
* Update go-plugin deps
* Update logging based on review comments
* Updates from review
* Unvendor logxi
* Remove null_logger.go
* redoing connection handling
* a little more cleanup
* empty implementation of rotation
* updating rotate signature
* signature update
* updating interfaces again :(
* changing back to interface
* adding templated url support and rotation for postgres
* adding correct username
* return updates
* updating statements to be a list
* adding error sanitizing middleware
* fixing log sanitizier
* adding postgres rotate test
* removing conf from rotate
* adding rotate command
* adding mysql rotate
* finishing up the endpoint in the db backend for rotate
* no more structs, just store raw config
* fixing tests
* adding db instance lock
* adding support for statement list in cassandra
* wip redoing interface to support BC
* adding falllback for Initialize implementation
* adding backwards compat for statements
* fix tests
* fix more tests
* fixing up tests, switching to new fields in statements
* fixing more tests
* adding mssql and mysql
* wrapping all the things in middleware, implementing templating for mongodb
* wrapping all db servers with error santizer
* fixing test
* store the name with the db instance
* adding rotate to cassandra
* adding compatibility translation to both server and plugin
* reordering a few things
* store the name with the db instance
* reordering
* adding a few more tests
* switch secret values from slice to map
* addressing some feedback
* reinstate execute plugin after resetting connection
* set database connection to closed
* switching secret values func to map[string]interface for potential future uses
* addressing feedback
* Use version to determine plugin protocol to use
* Remove field from ServeOpts
* Fix missing assignment, handle errors
* contraint -> constraint
* Inject the version string from the vault side
* Fix the version check
* Add grpc support check to database plugins
* Default to use grpc unless missing env var or fail on contraint check
* Add GRPCSupport test
* Add greater than test case
* Add go-version dep
* Add grpc plugins
* Add grpc plugins
* Translate wrap info to/from proto
* Add nil checks
* Fix nil marshaling errors
* Provide logging through the go-plugin logger
* handle errors in the messages
* Update the TLS config so bidirectional connections work
* Add connectivity checks
* Restart plugin and add timeouts where context is not availible
* Add the response wrap data into the grpc system implementation
* Add leaseoptions to pb.Auth
* Add an error translator
* Add tests for translating the proto objects
* Fix rename of function
* Add tracing to plugins for easier debugging
* Handle plugin crashes with the go-plugin context
* Add test for grpcStorage
* Add tests for backend and system
* Bump go-plugin for GRPCBroker
* Remove RegisterLicense
* Add casing translations for new proto messages
* Use doneCtx in grpcClient
* Use doneCtx in grpcClient
* s/shutdown/shut down/
* Start work on passing context to backends
* More work on passing context
* Unindent logical system
* Unindent token store
* Unindent passthrough
* Unindent cubbyhole
* Fix tests
* use requestContext in rollback and expiration managers
* Start work on context aware backends
* Start work on moving the database plugins to gRPC in order to pass context
* Add context to builtin database plugins
* use byte slice instead of string
* Context all the things
* Move proto messages to the dbplugin package
* Add a grpc mechanism for running backend plugins
* Serve the GRPC plugin
* Add backwards compatibility to the database plugins
* Remove backend plugin changes
* Remove backend plugin changes
* Cleanup the transport implementations
* If grpc connection is in an unexpected state restart the plugin
* Fix tests
* Fix tests
* Remove context from the request object, replace it with context.TODO
* Add a test to verify netRPC plugins still work
* Remove unused mapstructure call
* Code review fixes
* Code review fixes
* Code review fixes
* Add backend plugin changes
* Fix totp backend plugin tests
* Fix logical/plugin InvalidateKey test
* Fix plugin catalog CRUD test, fix NoopBackend
* Clean up commented code block
* Fix system backend mount test
* Set plugin_name to omitempty, fix handleMountTable config parsing
* Clean up comments, keep shim connections alive until cleanup
* Include pluginClient, disallow LookupPlugin call from within a plugin
* Add wrapper around backendPluginClient for proper cleanup
* Add logger shim tests
* Add logger, storage, and system shim tests
* Use pointer receivers for system view shim
* Use plugin name if no path is provided on mount
* Enable plugins for auth backends
* Add backend type attribute, move builtin/plugin/package
* Fix merge conflict
* Fix missing plugin name in mount config
* Add integration tests on enabling auth backend plugins
* Remove dependency cycle on mock-plugin
* Add passthrough backend plugin, use logical.BackendType to determine lease generation
* Remove vault package dependency on passthrough package
* Add basic impl test for passthrough plugin
* Incorporate feedback; set b.backend after shims creation on backendPluginServer
* Fix totp plugin test
* Add plugin backends docs
* Fix tests
* Fix builtin/plugin tests
* Remove flatten from PluginRunner fields
* Move mock plugin to logical/plugin, remove totp and passthrough plugins
* Move pluginMap into newPluginClient
* Do not create storage RPC connection on HandleRequest and HandleExistenceCheck
* Change shim logger's Fatal to no-op
* Change BackendType to uint32, match UX backend types
* Change framework.Backend Setup signature
* Add Setup func to logical.Backend interface
* Move OptionallyEnableMlock call into plugin.Serve, update docs and comments
* Remove commented var in plugin package
* RegisterLicense on logical.Backend interface (#3017)
* Add RegisterLicense to logical.Backend interface
* Update RegisterLicense to use callback func on framework.Backend
* Refactor framework.Backend.RegisterLicense
* plugin: Prevent plugin.SystemViewClient.ResponseWrapData from getting JWTs
* plugin: Revert BackendType to remove TypePassthrough and related references
* Fix typo in plugin backends docs
