* auth/ldap: fix login errors
This fixes 2 ldap auth login errors
* Missing entity alias attribute value
* Vault relies on case insensitive user attribute keys for mapping user
attributes to entity alias metadata. This sets the appropriate
configs in the cap library.
* ldap group search anonymous bind regression
* Anonymous group searches can be rejected by some LDAP servers if
they contain a userDN. This sets the configs in the cap library to
specify unauthenticated binds for anonymous group searches should
exclude a DN.
Closes https://github.com/hashicorp/vault/issues/26171
Closes https://github.com/hashicorp/vault/issues/26183
* changelog
* go mod tidy
* go get cap/ldap@latest and go mod tidy
* Validate OCSP response is signed by expected issuer and serial number matches request
- There was a bug in the OCSP response signature logic, it properly
verified but kept around the ocspRes object around so we ignored
the errors found and passed the response object back up the stack.
- Now extract the verification logic into a dedicated function, if
it returns an error, blank the ocspRes response as we can't trust it.
- Address an issue that the OCSP requests from multiple servers were
clobbering each others responses as the index loop variable was not
properly captured.
- Add a missing validation that the response was for the serial number
we requested
* Add cl
* Support OCSP responses without a NextUpdate value set
- Validate that the ThisUpdate value is
properly prior to our current time and
if NextUpdate is set that, ThisUpdate is
before NextUpdate.
- If we don't have a value for NextUpdate just compare against ThisUpdate.
* Add ocsp_this_update_max_ttl support to cert auth
- Allow configuring a maximum TTL of the OCSP response based on the
ThisUpdate time like OpenSSL does
- Add test to validate that we don't cache OCSP responses with no NextUpdate
* Add cl
* Add missing ` in docs
* Rename ocsp_this_update_max_ttl to ocsp_this_update_max_age
* Missed a few TTL references
* Fix error message
* Address OCSP client caching issue
- The OCSP cache built into the client that is used by cert-auth
would cache the responses but when pulling out a cached value the
response wasn't validating properly and was then thrown away.
- The issue was around a confusion of the client's internal status
vs the Go SDK OCSP status integer values.
- Add a test that validates the cache is now used
* Add cl
* Fix PKI test failing now due to the OCSP cache working
- Remove the previous lookup before revocation as now the OCSP
cache works so we don't see the new revocation as we are actually
leveraging the cache
* add gosimport to make fmt and run it
* move installation to tools.sh
* correct weird spacing issue
* Update Makefile
Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com>
* fix a weird issue
---------
Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com>
Add WaitForMatchingMerkleRootsClients and Clients to sdk testcluster. Fix internal TestCluster.SetRootToken, which wasn't updating the builtin clients' token.
Initial version of an internal plugin interface for event subscription plugins,
and an AWS SQS plugin as an example.
Co-authored-by: Tom Proctor <tomhjp@users.noreply.github.com>
* Make plugin-specific env take precedence over sys env
* Expand the existing plugin env integration test
---------
Co-authored-by: Austin Gebauer <34121980+austingebauer@users.noreply.github.com>
* Base Binary Cert and CSR Parse functions.
* Add otherSANS parsing.
* Notate what doesn't exist on a CSR.
* Fix otherSans call err-checking and add basic-constriants to CSR
* Move BasicConstraint parsing to be optionally set.
* Refactored to use existing ParseBasicConstraintsExtension.
* Add handling for the ChangeSubjectName ext on CSR that is needed for EST
* Remove ChangeSubjectName - it's an attribute, not an extension, and there is no clean way to parse it, so pair down for now.
* Make these public methods, so they can be used in vault.
* Add unit tests for certutil.ParseCertificateToCreationParameters.
Also add unit tests for certutil.ParseCertificateToFields.
* Cleanup TestParseCertificate.
* Add unit tests for certutil.ParseCsrToCreationParameters and ParseCsrToFields.
* Fix return values for "add_basic_constraints" in certutil.ParseCsrToFields.
Add a test for parsing CSRs where "add_basic_constraints" is false.
* Clear up some todos.
* Add a test for certutil.ParseCertificateToCreationParameters for non-CA cert.
* Tweak TestParseCertificate/full_non_CA_cert.
* Basics of three remaining fields - keyUsage; extKeyUsage; PolicyIdentifiers
* Fix tests and err handling
* Add unit tests for policy_identifiers; ext_key_usage_oids; key_usage
* Add test on ext_key_usage_oids
* Remove duplicate usages elsewhere.
* Add error handling to csr-checks.
* Remove extranames on returned types.
* Remove useless function.
---------
Co-authored-by: Victor Rodriguez <vrizo@hashicorp.com>
* add new plugin wif fields to AWS Secrets Engine
* add changelog
* go get awsutil v0.3.0
* fix up changelog
* fix test and field parsing helper
* godoc on new test
* require role arn when audience set
* make fmt
---------
Co-authored-by: Austin Gebauer <agebauer@hashicorp.com>
Co-authored-by: Austin Gebauer <34121980+austingebauer@users.noreply.github.com>
Adds the ability to pin a version for a specific plugin type + name to enable an easier plugin upgrade UX. After pinning and reloading, that version should be the only version in use.
No HTTP API implementation yet for managing pins, so no user-facing effects yet.
* Implement raft-wal
* go mod tidy
* add metrics, fix a panic
* fix the panic for real this time
* PR feedback
* refactor tests to use a helper and reduce duplication
* add a test to verify we don't use raft-wal if raft.db exists
* add config to enable the verifier
* add tests for parsing verification intervals
* run the verifier in the background
* wire up the verifier
* go mod tidy
* refactor config parsing
* remove unused function
* trying to get the verifier working
* wire up some more verifier bits
* sorted out an error, added a new test, lots of debug logging that needs to come out
* fix a bug and remove all the debugging statements
* make sure we close raft-wal stablestore too
* run verifier tests for both boltdb and raft-wal
* PR feedback
* Vault 20270 docker test raft wal (#24463)
* adding a migration test from boltdb to raftwal and back
adding a migration test using snapshot restore
* feedback
* Update physical/raft/raft.go
Co-authored-by: Paul Banks <pbanks@hashicorp.com>
* PR feedback
* change verifier function
* make this shorter
* add changelog
* Fix Close behavior
* make supporting empty logs more explicit
* add some godocs
---------
Co-authored-by: hamid ghaf <hamid@hashicorp.com>
Co-authored-by: Hamid Ghaf <83242695+hghaf099@users.noreply.github.com>
Co-authored-by: Paul Banks <pbanks@hashicorp.com>
* Migration of OtherSANs Parsing Call to SDK helper from pki-issuer
* Based on PR feedback from Steve, remove internal variable, reference certutil directly.
We're on a quest to reduce our pipeline execution time to both enhance
our developer productivity but also to reduce the overall cost of the CI
pipeline. The strategy we use here reduces workflow execution time and
network I/O cost by reducing our module cache size and using binary
external tools when possible. We no longer download modules and build
many of the external tools thousands of times a day.
Our previous process of installing internal and external developer tools
was scattered and inconsistent. Some tools were installed via `go
generate -tags tools ./tools/...`,
others via various `make` targets, and some only in Github Actions
workflows. This process led to some undesirable side effects:
* The modules of some dev and test tools were included with those
of the Vault project. This leads to us having to manage our own
Go modules with those of external tools. Prior to Go 1.16 this
was the recommended way to handle external tools, but now
`go install tool@version` is the recommended way to handle
external tools that need to be build from source as it supports
specific versions but does not modify the go.mod.
* Due to Github cache constraints we combine our build and test Go
module caches together, but having our developer tools as deps in
our module results in a larger cache which is downloaded on every
build and test workflow runner. Removing the external tools that were
included in our go.mod reduced the expanded module cache by size
by ~300MB, thus saving time and network I/O costs when downloading
the module cache.
* Not all of our developer tools were included in our modules. Some were
being installed with `go install` or `go run`, so they didn't take
advantage of a single module cache. This resulted in us downloading
Go modules on every CI and Build runner in order to build our
external tools.
* Building our developer tools from source in CI is slow. Where possible
we can prefer to use pre-built binaries in CI workflows. No more
module download or tool compiles if we can avoid them.
I've refactored how we define internal and external build tools
in our Makefile and added several new targets to handle both building
the developer tools locally for development and verifying that they are
available. This allows for an easy developer bootstrap while also
supporting installation of many of the external developer tools from
pre-build binaries in CI. This reduces our network IO and run time
across nearly all of our actions runners.
While working on this I caught and resolved a few unrelated issue:
* Both our Go and Proto format checks we're being run incorrectly. In
CI they we're writing changes but not failing if changes were
detected. The Go was less of a problem as we have git hooks that
are intended to enforce formatting, however we drifted over time.
* Our Git hooks couldn't handle removing a Go file without failing. I
moved the diff check into the new Go helper and updated it to handle
removing files.
* I combined a few separate scripts and into helpers and added a few
new capabilities.
* I refactored how we install Go modules to make it easier to download
and tidy all of the projects go.mod's.
* Refactor our internal and external tool installation and verification
into a tools.sh helper.
* Combined more complex Go verification into `scripts/go-helper.sh` and
utilize it in the `Makefile` and git commit hooks.
* Add `Makefile` targets for executing our various tools.sh helpers.
* Update our existing `make` targets to use new tool targets.
* Normalize our various scripts and targets output to have a consistent
output format.
* In CI, install many of our external dependencies as binaries wherever
possible. When not possible we'll build them from scratch but not mess
with the shared module cache.
* [QT-641] Remove our external build tools from our project Go modules.
* [QT-641] Remove extraneous `go list`'s from our `set-up-to` composite
action.
* Fix formatting and regen our protos
Signed-off-by: Ryan Cragun <me@ryan.ec>
I have an upcoming PR for event notifications that needs similar
exponential backoff logic, and I prefer the API and logic in the
auto-auth exponential backoff rather than that of
github.com/cenkalti/backoff/v3.
This does have a small behavior change: the auto-auth min backoff
will now be randomly reduced by up to 25% on the first call. This is
a desirable property to avoid thundering herd problems, where a bunch
of agents won't all try have the same retry timeout.
* PKI refactoring to start breaking apart monolith into sub-packages
- This was broken down by commit within enterprise for ease of review
but would be too difficult to bring back individual commits back
to the CE repository. (they would be squashed anyways)
- This change was created by exporting a patch of the enterprise PR
and applying it to CE repository
* Fix TestBackend_OID_SANs to not be rely on map ordering
* Pulls in github.com/go-secure-stdlib/plugincontainer@v0.3.0 which exposes a new `Config.Rootless` option to opt in to extra container configuration options that allow establishing communication with a non-root plugin within a rootless container runtime.
* Adds a new "rootless" option for plugin runtimes, so Vault needs to be explicitly told whether the container runtime on the machine is rootless or not. It defaults to false as rootless installs are not the default.
* Updates `run_config.go` to use the new option when the plugin runtime is rootless.
* Adds new `-rootless` flag to `vault plugin runtime register`, and `rootless` API option to the register API.
* Adds rootless Docker installation to CI to support tests for the new functionality.
* Minor test refactor to minimise the number of test Vault cores that need to be made for the external plugin container tests.
* Documentation for the new rootless configuration and the new (reduced) set of restrictions for plugin containers.
* As well as adding rootless support, we've decided to drop explicit support for podman for now, but there's no barrier other than support burden to adding it back again in future so it will depend on demand.
* wip
* Work on the tuneable allowance and some bugs
* Call handleCancellableRequest instead, which gets the audit order more correct and includes the preauth response
* Get rid of no longer needed operation
* Phew, this wasn't necessary
* Add auth error handling by the backend, and fix a bug with handleInvalidCredentials
* Cleanup req/resp naming
* Use the new form, and data
* Discovered that tokens werent really being checked because isLoginRequest returns true for the re-request into the backend, when it shouldnt
* Add a few more checks in the delegated request handler for bad inputs
- Protect the delegated handler from bad inputs from the backend such
as an empty accessor, a path that isn't registered as a login request
- Add similar protections for bad auth results as we do in the normal
login request paths. Technically not 100% needed but if somehow the
handleCancelableRequest doesn't use the handleLoginRequest code path
we could get into trouble in the future
- Add delegated-auth-accessors flag to the secrets tune command and
api-docs
* Unit tests and some small fixes
* Remove transit preauth test, rely on unit tests
* Cleanup and add a little more commentary in tests
* Fix typos, add another failure use-case which we reference a disabled auth mount
* PR Feedback
- Use router to lookup mount instead of defining a new lookup method
- Enforce auth table types and namespace when mount is found
- Define a type alias for the handleInvalidCreds
- Fix typos/grammar
- Clean up globals in test
* Additional PR feedback
- Add test for delegated auth handler
- Force batch token usage
- Add a test to validate failures if a non-batch token is used
- Check for Data member being nil in test cases
* Update failure error message around requiring batch tokens
* Trap MFA requests
* Reword some error messages
* Add test and fixes for delegated response wrapping
* Move MFA test to dedicated mount
- If the delegated auth tests were running in parallel, the MFA test
case might influence the other tests, so move the MFA to a dedicated
mount
* PR feedback: use textproto.CanonicalMIMEHeaderKey
- Change the X-Vault-Wrap-Ttl constant to X-Vault-Wrap-TTL
and use textproto.CanonicalMIMEHeaderKey to format it
within the delete call.
- This protects the code around changes of the constant typing
* PR feedback
- Append Error to RequestDelegatedAuth
- Force error interface impl through explicit nil var assignment on
RequestDelegatedAuthError
- Clean up test factory and leverage NewTestSoloCluster
- Leverage newer maps.Clone as this is 1.16 only
---------
Co-authored-by: Scott G. Miller <smiller@hashicorp.com>
* Handle expired OCSP responses from server
- If a server replied with what we considered an expired OCSP response (nextUpdate is now or in the past), and it was our only response we would panic due to missing error handling logic.
* Add cl
* Add test to demonstrate a split-brain active node when using Consul
* Add Consul session check to prevent split-brain updates
* It's not right
Co-authored-by: Josh Black <raskchanky@gmail.com>
---------
Co-authored-by: Josh Black <raskchanky@gmail.com>
* Ignore nonces when encrypting without convergence or with convergence versions > 1
* Honor nonce use warning in non-FIPS modes
* Revert "Honor nonce use warning in non-FIPS modes"
This reverts commit 2aee3dbdc11c4e333ecb20503539c7993b24ee57.
* Add a test func that removes a nonce when not needed
* err out rather than ignore the nonce
* Alter unit test to cover, also cover convergent version 3
* More unit test work
* Fix test 14
* changelog
* tests not already in a nonce present path
* Update unit test to not assume warning when nonce provided incorrectly
* remove unused test field
* Fix auto-squash events experiments
When #22835 was merged, it was auto-squashed, so the `experiments`
import was removed, but the test still referenced it.
This removes the (now unnecessary) experiment from the test.
* Allow nonces for managed keys, because we have no way of knowing if the backing cipher/mode needs one
---------
Co-authored-by: Christopher Swenson <christopher.swenson@hashicorp.com>
* Also makes plugin directory optional when registering container plugins
* And threads plugin runtime settings through to plugin execution config
* Add runsc to github runner for plugin container tests
