* Return the proper serial number in OCSP verification errors
- We returned the issuer's certificate number instead of the serial
number of the actual certificate we validated from an OCSP request.
- The problematic serial number within the error are never shown
currently in Vault. The only user of this library is cert-auth
which swallows errors around revoked certificates and returns
a boolean false instead of the actual error message.
* Add cl
* Use previously formatted serial in error msg
The previous logic would consider not normalize casing before comparing
the policy names which meant that a token associated to a policy with
an uppercase could not be renewed for the following auth methods:
- AppID
- Cert
- GitHub
- LDAP
- Okta
- Radius
- Userpass
Co-authored-by: Violet Hynes <violet.hynes@hashicorp.com>
* Do not generate HMAC keys for CMAC keys on calls to Upgrade
- Missed during the initial development of the Transit CMAC feature,
on initial key creation we did not generate HMAC keys when the key
type was CMAC. The call to the policy's Upgrade function though
would treat this key as requiring an upgrade and add one back.
- Fix this by adding an HMACSupported argument and verifying
on upgrade for HMAC creation that the key type supports HMAC
- Add generic test that verifies we aren't changing a key type iota
value, string it defined and the proper operation of HMACSupported
and CMACSupported functions
* Add warning to test variable
* add key types for cmac for transit key creation
* add test for key creation
* fix test logic and add cases
* fix logic for hmac
* add go doc
* fix key size and add check for HMAC key
We have many hand-written String() methods (and similar) for enums.
These require more maintenance and are more error-prone than using
automatically generated methods. In addition, the auto-generated
versions can be more efficient.
Here, we switch to using https://github.com/loggerhead/enumer, itself
a fork of https://github.com/diegostamigni/enumer, no longer maintained,
and a fork of the mostly standard tool
https://pkg.go.dev/golang.org/x/tools/cmd/stringer.
We use this fork of enumer for Go 1.20+ compatibility and because
we require the `-transform` flag to be able to generate
constants that match our current code base.
Some enums were not targeted for this change:
* auth/ldap: fix login errors
This fixes 2 ldap auth login errors
* Missing entity alias attribute value
* Vault relies on case insensitive user attribute keys for mapping user
attributes to entity alias metadata. This sets the appropriate
configs in the cap library.
* ldap group search anonymous bind regression
* Anonymous group searches can be rejected by some LDAP servers if
they contain a userDN. This sets the configs in the cap library to
specify unauthenticated binds for anonymous group searches should
exclude a DN.
Closes https://github.com/hashicorp/vault/issues/26171
Closes https://github.com/hashicorp/vault/issues/26183
* changelog
* go mod tidy
* go get cap/ldap@latest and go mod tidy
* Validate OCSP response is signed by expected issuer and serial number matches request
- There was a bug in the OCSP response signature logic, it properly
verified but kept around the ocspRes object around so we ignored
the errors found and passed the response object back up the stack.
- Now extract the verification logic into a dedicated function, if
it returns an error, blank the ocspRes response as we can't trust it.
- Address an issue that the OCSP requests from multiple servers were
clobbering each others responses as the index loop variable was not
properly captured.
- Add a missing validation that the response was for the serial number
we requested
* Add cl
* Support OCSP responses without a NextUpdate value set
- Validate that the ThisUpdate value is
properly prior to our current time and
if NextUpdate is set that, ThisUpdate is
before NextUpdate.
- If we don't have a value for NextUpdate just compare against ThisUpdate.
* Add ocsp_this_update_max_ttl support to cert auth
- Allow configuring a maximum TTL of the OCSP response based on the
ThisUpdate time like OpenSSL does
- Add test to validate that we don't cache OCSP responses with no NextUpdate
* Add cl
* Add missing ` in docs
* Rename ocsp_this_update_max_ttl to ocsp_this_update_max_age
* Missed a few TTL references
* Fix error message
* Address OCSP client caching issue
- The OCSP cache built into the client that is used by cert-auth
would cache the responses but when pulling out a cached value the
response wasn't validating properly and was then thrown away.
- The issue was around a confusion of the client's internal status
vs the Go SDK OCSP status integer values.
- Add a test that validates the cache is now used
* Add cl
* Fix PKI test failing now due to the OCSP cache working
- Remove the previous lookup before revocation as now the OCSP
cache works so we don't see the new revocation as we are actually
leveraging the cache
* add gosimport to make fmt and run it
* move installation to tools.sh
* correct weird spacing issue
* Update Makefile
Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com>
* fix a weird issue
---------
Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com>
Add WaitForMatchingMerkleRootsClients and Clients to sdk testcluster. Fix internal TestCluster.SetRootToken, which wasn't updating the builtin clients' token.
Initial version of an internal plugin interface for event subscription plugins,
and an AWS SQS plugin as an example.
Co-authored-by: Tom Proctor <tomhjp@users.noreply.github.com>
* Make plugin-specific env take precedence over sys env
* Expand the existing plugin env integration test
---------
Co-authored-by: Austin Gebauer <34121980+austingebauer@users.noreply.github.com>
* Base Binary Cert and CSR Parse functions.
* Add otherSANS parsing.
* Notate what doesn't exist on a CSR.
* Fix otherSans call err-checking and add basic-constriants to CSR
* Move BasicConstraint parsing to be optionally set.
* Refactored to use existing ParseBasicConstraintsExtension.
* Add handling for the ChangeSubjectName ext on CSR that is needed for EST
* Remove ChangeSubjectName - it's an attribute, not an extension, and there is no clean way to parse it, so pair down for now.
* Make these public methods, so they can be used in vault.
* Add unit tests for certutil.ParseCertificateToCreationParameters.
Also add unit tests for certutil.ParseCertificateToFields.
* Cleanup TestParseCertificate.
* Add unit tests for certutil.ParseCsrToCreationParameters and ParseCsrToFields.
* Fix return values for "add_basic_constraints" in certutil.ParseCsrToFields.
Add a test for parsing CSRs where "add_basic_constraints" is false.
* Clear up some todos.
* Add a test for certutil.ParseCertificateToCreationParameters for non-CA cert.
* Tweak TestParseCertificate/full_non_CA_cert.
* Basics of three remaining fields - keyUsage; extKeyUsage; PolicyIdentifiers
* Fix tests and err handling
* Add unit tests for policy_identifiers; ext_key_usage_oids; key_usage
* Add test on ext_key_usage_oids
* Remove duplicate usages elsewhere.
* Add error handling to csr-checks.
* Remove extranames on returned types.
* Remove useless function.
---------
Co-authored-by: Victor Rodriguez <vrizo@hashicorp.com>
* add new plugin wif fields to AWS Secrets Engine
* add changelog
* go get awsutil v0.3.0
* fix up changelog
* fix test and field parsing helper
* godoc on new test
* require role arn when audience set
* make fmt
---------
Co-authored-by: Austin Gebauer <agebauer@hashicorp.com>
Co-authored-by: Austin Gebauer <34121980+austingebauer@users.noreply.github.com>
Adds the ability to pin a version for a specific plugin type + name to enable an easier plugin upgrade UX. After pinning and reloading, that version should be the only version in use.
No HTTP API implementation yet for managing pins, so no user-facing effects yet.
* Implement raft-wal
* go mod tidy
* add metrics, fix a panic
* fix the panic for real this time
* PR feedback
* refactor tests to use a helper and reduce duplication
* add a test to verify we don't use raft-wal if raft.db exists
* add config to enable the verifier
* add tests for parsing verification intervals
* run the verifier in the background
* wire up the verifier
* go mod tidy
* refactor config parsing
* remove unused function
* trying to get the verifier working
* wire up some more verifier bits
* sorted out an error, added a new test, lots of debug logging that needs to come out
* fix a bug and remove all the debugging statements
* make sure we close raft-wal stablestore too
* run verifier tests for both boltdb and raft-wal
* PR feedback
* Vault 20270 docker test raft wal (#24463)
* adding a migration test from boltdb to raftwal and back
adding a migration test using snapshot restore
* feedback
* Update physical/raft/raft.go
Co-authored-by: Paul Banks <pbanks@hashicorp.com>
* PR feedback
* change verifier function
* make this shorter
* add changelog
* Fix Close behavior
* make supporting empty logs more explicit
* add some godocs
---------
Co-authored-by: hamid ghaf <hamid@hashicorp.com>
Co-authored-by: Hamid Ghaf <83242695+hghaf099@users.noreply.github.com>
Co-authored-by: Paul Banks <pbanks@hashicorp.com>
* Migration of OtherSANs Parsing Call to SDK helper from pki-issuer
* Based on PR feedback from Steve, remove internal variable, reference certutil directly.
We're on a quest to reduce our pipeline execution time to both enhance
our developer productivity but also to reduce the overall cost of the CI
pipeline. The strategy we use here reduces workflow execution time and
network I/O cost by reducing our module cache size and using binary
external tools when possible. We no longer download modules and build
many of the external tools thousands of times a day.
Our previous process of installing internal and external developer tools
was scattered and inconsistent. Some tools were installed via `go
generate -tags tools ./tools/...`,
others via various `make` targets, and some only in Github Actions
workflows. This process led to some undesirable side effects:
* The modules of some dev and test tools were included with those
of the Vault project. This leads to us having to manage our own
Go modules with those of external tools. Prior to Go 1.16 this
was the recommended way to handle external tools, but now
`go install tool@version` is the recommended way to handle
external tools that need to be build from source as it supports
specific versions but does not modify the go.mod.
* Due to Github cache constraints we combine our build and test Go
module caches together, but having our developer tools as deps in
our module results in a larger cache which is downloaded on every
build and test workflow runner. Removing the external tools that were
included in our go.mod reduced the expanded module cache by size
by ~300MB, thus saving time and network I/O costs when downloading
the module cache.
* Not all of our developer tools were included in our modules. Some were
being installed with `go install` or `go run`, so they didn't take
advantage of a single module cache. This resulted in us downloading
Go modules on every CI and Build runner in order to build our
external tools.
* Building our developer tools from source in CI is slow. Where possible
we can prefer to use pre-built binaries in CI workflows. No more
module download or tool compiles if we can avoid them.
I've refactored how we define internal and external build tools
in our Makefile and added several new targets to handle both building
the developer tools locally for development and verifying that they are
available. This allows for an easy developer bootstrap while also
supporting installation of many of the external developer tools from
pre-build binaries in CI. This reduces our network IO and run time
across nearly all of our actions runners.
While working on this I caught and resolved a few unrelated issue:
* Both our Go and Proto format checks we're being run incorrectly. In
CI they we're writing changes but not failing if changes were
detected. The Go was less of a problem as we have git hooks that
are intended to enforce formatting, however we drifted over time.
* Our Git hooks couldn't handle removing a Go file without failing. I
moved the diff check into the new Go helper and updated it to handle
removing files.
* I combined a few separate scripts and into helpers and added a few
new capabilities.
* I refactored how we install Go modules to make it easier to download
and tidy all of the projects go.mod's.
* Refactor our internal and external tool installation and verification
into a tools.sh helper.
* Combined more complex Go verification into `scripts/go-helper.sh` and
utilize it in the `Makefile` and git commit hooks.
* Add `Makefile` targets for executing our various tools.sh helpers.
* Update our existing `make` targets to use new tool targets.
* Normalize our various scripts and targets output to have a consistent
output format.
* In CI, install many of our external dependencies as binaries wherever
possible. When not possible we'll build them from scratch but not mess
with the shared module cache.
* [QT-641] Remove our external build tools from our project Go modules.
* [QT-641] Remove extraneous `go list`'s from our `set-up-to` composite
action.
* Fix formatting and regen our protos
Signed-off-by: Ryan Cragun <me@ryan.ec>
I have an upcoming PR for event notifications that needs similar
exponential backoff logic, and I prefer the API and logic in the
auto-auth exponential backoff rather than that of
github.com/cenkalti/backoff/v3.
This does have a small behavior change: the auto-auth min backoff
will now be randomly reduced by up to 25% on the first call. This is
a desirable property to avoid thundering herd problems, where a bunch
of agents won't all try have the same retry timeout.
