* vault: lazily materialize external tokens
Implement stub-first auth flow to avoid token-store writes on read-only requests, materialize only on lease registration, cache materialized token IDs by fingerprint, and add coverage for standby/leader materialization behavior.
* vault: align lazy JWT materialization with main
Adjust standby materialization behavior to main branch APIs, keep lease-time forwarding conversion, and update enterprise tests to match current standby error paths.
* vault: align jwt lazy materialization follow-ups
Apply naming and test-practice follow-ups, document cache-size rationale, simplify stub-use gating, and add external JWT e2e/benchmark coverage that measures raft writes for passthrough vs leased flows.
* vault: sanitize jwt lazy materialization logs
* vault: move jwt stub audit labels to ent
* vault: keep ent token type in audit logs
Remove audit token_type override and jwt_stub audit labeling so enterprise JWT tokens are consistently reported as ent while preserving existing request handling behavior.
* vault: rename unpersisted JWT token state
Rename IsStub/JwtStub to IsUnpersisted/JwtUnpersisted and switch serialized key to jwt_unpersisted for clearer storage semantics.
* vault: use storage-backed JWT token naming
Rename JWT token persistence marker to IsStorageBacked/JwtStorageBacked and clarify docs that passthrough JWT requests may never write token state to storage.
* vault: address jwt token materialization regressions
Fix storage-backed checks for non-ent tokens, materialize JWTs for lookup endpoints, and add lookup-self regression coverage.
* vault: thread perf standby state into jwt materialization
Pass perf standby state through materializeEnterpriseTokenForLease call sites to avoid unsynchronized state reads in request handling paths.
* test: harden flaky enterprise timing checks
Increase timing tolerance in flaky CI tests without changing runtime behavior.
* test: document CI timing tolerance in flaky tests
Add comments clarifying widened waits are for CI scheduling jitter and asynchronous convergence, not behavior changes.
* vault: remove sensitive token logging
Drop clear-text token value from SSC token MAC mismatch debug logging and keep census test unchanged per request.
* vault: tighten request logging for security scan
Remove remaining error-derived logging fields in request handling token/JWT revoke paths to satisfy clear-text sensitive logging findings.
* vault: use typed no-rpc-client errors
Replace string-based no-rpc-client error matching with a dedicated error type and update perf-standby callsites to return it.
* test: harden perf-standby billing race assertions
Allow early active-node counts up to the number of standby operations before the eventual RPC delivery assertion, to avoid race-only CI flakes without changing behavior.
* revert: undo billing test race assertion changes
Revert the prior billing test hardening commit per user request to keep billing tests unchanged in PR #12909.
* vault: restore fetchCeilingPolicies after rebase
Re-add the enterprise fetchCeilingPolicies method from main that was dropped during rebase conflict resolution, fixing build failure in request_handling.go.
* vault: materialize JWT tokens for cubbyhole requests
Ensure enterprise JWT tokens are materialized for cubbyhole paths in addition to token lookup endpoints, preserving expected persisted token-entry behavior for cubbyhole-backed request flows and token-entry JWT tests.
* vault: restore JWT token headers on perf-standby forward
* tests: fix jwt passthrough profile setup
* vault: clarify forwarding helper docs
* vault: cover jwt standby token endpoint behavior
* vault: remove enterprise JWT wording in comments
* vault: preserve materialized token request identity
* vault: stabilize external JWT readonly raft assertion
* vault: address PR 12909 review feedback
* tests: remove redundant NewTestCluster lifecycle calls
* vault: sanitize request handling token error logs
* vault: remove sensitive error fields in ent token cleanup logs
* vault: restore IsJWT wording
* vault: migrate jwt materialization tests to NewTestCluster
* tests: skip external jwt benchmark in CI
* Run make fmt
* Address PR review feedback
* Remove benchmark CI skip
* Move JWT materialization tests to external suite
---------
Co-authored-by: Bianca <48203644+biazmoreira@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Update config.go
* added validation and parsing
* tests
* move pki external config structs and validation into separate file
* update copywrite
* update configuration
* updates
* Moved tests to pki_external_config.go, comments, refactoring
* refactor
* add tests
* linter fix
* Consolidate to table tests
* consolidate to table tests
* remove APIVersion from PKIExternalCA
* added comments for explaining each struct
* Added ParsePKIExternalCA Test
* Update tests
* Added remaining constraints
* Added destination.template field
* changes
* Added validateListenerAddr
* refactor
* more comments
* changes
* Check for duplicates across blocks
* Make RSA bits a required field
* moved template to the top level
* added comment for test explanation
* move template to the top level
* Move pki config into pkiexternalca directory
* fix linting error
* move pkiconfig back into config folder
* fix failing unit tests
* added comments
* update to preserve order of templatePKIExternalCARefs
* Added comment descriptions for each struct member
* update to include warning
* bring in warning logger from upstream into the pki config parser
* Set default umask to 077
* added comments to each field in agent config
* execute tests in parallel
* combine tests into Validate
* Use assertion error func for tests
* assert error strings
* Removed warning for now
* removed normalization on values during validation
* added tests to ensure that user values are not overridden
* remove testparse
* Update command/agent/config/config.go
* change improvement to feature in changelog
* updated to add line number in error
* Added _ent suffix to files
* Implement CA manager for ACME-based workflows (#12827)
* Implement CA manager for ACME-based workflows
* refactor tests into table tests
* update with suggestions
* format
* fix challenge cleanup
* make fmt
* update with suggestions
* add _ent + build flags
* Add a runtime component for pkiexternalca (#12838)
* Implement CA manager for ACME-based workflows
* Add a runtime component for pkiexternalca
* make fmt
* refactor tests into table tests
* update with suggestions
* format
* fix challenge cleanup
* make fmt
* update with suggestions
* update with suggestions
* add _ent + build flags
* fix linters
* delete duplicate files
* fix changelog
* rename test files
* fix linter
* try to bypass false positive linter err
* fix
* Rename file
* fix linter
* fix linter
* remove go:build enterprise commends from _ent files
* update order statuses to use kebab case + fix scanner failures
* add missing order status
* Template Integration For pki_external_ca resources (#13069)
* Implement CA manager for ACME-based workflows
* Add a runtime component for pkiexternalca
* make fmt
* refactor tests into table tests
* update with suggestions
* initial commit
* fix test failure
* changes
* remove logger check
* remove redundant config by name check
* convert to table tests
* added comments
* updates
* Fix tests
* fix nil pointer issue
* move changes to _ent files
* remove ce duplicate files
* updates
* update template.go
* added changelog.txt
* create template_pem_ent_test.go
* added comment explanation
* update ca_manager_ent.go
* update changelog
* separate ce stubs into server_ce.go and common code into server.go
* Moved helper functions to bottom of test file. Added godocs.
* Make pkiExternalCA name required in template
* remove go:build enterprise commends from _ent files
* rename to template_pem_ent
* include ent tag in server_ent.go
* remove enterprise tag comment from server_ent.go
* create pki_external_config_ce.go
* update template_pem_ent_integration_test.go
* rename integration test
---------
---------
Co-authored-by: Jaired Jawed <jaired.jawed@hashicorp.com>
Co-authored-by: Ben Ash <32777270+benashz@users.noreply.github.com>
Co-authored-by: Zlaticanin <60530402+Zlaticanin@users.noreply.github.com>
Co-authored-by: Milena Zlaticanin <Milena.Zlaticanin@ibm.com>
* no-op commit
* Backport ce: Import PKI External CA plugin
* Rename from pki_external_ca to pki-external-ca
---------
Co-authored-by: Steven Clark <steven.clark@hashicorp.com>
* adding ibm tests for ent files
* adding debug commands
* adding code changes
* adding reload tests
* remove settings.json
* remove ryboe q
* changing isHashicorpLicense to isIBMLicense and moving DiagnoseCheckLicenseGeneration to core_util_common.go
* fix test
* reverting non-license related tests
* reverting non-license related tests
* removing hashicorp license test
* modify reload server_ent_test.go
* change ibm-license paths
* adding census reload server test
* moving LicensingEntitlementSelectionConfig to core_util_common.go
* add EntReloadLicenseAndConfig to stubs
* fix operator diagnose bug
* move bug fix into ce and ent files
* add more ibm test cases
* Update command/command_testonly/server_testonly_ent_test.go
* address comments
* make fmt
---------
Co-authored-by: akshya96 <87045294+akshya96@users.noreply.github.com>
Co-authored-by: Jenny Deng <jenny.deng@hashicorp.com>
* adding ibm tests for ent files
* changing isHashicorpLicense to isIBMLicense and moving DiagnoseCheckLicenseGeneration to core_util_common.go
* reverting non-license related tests
* removing hashicorp license test
Co-authored-by: akshya96 <87045294+akshya96@users.noreply.github.com>
* refactor dependencies and removes disallowed vault imports from builtin Okta auth (#10965)
* move SkipUnlessEnvVarsSet from vault/helper/testhelpers/ to vault/sdk/helper/testhelpers
* use unittest framework from vault-testing-stepwise module in place of sdk/logical
* refactor SkipUnlessEnvVarsSet() and NewAssertAuthPoliciesFunc() to sdk
* bump docker API version to 1.44 matching 2f33549
---------
Co-authored-by: Thy Ton <maithytonn@gmail.com>
* backport: Add override_pinned_version support on tune and enable for secret and auth (#9719)
* fix entWrappedAuthPath() and entWrappedMountsPath() oss stubs
---------
Co-authored-by: Thy Ton <maithytonn@gmail.com>
* license: update headers to IBM Corp.
* `make proto`
* update offset because source file changed
Signed-off-by: Ryan Cragun <me@ryan.ec>
Co-authored-by: Ryan Cragun <me@ryan.ec>
Re-enable some tests that were being skipped due to flakiness. Fix flakiness stemming from not waiting until new license was applied. Fix race in AOP code.
Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com>
* Fix unsetting sys tunable values (on ent).
* Remove commented test, add GoDoc for test.
* Handle empty slices better (PR feedback).
* Fetch Auth endpoint without listing (PR feedback).
* Fatal vs. Error
* Add GetAuth instead of ListAuth
* Fix error format error. Oops!
* One more list->get auth. Remove extra check.
* Updated TuneMountWithContextAllowNil to use a struct (with all pointers).
* Allow setting empty values for userLockoutConfig too - use new struct.
* Extra pointer.
* Remove useless functions.
* Simple test to ensure any field we can set we can update and vice-versa.
* Add json tag checks.
Co-authored-by: Kit Haines <khaines@mit.edu>
* Add additional validations to the destination and properties of file audit sinks
* changelog
* docs
* Revert "docs"
This reverts commit c2e8f7608ede78adf96726100013a45831712e13.
* cli: only set default command parameter to plugin name if sha256 is provided
* api: write warnings to RegisterPluginResponse, propagate up to cli
* api: filter out 'Endpoint replaced the value of these parameters' warning before returning in RegisterPluginWithContext
* docs
* add TODO on filtering that links to api type parameter deprecation ticket
* fix tests
* allocate filteredWarning slice only if there are warnings
* improve deferred resp close and early error return conditionals in RegisterPluginWithContext
* refer to sha256 as cli option -sha256 in command cli usage
* break up ui error lines for sha256 and version flag check
* consolidate if statements for sha256 and command, oci_image check in cli
* consolidate if statements for sha256 and command, oci_image check in api
* new RegisterPluginV2 and RegisterPluginWithContextV2 api client functions for backward compatibility
* add changelog
* more descriptive changelog
* rename RegisterPluginV2 to RegisterPluginDetailed and RegisterPluginWithContextV2 to RegisterPluginWithContextDetailed
* return nil, nil if no warnings to preserve status code
* fix eof from decoding (check if no content before decoding)
* doc for RegisterPluginResponse
* only validate plugin.Command in plugin catalog set for downloaded and binary plugins, which rely on plugin.Command input; extracted artifact plugins don't rely on plugin.Command input
* Update website/content/api-docs/system/plugins-catalog.mdx
Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
* Update website/content/api-docs/system/plugins-catalog.mdx
Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
* Update website/content/api-docs/system/plugins-catalog.mdx
Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
* Update website/content/docs/commands/plugin/register.mdx
Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
* Update website/content/docs/commands/plugin/register.mdx
Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
* Update website/content/docs/commands/plugin/register.mdx
Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
* Update website/content/docs/commands/plugin/register.mdx
Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
* move up enterprise note on plugin register command doc
* [DOCS] Editorial suggestions for PR #30811 (#31111)
* suggestions
* move common reqs to a partial
* fix typo
* tweak reqs
* Update website/content/partials/plugins/prepare-plugin.mdx
Co-authored-by: helenfufu <25168806+helenfufu@users.noreply.github.com>
* Update website/content/partials/plugins/prepare-plugin.mdx
Co-authored-by: helenfufu <25168806+helenfufu@users.noreply.github.com>
* Update website/content/partials/plugins/prepare-plugin.mdx
Co-authored-by: helenfufu <25168806+helenfufu@users.noreply.github.com>
* tweak feedback
* remove deprecation
* Update website/content/partials/plugins/common-requirements.mdx
Co-authored-by: helenfufu <25168806+helenfufu@users.noreply.github.com>
* save
* Update website/content/docs/plugins/rollback.mdx
Co-authored-by: helenfufu <25168806+helenfufu@users.noreply.github.com>
* Update website/content/docs/plugins/upgrade.mdx
Co-authored-by: helenfufu <25168806+helenfufu@users.noreply.github.com>
* fix formatting
---------
Co-authored-by: helenfufu <25168806+helenfufu@users.noreply.github.com>
---------
Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
Bump `hashicorp/go-discover` to v1.1.0. The updated module includes a
fix for `discover.Config.String()` where the marshaled configuration
was not properly quoting some values that includes equals characters. In
practice this could break some previously valid Raft `auto_join`
configuration.
Signed-off-by: Ryan Cragun <me@ryan.ec>
* upgrade hcl dependency on api pkg
This upgrades the hcl dependency for the API pkg,
and adapts its usage so users of our API pkg are
not affected. There's no good way of communicating
a warning via a library call so we don't.
The tokenHelper which is used by all Vault CLI
commands in order to create the Vault client, as
well as directly used by the login and server
commands, is implemented on the api pkg, so this
upgrade also affects all of those commands. Seems
like this was only moved to the api pkg because
the Terraform provider uses it, and I thought
creating a full copy of all those files back under
command would be too much spaghetti.
Also leaving some TODOs to make next deprecation
steps easier.
* upgrade hcl dependency in vault and sdk pkgs
* upgrade hcl dependency in vault and sdk pkgs
* add CLI warnings to commands that take a config
- vault agent (unit test on CMD warning)
- vault proxy (unit test on CMD warning)
- vault server (no test for the warning)
- vault operator diagnose (no tests at all, uses the
same function as vault server
* ignore duplicates on ParseKMSes function
* Extend policy parsing functions and warn on policy store
* Add warning on policy fmt with duplicate attributes
* Add warnings when creating/updating policy with duplicate HCL attrs
* Add log warning when switchedGetPolicy finds duplicate attrs
Following operations can trigger this warning when they run into a policy
with duplicate attributes:
* replication filtered path namespaces invalidation
* policy read API
* building an ACL (for many different purposes like most authZ operations)
* looking up DR token policies
* creating a token with named policies
* when caching the policies for all namespaces during unseal
* Print log warnings when token inline policy has duplicate attrs
No unit tests on these as new test infra would have to be built on all.
Operations affected, which will now print a log warning when the retrieved
token has an inline policy with duplicate attributes:
* capabilities endpoints in sys mount
* handing events under a subscription with a token with duplicate
attrs in inline policies
* token used to create another token has duplicate attrs in inline
policies (sudo check)
* all uses of fetchACLTokenEntryAndEntity when the request uses a
token with inline policies with duplicate attrs. Almost all reqs
are subject to this
* when tokens are created with inline policies (unclear exactly how that
can happen)
* add changelog and deprecation notice
* add missing copywrite notice
* fix copy-paste mistake
good thing it was covered by unit tests
* Fix manual parsing of telemetry field in SharedConfig
This commit in the hcl library was not in the
v1.0.1-vault-5 version we're using but is
included in v1.0.1-vault-7:
e80118accb
This thing of reusing when parsing means that
our approach of manually re-parsing fields
on top of fields that have already been parsed
by the hcl annotation causes strings (maybe
more?) to concatenate.
Fix that by removing annotation. There's
actually more occurrences of this thing of
automatically parsing something that is also
manually parsing. In some places we could
just remove the boilerplate manual parsing, in
others we better remove the auto parsing, but
I don't wanna pull at that thread right now. I
just checked that all places at least fully
overwrite the automatically parsed field
instead of reusing it as the target of the
decode call. The only exception is the AOP
field on ent but that doesn't have maps or
slices, so I think it's fine.
An alternative approach would be to ensure
that the auto-parsed value is discarded,
like the current parseCache function does
note how it's template not templates
* Fix linter complaints
* Update command/base_predict.go
Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com>
* address review
* remove copywrite headers
* re-add copywrite headers
* make fmt
* Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx
Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
* Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx
Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
* Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx
Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
* undo changes to deprecation.mdx
* remove deprecation doc
* fix conflict with changes from main
---------
Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com>
Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
* add automatic irrevocable lease removal features
* cleanup
* pull out ent changes
* strip out more stuff
* quiet CI screaming
* fix another test
* remove cl
* update irrevocable lease removal timing logic
* whoops
* require explicit value for disable_mlock
* set disable_mlock back to true for all docker tests
* fix build error
* update test config files
* change explicit mlock check to apply to integrated storage only.
* formatting and typo fixes
* added test for raft
* remove erroneous test
* remove unecessary doc line
* remove unecessary var
* pr suggestions
* test compile fix
* add mlock config value to enos tests
* enos lint
* update enos tests to pass disable_mlock value
* move mlock error to runtime to check for env var
* fixed mlock config detection logic
* call out mlock on/off tradeoffs to docs
* rewording production hardening section on mlock for clarity
* update error message when missing disable_mlock value to help customers with the previous default
* fix config doc error and update production-hardening doc to align with existing recommendations.
* remove extra check for mlock config value
* fix docker recovery test
* Update changelog/29974.txt
Explicitly call out that Vault will not start without disable_mlock included in the config.
Co-authored-by: Kuba Wieczorek <kuba.wieczorek@hashicorp.com>
* more docker test experimentation.
* passing disable_mlock into test cluster
* add VAULT_DISABLE_MLOCK envvar to docker tests and pass through the value
* add missing envvar for docker env test
* upate additional docker test disable_mlock values
* Apply suggestions from code review
Use active voice.
Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
---------
Co-authored-by: Kuba Wieczorek <kuba.wieczorek@hashicorp.com>
Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
* apply oss changes
* comment fix
* fix TestOperatorUsageCommandRun by defining billing start via license
* update go mod
* revert the changes in operator usage test
* fix operator usage test
* fix acme regeneration tests
* revert the changes for activity testonly test
* fix activity testonly tests
* seperate tests into ce and ent
* move 2 more tests to oss and ent
* remove left over test from common
* updates after feedback
* updates
* added unit tests to tests oss get start and end time function
* bring updates from ent
* carry over updates from ent pr
* fix the wording in ce warning
* add a dot to ent warning
* update comment
* revert go mod and go sum changes, remove the unintended oss changes patch
* add changelog entree for ce
`go-discover` supports being configured with some configuration strings
that include double-quotes, backslashes and escapes. As such, we now use
its own parser when normalizing `auto_join` config that may have
addresses.
Signed-off-by: Ryan Cragun <me@ryan.ec>
This is a follow-up to our initial work[0] to address RFC-5952 §4 conformance for IPv6 addresses in Vault. The initial pass focused on the vault server configuration and start-up routines. This follow-up focuses on Agent and Proxy, with a few minor improvements for server.
The approach generally mirrors the server implementation but also adds support for normalization with CLI configuration overrides.
One aspect we do not normalize currently is Agent/Proxy client creation to the Vault server with credentials taken from environment variables, as it would require larger changes to the `api` module. In practice this ought to be fine for the majority of cases.
[0]: https://github.com/hashicorp/vault/pull/29228
