* upgrade hcl dependency on api pkg
This upgrades the hcl dependency for the API pkg,
and adapts its usage so users of our API pkg are
not affected. There's no good way of communicating
a warning via a library call so we don't.
The tokenHelper which is used by all Vault CLI
commands in order to create the Vault client, as
well as directly used by the login and server
commands, is implemented on the api pkg, so this
upgrade also affects all of those commands. Seems
like this was only moved to the api pkg because
the Terraform provider uses it, and I thought
creating a full copy of all those files back under
command would be too much spaghetti.
Also leaving some TODOs to make next deprecation
steps easier.
* upgrade hcl dependency in vault and sdk pkgs
* upgrade hcl dependency in vault and sdk pkgs
* add CLI warnings to commands that take a config
- vault agent (unit test on CMD warning)
- vault proxy (unit test on CMD warning)
- vault server (no test for the warning)
- vault operator diagnose (no tests at all, uses the
same function as vault server
* ignore duplicates on ParseKMSes function
* Extend policy parsing functions and warn on policy store
* Add warning on policy fmt with duplicate attributes
* Add warnings when creating/updating policy with duplicate HCL attrs
* Add log warning when switchedGetPolicy finds duplicate attrs
Following operations can trigger this warning when they run into a policy
with duplicate attributes:
* replication filtered path namespaces invalidation
* policy read API
* building an ACL (for many different purposes like most authZ operations)
* looking up DR token policies
* creating a token with named policies
* when caching the policies for all namespaces during unseal
* Print log warnings when token inline policy has duplicate attrs
No unit tests on these as new test infra would have to be built on all.
Operations affected, which will now print a log warning when the retrieved
token has an inline policy with duplicate attributes:
* capabilities endpoints in sys mount
* handing events under a subscription with a token with duplicate
attrs in inline policies
* token used to create another token has duplicate attrs in inline
policies (sudo check)
* all uses of fetchACLTokenEntryAndEntity when the request uses a
token with inline policies with duplicate attrs. Almost all reqs
are subject to this
* when tokens are created with inline policies (unclear exactly how that
can happen)
* add changelog and deprecation notice
* add missing copywrite notice
* fix copy-paste mistake
good thing it was covered by unit tests
* Fix manual parsing of telemetry field in SharedConfig
This commit in the hcl library was not in the
v1.0.1-vault-5 version we're using but is
included in v1.0.1-vault-7:
e80118accb
This thing of reusing when parsing means that
our approach of manually re-parsing fields
on top of fields that have already been parsed
by the hcl annotation causes strings (maybe
more?) to concatenate.
Fix that by removing annotation. There's
actually more occurrences of this thing of
automatically parsing something that is also
manually parsing. In some places we could
just remove the boilerplate manual parsing, in
others we better remove the auto parsing, but
I don't wanna pull at that thread right now. I
just checked that all places at least fully
overwrite the automatically parsed field
instead of reusing it as the target of the
decode call. The only exception is the AOP
field on ent but that doesn't have maps or
slices, so I think it's fine.
An alternative approach would be to ensure
that the auto-parsed value is discarded,
like the current parseCache function does
note how it's template not templates
* Fix linter complaints
* Update command/base_predict.go
Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com>
* address review
* remove copywrite headers
* re-add copywrite headers
* make fmt
* Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx
Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
* Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx
Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
* Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx
Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
* undo changes to deprecation.mdx
* remove deprecation doc
* fix conflict with changes from main
---------
Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com>
Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
* add automatic irrevocable lease removal features
* cleanup
* pull out ent changes
* strip out more stuff
* quiet CI screaming
* fix another test
* remove cl
* update irrevocable lease removal timing logic
* whoops
* require explicit value for disable_mlock
* set disable_mlock back to true for all docker tests
* fix build error
* update test config files
* change explicit mlock check to apply to integrated storage only.
* formatting and typo fixes
* added test for raft
* remove erroneous test
* remove unecessary doc line
* remove unecessary var
* pr suggestions
* test compile fix
* add mlock config value to enos tests
* enos lint
* update enos tests to pass disable_mlock value
* move mlock error to runtime to check for env var
* fixed mlock config detection logic
* call out mlock on/off tradeoffs to docs
* rewording production hardening section on mlock for clarity
* update error message when missing disable_mlock value to help customers with the previous default
* fix config doc error and update production-hardening doc to align with existing recommendations.
* remove extra check for mlock config value
* fix docker recovery test
* Update changelog/29974.txt
Explicitly call out that Vault will not start without disable_mlock included in the config.
Co-authored-by: Kuba Wieczorek <kuba.wieczorek@hashicorp.com>
* more docker test experimentation.
* passing disable_mlock into test cluster
* add VAULT_DISABLE_MLOCK envvar to docker tests and pass through the value
* add missing envvar for docker env test
* upate additional docker test disable_mlock values
* Apply suggestions from code review
Use active voice.
Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
---------
Co-authored-by: Kuba Wieczorek <kuba.wieczorek@hashicorp.com>
Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
* apply oss changes
* comment fix
* fix TestOperatorUsageCommandRun by defining billing start via license
* update go mod
* revert the changes in operator usage test
* fix operator usage test
* fix acme regeneration tests
* revert the changes for activity testonly test
* fix activity testonly tests
* seperate tests into ce and ent
* move 2 more tests to oss and ent
* remove left over test from common
* updates after feedback
* updates
* added unit tests to tests oss get start and end time function
* bring updates from ent
* carry over updates from ent pr
* fix the wording in ce warning
* add a dot to ent warning
* update comment
* revert go mod and go sum changes, remove the unintended oss changes patch
* add changelog entree for ce
`go-discover` supports being configured with some configuration strings
that include double-quotes, backslashes and escapes. As such, we now use
its own parser when normalizing `auto_join` config that may have
addresses.
Signed-off-by: Ryan Cragun <me@ryan.ec>
This is a follow-up to our initial work[0] to address RFC-5952 §4 conformance for IPv6 addresses in Vault. The initial pass focused on the vault server configuration and start-up routines. This follow-up focuses on Agent and Proxy, with a few minor improvements for server.
The approach generally mirrors the server implementation but also adds support for normalization with CLI configuration overrides.
One aspect we do not normalize currently is Agent/Proxy client creation to the Vault server with credentials taken from environment variables, as it would require larger changes to the `api` module. In practice this ought to be fine for the majority of cases.
[0]: https://github.com/hashicorp/vault/pull/29228
USGv6[0] requires implementing §4.1.1 of the NISTv6-r1 profile[1] for
IPv6-Only capabilities. This section requires that whenever Vault
displays IPv6 addresses (including CLI output, Web UI, logs, etc.) that
_all_ IPv6 addresses must conform to RFC-5952 §4 text representation
recommendations[2].
These recommendations do not prevent us from accepting RFC-4241[3] IPv6
addresses, however, whenever these same addresses are displayed they
must conform to the strict RFC-5952 §4 guidelines.
This PR implements handling of IPv6 address conformance in our
`vault server` routine. We handle conformance normalization for all
server, http_proxy, listener, seal, storage and telemetry
configuration where an input could contain an IPv6 address, whether
configured via an HCL file or via corresponding environment variables.
The approach I've taken is to handle conformance normalization at
parse time to ensure that all log output and subsequent usage
inside of Vaults various subsystems always reference a conformant
address, that way we don't need concern ourselves with conformance
later. This approach ought to be backwards compatible to prior loose
address configuration requirements, with the understanding that
going forward all IPv6 representation will be strict regardless of
what has been configured.
In many cases I've updated our various parser functions to call the
new `configutil.NormalizeAddr()` to apply conformance normalization.
Others required no changes because they rely on standard library URL
string output, which always displays IPv6 URLs in a conformant way.
Not included in this changes is any other vault exec mode other than
server. Client, operator commands, agent mode, proxy mode, etc. will
be included in subsequent changes if necessary.
[0]: https://www.nist.gov/publications/usgv6-profile
[1]: https://www.nist.gov/publications/nist-ipv6-profile
[2]: https://www.rfc-editor.org/rfc/rfc5952.html#section-4
[3]: https://www.rfc-editor.org/rfc/rfc4291
Signed-off-by: Ryan Cragun <me@ryan.ec>
* Fix "t.Fatal from a non-test goroutine" errors in cache_test.go
- t.Fatal(f) should not be called within a Go routine based on it's documentation and only from the main test's thread.
- In 1.24 this seems to cause build failures
* Address all "non-constant format string errors" from go vet
- Within 1.24 these now cause test builds to fail
…" from go vet
* Do not use static certificates for diagnose tests
* Fix operator command tests, move PKI CA creation code into testhelper lib
* Fix compilation error from refactoring
* Use DRBG based RSA key generation everywhere
* switch to the conditional generator
* Use DRBG based RSA key generation everywhere
* switch to the conditional generator
* Add an ENV var to disable the DRBG in a pinch
* update go.mod
* Use DRBG based RSA key generation everywhere
* switch to the conditional generator
* Add an ENV var to disable the DRBG in a pinch
* Use DRBG based RSA key generation everywhere
* update go.mod
* fix import
* Remove rsa2 alias, remove test code
* move cryptoutil/rsa.go to sdk
* move imports too
* remove makefile change
* rsa2->rsa
* more rsa2->rsa, remove test code
* fix some overzelous search/replace
* Update to a real tag
* changelog
* copyright
* work around copyright check
* work around copyright check pt2
* bunch of dupe imports
* missing import
* wrong license
* fix go.mod conflict
* missed a spot
* dupe import
* initial implementation of unseal trace
* close file if we fail to start the trace
didn't bother to check the error from traceFile.Close()
* use reloadable config instead of env var
* license
* remove leftover
* allow setting custom dir and remove new package
* bring back StartDebugTrace
after talking to Kuba it sounds like it's a good idea to try to move stuff out of core, so even if there's no immediate need for a generic debug trace function it's still fair to add it
* track postUnseal instead of unsealInternal
also some usability improvements from manual testing
* address PR comments
* address security review
there were concerns about using the /tmp directory because of permissions, or having a default dir at all, so now it's required to set a dir in order to generate the traces.
* add unit tests to StartDebugTrace
* move back to default dir
* document new parameters
* add tiny integration test
* avoid column in trace filename
sounds like it might be forbidden in Windows and possibly cause problems in some MacOS applications.
* address PR feedback
* add go doc to test
CI was complaining about missing comments on the new test function. It feels a bit silly to require this of tests but whatever XD
* fix tests
* Support trimming trailing slashes via a mount tuneable to support CMPv2
* changelog/
* Perform trimming in handleLoginRequest too
* Eagerly fetch the mount entry so we only test this once
* Add a mount match function that gets path and entry
* Update vault/request_handling.go
Co-authored-by: Steven Clark <steven.clark@hashicorp.com>
* more docs
* Some patches (from ENT) didnt apply
* patch fail
* Update vault/router.go
Co-authored-by: Steven Clark <steven.clark@hashicorp.com>
* PR feedback
* dupe
* another dupe
* Add support for enabling trim_request_trailing_slashes on mount creation
* Fix read mount api returning configuration for trim_request_trailing_slashes
* Fix test assertion
* Switch enable and tune arguments to BoolPtrVal to allow end-users to specify false flag
* Add trim-request-trailing-slashes to the auth enable API and CLI
---------
Co-authored-by: Steven Clark <steven.clark@hashicorp.com>
