* license: update headers to IBM Corp.
* `make proto`
* update offset because source file changed
Signed-off-by: Ryan Cragun <me@ryan.ec>
Co-authored-by: Ryan Cragun <me@ryan.ec>
Bump `hashicorp/go-discover` to v1.1.0. The updated module includes a
fix for `discover.Config.String()` where the marshaled configuration
was not properly quoting some values that includes equals characters. In
practice this could break some previously valid Raft `auto_join`
configuration.
Signed-off-by: Ryan Cragun <me@ryan.ec>
* upgrade hcl dependency on api pkg
This upgrades the hcl dependency for the API pkg,
and adapts its usage so users of our API pkg are
not affected. There's no good way of communicating
a warning via a library call so we don't.
The tokenHelper which is used by all Vault CLI
commands in order to create the Vault client, as
well as directly used by the login and server
commands, is implemented on the api pkg, so this
upgrade also affects all of those commands. Seems
like this was only moved to the api pkg because
the Terraform provider uses it, and I thought
creating a full copy of all those files back under
command would be too much spaghetti.
Also leaving some TODOs to make next deprecation
steps easier.
* upgrade hcl dependency in vault and sdk pkgs
* upgrade hcl dependency in vault and sdk pkgs
* add CLI warnings to commands that take a config
- vault agent (unit test on CMD warning)
- vault proxy (unit test on CMD warning)
- vault server (no test for the warning)
- vault operator diagnose (no tests at all, uses the
same function as vault server
* ignore duplicates on ParseKMSes function
* Extend policy parsing functions and warn on policy store
* Add warning on policy fmt with duplicate attributes
* Add warnings when creating/updating policy with duplicate HCL attrs
* Add log warning when switchedGetPolicy finds duplicate attrs
Following operations can trigger this warning when they run into a policy
with duplicate attributes:
* replication filtered path namespaces invalidation
* policy read API
* building an ACL (for many different purposes like most authZ operations)
* looking up DR token policies
* creating a token with named policies
* when caching the policies for all namespaces during unseal
* Print log warnings when token inline policy has duplicate attrs
No unit tests on these as new test infra would have to be built on all.
Operations affected, which will now print a log warning when the retrieved
token has an inline policy with duplicate attributes:
* capabilities endpoints in sys mount
* handing events under a subscription with a token with duplicate
attrs in inline policies
* token used to create another token has duplicate attrs in inline
policies (sudo check)
* all uses of fetchACLTokenEntryAndEntity when the request uses a
token with inline policies with duplicate attrs. Almost all reqs
are subject to this
* when tokens are created with inline policies (unclear exactly how that
can happen)
* add changelog and deprecation notice
* add missing copywrite notice
* fix copy-paste mistake
good thing it was covered by unit tests
* Fix manual parsing of telemetry field in SharedConfig
This commit in the hcl library was not in the
v1.0.1-vault-5 version we're using but is
included in v1.0.1-vault-7:
e80118accb
This thing of reusing when parsing means that
our approach of manually re-parsing fields
on top of fields that have already been parsed
by the hcl annotation causes strings (maybe
more?) to concatenate.
Fix that by removing annotation. There's
actually more occurrences of this thing of
automatically parsing something that is also
manually parsing. In some places we could
just remove the boilerplate manual parsing, in
others we better remove the auto parsing, but
I don't wanna pull at that thread right now. I
just checked that all places at least fully
overwrite the automatically parsed field
instead of reusing it as the target of the
decode call. The only exception is the AOP
field on ent but that doesn't have maps or
slices, so I think it's fine.
An alternative approach would be to ensure
that the auto-parsed value is discarded,
like the current parseCache function does
note how it's template not templates
* Fix linter complaints
* Update command/base_predict.go
Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com>
* address review
* remove copywrite headers
* re-add copywrite headers
* make fmt
* Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx
Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
* Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx
Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
* Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx
Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
* undo changes to deprecation.mdx
* remove deprecation doc
* fix conflict with changes from main
---------
Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com>
Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
* add automatic irrevocable lease removal features
* cleanup
* pull out ent changes
* strip out more stuff
* quiet CI screaming
* fix another test
* remove cl
* update irrevocable lease removal timing logic
* whoops
* require explicit value for disable_mlock
* set disable_mlock back to true for all docker tests
* fix build error
* update test config files
* change explicit mlock check to apply to integrated storage only.
* formatting and typo fixes
* added test for raft
* remove erroneous test
* remove unecessary doc line
* remove unecessary var
* pr suggestions
* test compile fix
* add mlock config value to enos tests
* enos lint
* update enos tests to pass disable_mlock value
* move mlock error to runtime to check for env var
* fixed mlock config detection logic
* call out mlock on/off tradeoffs to docs
* rewording production hardening section on mlock for clarity
* update error message when missing disable_mlock value to help customers with the previous default
* fix config doc error and update production-hardening doc to align with existing recommendations.
* remove extra check for mlock config value
* fix docker recovery test
* Update changelog/29974.txt
Explicitly call out that Vault will not start without disable_mlock included in the config.
Co-authored-by: Kuba Wieczorek <kuba.wieczorek@hashicorp.com>
* more docker test experimentation.
* passing disable_mlock into test cluster
* add VAULT_DISABLE_MLOCK envvar to docker tests and pass through the value
* add missing envvar for docker env test
* upate additional docker test disable_mlock values
* Apply suggestions from code review
Use active voice.
Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
---------
Co-authored-by: Kuba Wieczorek <kuba.wieczorek@hashicorp.com>
Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
`go-discover` supports being configured with some configuration strings
that include double-quotes, backslashes and escapes. As such, we now use
its own parser when normalizing `auto_join` config that may have
addresses.
Signed-off-by: Ryan Cragun <me@ryan.ec>
USGv6[0] requires implementing §4.1.1 of the NISTv6-r1 profile[1] for
IPv6-Only capabilities. This section requires that whenever Vault
displays IPv6 addresses (including CLI output, Web UI, logs, etc.) that
_all_ IPv6 addresses must conform to RFC-5952 §4 text representation
recommendations[2].
These recommendations do not prevent us from accepting RFC-4241[3] IPv6
addresses, however, whenever these same addresses are displayed they
must conform to the strict RFC-5952 §4 guidelines.
This PR implements handling of IPv6 address conformance in our
`vault server` routine. We handle conformance normalization for all
server, http_proxy, listener, seal, storage and telemetry
configuration where an input could contain an IPv6 address, whether
configured via an HCL file or via corresponding environment variables.
The approach I've taken is to handle conformance normalization at
parse time to ensure that all log output and subsequent usage
inside of Vaults various subsystems always reference a conformant
address, that way we don't need concern ourselves with conformance
later. This approach ought to be backwards compatible to prior loose
address configuration requirements, with the understanding that
going forward all IPv6 representation will be strict regardless of
what has been configured.
In many cases I've updated our various parser functions to call the
new `configutil.NormalizeAddr()` to apply conformance normalization.
Others required no changes because they rely on standard library URL
string output, which always displays IPv6 URLs in a conformant way.
Not included in this changes is any other vault exec mode other than
server. Client, operator commands, agent mode, proxy mode, etc. will
be included in subsequent changes if necessary.
[0]: https://www.nist.gov/publications/usgv6-profile
[1]: https://www.nist.gov/publications/nist-ipv6-profile
[2]: https://www.rfc-editor.org/rfc/rfc5952.html#section-4
[3]: https://www.rfc-editor.org/rfc/rfc4291
Signed-off-by: Ryan Cragun <me@ryan.ec>
* Do not use static certificates for diagnose tests
* Fix operator command tests, move PKI CA creation code into testhelper lib
* Fix compilation error from refactoring
* initial implementation of unseal trace
* close file if we fail to start the trace
didn't bother to check the error from traceFile.Close()
* use reloadable config instead of env var
* license
* remove leftover
* allow setting custom dir and remove new package
* bring back StartDebugTrace
after talking to Kuba it sounds like it's a good idea to try to move stuff out of core, so even if there's no immediate need for a generic debug trace function it's still fair to add it
* track postUnseal instead of unsealInternal
also some usability improvements from manual testing
* address PR comments
* address security review
there were concerns about using the /tmp directory because of permissions, or having a default dir at all, so now it's required to set a dir in order to generate the traces.
* add unit tests to StartDebugTrace
* move back to default dir
* document new parameters
* add tiny integration test
* avoid column in trace filename
sounds like it might be forbidden in Windows and possibly cause problems in some MacOS applications.
* address PR feedback
* add go doc to test
CI was complaining about missing comments on the new test function. It feels a bit silly to require this of tests but whatever XD
* fix tests
* Add a configuration flag for enabling multiseal (Seal HA), CE side
* imports
* no quotes
* get rid of dep on ent config
* Abstract enableMultiSeal for a build time switch
* license headers
* wip
* gate physical seal gen fetch by a param
* docs tweak, remove core flag
* updates from the ent pr
* update stub
* update test fixtures for enable_multiseal
* use accessor
* add a test fixture for non-multiseal diagnose
* remove debugging crtuch
* Do handle phys seal gen info even if multiseal is off, in order to facilitate enable/disable safeties
* more enabled flag handling
* Accept seal gen info if we were previously disabled, and persist it
* update unit test
* Validation happens postUnseal, so this test is invalid
* Dont continue setting conf if seal loading fails during SIGHUP
* Update website/content/docs/configuration/seal/seal-ha.mdx
Thanks, that does sound much clearer
Co-authored-by: Jason O'Donnell <2160810+jasonodonnell@users.noreply.github.com>
* use validation if previous gen was enabled
* unit test update
* stub SetMultisealEnabled
* bring over more changes from ent
* this was an unfix
---------
Co-authored-by: Jason O'Donnell <2160810+jasonodonnell@users.noreply.github.com>
* Initial oss-patch apply
* Added changelog
* Renamed changelog txt
* Added the imports to the handler file
* Added a check that no two ports are the same, and modified changelog
* Edited go sum entry
* Tidy up using go mod
* Use strutil instead
* Revert go sum and go mod
* Revert sdk go sum
* Edited go.sum to before
* Edited go.sum again to initial
* Revert changes
* Adding explicit MPL license for sub-package.
This directory and its subdirectories (packages) contain files licensed with the MPLv2 `LICENSE` file in this directory and are intentionally licensed separately from the BSL `LICENSE` file at the root of this repository.
* Adding explicit MPL license for sub-package.
This directory and its subdirectories (packages) contain files licensed with the MPLv2 `LICENSE` file in this directory and are intentionally licensed separately from the BSL `LICENSE` file at the root of this repository.
* Updating the license from MPL to Business Source License.
Going forward, this project will be licensed under the Business Source License v1.1. Please see our blog post for more details at https://hashi.co/bsl-blog, FAQ at www.hashicorp.com/licensing-faq, and details of the license at www.hashicorp.com/bsl.
* add missing license headers
* Update copyright file headers to BUS-1.1
* Fix test that expected exact offset on hcl file
---------
Co-authored-by: hashicorp-copywrite[bot] <110428419+hashicorp-copywrite[bot]@users.noreply.github.com>
Co-authored-by: Sarah Thompson <sthompson@hashicorp.com>
Co-authored-by: Brian Kassouf <bkassouf@hashicorp.com>
* add Link config, init, and capabilities
* add node status proto
* bump protoc version to 3.21.9
* make proto
* adding link tests
* remove wrapped link
* add changelog entry
* update changelog entry
* VAULT-1564 report in-flight requests
* adding a changelog
* Changing some variable names and fixing comments
* minor style change
* adding unauthenticated support for in-flight-req
* adding documentation for the listener.profiling stanza
* adding an atomic counter for the inflight requests
addressing comments
* addressing comments
* logging completed requests
* fixing a test
* providing log_requests_info as a config option to determine at which level requests should be logged
* removing a member and a method from the StatusHeaderResponseWriter struct
* adding api docks
* revert changes in NewHTTPResponseWriter
* Fix logging invalid log_requests_info value
* Addressing comments
* Fixing a test
* use an tomic value for logRequestsInfo, and moving the CreateClientID function to Core
* fixing go.sum
* minor refactoring
* protecting InFlightRequests from data race
* another try on fixing a data race
* another try to fix a data race
* addressing comments
* fixing couple of tests
* changing log_requests_info to log_requests_level
* minor style change
* fixing a test
* removing the lock in InFlightRequests
* use single-argument form for interface assertion
* adding doc for the new configuration paramter
* adding the new doc to the nav data file
* minor fix
* Customizing HTTP headers in the config file
* Add changelog, fix bad imports
* fixing some bugs
* fixing interaction of custom headers and /ui
* Defining a member in core to set custom response headers
* missing additional file
* Some refactoring
* Adding automated tests for the feature
* Changing some error messages based on some recommendations
* Incorporating custom response headers struct into the request context
* removing some unused references
* fixing a test
* changing some error messages, removing a default header value from /ui
* fixing a test
* wrapping ResponseWriter to set the custom headers
* adding a new test
* some cleanup
* removing some extra lines
* Addressing comments
* fixing some agent tests
* skipping custom headers from agent listener config,
removing two of the default headers as they cause issues with Vault in UI mode
Adding X-Content-Type-Options to the ui default headers
Let Content-Type be set as before
* Removing default custom headers, and renaming some function varibles
* some refacotring
* Refactoring and addressing comments
* removing a function and fixing comments
* save
* save
* save
* first round of the diagnose language pass
* capitalization
* first round of feedback
* fix bug in advise
* a few more nouns to verbs
* diagnose: Add seal transit tls check
* Fixing the path to the config file and the path to the cert files
* Addressing comment
* Addressing seal transit tls check comments
* Actually call config.Validate in diagnose
* Wire configuration checks into diagnose and fix resulting bugs.
* go mod vendor
* Merge to vendorless version
* Remove sentinel section to allow diagnose_ok to pass
* Fix unit tests
* raft file and quorum checks
* raft checks
* backup
* raft file checks test
* address comments and add more raft and file and process checks
* syntax issues
* modularize functions to compile differently on different os
* compile raft checks everywhere
* more build tag issues
* raft-diagnose
* correct file permission checks
* upgrade tests and add a getConfigOffline test that currently does not work
* comment
* update file checks method signature on windows
* Update physical/raft/raft_test.go
Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
* raft tests
* add todo comment for windows root ownership
* voter count message
* raft checks test fixes
Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
* Disk usage checks
* Move disk free earlier
* Move logic to helpers
* Bring over test logic from the ulimit PR
* imports
* Report error
* Get unit tests working
* initial refactoring of unseal step in run
* remove waitgroup
* remove waitgroup
* backup work
* backup
* backup
* completely modularize run and move into diagnose
* add diagnose errors for incorrect number of unseal keys
* comment tests back in
* backup
* first subspan
* finished subspanning but running into error with timeouts
* remove runtime checks
* meeting updates
* remove telemetry block
* roy comment
* subspans for seal finalization and wrapping diagnose latency checks
* fix storage latency test errors
* review comments
* use random uuid for latency checks instead of static id
* Expose unknown fields and duplicate sections as diagnose warnings
* section counts not needed, already handled
* Address PR feedback
* Prune more of the new fields before tests call deep.Equals
* Update go.mod
* Create helpers which integrate with OpenTelemetry for diagnose collection
* Go mod vendor
* consul tls checks
* draft for storage end to end check
* Comments
* Update vault/diagnose/helpers.go
Co-authored-by: swayne275 <swayne275@gmail.com>
* Add unit test/example
* tweak output
* More comments
* add spot check concept
* Get unit tests working on Result structs
* Fix unit test
* Get unit tests working, and make diagnose sessions local rather than global
* Comments
* Last comments
* No need for init
* :|
* Fix helpers_test
* cleaned up chan logic. Tests next.
* fix tests
* remove a comment
* tests
* remove a comment
* run direct access checks in diagnose command
* review comments
Co-authored-by: Scott G. Miller <smiller@hashicorp.com>
Co-authored-by: swayne275 <swayne275@gmail.com>
* sanity checks for tls config in diagnose
* backup
* backup
* backup
* added necessary tests
* remove comment
* remove parallels causing test flakiness
* comments
* small fix
* separate out config hcl test case into new hcl file
* newline
* addressed comments
* addressed comments
* addressed comments
* addressed comments
* addressed comments
* reload funcs should be allowed to be nil
* a few tests to the operator diagnose stub command
* a few tests to the operator diagnose stub command
* a few tests to the operator diagnose stub command
* empty commit to fix circle ci permissions issue
* empty commit to fix circle ci permissions issue
