USGv6[0] requires implementing §4.1.1 of the NISTv6-r1 profile[1] for
IPv6-Only capabilities. This section requires that whenever Vault
displays IPv6 addresses (including CLI output, Web UI, logs, etc.) that
_all_ IPv6 addresses must conform to RFC-5952 §4 text representation
recommendations[2].
These recommendations do not prevent us from accepting RFC-4241[3] IPv6
addresses, however, whenever these same addresses are displayed they
must conform to the strict RFC-5952 §4 guidelines.
This PR implements handling of IPv6 address conformance in our
`vault server` routine. We handle conformance normalization for all
server, http_proxy, listener, seal, storage and telemetry
configuration where an input could contain an IPv6 address, whether
configured via an HCL file or via corresponding environment variables.
The approach I've taken is to handle conformance normalization at
parse time to ensure that all log output and subsequent usage
inside of Vaults various subsystems always reference a conformant
address, that way we don't need concern ourselves with conformance
later. This approach ought to be backwards compatible to prior loose
address configuration requirements, with the understanding that
going forward all IPv6 representation will be strict regardless of
what has been configured.
In many cases I've updated our various parser functions to call the
new `configutil.NormalizeAddr()` to apply conformance normalization.
Others required no changes because they rely on standard library URL
string output, which always displays IPv6 URLs in a conformant way.
Not included in this changes is any other vault exec mode other than
server. Client, operator commands, agent mode, proxy mode, etc. will
be included in subsequent changes if necessary.
[0]: https://www.nist.gov/publications/usgv6-profile
[1]: https://www.nist.gov/publications/nist-ipv6-profile
[2]: https://www.rfc-editor.org/rfc/rfc5952.html#section-4
[3]: https://www.rfc-editor.org/rfc/rfc4291
Signed-off-by: Ryan Cragun <me@ryan.ec>
* initial implementation of unseal trace
* close file if we fail to start the trace
didn't bother to check the error from traceFile.Close()
* use reloadable config instead of env var
* license
* remove leftover
* allow setting custom dir and remove new package
* bring back StartDebugTrace
after talking to Kuba it sounds like it's a good idea to try to move stuff out of core, so even if there's no immediate need for a generic debug trace function it's still fair to add it
* track postUnseal instead of unsealInternal
also some usability improvements from manual testing
* address PR comments
* address security review
there were concerns about using the /tmp directory because of permissions, or having a default dir at all, so now it's required to set a dir in order to generate the traces.
* add unit tests to StartDebugTrace
* move back to default dir
* document new parameters
* add tiny integration test
* avoid column in trace filename
sounds like it might be forbidden in Windows and possibly cause problems in some MacOS applications.
* address PR feedback
* add go doc to test
CI was complaining about missing comments on the new test function. It feels a bit silly to require this of tests but whatever XD
* fix tests
* Add a configuration flag for enabling multiseal (Seal HA), CE side
* imports
* no quotes
* get rid of dep on ent config
* Abstract enableMultiSeal for a build time switch
* license headers
* wip
* gate physical seal gen fetch by a param
* docs tweak, remove core flag
* updates from the ent pr
* update stub
* update test fixtures for enable_multiseal
* use accessor
* add a test fixture for non-multiseal diagnose
* remove debugging crtuch
* Do handle phys seal gen info even if multiseal is off, in order to facilitate enable/disable safeties
* more enabled flag handling
* Accept seal gen info if we were previously disabled, and persist it
* update unit test
* Validation happens postUnseal, so this test is invalid
* Dont continue setting conf if seal loading fails during SIGHUP
* Update website/content/docs/configuration/seal/seal-ha.mdx
Thanks, that does sound much clearer
Co-authored-by: Jason O'Donnell <2160810+jasonodonnell@users.noreply.github.com>
* use validation if previous gen was enabled
* unit test update
* stub SetMultisealEnabled
* bring over more changes from ent
* this was an unfix
---------
Co-authored-by: Jason O'Donnell <2160810+jasonodonnell@users.noreply.github.com>
The flag `events.alpha1` will no longer do anything, but we keep it
to prevent breaking users who have it in their configurations or
startup flags, or if it is referenced in other code.
* Add -dev-tls-san flag
This is helpful when wanting to set up a dev server with TLS in Kubernetes
and any other situations where the dev server may not be the same machine
as the Vault client (e.g. in combination with some /etc/hosts entries)
* Automatically add (best-effort only) -dev-listen-address host to extraSANs
* Seal HA: Use new SealWrappedValue type to abstract seal wrapped values
Introduce SealWrappedValue to abstract seal wrapped values.
Make SealWrappedValue capable of marshalling into a BlobInfo, when there is
plaintext or a single encryption, or to a custom serialization consisting of a
header, length and a marshalled MultiWrapValue protobuf.
* Vault-13769: Support configuring and using multiple seals for unsealing
* Make sealWrapBackend start using multiple seals
* Make seal.Access no longer implement wrapping.Wrapper.
Instead, add the Encrypt and Decrypt methods to the Access interface.
* Make raft snapshot system use funcs SealWrapValue + UnsealWrapValue.
Move the snapshot.Sealer implementation to the vault package to
avoid circular imports.
* Update sealWrapBackend to use multiple seals for encryption.
Use all the encryption wrappers when storing seal wrapped values.
Try do decrypt using the highest priority wrapper, but try all
combinations of encrypted values and wrappers if necessary.
* Allow the use of multiple seals for entropy augmentation
Add seal_name variable in entropy stanza
Add new MultiSourcer to accommodate the new entropy augmentation behavior.
* Individually health check each wrapper, and add a sys/seal-backend-status endpoint.
* Address a race, and also a failed test mock that I didn't catch
* Track partial wrapping failures...
... where one or more but not all access.Encrypts fail for a given write.
Note these failures by adding a time ordered UUID storage entry containing
the path in a special subdirectory of root storage. Adds a callback
pattern to accomplish this, with certain high value writes like initial
barrier key storage not allowing a partial failure. The followup work
would be to detect return to health and iterate through these storage
entries, rewrapping.
* Add new data structure to track seal config generation (#4492)
* Add new data structure to track seal config generation
* Remove import cycle
* Fix undefined variable errors
* update comment
* Update setSeal response
* Fix setSealResponse in operator_diagnose
* Scope the wrapper health check locks individually (#4491)
* Refactor setSeal function in server.go. (#4505)
Refactor setSeal function in server.go.
* Decouple CreateSecureRandomReaderFunc from seal package.
Instead of using a list of seal.SealInfo structs, make
CreateSecureRandomReaderFunc use a list of new EntropySourcerInfo structs. This
brakes the denpency of package configutil on the seal package.
* Move SealGenerationInfo tracking to the seal Access.
* Move SealGenerationInfo tracking to the seal Access.
The SealGenerationInfo is now kept track by a Seal's Access instead of by the
Config object. The access implementation now records the correct generation
number on seal wrapped values.
* Only store and read SealGenerationInfo if VAULT_ENABLE_SEAL_HA_BETA is true.
* Add MultiWrapValue protobuf message
MultiWrapValue can be used to keep track of different encryptions of a value.
---------
Co-authored-by: Victor Rodriguez <vrizo@hashicorp.com>
* Use generation to determine if a seal wrapped value is up-to-date. (#4542)
* Add logging to seal Access implementation.
* Seal HA buf format run (#4561)
* Run buf format.
* Add buf.lock to ensure go-kms-wrapping module is imported.
* Vault-18958: Add unit tests for config checks
* Add safety logic for seal configuration changes
* Revert "Add safety logic for seal configuration changes"
This reverts commit 7fec48035a5cf274e5a4d98901716d08d766ce90.
* changes and tests for checking seal config
* add ent tests
* remove check for empty name and add type into test cases
* add error message for empty name
* fix no seals test
---------
Co-authored-by: divyapola5 <divya@hashicorp.com>
* Handle migrations between single-wrapper and multi-wrapper autoSeals
* Extract method SetPhysicalSealConfig.
* Extract function physicalSealConfig.
The extracted function is the only code now reading SealConfig entries from
storage.
* Extract function setPhysicalSealConfig.
The extracted function is the only code now writing SealConfig entries from
storage (except for migration from the old recovery config path).
* Move SealConfig to new file vault/seal_config.go.
* Add SealConfigType quasy-enumeration.
SealConfigType is to serve as the typed values for field SealConfig.Type.
* Rename Seal.RecoveryType to RecoverySealConfigType.
Make RecoverySealConfigType return a SealConfigType instead of a string.
* Rename Seal.BarrierType to BarrierSealConfigType.
Make BarrierSealConfigType return a SealConfigType.
Remove seal.SealType (really a two-step rename to SealConfigType).
* Add Seal methods ClearBarrierConfig and ClearRecoveryConfig.
* Handle autoseal <-> multiseal migrations.
While going between single-wrapper and multiple-wrapper autoseals are not
migrations that require an unwrap seal (such as going from shamir to autoseal),
the stored "barrier" SealConfig needs to be updated in these cases.
Specifically, the value of SealConfg.Type is "multiseal" for autoSeals that have
more than one wrapper; on the other hand, for autoseals with a single wrapper,
SealConfig.Type is the type of the wrapper.
* Remove error return value from NewAutoSeal constructor.
* Automatically rewrap partially seal wrapped values on an interval
* Add in rewrapping of partially wrapped values on an interval, regardless of seal health/status.
* Don't set SealGenerationInfo Rewrapped flag in the partial rewrap call.
* Unexport the SealGenerationInfo's Rewrapped field, add a mutex to it for thread safe access, and add accessor methods for it.
* Add a success callback to the manual seal rewrap process that updates the SealGenerationInfo's rewrapped field. This is done via a callback to avoid an import cycle in the SealRewrap code.
* Fix a failing seal wrap backend test which was broken by the unexporting of SealGenerationInfo's Rewrapped field.
* Nil check the seal rewrap success callback before calling it.
* Change SealGenerationInfo rewrapped parameter to an atomic.Bool rather than a sync.RWMutex for simplicity and performance.
* Add nil check for SealAccess before updating SealGenerationInfo rewrapped status during seal rewrap call.
* Update partial rewrap check interval from 10 seconds to 1 minute.
* Update a reference to SealGenerationInfo Rewrapped field to use new getter method.
* Fix up some data raciness in partial rewrapping.
* Account for possibly nil storage entry when retrieving partially wrapped value.
* Allow multi-wrapper autoSeals to include disabled seal wrappers.
* Restore propagation of wrapper configuration errors by setSeal.
Function setSeal is meant to propagate non KeyNotFound errors returned by calls
to configutil.ConfigureWrapper.
* Remove unused Access methods SetConfig and Type.
* Allow multi-wrapper autoSeals to include disabled seal wrappers.
Make it possible for an autoSeal that uses multiple wrappers to include disabled
wrappers that can be used to decrypt entries, but are skipped for encryption.
e an unwrapSeal when there are disabled seals.
* Fix bug with not providing name (#4580)
* add suffix to name defaults
* add comment
* only change name for disabled seal
* Only attempt to rewrap partial values when all seals are healthy.
* Only attempt to rewrap partial values when all seals are healthy.
* Change logging level from info to debug for notice about rewrap skipping based on seal health.
* Remove stale TODOs and commented out code.
---------
Co-authored-by: rculpepper <rculpepper@hashicorp.com>
Co-authored-by: Larroyo <95649169+DeLuci@users.noreply.github.com>
Co-authored-by: Scott G. Miller <smiller@hashicorp.com>
Co-authored-by: Divya Pola <87338962+divyapola5@users.noreply.github.com>
Co-authored-by: Matt Schultz <matt.schultz@hashicorp.com>
Co-authored-by: divyapola5 <divya@hashicorp.com>
Co-authored-by: Rachel Culpepper <84159930+rculpepper@users.noreply.github.com>
* Adding explicit MPL license for sub-package.
This directory and its subdirectories (packages) contain files licensed with the MPLv2 `LICENSE` file in this directory and are intentionally licensed separately from the BSL `LICENSE` file at the root of this repository.
* Adding explicit MPL license for sub-package.
This directory and its subdirectories (packages) contain files licensed with the MPLv2 `LICENSE` file in this directory and are intentionally licensed separately from the BSL `LICENSE` file at the root of this repository.
* Updating the license from MPL to Business Source License.
Going forward, this project will be licensed under the Business Source License v1.1. Please see our blog post for more details at https://hashi.co/bsl-blog, FAQ at www.hashicorp.com/licensing-faq, and details of the license at www.hashicorp.com/bsl.
* add missing license headers
* Update copyright file headers to BUS-1.1
* Fix test that expected exact offset on hcl file
---------
Co-authored-by: hashicorp-copywrite[bot] <110428419+hashicorp-copywrite[bot]@users.noreply.github.com>
Co-authored-by: Sarah Thompson <sthompson@hashicorp.com>
Co-authored-by: Brian Kassouf <bkassouf@hashicorp.com>
* add config changes for name and priority fields in seal stanza
* change env vars and fix tests
* add header and fix func call
* tweak limits on seals
* fix missing import
* add docstrings
* add max_entry_size to sanitized config output
* add changelog entry
* add test parallelism
* add inmem test case
* use named struct fields for TestSysConfigState_Sanitized cases
* Revert "Don't execute the seal recovery tests on ENT. (#18841)"
This reverts commit 990d3bacc2.
* Revert "Add the ability to unseal using recovery keys via an explicit seal option. (#18683)"
This reverts commit 2ffe49aab0.
* wip
* wip
* Got it 'working', but not happy about cleanliness yet
* Switch to a dedicated defaultSeal with recovery keys
This is simpler than trying to hijack SealAccess as before. Instead, if the operator
has requested recovery unseal mode (via a flag in the seal stanza), we new up a shamir
seal with the recovery unseal key path instead of the auto seal. Then everything proceeds
as if you had a shamir seal to begin with.
* Handle recovery rekeying
* changelog
* Revert go.mod redirect
* revert multi-blob info
* Dumb nil unmarshal target
* More comments
* Update vault/seal.go
Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com>
* Update changelog/18683.txt
Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com>
* pr feedback
* Fix recovery rekey, which needs to fetch root keys and restore them under the new recovery split
* Better comment on recovery seal during adjustSealMigration
* Make it possible to migrate from an auto-seal in recovery mode to shamir
* Fix sealMigrated to account for a recovery seal
* comments
* Update changelog/18683.txt
Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com>
* Address PR feedback
* Refactor duplicated migration code into helpers, using UnsealRecoveryKey/RecoveryKey where appropriate
* Don't shortcut the reast of seal migration
* get rid of redundant transit server cleanup
Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com>
* add core state lockd eadlock detection config option v2
* add changelog
* split out NewTestCluster function to maintain build flag
* replace long func with constant
* remove line
* rename file, and move where detect deadlock flag is set
* Use new parseutil helper: Safe variants
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Update parseutil to v0.1.5
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Fix additional integer overflow in command/server
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* port SSCT OSS
* port header hmac key to ent and generate token proto without make command
* remove extra nil check in request handling
* add changelog
* add comment to router.go
* change test var to use length constants
* remove local index is 0 check and extra defer which can be removed after use of ExternalID
* VAULT-1564 report in-flight requests
* adding a changelog
* Changing some variable names and fixing comments
* minor style change
* adding unauthenticated support for in-flight-req
* adding documentation for the listener.profiling stanza
* adding an atomic counter for the inflight requests
addressing comments
* addressing comments
* logging completed requests
* fixing a test
* providing log_requests_info as a config option to determine at which level requests should be logged
* removing a member and a method from the StatusHeaderResponseWriter struct
* adding api docks
* revert changes in NewHTTPResponseWriter
* Fix logging invalid log_requests_info value
* Addressing comments
* Fixing a test
* use an tomic value for logRequestsInfo, and moving the CreateClientID function to Core
* fixing go.sum
* minor refactoring
* protecting InFlightRequests from data race
* another try on fixing a data race
* another try to fix a data race
* addressing comments
* fixing couple of tests
* changing log_requests_info to log_requests_level
* minor style change
* fixing a test
* removing the lock in InFlightRequests
* use single-argument form for interface assertion
* adding doc for the new configuration paramter
* adding the new doc to the nav data file
* minor fix
- Add the kms_library configuration stanza to Vault's command/server
- Provide validation of keys and general configuration.
- Add initial kms_library configuration documentation
- Attempt at startup to verify we can read the configured HSM Library
- Hook in KmsLibrary config into the Validate to detect typo/unused keys
* Actually call config.Validate in diagnose
* Wire configuration checks into diagnose and fix resulting bugs.
* go mod vendor
* Merge to vendorless version
* Remove sentinel section to allow diagnose_ok to pass
* Fix unit tests
* Expose unknown fields and duplicate sections as diagnose warnings
* section counts not needed, already handled
* Address PR feedback
* Prune more of the new fields before tests call deep.Equals
* Update go.mod
Fix JSON encoding adding newlines.
This manifested itself when encoding config values, which all map to
strings. An extra new line would get added by json.Encode, which caused
other things to break with confusing error messagges. Switching to
json.Marshal seems to solve the problem.
