* Add a configuration flag for enabling multiseal (Seal HA), CE side
* imports
* no quotes
* get rid of dep on ent config
* Abstract enableMultiSeal for a build time switch
* license headers
* wip
* gate physical seal gen fetch by a param
* docs tweak, remove core flag
* updates from the ent pr
* update stub
* update test fixtures for enable_multiseal
* use accessor
* add a test fixture for non-multiseal diagnose
* remove debugging crtuch
* Do handle phys seal gen info even if multiseal is off, in order to facilitate enable/disable safeties
* more enabled flag handling
* Accept seal gen info if we were previously disabled, and persist it
* update unit test
* Validation happens postUnseal, so this test is invalid
* Dont continue setting conf if seal loading fails during SIGHUP
* Update website/content/docs/configuration/seal/seal-ha.mdx
Thanks, that does sound much clearer
Co-authored-by: Jason O'Donnell <2160810+jasonodonnell@users.noreply.github.com>
* use validation if previous gen was enabled
* unit test update
* stub SetMultisealEnabled
* bring over more changes from ent
* this was an unfix
---------
Co-authored-by: Jason O'Donnell <2160810+jasonodonnell@users.noreply.github.com>
* add gosimport to make fmt and run it
* move installation to tools.sh
* correct weird spacing issue
* Update Makefile
Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com>
* fix a weird issue
---------
Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com>
@mitchellh suggested we fork `cli` and switch to that.
Since we primarily use the interfaces in `cli`, and the new
fork has not changed those, this is (mostly) a drop-in replacement.
A small fix will be necessary for Vault Enterprise, I believe.
* Seal HA: Use new SealWrappedValue type to abstract seal wrapped values
Introduce SealWrappedValue to abstract seal wrapped values.
Make SealWrappedValue capable of marshalling into a BlobInfo, when there is
plaintext or a single encryption, or to a custom serialization consisting of a
header, length and a marshalled MultiWrapValue protobuf.
* Vault-13769: Support configuring and using multiple seals for unsealing
* Make sealWrapBackend start using multiple seals
* Make seal.Access no longer implement wrapping.Wrapper.
Instead, add the Encrypt and Decrypt methods to the Access interface.
* Make raft snapshot system use funcs SealWrapValue + UnsealWrapValue.
Move the snapshot.Sealer implementation to the vault package to
avoid circular imports.
* Update sealWrapBackend to use multiple seals for encryption.
Use all the encryption wrappers when storing seal wrapped values.
Try do decrypt using the highest priority wrapper, but try all
combinations of encrypted values and wrappers if necessary.
* Allow the use of multiple seals for entropy augmentation
Add seal_name variable in entropy stanza
Add new MultiSourcer to accommodate the new entropy augmentation behavior.
* Individually health check each wrapper, and add a sys/seal-backend-status endpoint.
* Address a race, and also a failed test mock that I didn't catch
* Track partial wrapping failures...
... where one or more but not all access.Encrypts fail for a given write.
Note these failures by adding a time ordered UUID storage entry containing
the path in a special subdirectory of root storage. Adds a callback
pattern to accomplish this, with certain high value writes like initial
barrier key storage not allowing a partial failure. The followup work
would be to detect return to health and iterate through these storage
entries, rewrapping.
* Add new data structure to track seal config generation (#4492)
* Add new data structure to track seal config generation
* Remove import cycle
* Fix undefined variable errors
* update comment
* Update setSeal response
* Fix setSealResponse in operator_diagnose
* Scope the wrapper health check locks individually (#4491)
* Refactor setSeal function in server.go. (#4505)
Refactor setSeal function in server.go.
* Decouple CreateSecureRandomReaderFunc from seal package.
Instead of using a list of seal.SealInfo structs, make
CreateSecureRandomReaderFunc use a list of new EntropySourcerInfo structs. This
brakes the denpency of package configutil on the seal package.
* Move SealGenerationInfo tracking to the seal Access.
* Move SealGenerationInfo tracking to the seal Access.
The SealGenerationInfo is now kept track by a Seal's Access instead of by the
Config object. The access implementation now records the correct generation
number on seal wrapped values.
* Only store and read SealGenerationInfo if VAULT_ENABLE_SEAL_HA_BETA is true.
* Add MultiWrapValue protobuf message
MultiWrapValue can be used to keep track of different encryptions of a value.
---------
Co-authored-by: Victor Rodriguez <vrizo@hashicorp.com>
* Use generation to determine if a seal wrapped value is up-to-date. (#4542)
* Add logging to seal Access implementation.
* Seal HA buf format run (#4561)
* Run buf format.
* Add buf.lock to ensure go-kms-wrapping module is imported.
* Vault-18958: Add unit tests for config checks
* Add safety logic for seal configuration changes
* Revert "Add safety logic for seal configuration changes"
This reverts commit 7fec48035a5cf274e5a4d98901716d08d766ce90.
* changes and tests for checking seal config
* add ent tests
* remove check for empty name and add type into test cases
* add error message for empty name
* fix no seals test
---------
Co-authored-by: divyapola5 <divya@hashicorp.com>
* Handle migrations between single-wrapper and multi-wrapper autoSeals
* Extract method SetPhysicalSealConfig.
* Extract function physicalSealConfig.
The extracted function is the only code now reading SealConfig entries from
storage.
* Extract function setPhysicalSealConfig.
The extracted function is the only code now writing SealConfig entries from
storage (except for migration from the old recovery config path).
* Move SealConfig to new file vault/seal_config.go.
* Add SealConfigType quasy-enumeration.
SealConfigType is to serve as the typed values for field SealConfig.Type.
* Rename Seal.RecoveryType to RecoverySealConfigType.
Make RecoverySealConfigType return a SealConfigType instead of a string.
* Rename Seal.BarrierType to BarrierSealConfigType.
Make BarrierSealConfigType return a SealConfigType.
Remove seal.SealType (really a two-step rename to SealConfigType).
* Add Seal methods ClearBarrierConfig and ClearRecoveryConfig.
* Handle autoseal <-> multiseal migrations.
While going between single-wrapper and multiple-wrapper autoseals are not
migrations that require an unwrap seal (such as going from shamir to autoseal),
the stored "barrier" SealConfig needs to be updated in these cases.
Specifically, the value of SealConfg.Type is "multiseal" for autoSeals that have
more than one wrapper; on the other hand, for autoseals with a single wrapper,
SealConfig.Type is the type of the wrapper.
* Remove error return value from NewAutoSeal constructor.
* Automatically rewrap partially seal wrapped values on an interval
* Add in rewrapping of partially wrapped values on an interval, regardless of seal health/status.
* Don't set SealGenerationInfo Rewrapped flag in the partial rewrap call.
* Unexport the SealGenerationInfo's Rewrapped field, add a mutex to it for thread safe access, and add accessor methods for it.
* Add a success callback to the manual seal rewrap process that updates the SealGenerationInfo's rewrapped field. This is done via a callback to avoid an import cycle in the SealRewrap code.
* Fix a failing seal wrap backend test which was broken by the unexporting of SealGenerationInfo's Rewrapped field.
* Nil check the seal rewrap success callback before calling it.
* Change SealGenerationInfo rewrapped parameter to an atomic.Bool rather than a sync.RWMutex for simplicity and performance.
* Add nil check for SealAccess before updating SealGenerationInfo rewrapped status during seal rewrap call.
* Update partial rewrap check interval from 10 seconds to 1 minute.
* Update a reference to SealGenerationInfo Rewrapped field to use new getter method.
* Fix up some data raciness in partial rewrapping.
* Account for possibly nil storage entry when retrieving partially wrapped value.
* Allow multi-wrapper autoSeals to include disabled seal wrappers.
* Restore propagation of wrapper configuration errors by setSeal.
Function setSeal is meant to propagate non KeyNotFound errors returned by calls
to configutil.ConfigureWrapper.
* Remove unused Access methods SetConfig and Type.
* Allow multi-wrapper autoSeals to include disabled seal wrappers.
Make it possible for an autoSeal that uses multiple wrappers to include disabled
wrappers that can be used to decrypt entries, but are skipped for encryption.
e an unwrapSeal when there are disabled seals.
* Fix bug with not providing name (#4580)
* add suffix to name defaults
* add comment
* only change name for disabled seal
* Only attempt to rewrap partial values when all seals are healthy.
* Only attempt to rewrap partial values when all seals are healthy.
* Change logging level from info to debug for notice about rewrap skipping based on seal health.
* Remove stale TODOs and commented out code.
---------
Co-authored-by: rculpepper <rculpepper@hashicorp.com>
Co-authored-by: Larroyo <95649169+DeLuci@users.noreply.github.com>
Co-authored-by: Scott G. Miller <smiller@hashicorp.com>
Co-authored-by: Divya Pola <87338962+divyapola5@users.noreply.github.com>
Co-authored-by: Matt Schultz <matt.schultz@hashicorp.com>
Co-authored-by: divyapola5 <divya@hashicorp.com>
Co-authored-by: Rachel Culpepper <84159930+rculpepper@users.noreply.github.com>
* Adding explicit MPL license for sub-package.
This directory and its subdirectories (packages) contain files licensed with the MPLv2 `LICENSE` file in this directory and are intentionally licensed separately from the BSL `LICENSE` file at the root of this repository.
* Adding explicit MPL license for sub-package.
This directory and its subdirectories (packages) contain files licensed with the MPLv2 `LICENSE` file in this directory and are intentionally licensed separately from the BSL `LICENSE` file at the root of this repository.
* Updating the license from MPL to Business Source License.
Going forward, this project will be licensed under the Business Source License v1.1. Please see our blog post for more details at https://hashi.co/bsl-blog, FAQ at www.hashicorp.com/licensing-faq, and details of the license at www.hashicorp.com/bsl.
* add missing license headers
* Update copyright file headers to BUS-1.1
* Fix test that expected exact offset on hcl file
---------
Co-authored-by: hashicorp-copywrite[bot] <110428419+hashicorp-copywrite[bot]@users.noreply.github.com>
Co-authored-by: Sarah Thompson <sthompson@hashicorp.com>
Co-authored-by: Brian Kassouf <bkassouf@hashicorp.com>
Move version out of SDK. For now it's a copy rather than move: the part not addressed by this change is sdk/helper/useragent.String, which we'll want to remove in favour of PluginString. That will have to wait until we've removed uses of useragent.String from all builtins.
This fixes a couple of references to loop variables in parallel tests
and deferred functions. When running a parallel test (calling
`t.Parallel()`) combined with the table-driven pattern, it's necessary
to copy the test case loop variable, otherwise only the last test case
is exercised. This is documented in the `testing` package:
https://pkg.go.dev/testing#hdr-Subtests_and_Sub_benchmarks
`defer` statements that invoke a closure should also not reference a
loop variable directly as the referenced value will change in each
iteration of the loop.
Issues were automatically found with the `loopvarcapture` linter.
* HCP link integration
* update configure-git.yml
* more OSS stuff
* removing internal repos
* adding a nil check
* removing config test to be included in ENT only
* updating hcp-sdk-go to v0.22.0
* remove Hostname and AuthURL link config params
Co-authored-by: Chris Capurso <1036769+ccapurso@users.noreply.github.com>
* OSS portion of wrapper-v2
* Prefetch barrier type to avoid encountering an error in the simple BarrierType() getter
* Rename the OveriddenType to WrapperType and use it for the barrier type prefetch
* Fix unit test
* save
* save
* save
* first round of the diagnose language pass
* capitalization
* first round of feedback
* fix bug in advise
* a few more nouns to verbs
* diagnose: Add seal transit tls check
* Fixing the path to the config file and the path to the cert files
* Addressing comment
* Addressing seal transit tls check comments
* tls verification bugfix
* tls verification bugfix
* allow diagnose fail to report status when there are also warnings
* allow diagnose fail to report status when there are also warnings
* Update vault/diagnose/helpers_test.go
Co-authored-by: swayne275 <swayne275@gmail.com>
* comments
Co-authored-by: swayne275 <swayne275@gmail.com>
* Fix diagnose panic when configuration file does not exist
* Addressing comments
* Update command/operator_diagnose.go
Co-authored-by: Hridoy Roy <roy@hashicorp.com>
Co-authored-by: Hridoy Roy <roy@hashicorp.com>
* Actually call config.Validate in diagnose
* Wire configuration checks into diagnose and fix resulting bugs.
* go mod vendor
* Merge to vendorless version
* Remove sentinel section to allow diagnose_ok to pass
* Fix unit tests
* raft file and quorum checks
* raft checks
* backup
* raft file checks test
* address comments and add more raft and file and process checks
* syntax issues
* modularize functions to compile differently on different os
* compile raft checks everywhere
* more build tag issues
* raft-diagnose
* correct file permission checks
* upgrade tests and add a getConfigOffline test that currently does not work
* comment
* update file checks method signature on windows
* Update physical/raft/raft_test.go
Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
* raft tests
* add todo comment for windows root ownership
* voter count message
* raft checks test fixes
Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
* initial refactoring of unseal step in run
* remove waitgroup
* remove waitgroup
* backup work
* backup
* backup
* completely modularize run and move into diagnose
* add diagnose errors for incorrect number of unseal keys
* comment tests back in
* backup
* first subspan
* finished subspanning but running into error with timeouts
* remove runtime checks
* merge main branch
* meeting updates
* remove telemetry block
* roy comment
* subspans for seal finalization and wrapping diagnose latency checks
* backup while I fix something else
* fix storage latency test errors
* runtime checks
* diagnose with timeout on seal
* wip
* wip
* Finish implementing advice handling and word wrapping
* Properly word wrap messages and warnings
* Remove debugging
* Remove debugging
* Remove unnecessary test
* unit test bug
* go vendor
This allows operators to run diagnose in scripts and detect the difference between success, warning, and failure.
Exit codes are now:
0: Success (no warnings)
1: Failure (some test failed)
2: Warning (some test warned)
3: User input failure such as a bad flag
4: Other error
* Segment out disk checks to disable on openbsd/arm
Also add a spot skipped helper.
* Expected results may be fewer than actual because of variable length tests like disk usage
* Move to os_common and build on windows
* Add ulimit check, and tidy unit test cases to avoid needing to have all results and perfect ordering
* Make order independent check recursive
* Fix unit tests
* Try a 5s request timeout
* Disk usage checks
* Move disk free earlier
* Move logic to helpers
* Bring over test logic from the ulimit PR
* imports
* Report error
* Get unit tests working
* initial refactoring of unseal step in run
* remove waitgroup
* remove waitgroup
* backup work
* backup
* backup
* completely modularize run and move into diagnose
* add diagnose errors for incorrect number of unseal keys
* comment tests back in
* backup
* first subspan
* finished subspanning but running into error with timeouts
* remove runtime checks
* meeting updates
* remove telemetry block
* roy comment
* subspans for seal finalization and wrapping diagnose latency checks
* fix storage latency test errors
* review comments
* use random uuid for latency checks instead of static id
* Create helpers which integrate with OpenTelemetry for diagnose collection
* Go mod vendor
* Comments
* Update vault/diagnose/helpers.go
Co-authored-by: swayne275 <swayne275@gmail.com>
* Add unit test/example
* tweak output
* More comments
* add spot check concept
* Get unit tests working on Result structs
* wip
* Fix unit test
* Get unit tests working, and make diagnose sessions local rather than global
* Comments
* Last comments
* No need for init
* :|
* Fix helpers_test
* wip
* wip
* wip
* Revendor otel
* Fix merge related problems
* imports
* Fix unit tests
Co-authored-by: swayne275 <swayne275@gmail.com>
* Add infrastructure for skipping tests
* Add infrastructure for skipping tests
* Set it
* Update vault/diagnose/helpers.go
Co-authored-by: swayne275 <swayne275@gmail.com>
* Implement type alias for test functions
Co-authored-by: swayne275 <swayne275@gmail.com>
* Create helpers which integrate with OpenTelemetry for diagnose collection
* Go mod vendor
* consul tls checks
* draft for storage end to end check
* Comments
* Update vault/diagnose/helpers.go
Co-authored-by: swayne275 <swayne275@gmail.com>
* Add unit test/example
* tweak output
* More comments
* add spot check concept
* Get unit tests working on Result structs
* Fix unit test
* Get unit tests working, and make diagnose sessions local rather than global
* Comments
* Last comments
* No need for init
* :|
* Fix helpers_test
* cleaned up chan logic. Tests next.
* fix tests
* remove a comment
* tests
* remove a comment
* run direct access checks in diagnose command
* review comments
Co-authored-by: Scott G. Miller <smiller@hashicorp.com>
Co-authored-by: swayne275 <swayne275@gmail.com>
* Create helpers which integrate with OpenTelemetry for diagnose collection
* Go mod vendor
* consul tls checks
* draft for storage end to end check
* Comments
* Update vault/diagnose/helpers.go
Co-authored-by: swayne275 <swayne275@gmail.com>
* Add unit test/example
* tweak output
* More comments
* add spot check concept
* Get unit tests working on Result structs
* Fix unit test
* Get unit tests working, and make diagnose sessions local rather than global
* Comments
* Last comments
* No need for init
* :|
* Fix helpers_test
* cleaned up chan logic. Tests next.
* fix tests
* remove a comment
* tests
* remove a comment
* cosmetic changes
Co-authored-by: Scott G. Miller <smiller@hashicorp.com>
Co-authored-by: swayne275 <swayne275@gmail.com>
