Commit Graph

90 Commits

Author SHA1 Message Date
Violet Hynes
3a46ecc389
Replace all remaining time.ParseDurations with parseutil.ParseDurationSeconds (#21357)
* Replace all time.ParseDurations with testutil.ParseDurationSeconds

* Changelog

* Import formatting

* Import formatting

* Import formatting

* Import formatting

* Semgrep rule that runs as part of CI
2023-06-20 14:37:46 -04:00
Nick Cabatoff
6b87a087af
Fix a panic I observed in a sealed node running CollectMetrics (#21249) 2023-06-15 15:02:20 +00:00
Victor Rodriguez
2d7efaef97
Convert seal.Access struct into a interface (OSS) (#20510)
* Move seal barrier type field from Access to autoSeal struct.

Remove method Access.SetType(), which was only being used by a single test, and
which can use the name option of NewTestSeal() to specify the type.

* Change method signatures of Access to match those of Wrapper.

* Turn seal.Access struct into an interface.

* Tweak Access implementation.

Change `access` struct to have a field of type wrapping.Wrapper, rather than
extending it.

* Add method Seal.GetShamirWrapper().

Add method Seal.GetShamirWrapper() for use by code that need to perform
Shamir-specific operations.
2023-05-04 14:22:30 -04:00
Hamid Ghaf
e55c18ed12
adding copyright header (#19555)
* adding copyright header

* fix fmt and a test
2023-03-15 09:00:52 -07:00
Josh Black
5b083266ef
Enable undo logs by default (#18692)
* Enable undo logs by default

* add consul test

* update go.mod/sum

* add a better non-existent key
2023-01-17 13:38:18 -08:00
Mike Palmiotto
af04fd08ee
Fix race in fsm.db (#18386)
We need to take a read lock when reading any of the FSM fields. Expose a
new fsm.Stats to handle a racy read and make sure we're consistently using
the f.db read lock wrappers.
2022-12-15 10:04:27 -05:00
Nick Cabatoff
35df9489c7
Move version out of SDK. (#14229)
Move version out of SDK.  For now it's a copy rather than move: the part not addressed by this change is sdk/helper/useragent.String, which we'll want to remove in favour of PluginString.  That will have to wait until we've removed uses of useragent.String from all builtins.
2022-12-07 13:29:51 -05:00
Tom Proctor
60f92bbeef
storage/raft: Add retry_join_as_non_voter config option (#18030) 2022-11-18 17:58:16 +00:00
Nick Cabatoff
ce74f4f1de
Add more raft metrics, emit more metrics on non-perf standbys (#12166)
Add some metrics helpful for monitoring raft cluster state.

Furthermore, we weren't emitting bolt metrics on regular (non-perf) standbys, and there were other metrics
in metricsLoop that would make sense to include in OSS but weren't.  We now have an active-node-only func,
emitMetricsActiveNode.  This runs metricsLoop on the active node.  Standbys and perf-standbys run metricsLoop
from a goroutine managed by the runStandby rungroup.
2022-10-07 09:09:08 -07:00
Josh Black
db71fdb087
only enable undo logs if all cluster members support it (#17378) 2022-10-06 11:24:16 -07:00
Josh Black
b64cae6419
Adjust raft transactions to be safer with get operations (#17151) 2022-09-16 09:35:48 -07:00
Josh Black
c1cf97adac
merkle sync undo logs (#17103) 2022-09-13 10:03:19 -07:00
Scott Miller
0d6a42c79e
OSS portion of wrapper-v2 (#16811)
* OSS portion of wrapper-v2

* Prefetch barrier type to avoid encountering an error in the simple BarrierType() getter

* Rename the OveriddenType to WrapperType and use it for the barrier type prefetch

* Fix unit test
2022-08-23 15:37:16 -04:00
swayne275
82f1216c83
Use %q for quoted strings where appropriate (#15216)
* change '%s' to %q where single vs double quotes shouldn't matter

* replace double quotes with %q in logs and errors
2022-08-03 12:32:45 -06:00
Josh Black
3d4c6ca1f0
autopilot_upgrade_version should be parseable (#15590) 2022-05-25 09:09:45 -07:00
Josh Black
de99f93820
Add autopilot automated upgrades and redundancy zones (#15521) 2022-05-20 16:49:11 -04:00
Nick Cabatoff
dc91661737
Raft: use a larger initial heartbeat/election timeout (#15042) 2022-04-29 08:32:16 -04:00
hghaf099
1740186e72
Fixing excessive unix file permissions (#14791)
* Fixing excessive unix file permissions

* CL

* reduce the permission from 750 to 700
2022-04-01 12:57:38 -04:00
hghaf099
b1012f2e36
VAULT-4240 time.After() in a select statement can lead to memory leak (#14814)
* VAULT-4240 time.After() in a select statement can lead to memory leak

* CL
2022-04-01 10:17:11 -04:00
davidadeleon
07eacd3649
Raft/fix raft telemetry metric unit (#13749)
Converting raft time metrics to Milliseconds over Default Nanoseconds to maintain consistency
2022-01-24 10:51:35 -05:00
Nick Cabatoff
e7120442bf
Use MAP_POPULATE for our bbolt mmaps (#13573)
* Use MAP_POPULATE for our bbolt mmaps, assuming the files fit in memory.  This should improve startup times when freelist sync is disabled.
2022-01-11 08:16:53 -05:00
Jim Kalafut
a72a5ff754
Rename master key to root key (#13324)
* See what it looks like to replace "master key" with "root key".  There are two places that would require more challenging code changes: the storage path `core/master`, and its contents (the JSON-serialized EncodedKeyringtructure.)

* Restore accidentally deleted line

* Add changelog

* Update root->recovery

* Fix test

Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com>
2021-12-06 17:12:20 -08:00
Nick Cabatoff
e30164ac3a
Prevent raft transactions from containing overlarge keys. (#13286) 2021-11-26 08:38:39 -05:00
Nick Cabatoff
859cf45d05
Return an error when trying to store a too-large key with Raft (#13282) 2021-11-25 14:07:03 -05:00
Josh Black
17cb1c601a
Add InitialMmapSize to bolt options (#13178) 2021-11-22 20:16:57 -08:00
Nick Cabatoff
9c15118a66
Check to make sure context isn't expired before doing a raft operation. (#12162) 2021-08-19 12:03:56 -04:00
Jeff Mitchell
861454e0ed
Migrate to sdk/internalshared libs in go-secure-stdlib (#12090)
* Swap sdk/helper libs to go-secure-stdlib

* Migrate to go-secure-stdlib reloadutil

* Migrate to go-secure-stdlib kv-builder

* Migrate to go-secure-stdlib gatedwriter
2021-07-15 20:17:31 -04:00
Hridoy Roy
25346e824e
Diagnose Storage Panic Bugfixes (#11923)
* partial

* fix raft panics and ensure checks are skipped if storage isnt initialized

* cleanup directories

* newline

* typo in nil check

* another nil check
2021-06-24 09:56:38 -07:00
Nick Cabatoff
8096e7d856
Remove fragile link to docs from code. (#11928) 2021-06-23 15:43:44 -04:00
Brian Kassouf
053a7f2fa2
raft: Set BatchApplyCh for more consistent batch sizes (#11907)
* raft: Set BatchApplyCh for more consistent batch sizes

* Add changelog file
2021-06-21 12:00:41 -07:00
Josh Black
625bf466f1
Add new boltdb options (#11895) 2021-06-21 11:35:40 -07:00
Brian Kassouf
43b09f27cc
physical/raft: Add a function that gets the offline, stale configuration (#11821)
* Add a function that gets the offline, stale configuration

* Fix comment
2021-06-11 10:25:02 -07:00
Lars Lehtonen
3857cdbeaf
physical: deprecate errwrap.Wrapf() (#11692) 2021-05-31 12:54:05 -04:00
Josh Black
b6cf9836e0
Switch to shared raft-boltdb library and add metrics (#11269) 2021-04-26 16:01:26 -07:00
Brian Kassouf
a24653cc5c
Run a more strict formatter over the code (#11312)
* Update tooling

* Run gofumpt

* go mod vendor
2021-04-08 09:43:39 -07:00
Hridoy Roy
2fd30841cd
Initial Diagnose Command for TLS and Listener [VAULT-1896, VAULT-1899] (#11249)
* sanity checks for tls config in diagnose

* backup

* backup

* backup

* added necessary tests

* remove comment

* remove parallels causing test flakiness

* comments

* small fix

* separate out config hcl test case into new hcl file

* newline

* addressed comments

* addressed comments

* addressed comments

* addressed comments

* addressed comments

* reload funcs should be allowed to be nil
2021-04-06 16:40:43 -07:00
Nick Cabatoff
8675d59010
Expose snapshot_interval tunable instead of setting it in prod code for the sake of a test. (#11160) 2021-03-19 15:41:42 -04:00
Brian Kassouf
5ff9af3a17
storage/raft: Ensure peers are informed of their correct suffrage when added with AutoPilot (#11155)
* storage/raft: Ensure peers are informed of their correct suffrage when added with AutoPilot

* Add test ensuring peer sets are equivalent
2021-03-19 11:53:50 -07:00
Nick Cabatoff
a880df97ba
Add a test for server stabilization (#11128) 2021-03-17 17:23:13 -04:00
Vishal Nayak
794000c821
Fix autopilot fsm race (#11091)
* Fix autopilot fsm race

* No need to grab backend's lock
2021-03-11 13:14:11 -05:00
Vishal Nayak
415890e79c
Autopilot: Server Stabilization, State and Dead Server Cleanup (#10856)
* k8s doc: update for 0.9.1 and 0.8.0 releases (#10825)

* k8s doc: update for 0.9.1 and 0.8.0 releases

* Update website/content/docs/platform/k8s/helm/configuration.mdx

Co-authored-by: Theron Voran <tvoran@users.noreply.github.com>

Co-authored-by: Theron Voran <tvoran@users.noreply.github.com>

* Autopilot initial commit

* Move autopilot related backend implementations to its own file

* Abstract promoter creation

* Add nil check for health

* Add server state oss no-ops

* Config ext stub for oss

* Make way for non-voters

* s/health/state

* s/ReadReplica/NonVoter

* Add synopsis and description

* Remove struct tags from AutopilotConfig

* Use var for config storage path

* Handle nin-config when reading

* Enable testing autopilot by using inmem cluster

* First passing test

* Only report the server as known if it is present in raft config

* Autopilot defaults to on for all existing and new clusters

* Add locking to some functions

* Persist initial config

* Clarify the command usage doc

* Add health metric for each node

* Fix audit logging issue

* Don't set DisablePerformanceStandby to true in test

* Use node id label for health metric

* Log updates to autopilot config

* Less aggressively consume config loading failures

* Return a mutable config

* Return early from known servers if raft config is unable to be pulled

* Update metrics name

* Reduce log level for potentially noisy log

* Add knob to disable autopilot

* Don't persist if default config is in use

* Autopilot: Dead server cleanup (#10857)

* Dead server cleanup

* Initialize channel in any case

* Fix a bunch of tests

* Fix panic

* Add follower locking in heartbeat tracker

* Add LastContactFailureThreshold to config

* Add log when marking node as dead

* Update follower state locking in heartbeat tracker

* Avoid follower states being nil

* Pull test to its own file

* Add execution status to state response

* Optionally enable autopilot in some tests

* Updates

* Added API function to fetch autopilot configuration

* Add test for default autopilot configuration

* Configuration tests

* Add State API test

* Update test

* Added TestClusterOptions.PhysicalFactoryConfig

* Update locking

* Adjust locking in heartbeat tracker

* s/last_contact_failure_threshold/left_server_last_contact_threshold

* Add disabling autopilot as a core config option

* Disable autopilot in some tests

* s/left_server_last_contact_threshold/dead_server_last_contact_threshold

* Set the lastheartbeat of followers to now when setting up active node

* Don't use config defaults from CLI command

* Remove config file support

* Remove HCL test as well

* Persist only supplied config; merge supplied config with default to operate

* Use pointer to structs for storing follower information

* Test update

* Retrieve non voter status from configbucket and set it up when a node comes up

* Manage desired suffrage

* Consider bucket being created already

* Move desired suffrage to its own entry

* s/DesiredSuffrageKey/LocalNodeConfigKey

* s/witnessSuffrage/recordSuffrage

* Fix test compilation

* Handle local node config post a snapshot install

* Commit to storage first; then record suffrage in fsm

* No need of local node config being nili case, post snapshot restore

* Reconcile autopilot config when a new leader takes over duty

* Grab fsm lock when recording suffrage

* s/Suffrage/DesiredSuffrage in FollowerState

* Instantiate autopilot only in leader

* Default to old ways in more scenarios

* Make API gracefully handle 404

* Address some feedback

* Make IsDead an atomic.Value

* Simplify follower hearbeat tracking

* Use uber.atomic

* Don't have multiple causes for having autopilot disabled

* Don't remove node from follower states if we fail to remove the dead server

* Autopilot server removals map (#11019)

* Don't remove node from follower states if we fail to remove the dead server

* Use map to track dead server removals

* Use lock and map

* Use delegate lock

* Adjust when to remove entry from map

* Only hold the lock while accessing map

* Fix race

* Don't set default min_quorum

* Fix test

* Ensure follower states is not nil before starting autopilot

* Fix race

Co-authored-by: Jason O'Donnell <2160810+jasonodonnell@users.noreply.github.com>
Co-authored-by: Theron Voran <tvoran@users.noreply.github.com>
2021-03-03 13:59:50 -05:00
Nick Cabatoff
22b486b651
OSS parts of the new client controlled consistency feature (#10974) 2021-02-24 06:58:10 -05:00
Vishal Nayak
405eced084
Revert "Read-replica instead of non-voter (#10875)" (#10890)
This reverts commit fc745670cf.
2021-02-10 16:41:58 -05:00
Vishal Nayak
fc745670cf
Read-replica instead of non-voter (#10875) 2021-02-10 09:58:18 -05:00
Brian Kassouf
ab471a06e8
storage/raft: Fix panic when no Join TLS info is being used (#10801) 2021-01-27 17:21:45 -08:00
Nick Cabatoff
cac2e00f2f
Add configuration to specify a TLS ServerName to use in the TLS handshake when performing a raft join. (#10698) 2021-01-19 17:54:28 -05:00
Aleksandr Bezobchuk
e4421691da
Merge PR #10192: Auto-Join: Configurable Scheme & Port (and add k8s provider) 2020-10-23 16:13:09 -04:00
Nick Cabatoff
96313ba114
OSS changes for enterprise automated snapshots (#10160) 2020-10-16 14:57:11 -04:00
Aleksandr Bezobchuk
b67da26279
Merge PR #10095: Integrated Storage Cloud Auto-Join 2020-10-13 16:26:39 -04:00
ncabatoff
066678f5f0
Logging tweaks, mostly for tests (#10028)
Fix some places where raft wasn't hooking into the core logger as it should.
Revisited the code that was setting the log level to Error during cleanup: it's normal for there to be a bunch of errors then, which makes it harder to see what went wrong up to the point where the test was deemed to have failed.  So now, instead of setting log level to Error, we actually stop logging altogether.  This only applies if the test didn't pass in its own logger during cluster creation, but we should be moving away from that anyway.
2020-09-23 13:40:00 -04:00