Commit Graph

1453 Commits

Author SHA1 Message Date
ncabatoff
60c7e22ddf
Extend agent template tests to also validate that updated templates get re-rendered. (#9097) 2020-05-29 13:36:59 -04:00
Thomas L. Kula
f31092e7ad
Allow auto_auth with templates without specifying a sink (#8812)
For situations where you want the Vault agent to handle one or more templates but do not require the acquired credentials elsewhere.

Modify the logic in SyncServer so that if there are no sinks, ignore any new credentials. Since SyncServer is responsible for shutting down the agent, make sure it still properly shuts down in this new situation.

Solves #7988
2020-05-26 13:52:14 -04:00
Alexander Bezobchuk
f425d5ba37
Merge PR #9078: Add go version to server message output 2020-05-26 12:28:51 -04:00
Alexander Bezobchuk
2e1f75a26e
Merge PR #9060: Support reading Raft TLS flags from file 2020-05-23 11:09:55 -04:00
Josh Black
af5338b485
Add a new "vault monitor" command (#8477)
Add a new "vault monitor" command

Co-authored-by: ncabatoff <ncabatoff@hashicorp.com>
Co-authored-by: Calvin Leung Huang <cleung2010@gmail.com>
Co-authored-by: Jeff Mitchell <jeffrey.mitchell@gmail.com>
2020-05-21 13:07:50 -07:00
Jason O'Donnell
dbaa9ec65f
agent/template: update consul-template dep (#9044) 2020-05-20 13:03:33 -04:00
Josh Black
f0c8b4c7a9
Fix JSON encoding adding newlines. (#8928)
Fix JSON encoding adding newlines.

This manifested itself when encoding config values, which all map to
strings. An extra new line would get added by json.Encode, which caused
other things to break with confusing error messagges. Switching to
json.Marshal seems to solve the problem.
2020-05-19 16:13:05 -07:00
ncabatoff
e52f34772a
Don't include username or password of proxy env vars when logging them. (#9022) 2020-05-19 10:07:46 -04:00
Brian Kassouf
b8799b3358
storage/raft: Add committed and applied indexes to the status output (#9011)
* storage/raft: Add committed and applied indexes to the status output

* Update api vendor

* changelog++

* Update http/sys_leader.go

Co-authored-by: Jim Kalafut <jkalafut@hashicorp.com>

Co-authored-by: Jim Kalafut <jkalafut@hashicorp.com>
2020-05-18 16:07:27 -07:00
Calvin Leung Huang
5894f65138
Refactor service registration (#8976)
* serivceregistration: refactor service registration logic to run later

* move state check to the internal func

* sr/kubernetes: update setInitialStateInternal godoc

* sr/kubernetes: remove return in setInitialState

* core/test: fix mockServiceRegistration

* address review feedback
2020-05-15 11:06:58 -07:00
Jeff Mitchell
91b09c09b5
Create configutil and move some common config and setup functions there (#8362) 2020-05-14 09:19:27 -04:00
Mark Gritter
d5b1d5d3a5
Metrics wrapper that adds the cluster name as a label. (#8961) 2020-05-12 21:00:59 -05:00
Vladimir Dimitrov
52cadd32d9
Added plugin reload function to api (#8777)
* Added plugin reload function to api

* Apply suggestions from code review

Co-Authored-By: Calvin Leung Huang <cleung2010@gmail.com>

* go mod vendor

* addressing comments

* addressing comments

* add docs

Co-authored-by: Calvin Leung Huang <cleung2010@gmail.com>
2020-05-04 11:14:23 -07:00
ncabatoff
124572bf80
Ensure that the .vault-token file writen by vault login always has the correct permissions and ownership. (#8867) 2020-04-27 19:55:13 -04:00
Calvin Leung Huang
e23cacb33a
seal/gcpkms: fix panic that occurs when the seal config map is nil (#8840) 2020-04-24 13:31:03 -07:00
Calvin Leung Huang
6c9d585fe2
http: ensure return after writing response by respondError (#8796) 2020-04-21 15:30:36 -07:00
Calvin Leung Huang
ed5e1218b9
raft: check for nil on concrete type in SetupCluster (#8784)
* raft: check for nil on concrete type in SetupCluster

* raft: move check to its own func

* raft: func cleanup

* raft: disallow disable_clustering = true when raft storage is used

* docs: update disable_clustering to mention new behavior
2020-04-21 13:45:07 -07:00
Mike Jarmy
1530c05430
Enable transit->shamir seal migration in Enterprise (#8737)
* Enable transit->shamir seal migration in Enterprise

* prove that we can stop the transit cluster after migration is complete
2020-04-14 10:36:46 -04:00
Vishal Nayak
bb4b2b024e
Fix segfault when seal block is removed (#8517) 2020-04-13 22:54:21 -04:00
Becca Petrin
1a16c61ed0
minor docs updates (#8725) 2020-04-13 10:03:05 -07:00
Vishal Nayak
4ee817b474
Support unwrapping tokens that does not contain data (#8714)
* Support unwrapping tokens that does not contain data

* s/token/secret
2020-04-09 14:09:23 -04:00
Calvin Leung Huang
78189b801d
raft: disallow ha_storage stanza when raft storage is used (#8707) 2020-04-08 13:07:42 -07:00
Calvin Leung Huang
ba1871d48e
test: port test changes to oss (#8694) 2020-04-07 11:26:51 -07:00
Mike Jarmy
bf7bcc1ad8
Expand seal migration tests (#8658)
* rename some seal migration tests

* add comments to seal migration tests

* add comments to seal migration tests

* rename sealhelper

* creat TestSealMigration_ShamirToTestSeal

* fix typo in test suite

* create TestSealMigration_TransitToTestSeal
2020-04-01 13:21:32 -04:00
Becca Petrin
7cb59dc326
allow delete-version-after to be reset to 0 (#8635) 2020-03-30 16:19:44 -07:00
Vishal Nayak
3dd3b20619
Raft cli docs (#8548)
* Raft cli docs

* s/raft/Raft

* s/raft/Raft

* Update website/pages/docs/commands/operator/raft.mdx

Co-Authored-By: Meggie <m.ladlow@gmail.com>

* Mention that shamir seal requires unseal keys

Co-authored-by: Meggie <m.ladlow@gmail.com>
2020-03-17 09:46:44 -04:00
Tommy Murphy
45f488bb4a
stackdriver: metric label extraction (#8073)
* stackdriver: use label extraction and add debug config

* go.mod: update go-metrics-stackdriver

* vendor go-metrics-stackdriver
2020-03-13 07:58:45 +01:00
Jeff Mitchell
c53e0c8ebf Remove misleading comment 2020-03-11 09:04:45 -04:00
Brian Kassouf
ccb0dd7c05
Rename raft configuration command to list-peers and make output easier to read (#8484)
* Make the output of raft configuration easier to read

* Rename raft configuration sub command to list-peers

* Update command/operator_raft_listpeers.go

Co-Authored-By: Calvin Leung Huang <cleung2010@gmail.com>

Co-authored-by: Calvin Leung Huang <cleung2010@gmail.com>
2020-03-06 14:40:50 -08:00
Brian Kassouf
05a754241d
Fix migrating away from raft storage (#8485) 2020-03-06 09:35:17 -08:00
ncabatoff
1ff460597f
Use public seal helper to create a transit server to handle autounseal. (#8395) 2020-02-20 10:05:23 -05:00
Jeff Mitchell
468ed7043a Move kv-builder to internalshared 2020-02-19 11:13:13 -05:00
Jeff Mitchell
5350e670d2 Bump API/SDK and adapt to move from SDK stuff 2020-02-15 14:58:05 -05:00
Gerardo Di Giacomo
0e8c6c2171
enabling TLS 1.3 support for TCP listeners (#8305)
* adding support for TLS 1.3 for TCP listeners

* removed test as CI uses go 1.12

* removed Cassandra support, added deprecation notice

* re-added TestTCPListener_tls13
2020-02-15 11:40:18 -08:00
Jason O'Donnell
7d550e24da
Add OpenLDAP Secret Plugin (#8360)
* Add openldap secret plugin

* go mod vendor

* Revert to go-ldap 3.1.3

* go mod vendor
2020-02-15 13:21:07 -05:00
Calvin Leung Huang
c73a776ba2
command/generate-root: support returning json format on decode and generate otp (#7870) 2020-02-14 18:26:49 -08:00
Joe Dollard
e1c0a1c959
provide vault server flag to exit on core shutdown (#7561)
* provide vault server flag to exit on core shutdown

* Update command/server.go

Co-Authored-By: Jeff Mitchell <jeffrey.mitchell@gmail.com>

Co-authored-by: Jeff Mitchell <jeffrey.mitchell@gmail.com>
Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
2020-02-14 18:07:31 -08:00
Jeff Mitchell
4bf92f2617
Switch output loggers in hclog instead of going through gatedlogger (#8228)
After flushing we no longer have to take a mutex hit for every log line.
2020-02-14 17:50:47 -08:00
Brian Kassouf
317bf94c7a
Port some changes back to OSS (#8359) 2020-02-14 16:39:13 -08:00
Clint
f0e4c56ed0
Docs enforce autoauth token (#8270)
* rename UseAutoAuthForce to ForceAutoAuth, because I think it reads better

* Document 'ForceAuthAuthToken' option for Agent Cache

* Update website/pages/docs/agent/caching/index.mdx

Co-Authored-By: Jim Kalafut <jkalafut@hashicorp.com>

* Add additional tests around use_auto_auth=force and add documentation

* remove note, it's no longer correct

Co-authored-by: Jim Kalafut <jim@kalafut.net>
2020-02-14 15:48:12 -06:00
Clint
e87f01845c
Guard against using Raft as a seperate HA Storage (#8239)
* Guard against using Raft as a seperate HA Storage

* Document that Raft cannot be used as a seperate ha_storage backend at this time

* remove duplicate imports from updating with master
2020-02-14 14:25:53 -06:00
Vishal Nayak
9f980ade31
Seal migration with Raft (#8103)
* Seal migration after unsealing

* Refactor migration fields migrationInformation in core

* Perform seal migration as part of postUnseal

* Remove the sleep logic

* Use proper seal in the unseal function

* Fix migration from Auto to Shamir

* Fix the recovery config missing issue

* Address the non-ha migration case

* Fix the multi cluster case

* Avoid re-running seal migration

* Run the post migration code in new leaders

* Fix the issue of wrong recovery being set

* Address review feedback

* Add more complete testing coverage for seal migrations.   (#8247)

* Add more complete testing coverage for seal migrations.  Also remove VAULT_ACC gate from some tests that just depend on docker, cleanup dangling recovery config in storage after migration, and fix a call in adjustCoreForSealMigration that seems broken.

* Fix the issue of wrong recovery key being set

* Adapt tests to work with multiple cores.

* Add missing line to disable raft join.

Co-authored-by: Vishal Nayak <vishalnayak@users.noreply.github.com>

* Fix all known issues

* Remove warning

* Review feedback.

* Revert my previous change that broke raft tests.  We'll need to come back and at least comment
this once we better understand why it's needed.

* Don't allow migration between same types for now

* Disable auto to auto tests for now since it uses migration between same types which is not allowed

* Update vault/core.go

Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>

* Add migration logs

* Address review comments

* Add the recovery config check back

* Skip a few steps if migration is already done

* Return from waitForLeadership if migration fails

Co-authored-by: ncabatoff <nick.cabatoff@gmail.com>
Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
2020-02-13 16:27:31 -05:00
Becca Petrin
0937a58ad7
Add Kubernetes service registration (#8249) 2020-02-13 09:56:29 -08:00
Jeff Malnick
942dd1ef9e
Add redshift database plugin (#8299)
* feat: add redshift database plugin

* build: update vendored libraries

* docs: add reference doc for redshift variant of the database secrets engine

* feat: set middlewear type name for better metrics naming (#8346)

Co-authored-by: Becca Petrin <beccapetrin@gmail.com>
2020-02-13 09:42:30 -08:00
Jeff Mitchell
6ca61fa265 Bump api/sdk and fix imports 2020-02-13 10:41:16 -05:00
ncabatoff
4d7bfb4927
Add support for telemetry option metrics_prefix (#8340)
When set, instead of starting with `vault_`, metrics will start with `${prefix}_`.
2020-02-13 09:00:32 -05:00
Daniel Spangenberg
75461a6522
Fix CLI namespace commands (#8315)
* Fix namespace commands help

* Fix useless prediction for namespace commands

* Add namespace prediction capability
2020-02-09 19:38:39 +01:00
Jim Kalafut
264c446de6
Bundle MongoDB Atlas (#8309) 2020-02-07 14:09:39 -08:00
Daniel Spangenberg
af3b19099f
Support CLI autocompletion for nested mounts (#8303)
* Support CLI autocompletion for nested mounts

* Add test for nested autocomplete prediction
2020-02-07 15:58:30 +01:00
Becca Petrin
b788b223d7
update from github.com/hashicorp/gokrb5 to github.com/jcmturner/gokrb5/v8 (#8296) 2020-02-05 14:23:22 -08:00
Alex Antonov
963e71c33e
Added flag to disable X-Vault-Token header proxy if client passes the token (#8101)
* Added flag to disable X-Vault-Token header proxy if client passes the token

* Reveresed the flag value to better match the name intent

* Introduced UseAutoAuthTokenRaw for Cache to support triplicate value of true/false/force

Co-authored-by: Clint <catsby@users.noreply.github.com>
2020-01-30 09:08:42 -06:00
Calvin Leung Huang
8f8ba51449
test: fix TestAgent_Template_Basic (#8257)
* test: fix TestAgent_Template_Basic

* test: fix TestAgent_Template_ExitCounter
2020-01-29 09:31:29 -08:00
Michel Vocks
b879d61f02
Fix Vault Agent Template TLS config parameters (#8243) 2020-01-28 10:59:31 +01:00
Jim Kalafut
060775d9e9
Run goimports (#8251) 2020-01-27 21:11:00 -08:00
Becca Petrin
d7d4084c86
Observer pattern for service registration interface (#8123)
* use observer pattern for service discovery

* update perf standby method

* fix test

* revert usersTags to being called serviceTags

* use previous consul code

* vault isnt a performance standby before starting

* log err

* changes from feedback

* add Run method to interface

* changes from feedback

* fix core test

* update example
2020-01-24 09:42:03 -08:00
Jeff Mitchell
e243a85038 Update sdk to use gatedwriter from it 2020-01-23 14:00:49 -05:00
Jeff Mitchell
bd840fe9ab
Update gated-writer logic a bit (#8227)
This is to smooth some other changes coming once
https://github.com/hashicorp/go-hclog/pull/56 lands
2020-01-23 13:57:18 -05:00
Vishal Nayak
eb5fba1ca1 Use Shamir as KeK when migrating from auto-seal to shamir (#8172)
* Use Shamir as KeK when migrating from auto-seal to shamir

* Use the correct number of shares/threshold for the migrated seal.

* Fix log message

* Add WaitForActiveNode to test

* Make test fail

* Minor updates

* Test with more shares and a threshold

* Add seal/unseal step to the test

* Update the logic that prepares seal migration (#8187)

* Update the logic that preps seal migration

* Add test and update recovery logic

Co-authored-by: ncabatoff <nick.cabatoff@gmail.com>
Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
2020-01-21 09:24:33 -08:00
Jeff Mitchell
9fb12a733e Update test var name and tidy 2020-01-16 20:18:59 -05:00
Jeff Mitchell
c9de89d87b
Pull wrapping creation to a var (#8137)
* Pull a func out to a var

* Funcouttovarextension (#8153)

* Update test

Co-authored-by: Lexman <Lexman42@users.noreply.github.com>
2020-01-16 13:10:30 -05:00
Vishal Nayak
1a2ec77a7a Raft retry join (#7856)
* Raft retry join

* update

* Make retry join work with shamir seal

* Return upon context completion

* Update vault/raft.go

Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>

* Address some review comments

* send leader information slice as a parameter

* Make retry join work properly with Shamir case. This commit has a blocking issue

* Fix join goroutine exiting before the job is done

* Polishing changes

* Don't return after a successful join during unseal

* Added config parsing test

* Add test and fix bugs

* minor changes

* Address review comments

* Fix build error

Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
2020-01-13 17:02:16 -08:00
Jeff Mitchell
157e805b97
Migrate built in auto seal to go-kms-wrapping (#8118) 2020-01-10 20:39:52 -05:00
Becca Petrin
d9d5745f60 fix AWS http import 2020-01-10 15:53:56 -08:00
Becca Petrin
820dfaf3bb
Add support for AWS instance metadata v2 (#8062)
* add support for aws instance metadata v2

* use GetDynamicData instead
2020-01-10 09:31:16 -08:00
Becca Petrin
56edb780e8
Add Kerberos auth agent (#7999)
* add kerberos auth agent

* strip old comment

* changes from feedback

* strip appengine indirect dependency
2020-01-09 14:56:34 -08:00
Michel Vocks
488be87f67
Fix error handling during client TLS config setup (#8025) 2019-12-18 11:22:15 +01:00
Becca Petrin
73fd8f314f
Add Kerberos SPNEGO auth plugin (#7908) 2019-12-11 11:18:37 -08:00
Mike Jarmy
df01a4307d
Introduce optional service_registration stanza (#7887)
* move ServiceDiscovery into methods

* add ServiceDiscoveryFactory

* add serviceDiscovery field to vault.Core

* refactor ConsulServiceDiscovery into separate struct

* cleanup

* revert accidental change to go.mod

* cleanup

* get rid of un-needed struct tags in vault.CoreConfig

* add service_discovery parser

* add ServiceDiscovery to config

* cleanup

* cleanup

* add test for ConfigServiceDiscovery to Core

* unit testing for config service_discovery stanza

* cleanup

* get rid of un-needed redirect_addr stuff in service_discovery stanza

* improve test suite

* cleanup

* clean up test a bit

* create docs for service_discovery

* check if service_discovery is configured, but storage does not support HA

* tinker with test

* tinker with test

* tweak docs

* move ServiceDiscovery into its own package

* tweak a variable name

* fix comment

* rename service_discovery to service_registration

* tweak service_registration config

* Revert "tweak service_registration config"

This reverts commit 5509920a8a.

* simplify naming

* refactor into ./serviceregistration/consul
2019-12-06 09:46:39 -05:00
Michel Vocks
f0ff4c9e4f
Add enable_hostname_label option to telementry stanza (#7902) 2019-12-02 09:26:46 +01:00
Andrea Scarpino
6c08800675 Fix typo in the documentation (#7938) 2019-11-26 15:57:51 -06:00
Calvin Leung Huang
b9d03ca99e
agent/template: give more time for test to poll on directory before f… (#7872)
* agent/template: give more time for test to poll on directory before failing

* refactor timeout bit
2019-11-25 09:07:38 -08:00
Calvin Leung Huang
a68fde98e8
command: fix namespace list help text (#7890) 2019-11-25 08:57:32 -08:00
Calvin Leung Huang
4ec6dbaec3
agent: add -exit-after-auth flag (#7920)
* agent: add -exit-after-auth flag

* use short timeout for tests to prevent long test runs on error

* revert sdk/go.mod
2019-11-21 14:46:15 -08:00
Clint
aaabeff479
Agent Template: check rendering to match expectations (#7899)
* add regression test for #7883

* Add logic to count render events and match them to expected

* remove the WAIT label and make some changes to remove the break statements

* change the 'randomness' of the templateContents test helper method
2019-11-20 14:39:20 -06:00
Clint
ec62abb951 Update how Vault Agent configures Consul Templates internal logger (#7822)
* fix up logger in Vault Agent Template

* update deps
2019-11-12 11:29:29 -08:00
Michel Vocks
dc1c9c7db1 Fix indefinitely hanging agent test (#7866) 2019-11-12 09:19:11 -08:00
Calvin Leung Huang
f4ea6d6653 debug: fix test typo (#7862) 2019-11-11 16:30:49 -08:00
Calvin Leung Huang
fcf30cc328 debug: ignore invalid target flags (#7860) 2019-11-11 15:32:11 -08:00
Clint
0e7604ca50
Vault Agent Template follow-ups (#7739)
* Vault Agent Template: parse templates  (#7540)

* add template config parsing, but it's wrong b/c it's not using mapstructure

* parsing consul templates in agent config

* add additional test to configuration parsing, to cover basics

* another test fixture, rework simple test into table

* refactor into table test

* rename test

* remove flattenKeys and add other test fixture

* Update command/agent/config/config.go

Co-Authored-By: Jim Kalafut <jkalafut@hashicorp.com>

* return the decode error instead of swallowing it

* Update command/agent/config/config_test.go

Co-Authored-By: Jim Kalafut <jkalafut@hashicorp.com>

* go mod tidy

* change error checking style

* Add agent template doc

* TemplateServer: render secrets with Consul Template (#7621)

* add template config parsing, but it's wrong b/c it's not using mapstructure

* parsing consul templates in agent config

* add additional test to configuration parsing, to cover basics

* another test fixture, rework simple test into table

* refactor into table test

* rename test

* remove flattenKeys and add other test fixture

* add template package

* WIP: add runner

* fix panic, actually copy templates, etc

* rework how the config.Vault is created and enable reading from the environment

* this was supposed to be a part of the prior commit

* move/add methods to testhelpers for converting some values to pointers

* use new methods in testhelpers

* add an unblock channel to block agent until a template has been rendered

* add note

* unblock if there are no templates

* cleanups

* go mod tidy

* remove dead code

* simple test to starT

* add simple, empty templates test

* Update package doc, error logs, and add missing close() on channel

* update code comment to be clear what I'm referring to

* have template.NewServer return a (<- chan) type, even though it's a normal chan, as a better practice to enforce reading only

* Update command/agent.go

Co-Authored-By: Jim Kalafut <jkalafut@hashicorp.com>

* update with test

* Add README and doc.go to the command/agent directory (#7503)

* Add README and doc.go to the command/agent directory

* Add link to website

* address feedback for agent.go

* updated with feedback from Calvin

* Rework template.Server to export the unblock channel, and remove it from the NewServer function

* apply feedback from Nick

* fix/restructure rendering test

* Add pointerutil package for converting types to their pointers

* Remove pointer helper methods; use sdk/helper/pointerutil instead

* update newRunnerConfig to use pointerutil and empty strings

* only wait for unblock if template server is initialized

* update test structure

* some test cleanup

* follow up tests

* remove debugging, fix issue in replacing runner config

* need to handle first render/token

* Simplify the blocking logic to support exit after auth

* fix channel name

* expand TestAgent_Template to include multiple scenarios

* cleanup

* test cleanups after feedback
2019-11-11 17:27:23 -06:00
Calvin Leung Huang
c439a15f80
server: automatically enable dev mode if dev-consul is provided (#7847) 2019-11-11 09:51:49 -08:00
Jeff Mitchell
6f1749a88e
Add ability to renew by accessor (#7817)
* Add renewing by accessor

* Add accessor renewing test and fix bug

* Update website docs

* Remove extra newline

* Add command-level test
2019-11-08 11:32:01 -05:00
Jim Kalafut
cb178b7e4f
Run go fmt (#7823) 2019-11-07 08:54:34 -08:00
Calvin Leung Huang
f1c1a13ce2 debug: check for nil secret on replication status response before accessing secret.Data (#7811) 2019-11-06 16:36:38 -05:00
Brian Shumate
b1d1662c25 Help text: update token create help about id value (#7792)
- Token id is no longer a 36 character UUID
2019-11-05 15:50:20 -08:00
ncabatoff
a87a8b6c75
Use port 0 as the listener port so the OS assigns one not in use (#7763) 2019-10-30 10:36:23 -04:00
Dilan Bellinghoven
281b84184e Add TLS server name to Vault stanza of Agent configuration (#7519) 2019-10-29 09:11:01 -04:00
Jeff Mitchell
79ae63e9ae Sync up Agent and API's renewers. (#7733)
* Sync up Agent and API's renewers.

This introduces a new type, LifetimeWatcher, which can handle both
renewable and non-renewable secrets, modeled after the version in Agent.
It allows the user to select behavior, with the new style being the
default when calling Start(), and old style if using the legacy Renew()
call.

No tests have been modified (except for reflect issues) and no other
code has been modified to make sure the changes are backwards
compatible.

Once this is accepted I'll pull the Agent version out.

* Move compat flags to NewRenewer

* Port agent to shared lifetime watcher lib
2019-10-28 17:28:59 -07:00
Calvin Leung Huang
e71d856c34
agent: fix data race on inmemSink's token (#7707)
* agent: fix data race on inmemSink's token

* use uber/atomic instead
2019-10-22 10:42:56 -07:00
Calvin Leung Huang
da984b2d48
cli: fix json output for namespace list command (#7705) 2019-10-21 13:38:37 -07:00
Clint
012c165b02
Vault Agent Template (#7652)
* Vault Agent Template: parse templates  (#7540)

* add template config parsing, but it's wrong b/c it's not using mapstructure

* parsing consul templates in agent config

* add additional test to configuration parsing, to cover basics

* another test fixture, rework simple test into table

* refactor into table test

* rename test

* remove flattenKeys and add other test fixture

* Update command/agent/config/config.go

Co-Authored-By: Jim Kalafut <jkalafut@hashicorp.com>

* return the decode error instead of swallowing it

* Update command/agent/config/config_test.go

Co-Authored-By: Jim Kalafut <jkalafut@hashicorp.com>

* go mod tidy

* change error checking style

* Add agent template doc

* TemplateServer: render secrets with Consul Template (#7621)

* add template config parsing, but it's wrong b/c it's not using mapstructure

* parsing consul templates in agent config

* add additional test to configuration parsing, to cover basics

* another test fixture, rework simple test into table

* refactor into table test

* rename test

* remove flattenKeys and add other test fixture

* add template package

* WIP: add runner

* fix panic, actually copy templates, etc

* rework how the config.Vault is created and enable reading from the environment

* this was supposed to be a part of the prior commit

* move/add methods to testhelpers for converting some values to pointers

* use new methods in testhelpers

* add an unblock channel to block agent until a template has been rendered

* add note

* unblock if there are no templates

* cleanups

* go mod tidy

* remove dead code

* simple test to starT

* add simple, empty templates test

* Update package doc, error logs, and add missing close() on channel

* update code comment to be clear what I'm referring to

* have template.NewServer return a (<- chan) type, even though it's a normal chan, as a better practice to enforce reading only

* Update command/agent.go

Co-Authored-By: Jim Kalafut <jkalafut@hashicorp.com>

* update with test

* Add README and doc.go to the command/agent directory (#7503)

* Add README and doc.go to the command/agent directory

* Add link to website

* address feedback for agent.go

* updated with feedback from Calvin

* Rework template.Server to export the unblock channel, and remove it from the NewServer function

* apply feedback from Nick

* fix/restructure rendering test

* Add pointerutil package for converting types to their pointers

* Remove pointer helper methods; use sdk/helper/pointerutil instead

* update newRunnerConfig to use pointerutil and empty strings

* only wait for unblock if template server is initialized

* drain the token channel in this test

* conditionally send on channel
2019-10-18 16:21:46 -05:00
ncabatoff
afcba41190
Shamir seals now come in two varieties: legacy and new-style. (#7694)
Shamir seals now come in two varieties: legacy and new-style. Legacy
Shamir is automatically converted to new-style when a rekey operation
is performed. All new Vault initializations using Shamir are new-style.

New-style Shamir writes an encrypted master key to storage, just like
AutoUnseal. The stored master key is encrypted using the shared key that
is split via Shamir's algorithm. Thus when unsealing, we take the key
fragments given, combine them into a Key-Encryption-Key, and use that
to decrypt the master key on disk. Then the master key is used to read
the keyring that decrypts the barrier.
2019-10-18 14:46:00 -04:00
Lexman
47a54ac8bc
oss changes for entropy augmentation feature (#7670)
* oss changes for entropy augmentation feature

* fix oss command/server/config tests

* update go.sum

* fix logical_system and http/ tests

* adds vendored files

* removes unused variable
2019-10-17 10:33:00 -07:00
Calvin Leung Huang
0b3777f2aa
Vault Debug (#7375)
* cli: initial work on debug; server-status target

* debug: add metrics capture target (#7376)

* check against DR secondary

* debug: add compression

* refactor check into preflight func

* debug: set short test time on tests, fix exit code bug

* debug: use temp dir for output on tests

* debug: use mholt/archiver for compression

* first pass on adding pprof

* use logger for output

* refactor polling target capture logic

* debug: poll and collect replication status

* debug: poll and collect host-info; rename output files and collection refactor

* fix comments

* add archive test; fix bugs found

* rename flag name to singular target

* add target output test; scaffold other tests cases

* debug/test: add pprof and index file tests

* debug/test: add min timing check tests

* debug: fix index gen race and collection goroutine race

* debug: extend archive tests, handle race between program exit and polling goroutines

* update docstring

* debug: correctly add to pollingWg

* debug: add config target support

* debug: don't wait on interrupt shutdown; add file exists unit tests

* move pprof bits into its goroutine

* debug: skip empty metrics and some pprof file creation if permission denied, add matching unit test

* address comments and feedback

* Vault debug using run.Group (#7658)

* debug: switch to use oklog/run.Group

* debug: use context to cancel requests and interrupt rungroups.

* debug: trigger the first interval properly

* debug: metrics collection should use metrics interval

* debug: add missing continue on metrics error

* debug: remove the use of buffered chan to trigger first interval

* debug: don't shadow BaseCommand's client, properly block on interval capture failures

* debug: actually use c.cachedClient everywhere

* go mod vendor

* debug: run all pprof in goroutines; bump pprof timings in tests to reduce flakiness

* debug: update help text
2019-10-15 15:39:19 -07:00
Vishal Nayak
2f4ae7f215
Recovery Mode (#7559)
* Initial work

* rework

* s/dr/recovery

* Add sys/raw support to recovery mode (#7577)

* Factor the raw paths out so they can be run with a SystemBackend.

# Conflicts:
#	vault/logical_system.go

* Add handleLogicalRecovery which is like handleLogical but is only
sufficient for use with the sys-raw endpoint in recovery mode.  No
authentication is done yet.

* Integrate with recovery-mode.  We now handle unauthenticated sys/raw
requests, albeit on path v1/raw instead v1/sys/raw.

* Use sys/raw instead raw during recovery.

* Don't bother persisting the recovery token.  Authenticate sys/raw
requests with it.

* RecoveryMode: Support generate-root for autounseals (#7591)

* Recovery: Abstract config creation and log settings

* Recovery mode integration test. (#7600)

* Recovery: Touch up (#7607)

* Recovery: Touch up

* revert the raw backend creation changes

* Added recovery operation token prefix

* Move RawBackend to its own file

* Update API path and hit it using CLI flag on generate-root

* Fix a panic triggered when handling a request that yields a nil response. (#7618)

* Improve integ test to actually make changes while in recovery mode and
verify they're still there after coming back in regular mode.

* Refuse to allow a second recovery token to be generated.

* Resize raft cluster to size 1 and start as leader (#7626)

* RecoveryMode: Setup raft cluster post unseal (#7635)

* Setup raft cluster post unseal in recovery mode

* Remove marking as unsealed as its not needed

* Address review comments

* Accept only one seal config in recovery mode as there is no scope for migration
2019-10-15 00:55:31 -04:00
Jim Kalafut
a23395d209
Add region parameter to AWS agent auto auth (#7632) 2019-10-14 11:56:46 -07:00
Mike Jarmy
77ceb7dde0
Vault Agent Cache Auto-Auth SSRF Protection (#7627)
* implement SSRF protection header

* add test for SSRF protection header

* cleanup

* refactor

* implement SSRF header on a per-listener basis

* cleanup

* cleanup

* creat unit test for agent SSRF

* improve unit test for agent SSRF

* add VaultRequest SSRF header to CLI

* fix unit test

* cleanup

* improve test suite

* simplify check for Vault-Request header

* add constant for Vault-Request header

* improve test suite

* change 'config' to 'agentConfig'

* Revert "change 'config' to 'agentConfig'"

This reverts commit 14ee72d21f.

* do not remove header from request

* change header name to X-Vault-Request

* simplify http.Handler logic

* cleanup

* simplify http.Handler logic

* use stdlib errors package
2019-10-11 18:56:07 -04:00
Brian Kassouf
68750b70a2
OSS portions of raft non-voters (#7634)
* OSS portions of raft non-voters

* add file

* Update vault/raft.go

Co-Authored-By: Vishal Nayak <vishalnayak@users.noreply.github.com>
2019-10-11 11:56:59 -07:00
Jason O'Donnell
8875c9ec08 bug: VAULT_CLUSTER_ADDR not used in raft (#7619)
* bug: VAULT_CLUSTER_ADDR not used in raft

* Make env take precedence
2019-10-10 11:42:59 -07:00
Calvin Leung Huang
656b113dbd
sys/config: config state endpoint (#7424)
* sys/config: initial work on adding config state endpoint

* server/config: add tests, fix Sanitized method

* thread config through NewTestCluster's config to avoid panic on dev modes

* properly guard endpoint against request forwarding

* add http tests, guard against panics on nil RawConfig

* ensure non-nil rawConfig on NewTestCluster cores

* update non-forwarding logic

* fix imports; use no-forward handler

* add missing config test fixture; update gitignore

* return sanitized config as a map

* fix test, use deep.Equal to check for equality

* fix http test

* minor comment fix

* config: change Sanitized to return snake-cased keys, update tests

* core: hold rlock when reading config; add docstring

* update docstring
2019-10-08 10:57:15 -07:00
Anton Soroko
7aa1ffa92e Remove unused code in kv_get.go (#7583) 2019-10-07 10:18:04 -04:00
Michel Vocks
993a1ae9f2
Add config parameter to allow unauthenticated metrics access (#7550)
* Implement config parameter to allow unathenticated metricss access

* Add unit test for unauthenticated metrics access parameter

* go mod tidy
2019-10-04 09:29:51 +02:00
Jim Kalafut
b68573fa74
Log proxy settings from environment on startup (#7528) 2019-09-30 08:46:42 -07:00
Ivan Kurnosov
0c86d909b4 Fixed github-prod path (#7516) 2019-09-26 08:46:41 -04:00
Jim Kalafut
6a14065f73
Ignore any existing token during CLI login (#7508)
Fixes #6694
2019-09-25 10:59:42 -07:00
Dilan Bellinghoven
13c56f5f92 chore: Do not need logger for command/agent/config.LoadConfig (#7496) 2019-09-19 13:03:30 -07:00
Jim Kalafut
d6135f743f Fix Agent handling of gzipped responses (#7470)
* Fix Agent handling of gzipped responses

Fixes #6606

* Only remove "gzip" member, if present

* Simplify to just removing Accept-Encoding altogether
2019-09-18 14:24:41 -07:00
Matthew Irish
545a4b1242
expose 'storage_type' on the sys/seal-status endpoint (#7486)
* expose 'storage_type' on the sys/seal-status endpoint

* add comments

* Update vault/core.go

Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>
2019-09-18 14:07:18 -05:00
Jeff Mitchell
f843c09dd1 Move renewer to internal package so tests don't barf 2019-09-09 16:09:53 -04:00
Jeff Mitchell
9ee93b6e5f
Support batch tokens in renewer and caching proxy for transit auto-unseal (#7441)
Currently this uses a fork of the api.Renewer code, which we should consolidate in 1.3.
2019-09-09 12:58:28 -07:00
Jim Kalafut
1e467e1063
Add OCI auth to builtin plugin registry (#7436) 2019-09-05 13:50:12 -07:00
Jim Kalafut
96b3af9f00
Bundle OCI Auth method (#7422) 2019-09-04 16:46:00 -07:00
Vu Pham
feafd1b388 OCI KMS auto-unseal plugin (#6950) 2019-09-04 12:40:54 -07:00
Vu Pham
df173e70fc Added OCI Object Storage Plugin (#6985) 2019-09-04 11:33:16 -07:00
Becca Petrin
392e41ca42
rename pcf to cf maintaining backwards compat (#7346) 2019-08-26 09:55:08 -07:00
ncabatoff
78b5e2100d
Make -dev-four-cluster an ent-only option and remove ent-specific testhelpers (#7215) 2019-08-23 11:53:18 -04:00
Jeff Malnick
d430d5fa8a
Allow setting file mode on vault agent sink file (#7275)
* feat: enable setting mode on vault agent sink file

* doc: update vault agent file sink with mode configuration
2019-08-21 20:41:55 -07:00
Tommy Murphy
1c146a1012 telemetry: add stackdriver metrics sink (#6957)
* telemetry: add stackdriver metrics sink

* telemetry: stackdriver go mod tidy
2019-08-20 14:47:08 -07:00
Calvin Leung Huang
d0d17ed40e command/server: fix TestLoadConfigFile_json2 test, fix hcl tags (#7300)
* command/server: fix TestLoadConfigFile_json2 test, fix hcl tags

Fixes test to call the equality check, and add missing values to the expected object. Fixes hcl tags in the Telemetry structs.

* fix PrometheusRetentionTime tag
2019-08-14 10:32:11 -04:00
John Morrissey
4bf623d475 avoid a panic on failed creds polling (#7238) 2019-08-01 16:22:15 -04:00
Brian Kassouf
e88721c7db
storage/raft: Support storage migration to raft storage (#7207)
* Support raft in the migration command

* Add comments
2019-07-29 13:05:43 -07:00
Jeff Mitchell
fe03781408 Add adjustment factor for number of backends for secrets test 2019-07-25 10:00:51 -04:00
Srikanth Venkatesh
94db194a4d fixed a typo in an exception message regarding absence of config files while starting up vault (#6816) 2019-07-24 12:58:48 -04:00
Jeff Mitchell
6ad21c6812 Make fmt 2019-07-24 12:41:28 -04:00
Jonathan Sokolowski
fb8ce19d90 Add -dev-no-store-token to vault server command (#7104)
When starting a vault dev server the token helper is invoked to store
the dev root token.
This option gives the user the ability to not store the token.

Storing the token can be undesirable in certain circumstances
(e.g.  running local tests) as the user's existing vault token is
clobbered without warning.

Fixes #1861
2019-07-24 12:41:07 -04:00
Jim Kalafut
ec42f5f219
Fix broken server config test (#7153) 2019-07-22 09:39:13 -07:00
Christian Muehlhaeuser
c70d05b026 Fixed a bunch of typos (#7146) 2019-07-18 21:10:15 -04:00
Mike Jarmy
be3e2a1288
Add 'log-format' CLI flag, along with associated config flag, for 'vault server' command. (#6840)
* Read config before creating logger when booting vault server

* Allow for specifying log output in JSON format in a config file, via a 'log_level' flag

* Create parser for log format flag

* Allow for specifying log format in a config file, via a 'log_format' flag. Also, get rid of 'log_json' flag.

* Add 'log-format' command line flag

* Update documentation to include description of log_format setting

* Tweak comment for VAULT_LOG_FORMAT environment variable

* add test for ParseEnvLogFormat()

* clarify how log format is set

* fix typos in documentation
2019-07-18 15:59:27 -04:00
Calvin Leung Huang
3e2cd485de
logical: add support for passing data to delete (#7139)
* logical: add support for passing data to delete

* add back raft bit

* add back raft bit

* update error message

* fix command delete tests
2019-07-18 10:42:36 -07:00
Michel Vocks
757b039ee4
Added operator raft and operator raft snapshot descriptions (#7106) 2019-07-16 09:31:00 +02:00
Becca Petrin
3cc7f4a68c
add a reader that takes stdin (#7074) 2019-07-05 13:36:44 -07:00
Mike Jarmy
c9775399cc
fix output-curl-string for 'vault kv patch' (#6848)
* fix output-curl-string for 'vault-kv-patch'

* improve comments
2019-07-03 09:03:35 -04:00
Michel Vocks
b3cc25f939
Add namespace config option to agent auto-auth config (#6988)
* Added namespace option to vault agent auto-auth method

* Implemented review feedback
2019-07-03 09:33:20 +02:00
Michael Gaffney
3ae451ec78
Merge branch 'master' into mgaffney/kv-delete-version-after 2019-07-02 17:27:36 -04:00
Michel Vocks
e7ed7399f8
Token identity support (#6267)
* Implemented token backend support for identity

* Fixed tests

* Refactored a few checks for the token entity overwrite. Fixed tests.

* Moved entity alias check up so that the entity and entity alias is only created when it has been specified in allowed_entity_aliases list

* go mod vendor

* Added glob pattern

* Optimized allowed entity alias check

* Added test for asterisk only

* Changed to glob pattern anywhere

* Changed response code in case of failure. Changed globbing pattern check. Added docs.

* Added missing token role get parameter. Added more samples

* Fixed failing tests

* Corrected some cosmetical review points

* Changed response code for invalid provided entity alias

* Fixed minor things

* Fixed failing test
2019-07-01 11:39:54 +02:00
Michael Gaffney
2dde85e43b Remove delete-version-after from kv put and undelete subcommands
Removes the optional parameter "delete-version-after" from the following
CLI subcommands:

- kv put
- kv undelete
- kv rollback
2019-06-27 14:17:46 -04:00
Michael Gaffney
12ecac24a9 Add delete-version-after to kv CLI subcommands
Adds a new optional parameter "delete-version-after" to the following
CLI subcommands:

- kv metadata put
- kv put
- kv undelete
- kv rollback
2019-06-27 14:17:46 -04:00
Jeff Mitchell
fe636fb73f
Add a deprecated helper to tokenutil and JSON output to path-help (#7006) 2019-06-27 12:56:31 -04:00
Aaron Donovan
b70f5c4923 Allow configuration of nonce for ec2 authentication (#6953)
* Allow configuration of nonce for ec2 authentication

* Addressing pull request comment
2019-06-27 11:06:20 -04:00
Jeff Mitchell
2ac94d5581
Add the ability to use a dev Consul node for dev storage (#6965) 2019-06-24 13:29:47 -04:00
Vishal Nayak
c356df69a1
raft join tls (#6932) 2019-06-21 17:41:07 -04:00
Vishal Nayak
863201b434
Raft CLI (#6893)
* raft cli

* Reuse the command's client

* Better response handling

* minor touchups
2019-06-20 21:32:00 -04:00
Jeff Mitchell
99f914ae6f Fix tests 2019-06-20 21:00:01 -04:00
Jeff Mitchell
8a308716d6 Make base predict test kmip friendly 2019-06-20 20:57:46 -04:00
Jeff Mitchell
a342dcbb29 Sync 2019-06-20 20:55:10 -04:00
Brian Kassouf
b435028f3f
Raft Storage Backend (#6888)
* Work on raft backend

* Add logstore locally

* Add encryptor and unsealable interfaces

* Add clustering support to raft

* Remove client and handler

* Bootstrap raft on init

* Cleanup raft logic a bit

* More raft work

* Work on TLS config

* More work on bootstrapping

* Fix build

* More work on bootstrapping

* More bootstrapping work

* fix build

* Remove consul dep

* Fix build

* merged oss/master into raft-storage

* Work on bootstrapping

* Get bootstrapping to work

* Clean up FMS and node-id

* Update local node ID logic

* Cleanup node-id change

* Work on snapshotting

* Raft: Add remove peer API (#906)

* Add remove peer API

* Add some comments

* Fix existing snapshotting (#909)

* Raft get peers API (#912)

* Read raft configuration

* address review feedback

* Use the Leadership Transfer API to step-down the active node (#918)

* Raft join and unseal using Shamir keys (#917)

* Raft join using shamir

* Store AEAD instead of master key

* Split the raft join process to answer the challenge after a successful unseal

* get the follower to standby state

* Make unseal work

* minor changes

* Some input checks

* reuse the shamir seal access instead of new default seal access

* refactor joinRaftSendAnswer function

* Synchronously send answer in auto-unseal case

* Address review feedback

* Raft snapshots (#910)

* Fix existing snapshotting

* implement the noop snapshotting

* Add comments and switch log libraries

* add some snapshot tests

* add snapshot test file

* add TODO

* More work on raft snapshotting

* progress on the ConfigStore strategy

* Don't use two buckets

* Update the snapshot store logic to hide the file logic

* Add more backend tests

* Cleanup code a bit

* [WIP] Raft recovery (#938)

* Add recovery functionality

* remove fmt.Printfs

* Fix a few fsm bugs

* Add max size value for raft backend (#942)

* Add max size value for raft backend

* Include physical.ErrValueTooLarge in the message

* Raft snapshot Take/Restore API  (#926)

* Inital work on raft snapshot APIs

* Always redirect snapshot install/download requests

* More work on the snapshot APIs

* Cleanup code a bit

* On restore handle special cases

* Use the seal to encrypt the sha sum file

* Add sealer mechanism and fix some bugs

* Call restore while state lock is held

* Send restore cb trigger through raft log

* Make error messages nicer

* Add test helpers

* Add snapshot test

* Add shamir unseal test

* Add more raft snapshot API tests

* Fix locking

* Change working to initalize

* Add underlying raw object to test cluster core

* Move leaderUUID to core

* Add raft TLS rotation logic (#950)

* Add TLS rotation logic

* Cleanup logic a bit

* Add/Remove from follower state on add/remove peer

* add comments

* Update more comments

* Update request_forwarding_service.proto

* Make sure we populate all nodes in the followerstate obj

* Update times

* Apply review feedback

* Add more raft config setting (#947)

* Add performance config setting

* Add more config options and fix tests

* Test Raft Recovery (#944)

* Test raft recovery

* Leave out a node during recovery

* remove unused struct

* Update physical/raft/snapshot_test.go

* Update physical/raft/snapshot_test.go

* fix vendoring

* Switch to new raft interface

* Remove unused files

* Switch a gogo -> proto instance

* Remove unneeded vault dep in go.sum

* Update helper/testhelpers/testhelpers.go

Co-Authored-By: Calvin Leung Huang <cleung2010@gmail.com>

* Update vault/cluster/cluster.go

* track active key within the keyring itself (#6915)

* track active key within the keyring itself

* lookup and store using the active key ID

* update docstring

* minor refactor

* Small text fixes (#6912)

* Update physical/raft/raft.go

Co-Authored-By: Calvin Leung Huang <cleung2010@gmail.com>

* review feedback

* Move raft logical system into separate file

* Update help text a bit

* Enforce cluster addr is set and use it for raft bootstrapping

* Fix tests

* fix http test panic

* Pull in latest raft-snapshot library

* Add comment
2019-06-20 12:14:58 -07:00
Becca Petrin
6d8e9ef638 fix test 2019-06-19 10:59:11 -07:00
Becca Petrin
51f52175ab merge master 2019-06-19 10:24:45 -07:00
Becca Petrin
07120c2faf update to latest plugin dependencies 2019-06-19 10:04:49 -07:00
Becca Petrin
b492bd3068 fix test 2019-06-18 09:36:25 -07:00
Michel Vocks
37f4b65962
Fixed wrong rekey recovery backup client API URL (#6841)
* Fixed wrong rekey recovery backup client API URL

* Fixed wrong rekey recovery backup client API URL delete

* Changed output for recovery backup key delete
2019-06-11 10:05:44 +02:00
Becca Petrin
a732b22f11 add PCF auth method, agent, and cli handler 2019-06-06 12:26:04 -07:00
Jeff Mitchell
f5baee5b55
Make flushing the cache race safe (#6828)
* Make flushing the cache race safe

* Remove noop aborts
2019-06-05 02:52:18 -04:00
Jeff Mitchell
72cb483406 Audit listing with format json returns json, not a string (#6776)
* Audit listing with format json returns json, not a string

Fixes #6775

* list, kv list and namespace list with format json returns json, not a string

* Changed audit list return code to 2 which aligns with other list commands return codes
2019-06-04 10:36:34 -07:00
Calvin Leung Huang
734564825f
agent/caching: support proxying request query parameters (#6772)
* agent/caching: support proxying request query parameters

* update comment

* rejig other agent log messages to output method followed by path
2019-05-22 09:21:47 -07:00
Jim Kalafut
7c6c6fd5c0
Fix gofmt (#6764) 2019-05-20 15:15:05 -07:00
Lexman
c365b074b8 removes a duplicated test case in server_test (#6740) 2019-05-20 09:54:41 -04:00
Lexman
90ff541065
http timeout values are configurable (#6666)
* http timeout fields are configurable

* move return statement for server config tests outside of range loop

* adds documentation for configurable listener http_* values

* fixed some formatting for the docs markdown
2019-05-10 10:41:42 -07:00
vishalnayak
0ace1bc612 Increase the ttl for the tests to pass 2019-05-08 12:10:48 -04:00
Travis Cosgrave
f250d779e0 Add certificate auto-auth method to vault agent (#6652)
* adding auto auth for cert auth methods

* Adding Docs for Cert Auto-auth method

* fixes errors in docs

* wrong documentation title

* repariting a few typos in the docs and being very clear about where the certificates should be configured

* clarifying the name parameter documentation

* Fixes gofmt issues in command/agent.go

* Fix typo in docs

* Add nil check to Config for cert auto-auth and add test with and without a specified name
2019-05-06 10:39:27 -04:00
Brian Kassouf
0c9b011709
Fix a panic caused by a nil response on kv pre-flight (#6675)
* Fix a panic caused by a nil response on kv pre-flight

* Fix imports
2019-05-03 09:10:41 -07:00
Jeff Mitchell
252aa1c528 Re-add stored-shares for init and spit out a warning (#6677) 2019-05-03 09:09:59 -07:00
Brian Kassouf
f6270bae64
Add mount UUID to the secret and auth list API responses (#6633) 2019-04-24 12:27:43 -07:00
Jeff Mitchell
71d635293d Move physical/file to sdk 2019-04-15 14:51:33 -04:00
Jeff Mitchell
8054cc660f Update to api 1.0.1 and sdk 0.1.8 2019-04-15 14:10:07 -04:00
Jeff Mitchell
278bdd1f4e
Switch to go modules (#6585)
* Switch to go modules

* Make fmt
2019-04-13 03:44:06 -04:00
Becca Petrin
936192c392
Merge pull request #6268 from hashicorp/6234-aws-region
Add region to CLI for generating AWS login data
2019-04-12 16:15:38 -07:00
Jeff Mitchell
6388efd4f0 Move useragent to sdk 2019-04-12 18:17:49 -04:00
Jeff Mitchell
a920c10be6 Move password to sdk 2019-04-12 18:12:13 -04:00
Jeff Mitchell
170521481d
Create sdk/ and api/ submodules (#6583) 2019-04-12 17:54:35 -04:00
Becca Petrin
cd37917df5 Merge branch 'opensource-master' into 6234-aws-region 2019-04-12 11:13:17 -07:00
Jeff Mitchell
bf2a7be2bc
Fix some more test failures and recursive locking (#6549) 2019-04-08 13:40:54 -04:00
Calvin Leung Huang
8a5003bbe5 agent/cache: add comment around APIProxy.Send return case 2019-04-05 13:58:12 -07:00
ncabatoff
baac2642a2 Agent auto auth wrapping new config checks (#6479)
* Simplify Run(): the function that was being sent over a channel doesn't
need to close over anything except latestToken, and we don't need to
create a new one each iteration.  Instead just pass the relevant items,
namely the token and sink to work on.

* Disallow the following config combinations:
1. auto_auth.method.wrap_ttl > 0 and multiple file sinks
2. auto_auth.method.wrap_ttl > 0 and single file sink with wrap_ttl > 0
3. auto_auth.method.wrap_ttl > 0 and cache.use_auto_auth_token = true

* Expose errors that occur when APIProxy is forwarding request to Vault.

* Fix merge issues.
2019-04-05 16:12:54 -04:00
Becca Petrin
043ca6263e merge master 2019-04-01 13:52:44 -07:00
Alessandro De Blasis
14138f66c9 agent: allow AppRole Auto-Auth when bind_secret_id = false (#6324)
* agent: allow AppRole Auto-Auth when bind_secret_id = false
2019-04-01 16:27:54 -04:00
ncabatoff
0b05c86f8c Add support in Agent for running cache+auto_auth without any sinks (#6468)
* Add support in Agent for running cache+auto_auth without any sinks
configured.

* Add missing fixture.
2019-04-01 16:26:41 -04:00
Calvin Leung Huang
93e1cd07b0 set configSeal.Type to the type specified via the env var if provided (#6453) 2019-03-25 16:52:20 -07:00
Jeff Mitchell
1944263af7 Correct migration case to Shamir where it's not explicit (#6458)
If you were migrating to Shamir but didn't specify a Shamir block
migration would fail. Being explicit is nice but it's also not really
obvious since you don't need the block normally.
2019-03-25 16:50:58 -07:00
Becca Petrin
389eedb598 fix typo (#6469) 2019-03-25 10:56:04 -07:00
stuio
66ef59957a minor improvements to unseal inline docs (#6449) 2019-03-21 05:57:46 -07:00
T.K
4bcf0ec963 changed misspelled english words (#6432) 2019-03-19 09:32:45 -04:00
ncabatoff
351327867f
Move listener config from 'cache' block to top-level 'listener' blocks. Allow cache without auto-auth. (#6421)
* Since we want to use the Agent listener for #6384, move listener config
from top-level 'cache' block to new top-level 'listeners' block.

* Make agent config allow cache and listener blocks without auto-auth
configured.
2019-03-15 14:58:53 -04:00
ncabatoff
e6cadbf7b1 Fix windows build by moving references to SIGUSR2 into a file not built (#6422)
on windows.
2019-03-15 10:12:48 -07:00
ncabatoff
ccfeef6688
Receiving a SIGUSR2 makes Vault log the running goroutines' stacks. (#6240)
* Receiving a SIGUSR2 makes Vault log the running goroutines' stacks.
2019-03-15 09:27:53 -04:00
Jim Kalafut
8de190f31d Fix improperly formatted help output strings (#6414) 2019-03-14 13:57:28 -07:00
Iskander (Alex) Sharipov
dcfdb3d5d5 all: fix no-op append calls (#6360)
Append call in form of `append(s)` has no effect,
it just returns `s`. Sometimes such invocation is a sign
of a programming error, so it's better to remove these.

Signed-off-by: Iskander Sharipov <quasilyte@gmail.com>
2019-03-14 13:40:30 -07:00
Philipp Bandow
6d19de044a ssh on windows: fix file handling problem (#6359) 2019-03-14 13:39:34 -07:00
Vishal Nayak
3c7c593bca Agent: Listener refactoring and socket file system permissions (#6397)
* Listener refactoring and file system permissions

* added listenerutil and move some common code there

* Added test for verifying socket file permissions

* Change default port of agent to 8200

* address review feedback

* Address review feedback

* Read socket options from listener config
2019-03-14 11:53:14 -07:00
Michel Vocks
c85198a563
Added warning for auto unseal init command (#6398)
* Added warning when init command uses auto unseal and -key-shares and -key-threshold was set

* Reworked

* Update command/operator_init.go

Co-Authored-By: michelvocks <michelvocks@gmail.com>
2019-03-13 18:06:39 +01:00
Calvin Leung Huang
27c655ef67
agent/caching: add X-Cache and Age headers (#6394)
* agent/caching: add X-Cache and Age headers, update Date header on cached resp

* Update command/agent/cache/lease_cache.go

Co-Authored-By: calvn <cleung2010@gmail.com>

* Update command/agent/cache/proxy.go

Co-Authored-By: calvn <cleung2010@gmail.com>
2019-03-12 13:21:02 -07:00
Calvin Leung Huang
f72078f237
agent/caching: use locks on LeaseCache.Send (#6374)
* agent/caching: use locks on LeaseCache.Send to prevent running multiple renewer goroutines

* update comment
2019-03-07 15:50:12 -08:00
Jeff Mitchell
92988f36a1 HSM -> Auto Unseal in init help output 2019-03-07 15:10:50 -05:00
Jim Kalafut
a7531526ab
Fix hanging on empty keys during operator migrate (#6371) 2019-03-07 11:48:48 -08:00
Christopher McNabb
d0442dbba5 Use proper pathSeparator for the operating system (#6347)
* Use proper pathSeparator for the operating system

When running on Windows use the backslash as the path separator, other wise use the forward slash
2019-03-07 09:51:10 -05:00
Vishal Nayak
f1fcc27d2d
Avoid redundant client creation (#6361) 2019-03-06 22:09:55 -05:00
Calvin Leung Huang
e57416c93d
Refactor handleCacheClear input params (#6350)
* Refactor handleCacheClear input params

* address review feedback

* fix handleCacheClear's token_accessor case
2019-03-06 18:08:19 -08:00
Calvin Leung Huang
ba86e6bbd5
agent/caching: proxy redirect and non-json responses; update tests (#6353)
* agent/caching: proxy redirect and non-json responses; update tests

* agent/caching: do not wrap error responses as internal errors, simply proxy them back as-is

* minor refactoring of APIProxy.Send logic

* add test case to ensure error response is not wrapped
2019-03-06 17:23:20 -08:00
Vishal Nayak
c00c00ba94
Agent Cache doc updates (#6331)
* Agent Cache doc updates

* doc update

* Add renewal management section

* doc updates

* paraphrase the orphan token case
2019-03-05 15:19:52 -05:00
Vishal Nayak
c14e61c16e
Change agent's port to 8007 (#6348) 2019-03-05 12:57:17 -05:00
Vishal Nayak
6719ede26a
Test request token overriding auto-auth case (#6346) 2019-03-05 12:49:58 -05:00
Becca Petrin
74037a92d0 merge master 2019-03-05 09:39:53 -08:00
Vishal Nayak
038ab04cd0
Refactor respond error function (#6343) 2019-03-05 11:43:30 -05:00
Calvin Leung Huang
a84520e7d3
Fix TestLeaseCache_SendCacheable (#6333) 2019-03-04 15:18:52 -08:00
Jeff Mitchell
213da13264 Add ability to migrate autoseal to autoseal (#5930)
* Add ability to migrate autoseal to autoseal

This adds the ability to migrate from shamir to autoseal, autoseal to
shamir, or autoseal to autoseal, by allowing multiple seal stanzas. A
disabled stanza will be used as the config being migrated from; this can
also be used to provide an unwrap seal on ent over multiple unseals.

A new test is added to ensure that autoseal to autoseal works as
expected.

* Fix test

* Provide default shamir info if not given in config

* Linting feedback

* Remove context var that isn't used

* Don't run auto unseal watcher when in migration, and move SetCores to SetSealsForMigration func

* Slight logic cleanup

* Fix test build and fix bug

* Updates

* remove GetRecoveryKey function
2019-03-04 14:11:56 -08:00
Jeff Mitchell
ad6641a40c Minor syncs 2019-03-04 13:35:22 -05:00
Calvin Leung Huang
39b8acb915
agent/caching: simplify orphan check; add orphan token creation tests (#6322) 2019-03-01 17:57:25 -08:00
Vishal Nayak
12b51ff859
Add locking around base context (#6321)
Got offline 👍 from Calvin.
2019-03-01 20:30:14 -05:00
Vishal Nayak
be555fccca
added client_key and client_cert options to the agent config (#6319) 2019-03-01 15:11:16 -05:00
Calvin Leung Huang
7cc75c5f38
agent/caching: refactor ContextInfo (#6311)
* agent/caching: refactor ContextInfo

* use NewContextInfo in NewLeaseCache
2019-03-01 11:20:36 -08:00
vishalnayak
cb71144203 Agent config vault block should be optional 2019-03-01 12:23:00 -05:00
Michel Vocks
9617832784
Print warning when 'tls_cipher_suites' includes blacklisted cipher suites (#6300)
* Implemented a warning when tls_cipher_suites includes only cipher suites which are not supprted by the HTTP/2 spec

* Added test for cipher suites

* Added hard fail on startup when all defined cipher suites are blacklisted. Added warning when some ciphers are blacklisted.

* Replaced hard failure with warning. Removed bad cipher util function and replaced it by external library.

* Added missing dependency. Fixed renaming of package name.
2019-03-01 16:48:06 +01:00
Vishal Nayak
6ce09bae6a
Support reading Vault's address from Agent's config file (#6306)
* Support reading Vault's address from Agent's config file

* use consts and switch

* Add tls options to agent config vault block

* Update command/agent/config/config.go

Co-Authored-By: vishalnayak <vishalnayak@users.noreply.github.com>

* remove fmt.Printfs
2019-02-28 17:29:28 -05:00
Chris Hoffman
2d0c3ff335 Transit Autounseal (#5995)
* Adding Transit Autoseal

* adding tests

* adding more tests

* updating seal info

* send a value to test and set current key id

* updating message

* cleanup

* Adding tls config, addressing some feedback

* adding tls testing

* renaming config fields for tls
2019-02-28 13:13:56 -08:00
Calvin Leung Huang
6934bb6f9d agent/caching: test renaming; add api-level cache-clear test (#6305)
* test renaming; add api-level cache-clear test

* fix import ordering
2019-02-28 10:05:55 -08:00
Calvin Leung Huang
e7365516f3
agent/caching: enable caching of leases generated by autoauth token (#6293)
* agent/caching: enable caching of leases generated by autoauth token

* add test for auth/token/create path

* update error message log

* Some minor updates

* add sleep timer for renewal logic to process
2019-02-27 13:14:58 -08:00
Vishal Nayak
13586e8e09
Agent Caching: Remove response updates using renewal output (#6303)
* Agent Caching: Remove response updates using renewal output

* Use renewal output channel to log the fact
2019-02-27 14:33:45 -05:00
vishalnayak
603199c2cc Change agent's default port number to 8100 2019-02-26 10:02:12 -05:00
Jim Kalafut
143412d58e
Fix agent test (#6281) 2019-02-21 21:24:16 -08:00
Calvin Leung Huang
2e0bef841a
agent/caching: use const for cache-clear path (#6271) 2019-02-21 14:53:34 -08:00
Calvin Leung Huang
05d4f97882
agent/caching: remove EvictByPrefix and EvictAll (#6269) 2019-02-21 14:52:10 -08:00
Becca Petrin
99bd2afbb3 allow aws region in cli login 2019-02-20 16:43:21 -08:00
Brian Kassouf
760954c077 make fmt 2019-02-20 12:12:21 -08:00
Vishal Nayak
1e0b6a0d88
AgentCache: Properly compute auto-auth token (#6264)
* Properly compute auto-auth token

* Use inmem sink to track auto-auth token

* update debug statement

* Only add inmem sink if using auto-auth token is allowed
2019-02-20 02:11:05 -05:00
Vishal Nayak
91deac7aab
Agent Cache: Some review feedback (#6257)
* Revamp agent cache client

* Update command/agent.go

Co-Authored-By: vishalnayak <vishalnayak@users.noreply.github.com>

* Agent cache auto auth token lookup case (#6258)

* agent cache auto auth token lookup case

* Use Blake2b256Hash instead of SHA256

* agent/cache: update cache-clear endpoint; use bytes.NewReader instead (#6259)

* agent/cache: update cache-clear endpoint; use bytes.NewReader instead

* agent/cache: Fix TestCache_ComputeIndexID after switching to blake2b

* agent/cache: Only parse response body if it's non-nil (#6260)

* Differently disable agent address in the API client

* Remove DisableAgent
2019-02-19 16:53:29 -05:00
Vishal Nayak
4be79200e3
address some review back (#6237) 2019-02-15 13:40:03 -05:00
Jim Kalafut
9ea53eb438 Fix JWT end-to-end test (#6244) 2019-02-15 09:39:23 -08:00
Jim Kalafut
c96ad1b30a Fix agent test (#6243) 2019-02-15 09:27:57 -08:00
Chris Hoffman
4ad8ef777c fixing operator unseal test (#6241) 2019-02-15 09:06:37 -08:00
Chris Hoffman
ec015ec6bb fixing command server tests (#6242) 2019-02-15 09:06:20 -08:00
Calvin Leung Huang
6e96688719 cacememdb: add LeaseToken and TokenParent tests for GetByPrefix 2019-02-15 07:55:09 -08:00
vishalnayak
b2411f2a63 Fix cachememdb test 2019-02-15 10:34:19 -05:00
Vishal Nayak
e39a5f28df
Vault Agent Cache (#6220)
* vault-agent-cache: squashed 250+ commits

* Add proper token revocation validations to the tests

* Add more test cases

* Avoid leaking by not closing request/response bodies; add comments

* Fix revoke orphan use case; update tests

* Add CLI test for making request over unix socket

* agent/cache: remove namespace-related tests

* Strip-off the auto-auth token from the lookup response

* Output listener details along with configuration

* Add scheme to API address output

* leasecache: use IndexNameLease for prefix lease revocations

* Make CLI accept the fully qualified unix address

* export VAULT_AGENT_ADDR=unix://path/to/socket

* unix:/ to unix://
2019-02-14 20:10:36 -05:00
Martin
5dd50ef281 Prometheus support on v1/sys/metrics endpoint (#5308)
* initial commit for prometheus and sys/metrics support

* Throw an error if prometheusRetentionTime is 0,add prometheus in devmode

* return when format=prometheus is used and prom is disable

* parse prometheus_retention_time from string instead of int

* Initialize config.Telemetry if nil

* address PR issues

* add sys/metrics framework.Path in a factory

* Apply requiredMountTable entries's MountConfig to existing core table

* address pr comments

* enable prometheus sink by default

* Move Metric-related code in a separate metricsutil helper
2019-02-14 12:46:59 -08:00
Jim Kalafut
9408c3b8a2 Fix unit tests broken with JWT plugin update 2019-02-14 12:14:18 -08:00
Jeff Mitchell
c11d7189de
Don't add kv by default in non-dev scenarios (#6109)
It's retained for tests though since most rely on it.
2019-02-14 11:55:32 -08:00
Jeff Mitchell
8a59b0ca06 Remove deprecated CLI commands (#6112) 2019-02-14 11:54:47 -08:00
Calvin Leung Huang
4db655a026 Merge branch 'master-oss' into 1.1-beta 2019-02-11 14:47:22 -08:00
Jim Kalafut
5cb9a8a5fa
Create alias and command for OIDC (#6206) 2019-02-11 13:37:55 -08:00
Giacomo Tirabassi
75266af3d3 remove panicking and added usage (#6208) 2019-02-11 11:19:08 -08:00
Jeff Mitchell
8381a9fb16
If the log level isn't set, don't display "(not set)" (#6183) 2019-02-07 03:48:50 -05:00
Jeff Mitchell
20423f7082
Add allowed_response_headers (#6115) 2019-02-05 16:02:15 -05:00
Jeff Mitchell
42c2857598 Fix test build 2019-02-05 12:26:25 -05:00
Jeff Mitchell
f404e0acd2
Add the ability to print curl commands from CLI (#6113) 2019-02-01 17:13:51 -05:00
Brian Kassouf
b7f3823ad1
Updates to recovery keys (#6152) 2019-02-01 11:29:55 -08:00
Jeff Mitchell
e774948d24 Fix build 2019-01-23 16:52:06 -05:00
Seth Vargo
f0ab6b525e Continuously attempt to unseal if sealed keys are supported (#6039)
* Add helper for checking if an error is a fatal error

The double-double negative was really confusing, and this pattern is used a few places in Vault. This negates the double negative, making the devx a bit easier to follow.

* Check return value of UnsealWithStoredKeys in sys/init

* Return proper error types when attempting unseal with stored key

Prior to this commit, "nil" could have meant unsupported auto-unseal, a transient error, or success. This updates the function to return the correct error type, signaling to the caller whether they should retry or fail.

* Continuously attempt to unseal if sealed keys are supported

This fixes a bug that occurs on bootstrapping an initial cluster. Given a collection of Vault nodes and an initialized storage backend, they will all go into standby waiting for initialization. After one node is initialized, the other nodes had no mechanism by which they "re-check" to see if unseal keys are present. This adds a goroutine to the server command which continually waits for unseal keys to exist. It exits in the following conditions:

- the node is unsealed
- the node does not support stored keys
- a fatal error occurs (as defined by Vault)
- the server is shutting down

In all other situations, the routine wakes up at the specified interval and attempts to unseal with the stored keys.
2019-01-23 16:34:34 -05:00
Noel Cower
21e062d9d8 Merge all configuration fields (#6028)
This changes (*Config).Merge to merge all fields of a Config.
Previously, when merging Configs, some configuration fields were
ignored and completely lost, including APIAddr, ClusterAddr, and
a couple boolean fields. This only occurs when using multiple config
files and does not affect single config files (even when loading from
a directory -- Merge is only called after a second file is loaded).

- Fix APIAddr not being merged.
- Fix ClusterAddr not being merged.
- Fix DisablePrintableCheck not being merged.
- Fix DisableClustering not being merged. The DisableClusteringRaw
  value is also preserved so that it can be used in overrides for
  storage fields.
- Use merged top-level config as storage field overrides.
- Update config dir test fixtures to set some fields missed by
  (*Config).Merge previously.
2019-01-23 11:27:21 -05:00
nathan r. hruby
e8a2ac5fce add influx plugin (#6021) 2019-01-09 17:25:40 -08:00
Jim Kalafut
a54603039d Run goimports across the repository (#6010)
The result will still pass gofmtcheck and won't trigger additional
changes if someone isn't using goimports, but it will avoid the
piecemeal imports changes we've been seeing.
2019-01-08 16:48:57 -08:00
Pawel Gorczynski
fe0bb20c04 Extended info message regarding VAULT_ADDR env variable for Windows when running server -dev (#5970)
* Improved info message for Windows when running `server -dev`

Improvement for https://github.com/hashicorp/vault/issues/3405

* output updated to:

You may need to set the following environment variable:
PowerShell:
    $env:VAULT_ADDR="http://127.0.0.1:8200"
cmd.exe:
    set VAULT_ADDR=http://127.0.0.1:8200
2018-12-20 10:27:21 -08:00