* Since we want to use the Agent listener for #6384, move listener config
from top-level 'cache' block to new top-level 'listeners' block.
* Make agent config allow cache and listener blocks without auto-auth
configured.
Append call in form of `append(s)` has no effect,
it just returns `s`. Sometimes such invocation is a sign
of a programming error, so it's better to remove these.
Signed-off-by: Iskander Sharipov <quasilyte@gmail.com>
* Listener refactoring and file system permissions
* added listenerutil and move some common code there
* Added test for verifying socket file permissions
* Change default port of agent to 8200
* address review feedback
* Address review feedback
* Read socket options from listener config
* Added warning when init command uses auto unseal and -key-shares and -key-threshold was set
* Reworked
* Update command/operator_init.go
Co-Authored-By: michelvocks <michelvocks@gmail.com>
* agent/caching: proxy redirect and non-json responses; update tests
* agent/caching: do not wrap error responses as internal errors, simply proxy them back as-is
* minor refactoring of APIProxy.Send logic
* add test case to ensure error response is not wrapped
* Add ability to migrate autoseal to autoseal
This adds the ability to migrate from shamir to autoseal, autoseal to
shamir, or autoseal to autoseal, by allowing multiple seal stanzas. A
disabled stanza will be used as the config being migrated from; this can
also be used to provide an unwrap seal on ent over multiple unseals.
A new test is added to ensure that autoseal to autoseal works as
expected.
* Fix test
* Provide default shamir info if not given in config
* Linting feedback
* Remove context var that isn't used
* Don't run auto unseal watcher when in migration, and move SetCores to SetSealsForMigration func
* Slight logic cleanup
* Fix test build and fix bug
* Updates
* remove GetRecoveryKey function
* Implemented a warning when tls_cipher_suites includes only cipher suites which are not supprted by the HTTP/2 spec
* Added test for cipher suites
* Added hard fail on startup when all defined cipher suites are blacklisted. Added warning when some ciphers are blacklisted.
* Replaced hard failure with warning. Removed bad cipher util function and replaced it by external library.
* Added missing dependency. Fixed renaming of package name.
* Adding Transit Autoseal
* adding tests
* adding more tests
* updating seal info
* send a value to test and set current key id
* updating message
* cleanup
* Adding tls config, addressing some feedback
* adding tls testing
* renaming config fields for tls
* agent/caching: enable caching of leases generated by autoauth token
* add test for auth/token/create path
* update error message log
* Some minor updates
* add sleep timer for renewal logic to process
* Properly compute auto-auth token
* Use inmem sink to track auto-auth token
* update debug statement
* Only add inmem sink if using auto-auth token is allowed
* vault-agent-cache: squashed 250+ commits
* Add proper token revocation validations to the tests
* Add more test cases
* Avoid leaking by not closing request/response bodies; add comments
* Fix revoke orphan use case; update tests
* Add CLI test for making request over unix socket
* agent/cache: remove namespace-related tests
* Strip-off the auto-auth token from the lookup response
* Output listener details along with configuration
* Add scheme to API address output
* leasecache: use IndexNameLease for prefix lease revocations
* Make CLI accept the fully qualified unix address
* export VAULT_AGENT_ADDR=unix://path/to/socket
* unix:/ to unix://
* initial commit for prometheus and sys/metrics support
* Throw an error if prometheusRetentionTime is 0,add prometheus in devmode
* return when format=prometheus is used and prom is disable
* parse prometheus_retention_time from string instead of int
* Initialize config.Telemetry if nil
* address PR issues
* add sys/metrics framework.Path in a factory
* Apply requiredMountTable entries's MountConfig to existing core table
* address pr comments
* enable prometheus sink by default
* Move Metric-related code in a separate metricsutil helper
* Add helper for checking if an error is a fatal error
The double-double negative was really confusing, and this pattern is used a few places in Vault. This negates the double negative, making the devx a bit easier to follow.
* Check return value of UnsealWithStoredKeys in sys/init
* Return proper error types when attempting unseal with stored key
Prior to this commit, "nil" could have meant unsupported auto-unseal, a transient error, or success. This updates the function to return the correct error type, signaling to the caller whether they should retry or fail.
* Continuously attempt to unseal if sealed keys are supported
This fixes a bug that occurs on bootstrapping an initial cluster. Given a collection of Vault nodes and an initialized storage backend, they will all go into standby waiting for initialization. After one node is initialized, the other nodes had no mechanism by which they "re-check" to see if unseal keys are present. This adds a goroutine to the server command which continually waits for unseal keys to exist. It exits in the following conditions:
- the node is unsealed
- the node does not support stored keys
- a fatal error occurs (as defined by Vault)
- the server is shutting down
In all other situations, the routine wakes up at the specified interval and attempts to unseal with the stored keys.
This changes (*Config).Merge to merge all fields of a Config.
Previously, when merging Configs, some configuration fields were
ignored and completely lost, including APIAddr, ClusterAddr, and
a couple boolean fields. This only occurs when using multiple config
files and does not affect single config files (even when loading from
a directory -- Merge is only called after a second file is loaded).
- Fix APIAddr not being merged.
- Fix ClusterAddr not being merged.
- Fix DisablePrintableCheck not being merged.
- Fix DisableClustering not being merged. The DisableClusteringRaw
value is also preserved so that it can be used in overrides for
storage fields.
- Use merged top-level config as storage field overrides.
- Update config dir test fixtures to set some fields missed by
(*Config).Merge previously.
The result will still pass gofmtcheck and won't trigger additional
changes if someone isn't using goimports, but it will avoid the
piecemeal imports changes we've been seeing.
* re-add plugin name for backwards compat
* add plugin name to table for backwards compat
* conditionally set the plugin name
* Update command/auth_list.go
Co-Authored-By: tyrannosaurus-becks <beccapetrin@posteo.net>
* Update command/secrets_list.go
Co-Authored-By: tyrannosaurus-becks <beccapetrin@posteo.net>
* update auth and secrets list commands
* add comments regarding deprecation
* Add support for custom JWT path in Agent: kubernetes auth
- add support for "token_path" configuration
- add a reader for mocking in tests
* add documentation for token_path
* Refactor mount tune to support upsert options values and unset options.
* Do not allow unsetting options map
* add secret tune version regression test
* Only accept valid options version
* s/meVersion/optVersion/
* Continue on plugin registration error in dev mode
* Continue only on unknown type error
* Continue only on unknown type error
* Print plugin registration error on exit
Co-Authored-By: calvn <cleung2010@gmail.com>
* Support registering plugin with name only
* Make RegisterPlugin backwards compatible
* Add CLI backwards compat command to plugin info and deregister
* Add server-side deprecation warnings if old read/dereg API endpoints are called
* Address feedback
Running 'vault auth' with no parameters was panicking:
panic: assignment to entry in nil map
github.com/hashicorp/vault/command/login.go:255 +0xdee
Now it will show help.
The addition of CheckMigration to the server startup process means
that physical backends in this test need to be able to respond to Get() without error.
We support this in the API as of 0.10.2 so read should support it too.
Trivially tested with some log info:
`core: data: data="map[string]interface {}{"zip":[]string{"zap", "zap2"}}"`
This allows it to authenticate once, then exit once all sinks have
reported success. Useful for things like an init container vs. a
sidecard container.
Also adds command-level testing of it.
* Add request timeouts in normal request path and to expirations
* Add ability to adjust default max request duration
* Some test fixes
* Ensure tests have defaults set for max request duration
* Add context cancel checking to inmem/file
* Fix tests
* Fix tests
* Set default max request duration to basically infinity for this release for BC
* Address feedback
* Tackle #4929 a different way
This turns c.sealed into an atomic, which allows us to call sealInternal
without a lock. By doing so we can better control lock grabbing when a
condition causing the standby loop to get out of active happens. This
encapsulates that logic into two distinct pieces (although they could
be combined into one), and makes lock guarding more understandable.
* Re-add context canceling to the non-HA version of sealInternal
* Return explicitly after stopCh triggered
Making this configurable is useful for windows users which may not be
using the default `ssh` executable. It also means that users can point to a
specify SSH executable if multiple are available.
* Add 'plugin list' command
* Add 'plugin register' command
* Add 'plugin deregister' command
* Use a shared plugin helper
* Add 'plugin read' command
* Rename to plugin info
* Add base plugin for help text
* Fix arg ordering
* Add docs
* Rearrange to alphabetize
* Fix arg ordering in example
* Don't use "sudo" in command description
* Add description flag to secrets and auth tune subcommands
* Allow empty description to be provided in secret and auth mount tune
* Use flagNameDescription
This change makes it so that if a lease is revoked through user action,
we set the expiration time to now and update pending, just as we do with
tokens. This allows the normal retry logic to apply in these cases as
well, instead of just erroring out immediately. The idea being that once
you tell Vault to revoke something it should keep doing its darndest to
actually make that happen.
* Allow lease_duration to be pulled out with -field
This also provides an easy way to verify that when -field is used we
don't string format the value.
This also changes the human string helper to accept more than one type
of incoming int.
* Address review feedback
* Allow max request size to be user-specified
This turned out to be way more impactful than I'd expected because I
felt like the right granularity was per-listener, since an org may want
to treat external clients differently from internal clients. It's pretty
straightforward though.
This also introduces actually using request contexts for values, which
so far we have not done (using our own logical.Request struct instead),
but this allows non-logical methods to still get this benefit.
* Switch to ioutil.ReadAll()
In current Vault server EncodedToken will always be populated regardless
of type (root, DR), so prioritize that, and properly refer to it as
Encoded Token instead of Root Token.
Additionally refer to the nonce as the Operation nonce instead of the
Root generation operation nonce since it's used for both strategies.
* Add an idle timeout for the server
Because tidy operations can be long-running, this also changes all tidy
operations to behave the same operationally (kick off the process, get a
warning back, log errors to server log) and makes them all run in a
goroutine.
This could mean a sort of hard stop if Vault gets sealed because the
function won't have the read lock. This should generally be okay
(running tidy again should pick back up where it left off), but future
work could use cleanup funcs to trigger the functions to stop.
* Fix up tidy test
* Add deadline to cluster connections and an idle timeout to the cluster server, plus add readheader/read timeout to api server
* Add `kv rollback`
Like `kv patch` this is more of a helper than anything else; it provides
a single command to fetch the current version (for CAS), read the
version you want to roll back to, and set it as the new version (using
CAS for safety).
This can be used when errors are happening early on to avoid them being
swallowed by logGate.
This also does a bit of cleanup of format env var checking --
helper/logging internally looks for this so it was totally unnecessary
since moving to hclog.
* Allow vault ssh to accept ssh commands in any ssh compatible format
Previously vault ssh required ssh commands to be in the format
`username@hostname <flags> command`. While this works just fine for human
users this breaks a lot of automation workflows and is not compatible
with the options that the ssh client supports.
Motivation
We currently run ansible which uses vault ssh to connect to hosts.
Ansible generates ssh commands with the format `ssh <flags> -o User=username hostname
command`. While this is a valid ssh command it currently breaks with
vault because vault expects the format to be `username@hostname`. To work
around this we currently use a wrapper script to parse the correct username being set
by ansible and translate this into a vault ssh compatible `username@hostname` format
Changes
* You can now specify arguments in any order that ssh client allows. All
arguments are passed directly to the ssh command and the format isn't
modified in any way.
* The username and port are parsed from the specified ssh command. It
will accept all of the options supported by the ssh command and also
will properly prefer `-p` and `user@` if both options are specified.
* The ssh port is only added from the vault credentials if it hasn't
been specified on the command line
* This changes the way policies are reported in audit logs.
Previously, only policies tied to tokens would be reported. This could
make it difficult to perform after-the-fact analysis based on both the
initial response entry and further requests. Now, the full set of
applicable policies from both the token and any derived policies from
Identity are reported.
To keep things consistent, token authentications now also return the
full set of policies in api.Secret.Auth responses, so this both makes it
easier for users to understand their actual full set, and it matches
what the audit logs now report.
* Fix writing to KVv2 root via `kv put`
The check that adds the API path wasn't taking into account the root,
e.g. if it's mounted at `kv`, `kv` and `kv/` would end up creating an
extra copy of the mount path in front, leading to paths like
`kv/data/kv`.
* Output warnings if they come back and fix a panic in metadata_get
* Also add to metadata put/delete
Don't set a default value for the UserKnownHostsFile flag.
Only append `-o UserKnownHostsFile` to the ssh command if it
has been specified by the user or vault ssh has set it based on another
flag (such as flagHostKeyMountPoint)
Fixes https://github.com/hashicorp/vault/issues/4672
This is implementing the same fix that was added for the CA mode for vault
ssh in https://github.com/hashicorp/vault/pull/3922
Using the IP address caused `Host` entries in the ssh_config to not
match anymore meaning you would need to hardcode all of your IP
addresses in your ssh config instead of using DNS to connect to hosts
This takes place in two parts, since working on this exposed an issue
with response wrapping when there is a raw body set. The changes are (in
diff order):
* A CurrentWrappingLookupFunc has been added to return the current
value. This is necessary for the lookahead call since we don't want the
lookahead call to be wrapped.
* Support for unwrapping < 0.6.2 tokens via the API/CLI has been
removed, because we now have backends returning 404s with data and can't
rely on the 404 trick. These can still be read manually via
cubbyhole/response.
* KV preflight version request now ensures that its calls is not
wrapped, and restores any given function after.
* When responding with a raw body, instead of always base64-decoding a
string value and erroring on failure, on failure we assume that it
simply wasn't a base64-encoded value and use it as is.
* A test that fails on master and works now that ensures that raw body
responses that are wrapped and then unwrapped return the expected
values.
* A flag for response data that indicates to the wrapping handling that
the data contained therein is already JSON decoded (more later).
* RespondWithStatusCode now defaults to a string so that the value is
HMAC'd during audit. The function always JSON encodes the body, so
before now it was always returning []byte which would skip HMACing. We
don't know what's in the data, so this is a "better safe than sorry"
issue. If different behavior is needed, backends can always manually
populate the data instead of relying on the helper function.
* We now check unwrapped data after unwrapping to see if there were raw
flags. If so, we try to detect whether the value can be unbase64'd. The
reason is that if it can it was probably originally a []byte and
shouldn't be audit HMAC'd; if not, it was probably originally a string
and should be. In either case, we then set the value as the raw body and
hit the flag indicating that it's already been JSON decoded so not to
try again before auditing. Doing it this way ensures the right typing.
* There is now a check to see if the data coming from unwrapping is
already JSON decoded and if so the decoding is skipped before setting
the audit response.
* Update kv command to use a preflight check
* Make the existing ui endpoint return the allowed mounts
* Add kv subcommand tests
* Enable `-field` in `vault kv get/put` (#4426)
* Enable `-field` in `vault kv get/put`
Fixes#4424
* Unify nil value handling
* Use preflight helper
* Update vkv plugin
* Add all the mount info when authenticated
* Add fix the error message on put
* add metadata test
* No need to sort the capabilities
* Remove the kv client header
* kv patch command (#4432)
* Fix test
* Fix tests
* Use permission denied instead of entity disabled
This always specifies a color UI, but explicitly marks the output as
noncolorable if we don't want color. This allows getting rid of our
hacky Output function in favor of cli's normal functions.
* Allow returning list information and other data in 404s.
On read it'll output data and/or warnings on a 404 if they exist. On
list, the same behavior; the actual 'vault list' command doesn't change
behavior though in terms of output unless there are no actual keys (so
it doesn't just magically show other data).
This corrects some assumptions in response_util and wrapping.go; it also
corrects a few places in the latter where it could leak a (useless)
token in some error cases.
* Use same 404 logic in delete/put too
* Add the same secret parsing logic to the KV request functions
* adding UI handlers and UI header configuration
* forcing specific static headers
* properly getting UI config value from config/environment
* fixing formatting in stub UI text
* use http.Header
* case-insensitive X-Vault header check
* fixing var name
* wrap both stubbed and real UI in header handler
* adding test for >1 keys
* logbridge with hclog and identical output
* Initial search & replace
This compiles, but there is a fair amount of TODO
and commented out code, especially around the
plugin logclient/logserver code.
* strip logbridge
* fix majority of tests
* update logxi aliases
* WIP fixing tests
* more test fixes
* Update test to hclog
* Fix format
* Rename hclog -> log
* WIP making hclog and logxi love each other
* update logger_test.go
* clean up merged comments
* Replace RawLogger interface with a Logger
* Add some logger names
* Replace Trace with Debug
* update builtin logical logging patterns
* Fix build errors
* More log updates
* update log approach in command and builtin
* More log updates
* update helper, http, and logical directories
* Update loggers
* Log updates
* Update logging
* Update logging
* Update logging
* Update logging
* update logging in physical
* prefixing and lowercase
* Update logging
* Move phyisical logging name to server command
* Fix som tests
* address jims feedback so far
* incorporate brians feedback so far
* strip comments
* move vault.go to logging package
* update Debug to Trace
* Update go-plugin deps
* Update logging based on review comments
* Updates from review
* Unvendor logxi
* Remove null_logger.go
Some commands didn't setup a local test server since they didn't need
it. Other commands didn't setup a local test server because Seth forgot.
Long story short, I kept seeing weird requests to my Vault server when I
ran tests, and that should never happen. This ensures all test requests
will go to a test Vault instance.
Benchmarks show this adds 0.4s to the command test suite.
* Add more cli subcommands
* Add metadata commands
* Add more subcommands
* Update cli
* Move archive commands to delete
* Add helpers for making http calls to the kv backend
* rename cli header
* Format the various maps from kv
* Add list command
* Update help text
* Add a command to enable versioning on a backend
* Rename enable-versions command
* Some review feedback
* Fix listing of top level keys
* Fix issue when metadata is nil
* Add test for lising top level keys
* Fix some typos
* Add a note about deleting all versions
* Add some requirements for versioned k/v
* Add a warning message when an upgrade is triggered
* Add path help values
* Make the kv header a const
* Add the uid to mount entry instead of options map
* Pass the backend aware uuid to the mounts and plugins
* Fix comment
* Add options to secret/auth enable and tune CLI commands (#4170)
* Switch mount/tune options to use TypeKVPairs (#4171)
* switching options to TypeKVPairs, adding bool parse for versioned flag
* flipping bool check
* Fix leases coming back from non-leased pluin kv store
* add a test for updating mount options
* Fix tests
* Add audit hmac values to AuthConfigInput and AuthConfigOutput, fix docs
* docs: Add ttl params to auth enable endpoint
* Rewording of go string to simply string
* Add audit hmac keys as CLI flags on auth/secrets enable
* Fix copypasta mistake
* WIP on auth-list endpoint
* Rename variable to be singular, add CLI flag, show value in auth and secrets list
* Add audit hmac keys to auth and secrets list
* Only set config values if they exist
* Fix http sys/auth tests
* More auth plugin_name test fixes
* Rename tag internal_ui_show_mount to _ui_show_mount
* Add tests
* Make endpoint unauthed
* Rename field to listing_visibility
* Add listing-visibility to cli tune commands
* Use ListingVisiblityType
* Fix type conversion
* Do not actually change token's value on testHttpGet
* Remove unused ListingVisibilityAuth, use const in pathInternalUIMountsRead
* Add audit hmac values to AuthConfigInput and AuthConfigOutput, fix docs
* docs: Add ttl params to auth enable endpoint
* Rewording of go string to simply string
* Add audit hmac keys as CLI flags on auth/secrets enable
* Fix copypasta mistake
* Add audit hmac keys to auth and secrets list
* Only set config values if they exist
* Fix http sys/auth tests
* More auth plugin_name test fixes
* Pass API values into MountEntry's config when creating auth/secrets mount
* Update usage wording
```
:~# vault status
Key Value
--- -----
Seal Type shamir
Sealed false
Total Shares 8
Threshold 2
Version 0.9.5
Cluster Name vault-cluster-8c85f1aa
Cluster ID aaaaaaaa-1111-2222-3333-444444444444
HA Enabled true
HA Cluster https://10.0.0.1:8201
HA Mode standby
Active Node Address: https://10.0.0.1:8200
```
`Active Node Address:` - is the only one with a colon at the end.
This PR fix that output style issue.
* Create a new command/config subpackage.
This PR extracts the functions associated with loading and parsing
configs, and the DefaultTokenHelper, into a command/config subpackage,
just like TokenHelpers are in the command/token subpackage. The goal is
to allow other clients (in this case, the Vault and Nomad Terraform
providers, but in theory any client that wants to lean on Vault's
default behaviour) to reuse this logic and not drift from Vault, without
vendoring the entirety of Vault.
To retain backwards compatibility, I didn't remove any functions from
the command package; I just copied them into the command/config package,
and update the functions in the command package to call through to the
config package.
* Allow formatted data when using -field and -format together.
As a special case, allows "data" to be passed in to get the entire data
struct output.
* If data exists in the output map use that instead when special casing
This PR adds a new Storage Backend for Triton's Object Storage - Manta
```
make testacc TEST=./physical/manta
==> Checking that code complies with gofmt requirements...
==> Checking that build is using go version >= 1.9.1...
go generate
VAULT_ACC=1 go test -tags='vault' ./physical/manta -v -timeout 45m
=== RUN TestMantaBackend
--- PASS: TestMantaBackend (61.18s)
PASS
ok github.com/hashicorp/vault/physical/manta 61.210s
```
Manta behaves differently to how S3 works - it has no such concepts of Buckets - it is merely a filesystem style object store
Therefore, we have chosen the approach of when writing a secret `foo` it will actually map (on disk) as foo/.vault_value
The reason for this is because if we write the secret `foo/bar` and then try and Delete a key using the name `foo` then Manta
will complain that the folder is not empty because `foo/bar` exists. Therefore, `foo/bar` is written as `foo/bar/.vault_value`
The value of the key is *always* written to a directory tree of the name and put in a `.vault_value` file.
* Use Colored UI if stdout is a tty
* Add format options to operator unseal
* Add format test on operator unseal
* Add -no-color output flag, and use BasicUi if no-color flag is provided
* Move seal status formatting logic to OutputSealStatus
* Apply no-color to warnings from DeprecatedCommands as well
* Add OutputWithFormat to support arbitrary data, add format option to auth list
* Add ability to output arbitrary list data on TableFormatter
* Clear up switch logic on format
* Add format option for list-related commands
* Add format option to rest of commands that returns a client API response
* Remove initOutputYAML and initOutputJSON, and use OutputWithFormat instead
* Remove outputAsYAML and outputAsJSON, and use OutputWithFormat instead
* Remove -no-color flag, use env var exclusively to toggle colored output
* Fix compile
* Remove -no-color flag in main.go
* Add missing FlagSetOutputFormat
* Fix generate-root/decode test
* Migrate init functions to main.go
* Add no-color flag back as hidden
* Handle non-supported data types for TableFormatter.OutputList
* Pull formatting much further up to remove the need to use c.flagFormat (#3950)
* Pull formatting much further up to remove the need to use c.flagFormat
Also remove OutputWithFormat as the logic can cause issues.
* Use const for env var
* Minor updates
* Remove unnecessary check
* Fix SSH output and some tests
* Fix tests
* Make race detector not run on generate root since it kills Travis these days
* Update docs
* Update docs
* Address review feedback
* Handle --format as well as -format
* Use SHA2-256 hash with prefix to upgrade the paths
* test the SHA1 upgrade to SHA256
* Remove hash identifier and the delimiter; use 's' instead
* Added API test to verify the correctness of the fix
* Fix broken test
* remove unneeded test
* external identity groups
* add local LDAP groups as well to group aliases
* add group aliases for okta credential backend
* Fix panic in tests
* fix build failure
* remove duplicated struct tag
* add test steps to test out removal of group member during renewals
* Add comment for having a prefix check in router
* fix tests
* s/parent_id/canonical_id
* s/parent/canonical in comments and errors
* Redo the API client quite a bit to make the behavior of NewClient more
predictable and add locking to make it safer to use with Clone() and if
multiple goroutines for some reason decide to change things.
Along the way I discovered that currently, the x/net/http2 package is
broke with the built-in h2 support in released Go. For those using
DefaultConfig (the vast majority of cases) this will be a non-event.
Others can manually call http2.ConfigureTransport as needed. We should
keep an eye on commits on that repo and consider more updates before
release. Alternately we could go back revisions but miss out on bug
fixes; my theory is that this is not a purposeful break and I'll be
following up on this in the Go issue tracker.
In a few tests that don't use NewTestCluster, either for legacy or other
reasons, ensure that http2.ConfigureTransport is called.
* Use tls config cloning
* Don't http2.ConfigureServer anymore as current Go seems to work properly without requiring the http2 package
* Address feedback
This removes all references I could find to:
- credential provider
- authentication backend
- authentication provider
- auth provider
- auth backend
in favor of the unified:
- auth method
If the value is being "piped", we don't print colors or the newline character at the end. If it's not, we still give users pretty when selecting a raw field/value.
* disable raw endpoint by default
* adding docs
* config option raw -> raw_storage_endpoint
* docs updates
* adding listing on raw endpoint
* reworking tests for enabled raw endpoints
* root protecting base raw endpoint
* Add basic autocompletion
* Add autocomplete to some common commands
* Autocomplete the generate-root flags
* Add information about autocomplete to the docs
Previously we lowercased names on ingress but not on lookup or delete
which could cause unexpected results. Now, just unilaterally lowercase
policy names on write and delete. On get, to avoid the performance hit
of always lowercasing when not necessary since it's in the critical
path, we have a minor optimization -- we check the LRU first before
normalizing. For tokens, because they're already normalized when adding
policies during creation, this should always work; it might just be
slower for API calls.
Fixes#3187
1. The current implementation of the SSH command is heavily tied to the
assumptions of OTP/dynamic key types. The SSH CA backend is
fundamentally a different approach to login and authentication. As a
result, there was some restructuring of existing methods to share more
code and state.
2. Each authentication method (ca, otp, dynamic) are now fully-contained
in their own handle* function.
3. -mode and -role are going to be required for SSH CA, and I don't
think the magical UX (and overhead) of guessing them is a good UX. It's
confusing as to which role and how Vault guesses. We can reduce 66% of
the API calls and add more declaration to the CLI by making -mode and
-role required. This commit adds warnings for that deprecation, but
these values are both required for CA type authentication.
4. The principal and extensions are currently fixed, and I personally
believe that's good enough for the first pass at this. Until we
understand what configuration options users will want, I think we should
ship with all the local extensions enabled. Users who don't want that
can generate the key themselves directly (current behavior) or submit
PRs to make the map of extensions customizable.
5. Host key checking for the CA backend is not currently implemented.
It's not strictly required at setup, so I need to think about whether it
belongs here.
This is not ready for merge, but it's ready for early review.
From the sshpass manpage:
> The -p option should be considered the least secure of all of sshpass's options. All system users can see the password in the command line with a simple "ps" command. Sshpass makes a minimal attempt to hide the password, but such attempts are doomed to create race conditions without actually solving the problem. Users of sshpass are encouraged to use one of the other password passing techniques, which are all more secure.
This PR changes the sshpass behavior to execute a subprocess with the
SSHPASS envvar (which is generally regarded as more secure) than using
the -p option.
* Store original request path in WrapInfo as CreationPath
* Add wrapping_token_creation_path to CLI output
* Add CreationPath to AuditResponseWrapInfo
* Fix tests
* Add and fix tests, update API docs with new sample responses
* Add backend plugin changes
* Fix totp backend plugin tests
* Fix logical/plugin InvalidateKey test
* Fix plugin catalog CRUD test, fix NoopBackend
* Clean up commented code block
* Fix system backend mount test
* Set plugin_name to omitempty, fix handleMountTable config parsing
* Clean up comments, keep shim connections alive until cleanup
* Include pluginClient, disallow LookupPlugin call from within a plugin
* Add wrapper around backendPluginClient for proper cleanup
* Add logger shim tests
* Add logger, storage, and system shim tests
* Use pointer receivers for system view shim
* Use plugin name if no path is provided on mount
* Enable plugins for auth backends
* Add backend type attribute, move builtin/plugin/package
* Fix merge conflict
* Fix missing plugin name in mount config
* Add integration tests on enabling auth backend plugins
* Remove dependency cycle on mock-plugin
* Add passthrough backend plugin, use logical.BackendType to determine lease generation
* Remove vault package dependency on passthrough package
* Add basic impl test for passthrough plugin
* Incorporate feedback; set b.backend after shims creation on backendPluginServer
* Fix totp plugin test
* Add plugin backends docs
* Fix tests
* Fix builtin/plugin tests
* Remove flatten from PluginRunner fields
* Move mock plugin to logical/plugin, remove totp and passthrough plugins
* Move pluginMap into newPluginClient
* Do not create storage RPC connection on HandleRequest and HandleExistenceCheck
* Change shim logger's Fatal to no-op
* Change BackendType to uint32, match UX backend types
* Change framework.Backend Setup signature
* Add Setup func to logical.Backend interface
* Move OptionallyEnableMlock call into plugin.Serve, update docs and comments
* Remove commented var in plugin package
* RegisterLicense on logical.Backend interface (#3017)
* Add RegisterLicense to logical.Backend interface
* Update RegisterLicense to use callback func on framework.Backend
* Refactor framework.Backend.RegisterLicense
* plugin: Prevent plugin.SystemViewClient.ResponseWrapData from getting JWTs
* plugin: Revert BackendType to remove TypePassthrough and related references
* Fix typo in plugin backends docs
* exclude /sys/leases/renew from registering with expiration manager
* adding sys/leases/renew to return full secret object, adding tests to catch renew errors
* Normalize "X arguments expected" messages
* Use "Vault" when referring to the product and "vault" when referring to an instance of the product
* Various minor tweaks to improve readability and/or provide clarity
Adds HUP support for audit log files to close and reopen. This makes it
much easier to deal with normal log rotation methods.
As part of testing this I noticed that HUP and other items that come out
of command/server.go are going to stderr, which is where our normal log
lines go. This isn't so much problematic with our normal output but as
we officially move to supporting other formats this can cause
interleaving issues, so I moved those to stdout instead.
* Provide base64 keys in addition to hex encoded.
Accept these at unseal/rekey time.
Also fix a bug where backup would not be honored when doing a rekey with
no operation currently ongoing.
This makes it easier to understand the expected lifetime without a
lookup call that uses the single use left on the token.
This also adds a couple of safety checks and for JSON uses int, rather
than int64, for the TTL for the wrapped token.
Vault will now register itself with Consul. The active node can be found using `active.vault.service.consul`. All standby vaults are available via `standby.vault.service.consul`. All unsealed vaults are considered healthy and available via `vault.service.consul`. Change in status and registration is event driven and should happen at the speed of a write to Consul (~network RTT + ~1x fsync(2)).
Healthy/active:
```
curl -X GET 'http://127.0.0.1:8500/v1/health/service/vault?pretty' && echo;
[
{
"Node": {
"Node": "vm1",
"Address": "127.0.0.1",
"TaggedAddresses": {
"wan": "127.0.0.1"
},
"CreateIndex": 3,
"ModifyIndex": 20
},
"Service": {
"ID": "vault:127.0.0.1:8200",
"Service": "vault",
"Tags": [
"active"
],
"Address": "127.0.0.1",
"Port": 8200,
"EnableTagOverride": false,
"CreateIndex": 17,
"ModifyIndex": 20
},
"Checks": [
{
"Node": "vm1",
"CheckID": "serfHealth",
"Name": "Serf Health Status",
"Status": "passing",
"Notes": "",
"Output": "Agent alive and reachable",
"ServiceID": "",
"ServiceName": "",
"CreateIndex": 3,
"ModifyIndex": 3
},
{
"Node": "vm1",
"CheckID": "vault-sealed-check",
"Name": "Vault Sealed Status",
"Status": "passing",
"Notes": "Vault service is healthy when Vault is in an unsealed status and can become an active Vault server",
"Output": "",
"ServiceID": "vault:127.0.0.1:8200",
"ServiceName": "vault",
"CreateIndex": 19,
"ModifyIndex": 19
}
]
}
]
```
Healthy/standby:
```
[snip]
"Service": {
"ID": "vault:127.0.0.2:8200",
"Service": "vault",
"Tags": [
"standby"
],
"Address": "127.0.0.2",
"Port": 8200,
"EnableTagOverride": false,
"CreateIndex": 17,
"ModifyIndex": 20
},
"Checks": [
{
"Node": "vm2",
"CheckID": "serfHealth",
"Name": "Serf Health Status",
"Status": "passing",
"Notes": "",
"Output": "Agent alive and reachable",
"ServiceID": "",
"ServiceName": "",
"CreateIndex": 3,
"ModifyIndex": 3
},
{
"Node": "vm2",
"CheckID": "vault-sealed-check",
"Name": "Vault Sealed Status",
"Status": "passing",
"Notes": "Vault service is healthy when Vault is in an unsealed status and can become an active Vault server",
"Output": "",
"ServiceID": "vault:127.0.0.2:8200",
"ServiceName": "vault",
"CreateIndex": 19,
"ModifyIndex": 19
}
]
}
]
```
Sealed:
```
"Checks": [
{
"Node": "vm2",
"CheckID": "serfHealth",
"Name": "Serf Health Status",
"Status": "passing",
"Notes": "",
"Output": "Agent alive and reachable",
"ServiceID": "",
"ServiceName": "",
"CreateIndex": 3,
"ModifyIndex": 3
},
{
"Node": "vm2",
"CheckID": "vault-sealed-check",
"Name": "Vault Sealed Status",
"Status": "critical",
"Notes": "Vault service is healthy when Vault is in an unsealed status and can become an active Vault server",
"Output": "Vault Sealed",
"ServiceID": "vault:127.0.0.2:8200",
"ServiceName": "vault",
"CreateIndex": 19,
"ModifyIndex": 38
}
]
```
No signal handler was setup to receive SIGINT. I didn't investigate to
see if signal(2) mask was setup (ala `SIG_IGN`) or if sigprocmask(2) is
being used, but in either case, the correct behavior is to capture and
treat SIGINT the same as SIGTERM. At some point in the future these two
signals may affect the running process differently, but we will clarify
that difference in the future.
One thing that has been a point of confusion for users is Vault's
response when deleting a key that does not actually exist in the system.
For example, consider:
$ vault delete secret/foo
Success! Deleted 'secret/foo'
This message is misleading if the secret does not exist, especially if
the same command is run twice in a row.
Obviously the reason for this is clear - returning an error if a secret
does not exist would reveal the existence of a secret (the same reason
everything on S3 is a 403 or why GitHub repos 404 instead of 403 if you
do not have permission to view them).
I think we can make the UX a little bit better by adding just a few
words to the output:
$ vault delete secret/foo
Success! Deleted 'secret/foo' if it existed
This makes it clear that the operation was only performed if the secret
existed, but it does not reveal any more information.
This came up twice, in two different training courses. The UX is a
little confusing here on the CLI. Users are used to running:
$ vault auth abcd-1234...
So when they auth using a method, the output leads them to believe the
need to "re-auth" as the generated token:
$ vault auth -method=userpass username=foo password=bar
Successfully authenticated!
token: defg-5678...
A number of users then run:
$ vault auth defg-5678
I've added some helpful text to hint this is not required if the method
is not "token".
This was a flawed test. Previously the test passed in a fixture that
corresponded to a CLI config file, not an actual policy. The test
_should_ have been failing, but it wasn't. This commit adds a new
fixture.
This does NOT apply to the backend config, since each backend config
could have a variation of options that differ based off of the
configured backend itself. This may be an optimization that can be made
in the future, but I think each backend should be responsible for
performing its own configuration validation instead of overloading the
config itself with this functionality.
In some situations, it can be impossible to revoke leases (for instance,
if someone has gone and manually removed users created by Vault). This
can not only cause Vault to cycle trying to revoke them, but it also
prevents mounts from being unmounted, leaving them in a tainted state
where the only operations allowed are to revoke (or rollback), which
will never successfully complete.
This adds a new endpoint that works similarly to `revoke-prefix` but
ignores errors coming from a backend upon revocation (it does not ignore
errors coming from within the expiration manager, such as errors
accessing the data store). This can be used to force Vault to abandon
leases.
Like `revoke-prefix`, this is a very sensitive operation and requires
`sudo`. It is implemented as a separate endpoint, rather than an
argument to `revoke-prefix`, to ensure that control can be delegated
appropriately, as even most administrators should not normally have
this privilege.
Fixes#1135
renew-self endpoint. Otherwise it will use the renew endpoint, even if
the token matches the client token.
Adds an -increment flag to allow increments even with no token passed
in.
Fixes#1150
This endpoint causes the node it's hit to step down from active duty.
It's a noop if the node isn't active or not running in HA mode. The node
will wait one second before attempting to reacquire the lock, to give
other nodes a chance to grab it.
Fixes#1093
auth-enable command help in the "Auth Enable Options" is suggesting
the usage of a non-existing command called 'auth-list' instead of
the correct one "auth -methods"
This change is almost perfectly compatible with the existing code,
except it's a little shorter because it uses a list of a available
formatters that must implement a `command.Formatter` interface.
Also added some basic formatting tests.
