* upgrade hcl dependency on api pkg
This upgrades the hcl dependency for the API pkg,
and adapts its usage so users of our API pkg are
not affected. There's no good way of communicating
a warning via a library call so we don't.
The tokenHelper which is used by all Vault CLI
commands in order to create the Vault client, as
well as directly used by the login and server
commands, is implemented on the api pkg, so this
upgrade also affects all of those commands. Seems
like this was only moved to the api pkg because
the Terraform provider uses it, and I thought
creating a full copy of all those files back under
command would be too much spaghetti.
Also leaving some TODOs to make next deprecation
steps easier.
* upgrade hcl dependency in vault and sdk pkgs
* upgrade hcl dependency in vault and sdk pkgs
* add CLI warnings to commands that take a config
- vault agent (unit test on CMD warning)
- vault proxy (unit test on CMD warning)
- vault server (no test for the warning)
- vault operator diagnose (no tests at all, uses the
same function as vault server
* ignore duplicates on ParseKMSes function
* Extend policy parsing functions and warn on policy store
* Add warning on policy fmt with duplicate attributes
* Add warnings when creating/updating policy with duplicate HCL attrs
* Add log warning when switchedGetPolicy finds duplicate attrs
Following operations can trigger this warning when they run into a policy
with duplicate attributes:
* replication filtered path namespaces invalidation
* policy read API
* building an ACL (for many different purposes like most authZ operations)
* looking up DR token policies
* creating a token with named policies
* when caching the policies for all namespaces during unseal
* Print log warnings when token inline policy has duplicate attrs
No unit tests on these as new test infra would have to be built on all.
Operations affected, which will now print a log warning when the retrieved
token has an inline policy with duplicate attributes:
* capabilities endpoints in sys mount
* handing events under a subscription with a token with duplicate
attrs in inline policies
* token used to create another token has duplicate attrs in inline
policies (sudo check)
* all uses of fetchACLTokenEntryAndEntity when the request uses a
token with inline policies with duplicate attrs. Almost all reqs
are subject to this
* when tokens are created with inline policies (unclear exactly how that
can happen)
* add changelog and deprecation notice
* add missing copywrite notice
* fix copy-paste mistake
good thing it was covered by unit tests
* Fix manual parsing of telemetry field in SharedConfig
This commit in the hcl library was not in the
v1.0.1-vault-5 version we're using but is
included in v1.0.1-vault-7:
e80118accb
This thing of reusing when parsing means that
our approach of manually re-parsing fields
on top of fields that have already been parsed
by the hcl annotation causes strings (maybe
more?) to concatenate.
Fix that by removing annotation. There's
actually more occurrences of this thing of
automatically parsing something that is also
manually parsing. In some places we could
just remove the boilerplate manual parsing, in
others we better remove the auto parsing, but
I don't wanna pull at that thread right now. I
just checked that all places at least fully
overwrite the automatically parsed field
instead of reusing it as the target of the
decode call. The only exception is the AOP
field on ent but that doesn't have maps or
slices, so I think it's fine.
An alternative approach would be to ensure
that the auto-parsed value is discarded,
like the current parseCache function does
note how it's template not templates
* Fix linter complaints
* Update command/base_predict.go
Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com>
* address review
* remove copywrite headers
* re-add copywrite headers
* make fmt
* Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx
Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
* Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx
Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
* Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx
Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
* undo changes to deprecation.mdx
* remove deprecation doc
* fix conflict with changes from main
---------
Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com>
Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
This is a follow-up to our initial work[0] to address RFC-5952 §4 conformance for IPv6 addresses in Vault. The initial pass focused on the vault server configuration and start-up routines. This follow-up focuses on Agent and Proxy, with a few minor improvements for server.
The approach generally mirrors the server implementation but also adds support for normalization with CLI configuration overrides.
One aspect we do not normalize currently is Agent/Proxy client creation to the Vault server with credentials taken from environment variables, as it would require larger changes to the `api` module. In practice this ought to be fine for the majority of cases.
[0]: https://github.com/hashicorp/vault/pull/29228
* add gosimport to make fmt and run it
* move installation to tools.sh
* correct weird spacing issue
* Update Makefile
Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com>
* fix a weird issue
---------
Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com>
* allow users to specify files for child process stdout/stderr
* added changelog
* check if exec config is nil
* fix test
* first attempt at a test
* revise test
* passing test
* added failing test
* Apply suggestions from code review
Co-authored-by: Anton Averchenkov <84287187+averche@users.noreply.github.com>
* code review suggestions
* always close log files
* refactor to use real files
* hopefully fixed tests
* add back bool gates so we don't close global stdout/stderr
* compare to os.Stdout/os.Stderr
* remove unused
---------
Co-authored-by: Anton Averchenkov <84287187+averche@users.noreply.github.com>
* Adding explicit MPL license for sub-package.
This directory and its subdirectories (packages) contain files licensed with the MPLv2 `LICENSE` file in this directory and are intentionally licensed separately from the BSL `LICENSE` file at the root of this repository.
* Adding explicit MPL license for sub-package.
This directory and its subdirectories (packages) contain files licensed with the MPLv2 `LICENSE` file in this directory and are intentionally licensed separately from the BSL `LICENSE` file at the root of this repository.
* Updating the license from MPL to Business Source License.
Going forward, this project will be licensed under the Business Source License v1.1. Please see our blog post for more details at https://hashi.co/bsl-blog, FAQ at www.hashicorp.com/licensing-faq, and details of the license at www.hashicorp.com/bsl.
* add missing license headers
* Update copyright file headers to BUS-1.1
* Fix test that expected exact offset on hcl file
---------
Co-authored-by: hashicorp-copywrite[bot] <110428419+hashicorp-copywrite[bot]@users.noreply.github.com>
Co-authored-by: Sarah Thompson <sthompson@hashicorp.com>
Co-authored-by: Brian Kassouf <bkassouf@hashicorp.com>
* added exec and env_template config/parsing
* add tests
* we can reuse ctconfig here
* do not create a non-nil map
* check defaults
* Apply suggestions from code review
Co-authored-by: Anton Averchenkov <84287187+averche@users.noreply.github.com>
* Apply suggestions from code review
Co-authored-by: Anton Averchenkov <84287187+averche@users.noreply.github.com>
* convert to list
Signed-off-by: Daniel Huckins <dhuckins@users.noreply.github.com>
* convert to list
Signed-off-by: Daniel Huckins <dhuckins@users.noreply.github.com>
* sig test
Signed-off-by: Daniel Huckins <dhuckins@users.noreply.github.com>
* add failing example
Signed-off-by: Daniel Huckins <dhuckins@users.noreply.github.com>
* add test for invalid signal
Signed-off-by: Daniel Huckins <dhuckins@users.noreply.github.com>
* Update command/agent/config/config.go
* use latest consul-template
* fix build
* fix test
* fix test fixtures
* make fmt
* test docs
* rename file
* env var -> environment variable
* default to SIGTERM
* empty line
* explicit naming
Signed-off-by: Daniel Huckins <dhuckins@users.noreply.github.com>
* clean typo
Signed-off-by: Daniel Huckins <dhuckins@users.noreply.github.com>
* replace $ HOME with /home/username in examples
* remove empty line
---------
Signed-off-by: Daniel Huckins <dhuckins@users.noreply.github.com>
Co-authored-by: Anton Averchenkov <84287187+averche@users.noreply.github.com>
Co-authored-by: Anton Averchenkov <anton.averchenkov@hashicorp.com>
* VAULT-11510 Vault Agent can start listeners without caching
* VAULT-11510 fix order of imports
* VAULT-11510 changelog
* VAULT-11510 typo and better switch
* VAULT-11510 update name
* VAULT-11510 New api_proxy stanza to configure API proxy
* VAULT-11510 First pass at API Proxy docs
* VAULT-11510 nav data
* VAULT-11510 typo
* VAULT-11510 docs update
* Work to unify log-file for agent/server and add rotation
* Updates to rotation code, tried to centralise the log config setup
* logging + tests
* Move LogFile to ShareConfig in test
* Docs
* Started work on adding log-file support to Agent
* Allow log file to be picked up and appended
* Use NewLogFile everywhere
* Tried to pull out the config aggregation from Agent.Run
Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com>
Uses a bufconn listener between consul-template and vault-agent when
caching is enabled and either templates or a listener is defined. This
means no listeners need to be defined in vault-agent for just
templating. Always routes consul-template through the vault-agent
cache (instead of only when persistent cache is enabled).
Uses a local transportDialer interface in config.Cache{}.
Co-authored-by: Tom Proctor <tomhjp@users.noreply.github.com>
Co-authored-by: Ben Ash <32777270+benashz@users.noreply.github.com>
* Customizing HTTP headers in the config file
* Add changelog, fix bad imports
* fixing some bugs
* fixing interaction of custom headers and /ui
* Defining a member in core to set custom response headers
* missing additional file
* Some refactoring
* Adding automated tests for the feature
* Changing some error messages based on some recommendations
* Incorporating custom response headers struct into the request context
* removing some unused references
* fixing a test
* changing some error messages, removing a default header value from /ui
* fixing a test
* wrapping ResponseWriter to set the custom headers
* adding a new test
* some cleanup
* removing some extra lines
* Addressing comments
* fixing some agent tests
* skipping custom headers from agent listener config,
removing two of the default headers as they cause issues with Vault in UI mode
Adding X-Content-Type-Options to the ui default headers
Let Content-Type be set as before
* Removing default custom headers, and renaming some function varibles
* some refacotring
* Refactoring and addressing comments
* removing a function and fixing comments
* agent: restart template runner on retry for unlimited retries
* template: log error message early
* template: delegate retries back to template if param is set to true
* agent: add and use the new template config stanza
* agent: fix panic, fix existing tests
* changelog: add changelog entry
* agent: add tests for exit_on_retry_failure
* agent: properly check on agent exit cases, add separate tests for missing key vs missing secrets
* agent: add note on difference between missing key vs missing secret
* docs: add docs for template_config
* Update website/content/docs/agent/template-config.mdx
Co-authored-by: Jason O'Donnell <2160810+jasonodonnell@users.noreply.github.com>
* Update website/content/docs/agent/template-config.mdx
Co-authored-by: Jason O'Donnell <2160810+jasonodonnell@users.noreply.github.com>
* Update website/content/docs/agent/template-config.mdx
Co-authored-by: Tom Proctor <tomhjp@users.noreply.github.com>
* Update website/content/docs/agent/template-config.mdx
Co-authored-by: Tom Proctor <tomhjp@users.noreply.github.com>
* Update website/content/docs/agent/template-config.mdx
Co-authored-by: Tom Proctor <tomhjp@users.noreply.github.com>
* docs: fix exit_on_retry_failure, fix Functionality section
* docs: update interaction title
* template: add internal note on behavior for persist case
* docs: update agent, template, and template-config docs
* docs: update agent docs on retry stanza
* Apply suggestions from code review
Co-authored-by: Jim Kalafut <jkalafut@hashicorp.com>
Co-authored-by: Theron Voran <tvoran@users.noreply.github.com>
* Update changelog/11775.txt
Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
* agent/test: rename expectExit to expectExitFromError
* agent/test: add check on early exits on the happy path
* Update website/content/docs/agent/template-config.mdx
Co-authored-by: Tom Proctor <tomhjp@users.noreply.github.com>
Co-authored-by: Jason O'Donnell <2160810+jasonodonnell@users.noreply.github.com>
Co-authored-by: Tom Proctor <tomhjp@users.noreply.github.com>
Co-authored-by: Jim Kalafut <jkalafut@hashicorp.com>
Co-authored-by: Theron Voran <tvoran@users.noreply.github.com>
Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
* Actually call config.Validate in diagnose
* Wire configuration checks into diagnose and fix resulting bugs.
* go mod vendor
* Merge to vendorless version
* Remove sentinel section to allow diagnose_ok to pass
* Fix unit tests
* Expose unknown fields and duplicate sections as diagnose warnings
* section counts not needed, already handled
* Address PR feedback
* Prune more of the new fields before tests call deep.Equals
* Update go.mod
Remove template_retry config section. Add new vault.retry section which only has num_retries field; if num_retries is 0 or absent, default it to 12 for backwards compat with pre-1.7 template retrying. Setting num_retries=-1 disables retries.
Configured retries are used for both templating and api proxy, though if template requests go through proxy (currently requires persistence enabled) we'll only configure retries for the latter to avoid duplicate retrying. Though there is some duplicate retrying already because whenever the template server does a retry when not going through the proxy, the Vault client it uses allows for 2 behind-the-scenes retries for some 400/500 http error codes.
Adds the option of a write-through cache, backed by boltdb
Co-authored-by: Theron Voran <tvoran@users.noreply.github.com>
Co-authored-by: Jason O'Donnell <2160810+jasonodonnell@users.noreply.github.com>
Co-authored-by: Calvin Leung Huang <cleung2010@gmail.com>
* feat(agent): add retry configuration for vault agent
* feat(agent): add test fixtures for retry
* fix(retry): move retry stanza to top level as template_retry
* fix(retry): add retry config to ServerConfig struct
* fix(retry): point config parser to parse template_retry instead of retry
* remove netlify config (#10711)
* Fix build (#10749)
* Move the declaration to a OSS build tag file to not have it collide w… (#10750)
* Move the declaration to a OSS build tag file to not have it collide with ent declarations
* Add comment
* Remove comment to trigger ci
* Unconditionally use the root namespace when calling sys/seal-status. (#10742)
* feat(agent): add retry configuration for vault agent
* feat(agent): add test fixtures for retry
* fix(retry): move retry stanza to top level as template_retry
* fix(retry): add retry config to ServerConfig struct
* fix(retry): point config parser to parse template_retry instead of retry
Co-authored-by: Hridoy Roy <roy@hashicorp.com>
Co-authored-by: Jeff Escalante <jescalan@users.noreply.github.com>
Co-authored-by: Vishal Nayak <vishalnayak@users.noreply.github.com>
Co-authored-by: Mark Gritter <mgritter@hashicorp.com>
* TOB-018 remediation
* Make key derivation an optional config flag, off by default, for backwards compatibility
* Fix unit tests
* Address some feedback
* Set config on unit test
* Fix another test failure
* One more conf fail
* Switch one of the test cases to not use a derive dkey
* wip
* comments
For situations where you want the Vault agent to handle one or more templates but do not require the acquired credentials elsewhere.
Modify the logic in SyncServer so that if there are no sinks, ignore any new credentials. Since SyncServer is responsible for shutting down the agent, make sure it still properly shuts down in this new situation.
Solves #7988
* rename UseAutoAuthForce to ForceAutoAuth, because I think it reads better
* Document 'ForceAuthAuthToken' option for Agent Cache
* Update website/pages/docs/agent/caching/index.mdx
Co-Authored-By: Jim Kalafut <jkalafut@hashicorp.com>
* Add additional tests around use_auto_auth=force and add documentation
* remove note, it's no longer correct
Co-authored-by: Jim Kalafut <jim@kalafut.net>
* Added flag to disable X-Vault-Token header proxy if client passes the token
* Reveresed the flag value to better match the name intent
* Introduced UseAutoAuthTokenRaw for Cache to support triplicate value of true/false/force
Co-authored-by: Clint <catsby@users.noreply.github.com>
* Vault Agent Template: parse templates (#7540)
* add template config parsing, but it's wrong b/c it's not using mapstructure
* parsing consul templates in agent config
* add additional test to configuration parsing, to cover basics
* another test fixture, rework simple test into table
* refactor into table test
* rename test
* remove flattenKeys and add other test fixture
* Update command/agent/config/config.go
Co-Authored-By: Jim Kalafut <jkalafut@hashicorp.com>
* return the decode error instead of swallowing it
* Update command/agent/config/config_test.go
Co-Authored-By: Jim Kalafut <jkalafut@hashicorp.com>
* go mod tidy
* change error checking style
* Add agent template doc
* TemplateServer: render secrets with Consul Template (#7621)
* add template config parsing, but it's wrong b/c it's not using mapstructure
* parsing consul templates in agent config
* add additional test to configuration parsing, to cover basics
* another test fixture, rework simple test into table
* refactor into table test
* rename test
* remove flattenKeys and add other test fixture
* add template package
* WIP: add runner
* fix panic, actually copy templates, etc
* rework how the config.Vault is created and enable reading from the environment
* this was supposed to be a part of the prior commit
* move/add methods to testhelpers for converting some values to pointers
* use new methods in testhelpers
* add an unblock channel to block agent until a template has been rendered
* add note
* unblock if there are no templates
* cleanups
* go mod tidy
* remove dead code
* simple test to starT
* add simple, empty templates test
* Update package doc, error logs, and add missing close() on channel
* update code comment to be clear what I'm referring to
* have template.NewServer return a (<- chan) type, even though it's a normal chan, as a better practice to enforce reading only
* Update command/agent.go
Co-Authored-By: Jim Kalafut <jkalafut@hashicorp.com>
* update with test
* Add README and doc.go to the command/agent directory (#7503)
* Add README and doc.go to the command/agent directory
* Add link to website
* address feedback for agent.go
* updated with feedback from Calvin
* Rework template.Server to export the unblock channel, and remove it from the NewServer function
* apply feedback from Nick
* fix/restructure rendering test
* Add pointerutil package for converting types to their pointers
* Remove pointer helper methods; use sdk/helper/pointerutil instead
* update newRunnerConfig to use pointerutil and empty strings
* only wait for unblock if template server is initialized
* drain the token channel in this test
* conditionally send on channel
* implement SSRF protection header
* add test for SSRF protection header
* cleanup
* refactor
* implement SSRF header on a per-listener basis
* cleanup
* cleanup
* creat unit test for agent SSRF
* improve unit test for agent SSRF
* add VaultRequest SSRF header to CLI
* fix unit test
* cleanup
* improve test suite
* simplify check for Vault-Request header
* add constant for Vault-Request header
* improve test suite
* change 'config' to 'agentConfig'
* Revert "change 'config' to 'agentConfig'"
This reverts commit 14ee72d21fff8027966ee3c89dd3ac41d849206f.
* do not remove header from request
* change header name to X-Vault-Request
* simplify http.Handler logic
* cleanup
* simplify http.Handler logic
* use stdlib errors package
