Bruno Oliveira de Souza 0b9157156f
VAULT-32657 deprecate duplicate attributes in HCL configs and policies (#30386)
* upgrade hcl dependency on api pkg

This upgrades the hcl dependency for the API pkg,
and adapts its usage so users of our API pkg are
not affected. There's no good way of communicating
a warning via a library call so we don't.

The tokenHelper which is used by all Vault CLI
commands in order to create the Vault client, as
well as directly used by the login and server
commands, is implemented on the api pkg, so this
upgrade also affects all of those commands. Seems
like this was only moved to the api pkg because
the Terraform provider uses it, and I thought
creating a full copy of all those files back under
command would be too much spaghetti.

Also leaving some TODOs to make next deprecation
steps easier.

* upgrade hcl dependency in vault and sdk pkgs

* upgrade hcl dependency in vault and sdk pkgs

* add CLI warnings to commands that take a config

- vault agent (unit test on CMD warning)
- vault proxy (unit test on CMD warning)
- vault server (no test for the warning)
- vault operator diagnose (no tests at all, uses the
same function as vault server

* ignore duplicates on ParseKMSes function

* Extend policy parsing functions and warn on policy store

* Add warning on policy fmt with duplicate attributes

* Add warnings when creating/updating policy with duplicate HCL attrs

* Add log warning when switchedGetPolicy finds duplicate attrs

Following operations can trigger this warning when they run into a policy
with duplicate attributes:
* replication filtered path namespaces invalidation
* policy read API
* building an ACL (for many different purposes like most authZ operations)
* looking up DR token policies
* creating a token with named policies
* when caching the policies for all namespaces during unseal

* Print log warnings when token inline policy has duplicate attrs

No unit tests on these as new test infra would have to be built on all.
Operations affected, which will now print a log warning when the retrieved
token has an inline policy with duplicate attributes:
* capabilities endpoints in sys mount
* handing events under a subscription with a token with duplicate
attrs in inline policies
* token used to create another token has duplicate attrs in inline
policies (sudo check)
* all uses of fetchACLTokenEntryAndEntity when the request uses a
token with inline policies with duplicate attrs. Almost all reqs
are subject to this
* when tokens are created with inline policies (unclear exactly how that
can happen)

* add changelog and deprecation notice

* add missing copywrite notice

* fix copy-paste mistake

good thing it was covered by unit tests

* Fix manual parsing of telemetry field in SharedConfig

This commit in the hcl library was not in the
v1.0.1-vault-5 version we're using but is
included in v1.0.1-vault-7:
e80118accb

This thing of reusing when parsing means that
our approach of manually re-parsing fields
on top of fields that have already been parsed
by the hcl annotation causes strings (maybe
more?) to concatenate.

Fix that by removing annotation. There's
actually more occurrences of this thing of
automatically parsing something that is also
manually parsing. In some places we could
just remove the boilerplate manual parsing, in
others we better remove the auto parsing, but
I don't wanna pull at that thread right now. I
just checked that all places at least fully
overwrite the automatically parsed field
instead of reusing it as the target of the
decode call. The only exception is the AOP
field on ent but that doesn't have maps or
slices, so I think it's fine.

An alternative approach would be to ensure
that the auto-parsed value is discarded,
like the current parseCache function does

note how it's template not templates

* Fix linter complaints

* Update command/base_predict.go

Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com>

* address review

* remove copywrite headers

* re-add copywrite headers

* make fmt

* Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx

Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>

* Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx

Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>

* Update website/content/partials/deprecation/duplicate-hcl-attributes.mdx

Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>

* undo changes to deprecation.mdx

* remove deprecation doc

* fix conflict with changes from main

---------

Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com>
Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
2025-05-23 16:02:07 -03:00

558 lines
17 KiB
Go

// Copyright (c) HashiCorp, Inc.
// SPDX-License-Identifier: BUSL-1.1
package configutil
import (
"context"
"crypto/rand"
"errors"
"fmt"
"io"
"os"
"regexp"
"slices"
"strings"
"github.com/hashicorp/errwrap"
"github.com/hashicorp/go-hclog"
"github.com/hashicorp/go-kms-wrapping/entropy/v2"
wrapping "github.com/hashicorp/go-kms-wrapping/v2"
aeadwrapper "github.com/hashicorp/go-kms-wrapping/wrappers/aead/v2"
"github.com/hashicorp/go-kms-wrapping/wrappers/alicloudkms/v2"
"github.com/hashicorp/go-kms-wrapping/wrappers/awskms/v2"
"github.com/hashicorp/go-kms-wrapping/wrappers/azurekeyvault/v2"
"github.com/hashicorp/go-kms-wrapping/wrappers/gcpckms/v2"
"github.com/hashicorp/go-kms-wrapping/wrappers/ocikms/v2"
"github.com/hashicorp/go-kms-wrapping/wrappers/transit/v2"
"github.com/hashicorp/go-multierror"
"github.com/hashicorp/go-secure-stdlib/parseutil"
"github.com/hashicorp/hcl"
"github.com/hashicorp/hcl/hcl/ast"
"github.com/hashicorp/vault/helper/random"
"github.com/hashicorp/vault/sdk/helper/strutil"
"github.com/hashicorp/vault/sdk/logical"
)
var (
ConfigureWrapper = configureWrapper
CreateSecureRandomReaderFunc = createSecureRandomReader
GetEnvConfigFunc = getEnvConfig
)
//go:generate enumer -type=EntropyMode -trimprefix=Entropy
// EntropyMode contains Entropy configuration for the server
type EntropyMode int
const (
EntropyUnknown EntropyMode = iota
EntropyAugmentation
KmsRenameDisabledSuffix = "-disabled"
)
type Entropy struct {
Mode EntropyMode
SealName string
}
type EntropySourcerInfo struct {
Sourcer entropy.Sourcer
Name string
}
// KMS contains KMS configuration for the server
type KMS struct {
UnusedKeys []string `hcl:",unusedKeys"`
Type string
// Purpose can be used to allow a string-based specification of what this
// KMS is designated for, in situations where we want to allow more than
// one KMS to be specified
Purpose []string `hcl:"-"`
Disabled bool
Config map[string]string
Priority int `hcl:"priority"`
Name string `hcl:"name"`
}
func (k *KMS) GoString() string {
return fmt.Sprintf("*%#v", *k)
}
func parseKMS(result *[]*KMS, list *ast.ObjectList, blockName string, maxKMS int) error {
if len(list.Items) > maxKMS {
return fmt.Errorf("only %d or less %q blocks are permitted", maxKMS, blockName)
}
seals := make([]*KMS, 0, len(list.Items))
for _, item := range list.Items {
key := blockName
if len(item.Keys) > 0 {
key = item.Keys[0].Token.Value().(string)
}
// We first decode into a map[string]interface{} because purpose isn't
// necessarily a string. Then we migrate everything else over to
// map[string]string and error if it doesn't work.
var m map[string]interface{}
if err := hcl.DecodeObject(&m, item.Val); err != nil {
return multierror.Prefix(err, fmt.Sprintf("%s.%s:", blockName, key))
}
var purpose []string
var err error
if v, ok := m["purpose"]; ok {
if purpose, err = parseutil.ParseCommaStringSlice(v); err != nil {
return multierror.Prefix(fmt.Errorf("unable to parse 'purpose' in kms type %q: %w", key, err), fmt.Sprintf("%s.%s:", blockName, key))
}
for i, p := range purpose {
purpose[i] = strings.ToLower(p)
}
delete(m, "purpose")
}
var disabled bool
if v, ok := m["disabled"]; ok {
disabled, err = parseutil.ParseBool(v)
if err != nil {
return multierror.Prefix(err, fmt.Sprintf("%s.%s:", blockName, key))
}
delete(m, "disabled")
}
var priority int
if v, ok := m["priority"]; ok {
priority, err = parseutil.SafeParseInt(v)
if err != nil {
return multierror.Prefix(fmt.Errorf("unable to parse 'priority' in kms type %q: %w", key, err), fmt.Sprintf("%s.%s", blockName, key))
}
delete(m, "priority")
if priority < 1 {
return multierror.Prefix(fmt.Errorf("invalid priority in kms type %q: %d", key, priority), fmt.Sprintf("%s.%s", blockName, key))
}
}
name := strings.ToLower(key)
// ensure that seals of the same type will have unique names for seal migration
if disabled {
name += KmsRenameDisabledSuffix
}
if v, ok := m["name"]; ok {
name, ok = v.(string)
if !ok {
return multierror.Prefix(fmt.Errorf("unable to parse 'name' in kms type %q: unexpected type %T", key, v), fmt.Sprintf("%s.%s", blockName, key))
}
delete(m, "name")
if !regexp.MustCompile("^[a-zA-Z0-9-_]+$").MatchString(name) {
return multierror.Prefix(errors.New("'name' field can only include alphanumeric characters, hyphens, and underscores"), fmt.Sprintf("%s.%s", blockName, key))
}
}
strMap := make(map[string]string, len(m))
for k, v := range m {
s, err := parseutil.ParseString(v)
if err != nil {
return multierror.Prefix(err, fmt.Sprintf("%s.%s:", blockName, key))
}
strMap[k], err = normalizeKMSSealConfigAddrs(key, k, s)
if err != nil {
return multierror.Prefix(err, fmt.Sprintf("%s.%s:", blockName, key))
}
}
seal := &KMS{
Type: strings.ToLower(key),
Purpose: purpose,
Disabled: disabled,
Priority: priority,
Name: name,
}
if len(strMap) > 0 {
seal.Config = strMap
}
seals = append(seals, seal)
}
*result = append(*result, seals...)
return nil
}
func ParseKMSes(d string) ([]*KMS, error) {
// Parse!
// TODO (HCL_DUP_KEYS_DEPRECATION): return to hcl.Parse once deprecation is done. For now just ignore duplicates on
// this unused function
obj, _, err := random.ParseAndCheckForDuplicateHclAttributes(d)
if err != nil {
return nil, err
}
// Start building the result
var result struct {
Seals []*KMS `hcl:"-"`
}
if err := hcl.DecodeObject(&result, obj); err != nil {
return nil, err
}
list, ok := obj.Node.(*ast.ObjectList)
if !ok {
return nil, fmt.Errorf("error parsing: file doesn't contain a root object")
}
if o := list.Filter("seal"); len(o.Items) > 0 {
if err := parseKMS(&result.Seals, o, "seal", 3); err != nil {
return nil, fmt.Errorf("error parsing 'seal': %w", err)
}
}
if o := list.Filter("kms"); len(o.Items) > 0 {
if err := parseKMS(&result.Seals, o, "kms", 3); err != nil {
return nil, fmt.Errorf("error parsing 'kms': %w", err)
}
}
return result.Seals, nil
}
// kmsSealAddressKeys maps seal key types to corresponding config keys whose
// values might contain URLs, IP addresses, or host:port addresses. All seal
// types must contain an entry here, otherwise our normalization check will fail
// when parsing the seal config. Seal types which do not contain such
// configurations ought to have an empty array as the value in the map.
var kmsSealAddressKeys = map[string][]string{
wrapping.WrapperTypeAliCloudKms.String(): {"domain"},
wrapping.WrapperTypeAwsKms.String(): {"endpoint"},
wrapping.WrapperTypeAzureKeyVault.String(): {"resource"},
wrapping.WrapperTypeGcpCkms.String(): {},
wrapping.WrapperTypeOciKms.String(): {"key_id", "crypto_endpoint", "management_endpoint"},
wrapping.WrapperTypePkcs11.String(): {},
wrapping.WrapperTypeTransit.String(): {"address"},
}
// normalizeKMSSealConfigAddrs takes a kms seal type, a config key, and its
// associated value and will normalize any URLs, IP addresses, or host:port
// addresses contained in the value if the config key is known in the
// kmsSealAddressKeys.
func normalizeKMSSealConfigAddrs(seal string, key string, value string) (string, error) {
keys, ok := kmsSealAddressKeys[seal]
if !ok {
return "", fmt.Errorf("unknown seal type %s", seal)
}
if slices.Contains(keys, key) {
return NormalizeAddr(value), nil
}
return value, nil
}
// mergeKMSEnvConfig takes a KMS and merges any normalized values set via
// environment variables.
func mergeKMSEnvConfig(configKMS *KMS) error {
envConfig := GetEnvConfigFunc(configKMS)
if len(envConfig) > 0 && configKMS.Config == nil {
configKMS.Config = make(map[string]string)
}
// transit is a special case, because some config values take precedence over env vars
if configKMS.Type == wrapping.WrapperTypeTransit.String() {
if err := mergeTransitConfig(configKMS.Config, envConfig); err != nil {
return err
}
} else {
for name, val := range envConfig {
var err error
configKMS.Config[name], err = normalizeKMSSealConfigAddrs(configKMS.Type, name, val)
if err != nil {
return err
}
}
}
return nil
}
func configureWrapper(configKMS *KMS, infoKeys *[]string, info *map[string]string, logger hclog.Logger, opts ...wrapping.Option) (wrapping.Wrapper, error) {
var wrapper wrapping.Wrapper
var kmsInfo map[string]string
var err error
// Get any seal config set as env variables and merge it into the KMS.
if err = mergeKMSEnvConfig(configKMS); err != nil {
return nil, err
}
switch wrapping.WrapperType(configKMS.Type) {
case wrapping.WrapperTypeShamir:
return wrapper, nil
case wrapping.WrapperTypeAead:
wrapper, kmsInfo, err = GetAEADKMSFunc(configKMS, opts...)
case wrapping.WrapperTypeAliCloudKms:
wrapper, kmsInfo, err = GetAliCloudKMSFunc(configKMS, opts...)
case wrapping.WrapperTypeAwsKms:
wrapper, kmsInfo, err = GetAWSKMSFunc(configKMS, opts...)
case wrapping.WrapperTypeAzureKeyVault:
wrapper, kmsInfo, err = GetAzureKeyVaultKMSFunc(configKMS, opts...)
case wrapping.WrapperTypeGcpCkms:
wrapper, kmsInfo, err = GetGCPCKMSKMSFunc(configKMS, opts...)
case wrapping.WrapperTypeOciKms:
if keyId, ok := configKMS.Config["key_id"]; ok {
opts = append(opts, wrapping.WithKeyId(keyId))
}
wrapper, kmsInfo, err = GetOCIKMSKMSFunc(configKMS, opts...)
case wrapping.WrapperTypeTransit:
wrapper, kmsInfo, err = GetTransitKMSFunc(configKMS, opts...)
case wrapping.WrapperTypePkcs11:
return nil, fmt.Errorf("KMS type 'pkcs11' requires the Vault Enterprise HSM binary")
default:
return nil, fmt.Errorf("Unknown KMS type %q", configKMS.Type)
}
if err != nil {
return nil, err
}
if infoKeys != nil && info != nil {
for k, v := range kmsInfo {
*infoKeys = append(*infoKeys, k)
(*info)[k] = v
}
}
return wrapper, nil
}
func GetAEADKMSFunc(kms *KMS, opts ...wrapping.Option) (wrapping.Wrapper, map[string]string, error) {
wrapper := aeadwrapper.NewWrapper()
wrapperInfo, err := wrapper.SetConfig(context.Background(), append(opts, wrapping.WithConfigMap(kms.Config))...)
if err != nil {
return nil, nil, err
}
info := make(map[string]string)
if wrapperInfo != nil {
str := "AEAD Type"
if len(kms.Purpose) > 0 {
str = fmt.Sprintf("%v %s", kms.Purpose, str)
}
info[str] = wrapperInfo.Metadata["aead_type"]
}
return wrapper, info, nil
}
func GetAliCloudKMSFunc(kms *KMS, opts ...wrapping.Option) (wrapping.Wrapper, map[string]string, error) {
wrapper := alicloudkms.NewWrapper()
wrapperInfo, err := wrapper.SetConfig(context.Background(), append(opts, wrapping.WithDisallowEnvVars(true), wrapping.WithConfigMap(kms.Config))...)
if err != nil {
// If the error is any other than logical.KeyNotFoundError, return the error
if !errwrap.ContainsType(err, new(logical.KeyNotFoundError)) {
return nil, nil, err
}
}
info := make(map[string]string)
if wrapperInfo != nil {
info["AliCloud KMS Region"] = wrapperInfo.Metadata["region"]
info["AliCloud KMS KeyID"] = wrapperInfo.Metadata["kms_key_id"]
if domain, ok := wrapperInfo.Metadata["domain"]; ok {
info["AliCloud KMS Domain"] = domain
}
}
return wrapper, info, nil
}
var GetAWSKMSFunc = func(kms *KMS, opts ...wrapping.Option) (wrapping.Wrapper, map[string]string, error) {
wrapper := awskms.NewWrapper()
wrapperInfo, err := wrapper.SetConfig(context.Background(), append(opts, awskms.WithDisallowEnvVars(true), wrapping.WithConfigMap(kms.Config))...)
if err != nil {
// If the error is any other than logical.KeyNotFoundError, return the error
if !errwrap.ContainsType(err, new(logical.KeyNotFoundError)) {
return nil, nil, err
}
}
info := make(map[string]string)
if wrapperInfo != nil {
info["AWS KMS Region"] = wrapperInfo.Metadata["region"]
info["AWS KMS KeyID"] = wrapperInfo.Metadata["kms_key_id"]
if endpoint, ok := wrapperInfo.Metadata["endpoint"]; ok {
info["AWS KMS Endpoint"] = endpoint
}
}
return wrapper, info, nil
}
func GetAzureKeyVaultKMSFunc(kms *KMS, opts ...wrapping.Option) (wrapping.Wrapper, map[string]string, error) {
wrapper := azurekeyvault.NewWrapper()
wrapperInfo, err := wrapper.SetConfig(context.Background(), append(opts, azurekeyvault.WithDisallowEnvVars(true), wrapping.WithConfigMap(kms.Config))...)
if err != nil {
// If the error is any other than logical.KeyNotFoundError, return the error
if !errwrap.ContainsType(err, new(logical.KeyNotFoundError)) {
return nil, nil, err
}
}
info := make(map[string]string)
if wrapperInfo != nil {
info["Azure Environment"] = wrapperInfo.Metadata["environment"]
info["Azure Vault Name"] = wrapperInfo.Metadata["vault_name"]
info["Azure Key Name"] = wrapperInfo.Metadata["key_name"]
}
return wrapper, info, nil
}
func GetGCPCKMSKMSFunc(kms *KMS, opts ...wrapping.Option) (wrapping.Wrapper, map[string]string, error) {
wrapper := gcpckms.NewWrapper()
wrapperInfo, err := wrapper.SetConfig(context.Background(), append(opts, wrapping.WithDisallowEnvVars(true), wrapping.WithConfigMap(kms.Config))...)
if err != nil {
// If the error is any other than logical.KeyNotFoundError, return the error
if !errwrap.ContainsType(err, new(logical.KeyNotFoundError)) {
return nil, nil, err
}
}
info := make(map[string]string)
if wrapperInfo != nil {
info["GCP KMS Project"] = wrapperInfo.Metadata["project"]
info["GCP KMS Region"] = wrapperInfo.Metadata["region"]
info["GCP KMS Key Ring"] = wrapperInfo.Metadata["key_ring"]
info["GCP KMS Crypto Key"] = wrapperInfo.Metadata["crypto_key"]
}
return wrapper, info, nil
}
func GetOCIKMSKMSFunc(kms *KMS, opts ...wrapping.Option) (wrapping.Wrapper, map[string]string, error) {
wrapper := ocikms.NewWrapper()
wrapperInfo, err := wrapper.SetConfig(context.Background(), append(opts, wrapping.WithDisallowEnvVars(true), wrapping.WithConfigMap(kms.Config))...)
if err != nil {
return nil, nil, err
}
info := make(map[string]string)
if wrapperInfo != nil {
info["OCI KMS KeyID"] = wrapperInfo.Metadata[ocikms.KmsConfigKeyId]
info["OCI KMS Crypto Endpoint"] = wrapperInfo.Metadata[ocikms.KmsConfigCryptoEndpoint]
info["OCI KMS Management Endpoint"] = wrapperInfo.Metadata[ocikms.KmsConfigManagementEndpoint]
info["OCI KMS Principal Type"] = wrapperInfo.Metadata["principal_type"]
}
return wrapper, info, nil
}
var GetTransitKMSFunc = func(kms *KMS, opts ...wrapping.Option) (wrapping.Wrapper, map[string]string, error) {
wrapper := transit.NewWrapper()
var prefix string
if p, ok := kms.Config["key_id_prefix"]; ok {
prefix = p
} else {
prefix = kms.Name
}
if !strings.HasSuffix(prefix, "/") {
prefix = prefix + "/"
}
wrapperInfo, err := wrapper.SetConfig(context.Background(), append(opts, wrapping.WithDisallowEnvVars(true), wrapping.WithConfigMap(kms.Config),
transit.WithKeyIdPrefix(prefix))...)
if err != nil {
// If the error is any other than logical.KeyNotFoundError, return the error
if !errwrap.ContainsType(err, new(logical.KeyNotFoundError)) {
return nil, nil, err
}
}
info := make(map[string]string)
if wrapperInfo != nil {
info["Transit Address"] = wrapperInfo.Metadata["address"]
info["Transit Mount Path"] = wrapperInfo.Metadata["mount_path"]
info["Transit Key Name"] = wrapperInfo.Metadata["key_name"]
if namespace, ok := wrapperInfo.Metadata["namespace"]; ok {
info["Transit Namespace"] = namespace
}
}
return wrapper, info, nil
}
func createSecureRandomReader(_ *SharedConfig, _ []*EntropySourcerInfo, _ hclog.Logger) (io.Reader, error) {
return rand.Reader, nil
}
func getEnvConfig(kms *KMS) map[string]string {
envValues := make(map[string]string)
var wrapperEnvVars map[string]string
switch wrapping.WrapperType(kms.Type) {
case wrapping.WrapperTypeAliCloudKms:
wrapperEnvVars = AliCloudKMSEnvVars
case wrapping.WrapperTypeAwsKms:
wrapperEnvVars = AWSKMSEnvVars
case wrapping.WrapperTypeAzureKeyVault:
wrapperEnvVars = AzureEnvVars
case wrapping.WrapperTypeGcpCkms:
wrapperEnvVars = GCPCKMSEnvVars
case wrapping.WrapperTypeOciKms:
wrapperEnvVars = OCIKMSEnvVars
case wrapping.WrapperTypeTransit:
wrapperEnvVars = TransitEnvVars
default:
return nil
}
for envVar, configName := range wrapperEnvVars {
val := os.Getenv(envVar)
if val != "" {
envValues[configName] = val
}
}
return envValues
}
func mergeTransitConfig(config map[string]string, envConfig map[string]string) error {
useFileTlsConfig := false
for _, varName := range TransitTLSConfigVars {
if _, ok := config[varName]; ok {
useFileTlsConfig = true
break
}
}
if useFileTlsConfig {
for _, varName := range TransitTLSConfigVars {
delete(envConfig, varName)
}
}
var err error
for varName, val := range envConfig {
// for some values, file config takes precedence
if strutil.StrListContains(TransitPrioritizeConfigValues, varName) && config[varName] != "" {
continue
}
config[varName], err = normalizeKMSSealConfigAddrs(wrapping.WrapperTypeTransit.String(), varName, val)
if err != nil {
return err
}
}
return nil
}
func (k *KMS) Clone() *KMS {
ret := &KMS{
UnusedKeys: k.UnusedKeys,
Type: k.Type,
Purpose: k.Purpose,
Config: k.Config,
Name: k.Name,
Disabled: k.Disabled,
Priority: k.Priority,
}
return ret
}