This fixes the enterprise failure of the test
```
=== FAIL: builtin/logical/pki TestCRLIssuerRemoval (0.00s)
crl_test.go:1456:
Error Trace: /home/runner/actions-runner/_work/vault-enterprise/vault-enterprise/builtin/logical/pki/crl_test.go:1456
Error: Received unexpected error:
Global, cross-cluster revocation queue cannot be enabled when auto rebuilding is disabled as the local cluster may not have the certificate entry!
Test: TestCRLIssuerRemoval
Messages: failed enabling unified CRLs on enterprise
```
* Clean up unused CRL entries when issuer is removed
When a issuer is removed, the space utilized by its CRL was not freed,
both from the CRL config mapping issuer IDs to CRL IDs and from the
CRL storage entry. We thus implement a two step cleanup, wherein
orphaned CRL IDs are removed from the config and any remaining full
CRL entries are removed from disk.
This relates to a Consul<->Vault interop issue (#22980), wherein Consul
creates a new issuer on every leadership election, causing this config
to grow. Deleting issuers manually does not entirely solve this problem
as the config does not fully reclaim space used in this entry.
Notably, an observation that when deleting issuers, the CRL was rebuilt
on secondary clusters (due to the invalidation not caring about type of
the operation); for consistency and to clean up the unified CRLs, we
also need to run the rebuild on the active primary cluster that deleted
the issuer as well.
This approach does allow cleanup on existing impacted clusters by simply
rebuilding the CRL.
Co-authored-by: Steven Clark <steven.clark@hashicorp.com>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add test case on CRL removal
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add changelog entry
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
---------
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Co-authored-by: Steven Clark <steven.clark@hashicorp.com>
* Adding explicit MPL license for sub-package.
This directory and its subdirectories (packages) contain files licensed with the MPLv2 `LICENSE` file in this directory and are intentionally licensed separately from the BSL `LICENSE` file at the root of this repository.
* Adding explicit MPL license for sub-package.
This directory and its subdirectories (packages) contain files licensed with the MPLv2 `LICENSE` file in this directory and are intentionally licensed separately from the BSL `LICENSE` file at the root of this repository.
* Updating the license from MPL to Business Source License.
Going forward, this project will be licensed under the Business Source License v1.1. Please see our blog post for more details at https://hashi.co/bsl-blog, FAQ at www.hashicorp.com/licensing-faq, and details of the license at www.hashicorp.com/bsl.
* add missing license headers
* Update copyright file headers to BUS-1.1
* Fix test that expected exact offset on hcl file
---------
Co-authored-by: hashicorp-copywrite[bot] <110428419+hashicorp-copywrite[bot]@users.noreply.github.com>
Co-authored-by: Sarah Thompson <sthompson@hashicorp.com>
Co-authored-by: Brian Kassouf <bkassouf@hashicorp.com>
* Refactor sign-intermediate API response
- Allow the sign-intermediate response handling code to be shared
across different API calls.
* Add missing cieps.go
- The Golang ACME library will retry on failures until the passed in
context times out. So if a test is broken it will never end until
the context expires.
- The timeouts are rather large to avoid some issues with slow CI
runners, that have already triggered on a nightly 32 bit run
* OpenAPI: Define default response structure for ListOperations
Almost all Vault ListOperation responses have an identical response
schema. Update the OpenAPI generator to know this, and remove a few
instances where that standard response schema had been manually
copy/pasted into place in individual endpoints.
* changelog
* Only render StandardListResponse schema, if an operation uses it
* Teach the response schema validation test helper about the default list schema too
---------
Co-authored-by: Anton Averchenkov <84287187+averche@users.noreply.github.com>
* Fix ACME tidy to not reference acmeCtx
acmeContext is useful for when we need to reference things with a ACME
base URL, but everything used in tidy doesn't need this URL as it is not
coming from an ACME request.
Refactor tidy to remove references to acmeContext, including dependent
functions in acme_state.go.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Remove spurious log message
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Draft Tidy Acme Test with Backdate Storage + Backdate Sysxsx
* Fixes to ACME tidy testing
Co-authored-by: kitography <khaines@mit.edu>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Correctly set account kid to update account status
Co-authored-by: kitography <khaines@mit.edu>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add TestTidyAcmeWithSafetyBuffer
Co-authored-by: kitography <khaines@mit.edu>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add test for disabling tidy operation
Co-authored-by: kitography <khaines@mit.edu>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add acme_account_safety_buffer to auto-tidy config
Resolve: #21872
Co-authored-by: kitography <khaines@mit.edu>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add tests verifying tidy safety buffers
Co-authored-by: kitography <khaines@mit.edu>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add changelog entry
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add account status validations and order cleanup tests
---------
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Co-authored-by: kitography <khaines@mit.edu>
Co-authored-by: Steve Clark <steven.clark@hashicorp.com>
* imprv: Add a parameter to allow ExtKeyUsage field usage from a role
* chore: Add the changelog entry
* imprv: Reword UI and changelog
* doc: Add allow_role_extkeyusage in parameter list
* imprv: Align variable names with existing fields/codebase
* Add unit test and tweak some labels
---------
Co-authored-by: Steve Clark <steven.clark@hashicorp.com>
* Refactor CRL writing config to passthrough cache
When reading the CRL config via API endpoint, always read through to the
disk, updating the cache in the process. Similarly, when writing to the
CRL config, read first from disk (updating the cache), and on write,
also write back through the cache, providing consistency without the
need to invalidate through markConfigDirty(...).
This will form the basis of the new pattern for config caching.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Refactor ACME writing config to passthrough cache
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
---------
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
- Updating existing entries in OSS so that this can be backported.
- The proper fix on main going forward it to remove these ENT only
entries and move them into the enterprise only file
* OpenAPI: Separate ListOperation from ReadOperation
Historically, since Vault's ReadOperation and ListOperation both map to
the HTTP GET method, their representation in the generated OpenAPI has
been a bit confusing.
This was partially mitigated some time ago, by making the `list` query
parameter express whether it was required or optional - but only in
a way useful to human readers - the human had to know, for example, that
the schema of the response body would change depending on whether `list`
was selected.
Now that there is an effort underway to automatically generate API
clients from the OpenAPI spec, we have a need to fix this more
comprehensively. Fortunately, we do have a means to do so - since Vault
has opinionated treatment of trailing slashes, linked to operations
being list or not, we can use an added trailing slash on the URL path to
separate list operations in the OpenAPI spec.
This PR implements that, and then fixes an operation ID which becomes
duplicated, with this change applied.
See also hashicorp/vault-client-go#174, a bug which will be fixed by
this work.
* Set further DisplayAttrs in auth/github plugin
To mask out more duplicate read/list functionality, now being separately
generated to OpenAPI client libraries as a result of this change.
* Apply requested changes to operation IDs
I'm not totally convinced its worth the extra lines of code, but
equally, I don't have strong feelings about it, so I'll just make the
change.
* Adjust logic to prevent any possibility of generating OpenAPI paths with doubled final slashes
Even in the edge case of improper use of regex patterns and operations.
* changelog
* Fix TestSudoPaths to pass again... which snowballed a bit...
Once I looked hard at it, I found it was missing several sudo paths,
which led to additional bug fixing elsewhere.
I might need to pull some parts of this change out into a separate PR
for ease of review...
* Fix other tests
* More test fixing
* Undo scope creep - back away from fixing sudo paths not shown as such in OpenAPI, at least within this PR
Just add TODO comments for now.
- Add a little more debug to try and understand how we could get the
following error I saw within GHA
```
path_tidy_test.go:769: Created root and leaf certificate, deleted leaf, but a got a certificate count of 2
```
- Fix some other issues discovered while running the test through
stress locally
To allow us to support CIEPS backend state, allow the backend to
contain enterprise only state variables. Also allow us to implement
enterprise only hooks into the various backend functions to initialize,
periodicFunc, cleanup and invalidate.
* Add backend format linting to pre-commit hook
By taking a slight penalty with each commit, we can ensure that
contributors follow the format behavior by default (if they run hooks),
making accidental PRs without proper formatting less likely.
Additionally, fix gofmtcheck to align with the Makefile, fixing the
corresponding fmtcheck target for use with the hook.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Fix formatting errors
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
---------
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
- There's a race within the Plugin reloading mechanism that isn't
trivial to address. To silence some of the failures, switch this
test to use sealing of the cores instead of the plugin reload
mechanism
- We've seen a few issues with bind's auto-loading of configuration
too quickly at bad times leading to it having partial configurations
or not all files/permissions being restored properly during it's read
attempt.
- See if the freeze/thaw rndc commands will help out with these timing
issues
* Fix OpenAPI spec definitions for PKI EAB APIs
- Do not generate duplicate operation ids for the various new-eab apis
- Fill out proper operation verb for eab delete call
- Pluralize operation verb for list-eab-keys api
- Fill out proper response data for new-eab and list-eab-keys
* Add cl
* openapi: Fix response schema for PKI Issue requests
* tests
* changelog
* another expiration for generate/rotate root
* more type fixes from @stevendpclark
* Replace all time.ParseDurations with testutil.ParseDurationSeconds
* Changelog
* Import formatting
* Import formatting
* Import formatting
* Import formatting
* Semgrep rule that runs as part of CI
* Supporting PR for Enterprise ACME PR cluster tests
- Some changes within the OSS test helpers to help in the ACME Enterprise test cases.
* Don't rename existing helper method to make oss/ent merge easier
* Adds automated ACME tests using Caddy.
* Do not use CheckSignatureFrom method to validate TLS-ALPN-01 challenges
* Uncomment TLS-ALPN test.
* Fix validation of tls-alpn-01 keyAuthz
Surprisingly, this failure was not caught by our earlier, but unmerged
acme.sh tests:
> 2023-06-07T19:35:27.6963070Z [32mPASS[0m builtin/logical/pkiext/pkiext_binary.Test_ACME/group/acme.sh_tls-alpn (33.06s)
from https://github.com/hashicorp/vault/pull/20987.
Notably, we had two failures:
1. The extension's raw value is not used, but is instead an OCTET
STRING encoded version:
> The extension has the following ASN.1 [X.680] format :
>
> Authorization ::= OCTET STRING (SIZE (32))
>
> The extnValue of the id-pe-acmeIdentifier extension is the ASN.1
> DER encoding [X.690] of the Authorization structure, which
> contains the SHA-256 digest of the key authorization for the
> challenge.
2. Unlike DNS, the SHA-256 is directly embedded in the authorization,
as evidenced by the `SIZE (32)` annotation in the quote above: we
were instead expecting this to be url base-64 encoded, which would
have a different size.
This failure was caught by Matt, testing with Caddy. :-)
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Quick gofmt run.
* Fix challenge encoding in TLS-ALPN-01 challenge tests
* Rename a PKI test helper that retrieves the Vault cluster listener's cert to distinguish it from the method that retrieves the PKI mount's CA cert. Combine a couple of Docker file copy commands into one.
---------
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Co-authored-by: Steve Clark <steven.clark@hashicorp.com>
Co-authored-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Ignore templated AIA on root generation
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add test case verifying that roots generate OK
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add changelog entry
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add warning on generation
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
---------
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Correctly validate ACME PoP against public key
ACME's proof of possession based revocation uses a signature from the
private key, but only sends the public copy along with the request.
Ensure the public copy matches the certificate, instead of failing to
cast it to a private key.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add ACME revocation tests
* Clarify commentary in acmeRevocationByPoP
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
---------
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Co-authored-by: Steve Clark <steven.clark@hashicorp.com>
* Allow CSRs with basic constraint extension with IsCA=false
- We previously forbid any CSR with a basic constraint extension within the CSR.
- It was discovered that some ACME clients (Proxmox ACME client) do send us this extension with a value of IsCA to false.
- So allow the extension to be set within the ACME CSR with
a value of IsCA set to false
- Add a test for both the IsCA=true and IsCA=false use-cases and make sure we don't actually set the extension back within the generated certificate.
* PR feedback
- Move basic constraint function to sdk, increase test coverage
- Error out on extra characters being returned from the asn1 unmarshalling.
* make fmt
* Signal ACME challenge engine if existing challenges were loaded
- Addresses an issue of existing challenges on disk not being processed until a new challenge is accepted when Vault restarts
- Move loading of existing challenges from the plugin's initialize method into the challenge engine's thread
- Add docker test that validates we addressed the issue and ACME works across standby nodes.
* Add cl
* Add ACME TLS-ALPN-01 Challenge validator to PKI
This adds support for verifying the last missing challenge type,
TLS-ALPN-01 challenges, using Go's TLS library. We wish to add this as
many servers (such as Caddy) support transparently renewing certificates
via this protocol, without influencing the contents of sites served.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Enable suggesting, validating tls-alpn-01 in PKI
Notably, while RFC 8737 is somewhat vague about what identifier types
can be validated with this protocol, it does restrict SANs to be only
DNSSans; from this, we can infer that it is not applicable for IP
typed identifiers. Additionally, since this must resolve to a specific
domain name, we cannot provision it for wildcard identifiers either.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Fix test expectations to allow ALPN challenges
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add tls-alpn-01 as a supported challenge to docs
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add test for tls-alpn-01 challenge verifier
This hacks the challenge engine to allow non-standard (non-443) ports,
letting us use a local server listener with custom implementation.
In addition to the standard test cases, we run:
- A test with a longer chain (bad),
- A test without a DNSSan (bad),
- A test with a bad DNSSan (bad),
- A test with some other SANs (bad),
- A test without a CN (good),
- A test without any leaf (bad), and
- A test without the extension (bad).
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add changelog entry
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Update builtin/logical/pki/acme_challenges.go
Co-authored-by: Alexander Scheel <alex.scheel@hashicorp.com>
---------
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Co-authored-by: Kit Haines <khaines@mit.edu>
* Allow not specifying CN on CSR
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add test case validating behavior
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add notice about failure to validate
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
---------
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
When writing DNS configs, make sure to push the zone file prior to
writing the reference to the zone in the named.conf.options file.
Otherwise, when adding the initial domain (or any subsequent domains,
which isn't really used by these tests), a race occurs between Docker,
writing the updated config files, and the bind daemon, detecting that
mtime has changed on the named.conf.options file and reloading it
and any referenced zone files.
This should fix the error seen in some tests:
> /etc/bind/named.conf:9: parsing failed: file not found
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Limit ACME issued certificates to a max of 90 days
- If the normal NotAfter date that is calculated from the mount/issuer/role TTL values is greater than 90 days, override the TTL value to a 90-day value.
* Add changelog
* Generate ACME EAB tokens that do not start with -
- To avoid people having issues copying EAB tokens and using them on command lines when they start with - from the base64 encoded values, append a prefix.
- Remove the key_bits data from the eab api, not really useful and now technically wrong
- Fix up some issues with tests not running in parallel.
- Update docs to reflect new EAB apis.
* Add ACME directory to the various EAB output APIs
* Update EAB token prefix to be divisable by 3
- Our decoded prefix was not divisable by 3, which meant the last
character might be tweaked by the rest of the input
* Attempt to resolve flaky test TestAcmeBasicWorkflow test
- Since we can't control the challenge engine, flush the validation records it leverages prior to manually updating the authorization/challenge statuses
```
path_acme_test.go:261: csr: &{[] [] [] [] 0 [] 0 0 <nil> CN=*.localdomain [] [] [] [localhost.localdomain *.localdomain] [] [] []}
path_acme_test.go:300:
Error Trace: /home/runner/actions-runner/_work/vault-enterprise/vault-enterprise/builtin/logical/pki/path_acme_test.go:300
Error: Received unexpected error:
403 urn:ietf:params:acme:error:orderNotReady: The request attempted to finalize an order that is not ready to be finalized: order is status pending, needs to be in ready state
Test: TestAcmeBasicWorkflow/role
Messages: failed finalizing order
```
* make fmt
* Change from default_role to default_directory_policy to allow future improvements.
* Helper functions
* Use the helper function and make fmt.
* Do not allow the zero-length role "".
* Semgrep doesn't like shadowing errors that are impossible to hit, so fix that.
* Add default to switch branches.
* Add/fix docs.
* Fix wrong requestedRole
* Fix ACME computed order status
* Return validation errors and status updates for authorizations
- We now populate the error field within challenges with the error results from the challenge
- Update the status of the challenge and authorizations to invalid when we give up on the challenge
- Verify that only a single challenge within a given authorization can be accepted to avoid race conditions.
This should help to prevent the issue of missing tidy configurations
in the future, by placing all related configuration options at the
top with common validation logic.
However, short from this approach is ensuring that each config option
can be specified independently. Thus, the test allows (for any added
and properly tracked tidy operations) verifying that we have enabled
proper storage/retention of that attribute.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Update EAB management urls underneath pki/eab
- It was decided that for ease of ACLing, the management
paths for EAB apis should be outside of the acme path
prefix
- Delete duplicated tests, rely on the proper cluster
based tests for EAB management.
* Update ACME EAB creation paths to be directory specific
- Make the EAB creation APIs directory specific.
- This commit is still missing the enforcement that
they can be redeemed on a specific path.
* Enforce EAB tokens per ACME directory context like accounts
- Do not allow an EAB from one ACME directory to be used
in another.
- Rework the ACME directory function to simply get the path from the request instead of parsing out the role/issuer name.
- Add some commentary around expectations if operators change issuer names
* Add an EAB certbot integration test
- Verify with the 3rd party certbot cli that our EAB workflow works as expected.
* Fix unit test
- Unit test wasn't setting up r.Path within the request
that we now use to determine the acme directory.
