* an attempt to add parsing of wilcards
* test coverage and changelog
* fix policy path
* fix failing tesT
* fix comments
* add in go doc comment
* update second internal ui resultant acl test
Co-authored-by: Angel Garbarino <Monkeychip@users.noreply.github.com>
Co-authored-by: Tony Wittinger <anwittin@users.noreply.github.com>
* first full pass with new filtering
* updates and making dropdowns searchable
* fixing tests
* updates, test fix
* update version dropdown
* update icons
* comments and cleanup
* filter fixes, update template and add test
* fix tests
* fix tests but not insane
* update, changelog
Co-authored-by: Dan Rivera <dan.rivera@hashicorp.com>
* actions(install-tools): include os and arch in cache key
When caching and/or restoring our tools we should include the os and
arch in the key to ensure that we don't accidentally download the wrong
tools on different runners.
We also update the nightlies to specifically cache arm64 before running
the tests.
* actionlint: add arm self-hosted runner keys
Signed-off-by: Ryan Cragun <me@ryan.ec>
Co-authored-by: Ryan Cragun <me@ryan.ec>
* actions: use self-hosted runners in hashicorp/vault
While it is recommended that we use self-hosted runners for every
workflow in private and internal accounts, this change was primarily
motivated by different runner types using different cache paths. By
using the same runner type everywhere we can avoid double caches of the
internal Vault tools.
* disable the terraform wrapper in ci-bootstrap to handle updated action
Signed-off-by: Ryan Cragun <me@ryan.ec>
Co-authored-by: Ryan Cragun <me@ryan.ec>
* copy changes WIP
* update descriptions and headers to match latest designs
* add details to method info table
* update test assertion
Co-authored-by: claire bontempo <68122737+hellobontempo@users.noreply.github.com>
* pipeline(copy-pr): cherry-pick commits instead of merging
* fix staticcheck for docs in pkg/github
Signed-off-by: Ryan Cragun <me@ryan.ec>
Co-authored-by: Ryan Cragun <me@ryan.ec>
* [VAULT-39671] tools: use github cache for external tools
We currently have some ~13 tools that we need available both locally for
development and in CI for building, linting, and formatting, and testing Vault.
Each branch that we maintain often uses the same set of tools but often pinned
to different versions.
For development, we have a `make tools` target that will execute the
`tools/tool.sh` installation script for the various tools at the correct pin.
This works well enough but is cumbersome if you’re working across many branches
that have divergent versions.
For CI the problem is speed and repetition. For each build job (~10) and Go test
job (16-52) we have to install most of the same tools for each job. As we have
extremely limited Github Actions cache we can’t afford to cache the entire vault
go build cache, so if we were to build them from source each time we incur a
penalty of downloading all of the modules and building each tool from source.
This yields about an extra 2 minutes per job to install all of the tools. We’ve
worked around this problem by writing composite actions that download pre-built
binaries of the same tools instead of building them from source. That usually
takes a few seconds. The downside of that approach is rate limiting, which
Github has become much more aggressive in enforcing.
That leads us to where we are before this work:
- For builds in the compatibility docker container: the tools are built from
source and cached as separate builder image layer. (usually fast as we get
cache hits, slow on cache misses)
- For builds that compile directly on the runner: the tools are installed on
each job runner by composite github actions (fast, uses API requests, prone
to throttling)
- For tests, they use the same composite actions to install the tools on each
job. (fast, uses API requests, prone to throttling)
This also leads to inconsistencies since there are two sources of truth: the
composite actions have their own version pin outside of those in `tools.sh`.
This has led to drift.
We previously tried to save some API requests and move all builds into
the container. That almost works but docker's build conatiner had a hard
time with some esoteric builds. We could special case it but it's a bandaid at
best.
A prior version of this work (VAULT-39654) investigated using `go tool`, but
there were some showstopper issues with that workflow that make it a non-starter
for us. Instead, we’ll attempt to use more actions cache to resolve the
throttling. This will allow us to have a single source of truth for tools, their
pins, and afford us the same speed on cache hits as we had previously without
downloading the tools from github releases thousands of times per day.
We add a new composite github action for installing our tools.
- On cache misses it builds the tools and installs them into a cacheable path.
- On cache hits it restore the cacheable path.
- It adds the tools to the GITHUB_PATH to ensure runner based jobs can find
them.
- For Docker builds it mounts the tools at `/opt/tools/bin` which is
part of the PATH in the container.
- It uses a cache key of the SHA of the tools directory along with the
working directory SHA which is required to deal with actions/cache
issues.
This results in:
- A single source of truth for tools and their pins
- A single cache for tools that can be re-used between all CI and build jobs
- No more Github API calls for tooling. *_Rate limiting will be a thing of
the past._*
Signed-off-by: Ryan Cragun <me@ryan.ec>
Co-authored-by: Ryan Cragun <me@ryan.ec>
* use "redirect" instead of "afterModel"
* fix styling of radio group buttons
* remove redundant route redirect
* wrap mount dropdown in loading conditional
* reuse parent redirect logic, delete unused outlet
* minor padding adjustments
* force restart tests
Co-authored-by: claire bontempo <68122737+hellobontempo@users.noreply.github.com>
* Updates post-install text to properly reflect the updated IPLA blurb
* Adds changelog for SMRE-733
Co-authored-by: ethel-hashicorp <ethel.evans@hashicorp.com>
Add a new `github` changed file group that includes everything in the
`.github` directory. Further refine the `pipeline` group to only
include scripts, workflows, and actions files in `.github`. We also move
the `CODEOWNERS` file into `.github/` to simplify `github` grouping.
As `build` logic responds to changes to the `pipeline` group this will
result in no longer building and testing everything for simple
changes in `github` that don't affect the pipeline.
Signed-off-by: Ryan Cragun <me@ryan.ec>
Co-authored-by: Ryan Cragun <me@ryan.ec>
When determining whether to skip a backport ref we currenly we have to
consider many factors:
- Whether or not there are changed files?
- If there are changed files, are some enterprise or CE?
- Are there some changed files that ought to be backported to inactive
branches?
- Is the target branch active or not?
We had a large test suite that covered _most_ of these cases but because
the changed file set determines a lot of behavior we were missing cases
where we ought to backport normal mixed changed file sets for no other
reason other than the branch is active. After fixing and normalizing the
tests I fixed the source bug which is that we didn't strip the branch
prefix from the ref version when checking branch activeness.
Signed-off-by: Ryan Cragun <me@ryan.ec>
Co-authored-by: Ryan Cragun <me@ryan.ec>
* make allowed and denied_parameters compare lists
* change name of env var
* add changelog
* linter fixes and unnecessary code removal
Co-authored-by: Bruno Oliveira de Souza <bruno.souza@hashicorp.com>
- Now that all branches are upgraded to Go 1.24 we no longer need
to guard against older branches from being upgraded for FIPS reasons.
Co-authored-by: Steven Clark <steven.clark@hashicorp.com>
The code that loads the trusted certificate cache for cert-based
authentication ignores any error that occurs while attempting to load
any of the certificates that it finds. Undoubtedly some deployments
have broken certificates or other non-certificate files stored in
their respective back-ends, and so this is important behavior: we
don't want to fail authentication just because `README.md` is not a
valid certificate!
In addition, because listing files and loading certificates is
expensive, the server maintains a cache of trusted certificates. This
cache is populated the first time it's needed, and then used for the
lifetime of the process. If a file fails to load as a certificate,
then it is simply not included in the cache.
These two things lead to a problem when using a backend that might be
subject to transient failures: a hiccough in the certificate loading
process can cause the server to establish a cache that is missing an
otherwise valid certificate. This can then lead to clients failing to
authenticate to the server, until such time as the server is restarted
and the cache reloaded.
This change makes the certificate cache more resilient to loading
failures, by caching partial successes. With this patch, the cache
behavior becomes:
- If the cache exists *and* is either complete or it is not yet time
to attempt to reload the certificates, then the cached results are
used without reservation.
- Otherwise we attempt to load the certificates from storage:
- If the cache does not already exist then a new, empty cache is
created.
- The storage is listed, we attempt to load everything in storage,
skipping things that we have already successfully loaded, and
skipping things that we cannot load, as usual.
- Once we have attempted to load everything from storage, if there
were any errors, we compute a deadline for retrying the load, with
an exponentially increasing delay. If there were no errors, then
the cache is considered complete, and there will be no retry.
This has the nice behavior that we recover from transient failures
eventually, while the exponential back-off ensures that we don't waste
too much time attempting to load certificates that can never be
loaded.
Co-authored-by: John Doty <john.doty@databricks.com>
Co-authored-by: Steven Clark <steven.clark@hashicorp.com>
- Instead of using a reserved oid from LetsEncrypt in our tests
and documentation (1.3.6.1.4.1.44947.1.2.4), use
1.3.6.1.4.1.32473.1.2.4, which is in the reserved space for docs
and examples based on RFC 5612
Co-authored-by: Steven Clark <steven.clark@hashicorp.com>
* Add O= restrictions in addition to OU= restrictions
* Add changelog
* Add goDoc to test
* Don't let test certificate expire.
Co-authored-by: Kit Haines <khaines@mit.edu>
[VAULT-39160] actions(hcp): add support for testing custom images on HCP (#9345)
Add support for running the `cloud` scenario with a custom image in the
int HCP environment. We support two new tags that trigger new
functionality. If the `hcp/build-image` tag is present on a PR at the
time of `build`, we'll automatically trigger a custom build for the int
environment. If the `hcp/test` tag is present, we'll trigger a custom
build and run the `cloud` scenario with the resulting image.
* Fix a bug in our custom build pattern to handle prerelease versions.
* pipeline(hcp): add `--github-output` support to `show image` and
`wait image` commands.
* enos(hcp/create_vault_cluster): use a unique identifier for HVN
and vault clusters.
* actions(enos-cloud): add workflow to execute the `cloud` enos
scenario.
* actions(build): add support for triggering a custom build and running
the `enos-cloud` scenario.
* add more debug logging and query without a status
* add shim build-hcp-image for CE workflows
Signed-off-by: Ryan Cragun <me@ryan.ec>
Co-authored-by: Ryan Cragun <me@ryan.ec>
* adding new locks
* adding initial testing
* changelog
* Update changelog/_9336.txt
* fixing test to use proper error output
* added additional locks on pops and last push.
* fixing tests since I cant use t.Fatalf in a go func
* adding rotation window, removing lock as reached a deadlock in test
* removing locking from the OnFailure
* removing unused locks.
---------
Co-authored-by: JMGoldsmith <spartanaudio@gmail.com>
Co-authored-by: John-Michael Faircloth <fairclothjm@users.noreply.github.com>
* Fix unsetting sys tunable values (on ent).
* Remove commented test, add GoDoc for test.
* Handle empty slices better (PR feedback).
* Fetch Auth endpoint without listing (PR feedback).
* Fatal vs. Error
* Add GetAuth instead of ListAuth
* Fix error format error. Oops!
* One more list->get auth. Remove extra check.
* Updated TuneMountWithContextAllowNil to use a struct (with all pointers).
* Allow setting empty values for userLockoutConfig too - use new struct.
* Extra pointer.
* Remove useless functions.
* Simple test to ensure any field we can set we can update and vice-versa.
* Add json tag checks.
Co-authored-by: Kit Haines <khaines@mit.edu>
* support wide width splash page
* add enable_self_enrollment param to mfa-method config
* build and implement mfa setup-card display only component
* fix transition bug navigating away from mfa method
* rename mfa card
* WIP implement self-enrollment workflow
* wip integration tests
* convert mfa-form to typescript
* remove unused import
* show alert whenver there is a QR code
* organze mfa steps into Mfa::VerifyForm and Mfa::SelfEnroll
* WIP stretch goals of mfa redesign
* add copyright headers
* update test
* add support for multiple constraints with self-enrollment
* remove comment
* fix multi-method UX
* fix state for failed validation
* remove changing button for error states
* add error handling and validation messages
* minor cleanup for params
* first round of cleanup and reorganization
* final round of logic cleanup and organization
* touch ups after testing with live backend
* fix comment
* final test cleanup!
* Apply suggestions from code review
* improve mirage error handling to more accurately mimic real failures
* add test coverage
* make qr rendering logic easier
* address PR feedback
* submit enroll form on enter, remove code digit number from copy, reset enroll state
Co-authored-by: claire bontempo <68122737+hellobontempo@users.noreply.github.com>
