* [VAULT-40165] pipeline(github): add `check go-mod-diff` command
Add `pipeline github check go-mod-diff` command that is capable of
creating a Go module diff between one-or-more go.mod files in two
different Github branches. There are flags for the owner, repo, and
branch for both the A and B sides of the diff, as well as the `--path`
or `-p` flag that can be specified any number of times with relative
paths in the repository of go.mod files to compare. We assume that the
path is the same in both repositories.
This work will be followed up with another PR that removes the
enterprise only go.mod file and enables Go module diff checking on pull
requests to CE branches that change the go toolchain.
Signed-off-by: Ryan Cragun <me@ryan.ec>
Co-authored-by: Ryan Cragun <me@ryan.ec>
* [VAULT-40043]: pipeline: add `go diff mod` command
Add a `pipeline go diff mod` command that is capable of comparing two
go.mod files at a directive level. We also support strict or lax
comparisons of several directives to flexible diff comparisons. This is
especially useful when you want to compare two go.mod files that have
some different dependencies (CE vs. Ent) but still want to compare
versions of like dependencies.
This command is not currently used in the pipeline but was useful in
developing the diff library that is used. Subsequent work will use the
library and be integrated into CI.
* review feedback
* one more comment fix
---------
Signed-off-by: Ryan Cragun <me@ryan.ec>
Co-authored-by: Ryan Cragun <me@ryan.ec>
* license: update headers to IBM Corp.
* `make proto`
* update offset because source file changed
Signed-off-by: Ryan Cragun <me@ryan.ec>
Co-authored-by: Ryan Cragun <me@ryan.ec>
* pipeline(changed-files): fix false positives for some files (#10239)
Signed-off-by: Ryan Cragun <me@ryan.ec>
* make fmt
Signed-off-by: Ryan Cragun <me@ryan.ec>
---------
Signed-off-by: Ryan Cragun <me@ryan.ec>
Co-authored-by: Ryan Cragun <me@ryan.ec>
* [VAULT-39424] pipeline(close-origin-pr): add support for closing the origin of copied PRs
When we copy a community contributed Pull Request to Enterprise the
source PR is effectively orphaned, leaving the original PR still
opened, the author unsure of what state the copied PR is in, and any
issues associated with it open.
When the copied PR is closed we ought to close the origin PR if it's
still open, and any other issues that might be associated with either
the origin PR or the copied PR.
We can also add comments to both PRs that include links to each other
and the squash commit to make discovery of the work visible to those
with access to both repos. Unfortunately there is no way to know what
the SHA will be when it's synced so we have to rely on the
'Co-Authored-By:' trailers in commit message.
There are some challenges to this:
- The automation should only execute when copied PRs are closed
- How to determine the origin PR from only the copied PR
- How to determine the PR's linked issues (which the v3 REST API does not expose)
We solved them by:
- Requiring the PR HEAD ref to start with `copy/`
- Encoding the origin PR information in the PR HEAD ref.
e.g. `copy/hashicorp/vault/31580/ryan/VAULT-39424-test-ce`
- Using the V4 GraphQL API to determine "closed issue references"
The result is a new `pipeline` CLI command that can close the origin PR,
all of the issues, and write status comments on each PR with links to
everything to establish omnidirectional linking in the Github UI.
```bash
pipeline github close origin-pull-request 9903
```
* fix feedback
---------
Signed-off-by: Ryan Cragun <me@ryan.ec>
Co-authored-by: Ryan Cragun <me@ryan.ec>
Occasionally it seems that the tools will get built and linked against
the platform glibc. We definitely do not want that.
Now we always disable CGO when building tools. While doing this I
realized that we could also strip debug symbols and reduce the size of
the tools significantly, so that is included as well.
Signed-off-by: Ryan Cragun <me@ryan.ec>
Co-authored-by: Ryan Cragun <me@ryan.ec>
When we copy a Pull Request from CE to Ent we already add a status
comment to the origin PR but we don't actually bubble up the information
to the workflow summary. Instead, render the copy PR output as a
markdown table and write it to the step summary.
Signed-off-by: Ryan Cragun <me@ryan.ec>
Co-authored-by: Ryan Cragun <me@ryan.ec>
* pipeline(copy-pr): cherry-pick commits instead of merging
* fix staticcheck for docs in pkg/github
Signed-off-by: Ryan Cragun <me@ryan.ec>
Co-authored-by: Ryan Cragun <me@ryan.ec>
Add a new `github` changed file group that includes everything in the
`.github` directory. Further refine the `pipeline` group to only
include scripts, workflows, and actions files in `.github`. We also move
the `CODEOWNERS` file into `.github/` to simplify `github` grouping.
As `build` logic responds to changes to the `pipeline` group this will
result in no longer building and testing everything for simple
changes in `github` that don't affect the pipeline.
Signed-off-by: Ryan Cragun <me@ryan.ec>
Co-authored-by: Ryan Cragun <me@ryan.ec>
When determining whether to skip a backport ref we currenly we have to
consider many factors:
- Whether or not there are changed files?
- If there are changed files, are some enterprise or CE?
- Are there some changed files that ought to be backported to inactive
branches?
- Is the target branch active or not?
We had a large test suite that covered _most_ of these cases but because
the changed file set determines a lot of behavior we were missing cases
where we ought to backport normal mixed changed file sets for no other
reason other than the branch is active. After fixing and normalizing the
tests I fixed the source bug which is that we didn't strip the branch
prefix from the ref version when checking branch activeness.
Signed-off-by: Ryan Cragun <me@ryan.ec>
Co-authored-by: Ryan Cragun <me@ryan.ec>
[VAULT-39160] actions(hcp): add support for testing custom images on HCP (#9345)
Add support for running the `cloud` scenario with a custom image in the
int HCP environment. We support two new tags that trigger new
functionality. If the `hcp/build-image` tag is present on a PR at the
time of `build`, we'll automatically trigger a custom build for the int
environment. If the `hcp/test` tag is present, we'll trigger a custom
build and run the `cloud` scenario with the resulting image.
* Fix a bug in our custom build pattern to handle prerelease versions.
* pipeline(hcp): add `--github-output` support to `show image` and
`wait image` commands.
* enos(hcp/create_vault_cluster): use a unique identifier for HVN
and vault clusters.
* actions(enos-cloud): add workflow to execute the `cloud` enos
scenario.
* actions(build): add support for triggering a custom build and running
the `enos-cloud` scenario.
* add more debug logging and query without a status
* add shim build-hcp-image for CE workflows
Signed-off-by: Ryan Cragun <me@ryan.ec>
Co-authored-by: Ryan Cragun <me@ryan.ec>
* license: add support for publishing artifacts to IBM PAO (#8366)
Signed-off-by: Ryan Cragun <me@ryan.ec>
Co-authored-by: brian shore <bshore@hashicorp.com>
Co-authored-by: Ethel Evans <ethel.evans@hashicorp.com>
Co-authored-by: Ryan Cragun <me@ryan.ec>
* [VAULT-39158] actions(build-hcp-image): various small fixes
Various small fixes to correctly trigger custom image builds and wait
for them to be available.
Signed-off-by: Ryan Cragun <me@ryan.ec>
Co-authored-by: Ryan Cragun <me@ryan.ec>
* [VAULT-39159]: pipeline: add support for querying HCP image service
In order to facilitate testing Vault Enterprise directly in HCP we need
tools to both request an image be built from a candidate build and to
also wait for the image to be available in order to execute test
scenarios with it. This PR adds a few new `pipeline` sub-commands that
can will be used for this purpose.
`pipeline github find workflow-artifact` can be used to find the path of
an artifact that matches the given filter criteria. You'll need to
provide a pull request number, workflow name, and either an exact
artifact name or a pattern. When providing a pattern only the first
match will be returned so make sure your regular expression is robust.
`pipeline hcp get image` will return the image information for an HCP
image. You will need to supply auth via the `HCP_USERNAME` and
`HCP_PASSWORD` environment variables in order to query the image
service. It also takes an enviroment flag so you can query the image
service in different environments.
`pipeline hcp wait image` is like `pipeline hcp get image` except that
it will continue to retry for a given timeout and with a given delay
between requests. In this way it can be used to wait for an image to be
available.
As part of this we also update our Go modules to the latest versions
that are compatible.
* [VAULT-39158]: actions(build-hcp-image): add workflow for building HCP images
* copywrite: add missing headers
* remove unused output
* address feedback
* allow prerelease artifacts
---------
Signed-off-by: Ryan Cragun <me@ryan.ec>
Co-authored-by: Ryan Cragun <me@ryan.ec>
As part of this we also update the pin of gotestsum to 1.12.3 to allow
for building it with Go 1.25.
Signed-off-by: Ryan Cragun <me@ryan.ec>
Co-authored-by: Ryan Cragun <me@ryan.ec>
* always fetch HEAD repo with it's clone URL
* use the number instead of ID when generating the summary table
Signed-off-by: Ryan Cragun <me@ryan.ec>
Co-authored-by: Ryan Cragun <me@ryan.ec>
Don't categorize changelog files that begin with an underscore as
enterprise only, otherwise they'll be removed when backporting changes
to CE.
Since we want to include links to commit SHAs in the changelog we have
to create the changelog in the context of CE and thus need to backport
all of those changes.
We also fix a few Go tests that hand not been updated to handle the
updated default inactive CE groups.
Signed-off-by: Ryan Cragun <me@ryan.ec>
Co-authored-by: Ryan Cragun <me@ryan.ec>
Docs have been moved since the tool was written so that exclusion is no
longer needed. Since the defaults were added the `pipeline` group has
expanded to include all `.github`, which we don't want to always
backport. It seems unlike that `pipeline` tooling changes are likely to
be required often on inactive branches so we'll exclude all together for
now.
Signed-off-by: Ryan Cragun <me@ryan.ec>
Co-authored-by: Ryan Cragun <me@ryan.ec>
Ubunut 20.04 is EOL. Per our support and package policies we no longer
need to develop or test for that platform.
Signed-off-by: Ryan Cragun <me@ryan.ec>
Add a new `pipeline github sync branches` command that can synchronize
two branches. We'll use this to synchronize the
`hashicorp/vault-enterprise/ce/*` branches with `hashicorp/vault/*`.
As the community repository is effectively a mirror of what is hosted in
Enterprise, a scheduled sync cadence is probably fine. Eventually we'll
hook the workflow and sync into the release pipeline to ensure that
`hashicorp/vault` branches are up-to-date when cutting community
releases.
As part of this I also fixed a few static analysis issues that popped up
when running `golangci-lint` and fixed a few smaller bugs.
Signed-off-by: Ryan Cragun <me@ryan.ec>
The standard JSON output of `pipeline release list active-versions
--format json` returns the configuration in the same schema as
`.release/versions.hcl`. While that is intended for the command, the
schema itself is not easy to use with the built-in functions in Github
Actions which often requires complex JQ queries to reshape the data to
be useful. Instead, add a `--github-output` flag that will
automatically write the active versions to `$GITHUB_OUTPUT` encoded as
JSON but with multiple useful top-level keys that make utilizing it in
Github Actions a breeze. This support will be utilized in future pull
requests.
Signed-off-by: Ryan Cragun <me@ryan.ec>
After the merge workflow has been reversed and branches hosted in
`hashicorp/vault` are downstream from community branches hosted in
`hashicorp/vault-enterprise`, most contributions to the source code
will originate in `hashicorp/vault-enterprise` and be backported to
a community branch in hosted in `hashicorp/vault-enterprise`. These
community branches will be considered the primary source of truth and
we'll automatically push changes from them to mirrors hosted in
`hashicorp/vault`.
This workflow ought to yield a massive efficiency boost for HashiCorp
contributors with access to `hashicorp/vault-enterprise`. Before the
workflow would look something like:
- Develop a change in vault-enterprise
- Manually extract relevant changes from your vault-enterprise branch
into a new community vault branch.
- Add any stubs that might be required so as to support any enterprise
only changes.
- Get the community change reviewed. If changes are necessary it often
means changing and testing them on both the enteprise and community
branches.
- Merge the community change
- Wait for it to sync to enterprise
- *Hope you changes have not broken the build*. If they have, fix the
build.
- Update your enterprise branch
- Get the enterprise branch reviewed again
- Merge enterprise change
- Deal with complicated backports.
After the workflow will look like:
- Develop the change on enterprise
- Get the change reviewed
- Address feedback and test on the same branch
- Merge the change
- Automation will extract community changes and create a community
backport PR for you depending on changes files and branch
activeness.
- Automation will create any enterprise backports for you.
- Fix any backport as necessary
- Merge the changes
- The pipeline will automatically push the changes to the community
branch mirror hosted in hashicorp/vault.
No more
- Duplicative reviews
- Risky merges
- Waiting for changes to sync from community to enterprise
- Manual decompistion of changes from enterprise and community
- *Doing the same PR 3 times*
- Dealing with a different backport process depending on which branches
are active or not.
These changes do come at cost however. Since they always originate from
`vault-enterprise` only HashiCorp employees can take advatange of the
workflow. We need to be able to support community contributions that
originate from the mirrors but retain attribution.
That's what this PR is designed to do. The community will be able to
open a pull request as normal and have it reviewed as such, but rather
than merging it into the mirror we'll instead copy the PR and open it
against the corresponding enterprise base branch and have it merged it
from there. The change will automatically get backported to the
community branch if necessary, which eventually makes it back to the
mirror in hashicorp/vault.
To handle our squash merge workflow while retaining the correct
attribution, we'll automatically create merge commits in the copied PR
that include `Co-Authored-By:` trailers for all commit authors on the
original PR.
We also take care to ensure that the HashiCorp maitainers that approve
the PR and/or are assigned to it are also assigned to the copied PR.
This change is only the tooling to enable it. The workflow that drives
it will be implemented in VAULT-34827.
Signed-off-by: Ryan Cragun <me@ryan.ec>
Ubuntu 20.04 has reached EOL and is no longer a supported runner host distro. Historically we've relied on it for our CGO builds as it contains an old enough version of glibc that we can retain compatibility with all of our supported distros and build on a single host distro. Rather than requiring a new RHEL 8 builder (or some equivalent), we instead build CGO binaries inside an Ubuntu 20.04 container along with its glibc and various C compilers.
I've separated out system package changes, the Go toolchain install, and external build tools tools install into different container layers so that the builder container used for each branch is maximally cacheable.
On cache misses these changes result in noticeably longer build times for CGO binaries. That is unavoidable with this strategy. Most of the time our builds will get a cache hit on all layers unless they've changed any of the following:
- .build/*
- .go-version
- .github/actions/build-vault
- tools/tools.sh
- Dockerfile
I've tried my best to reduce the cache space used by each layer. Currently our build container takes about 220MB of cache space. About half of that ought to be shared cache between main and release branches. I would expect total new cache used to be in the 500-600MB range, or about 5% of our total space.
Some follow-up idea that we might want to consider:
- Build everything inside the build container and remove the github actions that set up external tools
- Instead of building external tools with `go install`, migrate them into build scripts that install pre-built `linux/amd64` binaries
- Migrate external to `go tool` and use it in the builder container. This requires us to be on 1.24 everywhere so ought not be considered until that is a reality.
Signed-off-by: Ryan Cragun <me@ryan.ec>
Add a new `github create backport` sub-command that can create a
backport of a given pull request. The command has been designed around a
Github Actions workflow where it is triggered on a closed pull request
event with a guard that checks for merges:
```yaml
pull_request_target:
types: closed
jobs:
backport:
if: github.even.pull_request.merged
runs-on: "..."
```
Eventually this sub-command (or another similar one) can be used to
implemente backporting a CE pull request to the corresponding ce/*
branch in vault-enterprise. This functionality will be implemented in
VAULT-34827.
This backport runner has several new behaviors not present in the
existing backport assistant:
- If the source PR was made against an enterprise branch we'll assume
that we want create a CE backport.
- Enterprise only files will be automatically _removed_ from the CE
backport for you. This will not guarantee a working CE pull request
but does quite a bit of the heavy lifting for you.
- If the change only contains enterprise files we'll skip creating a
CE backport.
- If the corresponding CE branch is inactive (as defined in
.release/versions.hcl) then we will skip creating a backport in most
cases. The exceptions are changes that include docs, README, or
pipeline changes as we assume that even active branches will want
those changes.
- Backport labels still work but _only_ to enterprise PR's. It is
assumed that when the subsequent PRs are merged that their
corresponding CE backports will be created.
- Backport labels no longer include editions. They will now use the
same schema as active versions defined .release/verions.hcl. E.g.
`backport/1.19.x`. `main` is always assumed to be active.
- The runner will always try and update the source PR with a Github
comment regarding the status of each individual backport. Even if
one attempt at backporting fails we'll continue until we've
attempted all backports.
Signed-off-by: Ryan Cragun <me@ryan.ec>
While working on VAULT-34829 it became apparent that if our new backporter
could know which branches are active and which CE counterparts are active
then we could completely omit the need for `ce` backport labels and instead
automatically backport to corresponding CE branches that are active.
To facilitate that we can re-use our `.release/versions.hcl` file as it is
the current source of truth for our present backport assistant workflow.
Here we add a new `pipeline releases list versions` command that is capable
of decoding that file and optionally displaying it. It will be used in the
next PR that fully implements VAULT-34829.
As part of this work we refactors `pipeline releases` to include a new `list`
sub-command and moved both `list-active-versions` and `versions` to it.
We also include a few small fixes that were noticed:
- `.release/verions.hcl` was not up-to-date
- Our cached dynamic config was not getting recreated when the pipeline
tool changed. That has been fixed so now dynamic config should always
get recreated when the pipeline binary changes
- We now initialize a git client when using the `github` sub-command.
This will be used in more forthcoming work
- Update our changed file detection to resolve some incorrect groupings
- Add some additional changed file helpers that we be used in forthcoming
work
Signed-off-by: Ryan Cragun <me@ryan.ec>
As I was working on VAULT-34829 it became clear that we needed to solve
the problem of using Git from Go. Initially I tried to use the
go-git/go-git pure Go implementation of Git, but it lacked several
features that we needed.
The next best option seemed to be shelling out to Git. What started as a
simple way to execute Git commands with the requisite environment and
configuration led to the implementation.
A few notes:
- I did not add support for all flags for the implemented sub-commands
- I wanted it to handle automatic inject and configuration of PATs when
operating against private remote repositories in Github.
- I did not want to rely on the local machine having been configured.
The client that is built in is capable of configuring everything that
we need via environment variables.
- I chose to go the environment variable route for configuration as it's
the only way to not overwrite local configuration or set up our own
cred store.
As the git client ended up being ~50% of the work for VAULT-34829, I
decided to extract it out into its own PR to reduce the review burden.
NOTE: While we don't use it in CI, there is .golanci.yml present in the
the tree and it causes problems in editors that expect it to be V2. As
such I migrated it.
Signed-off-by: Ryan Cragun <me@ryan.ec>
* enos(artifactory): unify dev and test scenario artifactory metadata into new module
There was previously a lot of shared logic between
`build_artifactory_artifact` and `build_artifactory_package` as it
regards to building an artifact name. When it comes down to it, both
modules are very similar and their only major difference is searching
for any artifact (released or not) by either a combination of
`revision`, `edition`, `version`, and `type` vs. searching for a
released artifact with a combination of `version`, `edition`, and
`type`.
Rather than bolt on new `s390x` and `fips1403` artifact metadata to
both, I factored their metadata for package names and such into a
unified and shared `artifact/metadata` module that is now called by
both.
This was tricky as dev and test scenarios currently differ in what
we pass in as the `vault_version`, but we hope to remove that
difference soon. We also add metadata support for the forthcoming
FIPS 140-3.
This commit was tested extensively, along with other test scenarios
in support for `s390x but will be useful immediately for FIPS 140-3
so I've extracted it out.
Signed-off-by: Ryan Cragun <me@ryan.ec>
* Fix artifactory metadata before merge
The initial pass of the artifactory metadata was largely untested and
extracted from a different branch. After testing, this commit fixes a
few issues with the metadata module.
In order to test this I also had to fix an issue where AWS secrets
engine testing became a requirement but is impossible unless you exectue
against a blessed AWS account that has required roles. Instead, we now
make those verification opt-in via a new variable.
We also make some improvements to the pki-verify-certificates script so
that it works reliably against all our supported distros.
We also update our dynamic configuration to use the updated versions in
samples.
Signed-off-by: Ryan Cragun <me@ryan.ec>
* VAULT-34834: pipeline: add better heuristics for changed files
To fully support automated Enterprise to Community backports we need to
have better changed file detection for community and enterprise only
files. Armed with this metadata, future changes will be able to inspect
changed files and automatically remove enterprise only files when
creating the CE backports.
For this change we now have the following changed file groups:
- autopilot
- changelog
- community
- docs
- enos
- enterprise
- app
- gotoolchain
- pipeline
- proto
- tools
- ui
Not included in the change, but something I did while updating out
checkers was generate a list of files that included only in
vault-enterprise and run every path the enterprise detection rules
to ensure that they are categorized appropriately post changes in
VAULT-35431. While it's possible that they'll drift, our changed
file categorization is best effort anyway and changes will always
happen in vault-enterprise and require a developer to approve the
changes.
We've also included a few new files into the various groups and updated
the various workflows to use the new categories. I've also included a
small change to the pipeline composite action whereby we do not handle
Go module caching. This will greatly reduce work on doc-only branches
that need only ensure that the pipeline binary is compiled.
Signed-off-by: Ryan Cragun <me@ryan.ec>
* VAULT-34822: Add `pipeline github list changed-files`
Add a new `github list changed-files` sub-command to `pipeline` command and
integrate it into the pipeline. This replaces our previous
`changed-files.sh` script.
This command works quite a bit differently than the full checkout and
diff based solution we used before. Instead of checking out the base ref
and head ref and comparing a diff, we now provide either a pull request
number or git commit SHA and use the Github REST API to determine the
changed files.
This approach has several benefits:
- Not requiring a local checkout of the repo to get the list of
changed files. This yields a significant perfomance improvement in
`setup` jobs where we typically determine the changed files list.
- The CLI supports both PRs and commit SHAs.
- The implementation is portable and doesn't require any system tools
like `git` or `bash` to be installed.
- A much more advanced system for adding group metadata to the changed
files. These groupings are going to be used heavily in future
pipeline automation work and will be used to make required jobs
smarter.
The theoretical drawbacks:
- It requires a GITHUB_TOKEN and only works for remote branches or
commits in Github. We could eventually add a local diff sub-command
or option to work locally, but that was not required for what we're
trying to achieve here.
While the groupings that I added in this change are quite rudimentary,
the system will allow us to add additional groups with very little
overhead. I tried to make this change more or less a port of the old
system to enable future work. I did include one small change of
behavior, which is that we now build all extended targets if the
`go.mod` or `go.sum` files change. We do this to ensure that dependency
changes don't subtly result in some extended platform breakage.
Signed-off-by: Ryan Cragun <me@ryan.ec>
`gosimports` is the preferred style for module imports and it is
enforced via CI. I've found that things often manage to drift so I've
taken the liberty to update our pre-commit hook to verify our imports
formatting before a change is committed.
Along with updating the formatting helper I've also run `make fmt` to
resolve any formatting drift that managed to make it into the codebase.
Signed-off-by: Ryan Cragun <me@ryan.ec>
* VAULT-33074: add `github` sub-command to `pipeline`
Investigating test workflow failures is common task that engineers on the
sustaining rotation perform. This task often requires quite a bit of
manual labor by manually inspecting all failed/cancelled workflows in
the Github UI on per repo/branch/workflow basis and performing root cause
analysis.
As we work to improve our pipeline discoverability this PR adds a new `github`
sub-command to the `pipeline` utility that allows querying for such workflows
and returning either machine readable or human readable summaries in a single
place. Eventually we plan to automate sending a summary of this data to
an OTEL collector automatically but for now sustaining engineers can
utilize it to query for workflows with lots of various criteria.
A common pattern for investigating build/enos test failure workflows would be:
```shell
export GITHUB_TOKEN="YOUR_TOKEN"
go run -race ./tools/pipeline/... github list-workflow-runs -o hashicorp -r vault -d '2025-01-13..2025-01-23' --branch main --status failure build
```
This will list `build` workflow runs in `hashicorp/vault` repo for the
`main` branch with the `status` or `conclusion` of `failure` within the date
range of `2025-01-13..2025-01-23`.
A sustaining engineer will likely do this for both `vault` and
`vault-enterprise` repositories along with `enos-release-testing-oss` and
`enos-release-testing-ent` workflows in addition to `build` in order to
get a full picture of the last weeks failures.
You can also use this utility to summarize workflows based on other
statuses, branches, HEAD SHA's, event triggers, github actors, etc. For
a full list of filter arguments you can pass `-h` to the sub-command.
> [!CAUTION]
> Be careful not to run this without setting strict filter arguments.
> Failing to do so could result in trying to summarize way too many
> workflows resulting in your API token being disabled for an hour.
Signed-off-by: Ryan Cragun <me@ryan.ec>
As the Vault pipeline and release processes evolve over time, so too must the tooling that drives them. Historically we've utilized a combination of CI features and shell scripts that are wrapped into make targets to drive our CI. While this
approach has worked, it requires careful consideration of what features to use (bash in CI almost never matches bash in developer machines, etc.) and often requires a deep understanding of several CLI tools (jq, etc). `make` itself also has limitations in user experience, e.g. passing flags.
As we're all in on Github Actions as our pipeline coordinator, continuing to utilize and build CLI tools to perform our pipeline tasks makes sense. This PR adds a new CLI tool called `pipeline` which we can use to build new isolated tasks that we can string together in Github Actions. We intend to use this utility as the interface for future release automation work, see VAULT-27514.
For the first task in this new `pipeline` tool, I've chosen to build two small sub-commands:
* `pipeline releases list-versions` - Allows us to list Vault versions between a range. The range is configurable either by setting `--upper` and/or `--lower` bounds, or by using the `--nminus` to set the N-X to go back from the current branches version. As CE and ENT do not have version parity we also consider the `--edition`, as well as none-to-many `--skip` flags to exclude specific versions.
* `pipeline generate enos-dynamic-config` - Which creates dynamic enos configuration based on the branch and the current list of release versions. It takes largely the same flags as the `release list-versions` command, however it also expects a `--dir` for the enos directory and a `--file` where the dynamic configuration will be written. This allows us to dynamically update and feed the latest versions into our sampling algorithm to get coverage over all supported prior versions.
We then integrate these new tools into the pipeline itself and cache the dynamic config on a weekly basis. We also cache the pipeline tool itself as it will likely become a repository for pipeline specific tooling. The caching strategy for the `pipeline` tool itself will make most workflows that require it super fast.
Signed-off-by: Ryan Cragun <me@ryan.ec>
* change no export data status to be 204 instead of 400
* add identity metadata for JSON and CSV with column flattening
* add condition to nil-check-physical-storage-by-nsid semgrep rule
* add TestActivityLog_Export_CSV_Header test
* fix tests
* add changelog entry
