* go: resolve CVE-2026-39883 by upgrading go.opentelemetry.io/otel/sdk to v1.43.0
Signed-off-by: Ryan Cragun <me@ryan.ec>
Co-authored-by: Ryan Cragun <me@ryan.ec>
Resolve GHSA-x744-4wpc-v9h2 and GHSA-pxq6-2prw-chj9 in `vault` by replacing
`github.com/docker/docker` with `github.com/moby/moby/client` @ `v0.3.0` and
`github.com/moby/moby/api` @ `v1.54.0`. This is necessary as `docker/docker`
is no longer maintained and the fixes are not available in it.
Resolve GO-2026-4518, GHSA-x6gf-mpr2-68h6 and GHSA-jqcq-xjh3-6g23 by
upgrading to github.com/jackc/pgx/v5. This is necessary as v4 is not
longer maitained.
Signed-off-by: Ryan Cragun <me@ryan.ec>
Co-authored-by: Ryan Cragun <me@ryan.ec>
* [VAULT-43364] pipeline: add template generation support
Add a new `template` to the `pipeline generate` command tree. It allows
rendering Go text templates with pipeline requests available via context
functions. The new system is now product agnostic and can be used to
generate any template we wish. This will supersede the enos specific
configuration command.
We also add support for multiple cadences when fetching the list of
release versions. Previously it was assumed that we followed a minor
version bump cadence when fetching versions with an n-minus style lower
bound. Now we can specify the major or minor cadence. To support a
migration from one cadence to another you can also specify an prior
cadence and the version at which the transition happened. This allows
the n-3 reverse traversal to drop into the prior cadence if/when
necessary.
**Template Rendering System**
- New `pipeline generate template` command renders Go templates with
pipeline data access
- Supports stdin/stdout or file-based input/output
- Templates access version data via function calls rather than
pre-populated context
**Version Cadence Support**
- Added `VersionCadence` type with `minor` and `major` release cadence
tracking
- Supports cadence transitions (e.g., minor→major) with
`TransitionVersion` and `PriorCadence` fields
- Calculates version ranges respecting different release cadences
**Template Functions**
- `VersionsNMinus` / `VersionsBounded` - List versions with explicit
cadence parameter
- `VersionsNMinusTransition` / `VersionsBoundedTransition` - Handle
cadence transitions
- `ParseVersion`, `CompareVersions`, `FilterVersions` - Version
utilities
- All functions require cadence to be explicitly specified
**CLI Integration**
- `--version` and `--edition` flags expose current version/edition to
templates
- Templates reference these via `.Version` and `.Edition` context fields
**Enos Migration**
- Converted `enos-dynamic-config.hcl` to template-based generation
- Uses `VersionsNMinusTransition` to handle Vault's minor→major cadence
shift at 1.21.5
Signed-off-by: Ryan Cragun <me@ryan.ec>
Co-authored-by: Ryan Cragun <me@ryan.ec>
Upgrade `cloudflare/circl` to v1.6.3 to resolve CVE-2026-1229. We had
several transient dependencies that depend on various versions of
`circl` that also needed to be updated in order to resolve the latest
version everywhere.
- github.com/ProtonMail/go-crypto v1.2.0 => v1.3.0
- github.com/google/go-github v17 => v83/v83.0.0
- github.com/google/go-github/v81 => v83/v83.0.0
Signed-off-by: Ryan Cragun <me@ryan.ec>
Co-authored-by: Ryan Cragun <me@ryan.ec>
The `pipeline` utility started as collection of small CLI utilities that we found useful for the Vault CI/CD pipeline. Rather than engineering complex bash scripts in YAML blocks, instead, we could build small, reusable, testable actions and integrate the into a single binary. No more copying and pasting loads of bash from YAML, instead we can copy a single command and run the same thing locally that we can in CI.
As we've continued to invest in the utilities capability, it's become clear that other CI pipelines would benefit from the same functionality that we've been building. This change represents the first significant work to make the utility truly generic in a HashiCorp repo that utilizes CRT sense. Once all the Vault specifics have been extracted we hope to move the utility out of the repo and make it available everywhere.
The primary change here is to move our changed file grouping configuration out of the `changed` package entirely. Instead of checkers that are written as Go code, we have created a new configuration file for the `pipeline` utility called `pipeline.hcl` While there are certainly other things that will eventually be configurable here, the only thing we've added support for is `changed_files`, which allows configuring how to match a given changed files path to a group name.
The DSL is fairly simple:
```hcl
changed_files {
// One or more groups can be defined
group "group_name_label" {
// Zero or more ignore blocks can be defined
ignore {
base_dir = []
base_name = []
base_name_prefix = []
contains = []
extension = []
file = []
}
// One or more match blocks can be defined
match {
base_dir = []
base_name = []
base_name_prefix = []
contains = []
extension = []
file = []
}
}
}
```
For example,
```hcl
// Create a changed_files block where we can define our changed files groups
changed_files {
// Group blocks take one label which is the name of the group
group "app" {
// Groups can ignore based on some criteria.
ignore {
// In this instance, we'll ignore any file that begins with
// tools/pipeline. All paths will be relative to the git repository
// root directory. The joinpath() function is here to support paths
// that are agnostic to the operating systems path separator. While
// it's unlikely that you'll need them, several cty stdlib functions
// are available.
base_dir = [joinpath("tools", "pipeline")]
}
// Groups must define at least one match block.
match {
// This will match any file with the .go extension (except for
// those that will be excluded with our ignore directive aboe
extension = [".go"]
}
// Groups can contain more than one match block. If any of the match
// blocks meet their criteria the group will be associated with the
// changed file
match {
base_name = ["go.mod", "go.sum"]
}
// If groups have more than one attribute set, each attribute group
// must match in order for the match.
match {
// Here we only match files that contain "raft_autopilot" in the
// path with the .go extension
extension = [".go"]
contains = ["raft_autopilot"]
}
}
group "autopilot" {
// Ignore blocks have the same attributes as match blocks
match {
// The base directory.
base_dir = [
"changelog",
joinpath("tools", "codechecker"),
]
// The base of the file
base_name = ["README.md"]
// A prefix string match on a files name.
base_name_prefix = ["buf."]
// Any string match in the files full path
contains = [
"-ce",
"_ce",
"-oss",
"_oss",
]
// The file's extension
extension = [
".hcl",
".md",
".sh",
".yaml",
".yml",
]
// An exact file match
file = [
# These exist on CE branches to please Github Actions.
joinpath(".github", "workflows", "build-artifacts-ent.yml"),
joinpath(".github", "workflows", "backport-automation-ent.yml"),
]
}
}
}
```
The default location of the config is `.release/pipeline.hcl`. All of our prior checks have been migrated to the DSL file present in this change.
- We had several commands that used the changed files groups that were built into the library. This change requires us to instead load the configuration from the file and use the user defined groupings.
- Several commands now take some part of that configuration in the request type. When possible we use the version parsed by the root command and verify in the request body rather than attempt to load the configuration.
- We also refactor the loading and parsing of `.release/versions.hcl` in the same manner. Now we automatically parse the file in the default locations relative to the git repo root.
- Our root command now has two new flags `--pipeline-config` and `--versions-config` which allow specifying a default location for each file. Commands which previously accepted flags or args to configure the versions file have been updated to use the global root flags instead. We've also removed the previous implementation that would recursively search backwards from the working directory to find the `versions.hcl` file. Instead we only support loading the file from the default location relative to the Git repo root.
- All instances of changed `pipeline` command invocations have been update to support the new auto-loading of configuration.
- A new configuration sub-command with validation exists to quickly validate a configuration file. `pipeline config validate`
Signed-off-by: Ryan Cragun <me@ryan.ec>
Co-authored-by: Ryan Cragun <me@ryan.ec>
We've already deployed some changed file detection in the CI pipeline. It uses the Github API to fetch a list of all changed files on a PR and then run it through a simple groups categorization pass. It's been a useful strategy in the context of a Pull Request because it does not depend on the local state of the Git repo.
This commit introduces a local git-based file change detection and validation system for the pipeline tool, enabling developers to identify and validate changed files before pushing code. We intend to use the new tool in two primary ways:
- As a Git pre-push hook when pushing new or updated branches. (Implemented here)
- As part of the scheduled automated repository synchronization. (Up next, and it will use the same `git.CheckChangedFilesReq{}` implementation.
This will allow us to guard all pushes to `hashicorp/vault` and `ce/*` branches in `hashicorp/vault-enterprise`, whether run locally on a developer machine or in CI by our service user.
We introduce two new `pipeline` CLI commands:
- `pipeline git list changed-files`
- `pipeline git check changed-files`
Both support specifying what method of git inspection we want to use for the changed files list:
- **`--branch <branch>`**: Lists all files added in the entire history of a specific branch. We use this when pushing a _new_ branch.
- **`--range <range>`**: Lists all changed files within a commit range (e.g., `HEAD~5..HEAD`). We use this when updating an existing branch.
- **`--commit <sha>`**: Lists all changed files in a specific commit (using `git show`). This isn't actually used at all in the pre-push hook but it useful if you wish to inspect a single commit on your branch.
The behavior when passing the `range` and `commit` is similar. We inspect the changed file list either for one or many commits (but with slightly different implementations for efficiency and accuracy. The `branch` option is a bit different. We use it to inspect the branches entire history of changed files for enterprise files before pushing a new branch. We do this to ensure that our branch doesn't accidentally add and then subsequently remove enterprise files, leaving the contents in the history but nothing obvious in the diff.
Each command supports several different output formats. The default is the human readable text table, though `--format json` will write all of the details as valid JSON to STDOUT. When given the `--github-output` command each will write a more concise version of the JSON output to `$GITHUB_OUTPUT`. It differs from our standard JSON output as it has been formatted to be easier to use in Github Actions contexts without requiring complex filtering.
When run, changed files are automatically categorized into logical groups based on their file name, just like our existing changed file detection. A follow-up to this PR will introduce a configuration based system for classifying file groups. This will allow us to create generic support for changed file detection so that many repositories can adopt this pattern.
The major difference in behavior between the two new commands is that the `list` command will always list the changed files for the given method/target, while the `check` command requires one-or-more changed file groups that we want to disallow to be included via the `-g` flag. If any changed files match the given group(s) then the command will fail. That allows us to specify the `enterprise` group and disallow the command to succeed if any of the changed files match the group.
The pre-push git hook now uses this system to prevent accidental pushes, however, it requires the local machine to have the `pipeline` tool in the `$PATH`. This ought not be much of a requirement as a working Go toolchain is required for any Vault developer. When it is not present we explain in our error messages how to resolve the problem and direct them to our slack channel if they need further assistance.
Signed-off-by: Ryan Cragun <me@ryan.ec>
Co-authored-by: Ryan Cragun <me@ryan.ec>
* [VAULT-41857] pipeline(find-artifact): add support for finding artifacts from branches (#11799)
Add support for finding matching workflow artifacts from branches rather than PRs. This allows us to trigger custom HCP image builds from a branch rather than an PR. It also enables us to build and test the HCP image on a scheduled nightly cadence, which we've also enabled.
As part of these changes I also added support for specifying which environment you want to test and threaded it through the cloud scenario now that there are multiple variants. We also make the testing workflow workflow_dispatch-able so that we can trigger HVD testing for any custom image in any environment without building a new image.
Signed-off-by: Ryan Cragun <me@ryan.ec>
Co-authored-by: Ryan Cragun <me@ryan.ec>
Our test fixtures include some directives that are not always required.
Depending on the Go version you are using the results of tidying will
also be different. We include them in our test fixtures to verify parsing
parsing of the module files so we want to always keep them there.
Unfortunately, some tooling and editors will automatically tidy the files
if they are discovered. To prevent that we'll rename them to include a
different suffix.
Signed-off-by: Ryan Cragun <me@ryan.ec>
Co-authored-by: Ryan Cragun <me@ryan.ec>
Update the base images for all scenarios:
- RHEL: upgrade base image for 10 to 10.1
- RHEL: upgrade base image for 9 to 9.7
- SLES: upgrade base image for 15 to 15.7
- SLES: add SLES 16.0 to the matrix
- OpenSUSE: remove OpenSUSE Leap from the matrix
I ended up removing OpenSUSE because the images that we were on were rarely updated and that resulted in very slow scenarios because of package upgrades. Also, despite the latest release being in October I didn't find any public cloud images produced for the new version of Leap. We can consider adding it back later but I'm comfortable just leaving SLES 15 and 16 in there for that test coverage.
I also ended up fixing a bug in our integration host setup where we'd provision three nodes instead of one. That ought to result in many fewer instance provisions per scenario. I also had to make a few small tweaks in how we detected whether or not SELinux is enabled, as the prior implementation did not work for SLES 16.
Signed-off-by: Ryan Cragun <me@ryan.ec>
Co-authored-by: Ryan Cragun <me@ryan.ec>
A few smaller changes to `pipeline`:
- Change the regions that we use back to us-east-1 and us-west-2
- Don't backport anything to inactive branches. This behavior was a
relic of prior behavior and is no longer necessary.
- Fix the go mod tests that rely on a strangely formatted mod file
- Ignore the module fixtures when running `make go-mod-tidy`
- Run `make go-mod-tidy`
Signed-off-by: Ryan Cragun <me@ryan.ec>
* [VAULT-40165] pipeline(github): add `check go-mod-diff` command
Add `pipeline github check go-mod-diff` command that is capable of
creating a Go module diff between one-or-more go.mod files in two
different Github branches. There are flags for the owner, repo, and
branch for both the A and B sides of the diff, as well as the `--path`
or `-p` flag that can be specified any number of times with relative
paths in the repository of go.mod files to compare. We assume that the
path is the same in both repositories.
This work will be followed up with another PR that removes the
enterprise only go.mod file and enables Go module diff checking on pull
requests to CE branches that change the go toolchain.
Signed-off-by: Ryan Cragun <me@ryan.ec>
Co-authored-by: Ryan Cragun <me@ryan.ec>
* [VAULT-40043]: pipeline: add `go diff mod` command
Add a `pipeline go diff mod` command that is capable of comparing two
go.mod files at a directive level. We also support strict or lax
comparisons of several directives to flexible diff comparisons. This is
especially useful when you want to compare two go.mod files that have
some different dependencies (CE vs. Ent) but still want to compare
versions of like dependencies.
This command is not currently used in the pipeline but was useful in
developing the diff library that is used. Subsequent work will use the
library and be integrated into CI.
* review feedback
* one more comment fix
---------
Signed-off-by: Ryan Cragun <me@ryan.ec>
Co-authored-by: Ryan Cragun <me@ryan.ec>
* license: update headers to IBM Corp.
* `make proto`
* update offset because source file changed
Signed-off-by: Ryan Cragun <me@ryan.ec>
Co-authored-by: Ryan Cragun <me@ryan.ec>
* pipeline(changed-files): fix false positives for some files (#10239)
Signed-off-by: Ryan Cragun <me@ryan.ec>
* make fmt
Signed-off-by: Ryan Cragun <me@ryan.ec>
---------
Signed-off-by: Ryan Cragun <me@ryan.ec>
Co-authored-by: Ryan Cragun <me@ryan.ec>
* [VAULT-39424] pipeline(close-origin-pr): add support for closing the origin of copied PRs
When we copy a community contributed Pull Request to Enterprise the
source PR is effectively orphaned, leaving the original PR still
opened, the author unsure of what state the copied PR is in, and any
issues associated with it open.
When the copied PR is closed we ought to close the origin PR if it's
still open, and any other issues that might be associated with either
the origin PR or the copied PR.
We can also add comments to both PRs that include links to each other
and the squash commit to make discovery of the work visible to those
with access to both repos. Unfortunately there is no way to know what
the SHA will be when it's synced so we have to rely on the
'Co-Authored-By:' trailers in commit message.
There are some challenges to this:
- The automation should only execute when copied PRs are closed
- How to determine the origin PR from only the copied PR
- How to determine the PR's linked issues (which the v3 REST API does not expose)
We solved them by:
- Requiring the PR HEAD ref to start with `copy/`
- Encoding the origin PR information in the PR HEAD ref.
e.g. `copy/hashicorp/vault/31580/ryan/VAULT-39424-test-ce`
- Using the V4 GraphQL API to determine "closed issue references"
The result is a new `pipeline` CLI command that can close the origin PR,
all of the issues, and write status comments on each PR with links to
everything to establish omnidirectional linking in the Github UI.
```bash
pipeline github close origin-pull-request 9903
```
* fix feedback
---------
Signed-off-by: Ryan Cragun <me@ryan.ec>
Co-authored-by: Ryan Cragun <me@ryan.ec>
Occasionally it seems that the tools will get built and linked against
the platform glibc. We definitely do not want that.
Now we always disable CGO when building tools. While doing this I
realized that we could also strip debug symbols and reduce the size of
the tools significantly, so that is included as well.
Signed-off-by: Ryan Cragun <me@ryan.ec>
Co-authored-by: Ryan Cragun <me@ryan.ec>
When we copy a Pull Request from CE to Ent we already add a status
comment to the origin PR but we don't actually bubble up the information
to the workflow summary. Instead, render the copy PR output as a
markdown table and write it to the step summary.
Signed-off-by: Ryan Cragun <me@ryan.ec>
Co-authored-by: Ryan Cragun <me@ryan.ec>
* pipeline(copy-pr): cherry-pick commits instead of merging
* fix staticcheck for docs in pkg/github
Signed-off-by: Ryan Cragun <me@ryan.ec>
Co-authored-by: Ryan Cragun <me@ryan.ec>
Add a new `github` changed file group that includes everything in the
`.github` directory. Further refine the `pipeline` group to only
include scripts, workflows, and actions files in `.github`. We also move
the `CODEOWNERS` file into `.github/` to simplify `github` grouping.
As `build` logic responds to changes to the `pipeline` group this will
result in no longer building and testing everything for simple
changes in `github` that don't affect the pipeline.
Signed-off-by: Ryan Cragun <me@ryan.ec>
Co-authored-by: Ryan Cragun <me@ryan.ec>
When determining whether to skip a backport ref we currenly we have to
consider many factors:
- Whether or not there are changed files?
- If there are changed files, are some enterprise or CE?
- Are there some changed files that ought to be backported to inactive
branches?
- Is the target branch active or not?
We had a large test suite that covered _most_ of these cases but because
the changed file set determines a lot of behavior we were missing cases
where we ought to backport normal mixed changed file sets for no other
reason other than the branch is active. After fixing and normalizing the
tests I fixed the source bug which is that we didn't strip the branch
prefix from the ref version when checking branch activeness.
Signed-off-by: Ryan Cragun <me@ryan.ec>
Co-authored-by: Ryan Cragun <me@ryan.ec>
[VAULT-39160] actions(hcp): add support for testing custom images on HCP (#9345)
Add support for running the `cloud` scenario with a custom image in the
int HCP environment. We support two new tags that trigger new
functionality. If the `hcp/build-image` tag is present on a PR at the
time of `build`, we'll automatically trigger a custom build for the int
environment. If the `hcp/test` tag is present, we'll trigger a custom
build and run the `cloud` scenario with the resulting image.
* Fix a bug in our custom build pattern to handle prerelease versions.
* pipeline(hcp): add `--github-output` support to `show image` and
`wait image` commands.
* enos(hcp/create_vault_cluster): use a unique identifier for HVN
and vault clusters.
* actions(enos-cloud): add workflow to execute the `cloud` enos
scenario.
* actions(build): add support for triggering a custom build and running
the `enos-cloud` scenario.
* add more debug logging and query without a status
* add shim build-hcp-image for CE workflows
Signed-off-by: Ryan Cragun <me@ryan.ec>
Co-authored-by: Ryan Cragun <me@ryan.ec>
* license: add support for publishing artifacts to IBM PAO (#8366)
Signed-off-by: Ryan Cragun <me@ryan.ec>
Co-authored-by: brian shore <bshore@hashicorp.com>
Co-authored-by: Ethel Evans <ethel.evans@hashicorp.com>
Co-authored-by: Ryan Cragun <me@ryan.ec>
* [VAULT-39158] actions(build-hcp-image): various small fixes
Various small fixes to correctly trigger custom image builds and wait
for them to be available.
Signed-off-by: Ryan Cragun <me@ryan.ec>
Co-authored-by: Ryan Cragun <me@ryan.ec>
* [VAULT-39159]: pipeline: add support for querying HCP image service
In order to facilitate testing Vault Enterprise directly in HCP we need
tools to both request an image be built from a candidate build and to
also wait for the image to be available in order to execute test
scenarios with it. This PR adds a few new `pipeline` sub-commands that
can will be used for this purpose.
`pipeline github find workflow-artifact` can be used to find the path of
an artifact that matches the given filter criteria. You'll need to
provide a pull request number, workflow name, and either an exact
artifact name or a pattern. When providing a pattern only the first
match will be returned so make sure your regular expression is robust.
`pipeline hcp get image` will return the image information for an HCP
image. You will need to supply auth via the `HCP_USERNAME` and
`HCP_PASSWORD` environment variables in order to query the image
service. It also takes an enviroment flag so you can query the image
service in different environments.
`pipeline hcp wait image` is like `pipeline hcp get image` except that
it will continue to retry for a given timeout and with a given delay
between requests. In this way it can be used to wait for an image to be
available.
As part of this we also update our Go modules to the latest versions
that are compatible.
* [VAULT-39158]: actions(build-hcp-image): add workflow for building HCP images
* copywrite: add missing headers
* remove unused output
* address feedback
* allow prerelease artifacts
---------
Signed-off-by: Ryan Cragun <me@ryan.ec>
Co-authored-by: Ryan Cragun <me@ryan.ec>
As part of this we also update the pin of gotestsum to 1.12.3 to allow
for building it with Go 1.25.
Signed-off-by: Ryan Cragun <me@ryan.ec>
Co-authored-by: Ryan Cragun <me@ryan.ec>
* always fetch HEAD repo with it's clone URL
* use the number instead of ID when generating the summary table
Signed-off-by: Ryan Cragun <me@ryan.ec>
Co-authored-by: Ryan Cragun <me@ryan.ec>
Don't categorize changelog files that begin with an underscore as
enterprise only, otherwise they'll be removed when backporting changes
to CE.
Since we want to include links to commit SHAs in the changelog we have
to create the changelog in the context of CE and thus need to backport
all of those changes.
We also fix a few Go tests that hand not been updated to handle the
updated default inactive CE groups.
Signed-off-by: Ryan Cragun <me@ryan.ec>
Co-authored-by: Ryan Cragun <me@ryan.ec>
Docs have been moved since the tool was written so that exclusion is no
longer needed. Since the defaults were added the `pipeline` group has
expanded to include all `.github`, which we don't want to always
backport. It seems unlike that `pipeline` tooling changes are likely to
be required often on inactive branches so we'll exclude all together for
now.
Signed-off-by: Ryan Cragun <me@ryan.ec>
Co-authored-by: Ryan Cragun <me@ryan.ec>
Ubunut 20.04 is EOL. Per our support and package policies we no longer
need to develop or test for that platform.
Signed-off-by: Ryan Cragun <me@ryan.ec>
