mirror of
https://github.com/hashicorp/vault.git
synced 2025-08-19 13:41:10 +02:00
Context ------- Building and testing Vault artifacts on pull requests and merges is responsible for about 1/3rd of our overall spend on Vault CI. Of the artifacts that we ship as part of a release, we do Enos testing scenarios on the `linux/amd64` and `linux/arm64` binaries and their derivative artifacts. The extended build artifacts for non-Linux platforms or less common machine architectures are not tested at this time. They are built, notarized, and signed as part of every pull request update and merge. As we don't actually test these artifacts, the only gain we get from this rather expensive behavior is that we wont merge a change that would prevent Vault from building on one of the extended targets. Extended platform or architecture changes are quite rare, so performing this work as frequently as we do is costly in both monetary and developer time for little relative safety benefit. Goals ----- Rethink and implement how and when we build binaries and artifacts of Vault so that we can spend less money on repetitive work and while also reducing the time it takes for the build and test pipelines to complete. Solution -------- Instead of building all release artifacts on every push, we'll opt to build only our testable (core) artifacts. With this change we are introducing a bit of risk. We could merge a change that breaks an extended platform and only find out after the fact when we trigger a complete build for a release. We'll hedge against that risk by building all of the release targets on a scheduled cadence to ensure that they are still buildable. We'll make building all of the targets optional on any pull request by use of a `build/all` label on the pull request. Further considerations ---------------------- * We want to reduce the total number of workflows and runners for all of our pipelines if possible. As each workflow runner has infrastructure cost and runner time penalties, using a single runner over many is often preferred. * Many of our jobs runners have been optimized for cost and performance. We should simplify the choices of which runners to use. * CRT requires us to use the same build workflow in both CE and Ent. Historically that meant that modifying `build.yml` in CE would result in a merge conflict with `build.yml` in Ent, and break our merge workflows. * Workflow flow control in both `build.yml` and `ci.yml` can be quite complicated, as each needs to maintain compatibility whether executed as CE or Ent, and when triggered with various Github events like pull_request, push, and workflow_call, each with their own requirements. * Many jobs utilize similar patterns of flow control and metadata but are not reusable. * Workflow call depth has a maximum of four, so we need to be quite considerate when calling other workflows. * Called workflows can only have 10 inputs. Implementation -------------- * Refactor the `build.yml` workflow to be agnostic to whether or not it is executing in CE or Ent. That makes future updates to the build much easier as we won't have to worry about merge conflicts when the change is merged downstream. * Extract common steps in workflows into composite actions that we can reuse. * Fix bugs where some but not all workflows would use different Git references when building and testing a pull request. * We rewrite the application, docs, and UI change helpers as a composite action. This allows us to re-use this logic to make consistent behavior choices across build and CI. * We combine several `build.yml` and `ci.yml` jobs into our final job. This reduces the number of workflows required for the same behavior while saving time overall. * Update most of our action pins. Results ------- | Metric | Before | After | Diff | |-------------------|----------|---------|-------| | Duration: | ~14-18m | ~15-18m | ~ = | | Workflows: | 43 | 18 | - 58% | | Billable time: | ~1h15m | 16m | - 79% | | Saved artifacts: | 34 | 12 | - 65% | Infra costs should map closely to billable time. Network I/O costs should map closely to the workflow count. Storage costs should map directly with saved artifacts. We could probably get parity with duration by getting more clever with our UBI container build, as that's where we're seeing the increase. I'm not yet concerned as it takes roughly the same time for this job to complete as it did before. While the CI workflow was not the focus on the PR, some shared refactoring does show some marginal improvements there. | Metric | Before | After | Diff | |-------------------|----------|----------|--------| | Duration: | ~24m | ~12.75m | - 15% | | Workflows: | 55 | 47 | - 8% | | Billable time: | ~4h20m | ~3h36m | - 7% | Further focus on streamlining the CI workflows would likely result in a few more marginal improvements, but nothing on the order like we've seen with the build workflow. Signed-off-by: Ryan Cragun <me@ryan.ec>
181 lines
4.1 KiB
Bash
Executable File
181 lines
4.1 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# Copyright (c) HashiCorp, Inc.
|
|
# SPDX-License-Identifier: BUSL-1.1
|
|
|
|
|
|
# The ci-helper is used to determine build metadata, build Vault binaries,
|
|
# package those binaries into artifacts, and execute tests with those artifacts.
|
|
|
|
set -euo pipefail
|
|
|
|
# We don't want to get stuck in some kind of interactive pager
|
|
export GIT_PAGER=cat
|
|
|
|
# Get the build date from the latest commit since it can be used across all
|
|
# builds
|
|
function build_date() {
|
|
# It's tricky to do an RFC3339 format in a cross platform way, so we hardcode UTC
|
|
: "${DATE_FORMAT:="%Y-%m-%dT%H:%M:%SZ"}"
|
|
git show --no-show-signature -s --format=%cd --date=format:"$DATE_FORMAT" HEAD
|
|
}
|
|
|
|
# Get the revision, which is the latest commit SHA
|
|
function build_revision() {
|
|
git rev-parse HEAD
|
|
}
|
|
|
|
# Determine our repository by looking at our origin URL
|
|
function repo() {
|
|
basename -s .git "$(git config --get remote.origin.url)"
|
|
}
|
|
|
|
# Determine the artifact basename based on metadata
|
|
function artifact_basename() {
|
|
: "${PKG_NAME:="vault"}"
|
|
: "${GOOS:=$(go env GOOS)}"
|
|
: "${GOARCH:=$(go env GOARCH)}"
|
|
: "${VERSION_METADATA:="ce"}"
|
|
|
|
: "${VERSION:=""}"
|
|
if [ -z "$VERSION" ]; then
|
|
echo "You must specify the VERSION variable for this command" >&2
|
|
exit 1
|
|
fi
|
|
|
|
local version
|
|
version="$VERSION"
|
|
if [ "$VERSION_METADATA" != "ce" ]; then
|
|
version="${VERSION}+${VERSION_METADATA}"
|
|
fi
|
|
|
|
echo "${PKG_NAME}_${version}_${GOOS}_${GOARCH}"
|
|
}
|
|
|
|
# Bundle the dist directory into a zip
|
|
function bundle() {
|
|
: "${BUNDLE_PATH:=$(repo_root)/vault.zip}"
|
|
echo "--> Bundling dist/* to $BUNDLE_PATH..."
|
|
zip -r -j "$BUNDLE_PATH" dist/
|
|
}
|
|
|
|
# Determine the root directory of the repository
|
|
function repo_root() {
|
|
git rev-parse --show-toplevel
|
|
}
|
|
|
|
# Build the UI
|
|
function build_ui() {
|
|
local repo_root
|
|
repo_root=$(repo_root)
|
|
|
|
pushd "$repo_root"
|
|
mkdir -p http/web_ui
|
|
popd
|
|
pushd "$repo_root/ui"
|
|
yarn install
|
|
npm rebuild node-sass
|
|
yarn run build
|
|
popd
|
|
}
|
|
|
|
# Build Vault
|
|
function build() {
|
|
local revision
|
|
local build_date
|
|
local ldflags
|
|
local msg
|
|
|
|
# Get or set our basic build metadata
|
|
revision=$(build_revision)
|
|
build_date=$(build_date) #
|
|
: "${BIN_PATH:="dist/"}" #if not run by actions-go-build (enos local) then set this explicitly
|
|
: "${GO_TAGS:=""}"
|
|
: "${REMOVE_SYMBOLS:=""}"
|
|
|
|
(unset GOOS; unset GOARCH; go generate ./...)
|
|
|
|
# Build our ldflags
|
|
msg="--> Building Vault revision $revision, built $build_date..."
|
|
|
|
# Keep the symbol and dwarf information by default
|
|
if [ -n "$REMOVE_SYMBOLS" ]; then
|
|
ldflags="-s -w "
|
|
else
|
|
ldflags=""
|
|
fi
|
|
|
|
ldflags="${ldflags} -X github.com/hashicorp/vault/version.GitCommit=$revision -X github.com/hashicorp/vault/version.BuildDate=$build_date"
|
|
|
|
if [[ ${VERSION_METADATA+x} ]]; then
|
|
msg="${msg}, metadata ${VERSION_METADATA}"
|
|
ldflags="${ldflags} -X github.com/hashicorp/vault/version.VersionMetadata=$VERSION_METADATA"
|
|
fi
|
|
|
|
# Build vault
|
|
echo "$msg"
|
|
pushd "$(repo_root)"
|
|
mkdir -p dist
|
|
mkdir -p out
|
|
set -x
|
|
go env
|
|
go build -v -tags "$GO_TAGS" -ldflags "$ldflags" -o dist/
|
|
set +x
|
|
popd
|
|
}
|
|
|
|
# Prepare legal requirements for packaging
|
|
function prepare_legal() {
|
|
: "${PKG_NAME:="vault"}"
|
|
|
|
pushd "$(repo_root)"
|
|
mkdir -p dist
|
|
curl -o dist/EULA.txt https://eula.hashicorp.com/EULA.txt
|
|
curl -o dist/TermsOfEvaluation.txt https://eula.hashicorp.com/TermsOfEvaluation.txt
|
|
mkdir -p ".release/linux/package/usr/share/doc/$PKG_NAME"
|
|
cp dist/EULA.txt ".release/linux/package/usr/share/doc/$PKG_NAME/EULA.txt"
|
|
cp dist/TermsOfEvaluation.txt ".release/linux/package/usr/share/doc/$PKG_NAME/TermsOfEvaluation.txt"
|
|
popd
|
|
}
|
|
|
|
# Package version converts a vault version string into a compatible representation for system
|
|
# packages.
|
|
function version_package() {
|
|
awk '{ gsub("-","~",$1); print $1 }' <<< "$VAULT_VERSION"
|
|
}
|
|
|
|
# Run the CI Helper
|
|
function main() {
|
|
case $1 in
|
|
artifact-basename)
|
|
artifact_basename
|
|
;;
|
|
build)
|
|
build
|
|
;;
|
|
build-ui)
|
|
build_ui
|
|
;;
|
|
bundle)
|
|
bundle
|
|
;;
|
|
date)
|
|
build_date
|
|
;;
|
|
prepare-legal)
|
|
prepare_legal
|
|
;;
|
|
revision)
|
|
build_revision
|
|
;;
|
|
version-package)
|
|
version_package
|
|
;;
|
|
*)
|
|
echo "unknown sub-command" >&2
|
|
exit 1
|
|
;;
|
|
esac
|
|
}
|
|
|
|
main "$@"
|