vault/scripts/ci-helper.sh
Ryan Cragun 9a10689ca3
[QT-645] Restructure dev tools (#24559)
We're on a quest to reduce our pipeline execution time to both enhance
our developer productivity but also to reduce the overall cost of the CI
pipeline. The strategy we use here reduces workflow execution time and
network I/O cost by reducing our module cache size and using binary
external tools when possible. We no longer download modules and build
many of the external tools thousands of times a day.

Our previous process of installing internal and external developer tools
was scattered and inconsistent. Some tools were installed via `go
generate -tags tools ./tools/...`,
others via various `make` targets, and some only in Github Actions
workflows. This process led to some undesirable side effects:
  * The modules of some dev and test tools were included with those
    of the Vault project. This leads to us having to manage our own
    Go modules with those of external tools. Prior to Go 1.16 this
    was the recommended way to handle external tools, but now
    `go install tool@version` is the recommended way to handle
    external tools that need to be build from source as it supports
    specific versions but does not modify the go.mod.
  * Due to Github cache constraints we combine our build and test Go
    module caches together, but having our developer tools as deps in
    our module results in a larger cache which is downloaded on every
    build and test workflow runner. Removing the external tools that were
    included in our go.mod reduced the expanded module cache by size
    by ~300MB, thus saving time and network I/O costs when downloading
    the module cache.
  * Not all of our developer tools were included in our modules. Some were
    being installed with `go install` or `go run`, so they didn't take
    advantage of a single module cache. This resulted in us downloading
    Go modules on every CI and Build runner in order to build our
    external tools.
  * Building our developer tools from source in CI is slow. Where possible
    we can prefer to use pre-built binaries in CI workflows. No more
    module download or tool compiles if we can avoid them.

I've refactored how we define internal and external build tools
in our Makefile and added several new targets to handle both building
the developer tools locally for development and verifying that they are
available. This allows for an easy developer bootstrap while also
supporting installation of many of the external developer tools from
pre-build binaries in CI. This reduces our network IO and run time
across nearly all of our actions runners.

While working on this I caught and resolved a few unrelated issue:
* Both our Go and Proto format checks we're being run incorrectly. In
  CI they we're writing changes but not failing if changes were
  detected. The Go was less of a problem as we have git hooks that
  are intended to enforce formatting, however we drifted over time.
* Our Git hooks couldn't handle removing a Go file without failing. I
  moved the diff check into the new Go helper and updated it to handle
  removing files.
* I combined a few separate scripts and into helpers and added a few
  new capabilities.
* I refactored how we install Go modules to make it easier to download
  and tidy all of the projects go.mod's.
* Refactor our internal and external tool installation and verification
  into a tools.sh helper.
* Combined more complex Go verification into `scripts/go-helper.sh` and
  utilize it in the `Makefile` and git commit hooks.
* Add `Makefile` targets for executing our various tools.sh helpers.
* Update our existing `make` targets to use new tool targets.
* Normalize our various scripts and targets output to have a consistent
  output format.
* In CI, install many of our external dependencies as binaries wherever
  possible. When not possible we'll build them from scratch but not mess
  with the shared module cache.
* [QT-641] Remove our external build tools from our project Go modules.
* [QT-641] Remove extraneous `go list`'s from our `set-up-to` composite
  action.
* Fix formatting and regen our protos

Signed-off-by: Ryan Cragun <me@ryan.ec>
2024-01-09 17:50:46 +00:00

173 lines
4.0 KiB
Bash
Executable File

#!/usr/bin/env bash
# Copyright (c) HashiCorp, Inc.
# SPDX-License-Identifier: BUSL-1.1
# The ci-helper is used to determine build metadata, build Vault binaries,
# package those binaries into artifacts, and execute tests with those artifacts.
set -euo pipefail
# We don't want to get stuck in some kind of interactive pager
export GIT_PAGER=cat
# Get the build date from the latest commit since it can be used across all
# builds
function build_date() {
# It's tricky to do an RFC3339 format in a cross platform way, so we hardcode UTC
: "${DATE_FORMAT:="%Y-%m-%dT%H:%M:%SZ"}"
git show --no-show-signature -s --format=%cd --date=format:"$DATE_FORMAT" HEAD
}
# Get the revision, which is the latest commit SHA
function build_revision() {
git rev-parse HEAD
}
# Determine our repository by looking at our origin URL
function repo() {
basename -s .git "$(git config --get remote.origin.url)"
}
# Determine the artifact basename based on metadata
function artifact_basename() {
: "${PKG_NAME:="vault"}"
: "${GOOS:=$(go env GOOS)}"
: "${GOARCH:=$(go env GOARCH)}"
: "${VERSION:=""}"
if [ -z "$VERSION" ]; then
echo "You must specify the VERSION variable for this command" >&2
exit 1
fi
echo "${PKG_NAME}_${VERSION}_${GOOS}_${GOARCH}"
}
# Bundle the dist directory into a zip
function bundle() {
: "${BUNDLE_PATH:=$(repo_root)/vault.zip}"
echo "==> Bundling dist/* to $BUNDLE_PATH..."
zip -r -j "$BUNDLE_PATH" dist/
}
# Determine the root directory of the repository
function repo_root() {
git rev-parse --show-toplevel
}
# Build the UI
function build_ui() {
local repo_root
repo_root=$(repo_root)
pushd "$repo_root"
mkdir -p http/web_ui
popd
pushd "$repo_root/ui"
yarn install
npm rebuild node-sass
yarn run build
popd
}
# Build Vault
function build() {
local revision
local build_date
local ldflags
local msg
# Get or set our basic build metadata
revision=$(build_revision)
build_date=$(build_date) #
: "${BIN_PATH:="dist/"}" #if not run by actions-go-build (enos local) then set this explicitly
: "${GO_TAGS:=""}"
: "${REMOVE_SYMBOLS:=""}"
(unset GOOS; unset GOARCH; go generate ./...)
# Build our ldflags
msg="==> Building Vault revision $revision, built $build_date..."
# Keep the symbol and dwarf information by default
if [ -n "$REMOVE_SYMBOLS" ]; then
ldflags="-s -w "
else
ldflags=""
fi
ldflags="${ldflags} -X github.com/hashicorp/vault/version.GitCommit=$revision -X github.com/hashicorp/vault/version.BuildDate=$build_date"
if [[ ${VERSION_METADATA+x} ]]; then
msg="${msg}, metadata ${VERSION_METADATA}"
ldflags="${ldflags} -X github.com/hashicorp/vault/version.VersionMetadata=$VERSION_METADATA"
fi
# Build vault
echo "$msg"
pushd "$(repo_root)"
mkdir -p dist
mkdir -p out
set -x
go build -v -tags "$GO_TAGS" -ldflags "$ldflags" -o dist/
set +x
popd
}
# Prepare legal requirements for packaging
function prepare_legal() {
: "${PKG_NAME:="vault"}"
pushd "$(repo_root)"
mkdir -p dist
curl -o dist/EULA.txt https://eula.hashicorp.com/EULA.txt
curl -o dist/TermsOfEvaluation.txt https://eula.hashicorp.com/TermsOfEvaluation.txt
mkdir -p ".release/linux/package/usr/share/doc/$PKG_NAME"
cp dist/EULA.txt ".release/linux/package/usr/share/doc/$PKG_NAME/EULA.txt"
cp dist/TermsOfEvaluation.txt ".release/linux/package/usr/share/doc/$PKG_NAME/TermsOfEvaluation.txt"
popd
}
# Package version converts a vault version string into a compatible representation for system
# packages.
function version_package() {
awk '{ gsub("-","~",$1); print $1 }' <<< "$VAULT_VERSION"
}
# Run the CI Helper
function main() {
case $1 in
artifact-basename)
artifact_basename
;;
build)
build
;;
build-ui)
build_ui
;;
bundle)
bundle
;;
date)
build_date
;;
prepare-legal)
prepare_legal
;;
revision)
build_revision
;;
version-package)
version_package
;;
*)
echo "unknown sub-command" >&2
exit 1
;;
esac
}
main "$@"