This change extends the garbage collector for the build cache server to
remove cached release artifacts. Release artifacts are copied to the
official mirrors and do not need to remain on the build cache after a
release was published.
By default, the 10 latest releases of all channels (including LTS and
previous LTS) are kept.
Also excluded from garbage collection are:
- Emerging new major releases (i.e. major number larger than the
latest Alpha release)
- channel progressions (major number exists in the lists of releases to
keep but minor is bigger than any release)
- patch releases (major and minor exist in list of releases to keep but
patch level is newer than in any release)
- SDKs (tarballs and containers) of any release in the list of releases
to keep; i.e. the SDK in <MAJOR>.0.0 for any release to keep.
Signed-off-by: Thilo Fromm <thilofromm@microsoft.com>
This change adds a min_age parameter to the github CI SDK garbage
collector. The parameter specifies a minimum age (in days) for artifacts
to be garbage collected. NOTE that this can result in more artifacts
being kept than specified via the "keep" parameter if artifacts are
younger than min_age.
The change also has garbage_collect.sh pass the min_age parameter to
garbage_collect_github_ci_sdk.sh.
Signed-off-by: Thilo Fromm <thilofromm@microsoft.com>
This change improves the build cache garbage collector to remove
orphaned artifact directories - i.e. directories to which no version tag
exists in the scripts repo.
SDK containers built by Github actions (using update_sdk_container) are
igored by this change because these are handled in a separate garbage
collection script.
Also, a new command line parameter has been added to remove artifacts
older than the specified number of days (defaulting to 14):
- If neither number of builds nor max age is specified, the script
defaults to 50 builds to keep, and a max age of 14 days.
The max age overrides the number of builds to keep, so more than
50 builds may be kept.
- If only the number of builds to keep is specified, the max age is
set to "0" (i.e. today).
- If both are specified, max age again overrides number of builds to
keep.
Signed-off-by: Thilo Fromm <thilofromm@microsoft.com>
The functions are sourcing other files that define global variables,
so they will spill into the callers shell unnecessarily. We will also
add some functionality that uses traps in follow-up commits, so it's
good to limit the scope of traps too.
The garbage collect job should also clean up kola resources if a test
job failed to do so due to forced terminator or misbehavior. The
cleanup is done by "ore" which needs credentials like kola.
Run ore from the mantle container image. Unfortunately Docker does not
support Podman's --env-host option and the env vars had to be passed
explicitly. While --env-file=<(env) would work it contains a lot of
variables that cause the container to behave a bit weird.
The pipeline created two tags if an SDK was built, one for the SDK and
one for the OS build (which was a free-standing tag or a local state
that was equivalent to the existing tag of the same name). The
nightlies created update commits on the main branch, even if no change
was done, and on the release branches we lacked these commits.
Create the release tag in the nightly SDK bootstrap already and reuse
it for the nightly OS build. Instead of local state, checkout the
existing tags explicitly. Extend the nightly update commit logic to
cover release branches and detect if we can skip building because no
changes were done.
This change adds copying test results to the build cache server, and
adds respective deletion to the garbage collector.
Also, the patch fixes an issue with torcx publishing (manifest
publishing had arch hard-coded).
Signed-off-by: Thilo Fromm <thilo@kinvolk.io>
The original intention of the "binpkg" prefix in the CI binary package
cache URL was to separate packages from other build artifacts like
containers, images, and SDK tarballs. Motivation was to separate
developer content (binary packages) from CI automation artifacts
(everything else); since binary packages are not used by the CI.
This broke assumptions in scripts which use the binary host URL for
other things than packages - e.g. SDK tarballs or images. These
scripts would get a bincache URL with "binpkg/" prepended, while CI
automation would *not* use that prefix.
This change removes the use of "binpkg/" altogether since it would not
work as intended without more significant changes to build scripts.
garbage_collect.sh was using 'docker_vernum' where it should have been
using 'vernum' (as push_pkgs.sh does).
Also, make sure release directories are removed, not just packages.
Signed-off-by: Thilo Fromm <thilo@kinvolk.io>
This change adds a job for publishing binary packages to the build cache
server to the ci automation.
Also, setup_board is updated to use the buildcache package cache if a
nightly build version is detected.
Signed-off-by: flatcar-ci <infra+ci@flatcar-linux.org>
ci-automation builds on the SDK container and simplifies CI automation
build tasks (SDK bootstrap, SDK container, packages, image, VMs).
See ci-automation/README.md for a brief introduction.
Signed-off-by: Thilo Fromm <thilo@kinvolk.io>
