This change extends the garbage collector for the build cache server to
remove cached release artifacts. Release artifacts are copied to the
official mirrors and do not need to remain on the build cache after a
release was published.
By default, the 10 latest releases of all channels (including LTS and
previous LTS) are kept.
Also excluded from garbage collection are:
- Emerging new major releases (i.e. major number larger than the
latest Alpha release)
- channel progressions (major number exists in the lists of releases to
keep but minor is bigger than any release)
- patch releases (major and minor exist in list of releases to keep but
patch level is newer than in any release)
- SDKs (tarballs and containers) of any release in the list of releases
to keep; i.e. the SDK in <MAJOR>.0.0 for any release to keep.
Signed-off-by: Thilo Fromm <thilofromm@microsoft.com>
gensub is a GNU extension; however, POSIXLY_CORRECT is enforced in
systemd-run which triggers gawk's traditional / posix mode.
Unset it before calling gawk to make gensub available.
Signed-off-by: Thilo Fromm <thilofromm@microsoft.com>
This change adds a min_age parameter to the github CI SDK garbage
collector. The parameter specifies a minimum age (in days) for artifacts
to be garbage collected. NOTE that this can result in more artifacts
being kept than specified via the "keep" parameter if artifacts are
younger than min_age.
The change also has garbage_collect.sh pass the min_age parameter to
garbage_collect_github_ci_sdk.sh.
Signed-off-by: Thilo Fromm <thilofromm@microsoft.com>
This change improves the build cache garbage collector to remove
orphaned artifact directories - i.e. directories to which no version tag
exists in the scripts repo.
SDK containers built by Github actions (using update_sdk_container) are
igored by this change because these are handled in a separate garbage
collection script.
Also, a new command line parameter has been added to remove artifacts
older than the specified number of days (defaulting to 14):
- If neither number of builds nor max age is specified, the script
defaults to 50 builds to keep, and a max age of 14 days.
The max age overrides the number of builds to keep, so more than
50 builds may be kept.
- If only the number of builds to keep is specified, the max age is
set to "0" (i.e. today).
- If both are specified, max age again overrides number of builds to
keep.
Signed-off-by: Thilo Fromm <thilofromm@microsoft.com>
This change introduces build_dev_binpkgs, a script to build binary
packages for all dependencies of the devcontainer. This works around an
issue with build_packages, which doesn't - leading to build issues with
the devcontainer later on. This particularly happens for more complex
builds with the devcontainer.
Additionally, a call to build_dev_binpkgs has been added to the package
publishing step in ci-automation before binary packages are published.
Signed-off-by: Thilo Fromm <thilofromm@microsoft.com>
The refactored image changes script will eventually be run for the old
LTS version, so make sure that the script for that channel will get a
last release of old LTS instead of new LTS.
Image changes job needs a list of OEMIDs that are built for a specific
architecture. Similar information already existed in the
coreos-base/common-oem-files ebuild, so factor it out to a separate
file, so the image changes job does not need to source the entire
ebuild (or process it in other way), but rather source the smaller
file.
Instead of depending on default value of build_image's base_sysext
parameter, create a file that explicitly lists which base sysexts will
be built for each architecture. The file can be sourced by other
scripts that need this kind of information. Currently, image.sh and
image_changes.sh use this file.
This is to limit the amount of reports consisting purely of failures,
because some files were missing. And those files will be missing,
because an OEM might not even have any image for certain arches (like
digitalocean has no arm64 images).
It certainly does happen on GitHub Actions that the HEAD commit has no
tag. In such case, fake the tag from version file. The git tag in this
scenario is used to figure out a channel transition.
This avoid assuming that the current working directory in the
generate_image_changes_report function is actually toplevel directory
of the scripts repo.
It was only needed for the show-changes script. Now that show-changes
script allows to set the repos parent directory with an environment
variable, we set the variable instead of changing the working
directory.
The special Brightbox image uses the OpenStack userdata in Ignition but
lacked Afterburn usage. It actually works to use the OpenStack image and
directly which also enables Afterburn, thus we can drop the special
image.
Don't build a special image for Brightbox but recommend to use OpenStack
images directly. A symlink is added to help with the download of
hardcoded user scripts.
For Brightbox we can use the OpenStack image but the import only works
with unpacked images. After we enabled internal qcow2 compression the
.gz or .bz2 external compression doesn't provide any benefits and makes
the import more complicated.
Provide the OpenStack image without external compression in addition.
The other files are kept for now but we could also delete them if we
announce this in advance.
- updated github actions for runc, containerd, and docker to not handle
nonexistent ebuilds in app-torcx/ anymore
- removed spurious package_run_dependencies from build_image_util.sh
- build_sysext: generate pkginfo before mangle script runs
use zstd for compression; add cli flag to select compression
- ci_automation_common.sh: remove spurious `/` from match string
- coreos, board-packages, bootengine: bump ebuild revisions
- kernel commonconfig: add squashfs zstd support
Signed-off-by: Thilo Fromm <thilofromm@microsoft.com>
This change makes QEMU_UPDATE_PAYLOAD configurable via
ci-automation/settings.env where it was hard-wired before.
The change also fixes fall-out in qemu_update.sh by ensuring a local tmp
directory is created before it is used by the test.
Signed-off-by: Thilo Fromm <thilofromm@microsoft.com>
Switch to using a managed identity instead of file based credentials for
running kola/ore (not plume). This covers our test subscription, but not our
publishing subscription.
Signed-off-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>
