- updated github actions for runc, containerd, and docker to not handle
nonexistent ebuilds in app-torcx/ anymore
- removed spurious package_run_dependencies from build_image_util.sh
- build_sysext: generate pkginfo before mangle script runs
use zstd for compression; add cli flag to select compression
- ci_automation_common.sh: remove spurious `/` from match string
- coreos, board-packages, bootengine: bump ebuild revisions
- kernel commonconfig: add squashfs zstd support
Signed-off-by: Thilo Fromm <thilofromm@microsoft.com>
This change makes QEMU_UPDATE_PAYLOAD configurable via
ci-automation/settings.env where it was hard-wired before.
The change also fixes fall-out in qemu_update.sh by ensuring a local tmp
directory is created before it is used by the test.
Signed-off-by: Thilo Fromm <thilofromm@microsoft.com>
Switch to using a managed identity instead of file based credentials for
running kola/ore (not plume). This covers our test subscription, but not our
publishing subscription.
Signed-off-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>
There's a bug in show-changes script where it defaults to values with
single quotes in them. So the default scripts directory is not
"scripts" but "'scripts'". This will be fixed in show-scripts, but for
now work it around here by explicitly defining the directories.
First issue is that on Jenkins, the beginning of the output seems to
be eaten, leaving us only the final part of the reports. This looks
like an issue stemming from redirecting stdout to stdout with
">/dev/stdout". Special case the stdout by not redirecting anything in
such case.
Second issue is that errors printed by the tools we use for generating
the reports go to stderr, so they don't show in the report. So
redirect their stderr to stdout, so the possible errors are visible in
the report file too. We do not want to redirect the stderr of the
print_image_reports function, because that would also capture
debugging stuff from "set -x" that GitHub Actions are using.
We push a commit with the nightly SDK tag to the main branch if the
SDK was built from the main branch. Which is what happens when we
build the nightly intermediate SDK. The final nightly SDK is not built
from the main branch, but rather from the nightly intermediate SDK
tag. Both of them point to the exactly same commit, but the difference
is in what `git rev-parse --abbrev-ref HEAD` returns for each of
those. When the main branch is checked out, the command will return
"main". When the nightly intermediate SDK tag is checked out, the
command will return "HEAD". So when nightly final SDK is being built,
the command returns a string different than "main" and thus decides
not to push the commit with the final nightly SDK tag to the main
branch. Rework it to assume that if `git rev-parse HEAD` and `git
rev-parse origin/main` return the same commit hash (and it's the
nightly build and all that) then the commit should be pushed.
We use "origin/main" instead of just "main" just in case the main
branch was not checked out before, for some reason (may come up in
testing with different names for the main branch when testing).
The kola run didn't pick up the version that was set up in the build
because the git changes from that step are lost.
Redo the version setup in the kola run to use the same version, and
skip the kola update test if no update payload can be found. In the
future we should copy it over from the GitHub Action artifact.
The vendor tools on the OEM partition weren't updated. We now want to
ship them as systemd-sysext images which we can easily update. This
change extends the Flatcar A/B update mechanism to cover the OEM
systemd-sysext images. The same mechanism is also able to support
"official" Flatcar extensions, e.g., a ZFS extension.
Since https://github.com/flatcar/scripts/pull/950 was merged,
tarball files `flatcar-{packages,sdk}-*.tar.zst` have been created
with mode 0600 instead of 0644. As a result, the files with mode 0600
were uploaded to bincache, but afterwards `copy-to-origin.sh` that in
turn runs rsync from bincache to the origin server could not read the
tarballs.
To fix that, it is necessary to chmod from 0600 to 0644 to make it
readable by rsync during the release process.
All of that happens because zstd sets the mode of the output file to
0600 in case of temporary files to avoid race condition.
See also https://github.com/facebook/zstd/pull/1644,
https://github.com/facebook/zstd/pull/3432.
The official release variable is used to decide whether a build ID gets
appended to the FLATCAR_VERSION (or VERSION in os-release) or not. It
was set for the image job but not for the vms job, causing the
build_sysext script to get the build ID appended to the FLATCAR_VERSION
which causes a mismatch with the one from the image job.
Set the official release variable in the vms job as well.
We currently use gzip together with pigz (parallel gzip) for importing
container images, and this is a lengthy operation (takes multiple minutes). By
moving to zstd we gain on all fronts: zstd produces smaller files, and is
faster to decompress/compress then pigz while using less resources.
Signed-off-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>
`build_image` depends on accesss to the torcx manifest and the "content
addressable nature" of the directory. We currently rely on the torcx output
root structure being preserved in the container image.
While we're moving the torcx output root out of the container image, preserve
its contents so that they can be restored from bincache.
The SDK container bind mounts __build__/images to the containers image
directory, but the CI uses a different path for images. This causes issues when
building the oem sysext, because it requires mounting an overlayfs. The current
path (~/build/...) is an overlayfs and an ovlerayfs can't be an upper
directory.
Align the CONTAINER_IMAGE_ROOT and CONTAINER_TORCX_ROOT values with standard
practices to that oem sysext building in jenkins ci works.
Signed-off-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>
I seem to have problems with getting a network connectivity inside
QEMU vm when running the tests in the azure machine. I don't know
what's the cause, but for the dev container tests these problems can
be worked around by using the locally provided dev container
image. Make it possible by specifying QEMU_DEVCONTAINER_FILE in the
environment.
This change adds a "<summary>" sub-block to the "<details>" block in
markdown test reports.
The test run details output, which is used to print debug and error
messages of test runs in case of test failures, was meant to be
fold-able, aiding readability of test reports. This is implemented using
the "<details>" feature. However, we forgot to mark a "<summary>" line
within the "<details>" block, leading to the blocks not being fold-able
but instead being visible all the time.
Signed-off-by: Thilo Fromm <thilofromm@microsoft.com>
Forwarding parameters to another bash invocation through a string
interpreted as a bash script is a bit troublesome. It is not enough to
wrap a parameter like 'foo bar' in escaped double quotes (\") to avoid
it being split into two parameters by bash executing the script
string. It mostly works, but there's always a risk of having a path
where this breaks. It's rare
Wrapping into escaped quotes, be them double or single, also won't
work for passing an array of parameters, so it's even easier here to
trigger globbing or bracket expansion or another unwanted splitting of
supposedly one parameter into multiple. Globbing can be temporarily
disabled with 'set -f' or 'set -o noglob', but this still leaves all
the other special bash characters unescaped. So each parameter in the
array should be escaped before they are put into the script string.
The escaping can be done with `printf` and its '%q` formatter, so
let's do so. For single parameters it is as simple as
`foo_escaped=$(printf '%q' "${foo}")`, for arrays a loop needs to be
used.
The two-phase nightly builds create an intermediate tag first, which
didn't match the main nightly tag regexp before. Because of that, the
commit was not pushed to the main branch. The following final SDK
build had a version that matched the regexp, but the last commit (with
the intermediate tag on it) wasn't in main, and thus was also not
pushed.
This change adds markdown output support to tapfile helper.
tap_generate_report() has been refactored to use low-level output
functions to write tests; TAP and markdown output is supported and both
are generated by default. Also, it should be straightforward to add
other output formats by implementing the respective low level print
functions.
The markdown output is now used by run-kola-tests.yaml to generate step
output and, if run from a PR, add a comment with test results to the PR.
Signed-off-by: Thilo Fromm <thilofromm@microsoft.com>
This change updates the github actions kola test runner workflow to use
the new, separated artifacts produced by ci.yaml.
Further, it adds a fix for the devcontainer tests. Devcontainer and bin
packages used in the devcontainer tests are now served from a local
temporary web server.
The change also adds the qemu_update test and provides the respective
update payload.
Lastly, the tests now use a local torcx_manifest.json produced by
ci.yaml, which points to a torcx tarball also served by the local
temporary web server.
For the GitHub CI we have to use --qemu-skip-mangle because the LXC
containers don't have access to loop devices. Running with
--qemu-skip-mangle means that the serial console does not get captured
completely because systemd and dracut messages are missing, and thus we
don't catch these errors in kola.
Make the skipping conditional and use it in Jenkins at least for the
nightlies and releases.
The get_git_channel function failed to work which resulted in the
Alpha release job skipping the AWS publishing for the Alpha channel
because it defaulted to the developer channel as fallback when git
rejected to work on the directory owned by the build user while running
as root user. A new version of git caused this behavior change and also
prints an error message that explains to have to set safe.directory.
Set the git config entry safe.directory for the /work path when
entering the mantle container where git runs as root while working on
the directory owned by the build user.
Getting the contents of the directory in the buildcache involves using
rsync with some ssh invocation to log in as a bincache user. It's not
a thing that will work locally unless the user gets ahold of the SSH
key allowing the user to log in to buildcache as a bincache user.
Replace it by downloading two files that are actually needed for
building vms: an image file and the version file. This just uses curl
and is accessible for everyone.
This uses the new size-change-report.sh script to print out some
information about largest files being added/removed and files with
greatest increase/decrease in file size between two versions of the
image.
This relies on flatcar_production_image_initrd_contents.txt being
uploaded to the server. It also exports the WITHWTD environment
variable with a value 1, which will make the package-diff script to
try out the wtd contents file variant first.
To ensure that we can update from very old releases, add a test with a
fixed old release, here the Stable release that introduced arm64
support to have the same test logic for both architectures.
Recently we ran into sporadic corruption issues for AWS EC2 AMIs.
We use the streamOptimized VMDK format and it seems to cause problems
at the AWS side, regardless if created by qemu-img or vmdk-convert.
Switch to using the plain AMI images for uploading as workaround.
For embargoed releases it is useful to apply patches locally to build
with them before they are public. This allows to push the same patches
to the repo during the Flatcar release at the embargo lift. The result
is the same (as long as the scripts patches did not change parts of the
setup logic that was running before they got applied), we can just build
earlier and thus do the Flatcar release directly on the embargo lift
instead of having to wait with the build because it would require the
patches to be in the repos.
