This is to limit the amount of reports consisting purely of failures,
because some files were missing. And those files will be missing,
because an OEM might not even have any image for certain arches (like
digitalocean has no arm64 images).
It certainly does happen on GitHub Actions that the HEAD commit has no
tag. In such case, fake the tag from version file. The git tag in this
scenario is used to figure out a channel transition.
This avoid assuming that the current working directory in the
generate_image_changes_report function is actually toplevel directory
of the scripts repo.
It was only needed for the show-changes script. Now that show-changes
script allows to set the repos parent directory with an environment
variable, we set the variable instead of changing the working
directory.
The special Brightbox image uses the OpenStack userdata in Ignition but
lacked Afterburn usage. It actually works to use the OpenStack image and
directly which also enables Afterburn, thus we can drop the special
image.
Don't build a special image for Brightbox but recommend to use OpenStack
images directly. A symlink is added to help with the download of
hardcoded user scripts.
For Brightbox we can use the OpenStack image but the import only works
with unpacked images. After we enabled internal qcow2 compression the
.gz or .bz2 external compression doesn't provide any benefits and makes
the import more complicated.
Provide the OpenStack image without external compression in addition.
The other files are kept for now but we could also delete them if we
announce this in advance.
- updated github actions for runc, containerd, and docker to not handle
nonexistent ebuilds in app-torcx/ anymore
- removed spurious package_run_dependencies from build_image_util.sh
- build_sysext: generate pkginfo before mangle script runs
use zstd for compression; add cli flag to select compression
- ci_automation_common.sh: remove spurious `/` from match string
- coreos, board-packages, bootengine: bump ebuild revisions
- kernel commonconfig: add squashfs zstd support
Signed-off-by: Thilo Fromm <thilofromm@microsoft.com>
This change makes QEMU_UPDATE_PAYLOAD configurable via
ci-automation/settings.env where it was hard-wired before.
The change also fixes fall-out in qemu_update.sh by ensuring a local tmp
directory is created before it is used by the test.
Signed-off-by: Thilo Fromm <thilofromm@microsoft.com>
Switch to using a managed identity instead of file based credentials for
running kola/ore (not plume). This covers our test subscription, but not our
publishing subscription.
Signed-off-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>
There's a bug in show-changes script where it defaults to values with
single quotes in them. So the default scripts directory is not
"scripts" but "'scripts'". This will be fixed in show-scripts, but for
now work it around here by explicitly defining the directories.
First issue is that on Jenkins, the beginning of the output seems to
be eaten, leaving us only the final part of the reports. This looks
like an issue stemming from redirecting stdout to stdout with
">/dev/stdout". Special case the stdout by not redirecting anything in
such case.
Second issue is that errors printed by the tools we use for generating
the reports go to stderr, so they don't show in the report. So
redirect their stderr to stdout, so the possible errors are visible in
the report file too. We do not want to redirect the stderr of the
print_image_reports function, because that would also capture
debugging stuff from "set -x" that GitHub Actions are using.
We push a commit with the nightly SDK tag to the main branch if the
SDK was built from the main branch. Which is what happens when we
build the nightly intermediate SDK. The final nightly SDK is not built
from the main branch, but rather from the nightly intermediate SDK
tag. Both of them point to the exactly same commit, but the difference
is in what `git rev-parse --abbrev-ref HEAD` returns for each of
those. When the main branch is checked out, the command will return
"main". When the nightly intermediate SDK tag is checked out, the
command will return "HEAD". So when nightly final SDK is being built,
the command returns a string different than "main" and thus decides
not to push the commit with the final nightly SDK tag to the main
branch. Rework it to assume that if `git rev-parse HEAD` and `git
rev-parse origin/main` return the same commit hash (and it's the
nightly build and all that) then the commit should be pushed.
We use "origin/main" instead of just "main" just in case the main
branch was not checked out before, for some reason (may come up in
testing with different names for the main branch when testing).
The kola run didn't pick up the version that was set up in the build
because the git changes from that step are lost.
Redo the version setup in the kola run to use the same version, and
skip the kola update test if no update payload can be found. In the
future we should copy it over from the GitHub Action artifact.
The vendor tools on the OEM partition weren't updated. We now want to
ship them as systemd-sysext images which we can easily update. This
change extends the Flatcar A/B update mechanism to cover the OEM
systemd-sysext images. The same mechanism is also able to support
"official" Flatcar extensions, e.g., a ZFS extension.
Since https://github.com/flatcar/scripts/pull/950 was merged,
tarball files `flatcar-{packages,sdk}-*.tar.zst` have been created
with mode 0600 instead of 0644. As a result, the files with mode 0600
were uploaded to bincache, but afterwards `copy-to-origin.sh` that in
turn runs rsync from bincache to the origin server could not read the
tarballs.
To fix that, it is necessary to chmod from 0600 to 0644 to make it
readable by rsync during the release process.
All of that happens because zstd sets the mode of the output file to
0600 in case of temporary files to avoid race condition.
See also https://github.com/facebook/zstd/pull/1644,
https://github.com/facebook/zstd/pull/3432.
The official release variable is used to decide whether a build ID gets
appended to the FLATCAR_VERSION (or VERSION in os-release) or not. It
was set for the image job but not for the vms job, causing the
build_sysext script to get the build ID appended to the FLATCAR_VERSION
which causes a mismatch with the one from the image job.
Set the official release variable in the vms job as well.
We currently use gzip together with pigz (parallel gzip) for importing
container images, and this is a lengthy operation (takes multiple minutes). By
moving to zstd we gain on all fronts: zstd produces smaller files, and is
faster to decompress/compress then pigz while using less resources.
Signed-off-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>
`build_image` depends on accesss to the torcx manifest and the "content
addressable nature" of the directory. We currently rely on the torcx output
root structure being preserved in the container image.
While we're moving the torcx output root out of the container image, preserve
its contents so that they can be restored from bincache.
The SDK container bind mounts __build__/images to the containers image
directory, but the CI uses a different path for images. This causes issues when
building the oem sysext, because it requires mounting an overlayfs. The current
path (~/build/...) is an overlayfs and an ovlerayfs can't be an upper
directory.
Align the CONTAINER_IMAGE_ROOT and CONTAINER_TORCX_ROOT values with standard
practices to that oem sysext building in jenkins ci works.
Signed-off-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>
I seem to have problems with getting a network connectivity inside
QEMU vm when running the tests in the azure machine. I don't know
what's the cause, but for the dev container tests these problems can
be worked around by using the locally provided dev container
image. Make it possible by specifying QEMU_DEVCONTAINER_FILE in the
environment.
This change adds a "<summary>" sub-block to the "<details>" block in
markdown test reports.
The test run details output, which is used to print debug and error
messages of test runs in case of test failures, was meant to be
fold-able, aiding readability of test reports. This is implemented using
the "<details>" feature. However, we forgot to mark a "<summary>" line
within the "<details>" block, leading to the blocks not being fold-able
but instead being visible all the time.
Signed-off-by: Thilo Fromm <thilofromm@microsoft.com>
Forwarding parameters to another bash invocation through a string
interpreted as a bash script is a bit troublesome. It is not enough to
wrap a parameter like 'foo bar' in escaped double quotes (\") to avoid
it being split into two parameters by bash executing the script
string. It mostly works, but there's always a risk of having a path
where this breaks. It's rare
Wrapping into escaped quotes, be them double or single, also won't
work for passing an array of parameters, so it's even easier here to
trigger globbing or bracket expansion or another unwanted splitting of
supposedly one parameter into multiple. Globbing can be temporarily
disabled with 'set -f' or 'set -o noglob', but this still leaves all
the other special bash characters unescaped. So each parameter in the
array should be escaped before they are put into the script string.
The escaping can be done with `printf` and its '%q` formatter, so
let's do so. For single parameters it is as simple as
`foo_escaped=$(printf '%q' "${foo}")`, for arrays a loop needs to be
used.
The two-phase nightly builds create an intermediate tag first, which
didn't match the main nightly tag regexp before. Because of that, the
commit was not pushed to the main branch. The following final SDK
build had a version that matched the regexp, but the last commit (with
the intermediate tag on it) wasn't in main, and thus was also not
pushed.
This change adds markdown output support to tapfile helper.
tap_generate_report() has been refactored to use low-level output
functions to write tests; TAP and markdown output is supported and both
are generated by default. Also, it should be straightforward to add
other output formats by implementing the respective low level print
functions.
The markdown output is now used by run-kola-tests.yaml to generate step
output and, if run from a PR, add a comment with test results to the PR.
Signed-off-by: Thilo Fromm <thilofromm@microsoft.com>
This change updates the github actions kola test runner workflow to use
the new, separated artifacts produced by ci.yaml.
Further, it adds a fix for the devcontainer tests. Devcontainer and bin
packages used in the devcontainer tests are now served from a local
temporary web server.
The change also adds the qemu_update test and provides the respective
update payload.
Lastly, the tests now use a local torcx_manifest.json produced by
ci.yaml, which points to a torcx tarball also served by the local
temporary web server.
For the GitHub CI we have to use --qemu-skip-mangle because the LXC
containers don't have access to loop devices. Running with
--qemu-skip-mangle means that the serial console does not get captured
completely because systemd and dracut messages are missing, and thus we
don't catch these errors in kola.
Make the skipping conditional and use it in Jenkins at least for the
nightlies and releases.
The get_git_channel function failed to work which resulted in the
Alpha release job skipping the AWS publishing for the Alpha channel
because it defaulted to the developer channel as fallback when git
rejected to work on the directory owned by the build user while running
as root user. A new version of git caused this behavior change and also
prints an error message that explains to have to set safe.directory.
Set the git config entry safe.directory for the /work path when
entering the mantle container where git runs as root while working on
the directory owned by the build user.
Getting the contents of the directory in the buildcache involves using
rsync with some ssh invocation to log in as a bincache user. It's not
a thing that will work locally unless the user gets ahold of the SSH
key allowing the user to log in to buildcache as a bincache user.
Replace it by downloading two files that are actually needed for
building vms: an image file and the version file. This just uses curl
and is accessible for everyone.
This uses the new size-change-report.sh script to print out some
information about largest files being added/removed and files with
greatest increase/decrease in file size between two versions of the
image.
This relies on flatcar_production_image_initrd_contents.txt being
uploaded to the server. It also exports the WITHWTD environment
variable with a value 1, which will make the package-diff script to
try out the wtd contents file variant first.
To ensure that we can update from very old releases, add a test with a
fixed old release, here the Stable release that introduced arm64
support to have the same test logic for both architectures.
Recently we ran into sporadic corruption issues for AWS EC2 AMIs.
We use the streamOptimized VMDK format and it seems to cause problems
at the AWS side, regardless if created by qemu-img or vmdk-convert.
Switch to using the plain AMI images for uploading as workaround.
For embargoed releases it is useful to apply patches locally to build
with them before they are public. This allows to push the same patches
to the repo during the Flatcar release at the embargo lift. The result
is the same (as long as the scripts patches did not change parts of the
setup logic that was running before they got applied), we can just build
earlier and thus do the Flatcar release directly on the embargo lift
instead of having to wait with the build because it would require the
patches to be in the repos.
All invocations of gpg in ci-automation pass --batch as an argument except the
import. Be consistent by having it included everywhere. Additionally, since
ci-automation runs wrapped in a systemd service, no tty is available so batch
is needed for correctness.
The container image was only created if it didn't exist locally. This
would result in fixes not being in a downstream job that is scheduled
to a different worker node on Jenkins that has a stale copy.
For the build automation we will now always download the latest
container tar ball based on comparing the image ID from a new artifact,
and for registry images we pull the container image to make sure that
we don't use a stale copy when we rebuild.
When there is no SDK container image in the registry, the fallback
looks at bincache but bincache isn't backed up and may be cleaned of
old releases. While this won't be the regular case, the container
image registry may be unavailable (or renamed as happened now), or
people would like to rerun the image job which relies on the packages
container.
qemu_update vendor test was downloading a wrong LTS image when it was
testing the old LTS image. This is because it was using a current
symlink, which for LTS channel will always point to the new LTS. Old
LTS is available under current-${YEAR} symlink. We can get the
information about year from the lts-info file.
FLATCAR_VERSION and FLATCAR_SDK_VERSION are defined in the version
file, so it should be sourced before trying to use those. Here we try
to do it in a limited scope.
Also, SDK container link should use the dockerized version in a
directory name.
Currently we skip the nightly build if there are no changes. This
didn't work well because a new run doesn't fix any failure because the
rerun became a no-op.
Check if the main artifacts we expect from a step are found, as simple
heuristic on whether a rerun is needed.
I found a duplicate function and verified that it's the only one via
comm -12 <(sort ci-automation/ci_automation_common.sh) <(sort sdk_lib/sdk_container_common.sh) | grep function
I'm not sure if this is due to a case where we only import one but
can't import the other, hence I'm not deleting it now.
This failed when used from ( secret_to_file ... VAR ; cat $VAR )
because ( ) starts a new subshell PID and secret_to_file's returned
/proc/PID/fd/X path was then using the wrong PID.
the JSON object is passed from the Groovy script to the release script,
we just need to extract the correct AWS Marketplace product ID based on
the "<channel>-<arch>".
Exception for the stable-amd64 where we also need to get the stable-pro
product ID.
Signed-off-by: Mathieu Tortuyaux <mtortuyaux@microsoft.com>
The mantle plume tool has two steps, pre-release is the mere upload and
release is the publication. In the past this was used to run the tests
inbetween but we don't do this anymore.
Run plume pre-release and release in a single job. Since plume can't
push to GCS in our case, we upload the files to bincache. Also do the
cloudformation update which was previously done in
flatcar-build-scripts but could only be run after the sync to Origin.
It requires the "aws" tool in the mantle container until we implement
this in plume directly.
I made a mistake and wrote a version like main-3363-0.0-stuff (note a
dash instead of a dot after the first number). Surprisingly the build
chugged along just fine almost until the end of the image job - it
detected invalid version string when the job wanted to create a
version.txt file:
ERROR build_image: script called: build_image '--board=amd64-usr' '--group=developer' '--output_root=/home/sdk/build/images' '--only_store_compressed' '--torcx_root=/home/sdk/build/torcx' 'prodtar' 'container'
ERROR build_image: Backtrace: (most recent call is last)
ERROR build_image: file build_image, line 196, called: split_ver '3363' 'SPLIT'
ERROR build_image: file common.sh, line 192, called: die 'Invalid version string '3363''
ERROR build_image:
ERROR build_image: Error was:
ERROR build_image: Invalid version string '3363'
Let's have a stricter version check in the beginning of the build
process, so the process fails sooner rather than later.
Now URLs for torcx packages are always present in the torcx manifest,
but for releases they may be pointing to the origin server where the
packages will be eventually uploaded. At the time of running the
tests, those packages are still only in the build cache, so change the
URLs to point to the build cache, so the test can pass.
Torcx manifest may contain paths and URLs as locations of
packages. There are two kinds of packages - vendored and
extra. Vendored packages normally have two locations - path to the
directory inside the image where the package is (which is why it's
called vendored), and a URL to the package on some remote
server. Extra packages only have a URL. But the URLs are added only
when we tell the build_torcx_store script to upload the packages at
the same time, which is what the old build pipeline was doing. With
the new pipeline, the upload happens as a separate step, thus the
upload is disabled when invoking build_torcx_store, and so the
packages are not getting URLs set. This change went unnoticed, because
a kola test checking the generated torcx manifest was only checking if
there is at least one location, either path or URL, and all the new
releases have no extra packages, only vendored ones.
When backporting the new pipeline to old LTS, the kola tests started
to fail, because old LTS had one extra package, and this is how I
noticed the problem.
The old pipeline had a release job where mantle's plume release tool
was invoked to publish the cloud images.
Implement a release job in the new pipeline with the same goals and
eventually even more automation.
To review the image changes and the changelog more easily and in case
of fixes, iterate over it without rebuilding the image, move this logic
to its own file where a new job could call it.
Instead of printing failed tests like this:
Failed tests: kubeadm.v1.25.0.cilium.base
kubeadm.v1.24.1.cilium.base
Do it like this:
Failed tests:
kubeadm.v1.25.0.cilium.base
kubeadm.v1.24.1.cilium.base
We have set success to true when the test cycle was broken, which was
a hacky way to avoid printing the give up message. But this setting
success to true also meant that the script returned with status 0,
which is wrong.
Add another variable for controlling printing the give up message.
The new m3.small instance does not have official Flatcar support yet
but we can already cover it in our PXE boot release tests.
The c3.small instances are legacy and m3.small is the new smallest
type.
