We currently use gzip together with pigz (parallel gzip) for importing
container images, and this is a lengthy operation (takes multiple minutes). By
moving to zstd we gain on all fronts: zstd produces smaller files, and is
faster to decompress/compress then pigz while using less resources.
Signed-off-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>
`build_image` depends on accesss to the torcx manifest and the "content
addressable nature" of the directory. We currently rely on the torcx output
root structure being preserved in the container image.
While we're moving the torcx output root out of the container image, preserve
its contents so that they can be restored from bincache.
The SDK container bind mounts __build__/images to the containers image
directory, but the CI uses a different path for images. This causes issues when
building the oem sysext, because it requires mounting an overlayfs. The current
path (~/build/...) is an overlayfs and an ovlerayfs can't be an upper
directory.
Align the CONTAINER_IMAGE_ROOT and CONTAINER_TORCX_ROOT values with standard
practices to that oem sysext building in jenkins ci works.
Signed-off-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>
I seem to have problems with getting a network connectivity inside
QEMU vm when running the tests in the azure machine. I don't know
what's the cause, but for the dev container tests these problems can
be worked around by using the locally provided dev container
image. Make it possible by specifying QEMU_DEVCONTAINER_FILE in the
environment.
This change adds a "<summary>" sub-block to the "<details>" block in
markdown test reports.
The test run details output, which is used to print debug and error
messages of test runs in case of test failures, was meant to be
fold-able, aiding readability of test reports. This is implemented using
the "<details>" feature. However, we forgot to mark a "<summary>" line
within the "<details>" block, leading to the blocks not being fold-able
but instead being visible all the time.
Signed-off-by: Thilo Fromm <thilofromm@microsoft.com>
Forwarding parameters to another bash invocation through a string
interpreted as a bash script is a bit troublesome. It is not enough to
wrap a parameter like 'foo bar' in escaped double quotes (\") to avoid
it being split into two parameters by bash executing the script
string. It mostly works, but there's always a risk of having a path
where this breaks. It's rare
Wrapping into escaped quotes, be them double or single, also won't
work for passing an array of parameters, so it's even easier here to
trigger globbing or bracket expansion or another unwanted splitting of
supposedly one parameter into multiple. Globbing can be temporarily
disabled with 'set -f' or 'set -o noglob', but this still leaves all
the other special bash characters unescaped. So each parameter in the
array should be escaped before they are put into the script string.
The escaping can be done with `printf` and its '%q` formatter, so
let's do so. For single parameters it is as simple as
`foo_escaped=$(printf '%q' "${foo}")`, for arrays a loop needs to be
used.
The two-phase nightly builds create an intermediate tag first, which
didn't match the main nightly tag regexp before. Because of that, the
commit was not pushed to the main branch. The following final SDK
build had a version that matched the regexp, but the last commit (with
the intermediate tag on it) wasn't in main, and thus was also not
pushed.
This change adds markdown output support to tapfile helper.
tap_generate_report() has been refactored to use low-level output
functions to write tests; TAP and markdown output is supported and both
are generated by default. Also, it should be straightforward to add
other output formats by implementing the respective low level print
functions.
The markdown output is now used by run-kola-tests.yaml to generate step
output and, if run from a PR, add a comment with test results to the PR.
Signed-off-by: Thilo Fromm <thilofromm@microsoft.com>
This change updates the github actions kola test runner workflow to use
the new, separated artifacts produced by ci.yaml.
Further, it adds a fix for the devcontainer tests. Devcontainer and bin
packages used in the devcontainer tests are now served from a local
temporary web server.
The change also adds the qemu_update test and provides the respective
update payload.
Lastly, the tests now use a local torcx_manifest.json produced by
ci.yaml, which points to a torcx tarball also served by the local
temporary web server.
For the GitHub CI we have to use --qemu-skip-mangle because the LXC
containers don't have access to loop devices. Running with
--qemu-skip-mangle means that the serial console does not get captured
completely because systemd and dracut messages are missing, and thus we
don't catch these errors in kola.
Make the skipping conditional and use it in Jenkins at least for the
nightlies and releases.
The get_git_channel function failed to work which resulted in the
Alpha release job skipping the AWS publishing for the Alpha channel
because it defaulted to the developer channel as fallback when git
rejected to work on the directory owned by the build user while running
as root user. A new version of git caused this behavior change and also
prints an error message that explains to have to set safe.directory.
Set the git config entry safe.directory for the /work path when
entering the mantle container where git runs as root while working on
the directory owned by the build user.
Getting the contents of the directory in the buildcache involves using
rsync with some ssh invocation to log in as a bincache user. It's not
a thing that will work locally unless the user gets ahold of the SSH
key allowing the user to log in to buildcache as a bincache user.
Replace it by downloading two files that are actually needed for
building vms: an image file and the version file. This just uses curl
and is accessible for everyone.
This uses the new size-change-report.sh script to print out some
information about largest files being added/removed and files with
greatest increase/decrease in file size between two versions of the
image.
This relies on flatcar_production_image_initrd_contents.txt being
uploaded to the server. It also exports the WITHWTD environment
variable with a value 1, which will make the package-diff script to
try out the wtd contents file variant first.
To ensure that we can update from very old releases, add a test with a
fixed old release, here the Stable release that introduced arm64
support to have the same test logic for both architectures.
Recently we ran into sporadic corruption issues for AWS EC2 AMIs.
We use the streamOptimized VMDK format and it seems to cause problems
at the AWS side, regardless if created by qemu-img or vmdk-convert.
Switch to using the plain AMI images for uploading as workaround.
For embargoed releases it is useful to apply patches locally to build
with them before they are public. This allows to push the same patches
to the repo during the Flatcar release at the embargo lift. The result
is the same (as long as the scripts patches did not change parts of the
setup logic that was running before they got applied), we can just build
earlier and thus do the Flatcar release directly on the embargo lift
instead of having to wait with the build because it would require the
patches to be in the repos.
All invocations of gpg in ci-automation pass --batch as an argument except the
import. Be consistent by having it included everywhere. Additionally, since
ci-automation runs wrapped in a systemd service, no tty is available so batch
is needed for correctness.
The container image was only created if it didn't exist locally. This
would result in fixes not being in a downstream job that is scheduled
to a different worker node on Jenkins that has a stale copy.
For the build automation we will now always download the latest
container tar ball based on comparing the image ID from a new artifact,
and for registry images we pull the container image to make sure that
we don't use a stale copy when we rebuild.
When there is no SDK container image in the registry, the fallback
looks at bincache but bincache isn't backed up and may be cleaned of
old releases. While this won't be the regular case, the container
image registry may be unavailable (or renamed as happened now), or
people would like to rerun the image job which relies on the packages
container.
qemu_update vendor test was downloading a wrong LTS image when it was
testing the old LTS image. This is because it was using a current
symlink, which for LTS channel will always point to the new LTS. Old
LTS is available under current-${YEAR} symlink. We can get the
information about year from the lts-info file.
FLATCAR_VERSION and FLATCAR_SDK_VERSION are defined in the version
file, so it should be sourced before trying to use those. Here we try
to do it in a limited scope.
Also, SDK container link should use the dockerized version in a
directory name.
Currently we skip the nightly build if there are no changes. This
didn't work well because a new run doesn't fix any failure because the
rerun became a no-op.
Check if the main artifacts we expect from a step are found, as simple
heuristic on whether a rerun is needed.
I found a duplicate function and verified that it's the only one via
comm -12 <(sort ci-automation/ci_automation_common.sh) <(sort sdk_lib/sdk_container_common.sh) | grep function
I'm not sure if this is due to a case where we only import one but
can't import the other, hence I'm not deleting it now.
This failed when used from ( secret_to_file ... VAR ; cat $VAR )
because ( ) starts a new subshell PID and secret_to_file's returned
/proc/PID/fd/X path was then using the wrong PID.
the JSON object is passed from the Groovy script to the release script,
we just need to extract the correct AWS Marketplace product ID based on
the "<channel>-<arch>".
Exception for the stable-amd64 where we also need to get the stable-pro
product ID.
Signed-off-by: Mathieu Tortuyaux <mtortuyaux@microsoft.com>
