Image changes job needs a list of OEMIDs that are built for a specific
architecture. Similar information already existed in the
coreos-base/common-oem-files ebuild, so factor it out to a separate
file, so the image changes job does not need to source the entire
ebuild (or process it in other way), but rather source the smaller
file.
Instead of depending on default value of build_image's base_sysext
parameter, create a file that explicitly lists which base sysexts will
be built for each architecture. The file can be sourced by other
scripts that need this kind of information. Currently, image.sh and
image_changes.sh use this file.
This is to limit the amount of reports consisting purely of failures,
because some files were missing. And those files will be missing,
because an OEM might not even have any image for certain arches (like
digitalocean has no arm64 images).
It certainly does happen on GitHub Actions that the HEAD commit has no
tag. In such case, fake the tag from version file. The git tag in this
scenario is used to figure out a channel transition.
This avoid assuming that the current working directory in the
generate_image_changes_report function is actually toplevel directory
of the scripts repo.
It was only needed for the show-changes script. Now that show-changes
script allows to set the repos parent directory with an environment
variable, we set the variable instead of changing the working
directory.
The special Brightbox image uses the OpenStack userdata in Ignition but
lacked Afterburn usage. It actually works to use the OpenStack image and
directly which also enables Afterburn, thus we can drop the special
image.
Don't build a special image for Brightbox but recommend to use OpenStack
images directly. A symlink is added to help with the download of
hardcoded user scripts.
For Brightbox we can use the OpenStack image but the import only works
with unpacked images. After we enabled internal qcow2 compression the
.gz or .bz2 external compression doesn't provide any benefits and makes
the import more complicated.
Provide the OpenStack image without external compression in addition.
The other files are kept for now but we could also delete them if we
announce this in advance.
- updated github actions for runc, containerd, and docker to not handle
nonexistent ebuilds in app-torcx/ anymore
- removed spurious package_run_dependencies from build_image_util.sh
- build_sysext: generate pkginfo before mangle script runs
use zstd for compression; add cli flag to select compression
- ci_automation_common.sh: remove spurious `/` from match string
- coreos, board-packages, bootengine: bump ebuild revisions
- kernel commonconfig: add squashfs zstd support
Signed-off-by: Thilo Fromm <thilofromm@microsoft.com>
This change makes QEMU_UPDATE_PAYLOAD configurable via
ci-automation/settings.env where it was hard-wired before.
The change also fixes fall-out in qemu_update.sh by ensuring a local tmp
directory is created before it is used by the test.
Signed-off-by: Thilo Fromm <thilofromm@microsoft.com>
Switch to using a managed identity instead of file based credentials for
running kola/ore (not plume). This covers our test subscription, but not our
publishing subscription.
Signed-off-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>
There's a bug in show-changes script where it defaults to values with
single quotes in them. So the default scripts directory is not
"scripts" but "'scripts'". This will be fixed in show-scripts, but for
now work it around here by explicitly defining the directories.
First issue is that on Jenkins, the beginning of the output seems to
be eaten, leaving us only the final part of the reports. This looks
like an issue stemming from redirecting stdout to stdout with
">/dev/stdout". Special case the stdout by not redirecting anything in
such case.
Second issue is that errors printed by the tools we use for generating
the reports go to stderr, so they don't show in the report. So
redirect their stderr to stdout, so the possible errors are visible in
the report file too. We do not want to redirect the stderr of the
print_image_reports function, because that would also capture
debugging stuff from "set -x" that GitHub Actions are using.
We push a commit with the nightly SDK tag to the main branch if the
SDK was built from the main branch. Which is what happens when we
build the nightly intermediate SDK. The final nightly SDK is not built
from the main branch, but rather from the nightly intermediate SDK
tag. Both of them point to the exactly same commit, but the difference
is in what `git rev-parse --abbrev-ref HEAD` returns for each of
those. When the main branch is checked out, the command will return
"main". When the nightly intermediate SDK tag is checked out, the
command will return "HEAD". So when nightly final SDK is being built,
the command returns a string different than "main" and thus decides
not to push the commit with the final nightly SDK tag to the main
branch. Rework it to assume that if `git rev-parse HEAD` and `git
rev-parse origin/main` return the same commit hash (and it's the
nightly build and all that) then the commit should be pushed.
We use "origin/main" instead of just "main" just in case the main
branch was not checked out before, for some reason (may come up in
testing with different names for the main branch when testing).
The kola run didn't pick up the version that was set up in the build
because the git changes from that step are lost.
Redo the version setup in the kola run to use the same version, and
skip the kola update test if no update payload can be found. In the
future we should copy it over from the GitHub Action artifact.
The vendor tools on the OEM partition weren't updated. We now want to
ship them as systemd-sysext images which we can easily update. This
change extends the Flatcar A/B update mechanism to cover the OEM
systemd-sysext images. The same mechanism is also able to support
"official" Flatcar extensions, e.g., a ZFS extension.
Since https://github.com/flatcar/scripts/pull/950 was merged,
tarball files `flatcar-{packages,sdk}-*.tar.zst` have been created
with mode 0600 instead of 0644. As a result, the files with mode 0600
were uploaded to bincache, but afterwards `copy-to-origin.sh` that in
turn runs rsync from bincache to the origin server could not read the
tarballs.
To fix that, it is necessary to chmod from 0600 to 0644 to make it
readable by rsync during the release process.
All of that happens because zstd sets the mode of the output file to
0600 in case of temporary files to avoid race condition.
See also https://github.com/facebook/zstd/pull/1644,
https://github.com/facebook/zstd/pull/3432.
The official release variable is used to decide whether a build ID gets
appended to the FLATCAR_VERSION (or VERSION in os-release) or not. It
was set for the image job but not for the vms job, causing the
build_sysext script to get the build ID appended to the FLATCAR_VERSION
which causes a mismatch with the one from the image job.
Set the official release variable in the vms job as well.
We currently use gzip together with pigz (parallel gzip) for importing
container images, and this is a lengthy operation (takes multiple minutes). By
moving to zstd we gain on all fronts: zstd produces smaller files, and is
faster to decompress/compress then pigz while using less resources.
Signed-off-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>
`build_image` depends on accesss to the torcx manifest and the "content
addressable nature" of the directory. We currently rely on the torcx output
root structure being preserved in the container image.
While we're moving the torcx output root out of the container image, preserve
its contents so that they can be restored from bincache.
The SDK container bind mounts __build__/images to the containers image
directory, but the CI uses a different path for images. This causes issues when
building the oem sysext, because it requires mounting an overlayfs. The current
path (~/build/...) is an overlayfs and an ovlerayfs can't be an upper
directory.
Align the CONTAINER_IMAGE_ROOT and CONTAINER_TORCX_ROOT values with standard
practices to that oem sysext building in jenkins ci works.
Signed-off-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>
