this logic takes care of copying binary packages, SDK and packages
containers to cloudflare bucket in the `r2:flatcar/mirror/` location
Signed-off-by: Mathieu Tortuyaux <mtortuyaux@microsoft.com>
We got a DNS resolution problem due to UDP packets not going to the
configured server. For now try the host network (and otherwise maybe
a custom DNS server?).
But in any case we should not block the release on that and continue.
Signed-off-by: Kai Lueke <kailuke@microsoft.com>
The wrong variable was used to set up the mapping of the rclone config
into the rclone container and it wasn't set up in the right function.
Move it into the right function and use the right variable name but also
don't rely on /proc/PID/fd/FD to be mappable into the container but
instead use a regular temp file.
Signed-off-by: Kai Lueke <kailuke@microsoft.com>
rclone was previously called from the Mantle image but it's not the
case anymore because we need some environment variables (CHANNEL, ARCH,
etc.)
Let's switch to the `rclone` Docker image.
Signed-off-by: Mathieu Tortuyaux <mtortuyaux@microsoft.com>
The upload to R2 was added experimentally and we now want to make use of
it. The CHANNEL variable wasn't defined and it failed because of that.
Do the upload for all channels and set the variable up first. Existing
releases should get synced from the current Origin server via a FUSE
mount that we anyway want to rely on for the directory listing that
Caddy creates. Left to decide is how we manage the "current" version
but that is done manually anyway as of now.
Signed-off-by: Kai Lueke <kailuke@microsoft.com>
- Tighten the patterns used for nightly tags detection.
- Compare hashes instead of names to figure out if we are on top of a
branch (fixes the issue of no nightly tags reachable from the
release branches). Jenkins is doing `git fetch origin "${branch}";
git checkout FETCH_HEAD` and this was confusing the `git rev-parse
--abbrev-ref HEAD` code (it returned `HEAD` instead of `${branch}`).
- Account for possible multiple tags in a single commit.
- Made the tagging fail in dubious situations.
- Reindent the code, modernize a bit.
`show_changes_params` is not available in this lexical scope, we
should have been using `show_changes_params_ref`. This has worked so
far only because all the callers of the functions were passing
`show_changes_params` to be referenced by
`show_changes_params_ref`. Just a lucky happenstance.
Spotted by Chewi.
It happens that we have some leftovers instances running in an "error"
state (the error comes from the OpenStack scheduled deletion). This
leads to instance creation error during the test because quota limits
are hit.
Let's clean-up everything before running the new tests.
This won't impact tests from other channels as OpenStack is limited to
one CI job at a time.
Signed-off-by: Mathieu Tortuyaux <mtortuyaux@microsoft.com>
Hetzner is having some capacity issues[^1]:
- amd64: CPX plans (CPX11 to CPX51) - Falkenstein (FSN) and Nuremberg (NBG)
- arm64: CAX plans (CAX11 to CAX41) - Helsinki (HEL) and Nuremberg (NBG)
Let's switch the location:
* Helsinki (hel1) for amd64
* Keep Falkenstein (fsn1) for arm64
[^1]: https://status.hetzner.com/incident/aa5ce33b-faa5-4fd0-9782-fde43cd270cf
Signed-off-by: Mathieu Tortuyaux <mtortuyaux@microsoft.com>
In the test we should use the unzipped image, which is the one
documented.
This allows us to drop some modifications to our OpenStack instance.
Signed-off-by: Mathieu Tortuyaux <mtortuyaux@microsoft.com>
No need for garbage collection since one temporary project is allocated with 1h of
lifespan for each run.
Signed-off-by: Mathieu Tortuyaux <mtortuyaux@microsoft.com>
Co-authored-by: Julian Tölle <julian.toelle97@gmail.com>
This adds support for providing a value for the newly introduce
--azure-kola-vnet kola parameter through the environment. This parameter is
meant to indicate that kola is running inside of a vnet in Azure and the kola
created storage account will be restricted to being accessed from that vnet.
This lets us disable public access to storage accounts.
Needs a corresponding change to jenkins jobs, because we have no way of
determining what vnet a worker node is connected to programmatically. So it
needs to be defined by the job.
Signed-off-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>
After making flatcar_production_pxe.vmlinuz a symlink to
flatcar_production_image.vmlinuz the signature creation didn't work
because the target could not be found.
As we do with the generic image, download the kernel from bincache, too,
before starting the VM image build.
On Windows, the .bz2 compression format is not supported by native
tooling and external tools like 7zip need to be installed.
Switching to .zip compression, there will be no need for the extra step
of having external tools.
See: https://github.com/flatcar/Flatcar/issues/1009
Signed-off-by: Adrian Vladu <avladu@cloudbasesolutions.com>
The qemu and qemu_uefi_secure images have the same contents as the
qemu_uefi image which wastes space on the release server. A similar
case is the PXE vmlinuz which is the same as the regular one, too.
Set up symlinks for same images, and also detect this when compressing
to set up symlinks there as well. To reduce complexity, the qemu and
qemu_uefi_secure images are not supported anymore and the Jenkins or
GitHub CI will skip over them if specified. Users that build their own
images need to adapt, though.
This change extends the garbage collector for the build cache server to
remove cached release artifacts. Release artifacts are copied to the
official mirrors and do not need to remain on the build cache after a
release was published.
By default, the 10 latest releases of all channels (including LTS and
previous LTS) are kept.
Also excluded from garbage collection are:
- Emerging new major releases (i.e. major number larger than the
latest Alpha release)
- channel progressions (major number exists in the lists of releases to
keep but minor is bigger than any release)
- patch releases (major and minor exist in list of releases to keep but
patch level is newer than in any release)
- SDKs (tarballs and containers) of any release in the list of releases
to keep; i.e. the SDK in <MAJOR>.0.0 for any release to keep.
Signed-off-by: Thilo Fromm <thilofromm@microsoft.com>
gensub is a GNU extension; however, POSIXLY_CORRECT is enforced in
systemd-run which triggers gawk's traditional / posix mode.
Unset it before calling gawk to make gensub available.
Signed-off-by: Thilo Fromm <thilofromm@microsoft.com>
