The original intention of the "binpkg" prefix in the CI binary package
cache URL was to separate packages from other build artifacts like
containers, images, and SDK tarballs. Motivation was to separate
developer content (binary packages) from CI automation artifacts
(everything else); since binary packages are not used by the CI.
This broke assumptions in scripts which use the binary host URL for
other things than packages - e.g. SDK tarballs or images. These
scripts would get a bincache URL with "binpkg/" prepended, while CI
automation would *not* use that prefix.
This change removes the use of "binpkg/" altogether since it would not
work as intended without more significant changes to build scripts.
garbage_collect.sh was using 'docker_vernum' where it should have been
using 'vernum' (as push_pkgs.sh does).
Also, make sure release directories are removed, not just packages.
Signed-off-by: Thilo Fromm <thilo@kinvolk.io>
It is a back-port of the 3 commits in
https://github.com/flatcar-linux/scripts/pull/209
to flatcar-3033. The backport was required because of a conflict
in set_version.
This change adds a job for publishing binary packages to the build cache
server to the ci automation.
Also, setup_board is updated to use the buildcache package cache if a
nightly build version is detected.
Lastly, ci-automation/README.md is updated to reflect the changes. THe
readme now also elaborates on the binary package cache server (build
infrastructure).
Co-authored-by: Krzesimir Nowak <knowak@microsoft.com>
sdk_entry.sh is expected to be called by the root user, so we set USER
root:root. Also we add a "root" entry to passwd and group since it does
not exist in the SDK tarball.
Signed-off-by: Thilo Fromm <thilo@kinvolk.io>
In bce3bd9031fa78c79173f29bcac5bc49c81a03b4, we added support for podman
for building and running the SDK container. The presence of podman is
auto-detected in sdk_container_common.sh. However, podman is preverred
over docker, requiring users to use *sudo* (which podman requires and
docker does not).
This change uses docker when present, podman otherwise. It also improves
podman detection - 'podman' uses argv[0] in its version string, so if
'docker' is a symlink to 'podman', 'podman --version' output uses
'docker'. This broke the SDK container on hosts which have a 'docker'
symlink to 'podman' since 'podman' is then run w/o 'sudo'.
Signed-off-by: Thilo Fromm <thilo@kinvolk.io>
At least with Podman it's not possible to call "container rm" on a
running container without the force flag.
Add the force flag which is also used elsewhere already.
When the docker wrapper script for Podman is used, we need to
explicitly create a root user container with "sudo podman".
Podman also has its own bridge for root user containers which we need
to detect, and it requires to explicitly say to use the Docker Hub
Caddy image.
Add a "$docker" variable that uses sudo podman as needed, and also
check which bridge interface to use. The filter had to be changed
because it didn't work with Podman. Use the Docker Hub Caddy image
explicitly.
The creation of the target version file failed:
/home/sdk/sdk_entry.sh: line 32: /build/amd64-usr/etc/target-version.txt: Permission denied
Use root permissions to create the file.
This change ensures the binpkg host is updated if the board (OS) version
differs from the SDK version.
This is to ensure /build/[arch] uses the correct binary package cache.
Signed-off-by: Thilo Fromm <thilo@kinvolk.io>
For execution of the compiled binaries in /build/arm64-usr we rely on
qemu-user binfmt emulation and have to tell it where the root is with
QEMU_LD_PREFIX because build systems don't chroot into /build/arm64-usr
themselves (which also works just by chance on amd64 because we have
similar glibc versions and so on). The env var setup was done in
/etc/profile.d/qemu-aarch64.sh but is now not read anymore since the
container runs the shell not as login shell.
Add the login options to the bash and su calls when starting the
container.
For test builds the commit that updates the submodules can be free-
standing but for releases we need to push it to the branch and also
sign the tag.
Add optional arguments that are used by the tag-release script in
flatcar-build-scripts.
run_sdk_container uses the sourcetree version to decide whether to
re-use existing containers or create new ones. However, containers were
not matched by exact name - instead, plain --filter name="..." was used,
leading to prefix matching. This change updates name="..." to use
regular expressions for exact matching.
Signed-off-by: Thilo Fromm <thilo@kinvolk.io>
In 9fba5789f95bf79a393c5e35cb2346d3900e29ba we introduced
--torcx_output_root as an optional command line parameter
and had it default to "${DEFAULT_BUILD_ROOT}", inadvertently
diverging from the previous default, which was
"${DEFAULT_BUILD_ROOT}/torcx".
This change sets the correct default root "${DEFAULT_BUILD_ROOT}/torcx" to bring
build_packages back into alignment with build_image.
Stop relying on github redirects, they are a mixed blessing and using
them broke emerge-gitclone inside dev-container in silent way. The
script could not find a desired revision of portage-stable or
coreos-overlay, because it tried to pull from kinvolk instead of
flatcar-linux github org. The redirects seem to hinder fetching a
specific commit, so the script pulled something else (HEAD or main?).
Back-port for flatcar-3033 branch.
This change introduces a containerised SDK as a replacement for cork SDK
operations. It also simplifies versioning by removing the need for
manifest repos as well as usage of the "repo" tool by use of git
submodules for coreos-overlay and portage-stable.
The following feature scripts are added:
- run_sdk_container: Run a command in an SDK container, using the
current scripts repo + ebuild submodules.
current scripts repo + ebuild submodules.
- bootstrap_sdk_container / build_sdk_container_image: Bootstrap a new
SDK and create an SDK container from the resulting SDK tarball.
The following additions have been made to SDK scripts:
- setup_board: add --pkgdir parameter to use a custom binary packge
directory.
We override `PARALLEL_TESTS`, because kola run with PARALLEL_TESTS >= 4
causes the tests to provision >= 12 ARM servers at the same time. As the
da11 region does not have that many free ARM servers, the whole tests
will fail. With PARALLEL_TESTS=2 the total number of servers stays < 10.
In addition, we override `timeout` to 10 hours, because it takes more
than 8 hours to run all tests only with 2 tests in parallel.
Equinix Metal ARM server are not yet hourly available in the default `sv15` region
so we override the `PACKET_REGION` to `Dallas` since it's available in this region.
We do not override `PACKET_REGION` for both board on top level because we need to keep proximity
for PXE booting.
Signed-off-by: Mathieu Tortuyaux <mtortuyaux@microsoft.com>
