Commit Graph

4905 Commits

Author SHA1 Message Date
Gabriel Adrian Samfira
f126debb37 Allow specifying multiple compression formats
Add the ability to specify a comma separated list of compression formats.

Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
2022-03-07 12:01:21 +00:00
Gabriel Adrian Samfira
f0a535271c Use gzip compression for OpenStack images
This change makes the Jenkins job output openstack images using gzip
compression format. This allows OpenStack users to directly consume images
by simply specifying the URL to the image. Glance will then download the
image, unarchive it and add it to it's catalogue.

Fixes #575

Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
2022-03-07 12:01:21 +00:00
Gabriel Adrian Samfira
571de4064b Make image compression format configurable
This change adds a new flag called --image_compression_format which
allows us to output the final VM image as one of the supported formats:

bz2 (default), gz, zip or none

if the compression format is "none" or "", the image will not be compressed.

Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
2022-03-07 12:01:21 +00:00
flatcar-ci
05ae47afd7 New version: sdk-3170.0.0-nightly-20220306-0139 2022-03-06 04:46:43 +00:00
Kai Lüke
0cc95e3b3e
Merge pull request #244 from flatcar-linux/kai/set-group
Fix and improve channel handling
2022-03-04 14:22:10 +01:00
Kai Lueke
db7220eced ci-automation: set the channel from the git tag
For now we had only "developer" images in the new pipeline.
Based on the git tag like "alpha-1234.0.0" set the channel (group) for
the image and also use this logic when finding the channel in the QEMU
update test.
2022-03-04 13:49:18 +01:00
Thilo Fromm
68305d60f0
Merge pull request #243 from flatcar-linux/t-lo/fix-tap-parser-errorr
ci-automation: fix tap parser errors
2022-03-04 13:43:07 +01:00
flatcar-ci
44f1275126 New version: sdk-3168.0.0-nightly-20220304-0139 2022-03-04 04:53:45 +00:00
Kai Lueke
b12c5b8783 jenkins/images: find channel from base channel variable
Using the tags on the branch is not enough to find the channel we want
to be the dev build be related to.
Use the base channel variable which was introduced for this.
2022-03-03 16:31:53 +01:00
Thilo Fromm
eba1cdb4c2 ci-automation/tapfile_helper_lib.sh: fix CI TAP parse errors
This change fixes and adds more string chars escaping in the test error
debug output ("\" are removed and a bug in removing '"' is fixed),
addressing a parser errof the CI encountered when ingesting TAP output.

Furthermore, line numbering is shortened, and test names have a spurious
"-" prefix removed.

Signed-off-by: Thilo Fromm <thilo@kinvolk.io>
2022-03-03 16:29:38 +01:00
flatcar-ci
7755567f31 New version: sdk-3167.0.0-nightly-20220303-0139 2022-03-03 05:00:50 +00:00
Thilo Fromm
c94315eeba
Merge pull request #242 from flatcar-linux/t-lo/tapfile-helper-fix-test-output
ci-automation/tapfile_helper_lib.sh: read test output from file
2022-03-02 17:46:10 +01:00
Sayan Chowdhury
ee7e48cb14
Merge pull request #241 from flatcar-linux/sayan/merge-pro-in-regular
build_library/vm_image_util.sh: Pro image to use non-pro package & use flag
2022-03-02 20:44:03 +05:30
Thilo Fromm
2b2bfad5e1 ci-automation/tapfile_helper_lib.sh: use subshell, break lines after 200
Signed-off-by: Thilo Fromm <thilo@kinvolk.io>
2022-03-02 15:35:57 +01:00
Thilo Fromm
22af2876e6 ci-automation/tapfile_helper_lib.sh: read test output from file
This change updates the tapfile helper to read test output from a file
instead of passig it inline in the SQL INSERT statement.
This fixes an issue with large error output from tests, which breaks
tap_ingest_tapfile():
    ci-automation/tapfile_helper_lib.sh: line 31: /usr/bin/sqlite3: Argument list too long
Error observed with the cl.toolbox.dnf-install test, which can generate
8000 lines of output. Fix tested with the same output.

Signed-off-by: Thilo Fromm <thilo@kinvolk.io>
2022-03-02 13:02:52 +01:00
Kai Lüke
ef9914e06e
Merge pull request #235 from flatcar-linux/kai/add-update-test
ci-automation: add update test
2022-03-02 11:27:43 +01:00
Thilo Fromm
b72586c4de
Merge pull request #236 from flatcar-linux/t-lo/tool-for-fetching-build-stage-images
ci-automation/util/fetch_image.sh: fetch CI build stage image
2022-03-02 10:10:17 +01:00
Thilo Fromm
637e5d52ec
Apply suggestions of my favourite proofreader
Krzesimir continues to save me from embarrassing spelling mistakes 💙

Co-authored-by: Krzesimir Nowak <knowak@microsoft.com>
2022-03-02 10:09:59 +01:00
flatcar-ci
b50f030015 New version: sdk-3166.0.0-nightly-20220302-0139 2022-03-02 04:52:43 +00:00
flatcar-ci
16bd244b62 New version: sdk-3165.0.0-nightly-20220301-0139 2022-03-01 04:47:20 +00:00
Kai Lueke
a9c3a31efb ci-automation: add missing update test
The kola update test was missing. It is performed as update from the
old image to the newly built payload to ensure that the new image is
compatible for old clients.
2022-02-28 15:59:17 +01:00
flatcar-ci
eb2394b9ac New version: sdk-3164.0.0-nightly-20220228-0139 2022-02-28 04:51:57 +00:00
flatcar-ci
be6145f8e1 New version: sdk-3163.0.0-nightly-20220227-0139 2022-02-27 04:45:02 +00:00
flatcar-ci
f688a206bd New version: sdk-3162.0.0-nightly-20220226-0139 2022-02-26 04:50:56 +00:00
Thilo Fromm
88a4df98b1
Merge pull request #239 from flatcar-linux/t-lo/ci-automation-add-qemu-uefi-test
ci-automation/vendor-testing: add qemu_uefi
2022-02-25 12:15:01 +01:00
Thilo Fromm
1b4022d237
Merge pull request #237 from flatcar-linux/t-lo/container-builds-update-version-in-main-branch
ci-automation: SDK build updates version.txt in main branch
2022-02-25 12:14:24 +01:00
Thilo Fromm
af339721ef version.txt: bump to latest nightly for CI 2022-02-24 16:47:38 +01:00
Thilo Fromm
9afab3aac4 ci-automation/vendor-testing: add qemu_uefi
This change adds the qemu_uefi.sh vendor test. It reuses most of the
implementation in qemu.sh (qemu_uefi.sh is a soft-link to qemu.sh).

This also enables qemu testing for ARM64.

Signed-off-by: Thilo Fromm <thilo@kinvolk.io>
2022-02-24 13:54:58 +01:00
Thilo Fromm
2dcc461e17
Merge pull request #238 from flatcar-linux/t-lo/fix-test-report-update-script
ci-automation: fix test_update_reruns typo
2022-02-24 12:38:16 +01:00
Thilo Fromm
308a2a2ab6 ci-automation/sdk_bootstrap: check submodules for nightly branch push
Signed-off-by: Thilo Fromm <thilo@kinvolk.io>
2022-02-24 12:06:28 +01:00
Thilo Fromm
82da911c27 ci-automation/sdk_bootstrap: Only push to main in nightlies
Signed-off-by: Thilo Fromm <thilo@kinvolk.io>
2022-02-24 11:02:34 +01:00
Thilo Fromm
39b65765b4 ci-automation: fix test_update_reruns typo
Signed-off-by: Thilo Fromm <thilo@kinvolk.io>
2022-02-24 10:11:38 +01:00
Thilo Fromm
7cadae957a ci-automation: SDK build updates version.txt in main branch
This change has sdk_bootstrap update the origin branch when run from the
main branch, updating the SDK and OS version in 'main' for each SDK
bootstrap build.

Release / maintenance branches have the SDK version set in the
versionfile at release time. But main is never updated.

Updating the versionfile in main when a new SDK is built ensures that
dev branches based on main will also use the correct SDK version (e.g.
in subsequent CI builds).
2022-02-23 20:41:57 +01:00
Thilo Fromm
248ffcef03 ci-automation/util/fetch_image.sh: fetch CI build stage image
Signed-off-by: Thilo Fromm <thilo@kinvolk.io>
2022-02-23 19:25:12 +01:00
Thilo Fromm
368aa61796
Merge pull request #234 from flatcar-linux/t-lo/container-tests-add-per-image-summary-report
ci-automation/test*: per-image summary report, honour pre-set PARALLEL_TESTS
2022-02-23 17:42:25 +01:00
Thilo Fromm
e92521c166 ci-automation/test*: per-image summary, honor parallel
Signed-off-by: Thilo Fromm <thilo@kinvolk.io>
2022-02-23 11:30:54 +01:00
Thilo Fromm
5ac2418679
Merge pull request #232 from flatcar-linux/t-lo/container-tests-publish-results
ci-automation: publish test results, add to GC
2022-02-23 10:23:40 +01:00
Thilo Fromm
3253435d6c ci-automation/test.sh: publish to _kola_tmp, not debug/_kola_tmp
Signed-off-by: Thilo Fromm <thilo@kinvolk.io>
2022-02-22 18:27:36 +01:00
Thilo Fromm
3397167b5e
Update ci-automation/test.sh: fix typo.
Co-authored-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>
2022-02-22 18:13:15 +01:00
Mathieu Tortuyaux
e63a191c16
Merge pull request #222 from flatcar-linux/tormath1/timeout-em
kola/em: increase timeout
2022-02-22 18:04:31 +01:00
Thilo Fromm
8157bf0302 ci-automation: publish test results, add to GC
This change adds copying test results to the build cache server, and
adds respective deletion to the garbage collector.

Also, the patch fixes an issue with torcx publishing (manifest
publishing had arch hard-coded).

Signed-off-by: Thilo Fromm <thilo@kinvolk.io>
2022-02-22 16:22:57 +01:00
Thilo Fromm
db062b2e81
Merge pull request #230 from flatcar-linux/t-lo/add-sdk-container-image-update
update_sdk_container_image: New SDK container from existing image (minor / patch updates)
2022-02-22 15:45:36 +01:00
Thilo Fromm
ceba903d1b
Merge pull request #228 from flatcar-linux/t-lo/container-builds-add-testing
ci-automation: Add foundational testing primitives
2022-02-22 15:45:23 +01:00
Thilo Fromm
081df6cd2c ci-automtion/packages.sh: fix torcx URL, add manifest
Signed-off-by: Thilo Fromm <thilo@kinvolk.io>
2022-02-22 15:44:04 +01:00
Jeremi Piotrowski
f8efad73bf
Merge pull request #231 from flatcar-linux/jepio/build-image-disable-ebuild-locks
build_library/build_image_util: disable ebuild-locks when merging binary packages.
2022-02-22 13:52:46 +01:00
Jeremi Piotrowski
4cdacf5ae4 build_library/build_image_util: disable ebuild-locks when merging binary packages
Disable ebuild-locks for the emerge command that creates the image.
Ebuild-locks protect unsandboxed ebuild phases from running
concurrently, but also slow things down greatly when a lot of
concurrency would otherwise be possible. The image build phase merges a
big amount of binary packages, and I am not aware of us having any
phases that risk concurrently modifying shared files.

I have been testing this for the last months and have not seen any
failures. The time savings are significant: this cuts image build time
from 20m to 10m for me.

Signed-off-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>
2022-02-22 09:42:42 +00:00
Thilo Fromm
38d85729bf update_sdk_container_image: update SDK container image
This change introduces update_sdk_container_image, a script to generate
a new SDK container image based on an existing SDK container. The
script is meant to be used for minor / patch level SDK changes (like
test suite updates).

Signed-off-by: Thilo Fromm <thilo@kinvolk.io>
2022-02-21 20:06:30 +01:00
Thilo Fromm
4f39e0112f ci-automation/tests.sh: use http in torcx manifest
Use HTTP instead of https because Ignition does not recognise
letsencrypt certificates, leading to test breakage in
docker.torcx-manifest-pkgs.

Add a note in settings.env to explicitly call out HTTP requirement of
build cache server.

Signed-off-by: Thilo Fromm <thilo@kinvolk.io>
2022-02-21 17:23:53 +01:00
Thilo Fromm
0fa985b872 ci-automation/test.sh: stage torcx manifest
Signed-off-by: Thilo Fromm <thilo@kinvolk.io>
2022-02-21 16:30:30 +01:00
Thilo Fromm
1045fd5ac8 ci-automation/README.md: add docs for qemu test 2022-02-21 13:57:11 +01:00