Commit Graph

4697 Commits

Author SHA1 Message Date
Dongsu Park
173554db7a
Merge pull request #229 from flatcar-linux/dongsu/glsa-20220216
build_library: update allow list for GLSA checks 2022-02-16
2022-02-17 11:12:26 +01:00
Thilo Fromm
6c76bfa1cd
ci-automation/tapfile_helper_lib.sh: add @pothos' retcode fix
Co-authored-by: Kai Lüke <pothos@users.noreply.github.com>
2022-02-17 10:45:10 +01:00
Dongsu Park
580a5c6852 build_library: update GLSA allowlist
201908-24: polkit 0.120-r2, so not affected
201909-01: perl 5.34.0, so not affected
202003-26: python 3.9.8, so not affected
202005-09: python 3.9.8, so not affected
202006-03: perl 5.34.0, so not affected
202008-01: python 3.9.8, so not affected
202101-18: python 3.9.8, so not affected
202104-04: python 3.9.8, so not affected
202105-34: bash 5.1_p8, so not affected
202107-31: polkit 0.120-r2, so not affected
202107-48: systemd 250.3, so not affected
2022-02-17 10:44:28 +01:00
Dongsu Park
a900d579c4 *: blacklist/whitelist to denylist/allowlist
Replace blacklist with denylist, and whitelist with allowlist.
Use proper vocabulary in all over the code.
2022-02-17 10:44:25 +01:00
Thilo Fromm
5bfe2f395c
Apply @pothos' suggestions from code review
Co-authored-by: Kai Lüke <pothos@users.noreply.github.com>
2022-02-17 10:29:05 +01:00
Thilo Fromm
f6f44e2ca8 ci-automation: first stab at adding testing
Signed-off-by: Thilo Fromm <thilo@kinvolk.io>
2022-02-16 19:59:45 +01:00
Kai Lüke
221351927e
Merge pull request #226 from flatcar-linux/kai/package-diff
jenkins/images: print out image diff report
2022-02-16 12:10:30 +01:00
Kai Lueke
b949428185 jenkins/images: print out image diff report
Often a change results in unexpected effects on the image, e.g., when
a wrong package version gets chosen or the package installs files under
/etc, or binaries of library dependencies get pulled in. Besides
inspecting the image manually, the package-diff tool also gives
valuable insights.
Run the package-diff tool in a comparison to the last release and print
the image URL alongside for convenience.
2022-02-16 10:12:48 +01:00
Mathieu Tortuyaux
6d0d7ea2ba
Merge pull request #224 from flatcar-linux/tormath1/https
jenkins/kola: use httpS URL for PXE boot
2022-02-11 15:09:15 +01:00
Dongsu Park
0cedf256f7
Merge pull request #223 from flatcar-linux/dongsu/jenkins-dev-container-branch
jenkins: do not check out branches of coreos-overlay and portage-stable
2022-02-10 17:02:29 +01:00
Sayan Chowdhury
9cc851bc78
Merge pull request #225 from flatcar-linux/sayan/update-sdk-3139.0.0
New major release: 3139.0.0
2022-02-09 21:38:47 +05:30
flatcar-ci
1049e54af8
New major release: 3139.0.0 2022-02-09 15:40:21 +05:30
Mathieu Tortuyaux
bd30be56ee
jenkins/kola: use httpS URL for PXE boot
Follow-up of:
* https://github.com/flatcar-linux/mantle/pull/288
* https://github.com/flatcar-linux/Flatcar/issues/527

Signed-off-by: Mathieu Tortuyaux <mtortuyaux@microsoft.com>
2022-02-09 10:46:42 +01:00
Jeremi Piotrowski
c9ddda260b
Merge pull request #221 from flatcar-linux/jepio/azure-fixed-vhd
build_library: vm_image_util: create fixed-size VHDs for Azure
2022-02-08 16:23:43 +01:00
Dongsu Park
39301b007f jenkins: do not check out branches of coreos-overlay and portage-stable
The default branch of both repos, coreos-overlay and portage-stable,
should be `main`. If we checkout `master` branch, which contains
invalid source code that was deprecated many years ago, the build could
sometimes fail, e.g. when trying to build perl 5.26.2 with gcc 10.

Simply delete the code checking out branches, as the part is already
being handled in emerge-gitclone.
2022-02-08 12:26:47 +01:00
Mathieu Tortuyaux
5c4ac96f69
kola/em: increase timeout
number of test increased. While we don't have yet a way to reduce
testing time, let's increase the timeout.

Signed-off-by: Mathieu Tortuyaux <mtortuyaux@microsoft.com>
2022-02-06 11:56:12 +01:00
Jeremi Piotrowski
323c1b28b1 changelog: Add entry for Azure VHD format change 2022-02-02 10:01:19 +01:00
Jeremi Piotrowski
07568fb418 build_library: vm_image_util: create fixed-size VHDs for Azure
Azure requires disks to be fixed-size VHD files when uploading to blob storage
in order to create image/gallery objects from them. This is documented here[1].
To prevent mistakes from happening create disks in that format directly so that
any azure compatible tool can upload them, though azcopy is recommend because
it handles their sparseness best.

This has not been an issue for us so far because kola uses code from an older
utility that transparently handled the dynamic-to-fixed-size conversion for VHD
files (azure-vhd-utils). But people working with these things for the first
time fall into this trap.

[1]: https://docs.microsoft.com/en-us/azure/virtual-machines/linux/create-upload-generic#resizing-vhds.

Signed-off-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>
2022-02-01 17:52:09 +00:00
Mathieu Tortuyaux
98d964d689
Merge pull request #220 from flatcar-linux/tormath1/em-metro
equinixmetal: use metro instead of facility
2022-01-26 16:27:09 +01:00
Mathieu Tortuyaux
373976b1eb
jenkins/kola/packet: use metro instead of facility
Follow up of: https://github.com/flatcar-linux/mantle/pull/281

Signed-off-by: Mathieu Tortuyaux <mtortuyaux@microsoft.com>
2022-01-26 15:49:30 +01:00
Mathieu Tortuyaux
7ddf2121a7
Merge pull request #219 from flatcar-linux/sayan/update-sdk-3127.0.0
New major release: 3127.0.0
2022-01-26 11:35:18 +01:00
flatcar-ci
153a347999
New major release: 3127.0.0 2022-01-26 14:11:59 +05:30
Jeremi Piotrowski
45be4de050
Merge pull request #217 from flatcar-linux/jepio/pkg-list-torcx-components
list torcx components in package list.
2022-01-17 15:05:56 +01:00
Jeremi Piotrowski
3af02e323c build_library/torcx_manifest.sh: list all component pkgs in sources_on_disk()
source_on_disk() so far relied on the 'sourcePackage' field, which contains the
primary dependency of a torcx packge (app-torcx/docker ->
app-emulation/docker). Now the 'metaPackage' field (app-torcx/docker) is used,
which lets us look at RDEPENDS and figure out all packages that are indirectly
installed when installing a torcx package. torcx_dependencies() does just that,
so move it's definition to torcx_manifest.sh.

Signed-off-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>
2022-01-14 16:00:51 +00:00
Jeremi Piotrowski
499fade8d3 build_torcx_store: store additional 'metaPackage' field in torcx manifest
The torcx_manifest.json file currently has a 'sourcePackage' field which is
extracted from the first runtime dependency of the torcx package ebuild. This
is a convention, and causes sourcePackage to hold 'app-emulation/docker' for
the 'app-torcx/docker' package. This does not carry enough information to be
able to figure out what other packages are part of the torcx package.

Store an additional field, 'metaPackage', in the manifest which contains the
name of the torcx package. With the right ebuild it is then possible to figure
out what other packages are part of a given torcx package. This can then be
used to add that information to the image packages list.

Signed-off-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>
2022-01-14 15:46:08 +00:00
Jeremi Piotrowski
78254a18eb build_torcx_store: build torcx components in parallel
Instead of looping over the package list, pass all the packages to a single
emerge call and specify num jobs. This lets emerge build/install all of them in
parallel, shaving some time off the torcx build.

Signed-off-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>
2022-01-14 15:37:12 +00:00
Sayan Chowdhury
7d36fedee6
Merge pull request #216 from flatcar-linux/sayan/update-sdk-3115.0.0
New major release: 3115.0.0
2022-01-13 20:07:38 +05:30
flatcar-ci
6deacd7839
New major release: 3115.0.0 2022-01-13 19:50:37 +05:30
Kai Lueke
e188557195 changelog/README.md: specify current security fix section format
The entries added in changelog/security/ do not follow our existing
security section in the release notes:
  https://www.flatcar.org/releases/#release-3033.2.0

Document the structure and an example to use the right format that we
need for release note generation.
2022-01-12 17:29:54 +01:00
Thilo Fromm
0bc166c560
Merge pull request #215 from flatcar-linux/t-lo/enable-nightly-build-binpkgs
SDK container: enable binpkg cache for nightly builds
2022-01-12 16:05:04 +01:00
Thilo Fromm
719689992c SDK container: enable binpkg cache for nightly builds
Signed-off-by: Thilo Fromm <thilo@kinvolk.io>
2022-01-12 15:44:30 +01:00
Kai Lüke
f9b51461ab
Merge pull request #212 from flatcar-linux/kai/set-group-only-in-usr
set_lsb_release: only set update-engine GROUP in /usr, not /etc
2022-01-11 15:01:34 +01:00
Kai Lueke
0c9804a9a4 set_lsb_release: only set update-engine GROUP in /usr, not /etc
The default image group is already encoded in
/usr/share/flatcar/update.conf but it was written to
/etc/flatcar/update.conf as well. This can cause problems when the user
switches channels by forcing an update to a specific release from the
different channel (e.g., through the flatcar-update tool) as it leaves
the file under /etc/flatcar/update.conf out of sync with the new
channel version in /usr/share/flatcar/update.conf.

Since we don't really need to write a specific channel to /etc on new
images as we can rely on the value from /usr, we now leave any possible
overwriting of the value in /etc entirely to the user.
2022-01-11 15:00:43 +01:00
Thilo Fromm
e84e90e625
Merge pull request #213 from flatcar-linux/t-lo/ci-fix-nightly-binpkg-url
settings / ci-automation: remove "binpkg" prefix
2022-01-11 11:25:53 +01:00
Jeremi Piotrowski
43687783af
Merge pull request #214 from flatcar-linux/jepio/ignore-egg-info-symlink
build_library/check_root: ignore dangling '*egg-info' symlink
2022-01-11 11:25:21 +01:00
Jeremi Piotrowski
7caafa6cd6 build_library/check_root: ignore dangling '*egg-info' symlink
Since the update of dev-python/certifi, running the command
`./image_to_vm.sh --format gce --board=amd64-usr` fails due to a
dangling symlink. This symlink is located in
/usr/lib64/python3.9/site-packages and is not supposed to be installed
in the first place because of this INSTALL_MASK entry in
coreos-overlay/profiles/coreos/targets/generic/oem-aci/make.defaults:

  INSTALL_MASK="${INSTALL_MASK}
    /usr/*/python3*
  "

There is an open upstream bug that INSTALL_MASK doesn't work correctly on
symlinks (https://bugs.gentoo.org/678462).
The best we can do at this time is to ignore the dangling symlink.

Signed-off-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>
2022-01-11 10:02:53 +00:00
flatcar-ci
0bbae51a5e settings / ci-automation: remove "binpkg" prefix
The original intention of the "binpkg" prefix in the CI binary package
cache URL was to separate packages from other build artifacts like
containers, images, and SDK tarballs. Motivation was to separate
developer content (binary packages) from CI automation artifacts
(everything else); since binary packages are not used by the CI.

This broke assumptions in scripts which use the binary host URL for
other things than packages - e.g. SDK tarballs or images. These
scripts would get a bincache URL with "binpkg/" prepended, while CI
automation would *not* use that prefix.

This change removes the use of "binpkg/" altogether since it would not
work as intended without more significant changes to build scripts.
2022-01-11 09:56:21 +01:00
Thilo Fromm
bf053ed800
Merge pull request #211 from flatcar-linux/t-lo/checkout-imgrove-tag-handling
checkout: omit checking for tags in submodules
2022-01-10 15:46:53 +01:00
flatcar-ci
fa090f839e checkout: omit checking for tags in submodules
Signed-off-by: flatcar-ci <infra+ci@flatcar-linux.org>
2022-01-10 15:23:25 +01:00
Thilo Fromm
a2ae11cb3d
Merge pull request #210 from flatcar-linux/t-lo/garbage_collect-fix-binpkg-url
ci-automation/garbage_collect: fix binpkg url
2022-01-10 14:44:10 +01:00
Thilo Fromm
e076931c7d ci-automation/garbage_collect: fix binpkg url
garbage_collect.sh was using 'docker_vernum' where it should have been
using 'vernum' (as push_pkgs.sh does).

Also, make sure release directories are removed, not just packages.

Signed-off-by: Thilo Fromm <thilo@kinvolk.io>
2022-01-10 14:07:33 +01:00
Thilo Fromm
567a526a5d
Merge pull request #209 from flatcar-linux/t-lo/container-ci-builds-publish-nightly-packages
ci-automation + setup_board: publish and use binpkgs
2022-01-10 12:09:21 +01:00
Thilo Fromm
a6ddcda88e ci-automation: Apply suggestions from PR review
Co-authored-by: Krzesimir Nowak <knowak@microsoft.com>
2022-01-10 11:41:03 +01:00
Thilo Fromm
0ecd0be77a ci-automation/README.md: pkg publish, bin cache added
Signed-off-by: Thilo Fromm <thilo@kinvolk.io>
2022-01-07 17:47:50 +01:00
flatcar-ci
7d76cfedf7 ci-automation + setup_board: publish and use binpkgs
This change adds a job for publishing binary packages to the build cache
server to the ci automation.

Also, setup_board is updated to use the buildcache package cache if a
nightly build version is detected.

Signed-off-by: flatcar-ci <infra+ci@flatcar-linux.org>
2022-01-07 17:16:44 +01:00
Thilo Fromm
13a6f6e0eb
Merge pull request #202 from flatcar-linux/t-lo/add-checkout-helper
checkout: helper script for checking out branches and tags
2022-01-07 16:08:04 +01:00
Jeremi Piotrowski
a3ac30292a
Merge pull request #208 from flatcar-linux/jepio/azure-gallery
jenkins/kola/azure: make use of --azure-use-gallery parameter
2022-01-07 15:43:16 +01:00
Thilo Fromm
e9299efb36
checkout: fix typo in update strategy detection
Co-authored-by: Kai Lüke <pothos@users.noreply.github.com>
2022-01-07 15:26:17 +01:00
Jeremi Piotrowski
7cafff85f4 jenkins/kola/azure: make use of --azure-use-gallery parameter
Signed-off-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>
2022-01-07 13:56:32 +01:00
Thilo Fromm
301e80e7f5 checkout: helper script for checking out branches and tags
Signed-off-by: Thilo Fromm <thilo@kinvolk.io>
2022-01-07 13:54:10 +01:00