The os-release file was not only accessible through /usr/lib/ but
also through /usr/lib64 because "lib" was just a symlink.
Now that we split them up into two directories, add a compatibility
symlink in case /usr/lib64 was used to access os-release. A check
is added to also work without the split which is useful if the split
is not done for the SDK at the same time.
The standard location is /usr/lib/modules but on Flatcar "lib" was a
symlink to "lib64". Now this is going to be split up in separate
directories but with compatibility symlinks.
Add the new location to the ignore list.
Sysext images have a compatibility matching mechanism that searches for
the matching OS version or custom sysext level setting. On Flatcar
there is just the OS version set in /etc/os-release until now which
means that sysext images can't easily be used together with autoupdates
that change the OS version.
Define a sysext level for Flatcar so that users can refer to it instead
of the OS version when they have images that don't rely on a particular
Flatcar version.
Here an example of the now possible metadata:
/etc/extensions/NAME/usr/lib64/extension-release.d/extension-release.NAME
ID=flatcar
SYSEXT_LEVEL=1.0
and a symlink /etc/extensions/NAME/usr/lib → /etc/extensions/NAME/usr/lib64
to work around the problem that using lib/ as path destroys Flatcar's
lib → lib64 symlink.
In the future the matching logic hopefully gets more flexible because
now it is just a string comparison. Also, the architecture is not
matched either for now - we should work with upstream to improve this.
Closes: https://github.com/flatcar-linux/Flatcar/issues/643
Rename sztd to zst and amend the changelog. The zstd binary generates a
compressed file with the .zst extension by default.
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
This change adds a new flag called --image_compression_format which
allows us to output the final VM image as one of the supported formats:
bz2 (default), gz, zip or none
if the compression format is "none" or "", the image will not be compressed.
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
Disable ebuild-locks for the emerge command that creates the image.
Ebuild-locks protect unsandboxed ebuild phases from running
concurrently, but also slow things down greatly when a lot of
concurrency would otherwise be possible. The image build phase merges a
big amount of binary packages, and I am not aware of us having any
phases that risk concurrently modifying shared files.
I have been testing this for the last months and have not seen any
failures. The time savings are significant: this cuts image build time
from 20m to 10m for me.
Signed-off-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>
Package users nowadays get created through systemd-sysuser files.
Gentoo uses the acct-user|groups packages to allocate stable IDs for
these users. Since they get created at runtime, we have the problem
that they end up in /etc/passwd at boot time which would be fine if
they follow the acct-user allocations but it could also be that there
is a package that uses its own sysuser files, leading to dynamic ID
allocation which we can't control and may result in ugly user ID
mismatches that are hard to resolve again. Normally we intend to ship
all system users under /usr/share/baselayout/passwd so that /etc/passwd
is really left to the user's own entries.
Generate the /etc/passwd sysuser entries at image build time and move
these entries over to /usr/share/baselayout/passwd so that all
system users reside in this database. We should still ensure to have
acct-user packages for all system users or at least hardcoded user
IDs, therefore, add a check for that.
201908-24: polkit 0.120-r2, so not affected
201909-01: perl 5.34.0, so not affected
202003-26: python 3.9.8, so not affected
202005-09: python 3.9.8, so not affected
202006-03: perl 5.34.0, so not affected
202008-01: python 3.9.8, so not affected
202101-18: python 3.9.8, so not affected
202104-04: python 3.9.8, so not affected
202105-34: bash 5.1_p8, so not affected
202107-31: polkit 0.120-r2, so not affected
202107-48: systemd 250.3, so not affected
Azure requires disks to be fixed-size VHD files when uploading to blob storage
in order to create image/gallery objects from them. This is documented here[1].
To prevent mistakes from happening create disks in that format directly so that
any azure compatible tool can upload them, though azcopy is recommend because
it handles their sparseness best.
This has not been an issue for us so far because kola uses code from an older
utility that transparently handled the dynamic-to-fixed-size conversion for VHD
files (azure-vhd-utils). But people working with these things for the first
time fall into this trap.
[1]: https://docs.microsoft.com/en-us/azure/virtual-machines/linux/create-upload-generic#resizing-vhds.
Signed-off-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>
source_on_disk() so far relied on the 'sourcePackage' field, which contains the
primary dependency of a torcx packge (app-torcx/docker ->
app-emulation/docker). Now the 'metaPackage' field (app-torcx/docker) is used,
which lets us look at RDEPENDS and figure out all packages that are indirectly
installed when installing a torcx package. torcx_dependencies() does just that,
so move it's definition to torcx_manifest.sh.
Signed-off-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>
The torcx_manifest.json file currently has a 'sourcePackage' field which is
extracted from the first runtime dependency of the torcx package ebuild. This
is a convention, and causes sourcePackage to hold 'app-emulation/docker' for
the 'app-torcx/docker' package. This does not carry enough information to be
able to figure out what other packages are part of the torcx package.
Store an additional field, 'metaPackage', in the manifest which contains the
name of the torcx package. With the right ebuild it is then possible to figure
out what other packages are part of a given torcx package. This can then be
used to add that information to the image packages list.
Signed-off-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>
The default image group is already encoded in
/usr/share/flatcar/update.conf but it was written to
/etc/flatcar/update.conf as well. This can cause problems when the user
switches channels by forcing an update to a specific release from the
different channel (e.g., through the flatcar-update tool) as it leaves
the file under /etc/flatcar/update.conf out of sync with the new
channel version in /usr/share/flatcar/update.conf.
Since we don't really need to write a specific channel to /etc on new
images as we can rely on the value from /usr, we now leave any possible
overwriting of the value in /etc entirely to the user.
Since the update of dev-python/certifi, running the command
`./image_to_vm.sh --format gce --board=amd64-usr` fails due to a
dangling symlink. This symlink is located in
/usr/lib64/python3.9/site-packages and is not supposed to be installed
in the first place because of this INSTALL_MASK entry in
coreos-overlay/profiles/coreos/targets/generic/oem-aci/make.defaults:
INSTALL_MASK="${INSTALL_MASK}
/usr/*/python3*
"
There is an open upstream bug that INSTALL_MASK doesn't work correctly on
symlinks (https://bugs.gentoo.org/678462).
The best we can do at this time is to ignore the dangling symlink.
Signed-off-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>
This change adds a job for publishing binary packages to the build cache
server to the ci automation.
Also, setup_board is updated to use the buildcache package cache if a
nightly build version is detected.
Signed-off-by: flatcar-ci <infra+ci@flatcar-linux.org>
Stop relying on github redirects, they are a mixed blessing and using
them broke emerge-gitclone inside dev-container in silent way. The
script could not find a desired revision of portage-stable or
coreos-overlay, because it tried to pull from kinvolk instead of
flatcar-linux github org. The redirects seem to hinder fetching a
specific commit, so the script pulled something else (HEAD or main?).
Flatcar is in the NIST CPE dictionary. Let's programmatically build the
`CPE_NAME` in the build process in order to be scanned.
`CPE_NAME` is part of `/etc/os-release` with the following manual entry:
```
CPE_NAME=
A CPE name for the operating system, in URI binding syntax, following the Common Platform Enumeration Specification[2] as proposed by the NIST.
This field is optional. Example: "CPE_NAME="cpe:/o:fedoraproject:fedora:17""
...
[^2]: Common Platform Enumeration Specification
http://scap.nist.gov/specifications/cpe/
```
Which indicates that the current version of CPE is 2.3.
Closes: https://github.com/flatcar-linux/Flatcar/issues/536
Signed-off-by: Mathieu Tortuyaux <mtortuyaux@microsoft.com>
This updates the default settings in build scripts to use
https://mirror.release.flatcar-linux.net/
instead of the google storage bucket if no binhost or FLATCAR_DEV_BUILDS
is specified.
Defaults are updated for
* update_chroot (runs at SDK initialisation time)
* setup_board (creates /boards/[ARCH]/) chroots
* the development container
* set_version
The documentation says it always returns zero, which is not true -
portageq could return a non-zero return value and that would be the
return value of the function. Fix the function to actually follow the
documentation - apparently the function should just return an empty
string in case of failure (like package not found).
It has some weird semantics that seem to trip us up after updating
bash to 5.1. We tried to use it inside functions to clean up some
stuff after function returns. This can be emulated with an EXIT trap
within a subshell. Fortunately all the users of the RETURN trap were
not setting any global variables - modifications of such variables are
local to the subshell and are lost when the subshell exits.
Now that GLSA metadata was updated as of 2021-09-03, we need to
add the following entries to the GLSA allow list, to avoid build
failures caused by `glsa-check -t all`.
202006-03: perl 5.26.2, only SDK, allowlist
202008-01: python 2.7.15 & 3.6.5, only SDK, allowlist
202101-18: python 2.7.15 & 3.6.5, only SDK, allowlist
202104-04: python 2.7.15 & 3.6.5, only SDK, allowlist
202105-22: samba 4.12.9, not affected, samba has no ldap flag, no smbd.
202105-34: bash 4.3, non-trivial to update
202107-31: polkit 0.113, in-progress
202107-48: systemd 247.9, backported the fixes to v247.9.
201904-13: git 2.26.3, so not affected
201909-08: dbus 1.12.20, so not affected
201911-01: openssh 8.6, so not affected
202003-12: sudo 1.9.5, so not affected
202003-20: systemd 246+, so not affected
202003-24: file 5.39, so not affected
202003-30: git 2.26.3, so not affected
202003-31: gdb 9.2, so not affected
202003-52: samba 4.12.9, so not affected
202004-10: openssl 1.1.1l, so not affected
202004-13: git 2.26.3, so not affected
202005-02: qemu 5.2, so not affected
disk_util sometimes bails out during build with an ASCII conversion
error:
Traceback (most recent call last):
File "/mnt/host/source/src/scripts/build_library/disk_util", line 1114, in <module>
main(sys.argv)
File "/mnt/host/source/src/scripts/build_library/disk_util", line 1110, in main
options.func(options)
File "/mnt/host/source/src/scripts/build_library/disk_util", line 779, in Verity
Tune2fsReadWrite(options, part, disable_rw=True)
File "/mnt/host/source/src/scripts/build_library/disk_util", line 716, in Tune2fsReadWrite
image.write(chr(flag_value))
UnicodeEncodeError: 'ascii' codec can't encode character '\xff' in position 0: ordinal not in range(128)
Curiously, the error does not reproduce every time (though the code
leading to the error is straightforward).
This change converts the integer to be written to a byte array (of size
1) instead of using chr(). Also, the file to be written is explicitly
opened in binary mode.
Signed-off-by: Thilo Fromm <thilo@kinvolk.io>
This file used to be imported by scripts coreos-base/cros-devutils,
which we have dropped. Now it is imported only from some other script
in build library so move it there. This leaves lib as a directory
where we keep shflags library.
This is some fallout from converting scripts from python2 to
python3. Output received from the functions in subprocess module now
return bytearrays, but we operate on them as if they were a text. So
decode the bytearrays to strings. Otherwise we are either getting some
junk values passed to the command line utilities (for example:
`b'/dev/loop2'` instead of `/dev/loop2`), or exceptions are thrown,
because a function expected a string.
This is just a conversion done by 2to3 with a manual updates of
shebangs to mention python3 explicitly. The fixups for bytearray vs
string issues will follow up.
