The new build pipeline compresses images already but uploaded both the
compressed and uncompressed files because the whole build folder gets
uploaded.
Add a new flag "--only_store_compressed" to the image generation which
deletes the uncompressed file after compression is done. Uncompressed
images are still supported if specified in the flag
"image_compression_formats".
Closes https://github.com/flatcar-linux/Flatcar/issues/793
Rename sztd to zst and amend the changelog. The zstd binary generates a
compressed file with the .zst extension by default.
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
This change adds a new flag called --image_compression_format which
allows us to output the final VM image as one of the supported formats:
bz2 (default), gz, zip or none
if the compression format is "none" or "", the image will not be compressed.
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
It has some weird semantics that seem to trip us up after updating
bash to 5.1. We tried to use it inside functions to clean up some
stuff after function returns. This can be emulated with an EXIT trap
within a subshell. Fortunately all the users of the RETURN trap were
not setting any global variables - modifications of such variables are
local to the subshell and are lost when the subshell exits.
Torcx is special in that it wishes to be uploaded under a prefixed
directory (torcx), typically wishes to be downloaded from there, but
ultimately wants to be downloaded from a location without that prefix.
In fact, I expect during a normal release process, it will be uploaded
with that prefix to the build bucket, copied without that prefix to the
final bucket (during pre-release), and then finally downloaded without
the prefix.
I think this set of variables ends up being the cleanest way to
represent this complexity.
This variable was semi-deprecated ages ago so `version.txt` could follow
a similar variable naming pattern to `os-release`. Finally drop usage of
it here in favor of `$COREOS_VERSION`.
SDK tarballs have a .DIGESTS file but it is created by catalyst instead
of the upload_image function. In order to support plain GPG signing but
not avoid re-generating .DIGESTS we need to move that code out of
upload_image to a new function. upload_files shouldn't do it itself
because it is also used for portage binary packages which shouldn't be
signed (there is no point, nothing would verify the signatures).
I started to move board files under a boards/ directory similar to how
the SDK is under sdk/ but didn't do so everywhere. This should finish
the job so everything is consistent now.
Note: This prefix is only used in developer and buildbot uploads. When
final releases are copied to $channel.release.core-os.net it doesn't use
the prefix since a) I already published urls without the prefix and b)
no sdk files are ever posted to the public release locations.
Since the new bucket scheme uploads images to a private staging area
first we need to configure the final location to generate vagrant json
metadata correctly.
- Automated builds drop SDK and binary packages into
gs://builds.developer.core-os.net/ and the new download URL is
http://builds.developer.core-os.net/ (COREOS_DEV_BUILDS)
- Change default upload path to gs://users.developer.core-os.net/ for
misc developer builds. Official builds go elsewhere and will just be
configured in buildbot/jenkins so some COREOS_OFFICIAL stuff is gone.
- Automated builds of images go to a private bucket,
gs://builds.release.core-os.net which later gets copied to
gs://alpha.release.core-os.net and friends by core_promote.
The .DIGESTS format is clunky and annoying. It also requires uses to
perform two steps to verify images using GPG. Instead support signing
all files directly so there is no need for .DIGESTS.
The old DIGESTS code will remain in place for now but after a few
releases I plan on deleting it.
While attempting to fix the easy to mix up DIGESTS names in feb59db9f I
stumbled across yet another way that the DIGESTS names were a bit
unpredictable: previously a .bz2 got stuck into the file name when
upload_images automatically compressed some file types. The new code
missed this and never added the .bz2. Correct this now, both for
image_to_vm which has a pile of glue to keep the legacy names in place
for now and build_image which I never intended to change.
Make it possible for other scripts to share the same value for our
release repository and equally easy to override with a custom value.
Also allow setting the root from the command line in addition to the
environment. Usually --upload_root is better to use than --upload_path.
For multi-file uploads we should explicitly declare what the name of the
.DIGESTS file should be instead of using the first file name. Relying on
the ordering was subtle and easy to break.
Sync up bootstrap_sdk with other tools by using the common upload
functions. As part of this refactor release_util a bit to provide a
truly generic upload function.
This will be used to upload the latest images built from master, we
don't need every build so we just want to upload to a 'master'
directory, not one named for the current version.
