armbian_build/lib/functions/artifacts/artifacts-reversion.sh
Ricardo Pardini 2142f73b97 hashed-OCI-revisioned-debs: introduce "reversioning" of .deb packages
> tl-dr:
> - maximize OCI cache hit ratio across nightlies/releases/PRs/etc;
> - publish simple `Version:`'s that don't include a crazy hash in repo and images
> - introduce `output/packages-hashed` directory
> - radically change the `output/debs` directory structure

- simplify artifact's `prepare_version()` method for `deb` and `deb-tar` artifacts:
  - `artifact_base_dir` and `artifact_final_file` will now be auto-calculated; thus removed from each artifact (except `rootfs`)
  - `artifact_deb_repo` ("global", "jammy", "bookworm") is now required; "global" means common across all RELEASES
  - `artifact_deb_arch` is now required, "all" is arch-independent, otherwise use `${ARCH}`
  - `artifact_map_debs` is now auto-calculated based on the above, and shouldn't be specified manually
  - `artifact_final_version_reversioned` is optional, and can force the final version of the artifact (specific for the `base-files` case)
  - artifacts that need special handling for reversioning can add function names to `artifact_debs_reversion_functions` array (`base-files` and `bsp-cli` cases)
  - artifacts `prepare_version()` should set `artifact_version`, but _never_ include it in other variables; `artifact_version` is now changed by framework after `prepare_version()` returns
- no longer use/refer/mention `${REVISION}` when building packages. All packages should be `${REVISION}`-agnostic.
- `${REVISION}` (actually, `artifact_final_version_reversioned`) will be automatically swapped in the `control` file during reversioning
- `fakeroot_dpkg_deb_build()` now takes exactly two arguments: the directory to pack, and the deb ID (key of `artifact_map_packages` dict); add this change in all the artifact's code for this
- `obtain_complete_artifact()`:
  - automatically adds `-Rxxxx` "revisioning-hash" to `artifact_version`, by hashing the revisioning functions and any `artifact_debs_reversion_functions` set
  - calculates more complex subdirectory paths for both the `output/packages-hashed` and `output/debs`/`output/debs-beta` directories
    - with the new subdirectories we can be sure a re-version is already done correctly and can skip it (eg, for partial `download-debs` re-runs)
    - in the future we can automatically clean/remove old versions that are no longer relevant based on the dir structure
    - exports a lot more information to JSON, including the new subdirectory paths
  - comment-out code that implemented `skip_unpack_if_found_in_caches`, I'm very unsure why we had this in the first place
- `obtain_artifact_from_remote_cache()`
  - for `deb` type artifacts, OCI won't preserve the subdirectory structure, so move downloaded files to the correct subdirectory manually
  - this is not needed for `deb-tar`, since that can preserve the dir structure itself
- introduce `artifacts-reversion.sh` and its main function `artifact_reversion_for_deployment()`
  - this has the logic for reversioning .deb's, by `ar`-unpacking them, changing `control.tar` (and possibly `data.tar`), handling `.xz` compression, etc.
  - also handles hashing those functions, for consistency. Any changes in reversioning code actually change the artifact itself so we're not caught by surprise
  - by default, it changes `control` file only:
    - replace `Version:` (which is the hash-version originally) with `artifact_final_version_reversioned` (which is mostly just `${REVISION}`)
    - add a custom field `Armbian-Original-Hash:` with the original hash-version
  - `artifact_reversion_for_deployment()` is called by
    - new CLI wrapper `cli_obtain_complete_artifact()`, used for CLI building of specific artifact, but also for `download-artifact`
    - `build_artifact_for_image()` used during image build
- `armbian-bsp-cli-deb.sh`: move `${REVISION}` related stuff from the main package build to new reversioning functions.
- `artifact-armbian-base-files.sh`: move `${REVISION}` related stuff from the main package build to new reversioning functions.
- `kernel`:
  - add some custom fields to `DEBIAN/control`:
    - `Armbian-Kernel-Version:` / `Armbian-Kernel-Version-Family:` (for future use: cleanup of usage of `Source: ` field which should be removed)
  - declutter the `Description:` field, moving long description out of the first line
  - obtain `IMAGE_INSTALLED_KERNEL_VERSION` from the reversioned deb (this is still a hack and has not been fixed)
- `uboot`:
  - declutter the `Description:` field, moving long description out of the first line
  - use the reversioned .deb when deploying u-boot to the image
- `main_default_build_packages()` now stores reversioned values and complete paths to reversioned .deb's
- `list_installed_packages()` now compares custom field `Armbian-Original-Hash: `, and not the `Version:` to make sure debs in the image are the ones we want
- `install_artifact_deb_chroot()` is a new wrapper around `install_deb_chroot()` for easy handling of reversioned debs
  - use it everywhere `install_deb_chroot()` was used in `distro-agnostic.sh` and `distro-specific.sh`
2023-08-12 09:58:32 +02:00

157 lines
6.5 KiB
Bash

#
# SPDX-License-Identifier: GPL-2.0
# Copyright (c) 2023 Ricardo Pardini <ricardo@pardini.net>
# This file is a part of the Armbian Build Framework https://github.com/armbian/build/
#
function artifact_reversion_for_deployment() {
standard_artifact_reversion_for_deployment "${artifact_debs_reversion_functions[@]}"
}
function artifact_calculate_reversioning_hash() {
declare hash_functions="undetermined"
declare -a all_functions=("standard_artifact_reversion_for_deployment" "standard_artifact_reversion_for_deployment_one_deb")
all_functions+=("artifact_deb_reversion_unpack_data_deb" "artifact_deb_reversion_repack_data_deb")
all_functions+=("${artifact_debs_reversion_functions[@]}")
calculate_hash_for_function_bodies "${all_functions[@]}" # sets hash_functions
artifact_reversioning_hash="${hash_functions}" # outer scope
return 0
}
function standard_artifact_reversion_for_deployment() {
display_alert "Reversioning package" "re-version '${artifact_name}(${artifact_type})::${artifact_version}' to '${artifact_final_version_reversioned}'" "info"
declare artifact_mapped_deb one_artifact_deb_package
for one_artifact_deb_package in "${!artifact_map_packages[@]}"; do
declare artifact_mapped_deb="${artifact_map_debs["${one_artifact_deb_package}"]}"
declare hashed_storage_deb_full_path="${PACKAGES_HASHED_STORAGE}/${artifact_mapped_deb}"
if [[ ! -f "${hashed_storage_deb_full_path}" ]]; then
exit_with_error "hashed storage does not have ${hashed_storage_deb_full_path}"
fi
display_alert "Found hashed storage file" "'${artifact_mapped_deb}': ${hashed_storage_deb_full_path}" "debug"
# find the target dir and full path to the reversioned file
declare deb_versioned_rel_path="${artifact_map_debs_reversioned["${one_artifact_deb_package}"]}"
declare deb_versioned_full_path="${DEB_STORAGE}/${deb_versioned_rel_path}"
declare deb_versioned_dirname
deb_versioned_dirname="$(dirname "${deb_versioned_full_path}")"
run_host_command_logged mkdir -p "${deb_versioned_dirname}"
# since the full versioned path includes the original hash, if the file already exists, we can trust
# it's the correct one, and skip reversioning.
if [[ -f "${deb_versioned_full_path}" ]]; then
display_alert "Skipping reversioning" "deb: ${deb_versioned_full_path} already exists" "debug"
continue
fi
# call function for each deb, pass parameters
standard_artifact_reversion_for_deployment_one_deb "${@}"
# make sure reversioning produced the expected file
if [[ ! -f "${deb_versioned_full_path}" ]]; then
exit_with_error "reversioning did not produce the expected file: ${deb_versioned_full_path}"
fi
done
return 0
}
function standard_artifact_reversion_for_deployment_one_deb() {
display_alert "Will repack" "one_artifact_deb_package: ${one_artifact_deb_package}" "debug"
display_alert "Will repack" "hashed_storage_deb_full_path: ${hashed_storage_deb_full_path}" "debug"
display_alert "Will repack" "deb_versioned_full_path: ${deb_versioned_full_path}" "debug"
display_alert "Will repack" "artifact_version: ${artifact_version}" "debug"
declare cleanup_id="" unpack_dir=""
prepare_temp_dir_in_workdir_and_schedule_cleanup "reversion-${artifact_name}" cleanup_id unpack_dir # namerefs
declare deb_contents_dir="${unpack_dir}/deb-contents"
mkdir -p "${deb_contents_dir}"
# unpack the hashed_storage_deb_full_path .deb, which is just an "ar" file, to the deb_contents_dir
run_host_command_logged ar x "${hashed_storage_deb_full_path}" --output="${deb_contents_dir}"
# find out if compressed or not, and store for future recompressing
control_compressed=""
if [[ -f "${deb_contents_dir}/control.tar.xz" ]]; then
control_compressed=".xz"
run_host_command_logged xz -d "${deb_contents_dir}/control.tar.xz" # decompress
fi
# untar the control into its own specific dir
declare control_dir="${unpack_dir}/control"
mkdir -p "${control_dir}"
run_host_command_logged tar -xf "${deb_contents_dir}/control.tar" --directory="${control_dir}"
# prepare for unpacking the data tarball as well
declare data_dir="${unpack_dir}/data"
mkdir -p "${data_dir}"
declare data_compressed=""
if [[ -f "${deb_contents_dir}/data.tar.xz" ]]; then
data_compressed=".xz"
fi
# Hack at the control file...
declare control_file="${control_dir}/control"
declare control_file_new="${control_dir}/control.new"
# Replace "Version: " field with our own
sed -e "s/^Version: .*/Version: ${artifact_final_version_reversioned}/" "${control_file}" > "${control_file_new}"
echo "Armbian-Original-Hash: ${artifact_version}" >> "${control_file_new}" # non-standard field.
for one_reversion_function_name in "${@}"; do
display_alert "reversioning" "call custom function: '${one_reversion_function_name}'" "debug"
"${one_reversion_function_name}" "${one_artifact_deb_package}"
done
# Show a nice diff using batcat if debugging
if [[ "${SHOW_DEBUG}" == "yes" ]]; then
diff -u "${control_file}" "${control_file_new}" > "${unpack_dir}/control.diff" || true
run_tool_batcat "${unpack_dir}/control.diff"
fi
# Move new control on top of old
run_host_command_logged mv "${control_file_new}" "${control_file}"
run_host_command_logged rm "${deb_contents_dir}/control.tar"
cd "${control_dir}" || exit_with_error "cray-cray about control_dir ${control_dir}"
run_host_command_logged tar cf "${deb_contents_dir}/control.tar" .
# if it was compressed to begin with, recompress...
if [[ "${control_compressed}" == ".xz" ]]; then
run_host_command_logged xz "${deb_contents_dir}/control.tar"
fi
# re-ar the whole .deb back in place, using the new version for filename.
run_host_command_logged ar rcs "${deb_versioned_full_path}" \
"${deb_contents_dir}/debian-binary" \
"${deb_contents_dir}/control.tar${control_compressed}" \
"${deb_contents_dir}/data.tar${data_compressed}"
done_with_temp_dir "${cleanup_id}" # changes cwd to "${SRC}" and fires the cleanup function early
return 0
}
function artifact_deb_reversion_unpack_data_deb() {
if [[ "${data_compressed}" == ".xz" ]]; then
run_host_command_logged xz -d "${deb_contents_dir}/data.tar.xz" # decompress
fi
run_host_command_logged tar -xf "${deb_contents_dir}/data.tar" --directory="${data_dir}"
}
function artifact_deb_reversion_repack_data_deb() {
run_host_command_logged rm "${deb_contents_dir}/data.tar"
cd "${data_dir}" || exit_with_error "cray-cray about data_dir ${data_dir}"
run_host_command_logged tar cf "${deb_contents_dir}/data.tar" .
# if it was compressed to begin with, recompress...
if [[ "${data_compressed}" == ".xz" ]]; then
run_host_command_logged xz "${deb_contents_dir}/data.tar"
fi
}