* Refactor armbian-config into install as external package
- https://github.armbian.com/configng repository is always present
- package can still be removed from the install list
Since this tool will have deeper integration with Armbian, we should probably drop possibility to remove the tool from the list.
* Enable armbian-config as extension
- incl some indicative fdt/extlinux/gpt info
- include full config and defconfig for each target
- mark the old .config file as legacy (it was always the last-target's config)
- fix: savedefconfig after build, not before (as target_map might change it)
- fix: some very old u-boots (2011, odroidc1) do not have `savedefconfig` at all, so make it optional
- fix: same for `.config` -- very old u-boots worked different (`boards.cfg`?)
- Fixes: 71296430fa9e849098baa5da2cb0734580b12be9 -- which only set `SKIP_ARMBIAN_REPO=yes` when `rootfs` cli shortcut used
- GHA artifact pipelines are generic across all artifacts, and don't use the cli shortcuts
- instead, they use the `artifact` shortcut, with the `WHAT=rootfs` parameter
- so in GHA pipelines `SKIP_ARMBIAN_REPO=yes` was not taking effect
- rootfs artifact itself now enforces `SKIP_ARMBIAN_REPO` to be `yes` and makes it readonly
- also add `SKIP_ARMBIAN_REPO` to the general change-tracking for more clarity about configs and hooks etc
- add change tracking calls before/after generic artifact config adapter for even more clarity
- both artifacts use functions in the bsp/utils-bsp.sh file, which wasn't being hashed
- bsp-cli hashed non-existent 'packages/bsp-cli' directory, remove
- bsp-cli: be verbose when rsync'ing `packages/bsp/common`
- force sync in a few strategic spots
- utils-bsp.sh: be verbose when copying files (incl `config/optional` stuff)
* Re-add functionality to create linux-libc-dev when building kernel packages.
* linux-libc-dev: append -${BRANCH}-${LINUXFAMILY} to package name and update control file
- This way we can safely include BOOT_SOC in /etc/armbian-release as done in the previous commit
- this will cause a rebuild of _all_ bsp-cli packages, even those for boards that don't define BOOT_SOC
- hopefully one day we will have BOOT_SOC ("What is the SoC?") defined for all boards
- for now, it will be empty for those that don't define it, which should be harmless
- example: `./compile.sh BOARD=xxxxx BRANCH=edge kernel-dtb`
- outputs preprocessed DTS source for board in question to `output/`
- also outputs same preprocessed DTS source, ran through `dtc` with input and output DTS formats for "normalized" comparisions
* meson-s4t7: bump u-boot to khadas-vims-u-boot-2019.01-v1.6-release
* Use khadas default bootargs as much as possible
* Add new hook to allow copying code into kernel
* meson-s4t7: legacy: Switch to 5.15 kernel
* meson-s4t7: add kernel-config for 5.15 kernel
* device tree overlays for 5.15 kernel for vim1s and vim4
* restructure packaging of bsp files for vim1s/vim4
* silence vblank warning on boot
* Remove display workaround as it doesn't work with 5.15 kernel
* Remove 5.4 kernel patches
* Set dpkg vendor to Armbian in all images
We already set to Armbian, but we didn't set default link. This fixes it for both, Debian and Ubuntu.
* Adjust post install scripts to set correct link on upgrade
- also, for extension style hook `post_uboot_custom_postprocess`, don't do it 3 times, just once
- this commit will cause rebuild of all u-boots
- and that's a good thing, many custom changes in family code (eg ddr change in rk322x) were being ignored
- thanks @paolosabatino
- OCI tags can't have "+" or "~" so replace those with "--" before using in OCI tag
- apt (repo) version will have whatever upstream has, including "+" and/or "~"
- `kernel-patches-to-git` wasn't providing the needed `KERNEL_GIT_SHA1` for kernel drivers
- refactor `obtain_kernel_git_info_and_makefile()` out of `artifact_kernel_prepare_version()` so we can reuse
- introduce `rewrite-kernel-patches`, which is just an alias to `kernel-patches-to-git` with `REWRITE_PATCHES=yes`
> tl-dr: only deploys to remote OCI if `UPLOAD_TO_OCI_ONLY=yes`; stop leaving junk behind in local cache in many situations
- simplify CLI artifact building parameters and behaviour
- `ARTIFACT_USE_CACHE` is now deprecated, and its behaviour is the default
- for _any_ uploading to OCI to occur, `UPLOAD_TO_OCI_ONLY=yes` **must** be present; in this case, reversioning is not done
- `FORCE_ARTIFACTS_DOWNLOAD` is completely removed (use `download-artifact` instead)
- `cli_obtain_complete_artifact()`'s and `build_artifact_for_image()`'s reversioning is now moved to common `obtain_complete_artifact()`
- `standard_artifact_reversion_for_deployment()`:
- check for hashed deb existence only if reversioned does not exist
- touch the reversioned file if it already exists; helps to clean up junk later
- delete hashed version after reversioning, so we don't leave trash behind
- unless in `download-artifact` mode, which `touch`es the hashed version
- we can later delete old files from packages-hashed to keep junk under control
- refactor `obtain_complete_artifact()`
- extract function `artifact_dump_json_info()` since obtain is large enough already
- when deploying to remote, always ignore all local and remote caches
- introduce `artifact_is_available_in_revisioned_local_cache()`
- if not deploying to remote, and revisioned cache exists, use it directly
- if deploying to remote, reversioned is not checked and not created
- if deploying to remote, force `DEB_COMPRESS=xz`
- if deploying to remote, completely remove the local cache base dir after upload is done (no more junk leftover)
> tl-dr:
> - maximize OCI cache hit ratio across nightlies/releases/PRs/etc;
> - publish simple `Version:`'s that don't include a crazy hash in repo and images
> - introduce `output/packages-hashed` directory
> - radically change the `output/debs` directory structure
- simplify artifact's `prepare_version()` method for `deb` and `deb-tar` artifacts:
- `artifact_base_dir` and `artifact_final_file` will now be auto-calculated; thus removed from each artifact (except `rootfs`)
- `artifact_deb_repo` ("global", "jammy", "bookworm") is now required; "global" means common across all RELEASES
- `artifact_deb_arch` is now required, "all" is arch-independent, otherwise use `${ARCH}`
- `artifact_map_debs` is now auto-calculated based on the above, and shouldn't be specified manually
- `artifact_final_version_reversioned` is optional, and can force the final version of the artifact (specific for the `base-files` case)
- artifacts that need special handling for reversioning can add function names to `artifact_debs_reversion_functions` array (`base-files` and `bsp-cli` cases)
- artifacts `prepare_version()` should set `artifact_version`, but _never_ include it in other variables; `artifact_version` is now changed by framework after `prepare_version()` returns
- no longer use/refer/mention `${REVISION}` when building packages. All packages should be `${REVISION}`-agnostic.
- `${REVISION}` (actually, `artifact_final_version_reversioned`) will be automatically swapped in the `control` file during reversioning
- `fakeroot_dpkg_deb_build()` now takes exactly two arguments: the directory to pack, and the deb ID (key of `artifact_map_packages` dict); add this change in all the artifact's code for this
- `obtain_complete_artifact()`:
- automatically adds `-Rxxxx` "revisioning-hash" to `artifact_version`, by hashing the revisioning functions and any `artifact_debs_reversion_functions` set
- calculates more complex subdirectory paths for both the `output/packages-hashed` and `output/debs`/`output/debs-beta` directories
- with the new subdirectories we can be sure a re-version is already done correctly and can skip it (eg, for partial `download-debs` re-runs)
- in the future we can automatically clean/remove old versions that are no longer relevant based on the dir structure
- exports a lot more information to JSON, including the new subdirectory paths
- comment-out code that implemented `skip_unpack_if_found_in_caches`, I'm very unsure why we had this in the first place
- `obtain_artifact_from_remote_cache()`
- for `deb` type artifacts, OCI won't preserve the subdirectory structure, so move downloaded files to the correct subdirectory manually
- this is not needed for `deb-tar`, since that can preserve the dir structure itself
- introduce `artifacts-reversion.sh` and its main function `artifact_reversion_for_deployment()`
- this has the logic for reversioning .deb's, by `ar`-unpacking them, changing `control.tar` (and possibly `data.tar`), handling `.xz` compression, etc.
- also handles hashing those functions, for consistency. Any changes in reversioning code actually change the artifact itself so we're not caught by surprise
- by default, it changes `control` file only:
- replace `Version:` (which is the hash-version originally) with `artifact_final_version_reversioned` (which is mostly just `${REVISION}`)
- add a custom field `Armbian-Original-Hash:` with the original hash-version
- `artifact_reversion_for_deployment()` is called by
- new CLI wrapper `cli_obtain_complete_artifact()`, used for CLI building of specific artifact, but also for `download-artifact`
- `build_artifact_for_image()` used during image build
- `armbian-bsp-cli-deb.sh`: move `${REVISION}` related stuff from the main package build to new reversioning functions.
- `artifact-armbian-base-files.sh`: move `${REVISION}` related stuff from the main package build to new reversioning functions.
- `kernel`:
- add some custom fields to `DEBIAN/control`:
- `Armbian-Kernel-Version:` / `Armbian-Kernel-Version-Family:` (for future use: cleanup of usage of `Source: ` field which should be removed)
- declutter the `Description:` field, moving long description out of the first line
- obtain `IMAGE_INSTALLED_KERNEL_VERSION` from the reversioned deb (this is still a hack and has not been fixed)
- `uboot`:
- declutter the `Description:` field, moving long description out of the first line
- use the reversioned .deb when deploying u-boot to the image
- `main_default_build_packages()` now stores reversioned values and complete paths to reversioned .deb's
- `list_installed_packages()` now compares custom field `Armbian-Original-Hash: `, and not the `Version:` to make sure debs in the image are the ones we want
- `install_artifact_deb_chroot()` is a new wrapper around `install_deb_chroot()` for easy handling of reversioned debs
- use it everywhere `install_deb_chroot()` was used in `distro-agnostic.sh` and `distro-specific.sh`
- this should avoid (late) patching errors that might happen during a point release bump like `6.4.5` -> `6.4.6` cos we'd be using the wrong cached drivers patch
- using the SHA1 will instead (possibly) trigger the "real patching failure", during drivers-harness when building a new driver patch cache
- also try to cleanup old caches in the old format so we've not many leftovers -- each patch is ~150mb
