armbian_build/lib/functions/compilation/utils-compilation.sh
Ricardo Pardini b25b3cf499
armbian-next: lib changes - MEGASQUASH - squashed changes from c9cf3fc241cfb4c872f4aef7bbc41d5854db7ea3 to 6809de3d6063cb041205a8318e19da6a4dee68c9 ref extensions_08_10_2022_pre_v30
- also compile.sh
- shellfmt lib
- split off shell and python tools under lib
- revert removal of stuff a-n no longer uses (ref. compilation): general packaging, mkdeb etc
- editoconfig split off
- extension changes split off
- sources and sources/families split off
- some undue stuff removed or split

armbian-next: manual merge (30) of lib changes between 882f995e21 and 31ac6383e1

armbian-next: manual merge (30) of family/board changes between 882f995e21 and 31ac6383e1

armbian-next: manual merge (29) of family/board changes between 3435c46367 and 882f995e21 (A LOT!)

armbian-next: manual merge (29) of lib changes between 3435c46367 and 882f995e21 (A LOT!)

armbian-next: manual merge (28) of lib changes between revisions af6ceee6c5 and 38df56fbf3

armbian-next: manual merge (28) of sources/families changes between revisions af6ceee6c5 and 38df56fbf3

armbian-next: manual merge (27) of `lib` changes between revisions 9c52562176 and af6ceee6c5

armbian-next: manual merge (27) of `sources/families` changes between revisions 9c52562176 and af6ceee6c5

armbian-next: move `ROOTFSCACHE_VERSION` resolution from GitHub from `main-config` down to `create-cache`

- this way config does not depend on remote...

armbian-next: move `ARMBIAN_MIRROR` selection (network) from `main-config` to `prepare-host`

- this way CONFIG_DEFS_ONLY can run without touching the network

armbian-next: manual merge (26) of MD5-checking via debsums (3955) re-imagined

- @TODO make sure

armbian-next: manual merge (26) of sources/families changes between revisions 20ee8c5450 and 9c52562176

armbian-next: manual merge (26) of lib changes between revisions 20ee8c5450 and 9c52562176

- @TODO NOT including the md5/debsums check, that needs further rewrite

armbian-next: manual merge (25) of lib changes between revisions fe972621c6 and 20ee8c5450

- @TODO hmm Igor is now going out to the network for rootfs cache version during configuration phase!!! BAD BAD BAD

armbian-next: manual merge (25) of family changes between revisions fe972621c6 and 20ee8c5450

armbian-next: manual merge (24) of families changes between revisions 9ca9120420 and 560531a635

armbian-next: manual merge (24) of lib changes between revisions 9ca9120420 and 560531a635

armbian-next: manual merge (23) of all changes between revisions 17b4fb913c and 9ca9120420

armbian-next: manual merge (22) of all changes between revisions 0eb8fe7497 and 1dddf78cd0

- @TODO EXCEPT the insanity about locales/eval/VERYSILENT in #3850, requires deep review

armbian-next: manual merge (21) of all changes between revisions e7d7dab1bb and 0eb8fe7497

armbian-next: fix: patching CREATE_PATCHES=yes

- needed to create output dir

armbian-next: add `python2-dev` dep for old uboots

- cleanup some comments

armbian-next: manual merge (20) of all changes between revisions 6b72ae3c86 and 247c4c45fd

armbian-next: fix: pass `TERM` to kernel's make, so `make menuconfig` can work

armbian-next: fix: git: read commit UNIX timestamp/local date correctly

- `checked_out_revision_ts` was correct; git outputs `%ct` as a UNIX timestamp, UTC-based
- `checked_out_revision_mtime` was incorrect: git output it without converting to local time
- manually convert using `date @xx` so it has correct local time, whatever it is.
- add debugging to `get_file_modification_time()` too

armbian-next: abstract `$QEMU_BINARY` to `qemu-static.sh`: `deploy_qemu_binary_to_chroot()`/`undeploy_qemu_binary_from_chroot()`

- add hackish logic to avoid removing binary that would be needed if image actually contains `qemu-user-static` package

armbian-next: fix `uuidgen` basic dep check; use fake bash `$RANDOM` if uuidgen not available

- not good: we need uuidgen to begin logging, but it may not be installed yet. workaround.

armbian-next: retry 3 times download-only also for `PACKAGE_LIST_BOARD`

- acng is really not helping

armbian-next: allow customizing UBUNTU_MIRROR (ports mirror) with `CUSTOM_UBUNTU_MIRROR_ARM64=host/path`

armbian-next: WiP: kernel make via `env -i` for clean env; show produced /boot tree

armbian-next: manual merge (19) of all changes between revisions b23498b949 and e621d25adc

- the ssh firstrun revert stuff mostly

armbian-next: *breaking change* remove `LIB_TAG` and `.ignore_changes` completely

- one day should be replaced with an "update checker" extension, or even "update-enforcer"
- for now this just causes chaos

armbian-next: `python2` is required for some u-boot builds

- would be "use `python-is-python2` so /usr/bin/python exists and points to Python 2.x" but Jammy does not have that anymore
- python2 is required for some u-boot builds.
- that said, python 2.x is deprecated for a while and needs work thus @TODO

armbian-next: bump Python info gatherer to RELEASE=jammy too

armbian-next: add `KERNEL_MAJOR_MINOR` info to `media` kernel (@balbes150)

- 5.18 is not yet released so might be a problem here

armbian-next: allow to skip submodules during `fetch_from_repo`; introduce hook `fetch_custom_uboot`

- via GIT_SKIP_SUBMODULES=yes, which disables all submodules everywhere
- via UBOOT_GIT_SKIP_SUBMODULES=yes, which disables fetching of submodules during uboot fetch (hidden rkbins anyone?)
- extension hook `fetch_custom_uboot` so we can fetch our own stuff if needed

armbian-next: `initrd` caching fixes (always enable hook; if cache hit, convert to uImage too)

armbian-next: introduce `initramfs`/`initrd` caching

- using hashes of (hopefully) all involved files
- cache hits are rewarded with sprinkly hearts.
  - why? this proves we got a reproducible kernel modules build!
  - also, you just saved yourself 2-10 minutes of pain

armbian-next: manual merge (18) of changes between revisions 08cf31de73 and c8855aa08d

- heh; most bash code changes are for things already done in -next, or no longer used
- some version bumps, etc

armbian-next: cleanup entrypoint and shuffle `prepare_host_basic()` into logging section

armbian-next: *breaking change* add global extlinux killswitch `ALLOW_EXTLINUX`

- unless you set `ALLOW_EXTLINUX=yes`, then `SRC_EXTLINUX` will be disabled globally.
- add a bunch of logging regarding extlinux, armbianEnv and bootscripts for clarity during build
- this is due to nand-sata-install problems with extlinux
- some boards _only work_ with extlinux; we'll have to handle it later

armbian-next: extensions: `image-output-{qcow2|ovf}`: virtual output formats

- which use `qemu-utils` for `qemu-img` conversion of the .img

armbian-next: extension: `kernel-localmodconfig`: faster/slimmer kernel builds with `make localmodconfig`

armbian-next: extension: `cleanup-space-final-image`: zerofree, slim down firmware, show used space

armbian-next: introduce `do_with_ccache_statistics` and use it for kernel compile

- some TODOs
- better logging for .config copying

armbian-next: *breaking change* really disable apt sources for non-desktop builds

armbian-next: fix: don't manage apt-cacher-ng if told NOT to, not the other way around

armbian-next: `JUST_UBOOT=yes` + hooks `build_custom_uboot()`/`post_write_uboot_platform()`

- post_write_uboot_platform()
  - only runs during build, for now (not on device)
- build_custom_uboot()
  - allow fully custom, extension driven, building of u-boot
  - also partial preparation of uboot source combined with default Armbian build
- HACK: u-boot: downgrade some errors to warnings via KCFLAGS
- fix copy of atf bins to uboot, don't do it if atf's not there

armbian-next: fix: no use testing the host for resolvconf if we're manipulating the SDCARD

armbian-next: sunxi_common: avoid shortcircuit error on family_tweaks_bsp when family_tweaks_bsp_s is not defined

armbian-next: fix: add `zstd` and `parallel` to hostdeps

armbian-next: manual merge (17) of all changes between revisions 64410fb74b and 08cf31de73

- changes about `git safe dir` ignored, I've done the same in a different way
- hash calculation changes ignored, fasthash is completely different

armbian-next: add `crossbuild-essential-armel` so `arm-linux-gnueabi-gcc` is available with system toolchains

- need to for some ATF builds, at least.

armbian-next: rockchip64_common: lotsa logging and debugging

- supposedly no practical changes

armbian-next: grub: better logging

armbian-next: fix for chaos caused by git's fix of CVE-2022-24765 otherwise "fatal: unsafe repository"

- might not be the best solution, but it's the only one I found

partitioning: fix: don't try fixing a bootscript that's not there

- this fixes a bug when "rootpart=2" without rootpart 1 being /boot

armbian-next: cleanups: umount tmpfs-based $SDCARD during cleanup too

armbian-next: indented heredoc, no functional changes

armbian-next: fix shortcircuit as last statement in case of extlinux

- yes, I wasted 3 hours on this tiny bit, so *you* don't have to!
- better logging for rootfs `mkfs` et al
- introduce `PRESERVE_SDCARD_MOUNT=yes` to preserve SDCARD, MOUNT, and LOOP for debugging

armbian-next: kernel-headers: less verbose, trimmed down tools a bit (perf and testing)

khadas-vim3l: add asound.state for Khadas VIM3L

armbian-next: introduce hook `extension_finish_config()` - late hook for ext configuration

- `extension_finish_config()` is the last thing done in config phase
    - use it for determining stuff based on kernel version details, package names, etc
- also tune down some logging which was too verbose
- CI logs with no ANSI escape codes

armbian-next: shuffle around code and logic of `add_desktop_package_sources()`

- @TODO: still needs proper asset logging for sources.list(.d)
- @TODO: tunes down adding of sources/packages to CLI builds, check with Igor

armbian-next: 4.x can't build objtool in kernel-headers; allow for handling that later

- 4.x has a lot more obtuse dependencies
- introduce KERNEL_HAS_WORKING_HEADERS calculated based on KERNEL_MAJOR_MINOR

armbian-next: downgrade `error=misleading-indentation` to warning

- some 4.x kernels patches are really messy
- newer gcc's make that an error now

armbian-next: *allow cross compilation*, even the so-called "reverse cross-compile" (amd64 on arm64)

armbian-next: add `zfs` extension, which installs headers and builds ZFS via DKMS in chroot

- similar to how `nvidia` extension does it

armbian-next: x86: enable `nvidia` extension for all releases (only desktop)

armbian-next: `headers-debian-byteshift.patch` is dead; long-live cross-compiled source-only kernel-headers

- kernel-headers package now only includes _sources_
- postinst does the compilation and preparation for DKMS compatibility
- `tools` dir is included now, which includes the byteshift utilities
- handle special scripts/module.lds case after 5.10
- tested on a 6 combinations of `x86` / `arm64` / `armhf` (3x targets, 2x hosts)
- @TODO: we might be able to reduce the size of tools a bit (perf/tests/etc)
- @TODO: still missing ARCH vs ARCHITECTURE vs SRC_ARCH clarity elsewhere

armbian-next: allow `use_clean_environment=yes` for `chroot_sdcard_apt_get()` and descendants

- this causes command to be run under `env -i`, for a clean environment

armbian-next: manual merge (16) of all changes between revisions be9b5156a4 and 2a8e1ecac1

- many `traps` ignored: we don't use them anymore

armbian-next: fix logging for apt sources/gpg keys

armbian-next: don't leak `if_error_xxx` vars across runner helper invocations; always clean then (even if no error)

- also: fix wireguard-tools install, had a double parameter there

bcm2711: rpi4b: add `pi-bluetooth` which provides working Bluetooth

armbian-next: fixes for (non-)logging when interactively configuring kernel (`KERNEL_CONFIGURE=yes`)

armbian-next: move `lz4` rootfs caches to `zstd`, multithreaded

armbian-next: customize.sh: error handling, do not mount overlay if it doesn't exist

armbian-next: extra info for runners; `if_error_detail_message` and `if_error_find_files_sdcard` globals

- those are unset after running any command
- if error occur, message and/or found files will be included in log, for clarity

armbian-next: manual merge (15) of all changes between revisions 0f7200c793 and 101eaec907

armbian-next: better logging for `rsync` calls everywhere

- make rsync verbose

armbian-next: downloads: skip download if no `ARMBIAN_MIRROR` nor `DOWNLOAD_MIRROR`; less logs

armbian-next: update rockchip.conf from master and use runners

armbian-next: update mvebu64.conf from master and use functions

armbian-next: git: fix `fetch_from_repo` with actual submodules usage

armbian-next: `armbian-next`ify the `nvidia` extension after rebase from master

- driver version is configurable via `NVIDIA_DRIVER_VERSION`
- use runner function to log/error-handle/use apt cache/etc

rpi4b: there's no legacy branch anymore, remove it from KERNEL_TARGET

armbian-next: `download_and_verify` non-error handled; logging is messy [WiP] [HACK]

armbian-next: logging: let ANSI colors pass to logfile; CALLER_PID instead of BASHPID in subshell

armbian-next: enable HTTPS CONNECT in Armbian-managed apt-cacher-ng configuration

- PPAs require it

armbian-next: don't loop forever if we can't obtain ARMBIAN_MIRROR from redirector

- also, don't even try to do it if `SKIP_ARMBIAN_REPO=yes`

armbian-next: manual merge (14) of all changes between revisions 13469fd8a9 and 09e416e31c

- also editorconfig and compile.sh (root) changes

armbian-next: *much* improved logging to HTML; log archiving; consistency

- keep only current logfile
- log to LOGFILE also if SHOW_LOG=yes
- log cmd runtime and success/error directly in runner

armbian-next: *breaking change* use `MemAvailable` (not including swap) and up requirements for tmpfs

- of course add debugging logs
- rename vars
- should really only use this if we've really a lot of completely free RAM to spare
- otherwise OOM killer comes knocking
- or swapping to disk, that is counter-productive

armbian-next: *breaking change* `DEB_COMPRESS=none` by default if not running in CI/GHA

armbian-next: *breaking change* `CLEAN_LEVEL=make` is no more; new `make-kernel`, `make-atf`, `make-uboot`

- allows individual control of what to clean
- this effectively disables `make clean` by default
- rebuilds work and timestamping works for patching, so no reason to clean everytime by default.

armbian-next: refactor `prepare_host`, give `apt-cacher-ng` some much needed attention

- library dir for host-related stuff, pull it out of "general" finally

armbian-next: hostdeps: all toolchains via `crossbuild-essential-arm64`/`armhf`/`amd64`

- trying to sort out hostdeps for Jammy [WiP]

armbian-next: remove `eatmydata` usage, leftover from failed tries to make git faster

armbian-next: fix git origin check, recreate working copy if origin does not match

- fix cold bundle https download progress reporting

armbian-next: finally consolidating logs into output/logs; colorized HTML logs

armbian-next: introduce `do_with_retries()` and use it for apt remote operations during image build

armbian-next: another round of logging tuning/fixes; log assets; git logging

- introduce `do_with_log_asset()` and `LOG_ASSET=xxx`
- separate "git" logging level
- add `trap_handler_cleanup_destimg()` to cleanup DESTIMG

armbian-next: kernel: use parallel compressors; reproducible kernel builds

- also remove leftover mkdebian/builddeb parameters in make invocation
- add pbzip2 to hostdeps

armbian-next: tuning logging for timestamp/fasthash related stuff which is very verbose

- idea is to not completely overwhelm `SHOW_DEBUG=yes` case
- make patching quieter and use file instead of stdin
- set checked_out_revision_ts during git checkout (timestamp version of _mtime)
- timestamp | fasthash logging level (via `SHOW_FASTHASH=yes`)

armbian-next: completely remove mkdebian/builddeb/general-packaging kernel packaging stuff

armbian-next: manual merge (12) of all changes between revisions 34d4be6b7b and 5fe0f36aa8

armbian-next: introduce `PRESERVE_WORKDIR=yes` for heavy debugging

armbian-next: packaging linux-headers again

- do NOT use any output from `make headers_install` - that's for libc headers
- grabs "headers" (and binary tools) directly from the kernel build tree, not install target
- does not produce headers if cross compiling, for now
- produces kernel-headers package for the architecture on which it was built
- doing a single make invocation with build and install for packaging
  - using 'make all' in place of vmlinuz/bzImage/image/zImage

armbian-next: apt download-only retried 3 times before installing main packages

armbian-next: fix `VER=` read from kernel-image package, also add `linux` default

armbian-next: some logging for atf compilation

armbian-next: rewrite hostdeps as array, add armhf toolchains

armbian-next: distro-agnostic: cleanups

armbian-next: armbianEnv hooks/debugs (bsp / image)

armbian-next: rpi: completely remove dtb hacks, allowing flash-kernel to work again

armbian-next: refactor new kernel packaging; add linux-dtb package back in finally, upgrades untested

armbian-next: refactor new kernel packaging; extract hook helper, fix kernel symlink

armbian-next: refactor new kernel packaging; add linux-dtb package back in finally, all hooks untested

flash-kernel: fix short-circuits as last statement in functions

armbian-next: do not force `SHOW_LOG=yes` if `CI=true`; let's _trust_ logging and error handling works

armbian-next: back out of setting mtime to the revision time during git checkout.

- of course this causes huge recompiles when wanted revision moves forward

armbian-next: sync 'config' dir from master revision ed589b248a

- this is _also_ getting out of hand... gotta merge soon

armbian-next: sync 'packages' dir from master revision ed589b248a

armbian-next: manual merge (11) of all lib/*.sh changes between revisions 3305d45b81 and ed589b248a

armbian-next: more refactorings, general logging; fixes; split image stuff

- logging flowing correct to LOGDIR, still needs packaging

armbian-next: complete removal of usages of `LOG_SUBPATH`; 100% error handled

- loose ends, use new LOGDIR
- remove the last shortcircuit in extensions execution, now it's 100% error handled
- many logging fixes
- still missing: final log consolidation/cleanup

logging: blue groups if `SHOW_DEBUG=yes` or `SHOW_GROUPS=yes` (console equivalent of CI's avocado)

armbian-next: shut down some too-verbose logging: logo building and update-initramfs

armbian-next: git/patching, kernel: use date from git as mtime minimum for patched files

- use revision's date from git log as mtime for all fetch_from_repo
- fix patched files date at least checkout date, otherwise some patches never build

armbian-next: first attempt at kernel packaging rework; just linux-image pkg, no dtbs yet

- correctly predict KERNELRELEASE, put image-dtbs in the right spot for flash-kernel
- remove dpkg-gencontrol, do it all directly

armbian-next: rework kernel source packaging, now exporting from git, to .tar.zst

- compress with zstdmt (multi-thread zstd), remove pv indicator, it's much faster anyway
- export from git (soon will have all patches et al too)
- better logging, show pkg name
- much, much faster due to zstdmt and deb with none compression

armbian-next: a bit atrocious, nameref loaded, `get_list_of_all_buildable_boards()`

- in the process, add support for userpatches/config structure mirroring core, for easy upstreaming

armbian-next: make `SKIP_EXTERNAL_TOOLCHAINS=yes` default. lets see what breaks [WiP]

armbian-next: keeping stdout clean, use display_alert() for cleanup logging

armbian-next: library cleanups; remove `build-all.sh` for good; bring `patching.sh` back

armbian-next: `interactive_desktop_main_configuration()` and stderr'ed + error handl'ed dialog

- use redirection to fd 3 for dialog, now cleanly on stderr
- `show_menu()` -> `dialog_menu()` et al
- interactive configuration now works again!

armbian-next: logging: `SHOW_PIDS=yes`

armbian-next: refactor and error-handle most of desktop configuration, incl menus/dialog

- `dialog_if_terminal_set_vars()` in place of `dialog_if_terminal()`

[WiP] ddk stuff, allow if not in `$KERNEL_TARGET`

armbian-next: split `compile_kernel()` function into smaller functions (+logging)

- `do_with_logging()` galore, much better error reporting for kernel
- `do_with_hooks()` is for the future, just a placeholder for now

armbian-next: `do_with_hooks()` placeholder for future ideas

armbian-next: logging: small refactor and `do_with_logging` admit it does not do error control

armbian-next: fix: traps: `trap_manager_error_handled` is integer (`-i`) not array (`-a`)

armbian-next: sunxi-tools: fix logging for sunxi-tools compilation

armbian-next: runners now run bash with `-o pipefail` in addition to `-e`

- attention, only affects stuff run through the functions in runners.sh

armbian-next: kernel: reduce logging clutter (CC,LD,AR)

- hide fasthash_debug under `SHOW_FASTHASH`

armbian-next: `armhf` should make `zImage` -- or should it?

armbian-next: show logs through ccze; avoid ANSI escapes in file; `SHOW_xxx` control

- `SHOW_DEBUG` shows the debug level
- `SHOW_COMMAND` shows all invoked commands through the runners
- `SHOW_TRAPS` to show 'cleanup' and 'trap' level
- `SHOW_TIMING` to show $SECONDS but pretty
- replace hardcoded traps/cleanups logging

armbian-next: add `KERNEL_MAJOR_MINOR=x.z` to every family, manually from the `KERNELBRANCH`

armbian-next: cold/warm bundles import/download/export for fetch_from_repo

- warm remote, if present, can be exported shallow
- if warm remote bundle is present, can be imported shallow too
- fallback to cold bundle if warm not present
- export (from cold, if exists + warm) shallow bundle
- use temp clone and DATE (not rev or tag) for shallowing, WORKS!
- info JSON/CSV, include "config_ok" true/false, kernel/uboot info
  - include logs for failed configs too
  - core reports ARMBIAN_WILL_BUILD_KERNEL and ARMBIAN_WILL_BUILD_UBOOT now with package names

armbian-next: `KERNELDIR` is out, `KERNEL_MAJOR_MINOR` is in for all `meson64`, `rpi4b` and `uefi`

armbian-next: new kernel workdir layout: cache/sources/kernel/<ARCH>-<KERNEL_MAJOR_MINOR>-<LINUXFAMILY>

- `GIT_FIXED_WORKDIR` is used to ignore 2nd param and use a specific dir
- this now REQUIRES `KERNEL_MAJOR_MINOR` to be set.
- prepare some `WARM_REMOTE_NAME` and related, based on it

armbian-next: JUST_KERNEL=yes (opposed to KERNEL_ONLY=yes) is really just the kernel build

armbian-next: fetch_from_repos now done when actually compiling atf/uboot/kernel, not before

- lib regen after removing empty files (sources.sh and errors.sh are now gone)

armbian-next: linux: back to Torvalds bundle, no tags; reminder about export

armbian-next: full cached kernel build; refactor all make's in a single place, even for packaging

- 2nd+ runs build in less than a minute
- kernel: compile and package in a single step, more efficient?
- KERNEL_BUILD_DTBS yes/no to build or not dtbs, replaces KERNEL_EXTRA_TARGETS
- dtbs_install, modules_install and headers_install now called by Armbian, not packaging
- kernel with split, but identical, build and install makes for modules/headers/dtbs
- make mkdebian and builddeb idempotent as possible
- keep a lot more cache, specially 'debian' folder
- filtering logging of install stuff
- might be a few leftovers, revisit gains with packaging later
  - keeping the arm64 makefile Image.gz vs Image hack
  - fix order of packaging patch byteshift, but still there
  - cleaning of scripts tools on cross compile removed (!)

armbian-next: minor stylistic changes that I end up doing while working on other stuff

- I am `OFFLINE_WORK`'ing, I don't wanna waste 3 seconds, thanks
- OCD-fix of double `local` declarations

[giga-wip] rework patching, introducing fasthash

[wip] git: experiment with stable kernel bundle, and all tags. nice, but for what?

- also: never delete working copy, exit with error instead.

[wip] disable make clean during packaging. I wanna rebuild fast, always [NO PR?]

armbian-next: export CHOSEN_KERNEL_WITH_ARCH for reporting info

- fix info gathering, parse all boards first, and stop if some failed
- fix KERNEL_TARGET regex by including optional "export "
- add export from info to CSV, very basic stuff, but works

[squash] remove ddk bullshit from KERNEL_TARGET

armbian-next: remove file logging of aggregation stuff. config phase can't touch disk anymore.

[WiP] git cold bundle; no: shallow clones/fetched; yes: locally packed repo

armbian-next: reorder functions in file, they have a ~logical call-tree order

armbian-next: move `fingerprint_image()` out of `git.sh` into its own file

logging: fix for double stderr redirect during `fakeroot_dpkg_deb_build`

logging: subdued "command" logging for debugging low level cmd invocations ("frog")

armbian-next: when showing log, emit all host-side invocations as `debug` too.

[WiP] trap handler abstraction, somewhat works!

armbian-next: manual merge (10) of all lib/*.sh changes between revisions a4ae3a2270 and 3305d45b81 - but NOT the git unshallow stuff, that will be rewritten

armbian-next: trapmanager pt1, identifying spots for trap manager intervention

armbian-next: `install_pkg_deb` -> `install_host_side_packages` which is completely rewritten version

- much simplified; compare installed packages vs wanted, and only update if some missing

armbian-next: force u-boot and kernel's gcc to output colors to make easy to spot warnings and errors

docker: pass the `CI` env var through Docker invocation, for GitHub Actions

armbian-next: avoid warning if "file" utility not installed

- should not happen, but better safe than sorry

armbian-next: disable long_running `pv` progress bar for custom case too

- will rework later, for now pipe causes subshell and caos

armbian-next: if `CI=true` then `SHOW_LOG=yes` always

docker: add arm64 toolchain to Dockerfile; warn, but don't break, on modprobe failure

armbian-next: docker: use ubuntu:rolling, fix deps, use `$SRC/cache` as container's cache dir

armbian-next: logging fixes (padding, don't show error more than once, don't remove trap)

armbian-next: fixes for early error handling and logging

- split stacktrace-related functions into their own lib file
- simplify the traps
- some stacktrace formatting for readability

armbian-next: fix: don't `trap` `ERR` twice, it causes bash to go bananas regarding `caller`

armbian-next: `UPSTEM_VER` -> `UBOOT_REPO_VERSION` and related fixes

armbian-next: oops, fix some non-lib changes I missed, up to revision ff4346c468

armbian-next: manual merge (9) of all lib/*.sh changes between revisions 3b7f5b1f34 and ff4346c468

armbian-next: more error handling fixes. avoid shortcircuits.

- store previous error message in `MSG_IF_ERROR` (still to be actually shown somewhere during error)

armbian-next: more error handling fixes. avoid subshells, shortcircuits, and pipes

- add `CFLAGS=-fdiagnostics-color=always` to kernel compile; would need also for u-boot soon

WiP: indexing JSON into OpenSearch, all-hardcoded version

rpi: add DTB symlink in Debian/Ubuntu standard location /lib/firmware/$version/device-tree; remove build-time-only hacks

- this allows us to remove the most horrible hack
- should allow for correctly working DTB upgrades
- should NOT impact other families, although a new symlink will be created, nothing uses it.

rpi: fix: flash-kernel fix to ignore kernel 'flavour' for all raspi's

armbian-next: don't try to remove packages that are not installed to begin with

- much faster
- new chroot_sdcard_with_stdout() runner, without bash or any escaping.

armbian-next: don't try to enable systemd services for units that don't exist

- those might be removed by a bsp extension, so check for existence before trying to enable

armbian-next: don't error/warn on failure to enable bootsplash when systemd units missing

armbian-next: use indented HEREDOCS for all call_extension_method's

armbian-next: manual merge (8) of all lib/*.sh changes between revisions 1d499d9ac2 and 3b7f5b1f34

armbian-next: manual merge (7) of all lib/*.sh changes between revisions d885bfc97d and 1d499d9ac2

armbian-next: manual merge (6) of all lib/*.sh changes between revisions c7f3c239fe and d885bfc97d

armbian-next: avoid writing to disk during configuration; `ANSI_COLOR=none` logging; make CONFIG_DEFS_ONLY=yes runnable without sudo

- when `CONFIG_DEFS_ONLY=yes`, avoid writing the config summary output.log file.
  - refactor that into a function as to be easy to if-out-of
  - don't write to disk during aggregate_content() if `CONFIG_DEFS_ONLY=yes`
  - don't write to disk during show_checklist_variables() if `CONFIG_DEFS_ONLY=yes`
  - don't write to disk during write_deboostrap_list_debug_log() if `CONFIG_DEFS_ONLY=yes`
  - don't compress and rotate logs if `CONFIG_DEFS_ONLY=yes`
- don't pretend to be handling errors we can't handle during var capture
- I foresee a world we can build all .debs without sudo
- and a some kind of split of codebase entrypoint due to that future feature
- some python info.py enhancements, not ready yet

armbian-next: shellfmt and regen library (after rebase from master n.5)

tools/shellfmt.sh: exclude "cache" and ".tmp" from formatting, for obvious reasons

tools/gen-library.sh: sort function files, so it does not keep changing between runs on different machines.

- order should not be important, since files only contain functions, but avoid git churn

armbian-next: manual merge (5) of all lib/*.sh changes between revisions 1b18df3c83 and e7962bb2b5

- most PKG_PREFIX work was already done

armbian-next: `TMPDIR` for all, many logging fixes, error handling: leave-no-garbage-behind without needing traps.

- set `MOUNT_UUID` and `WORKDIR`/`MOUNT`/`SDCARD`/`EXTENSION_MANAGER_TMP_DIR`/`DESTIMG` early in do_main_configuration()
  - but, they're just _set_ there, dirs are not created early, but on demand later
  - still @TODO: actually clean those during error trap. (unhappy path leaves garbage still)
  - but does not leave garbage behind during "successful" runs at least (happy path works)
- actually export `TMPDIR` (== `WORKDIR`) during start of build (not config!), so all `mktemp` are subject to it
  - `runners.sh` has helpers to avoid passing `TMPDIR` to chroot. Use the helpers! don't call `chroot` directly.
  - don't trap/cleanup individual `mktemp` temp dirs during .deb packaging's, all is handled at once now.
  - kernel packaging, for example, automatically picks up `TMPDIR` too. So now hosts `/tmp` is mostly left alone.
- fix some "infodumps" that are done into `.log` files directly.
- don't use sudo if `CONFIG_DEFS_ONLY=yes`; we'll only be collecting info, not doing anything.
- simpler logging for `rsync` operations (just dump to stdout, logging will handle it!)
- use padded counter for section logfiles, so we know which order things ran. exported as `CURRENT_LOGGING_COUNTER`
- no reason to use `apt-get` with `-yqq` anymore, since all logging is handled, so now `-y` by default
- desktop: using runners helpers for rootfs-desktop.sh, which should help a lot with acng caching and finding of problems
- extensions: correctly cleanup temp stuff, extensions has its own tmp/workdir now, and is always cleaned up at end of build.

armbian-next: bye `PKG_PREFIX`, hello `run_host_x86_binary_logged()` wrapper function; better error handling

- we've x86-only utilities that might need to be run on non-x86 build machines
- previously duplicated logic in PKG_PREFIX variable refactored into logged function
- added centralized debug logging
- replace all PKG_PREFIX usage with the new wrapper function, which already handles logging and errors.
  - mostly FIP tooling invocations
  - but also the boot_logo image builder
  - wrapper function delegates to common `run_host_command_logged`
- wrap other FIP invocations with `run_host_command_logged` too, for tidy logging
- avoid using conditionals when invoking functions; that completely disables error handling inside the called function
- use explicit bash opts instead of shortcuts like `set -e`
- a _lot_ of debug logging added

armbian-next: always use UPPERCASE labels for FAT32 UEFI filesystems (rpi4b, uefi-*)

armbian-next: shellfmt after rebase onto master

armbian-next: manual merge (4) of all lib/*.sh changes between revisions 23afccf56e and e610f00bc7

- plus ooops

atf: fix for `set -e` mode; fix CROSS_COMPILE quoting

Signed-off-by: Ricardo Pardini <ricardo@pardini.net>

logging: predict the future location of .img file

- otherwise it's really unhelpful

Signed-off-by: Ricardo Pardini <ricardo@pardini.net>

uefi: alias `BRANCH=ddk` to `current`'s `DISTRO_GENERIC_KERNEL=yes`

- no real change, just to match rpi4b's BRANCH=style
- opens space for Armbian-built `current` soon

Signed-off-by: Ricardo Pardini <ricardo@pardini.net>

rpi: `legacy`->`ddk` (distro default kernel), remove overclock

- common vars in bcm2711.conf moved to top
- removed overclock/overvolt that was leftover my old setup
- confirmed: works with rpi3b too, should work with CM4/CM3 and others
- use valid UPPERCASE FAT label for RPICFG (in place of `rpicfg`)

Signed-off-by: Ricardo Pardini <ricardo@pardini.net>

armbian-next: shellfmt again after rebase

Signed-off-by: Ricardo Pardini <ricardo@pardini.net>

armbian-next: manual merge (3) of all lib/*.sh changes between revisions 1035905760 and e4e4ab0791

- missed non lib change on Several improvements for RPi builds (#3391)
- I just realized I will have to drop all non lib changes

rockship: fixes for `set -e` mode in rockship armhf family and bsp tweaks

Signed-off-by: Ricardo Pardini <ricardo@pardini.net>

armhf: enable building armhf targets on amd64 using system toolchains

- SKIP_EXTERNAL_TOOLCHAINS=yes on amd64 should use the same system toolchains as an arm64 build

Signed-off-by: Ricardo Pardini <ricardo@pardini.net>

logging: better logging about family_tweaks and family_tweaks_bsp

Signed-off-by: Ricardo Pardini <ricardo@pardini.net>

kernel: unblock cross compilation, warn about headers package

Signed-off-by: Ricardo Pardini <ricardo@pardini.net>

logging: fixes for sunxi/megous stuff with `set -e`

Signed-off-by: Ricardo Pardini <ricardo@pardini.net>

logging: fix shellcheck references generation

Signed-off-by: Ricardo Pardini <ricardo@pardini.net>

logging: manual merge (2) of all lib/*.sh changes between revisions 117633687e and 3083038855

Signed-off-by: Ricardo Pardini <ricardo@pardini.net>

logging: renaming function files a bit more consistently

Signed-off-by: Ricardo Pardini <ricardo@pardini.net>

logging: removing leftover empty file after all the moving around

Signed-off-by: Ricardo Pardini <ricardo@pardini.net>

logging: really insist on set -e during library loading

Signed-off-by: Ricardo Pardini <ricardo@pardini.net>

logging: shellfmt again after rebasing master

Signed-off-by: Ricardo Pardini <ricardo@pardini.net>

logging: manual merge of all lib/*.sh changes between revisions f6143eff67 and f3388b9aee

Signed-off-by: Ricardo Pardini <ricardo@pardini.net>

logging: generic do_capturing_defs wrapper; Python parser

- enabled by passing CONFIG_DEFS_ONLY=yes; in this case does not build anything
- [WiP] Python3 info reader / matrix expander
  - multithreaded version

Signed-off-by: Ricardo Pardini <ricardo@pardini.net>

logging: move some interactive parts of config into its own functions

- mostly from config-prepare;
- there is still a lot of others in main-config

Signed-off-by: Ricardo Pardini <ricardo@pardini.net>

logging: use chroot_custom for grub and flash-kernel extension logging

Signed-off-by: Ricardo Pardini <ricardo@pardini.net>

logging: use line buffering, fix runner output color for GHA

Signed-off-by: Ricardo Pardini <ricardo@pardini.net>

logging: wrap dpkg-deb; set TMPDIR (not in chroot); refactor kernel make

- And a huge amount of @TODO's added
- Add "debug" and "deprecation" `display_alert()` levels
- insist that `install_common` is now `install_distribution_agnostic`
- unrelated: realtek 8822CS is EXTRAWIFI=yes only now, sorry.
- many debug statements for desktop

Signed-off-by: Ricardo Pardini <ricardo@pardini.net>

logging: don't bail out on patching error

Signed-off-by: Ricardo Pardini <ricardo@pardini.net>

logging: bunch of fixes; no-stdin; traps; better stacks

- mostly no-stdin dialog handling (desktop et al)
- let ERR trap run together with unmount trap (EXIT etc)

Signed-off-by: Ricardo Pardini <ricardo@pardini.net>

logging: trap ERR very early, pass-in caller info

Signed-off-by: Ricardo Pardini <ricardo@pardini.net>

armbian-next: huge refactor, shellfmt, codesplit, logging/error handling

- *this commit changes most/all the lines of bash code in armbian/build*
- *author is set to IgorPec for historical reasons, rpardini is to blame for the actual changes*
- logging: refactorings, pt.4: autogen lib, shellfmt tool, extract cli
  - shellfmt: auto-downloader and runner of shellfmt
    - darwin/linux
    - amd64/arm64
    - find ~correct files to format
    - run formatting
    - check formatting soon
  - refactor compile's CLI stuff out into function/cli
  - gen, and use genned library with tool
- logging: refactoring pt3: HUGE split of everything into everything else
  - plus rebasing fixes
- logging: refactorings, pt. 2: error handling
  - fix: no errors during umount_chroot()
  - no progress for CI=true builds
  - WiP disable kernel hashing. too crazy.
  - a few builds now working with "set -e"
  - wtf. or something
  - kernel logging and long_running stuff - a mess - needs moving around in files
  - rewrite uboot compile loop without using subshells. remove ALL traps. refactor host command
  - better logging for u-boot
  - more fixes, u-boot
  - more fixes for logging et al
  - git stuff
  - many (many) fixes
  - new color scheme
  - a monster. make sure unmounted at the end. remove set -e's, to-be-readded.
  - remove set -e "for safety reasons"
  - more alerts. we gotta know whats failing
  - some more logging stuff and fixes for error checking
  - more logging and error handling stuff
  - fixes; some set -e's
  - more logging stuff
- logging: refactoring codebase, pt.1: functions-only
  - Refactor the hell out of everything into functions
  - rename build-all-ng to build-multi; other fixes, extensions init
  - slight nudge
  - some were already good, like this one.
  - syntax fixes
  - some need a little nudge
  - another clean one
  - some just need a better name (and splitting later)
  - syntax fixes
  - some were already good, like this desktop one
  - some were already good, like this other one
  - some were already good, like this one.
  - debootstrap is gone.
  - extract functions from compile.sh
  - add logging to main_default_build
  - more stuff
  - cleanups and refactors of main.sh
- logging: first steps
- logging: pt. 0: shellfmt everything

- add riscv64 do SRC_ARCH/ARCH/ARCHITECTURE mess; add warn
2023-02-18 07:38:46 -03:00

120 lines
4.1 KiB
Bash

#!/usr/bin/env bash
#
# Copyright (c) 2013-2021 Igor Pecovnik, igor.pecovnik@gma**.com
#
# This file is licensed under the terms of the GNU General Public
# License version 2. This program is licensed "as is" without any
# warranty of any kind, whether express or implied.
#
# This file is a part of the Armbian build script
# https://github.com/armbian/build/
# Functions:
# compile_atf
# compile_uboot
# compile_kernel
# compile_firmware
# compile_armbian-config
# compile_xilinx_bootgen
# grab_version
# find_toolchain
# advanced_patch
# process_patch_file
# userpatch_create
# overlayfs_wrapper
grab_version() {
local ver=()
ver[0]=$(grep "^VERSION" "${1}"/Makefile | head -1 | awk '{print $(NF)}' | grep -oE '^[[:digit:]]+' || true)
ver[1]=$(grep "^PATCHLEVEL" "${1}"/Makefile | head -1 | awk '{print $(NF)}' | grep -oE '^[[:digit:]]+' || true)
ver[2]=$(grep "^SUBLEVEL" "${1}"/Makefile | head -1 | awk '{print $(NF)}' | grep -oE '^[[:digit:]]+' || true)
ver[3]=$(grep "^EXTRAVERSION" "${1}"/Makefile | head -1 | awk '{print $(NF)}' | grep -oE '^-rc[[:digit:]]+' || true)
echo "${ver[0]:-0}${ver[1]:+.${ver[1]}}${ver[2]:+.${ver[2]}}${ver[3]}"
return 0
}
# find_toolchain <compiler_prefix> <expression>
#
# returns path to toolchain that satisfies <expression>
#
find_toolchain() {
[[ "${SKIP_EXTERNAL_TOOLCHAINS}" == "yes" ]] && {
echo "/usr/bin"
return
}
local compiler=$1
local expression=$2
local dist=10
local toolchain=""
# extract target major.minor version from expression
local target_ver
target_ver=$(grep -oE "[[:digit:]]+\.[[:digit:]]" <<< "$expression")
for dir in "${SRC}"/cache/toolchain/*/; do
# check if is a toolchain for current $ARCH
[[ ! -f ${dir}bin/${compiler}gcc ]] && continue
# get toolchain major.minor version
local gcc_ver
gcc_ver=$("${dir}bin/${compiler}gcc" -dumpversion | grep -oE "^[[:digit:]]+\.[[:digit:]]")
# check if toolchain version satisfies requirement
awk "BEGIN{exit ! ($gcc_ver $expression)}" > /dev/null || continue
# check if found version is the closest to target
# may need different logic here with more than 1 digit minor version numbers
# numbers: 3.9 > 3.10; versions: 3.9 < 3.10
# dpkg --compare-versions can be used here if operators are changed
local d
d=$(awk '{x = $1 - $2}{printf "%.1f\n", (x > 0) ? x : -x}' <<< "$target_ver $gcc_ver")
if awk "BEGIN{exit ! ($d < $dist)}" > /dev/null; then
dist=$d
toolchain=${dir}bin
fi
done
echo "$toolchain"
}
# overlayfs_wrapper <operation> <workdir> <description>
#
# <operation>: wrap|cleanup
# <workdir>: path to source directory
# <description>: suffix for merged directory to help locating it in /tmp
# return value: new directory
#
# Assumptions/notes:
# - Ubuntu Xenial host
# - /tmp is mounted as tmpfs
# - there is enough space on /tmp
# - UB if running multiple compilation tasks in parallel
# - should not be used with CREATE_PATCHES=yes
#
overlayfs_wrapper() {
local operation="$1"
if [[ $operation == wrap ]]; then
local srcdir="$2"
local description="$3"
mkdir -p /tmp/overlay_components/ /tmp/armbian_build/
local tempdir workdir mergeddir
tempdir=$(mktemp -d --tmpdir="/tmp/overlay_components/") # @TODO: WORKDIR? otherwise uses host's root disk, which might be small
workdir=$(mktemp -d --tmpdir="/tmp/overlay_components/")
mergeddir=$(mktemp -d --suffix="_$description" --tmpdir="/tmp/armbian_build/")
mount -t overlay overlay -o lowerdir="$srcdir",upperdir="$tempdir",workdir="$workdir" "$mergeddir"
# this is executed in a subshell, so use temp files to pass extra data outside
echo "$tempdir" >> /tmp/.overlayfs_wrapper_cleanup
echo "$mergeddir" >> /tmp/.overlayfs_wrapper_umount
echo "$mergeddir" >> /tmp/.overlayfs_wrapper_cleanup
echo "$mergeddir"
return
fi
if [[ $operation == cleanup ]]; then
if [[ -f /tmp/.overlayfs_wrapper_umount ]]; then
for dir in $(< /tmp/.overlayfs_wrapper_umount); do
[[ $dir == /tmp/* ]] && umount -l "$dir" > /dev/null 2>&1
done
fi
if [[ -f /tmp/.overlayfs_wrapper_cleanup ]]; then
for dir in $(< /tmp/.overlayfs_wrapper_cleanup); do
[[ $dir == /tmp/* ]] && rm -rf "$dir"
done
fi
rm -f /tmp/.overlayfs_wrapper_umount /tmp/.overlayfs_wrapper_cleanup
fi
}