Commit Graph

98 Commits

Author SHA1 Message Date
Ricardo Pardini
d8016c49b6
armbian-next: fix ATF build under binutils 2.39+ via TF_LDFLAGS=--no-warn-rwx-segment 2023-02-18 07:43:48 -03:00
Ricardo Pardini
4dd278f07d
armbian-next: compile_firmware(): use new temp dir helpers (saves 2Gb+ in WORKDIR), fix typos 2023-02-18 07:43:32 -03:00
Ricardo Pardini
92d902c883
armbian-next: drop dead/duplicated code for bootsplash and mark kernel-drivers.sh (which includes bootsplash) as dead code 2023-02-18 07:43:23 -03:00
Ricardo Pardini
2455c55dac
armbian-next: firmware: don't build -full firmware if not on CI/noninteractive and board's not going to use it
- move `fakeroot_dpkg_deb_build()` from runners.sh to new utils-dpkgdeb.sh
  - and make it debug show the size of the source
- clean off `.git` directory earlier for full build
- @TODO: get rid of the "hardlinks" funky biz in there and not-copy `.git` to begin with
2023-02-18 07:43:18 -03:00
Ricardo Pardini
f2bd68afde
armbian-next: kernel: cleanup bundle after patching succeeded, not after build success
- for extra gigabytes of space, earlier
2023-02-18 07:43:17 -03:00
Ricardo Pardini
2d9f9216eb
armbian-next: introduce tmpfs-utils.sh; put LOGDIR and WORKDIR under tmpfs; set CCACHE_TEMPDIR under WORKDIR
- introduce generic `prepare_tmpfs_for()`, which manages it's own cleanup/dir removal
- use it for `WORKDIR` (which is `TMPDIR`) and `LOGDIR`
  - adapt previous cleanup handlers for those, so they delete their contents but not the dir itself (which might be mounted)
- also: make `ARMBIAN_LOG_CLI_ID` readonly together with other superglobals
- set `XDG_RUNTIME_DIR` & `XDG_RUNTIME_DIR` together with `TMPDIR`
- kernel-make.sh: pass `CCACHE_TEMPDIR` down to Kernel make (thanks @the-Going)
2023-02-18 07:43:09 -03:00
Ricardo Pardini
7f21e7d5f3
armbian-next: ccache: show more in ccache debugging 2023-02-18 07:43:07 -03:00
Ricardo Pardini
78fd35a1e3
armbian-next: better logging for kernel_prepare_bare_repo_from_oras_gitball(); remove dead code
- remove dead code used to fetch tags, back from pre-bundle days
2023-02-18 07:42:51 -03:00
Ricardo Pardini
7beeae6219
armbian-next: Python tooling: use consolidated+hashed+cached pip base/pycache; don't pip during Dockerfile build, nor cli-requirements
- consolidate at `prepare_python_and_pip()`
- sanity check for Python version 3.9+ regardless of HOSTRELEASE
- TODO: pip vs sudo/root: need pip 22.2+ to curb warning, not doing it
2023-02-18 07:42:44 -03:00
Ricardo Pardini
53266feff8
armbian-next: kernel: deterministic .config mtime handling; kernel build logging sections reorg
- .config is now compared with previous one byte-wise, diff shown, and mtime preserved directly via `cp -p`
- this does away with the previous mtime-based BS I had written during the fasthash era (and thus remove file-mtime.sh, regen lib)
- split some functions, to better control the interactive piece in the _middle_ of kernel config process
- no more `kernel_config_maybe_interactive()`, now `kernel_config()` that controls its own logging sections
- extract `kernel_determine_toolchain()` out of config into make
- introduce internal hook `armbian_kernel_config()` that runs before custom version (for .config defaults, coming soon)
2023-02-18 07:42:39 -03:00
Ricardo Pardini
c99e06a84b
armbian-oleg: logging: disable debugging (set -x) of generated postinst/etc kernel scripts (board-side, but also chroot logging) 2023-02-18 07:42:38 -03:00
Ricardo Pardini
6b09e97b4f
armbian-oleg: logging: drastically curb / make more useful kernel packaging logging 2023-02-18 07:42:37 -03:00
Ricardo Pardini
00e1dec8a2
armbian-next: odroidxu4 vs kernel: move firmware hack to family using extension hook
- @TODO: is this even needed/used anymore?
2023-02-18 07:42:36 -03:00
Ricardo Pardini
62bc394738
armbian-oleg: curb logging from building armbian-plymouth-theme; do it in a logging section, like the others 2023-02-18 07:42:33 -03:00
Ricardo Pardini
e8306a8712
armbian-oleg: curb logging from building armbian-firmware 2023-02-18 07:42:32 -03:00
Ricardo Pardini
d979c73c1d
armbian-next: split compilation/debs.sh; compile_xilinx_bootgen() moved to family 2023-02-18 07:42:31 -03:00
Ricardo Pardini
2c0c968645
armbian-oleg: curb Python launcher logs 2023-02-18 07:42:24 -03:00
Ricardo Pardini
47af820f5d
armbian-oleg: junk: drastically reduce host-side dependencies / "remove junk"
- remove a lot of hopefully / hardly confirmed, unused dependencies
- don't use crossbuild-essential-xxx; avoid the c++ compiler that comes with it, install gcc-only instead
- hostdeps: use `libc6-dev make dpkg-dev gcc` (without `g++`) instead of `build-essential`
- drop `btrfs-progs` and `f2fs-tools`  (@TODO add in extension when/whereused)
- more: drop `cryptsetup` (@TODO add in extension when/whereused)
- don't be too quiet when doing apt-update for hostdeps
2023-02-18 07:42:14 -03:00
Ricardo Pardini
e6b1c4d875
armbian-next: CLEAN_LEVEL=make-kernel now does git clean -xfd instead, faster and 100% clean 2023-02-18 07:42:07 -03:00
Ricardo Pardini
4f743caab8
armbian-next: drivers-harness: force -c commit.gpgsign=false during git commit 2023-02-18 07:42:06 -03:00
Ricardo Pardini
e084991abc
armbian-next: copy modified config + defconfig back from where it came from when KERNEL_CONFIGURE=yes 2023-02-18 07:41:57 -03:00
Ricardo Pardini
ac81b410e0
armbian-next: [focal-host] don't use git format-patch's --no-encode-email-headers which is not supported on focal's git 2023-02-18 07:41:51 -03:00
Ricardo Pardini
eda272045d
armbian-next: [focal-host] remove usages of declare -I, not supported under focal's bash
- this is bad/terrible and will cause a lot of shellcheck errors... but such is life
2023-02-18 07:41:49 -03:00
Ricardo Pardini
6cb12a3397
armbian-next: [focal-host] try to support focal-like host OS's again, via python3.9 (and drop buster completely)
- completely remove support for building under `buster` -- that's way too old, sorry.
- de-hardcode `python3` invocations, instead use `python3_binary_path` set by `prepare_python3_binary_for_python_tools()`
- juggle `$HOSTRELEASE`: read from actual host, or determined from Docker image name (during Dockerfile build)
- TL;DR: include and use `python3.9` for focal-like host OS's
2023-02-18 07:41:47 -03:00
Ricardo Pardini
61d4691e09
armbian-next: drivers-harness: make sure cached patch is valid; write it first to tmp file; redirect inside subshell
- otherwise if format-patch failed, it would still create the (invalid) file which would be used next time
2023-02-18 07:41:45 -03:00
Ricardo Pardini
92a4296a62
armbian-next: drivers-harness: pass HOME/PATH explicitly to git commit/format-patch 2023-02-18 07:41:44 -03:00
Ricardo Pardini
60fb19b48d
Obsolete: kernel-headers via full source + make modules_prepare for 4.19 kernels. [later removed] 2023-02-18 07:41:34 -03:00
Ricardo Pardini
68fafebf9f
Obsolete: try for working armhf kernel-headers. leading nowhere. move to kernel-source / make modules_prepare later. [later re-done] 2023-02-18 07:41:33 -03:00
Ricardo Pardini
b8e595aef4
armbian-next: drivers-harness: include the revert-IPX patch as a "driver" before all others 2023-02-18 07:41:28 -03:00
Ricardo Pardini
c18357fe56
armbian-next: drivers-harness: remove commit-date, do proper logging & escaping for git add/commit/format-patch
- seems like I indulged in the same mistakes I've been correcting for years ("&> /dev/null") - d'oh
2023-02-18 07:41:27 -03:00
Ricardo Pardini
8f3fd355eb
armbian-next: tune logging in multiple places, there was a lot of redundant/un-needed info's 2023-02-18 07:41:22 -03:00
Ricardo Pardini
7987dad34d
armbian-next: Python tools (aggregation/patching): use separate pycache dir in ./cache; use unbuffered stdout/stderr 2023-02-18 07:41:21 -03:00
Ricardo Pardini
9973010972
armbian-next: Python patching: pass $HOME/$PATH to Python, so git can find .gitconfig and Python can find git
- add sanity check before handling the Git repo in Python (call `status()` to make sure access is OK)
2023-02-18 07:41:16 -03:00
Ricardo Pardini
9370c19a44
armbian-next: drop old code for patching & fasthash
- split the modification time stuff to general
2023-02-18 07:41:05 -03:00
Ricardo Pardini
9200a2ce0c
armbian-next: bring find_toolchain() back from the dead; force SKIP_EXTERNAL_TOOLCHAINS=no to force use external toolchains
- handle a lot of corner cases, some toolchains are i386-only and won't run on modern x64 systems
2023-02-18 07:40:58 -03:00
Ricardo Pardini
770d508ca0
armbian-next: Python patching delusion pt 5 + EXTRAWIFI patch generator/harness + consistent patch mtime targets
- Python patching: tune some logging all-around
- Python patching: *FINALLY* set the dates on the patched files to `max(patch_date, root_makefile_date)` thus making lighting-fast rebuilds again
- new EXTRAWIFI patch generator harness; Python patching EXTRA_PATCH_FILES_FIRST
- Python patching: use temp file for patching rejects; clean it up
- new EXTRAWIFI patch generator harness; Python: mark drivers as autogen, don't split or parse them, read as bytes, apply bytes directly
- new EXTRAWIFI patch generator harness; somewhat-works, but patches are per-family
- Python patching: add the `cache/patch` as a mountpoint
  - Darwin Docker performance is 20x with namedvolume; the cached patches are very large
2023-02-18 07:40:55 -03:00
Ricardo Pardini
97f6836705
armbian-next: Python patching delusion, pt4: series & better markdown
- Python patching: hopefully better Markdown: split series dir; detect more status/problems; arche only for missing Subject
  - Python patching: archeology only for missing Subject:, not description; clarify CLI pushing
  - Python patching: use `{file_base_name}.patch` for archeology as `file_name` might include a dir
- Python patching: fix: don't skip board/target patches for no reason
- Python patching: fix for series.conf patches (sunxi/sunxi64): don't sort; mark as series and show on summary
- Python patching: don't fail if archeology found no commits (but add warning)
- Python patching: CLI command `kernel-patches-to-git` with archeology summary, and git pushing of results & summary
  - Python patching: patches-to-git small fixes, auto-push if it's rpardini
  - Python patching: add `patch-kernel` CLI command
- Python patching: commit README.md & gh-pages workflow when apply_patches_to_git
  - Python patching: hopefully better markdown
- Python patching: `git add` everything all at once, for speed
2023-02-18 07:40:54 -03:00
Ricardo Pardini
2c0e9182ed
armbian-next: introduce ORAS-based kernel git bare tree seeding/bundles
- armbian-next: fixes to ORAS-related logging
- armbian-next: actually enable ORAS-based kernel git bare tree seeding/bundles; enable cleaning of bundle articfacts after confirmed working
- armbian-next: introduce ORAS-based kernel git bare tree seeding/bundles (although it is a .tar, not a bundle); this is 20x faster than cloning
- armbian-next: introduce `ORAS` tooling; pull and push functions & downloader/launcher
2023-02-18 07:40:53 -03:00
Ricardo Pardini
76e276c6a9
armbian-next: Python patching delusion, pt1 & pt2 & pt3
- WiP: Python patching delusion, pt 1: finding & parsing patches; apply & git commit with pygit2; Markdown summaries (also for aggregation); git-to-patches tool
  - Python: Markdown aggregation and patching summaries; collapsible; SummarizedMarkdownWriter
  - Python: Markdown aggregation and patching summaries
  - Python: reorg a bit into common/armbian_utils; define the `ASSET_LOG_BASE` in preparation for Markdown delusion
  - Python patching: initial apply patches & initial commit patches to git (using pygit2)
  - Python patching: add basic `series.conf` support
  - Python patching: force use of utf-8; better error handling; use realpath of dirs
  - Python patching: `git-to-patches` initial hack. not proud. half-reused some of the patches-to-git
  - Python patching: "tag" the git commits with info for extracting later; introduce REWRITE_PATCHES/rewrite_patches_in_place
  - Python patching: commented-out, recover-bad-patches hacks
  - Python patching: shorten the signature
  - Python patching: allow BASE_GIT_TAG as well as BASE_GIT_REVISION
  - Python patching: git-archeology for patches missing descriptions; avoid UTF-8 in header/desc (not diff)
  - Python patching: use modern-er email.utils.parsedate_to_datetime to parse commit date
  - Python patching: unify PatchInPatchFile; better git-commiting; re-exporting patches from Git (directly)
  - Python patching: switch to GitPython
    - GitPython is like 100x slower than pygit2, but actually allows for date & committer
    - also allows to remove untracked files before starting
  - Python aggregation: fix missing `AGGREGATED_APT_SOURCES_DICT`
  - Python patching: add `unidecode` dependency to pip3 install
  - Python patching: don't try archeology if SRC is not a Git Repo (eg, in Docker)
  - Python patching: don't try archeology if not applying patches to git
- WiP: Python patching delusion, pt2: actually use for u-boot & kernel patching
  - Python patching: much better problem handling/logging; lenient with recreations (kernel)
  - Python patching: don't force SHOW_LOG for u-boot patching
  - Python patching: don't bomb for no reason when there are no patches to apply
  - Python patching: fully (?) switch kernel patching to Python
  - Python patching: more logging fixups
  - Python patching: capture `kernel_git_revision` from `fetch_from_repo()`'s `checked_out_revision`
  - Python patching: fully switch u-boot patching to Python
  - Python aggregation/patching: colored logging; patching: always reset to git revision
  - Python aggregation/patching: better logging; introduce u-boot Python patching
- Python patching pt3: recovers and better Markdown
  - Python patching: detect, and rescue, `wrong_strip_level` problem; don't try to export patches that didn't apply, bitch instead
  - Python patching: Markdown patching summary table, complete with emoji
  - Python patching: include the problem breakdown in Markdown summary
  - Python patching: sanity check against half-bare, half-mbox patches
  - Python patching: try to recover from 1) bad utf-8 encoded patches; 2) bad unidiff patches; add a few sanity checks
  - Python patching: try, and fail, to apply badly utf-8 encoded patches directly as bytes [reverted]
  - Python patching: try to recover from patch *parse* failures; show summary; better logging
      - set `GIT_ARCHEOLOGY=yes` to do archeology, default not

- armbian-next: Python `pip` dependencies handling, similar to `hostdeps`
  - same scheme for Dockerfile caching
  - @TODO: still using global/shared environment; should move to a dir under `cache` or some kinda venv
- WiP: add `python3-pip` to hostdeps; remove `python-setuptools`
  - remove `python-setuptools` (Python2, no longer exists in Sid) from hostdeps
  - add `python3-pip` to hostdeps; part of virtualenv saga
- WiP: split `kernel.sh` a bit, into `kernel-patching.sh`, `kernel-config.sh` and `kernel-make.sh`
  - `advanced_patch()`: rename vars for clarity; no real changes
- Python patching: introduce FAST_ARCHEOLOGY; still trying for Markdown links
2023-02-18 07:40:52 -03:00
Ricardo Pardini
fd043d707a
armbian-next: squash some warn loggings that were undue; remove $SUDO_USER handling in prepare-host.sh
- squash warning about `NO_APT_CACHER` when in Docker
2023-02-18 07:40:43 -03:00
Ricardo Pardini
074857ede3
armbian-next: u-boot: use git worktree for u-boot; full mainline clone for bare
- share a single `.git` for all u-boots
- all fetches are done against it, maximizing cache hit ratio drastically
- also reduces the size of each working copy by more than 70%
- split uboot's git stuff into uboot-git.sh
- split `uboot_prepare_git()` from `compile_uboot()`
2023-02-18 07:40:39 -03:00
Ricardo Pardini
84c0bf0f7d
armbian-next: kernel: use git worktree & kernel.org clone bundles for kernel working copies
- now we've a single `.git` for all kernels with all branches; a "master" cache
- back to full stable git bundle usage from kernel.org
- bye bye, "gitballs"
- split kernel-related git stuff into `kernel-git.sh`
- introduce `ARMBIAN_MOUNTPOINTS_DEPRECATED` @TODO actually use for cleaning non-Docker
- add `axel` hostdep for multi-conn download of bundle, which comes from a CDN.
- deprecate old mountpoints for gitballs and linux-kernel
- move cleaning of old sources/kernel to mountpoints code
2023-02-18 07:40:38 -03:00
Ricardo Pardini
4a9529dd34
armbian-next: qemu-uboot-arm64 and qemu-uboot-x86 new boards
- sharing most UEFI code, will replace the `virtual` one soon
- x86: patch uboot defconfig to use the `q35` machine type, not `i440fx`
  - separate x86 bootscript, due to non-uInitrd-ness of it
  - hack ramdisk load address both in u-boot source and bootscript
  - use 32-bit u-boot, not 64-bit
- grub: introduce `UEFI_GRUB=skip`, does not deploy GRUB (but does the kernel packages, etc)
- auto-enable qcow2 output for these
- works with both distro's and Armbian's kernels
2023-02-18 07:40:33 -03:00
Ricardo Pardini
b951f7b873
armbian-next: u-boot: introduce hook pre_config_uboot_target()
- useful for hacking at Python versions and other compilation-time hacks
2023-02-18 07:40:32 -03:00
Ricardo Pardini
b2d02071bd
armbian-next: better ccache logs/stats/Docker-support/use in u-boot/kernel
- introduce `SHOW_CCACHE=yes` for detailed ccache statistics and logging
- fix: kernel build is done under "env -i", so pass CCACHE_DIR down to Make if CCACHE_DIR is set...
- split from kernel.sh, show stats also for u-boot targets; show compile time
- add volume definition (under `${SRC}/cache/ccache`); auto-use that in Docker builds via `CCACHE_DIR`
- better logging for `do_with_ccache_statistics()`
- there's some CCACHE_DIR code from before; unify @TODO
2023-02-18 07:40:19 -03:00
Ricardo Pardini
da0b3080a0
armbian-next: distcc: optionally using distcc for kernel build 2023-02-18 07:40:17 -03:00
Ricardo Pardini
81376df4b7
armbian-next: u-boot: use scripts/config instead of seding into the .config (BOOTDELAY, CONFIG_LOGLEVEL=6, etc); v2022.10+ only 2023-02-18 07:40:16 -03:00
Ricardo Pardini
cc62d4932a
armbian-next: introduce ARMBIAN_FIRMWARE_GIT_SOURCE and _BRANCH to allow custom firmware repository coordinates 2023-02-18 07:40:14 -03:00
Ricardo Pardini
291b02577a
armbian-next: rename compile_plymouth-theme-armbian() to compile_plymouth_theme_armbian() for consistency 2023-02-18 07:40:08 -03:00
Ricardo Pardini
d502402c1a
armbian-next: show more complete gcc versions in logs 2023-02-18 07:40:04 -03:00