Commit Graph

236 Commits

Author SHA1 Message Date
Ricardo Pardini
eda272045d
armbian-next: [focal-host] remove usages of declare -I, not supported under focal's bash
- this is bad/terrible and will cause a lot of shellcheck errors... but such is life
2023-02-18 07:41:49 -03:00
Ricardo Pardini
6dda660fb6
armbian-next: [focal-host] try to support focal-like host OS's again, by avoiding git's --no-auto-maintenance 2023-02-18 07:41:48 -03:00
Ricardo Pardini
6cb12a3397
armbian-next: [focal-host] try to support focal-like host OS's again, via python3.9 (and drop buster completely)
- completely remove support for building under `buster` -- that's way too old, sorry.
- de-hardcode `python3` invocations, instead use `python3_binary_path` set by `prepare_python3_binary_for_python_tools()`
- juggle `$HOSTRELEASE`: read from actual host, or determined from Docker image name (during Dockerfile build)
- TL;DR: include and use `python3.9` for focal-like host OS's
2023-02-18 07:41:47 -03:00
Ricardo Pardini
f167864f2c
armbian-next: split of config-prepare into multiple functions, allowing for logging of all non-interactive sections
- introduce `do_with_conditional_logging()` which only starts logging sections if `do_logging=no`
- with this we should get complete logs (ofc except for the interactive sections)
2023-02-18 07:41:46 -03:00
Ricardo Pardini
61d4691e09
armbian-next: drivers-harness: make sure cached patch is valid; write it first to tmp file; redirect inside subshell
- otherwise if format-patch failed, it would still create the (invalid) file which would be used next time
2023-02-18 07:41:45 -03:00
Ricardo Pardini
92a4296a62
armbian-next: drivers-harness: pass HOME/PATH explicitly to git commit/format-patch 2023-02-18 07:41:44 -03:00
Ricardo Pardini
0e8bddd1e8
armbian-next: cli: config-dump: can now run on any bash/coreutils-enabled machine; remove some useless vars from dump 2023-02-18 07:41:42 -03:00
Ricardo Pardini
c33b59d61b
armbian-next: config: reduce bash scope & remove some vars un-needed outside of config 2023-02-18 07:41:40 -03:00
Ricardo Pardini
166a34da86
armbian-next: avoid linux-version during configuration; declare -g instead of export
- config might run on a non-Linux, bash-capable, GNU coreutils enabled machine, so avoid `linux-version`
- use `declare -g` instead of `export` for most of `prepare_and_config_main_build_single()`
2023-02-18 07:41:40 -03:00
Ricardo Pardini
f252138693
armbian-next: call hook post_aggregate_packages() only after actually aggregating 2023-02-18 07:41:38 -03:00
Ricardo Pardini
6c027ba552
armbian-next: delete old aggregation code from config 2023-02-18 07:41:37 -03:00
Ricardo Pardini
67108e31c0
armbian-next: move var HOSTRELEASE from config to prepare_host() 2023-02-18 07:41:36 -03:00
Ricardo Pardini
6797016cfb
armbian-next: move compilation vars (chtreads, ccache stuff) from config to compile 2023-02-18 07:41:35 -03:00
Ricardo Pardini
60fb19b48d
Obsolete: kernel-headers via full source + make modules_prepare for 4.19 kernels. [later removed] 2023-02-18 07:41:34 -03:00
Ricardo Pardini
68fafebf9f
Obsolete: try for working armhf kernel-headers. leading nowhere. move to kernel-source / make modules_prepare later. [later re-done] 2023-02-18 07:41:33 -03:00
Ricardo Pardini
59278d1869
armbian-next: remove spurious debug logging with no useful info "Error-related files found" 2023-02-18 07:41:32 -03:00
Ricardo Pardini
b8e595aef4
armbian-next: drivers-harness: include the revert-IPX patch as a "driver" before all others 2023-02-18 07:41:28 -03:00
Ricardo Pardini
c18357fe56
armbian-next: drivers-harness: remove commit-date, do proper logging & escaping for git add/commit/format-patch
- seems like I indulged in the same mistakes I've been correcting for years ("&> /dev/null") - d'oh
2023-02-18 07:41:27 -03:00
Ricardo Pardini
ec2e6c42be
armbian-next: happy new year, bump Igor's copyright 2023-02-18 07:41:25 -03:00
Ricardo Pardini
43f268fdcb
armbian-next: remove very old extlinux killswitch 2023-02-18 07:41:24 -03:00
Ricardo Pardini
c61b985417
armbian-next: better logging in distro-agnostic.sh
- still _a lot_ to fix
2023-02-18 07:41:23 -03:00
Ricardo Pardini
8f3fd355eb
armbian-next: tune logging in multiple places, there was a lot of redundant/un-needed info's 2023-02-18 07:41:22 -03:00
Ricardo Pardini
7987dad34d
armbian-next: Python tools (aggregation/patching): use separate pycache dir in ./cache; use unbuffered stdout/stderr 2023-02-18 07:41:21 -03:00
Ricardo Pardini
59031be6b3
armbian-next: only put systemd-resolved's symlink in place if _not_ using NetworkManager
- why is it even installed? dunno, but fix anyway.
2023-02-18 07:41:20 -03:00
Ricardo Pardini
f8b141cdf0
armbian-next: fix: only aggregate packages if building images (AR-1471) 2023-02-18 07:41:19 -03:00
Ricardo Pardini
690ca4df7c
armbian-next: add (trap) cleanup handler to reset ./tmp and ./output to original pre-sudo UID before exiting, even with failure/abort 2023-02-18 07:41:17 -03:00
Ricardo Pardini
9973010972
armbian-next: Python patching: pass $HOME/$PATH to Python, so git can find .gitconfig and Python can find git
- add sanity check before handling the Git repo in Python (call `status()` to make sure access is OK)
2023-02-18 07:41:16 -03:00
Ricardo Pardini
12b31afa51
armbian-next: git: mark submodule's dirs as safe before recursive fetch_from_repo() 2023-02-18 07:41:15 -03:00
Ricardo Pardini
3bc8afd593
armbian-next: refactor losetup -d ... into free_loop_device_insistent() which retries 10 times 2023-02-18 07:41:14 -03:00
Ricardo Pardini
f679b81b10
armbian-next: git: use --recurse-submodules=no for fetch; fetch_from_repo() handles submodules separately
- also remove `du` working copy size debugging
2023-02-18 07:41:13 -03:00
Ricardo Pardini
65da2a697f
armbian-next: remove @TODO for stuff already done 2023-02-18 07:41:12 -03:00
Ricardo Pardini
ddbb91653f
armbian-next: reset_uid_owner for the markdown log file too 2023-02-18 07:41:11 -03:00
Ricardo Pardini
2f9a4470c7
armbian-next: validate cmdline params against malformed
- eg: `./compile.sh $ERROR=here` is caught now
2023-02-18 07:41:10 -03:00
Ricardo Pardini
093e60d111
armbian-next: exec into sudo when relaunching (AR-1452)
- with `exec`, `sudo` replaces the current shell, and thus never returns
2023-02-18 07:41:09 -03:00
Ricardo Pardini
9370c19a44
armbian-next: drop old code for patching & fasthash
- split the modification time stuff to general
2023-02-18 07:41:05 -03:00
Ricardo Pardini
143ba996ca
armbian-next: git: setup the worktree first before trying to ensure safe 2023-02-18 07:41:04 -03:00
Ricardo Pardini
5384a7dcbc
armbian-next: ignore git errors due to missing 'origin' remote and/or revisions of ${SRC}/.git during main-config 2023-02-18 07:41:03 -03:00
Ricardo Pardini
85009a6431
armbian-next: git worktree: forcefully fix the worktree's reference to the bare repo, and vice-versa 2023-02-18 07:41:02 -03:00
Ricardo Pardini
e607e9e54e
armbian-next: remove spurious output.log redirections in downloads.sh; warn that download_and_verify() is unhandled in armbian-next 2023-02-18 07:41:01 -03:00
Ricardo Pardini
bd04f71a05
armbian-next: Revert "Fix issues 4589 (#4614)" (installing HEADERS dependencies manually)
- This reverts commit 36d9420d
- this makes no sense
- we don't do adhoc installs in armbian-next
2023-02-18 07:41:00 -03:00
Ricardo Pardini
9200a2ce0c
armbian-next: bring find_toolchain() back from the dead; force SKIP_EXTERNAL_TOOLCHAINS=no to force use external toolchains
- handle a lot of corner cases, some toolchains are i386-only and won't run on modern x64 systems
2023-02-18 07:40:58 -03:00
Ricardo Pardini
39ac9e00b1
armbian-next: docker: don't try to double init when recovering from failed pull 2023-02-18 07:40:57 -03:00
Ricardo Pardini
7842113c1d
armbian-next: make sure at least one board config file is sourced, otherwise misspell BOARD= is hard to find 2023-02-18 07:40:56 -03:00
Ricardo Pardini
770d508ca0
armbian-next: Python patching delusion pt 5 + EXTRAWIFI patch generator/harness + consistent patch mtime targets
- Python patching: tune some logging all-around
- Python patching: *FINALLY* set the dates on the patched files to `max(patch_date, root_makefile_date)` thus making lighting-fast rebuilds again
- new EXTRAWIFI patch generator harness; Python patching EXTRA_PATCH_FILES_FIRST
- Python patching: use temp file for patching rejects; clean it up
- new EXTRAWIFI patch generator harness; Python: mark drivers as autogen, don't split or parse them, read as bytes, apply bytes directly
- new EXTRAWIFI patch generator harness; somewhat-works, but patches are per-family
- Python patching: add the `cache/patch` as a mountpoint
  - Darwin Docker performance is 20x with namedvolume; the cached patches are very large
2023-02-18 07:40:55 -03:00
Ricardo Pardini
97f6836705
armbian-next: Python patching delusion, pt4: series & better markdown
- Python patching: hopefully better Markdown: split series dir; detect more status/problems; arche only for missing Subject
  - Python patching: archeology only for missing Subject:, not description; clarify CLI pushing
  - Python patching: use `{file_base_name}.patch` for archeology as `file_name` might include a dir
- Python patching: fix: don't skip board/target patches for no reason
- Python patching: fix for series.conf patches (sunxi/sunxi64): don't sort; mark as series and show on summary
- Python patching: don't fail if archeology found no commits (but add warning)
- Python patching: CLI command `kernel-patches-to-git` with archeology summary, and git pushing of results & summary
  - Python patching: patches-to-git small fixes, auto-push if it's rpardini
  - Python patching: add `patch-kernel` CLI command
- Python patching: commit README.md & gh-pages workflow when apply_patches_to_git
  - Python patching: hopefully better markdown
- Python patching: `git add` everything all at once, for speed
2023-02-18 07:40:54 -03:00
Ricardo Pardini
2c0e9182ed
armbian-next: introduce ORAS-based kernel git bare tree seeding/bundles
- armbian-next: fixes to ORAS-related logging
- armbian-next: actually enable ORAS-based kernel git bare tree seeding/bundles; enable cleaning of bundle articfacts after confirmed working
- armbian-next: introduce ORAS-based kernel git bare tree seeding/bundles (although it is a .tar, not a bundle); this is 20x faster than cloning
- armbian-next: introduce `ORAS` tooling; pull and push functions & downloader/launcher
2023-02-18 07:40:53 -03:00
Ricardo Pardini
76e276c6a9
armbian-next: Python patching delusion, pt1 & pt2 & pt3
- WiP: Python patching delusion, pt 1: finding & parsing patches; apply & git commit with pygit2; Markdown summaries (also for aggregation); git-to-patches tool
  - Python: Markdown aggregation and patching summaries; collapsible; SummarizedMarkdownWriter
  - Python: Markdown aggregation and patching summaries
  - Python: reorg a bit into common/armbian_utils; define the `ASSET_LOG_BASE` in preparation for Markdown delusion
  - Python patching: initial apply patches & initial commit patches to git (using pygit2)
  - Python patching: add basic `series.conf` support
  - Python patching: force use of utf-8; better error handling; use realpath of dirs
  - Python patching: `git-to-patches` initial hack. not proud. half-reused some of the patches-to-git
  - Python patching: "tag" the git commits with info for extracting later; introduce REWRITE_PATCHES/rewrite_patches_in_place
  - Python patching: commented-out, recover-bad-patches hacks
  - Python patching: shorten the signature
  - Python patching: allow BASE_GIT_TAG as well as BASE_GIT_REVISION
  - Python patching: git-archeology for patches missing descriptions; avoid UTF-8 in header/desc (not diff)
  - Python patching: use modern-er email.utils.parsedate_to_datetime to parse commit date
  - Python patching: unify PatchInPatchFile; better git-commiting; re-exporting patches from Git (directly)
  - Python patching: switch to GitPython
    - GitPython is like 100x slower than pygit2, but actually allows for date & committer
    - also allows to remove untracked files before starting
  - Python aggregation: fix missing `AGGREGATED_APT_SOURCES_DICT`
  - Python patching: add `unidecode` dependency to pip3 install
  - Python patching: don't try archeology if SRC is not a Git Repo (eg, in Docker)
  - Python patching: don't try archeology if not applying patches to git
- WiP: Python patching delusion, pt2: actually use for u-boot & kernel patching
  - Python patching: much better problem handling/logging; lenient with recreations (kernel)
  - Python patching: don't force SHOW_LOG for u-boot patching
  - Python patching: don't bomb for no reason when there are no patches to apply
  - Python patching: fully (?) switch kernel patching to Python
  - Python patching: more logging fixups
  - Python patching: capture `kernel_git_revision` from `fetch_from_repo()`'s `checked_out_revision`
  - Python patching: fully switch u-boot patching to Python
  - Python aggregation/patching: colored logging; patching: always reset to git revision
  - Python aggregation/patching: better logging; introduce u-boot Python patching
- Python patching pt3: recovers and better Markdown
  - Python patching: detect, and rescue, `wrong_strip_level` problem; don't try to export patches that didn't apply, bitch instead
  - Python patching: Markdown patching summary table, complete with emoji
  - Python patching: include the problem breakdown in Markdown summary
  - Python patching: sanity check against half-bare, half-mbox patches
  - Python patching: try to recover from 1) bad utf-8 encoded patches; 2) bad unidiff patches; add a few sanity checks
  - Python patching: try, and fail, to apply badly utf-8 encoded patches directly as bytes [reverted]
  - Python patching: try to recover from patch *parse* failures; show summary; better logging
      - set `GIT_ARCHEOLOGY=yes` to do archeology, default not

- armbian-next: Python `pip` dependencies handling, similar to `hostdeps`
  - same scheme for Dockerfile caching
  - @TODO: still using global/shared environment; should move to a dir under `cache` or some kinda venv
- WiP: add `python3-pip` to hostdeps; remove `python-setuptools`
  - remove `python-setuptools` (Python2, no longer exists in Sid) from hostdeps
  - add `python3-pip` to hostdeps; part of virtualenv saga
- WiP: split `kernel.sh` a bit, into `kernel-patching.sh`, `kernel-config.sh` and `kernel-make.sh`
  - `advanced_patch()`: rename vars for clarity; no real changes
- Python patching: introduce FAST_ARCHEOLOGY; still trying for Markdown links
2023-02-18 07:40:52 -03:00
Ricardo Pardini
c251073b78
armbian-next: introduce internal base tmp dir; check it early for sanity ref noexec and nodev mount options 2023-02-18 07:40:51 -03:00
Ricardo Pardini
0c5b10a77b
armbian-next: introduce SKIP_ARMBIAN_ROOTFS_CACHE which is =yes by default
- that is in addition to `SKIP_ARMBIAN_REPO` which is not set by default
2023-02-18 07:40:50 -03:00
Ricardo Pardini
b4da0c869c
armbian-next: a few fixes (shortcircuits...) and enhancements for the GUI building; thanks NicoD 2023-02-18 07:40:49 -03:00