217 Commits

Author SHA1 Message Date
Ricardo Pardini
43f268fdcb
armbian-next: remove very old extlinux killswitch 2023-02-18 07:41:24 -03:00
Ricardo Pardini
c61b985417
armbian-next: better logging in distro-agnostic.sh
- still _a lot_ to fix
2023-02-18 07:41:23 -03:00
Ricardo Pardini
8f3fd355eb
armbian-next: tune logging in multiple places, there was a lot of redundant/un-needed info's 2023-02-18 07:41:22 -03:00
Ricardo Pardini
7987dad34d
armbian-next: Python tools (aggregation/patching): use separate pycache dir in ./cache; use unbuffered stdout/stderr 2023-02-18 07:41:21 -03:00
Ricardo Pardini
59031be6b3
armbian-next: only put systemd-resolved's symlink in place if _not_ using NetworkManager
- why is it even installed? dunno, but fix anyway.
2023-02-18 07:41:20 -03:00
Ricardo Pardini
f8b141cdf0
armbian-next: fix: only aggregate packages if building images (AR-1471) 2023-02-18 07:41:19 -03:00
Ricardo Pardini
690ca4df7c
armbian-next: add (trap) cleanup handler to reset ./tmp and ./output to original pre-sudo UID before exiting, even with failure/abort 2023-02-18 07:41:17 -03:00
Ricardo Pardini
9973010972
armbian-next: Python patching: pass $HOME/$PATH to Python, so git can find .gitconfig and Python can find git
- add sanity check before handling the Git repo in Python (call `status()` to make sure access is OK)
2023-02-18 07:41:16 -03:00
Ricardo Pardini
12b31afa51
armbian-next: git: mark submodule's dirs as safe before recursive fetch_from_repo() 2023-02-18 07:41:15 -03:00
Ricardo Pardini
3bc8afd593
armbian-next: refactor losetup -d ... into free_loop_device_insistent() which retries 10 times 2023-02-18 07:41:14 -03:00
Ricardo Pardini
f679b81b10
armbian-next: git: use --recurse-submodules=no for fetch; fetch_from_repo() handles submodules separately
- also remove `du` working copy size debugging
2023-02-18 07:41:13 -03:00
Ricardo Pardini
65da2a697f
armbian-next: remove @TODO for stuff already done 2023-02-18 07:41:12 -03:00
Ricardo Pardini
ddbb91653f
armbian-next: reset_uid_owner for the markdown log file too 2023-02-18 07:41:11 -03:00
Ricardo Pardini
2f9a4470c7
armbian-next: validate cmdline params against malformed
- eg: `./compile.sh $ERROR=here` is caught now
2023-02-18 07:41:10 -03:00
Ricardo Pardini
093e60d111
armbian-next: exec into sudo when relaunching (AR-1452)
- with `exec`, `sudo` replaces the current shell, and thus never returns
2023-02-18 07:41:09 -03:00
Ricardo Pardini
9370c19a44
armbian-next: drop old code for patching & fasthash
- split the modification time stuff to general
2023-02-18 07:41:05 -03:00
Ricardo Pardini
143ba996ca
armbian-next: git: setup the worktree first before trying to ensure safe 2023-02-18 07:41:04 -03:00
Ricardo Pardini
5384a7dcbc
armbian-next: ignore git errors due to missing 'origin' remote and/or revisions of ${SRC}/.git during main-config 2023-02-18 07:41:03 -03:00
Ricardo Pardini
85009a6431
armbian-next: git worktree: forcefully fix the worktree's reference to the bare repo, and vice-versa 2023-02-18 07:41:02 -03:00
Ricardo Pardini
e607e9e54e
armbian-next: remove spurious output.log redirections in downloads.sh; warn that download_and_verify() is unhandled in armbian-next 2023-02-18 07:41:01 -03:00
Ricardo Pardini
bd04f71a05
armbian-next: Revert "Fix issues 4589 (#4614)" (installing HEADERS dependencies manually)
- This reverts commit 36d9420d
- this makes no sense
- we don't do adhoc installs in armbian-next
2023-02-18 07:41:00 -03:00
Ricardo Pardini
9200a2ce0c
armbian-next: bring find_toolchain() back from the dead; force SKIP_EXTERNAL_TOOLCHAINS=no to force use external toolchains
- handle a lot of corner cases, some toolchains are i386-only and won't run on modern x64 systems
2023-02-18 07:40:58 -03:00
Ricardo Pardini
39ac9e00b1
armbian-next: docker: don't try to double init when recovering from failed pull 2023-02-18 07:40:57 -03:00
Ricardo Pardini
7842113c1d
armbian-next: make sure at least one board config file is sourced, otherwise misspell BOARD= is hard to find 2023-02-18 07:40:56 -03:00
Ricardo Pardini
770d508ca0
armbian-next: Python patching delusion pt 5 + EXTRAWIFI patch generator/harness + consistent patch mtime targets
- Python patching: tune some logging all-around
- Python patching: *FINALLY* set the dates on the patched files to `max(patch_date, root_makefile_date)` thus making lighting-fast rebuilds again
- new EXTRAWIFI patch generator harness; Python patching EXTRA_PATCH_FILES_FIRST
- Python patching: use temp file for patching rejects; clean it up
- new EXTRAWIFI patch generator harness; Python: mark drivers as autogen, don't split or parse them, read as bytes, apply bytes directly
- new EXTRAWIFI patch generator harness; somewhat-works, but patches are per-family
- Python patching: add the `cache/patch` as a mountpoint
  - Darwin Docker performance is 20x with namedvolume; the cached patches are very large
2023-02-18 07:40:55 -03:00
Ricardo Pardini
97f6836705
armbian-next: Python patching delusion, pt4: series & better markdown
- Python patching: hopefully better Markdown: split series dir; detect more status/problems; arche only for missing Subject
  - Python patching: archeology only for missing Subject:, not description; clarify CLI pushing
  - Python patching: use `{file_base_name}.patch` for archeology as `file_name` might include a dir
- Python patching: fix: don't skip board/target patches for no reason
- Python patching: fix for series.conf patches (sunxi/sunxi64): don't sort; mark as series and show on summary
- Python patching: don't fail if archeology found no commits (but add warning)
- Python patching: CLI command `kernel-patches-to-git` with archeology summary, and git pushing of results & summary
  - Python patching: patches-to-git small fixes, auto-push if it's rpardini
  - Python patching: add `patch-kernel` CLI command
- Python patching: commit README.md & gh-pages workflow when apply_patches_to_git
  - Python patching: hopefully better markdown
- Python patching: `git add` everything all at once, for speed
2023-02-18 07:40:54 -03:00
Ricardo Pardini
2c0e9182ed
armbian-next: introduce ORAS-based kernel git bare tree seeding/bundles
- armbian-next: fixes to ORAS-related logging
- armbian-next: actually enable ORAS-based kernel git bare tree seeding/bundles; enable cleaning of bundle articfacts after confirmed working
- armbian-next: introduce ORAS-based kernel git bare tree seeding/bundles (although it is a .tar, not a bundle); this is 20x faster than cloning
- armbian-next: introduce `ORAS` tooling; pull and push functions & downloader/launcher
2023-02-18 07:40:53 -03:00
Ricardo Pardini
76e276c6a9
armbian-next: Python patching delusion, pt1 & pt2 & pt3
- WiP: Python patching delusion, pt 1: finding & parsing patches; apply & git commit with pygit2; Markdown summaries (also for aggregation); git-to-patches tool
  - Python: Markdown aggregation and patching summaries; collapsible; SummarizedMarkdownWriter
  - Python: Markdown aggregation and patching summaries
  - Python: reorg a bit into common/armbian_utils; define the `ASSET_LOG_BASE` in preparation for Markdown delusion
  - Python patching: initial apply patches & initial commit patches to git (using pygit2)
  - Python patching: add basic `series.conf` support
  - Python patching: force use of utf-8; better error handling; use realpath of dirs
  - Python patching: `git-to-patches` initial hack. not proud. half-reused some of the patches-to-git
  - Python patching: "tag" the git commits with info for extracting later; introduce REWRITE_PATCHES/rewrite_patches_in_place
  - Python patching: commented-out, recover-bad-patches hacks
  - Python patching: shorten the signature
  - Python patching: allow BASE_GIT_TAG as well as BASE_GIT_REVISION
  - Python patching: git-archeology for patches missing descriptions; avoid UTF-8 in header/desc (not diff)
  - Python patching: use modern-er email.utils.parsedate_to_datetime to parse commit date
  - Python patching: unify PatchInPatchFile; better git-commiting; re-exporting patches from Git (directly)
  - Python patching: switch to GitPython
    - GitPython is like 100x slower than pygit2, but actually allows for date & committer
    - also allows to remove untracked files before starting
  - Python aggregation: fix missing `AGGREGATED_APT_SOURCES_DICT`
  - Python patching: add `unidecode` dependency to pip3 install
  - Python patching: don't try archeology if SRC is not a Git Repo (eg, in Docker)
  - Python patching: don't try archeology if not applying patches to git
- WiP: Python patching delusion, pt2: actually use for u-boot & kernel patching
  - Python patching: much better problem handling/logging; lenient with recreations (kernel)
  - Python patching: don't force SHOW_LOG for u-boot patching
  - Python patching: don't bomb for no reason when there are no patches to apply
  - Python patching: fully (?) switch kernel patching to Python
  - Python patching: more logging fixups
  - Python patching: capture `kernel_git_revision` from `fetch_from_repo()`'s `checked_out_revision`
  - Python patching: fully switch u-boot patching to Python
  - Python aggregation/patching: colored logging; patching: always reset to git revision
  - Python aggregation/patching: better logging; introduce u-boot Python patching
- Python patching pt3: recovers and better Markdown
  - Python patching: detect, and rescue, `wrong_strip_level` problem; don't try to export patches that didn't apply, bitch instead
  - Python patching: Markdown patching summary table, complete with emoji
  - Python patching: include the problem breakdown in Markdown summary
  - Python patching: sanity check against half-bare, half-mbox patches
  - Python patching: try to recover from 1) bad utf-8 encoded patches; 2) bad unidiff patches; add a few sanity checks
  - Python patching: try, and fail, to apply badly utf-8 encoded patches directly as bytes [reverted]
  - Python patching: try to recover from patch *parse* failures; show summary; better logging
      - set `GIT_ARCHEOLOGY=yes` to do archeology, default not

- armbian-next: Python `pip` dependencies handling, similar to `hostdeps`
  - same scheme for Dockerfile caching
  - @TODO: still using global/shared environment; should move to a dir under `cache` or some kinda venv
- WiP: add `python3-pip` to hostdeps; remove `python-setuptools`
  - remove `python-setuptools` (Python2, no longer exists in Sid) from hostdeps
  - add `python3-pip` to hostdeps; part of virtualenv saga
- WiP: split `kernel.sh` a bit, into `kernel-patching.sh`, `kernel-config.sh` and `kernel-make.sh`
  - `advanced_patch()`: rename vars for clarity; no real changes
- Python patching: introduce FAST_ARCHEOLOGY; still trying for Markdown links
2023-02-18 07:40:52 -03:00
Ricardo Pardini
c251073b78
armbian-next: introduce internal base tmp dir; check it early for sanity ref noexec and nodev mount options 2023-02-18 07:40:51 -03:00
Ricardo Pardini
0c5b10a77b
armbian-next: introduce SKIP_ARMBIAN_ROOTFS_CACHE which is =yes by default
- that is in addition to `SKIP_ARMBIAN_REPO` which is not set by default
2023-02-18 07:40:50 -03:00
Ricardo Pardini
b4da0c869c
armbian-next: a few fixes (shortcircuits...) and enhancements for the GUI building; thanks NicoD 2023-02-18 07:40:49 -03:00
Ricardo Pardini
b606ae9715
armbian-next: docker: fix when using buildx, always pass --load so produced image is actually available to Docker proper 2023-02-18 07:40:48 -03:00
Ricardo Pardini
2d563312bd
armbian-next: fixes for interactive config; change wording of stuff and sort the list of boards in "expert" mode 2023-02-18 07:40:47 -03:00
Ricardo Pardini
66747a12e1
armbian-next: disable cpufrequtils by default; to enable set CPUFREQUTILS_ENABLE=true [HIGHLY CONTENTIOUS CHANGE] 2023-02-18 07:40:46 -03:00
Ricardo Pardini
c2af130a94
armbian-next: fix all shellcheck errors in lib/ 2023-02-18 07:40:44 -03:00
Ricardo Pardini
fd043d707a
armbian-next: squash some warn loggings that were undue; remove $SUDO_USER handling in prepare-host.sh
- squash warning about `NO_APT_CACHER` when in Docker
2023-02-18 07:40:43 -03:00
Ricardo Pardini
b386e4615e
armbian-next: introduce INSTALL_HOST_TOOLS=no to skip rkbin, fips, etc. for fast(er) development 2023-02-18 07:40:42 -03:00
Ricardo Pardini
50b6b232d9
armbian-next: git: fetch_from_repo(): determine, use for checkout, and set the SHA1 rev in checked_out_revision var 2023-02-18 07:40:41 -03:00
Ricardo Pardini
52d89e132b
armbian-next: logging: add Markdown logging output
- armbian-next: logging: add Markdown logging output; collapsible
2023-02-18 07:40:40 -03:00
Ricardo Pardini
074857ede3
armbian-next: u-boot: use git worktree for u-boot; full mainline clone for bare
- share a single `.git` for all u-boots
- all fetches are done against it, maximizing cache hit ratio drastically
- also reduces the size of each working copy by more than 70%
- split uboot's git stuff into uboot-git.sh
- split `uboot_prepare_git()` from `compile_uboot()`
2023-02-18 07:40:39 -03:00
Ricardo Pardini
84c0bf0f7d
armbian-next: kernel: use git worktree & kernel.org clone bundles for kernel working copies
- now we've a single `.git` for all kernels with all branches; a "master" cache
- back to full stable git bundle usage from kernel.org
- bye bye, "gitballs"
- split kernel-related git stuff into `kernel-git.sh`
- introduce `ARMBIAN_MOUNTPOINTS_DEPRECATED` @TODO actually use for cleaning non-Docker
- add `axel` hostdep for multi-conn download of bundle, which comes from a CDN.
- deprecate old mountpoints for gitballs and linux-kernel
- move cleaning of old sources/kernel to mountpoints code
2023-02-18 07:40:38 -03:00
Ricardo Pardini
a51448a1da
armbian-next: non-Docker cleaning of deprecated mountpoints (before general_cleaning) 2023-02-18 07:40:37 -03:00
Ricardo Pardini
568f6aecd0
armbian-next: logging: helpers for handling special logging if stdout is a terminal (user-interactive)
- introduce `if_user_on_terminal_and_not_logging_add()`;
- also the opposite, `if_user_not_on_terminal_or_is_logging_add()`
- unset `CURRENT_LOGGING_SECTION` when exiting a logging section
- introduce `do_with_logging_unless_user_terminal()`
2023-02-18 07:40:36 -03:00
Ricardo Pardini
9657ceac9b
armbian-next: docker: implement docker-purge and docker-shell cli commands; deprecated mountpoints 2023-02-18 07:40:35 -03:00
Ricardo Pardini
277ce40e6f
armbian-next: fix for grub's DISTRO_GENERIC_KERNEL on Debian (not Ubuntu) during update-initramfs
- `initrd`: when kernel not found, list the installed kernels
- armbian-next: small fix for `grub` extension naming
2023-02-18 07:40:34 -03:00
Ricardo Pardini
4a9529dd34
armbian-next: qemu-uboot-arm64 and qemu-uboot-x86 new boards
- sharing most UEFI code, will replace the `virtual` one soon
- x86: patch uboot defconfig to use the `q35` machine type, not `i440fx`
  - separate x86 bootscript, due to non-uInitrd-ness of it
  - hack ramdisk load address both in u-boot source and bootscript
  - use 32-bit u-boot, not 64-bit
- grub: introduce `UEFI_GRUB=skip`, does not deploy GRUB (but does the kernel packages, etc)
- auto-enable qcow2 output for these
- works with both distro's and Armbian's kernels
2023-02-18 07:40:33 -03:00
Ricardo Pardini
b951f7b873
armbian-next: u-boot: introduce hook pre_config_uboot_target()
- useful for hacking at Python versions and other compilation-time hacks
2023-02-18 07:40:32 -03:00
Ricardo Pardini
42e26e9ade
armbian-next: hostdeps: python2/3: add setuptools (required by some u-boot configs) 2023-02-18 07:40:31 -03:00
Ricardo Pardini
a752ac8899
armbian-next: REVERT: remove the BUILD_ONLY build-tasks.sh which makes no sense in armbian-next 2023-02-18 07:40:29 -03:00
Ricardo Pardini
64e298f78f
armbian-next: git: add origin remote if it is empty 2023-02-18 07:40:28 -03:00