Commit Graph

75 Commits

Author SHA1 Message Date
Ricardo Pardini
f75b9c110f aggregation: fix: don't eat value if empty during bash_string_multiline(); write end-of-file comment; show aggregation bash results if debugging 2023-03-31 09:12:44 +02:00
Ricardo Pardini
a77d5f3c70 fix: declare some variables that were leaking into global namespace for no reason
- especially `for VAR in ...` and VAR leaks sans the declare
- rename `fragment_manager_cleanup_file` to `extension_manager_cleanup_file`
2023-03-31 09:12:44 +02:00
Ricardo Pardini
630ee512cb stop the build if LIB_TAG is set at all; some users expect it to work while it doesn't 2023-03-17 08:32:11 +01:00
Ricardo Pardini
6b965c605c completely remove any traces of KERNEL_ONLY and exit with error if set 2023-03-17 08:32:11 +01:00
Ricardo Pardini
ff657fcf90 artifacts (all): require prefixing of artifact_version with artifact_prefix_version (which is ${REVISION}--) 2023-03-17 08:32:11 +01:00
Igor Pečovnik
b7b8eb7b72
Add / modify (c) in bash scripts (#4922)
* Add / modify (c) in bash scripts

Signed-off-by: Igor <igor@armbian.com>

* Add (c) to the source config files

---------

Signed-off-by: Igor <igor@armbian.com>
2023-03-09 18:30:40 +01:00
Igor Pečovnik
b7921eb53e
Rename config file "only_for" into "architecture" (#4861)
* Rename only_for def into architectures
* Rename def with typo arhitectures -> architectures
2023-02-24 20:36:27 +01:00
Ricardo Pardini
68e20a4bf4
qemu-uboot-xxx: fix missed rootfs cache due to pkg removals 2023-02-18 07:46:41 -03:00
Ricardo Pardini
ca6bf9046b
armbian-next: remove all traces of Vagrant and FEL boot 2023-02-18 07:46:00 -03:00
Ricardo Pardini
49947ec373
armbian-next: aggregation: don't include packages.additional if BUILD_MINIMAL=yes 2023-02-18 07:45:58 -03:00
Ricardo Pardini
70e68b619c
armbian-next: swap github in as default u-boot mirror; allow denx to use upstream's gitlab
- @TODO still need to change mirror swap to usage instead of config
2023-02-18 07:45:37 -03:00
Ricardo Pardini
388c76c91b
armbian-next: shellcheck: squash more shellcheck warnings; some long-lost variables being used; some unset's that are better reset's, etc 2023-02-18 07:45:33 -03:00
Ricardo Pardini
68d76d6f3a
armbian-next: squash some shellcheck warnings; don't local and assign subshell value in same statement, that masks errors (declare/local almost-always works) 2023-02-18 07:45:31 -03:00
Ricardo Pardini
bb6975707a
armbian-next: take a breath: config might mark aggregation required; aggregation requires prepare_host, and thus runs in default-build 2023-02-18 07:45:21 -03:00
Ricardo Pardini
bef7f8fee4
armbian-next: artifacts: introduce kernel/u-boot artifacts; git ref2info; "memoizer"; some hashing
- artifacts: u-boot/kernel - pt7 - adapt legacy/artifact versions; use common `capture_rename_legacy_debs_into_artifacts()`
- artifacts: u-boot - pt6: add artifact for u-boot
  - use artifact version / reason in actual u-boot .deb if present
- artifacts: kernel - pt5: tune kernel version, refactor
- artifacts: kernel - pt4: squash unrelated bugs that show up; move `prepare_compilation_vars()` to default build
- artifacts: kernel - pt3: drivers+patches+.config hashing
  - split file hashing function from drivers-harness; fix it so filenames are relative and sorted; sort from ${SRC}, always
  - aplit prepare_kernel_config_core_or_userpatches() from `kernel_config_initialize()`
- artifacts: kernel - pt2:  memoizing git ref2info
- artifacts: kernel - pt1:  versioning
2023-02-18 07:45:20 -03:00
Ricardo Pardini
19aa7e95cf
armbian-next: introduce assert_prepared_host() and assert before using any Python, to un-mask any further fallacies 2023-02-18 07:45:19 -03:00
Ricardo Pardini
16eb5eb50b
armbian-next: aggregate_packages out of main_default_start_build(); introduce assert_requires_aggregation() 2023-02-18 07:45:03 -03:00
Ricardo Pardini
78dcf01c92
armbian-next: cleanup, kill and add to-do's, remove dead code, turn down logging, squash future shortcircuit bugs
- killed `[[ $ROOTFS_TYPE != ext4 ]] && display_alert "Assuming ${BOARD} ${BRANCH} kernel supports ${ROOTFS_TYPE}" "" "wrn"`
  - which definitely didn't belong in rootfs
- disable usage of run_host_command_logged_long_running for kernel-make
- stop lying about long_running stuff being any different from non-long_running versions
- stop lying about 'set -e' when there's still a bunch of pipes for "pv" and stuff all around
2023-02-18 07:44:33 -03:00
Ricardo Pardini
d17157885c
armbian-next: rootfs: converge rootfs-cache and rootfs together; exchange BOARD for ARCH 2023-02-18 07:44:28 -03:00
Ricardo Pardini
6077d40f9b
armbian-next: rootfs: build rootfs from CLI, sans-BOARD, sans-FAMILY, etc -- rootfs are meant to be shared
- require ARCH/RELEASE for rootfs cli
- disallow BOARD/LINUXFAMILY/BOARDFAMILY being set for rootfs cli
- split `prep_conf_main_only_rootfs()` instead of `prep_conf_main_build_single()` for rootfs-only stuff
- run both conf and build without stdin for rootfs CLI (no interactive allowed)
- don't blindly set `ARCH=armhf` for no reason in main-config
- allow skipping family stuff in main-config, via `allow_no_family=yes` (used only in `prep_conf_main_only_rootfs()`)
- kill double loading of userpatches family, not needed/old code
- allow skipping kernel stuff via `skip_kernel=yes` in `config_pre_main()`  (used only in `prep_conf_main_only_rootfs()`)
2023-02-18 07:44:26 -03:00
Ricardo Pardini
28d89e2c91
armbian-next: rootfs: new CLI command for rootfs cache building; further refactorings
- _told ya it would all make sense eventually..._
- introduce new, real, non-alias `rootfs` command in `cli-rootfs` -- this _only_ builds rootfs, nothing else
- no more `main_default_build_single()`; instead `full_build_packages_rootfs_and_image()`
- introduce `do_with_default_build()` wrapper, for things that need host prepared + aggregation to run
- CLI-specific parts of `main_default_build_single()` moved to cli-build
- put prepare/cleanup pair of `prepare_rootfs_build_params_and_trap()` and `trap_handler_cleanup_rootfs_and_image()` into new `trap-rootfs.sh` (this needs further splitting for the image part)
- refactor `get_or_create_rootfs_cache_chroot_sdcard()` into `build_rootfs_only()`, `calculate_rootfs_cache_id()`
- update library with new files
2023-02-18 07:44:22 -03:00
Ricardo Pardini
7f16064c23
armbian-next: interactive: info msg before (slow) gathering of all boards 2023-02-18 07:44:12 -03:00
Ricardo Pardini
2c9582f9ba
armbian-next: add Debian Ports keyring hostdep to strap Debian riscv64 from ports
- rpardini: actually use ports keyring for Debian Risc-V
- `risc-v`: correctly include `debian-ports-archive-keyring` in Risc-V CLI packages
  - include in debootstrap list, via `config/optional/architectures/riscv64/_config/cli/_all_distributions/main/packages`
  - can't have two `--include` in debootstrap call
- riscv64: move `debian-ports-archive-keyring` from main to debootstrap; also only for `sid`
2023-02-18 07:43:49 -03:00
Ricardo Pardini
2a68cc6e9e
armbian-next: optimize for CONFIG_DEFS_ONLY=yes, skipping stacktraces/git info 2023-02-18 07:42:59 -03:00
Ricardo Pardini
7879895049
armbian-next: core extensions for xfs / f2fs / brtfs / LUKS/cryptroot hostdeps; enable in main-config 2023-02-18 07:42:46 -03:00
Ricardo Pardini
7beeae6219
armbian-next: Python tooling: use consolidated+hashed+cached pip base/pycache; don't pip during Dockerfile build, nor cli-requirements
- consolidate at `prepare_python_and_pip()`
- sanity check for Python version 3.9+ regardless of HOSTRELEASE
- TODO: pip vs sudo/root: need pip 22.2+ to curb warning, not doing it
2023-02-18 07:42:44 -03:00
Ricardo Pardini
461f82f95a
armbian-next: completely remove acl (which provided getfacl) and it's usages (basic deps) 2023-02-18 07:42:42 -03:00
Ricardo Pardini
b5a777c4ac
armbian-next: remove REPO_STORAGE and REPO_CONFIG -- unused 2023-02-18 07:42:40 -03:00
Ricardo Pardini
2c0c968645
armbian-oleg: curb Python launcher logs 2023-02-18 07:42:24 -03:00
Ricardo Pardini
b03530dacd
armbian-oleg: better logs; only include non-empty .log files 2023-02-18 07:42:22 -03:00
Ricardo Pardini
93c8ce99af
armbian-next: apt-cacher-ng is now optional and activated via MANAGE_ACNG=yes; drop NO_APT_CACHER
- acng is constant source of headaches
- we don't need it anymore, since we have local caching now
- You can also have acng running on a different machine/container and set `APT_PROXY_ADDR=xxx:3142` if you're running a build farm
2023-02-18 07:42:12 -03:00
Ricardo Pardini
d4d3c1346c
armbian-oleg: split hostdeps again, full of ifs, and reduce the minimum set of pkgs for Oleg-conditions while trying to keep Docker full
- probably breaks a lot of stuff; c++ compilers, distcc, possibly others
- split into late_prepare_host_dependencies / early_prepare_host_dependencies calling adaptative_prepare_host_dependencies() full of if's

also:
- calc deps out of configuration
- reorg checks into obtain_and_check_host_release_and_arch()
2023-02-18 07:42:11 -03:00
Ricardo Pardini
b380ba4d86
armbian-next: further split of config-prepare() and do_main_configuration() into do_extra_configuration() - fixes interactive desktop 2023-02-18 07:42:03 -03:00
Ricardo Pardini
d152574ea8
armbian-next: move aggregation to after prepare_host(), and thus out of config, and with correct Python deps 2023-02-18 07:41:53 -03:00
Ricardo Pardini
6cb12a3397
armbian-next: [focal-host] try to support focal-like host OS's again, via python3.9 (and drop buster completely)
- completely remove support for building under `buster` -- that's way too old, sorry.
- de-hardcode `python3` invocations, instead use `python3_binary_path` set by `prepare_python3_binary_for_python_tools()`
- juggle `$HOSTRELEASE`: read from actual host, or determined from Docker image name (during Dockerfile build)
- TL;DR: include and use `python3.9` for focal-like host OS's
2023-02-18 07:41:47 -03:00
Ricardo Pardini
f167864f2c
armbian-next: split of config-prepare into multiple functions, allowing for logging of all non-interactive sections
- introduce `do_with_conditional_logging()` which only starts logging sections if `do_logging=no`
- with this we should get complete logs (ofc except for the interactive sections)
2023-02-18 07:41:46 -03:00
Ricardo Pardini
c33b59d61b
armbian-next: config: reduce bash scope & remove some vars un-needed outside of config 2023-02-18 07:41:40 -03:00
Ricardo Pardini
f252138693
armbian-next: call hook post_aggregate_packages() only after actually aggregating 2023-02-18 07:41:38 -03:00
Ricardo Pardini
6c027ba552
armbian-next: delete old aggregation code from config 2023-02-18 07:41:37 -03:00
Ricardo Pardini
67108e31c0
armbian-next: move var HOSTRELEASE from config to prepare_host() 2023-02-18 07:41:36 -03:00
Ricardo Pardini
ec2e6c42be
armbian-next: happy new year, bump Igor's copyright 2023-02-18 07:41:25 -03:00
Ricardo Pardini
43f268fdcb
armbian-next: remove very old extlinux killswitch 2023-02-18 07:41:24 -03:00
Ricardo Pardini
8f3fd355eb
armbian-next: tune logging in multiple places, there was a lot of redundant/un-needed info's 2023-02-18 07:41:22 -03:00
Ricardo Pardini
7987dad34d
armbian-next: Python tools (aggregation/patching): use separate pycache dir in ./cache; use unbuffered stdout/stderr 2023-02-18 07:41:21 -03:00
Ricardo Pardini
f8b141cdf0
armbian-next: fix: only aggregate packages if building images (AR-1471) 2023-02-18 07:41:19 -03:00
Ricardo Pardini
5384a7dcbc
armbian-next: ignore git errors due to missing 'origin' remote and/or revisions of ${SRC}/.git during main-config 2023-02-18 07:41:03 -03:00
Ricardo Pardini
76e276c6a9
armbian-next: Python patching delusion, pt1 & pt2 & pt3
- WiP: Python patching delusion, pt 1: finding & parsing patches; apply & git commit with pygit2; Markdown summaries (also for aggregation); git-to-patches tool
  - Python: Markdown aggregation and patching summaries; collapsible; SummarizedMarkdownWriter
  - Python: Markdown aggregation and patching summaries
  - Python: reorg a bit into common/armbian_utils; define the `ASSET_LOG_BASE` in preparation for Markdown delusion
  - Python patching: initial apply patches & initial commit patches to git (using pygit2)
  - Python patching: add basic `series.conf` support
  - Python patching: force use of utf-8; better error handling; use realpath of dirs
  - Python patching: `git-to-patches` initial hack. not proud. half-reused some of the patches-to-git
  - Python patching: "tag" the git commits with info for extracting later; introduce REWRITE_PATCHES/rewrite_patches_in_place
  - Python patching: commented-out, recover-bad-patches hacks
  - Python patching: shorten the signature
  - Python patching: allow BASE_GIT_TAG as well as BASE_GIT_REVISION
  - Python patching: git-archeology for patches missing descriptions; avoid UTF-8 in header/desc (not diff)
  - Python patching: use modern-er email.utils.parsedate_to_datetime to parse commit date
  - Python patching: unify PatchInPatchFile; better git-commiting; re-exporting patches from Git (directly)
  - Python patching: switch to GitPython
    - GitPython is like 100x slower than pygit2, but actually allows for date & committer
    - also allows to remove untracked files before starting
  - Python aggregation: fix missing `AGGREGATED_APT_SOURCES_DICT`
  - Python patching: add `unidecode` dependency to pip3 install
  - Python patching: don't try archeology if SRC is not a Git Repo (eg, in Docker)
  - Python patching: don't try archeology if not applying patches to git
- WiP: Python patching delusion, pt2: actually use for u-boot & kernel patching
  - Python patching: much better problem handling/logging; lenient with recreations (kernel)
  - Python patching: don't force SHOW_LOG for u-boot patching
  - Python patching: don't bomb for no reason when there are no patches to apply
  - Python patching: fully (?) switch kernel patching to Python
  - Python patching: more logging fixups
  - Python patching: capture `kernel_git_revision` from `fetch_from_repo()`'s `checked_out_revision`
  - Python patching: fully switch u-boot patching to Python
  - Python aggregation/patching: colored logging; patching: always reset to git revision
  - Python aggregation/patching: better logging; introduce u-boot Python patching
- Python patching pt3: recovers and better Markdown
  - Python patching: detect, and rescue, `wrong_strip_level` problem; don't try to export patches that didn't apply, bitch instead
  - Python patching: Markdown patching summary table, complete with emoji
  - Python patching: include the problem breakdown in Markdown summary
  - Python patching: sanity check against half-bare, half-mbox patches
  - Python patching: try to recover from 1) bad utf-8 encoded patches; 2) bad unidiff patches; add a few sanity checks
  - Python patching: try, and fail, to apply badly utf-8 encoded patches directly as bytes [reverted]
  - Python patching: try to recover from patch *parse* failures; show summary; better logging
      - set `GIT_ARCHEOLOGY=yes` to do archeology, default not

- armbian-next: Python `pip` dependencies handling, similar to `hostdeps`
  - same scheme for Dockerfile caching
  - @TODO: still using global/shared environment; should move to a dir under `cache` or some kinda venv
- WiP: add `python3-pip` to hostdeps; remove `python-setuptools`
  - remove `python-setuptools` (Python2, no longer exists in Sid) from hostdeps
  - add `python3-pip` to hostdeps; part of virtualenv saga
- WiP: split `kernel.sh` a bit, into `kernel-patching.sh`, `kernel-config.sh` and `kernel-make.sh`
  - `advanced_patch()`: rename vars for clarity; no real changes
- Python patching: introduce FAST_ARCHEOLOGY; still trying for Markdown links
2023-02-18 07:40:52 -03:00
Ricardo Pardini
b4da0c869c
armbian-next: a few fixes (shortcircuits...) and enhancements for the GUI building; thanks NicoD 2023-02-18 07:40:49 -03:00
Ricardo Pardini
2d563312bd
armbian-next: fixes for interactive config; change wording of stuff and sort the list of boards in "expert" mode 2023-02-18 07:40:47 -03:00
Ricardo Pardini
adce39d3b7
armbian-next: aggregation.py rewrite; introduce package-lists.sh; usage in core lib 2023-02-18 07:40:22 -03:00