Commit Graph

55 Commits

Author SHA1 Message Date
Ricardo Pardini
d95478281b
armbian-next: fix ANSI logs, use reset at strategic places 2023-02-18 07:44:06 -03:00
Ricardo Pardini
eb30aac310
armbian-next: extensions: move 'extensions.sh' into library; replace manual source with init function
- wraps globals in `extension_manager_declare_globals()` -- and make them actually global
- whoever wrote this original code is insane, add copyright
2023-02-18 07:44:05 -03:00
Ricardo Pardini
22074a0cde
armbian-next: logging: new log file / summary nomenclature 2023-02-18 07:44:01 -03:00
Ricardo Pardini
a6182bbd89
armbian-next: docker: archive logfiles again; warn & greet testers when archiving many old logfiles 2023-02-18 07:44:00 -03:00
Ricardo Pardini
72c61f2f68
armbian-next: replace bare sync invocations by introducing wait_for_disk_sync(); sync early at start of build
- wait 10s for sync call; otherwise warn user to be patient
- wait_for_sync "reason for sync" everywhere
- add @TODO for /run/user/0 etc and XDG_RUNTIME_DIR in chroot
2023-02-18 07:43:53 -03:00
Ricardo Pardini
8af14d4f8c
armbian-next: countdown: split off file for countdown functions (countdown-to-abort and countdown-to-continue)
- introduce `countdown.sh`, move `traps.sh::exit_if_countdown_not_aborted()` there, introduce `exit_if_countdown_not_aborted()`
2023-02-18 07:43:50 -03:00
Ricardo Pardini
8cedc82ec2
armbian-next: hopefully improved error logging; avoid logging errors up to 3x; subdued red stacktraces
- reorg `display_alert()` so more frequent levels are handled first
- improve some ANSI color usage, still a mess
- don't write failed commands stackstrace as warnings to screen -- just to file (yes it's redundant, except if in test)
- more useful / subdued stacktraces, to make actual error more obvious...
2023-02-18 07:43:43 -03:00
Ricardo Pardini
242a5dad1f
armbian-next: somewhat-better handling/logging when using Docker (don't show two logfiles to inspect)
- do not show path to Docker run-host's logfile, unless the Docker run itself didn't produce logs
- introduce `global_final_exit_code` to allow to maintain/preserve Docker's exit code without triggering an error host-side
2023-02-18 07:43:41 -03:00
Ricardo Pardini
bf3f346bee
armbian-next: still fighting tee leaking: now I think I won, again
- avoid using the "last resort, use lazy umount" by killing tee_pid _and descendants_ before hand. Works!
- introduce `get_descendants_of_pid_array()` using recursive trick to get all descendants
- do not `unset CURRENT_LOGFILE` -- during `logging_error_show_log()` (from SHOW_LOG=no days)
  - do `unset CURRENT_LOGFILE` immediately before processing logs -- although it magically survives. mysteries of traps in bash...
- `declare -g` instead of `export` for `start_logging_section()`
- don't `check_and_close_fd_13()` in `run_cleanup_handlers()`
- do run `check_and_close_fd_13()`, but only after archiving the old logs (in logging trap!)
- add a default error for unhandled TRAP's -- of which should be done
- fix reset of `global_tee_pid=0` in section-logging
2023-02-18 07:43:40 -03:00
Ricardo Pardini
ec7639e518
armbian-next: severe bug, there was no prefix-emoji for Windows 2023-02-18 07:43:33 -03:00
Ricardo Pardini
62d948b1b4
armbian-next: still fighting tee leaking under duress, and I think I won
- call `check_and_close_fd_13()` directly when `exit_with_error()` called, don't wait for trap
- store PID spawned by `do_with_logging()` under `$global_tee_pid`
- `check_and_close_fd_13()`:
  - sprinkle with `sync` to give a chance for stuff to die gracefully
  - check if `$global_tee_pid` is set and actually a running PID, if so, kill it
  - do not blindly "wait" for stuff, instead, kill it and wait for it to die -- 100% more likely to not hang ;-)
2023-02-18 07:43:31 -03:00
Ricardo Pardini
35fbb7ed6c
armbian-next: small cleanups: squash typos / add function keyword / add types / mark dead code 2023-02-18 07:43:28 -03:00
Ricardo Pardini
a6a5152695
armbian-next: chroot_sdcard_apt_get_update() to replace and better log chroot_sdcard_apt_get update
- also add retries where it was missing
- @TODO: desktop_postinstall() is a mess, needs aggregation work
2023-02-18 07:43:19 -03:00
Ricardo Pardini
2455c55dac
armbian-next: firmware: don't build -full firmware if not on CI/noninteractive and board's not going to use it
- move `fakeroot_dpkg_deb_build()` from runners.sh to new utils-dpkgdeb.sh
  - and make it debug show the size of the source
- clean off `.git` directory earlier for full build
- @TODO: get rid of the "hardlinks" funky biz in there and not-copy `.git` to begin with
2023-02-18 07:43:18 -03:00
Ricardo Pardini
48c0e9acf4
armbian-next: severe bug, countdown counted up 2023-02-18 07:43:16 -03:00
Ricardo Pardini
9d26ee1373
armbian-next: logging: squash nasty leaking file descriptor bug due to "tee"
- check and make sure the fd is closed before processing and/or deleting logs.
- change from fd 3 to 13, for extra "this is evil". is it? is 13 an evil number? you decide.
2023-02-18 07:43:14 -03:00
Ricardo Pardini
bf891d54d9
armbian-next: split prepare_host(), fix .tmp reset trap
- `prepare_host()`: split; do checks earlier and allow them to be interactive
- introduce `exit_if_countdown_not_aborted()` for "Low Disk Space" and other critical conditions
- split `prepare_host()` into interactive & non-interactive parts
- split off `clean_deprecated_mountpoints()` from prepare into `cleaning.sh`
- introduce and use `reset_uid_owner_non_recursive()` for `.tmp` reset in trap, to avoid disasters
- add more logging sections to default-build.sh, avoid unlogged parts
2023-02-18 07:43:10 -03:00
Ricardo Pardini
2d9f9216eb
armbian-next: introduce tmpfs-utils.sh; put LOGDIR and WORKDIR under tmpfs; set CCACHE_TEMPDIR under WORKDIR
- introduce generic `prepare_tmpfs_for()`, which manages it's own cleanup/dir removal
- use it for `WORKDIR` (which is `TMPDIR`) and `LOGDIR`
  - adapt previous cleanup handlers for those, so they delete their contents but not the dir itself (which might be mounted)
- also: make `ARMBIAN_LOG_CLI_ID` readonly together with other superglobals
- set `XDG_RUNTIME_DIR` & `XDG_RUNTIME_DIR` together with `TMPDIR`
- kernel-make.sh: pass `CCACHE_TEMPDIR` down to Kernel make (thanks @the-Going)
2023-02-18 07:43:09 -03:00
Ricardo Pardini
231e179703
armbian-next: traps: allow for cleanup handlers with arguments; unify around run_one_cleanup_handler()
- show error if trap handler fails
2023-02-18 07:43:06 -03:00
Ricardo Pardini
43b4f45018
armbian-next: introduce POOR_MAN_PROFILER=yes (under ANSI_COLOR=none)
- turns out EPOCHREALTIME can be comma separated, and have multiple leading zeroes
2023-02-18 07:43:00 -03:00
Ricardo Pardini
2a68cc6e9e
armbian-next: optimize for CONFIG_DEFS_ONLY=yes, skipping stacktraces/git info 2023-02-18 07:42:59 -03:00
Ricardo Pardini
ef36016ee4
armbian-next: logging: reinit logging after processing cmdline parameters
- so `DEBUG=yes` actually works
2023-02-18 07:42:34 -03:00
Ricardo Pardini
898d567ae0
armbian-oleg: if commands fail, write to log in bright red 2023-02-18 07:42:30 -03:00
Ricardo Pardini
68d8f10666
armbian-oleg: make chroot_sdcard_apt_get_install_dry_run() logging more useful by filtering the noise out 2023-02-18 07:42:29 -03:00
Ricardo Pardini
c7c7d2e0d6
armbian-oleg: drastically curb apt-get logging when running in chroot 2023-02-18 07:42:28 -03:00
Ricardo Pardini
2c0c968645
armbian-oleg: curb Python launcher logs 2023-02-18 07:42:24 -03:00
Ricardo Pardini
95ffd76b3b
armbian-oleg: curb apt-get's logging with -q / -qq 2023-02-18 07:42:23 -03:00
Ricardo Pardini
b03530dacd
armbian-oleg: better logs; only include non-empty .log files 2023-02-18 07:42:22 -03:00
Ricardo Pardini
d24f363187
armbian-oleg: logging: new logfile format; SHOW_LOG=yes default; introduce DEBUG=yes
- default `SHOW_LOG=yes` if user on terminal
- shortcut: `DEBUG=yes` defaults both `SHOW_LOG=yes` & `SHOW_DEBUG=yes`
- much simpler logging format for log file
- skip all display_alert()'s more verbose than DEBUG from logfile
  - set `DEBUG=yes` to log everything again
- skip Markdown assets from being included in ANSI log
- use of `ccze`
- introduce internal var for controlling COMMAND-level log
- remove unused `run_on_sdcard()` that somehow was left there
- drop `export_html_logs()` and `EXPORT_HTML_LOG` -- are no more
2023-02-18 07:42:20 -03:00
Ricardo Pardini
7493ce1b09
armbian-next: long overdue split of logging.sh 2023-02-18 07:42:19 -03:00
Ricardo Pardini
93c8ce99af
armbian-next: apt-cacher-ng is now optional and activated via MANAGE_ACNG=yes; drop NO_APT_CACHER
- acng is constant source of headaches
- we don't need it anymore, since we have local caching now
- You can also have acng running on a different machine/container and set `APT_PROXY_ADDR=xxx:3142` if you're running a build farm
2023-02-18 07:42:12 -03:00
Ricardo Pardini
f167864f2c
armbian-next: split of config-prepare into multiple functions, allowing for logging of all non-interactive sections
- introduce `do_with_conditional_logging()` which only starts logging sections if `do_logging=no`
- with this we should get complete logs (ofc except for the interactive sections)
2023-02-18 07:41:46 -03:00
Ricardo Pardini
59278d1869
armbian-next: remove spurious debug logging with no useful info "Error-related files found" 2023-02-18 07:41:32 -03:00
Ricardo Pardini
65da2a697f
armbian-next: remove @TODO for stuff already done 2023-02-18 07:41:12 -03:00
Ricardo Pardini
ddbb91653f
armbian-next: reset_uid_owner for the markdown log file too 2023-02-18 07:41:11 -03:00
Ricardo Pardini
52d89e132b
armbian-next: logging: add Markdown logging output
- armbian-next: logging: add Markdown logging output; collapsible
2023-02-18 07:40:40 -03:00
Ricardo Pardini
568f6aecd0
armbian-next: logging: helpers for handling special logging if stdout is a terminal (user-interactive)
- introduce `if_user_on_terminal_and_not_logging_add()`;
- also the opposite, `if_user_not_on_terminal_or_is_logging_add()`
- unset `CURRENT_LOGGING_SECTION` when exiting a logging section
- introduce `do_with_logging_unless_user_terminal()`
2023-02-18 07:40:36 -03:00
Ricardo Pardini
adce39d3b7
armbian-next: aggregation.py rewrite; introduce package-lists.sh; usage in core lib 2023-02-18 07:40:22 -03:00
Ricardo Pardini
b2d02071bd
armbian-next: better ccache logs/stats/Docker-support/use in u-boot/kernel
- introduce `SHOW_CCACHE=yes` for detailed ccache statistics and logging
- fix: kernel build is done under "env -i", so pass CCACHE_DIR down to Make if CCACHE_DIR is set...
- split from kernel.sh, show stats also for u-boot targets; show compile time
- add volume definition (under `${SRC}/cache/ccache`); auto-use that in Docker builds via `CCACHE_DIR`
- better logging for `do_with_ccache_statistics()`
- there's some CCACHE_DIR code from before; unify @TODO
2023-02-18 07:40:19 -03:00
Ricardo Pardini
93f0545519
armbian-next: the great aggregation rewrite: pt 0: SHOW_AGGREGATION debug logging 2023-02-18 07:40:13 -03:00
Ricardo Pardini
c4eaf49f3d
armbian-next: cleanup extensions logging, using display_alert new levels extensions (SHOW_EXTENSIONS=yes) and extensionstrace (SHOW_EXTENSIONS_TRACE=yes)
- DEBUG_EXTENSION_CALLS is gone. use SHOW_EXTENSIONS instead.
- LOG_ALL_HOOK_TRACES is gone. use SHOW_EXTENSIONS_TRACE instead
- LOG_ENABLE_EXTENSION is gone. use SHOW_DEBUG=yes instead
2023-02-18 07:39:54 -03:00
Ricardo Pardini
a612a03678
armbian-next: retry 3 times for loop device's size, then give up; introduce RETRY_RUNS in do_with_retries() 2023-02-18 07:39:47 -03:00
Ricardo Pardini
d24d3327a8
armbian-next: the great cli entrypoint (+docker) rewrite; introduce USE_LOCAL_APT_DEB_CACHE replacing apt-cacher-ng
- armbian-next: introduce `USE_LOCAL_APT_DEB_CACHE` (default `=yes`) as alternative/in addition to `apt-cacher-ng` (eg, in Docker)
  - this uses `cache/aptcache/${RELEASE}-${ARCH}` (in the host) for
      - apt cache, by bind-mounting it to `${SDCARD}/var/cache/apt` in the `chroot_sdcard_apt_get()` runner and its usages
      - debootstrap, by passing it `--cache-dir`
  - utility function to help understand what is happening to cache during usage
  - apt itself mantains this cache, removing old packages when new ones are installed. apt does this _by default_
      - introduce `DONT_MAINTAIN_APT_CACHE=yes` to skip out of automatic apt maintenance of apt cache, eg, during `remove`s
      - don't do `apt clean` and such if using local cache, that would clean the cache, not the chroot
  - clean up `install_deb_chroot()` a little, find an unrelated bug there
- WiP: the great cli entrypoint (+docker) rewrite, Phase 6: relaunching structure; re-pass ARMBIAN_BUILD_UUID; use ARMBIAN_COMMAND for log filename; fix for output/logs dir perms
- WiP: the great cli entrypoint (+docker) rewrite, Phase 5: cleanups 4/x; better logging, check & force `DEST_LANG`
- WiP: the great cli entrypoint (+docker) rewrite, Phase 5: cleanups 3/x; don't write to stderr in generated Dockerfile
  - it's `drastic red` on non-buildx dockers
- WiP: the great cli entrypoint (+docker) rewrite, Phase 5: cleanups 2/x, logging
- WiP: the great cli entrypoint (+docker) rewrite, Phase 5: cleanups 1/x
  - source configs in a logging section.
  - Docker: silent, fast retries to make sure `docker system df` works
  - shut-up `chown` (no `-v`) output related to  `SET_OWNER_TO_UID`
  - ask user to wait while `DESTIMG` is rsync'ed to `FINALDEST` -- it's potentially very slow
  - use green apple for Mac logging, instead of red apple which might imply error...
- WiP: the great cli entrypoint (+docker) rewrite, Phase 4: run as non-root, maybe-with-Docker
  - introduce `is_docker_ready_to_go()`; if it is, and we're not root, use Docker instead of sudo. <- GOOD IDEA? BAD IDEA? lol
  - introduce `SET_OWNER_TO_UID` var to be passed to Docker/sudo so written files are owned by the launching user, not root.
    - introduce `mkdir_recursive_and_set_uid_owner()` and `reset_uid_owner()` to reset owner based on `SET_OWNER_TO_UID`
    - use it for userpatches files created, logs, and output files, including images and debs.
  - @TODOs ref. `$SUDO_USER` which I think the old version of this?
  - add a lot of @TODOs, ref being able to relaunch something that's not `build` inside Docker, also add/change params and configs and command.
    - initially add `ARMBIAN_DOCKER_RELAUNCH_EXTRA_ARGS`
- WiP: the great cli entrypoint (+docker) rewrite, Phase 3: rpardini is demented, v3
- WiP: the great cli entrypoint (+docker) rewrite, Phase 2: rpardini is demented
- WiP: the great cli entrypoint (+docker) rewrite, Phase 1
- armbian-next: WiP: Docker: actually use the GHA-image as base; pull it every 24hs.
  - using image in my private repo.
  - this has significant speedup to "start building time" on the 1st run
  - move some Linux specific stuff to its own if
  - add comments and todo
- armbian-next: WiP: Docker, high-WiP, beginnings of Armbian mount dict, with linux/darwin preferences
- armbian-next: WiP: Docker, configure `BUILDKIT_COLORS`
- armbian-next: WiP: Docker, make docker image from Dockerfile more compact by flattening layers
- armbian-next: `logging`: add whale indicator if build running under Docker
- armbian-next: WiP: `docker`: working with `bookworm`, `sid`, and `jammy` on Darwin & Linux; works with `bullseye` on Linux only
- armbian-next: WiP: `docker`: force ARMBIAN_RUNNING_IN_CONTAINER both in Dockerfile and passed as `--env`; apt update and install in same layer; back to jammy
- armbian-next: introduce `armbian_is_running_in_container()` and `armbian_is_host_running_systemd()`, replacing `systemd-detect-virt` in multiple spots
- WiP: try with debian:bullseye -- can't detect docker at all
- armbian-next: WiP: 2nd stab at new Docker support; Darwin still works; Linux `docker.io` working
  - gen .dockerignore together with Dockerfile
  - split in funcs
  - hacks for Linux and `/dev/loop` stuff, CONTAINER_COMPAT=yes
  - mac still works, Linux stuff would break it but I if'fed
- armbian-next: the secrets of `CONTAINER_COMPAT` revealed; add size checking to check_loop_device() and avoid retry when `mknod`ing
  - this fails for the right reasons now, causing retries, which are then retried and work ;-)
  - this is related to building under Docker on Linux, using docker.io package (not docker-ce)
- armbian-next: remove `.dockerignore` and add it to `.gitignore`; it's going to be auto-generated
- armbian-next: `.dockerignore`: Docker context should only have minimal files and folders, to speed up Dockerfile build
  - IMPORTANT: `.dockerignore` is going to be generated from now on: so this is the last commit with changes before removal
-  armbian-next: WiP: initial stab at new Docker support; really run the passed cmdline; add Dockerfile to gitignore
-  armbian-next: WiP: initial stab at new Docker support; generate Dockerfile; introduce REQUIREMENTS_DEFS_ONLY
  - uses REQUIREMENTS_DEFS_ONLY
  - works on Docker Desktop on Mac;
  - linux TBA
- armbian-next: don't error out if `.git` not present; other small fixes
- armbian-next: general "work or at least don't misbehave when run on a very bare ubuntu:latest instance"
  - can't assume things, for example:
  - that `sudo` will be available; it might not, and might be already root, no reason to fail
  - that `/etc/timezone` will exist
  - that `systemd-detect-virt` will be available
  - that `git` will be available
  - that `locale-gen` will be available
2023-02-18 07:39:43 -03:00
Ricardo Pardini
8f26db138a
armbian-next: traps: don't duplicate error message/stacktrace when error occurs in subshell; instead indicate clearly SUBSHELL 2023-02-18 07:39:41 -03:00
Ricardo Pardini
775cd6c619
armbian-next: logging: don't bomb due to lack of git or zstdmt
- logging might run super early when dependencies are not installed
- or say in a container, without git
2023-02-18 07:39:40 -03:00
Ricardo Pardini
7b47e79780
armbian-next: introduce ARMBIAN_LOGS_TO_JOURNAL and ARMBIAN_LOGS_JOURNAL_IDENTIFIER (logs display_alert to journald) 2023-02-18 07:39:23 -03:00
Ricardo Pardini
2196e957f0
armbian-next: logs: avoid errors by cwd'ing to SRC 2023-02-18 07:39:13 -03:00
Ricardo Pardini
f07ec572d8
armbian-next: fix logging headers in HTML/ANSI log output 2023-02-18 07:39:09 -03:00
Ricardo Pardini
b0cb5c270f
armbian-next: traps: cleanups: run cleanups in the reverse order they were added; allow running one single cleanup and removing from list 2023-02-18 07:39:07 -03:00
Ricardo Pardini
8ce52db74e
armbian-next: back to ANSI logs by default; HTML logs only by EXPORT_HTML_LOG=yes 2023-02-18 07:39:01 -03:00