4 Commits

Author	SHA1	Message	Date
Ricardo Pardini	76e276c6a9	armbian-next: Python patching delusion, pt1 & pt2 & pt3 ... - WiP: Python patching delusion, pt 1: finding & parsing patches; apply & git commit with pygit2; Markdown summaries (also for aggregation); git-to-patches tool - Python: Markdown aggregation and patching summaries; collapsible; SummarizedMarkdownWriter - Python: Markdown aggregation and patching summaries - Python: reorg a bit into common/armbian_utils; define the `ASSET_LOG_BASE` in preparation for Markdown delusion - Python patching: initial apply patches & initial commit patches to git (using pygit2) - Python patching: add basic `series.conf` support - Python patching: force use of utf-8; better error handling; use realpath of dirs - Python patching: `git-to-patches` initial hack. not proud. half-reused some of the patches-to-git - Python patching: "tag" the git commits with info for extracting later; introduce REWRITE_PATCHES/rewrite_patches_in_place - Python patching: commented-out, recover-bad-patches hacks - Python patching: shorten the signature - Python patching: allow BASE_GIT_TAG as well as BASE_GIT_REVISION - Python patching: git-archeology for patches missing descriptions; avoid UTF-8 in header/desc (not diff) - Python patching: use modern-er email.utils.parsedate_to_datetime to parse commit date - Python patching: unify PatchInPatchFile; better git-commiting; re-exporting patches from Git (directly) - Python patching: switch to GitPython - GitPython is like 100x slower than pygit2, but actually allows for date & committer - also allows to remove untracked files before starting - Python aggregation: fix missing `AGGREGATED_APT_SOURCES_DICT` - Python patching: add `unidecode` dependency to pip3 install - Python patching: don't try archeology if SRC is not a Git Repo (eg, in Docker) - Python patching: don't try archeology if not applying patches to git - WiP: Python patching delusion, pt2: actually use for u-boot & kernel patching - Python patching: much better problem handling/logging; lenient with recreations (kernel) - Python patching: don't force SHOW_LOG for u-boot patching - Python patching: don't bomb for no reason when there are no patches to apply - Python patching: fully (?) switch kernel patching to Python - Python patching: more logging fixups - Python patching: capture `kernel_git_revision` from `fetch_from_repo()`'s `checked_out_revision` - Python patching: fully switch u-boot patching to Python - Python aggregation/patching: colored logging; patching: always reset to git revision - Python aggregation/patching: better logging; introduce u-boot Python patching - Python patching pt3: recovers and better Markdown - Python patching: detect, and rescue, `wrong_strip_level` problem; don't try to export patches that didn't apply, bitch instead - Python patching: Markdown patching summary table, complete with emoji - Python patching: include the problem breakdown in Markdown summary - Python patching: sanity check against half-bare, half-mbox patches - Python patching: try to recover from 1) bad utf-8 encoded patches; 2) bad unidiff patches; add a few sanity checks - Python patching: try, and fail, to apply badly utf-8 encoded patches directly as bytes [reverted] - Python patching: try to recover from patch *parse* failures; show summary; better logging - set `GIT_ARCHEOLOGY=yes` to do archeology, default not - armbian-next: Python `pip` dependencies handling, similar to `hostdeps` - same scheme for Dockerfile caching - @TODO: still using global/shared environment; should move to a dir under `cache` or some kinda venv - WiP: add `python3-pip` to hostdeps; remove `python-setuptools` - remove `python-setuptools` (Python2, no longer exists in Sid) from hostdeps - add `python3-pip` to hostdeps; part of virtualenv saga - WiP: split `kernel.sh` a bit, into `kernel-patching.sh`, `kernel-config.sh` and `kernel-make.sh` - `advanced_patch()`: rename vars for clarity; no real changes - Python patching: introduce FAST_ARCHEOLOGY; still trying for Markdown links	2023-02-18 07:40:52 -03:00
Ricardo Pardini	75de1c2d82	armbian-next: remove usage of REQUIREMENTS_DEFS_ONLY; initialize extensions for requirements cli command & docker pre-game deps	2023-02-18 07:40:07 -03:00
Ricardo Pardini	bc7d388b49	armbian-next: cli: a bit more untangling of re-launching; introduce ARMBIAN_CLI_RELAUNCH_COMMAND and cli_standard_relaunch_docker_or_sudo() ... - `configdump` cli: force root/docker	2023-02-18 07:40:06 -03:00
Ricardo Pardini	d24d3327a8	armbian-next: the great cli entrypoint (+docker) rewrite; introduce USE_LOCAL_APT_DEB_CACHE replacing apt-cacher-ng ... - armbian-next: introduce `USE_LOCAL_APT_DEB_CACHE` (default `=yes`) as alternative/in addition to `apt-cacher-ng` (eg, in Docker) - this uses `cache/aptcache/${RELEASE}-${ARCH}` (in the host) for - apt cache, by bind-mounting it to `${SDCARD}/var/cache/apt` in the `chroot_sdcard_apt_get()` runner and its usages - debootstrap, by passing it `--cache-dir` - utility function to help understand what is happening to cache during usage - apt itself mantains this cache, removing old packages when new ones are installed. apt does this _by default_ - introduce `DONT_MAINTAIN_APT_CACHE=yes` to skip out of automatic apt maintenance of apt cache, eg, during `remove`s - don't do `apt clean` and such if using local cache, that would clean the cache, not the chroot - clean up `install_deb_chroot()` a little, find an unrelated bug there - WiP: the great cli entrypoint (+docker) rewrite, Phase 6: relaunching structure; re-pass ARMBIAN_BUILD_UUID; use ARMBIAN_COMMAND for log filename; fix for output/logs dir perms - WiP: the great cli entrypoint (+docker) rewrite, Phase 5: cleanups 4/x; better logging, check & force `DEST_LANG` - WiP: the great cli entrypoint (+docker) rewrite, Phase 5: cleanups 3/x; don't write to stderr in generated Dockerfile - it's `drastic red` on non-buildx dockers - WiP: the great cli entrypoint (+docker) rewrite, Phase 5: cleanups 2/x, logging - WiP: the great cli entrypoint (+docker) rewrite, Phase 5: cleanups 1/x - source configs in a logging section. - Docker: silent, fast retries to make sure `docker system df` works - shut-up `chown` (no `-v`) output related to `SET_OWNER_TO_UID` - ask user to wait while `DESTIMG` is rsync'ed to `FINALDEST` -- it's potentially very slow - use green apple for Mac logging, instead of red apple which might imply error... - WiP: the great cli entrypoint (+docker) rewrite, Phase 4: run as non-root, maybe-with-Docker - introduce `is_docker_ready_to_go()`; if it is, and we're not root, use Docker instead of sudo. <- GOOD IDEA? BAD IDEA? lol - introduce `SET_OWNER_TO_UID` var to be passed to Docker/sudo so written files are owned by the launching user, not root. - introduce `mkdir_recursive_and_set_uid_owner()` and `reset_uid_owner()` to reset owner based on `SET_OWNER_TO_UID` - use it for userpatches files created, logs, and output files, including images and debs. - @TODOs ref. `$SUDO_USER` which I think the old version of this? - add a lot of @TODOs, ref being able to relaunch something that's not `build` inside Docker, also add/change params and configs and command. - initially add `ARMBIAN_DOCKER_RELAUNCH_EXTRA_ARGS` - WiP: the great cli entrypoint (+docker) rewrite, Phase 3: rpardini is demented, v3 - WiP: the great cli entrypoint (+docker) rewrite, Phase 2: rpardini is demented - WiP: the great cli entrypoint (+docker) rewrite, Phase 1 - armbian-next: WiP: Docker: actually use the GHA-image as base; pull it every 24hs. - using image in my private repo. - this has significant speedup to "start building time" on the 1st run - move some Linux specific stuff to its own if - add comments and todo - armbian-next: WiP: Docker, high-WiP, beginnings of Armbian mount dict, with linux/darwin preferences - armbian-next: WiP: Docker, configure `BUILDKIT_COLORS` - armbian-next: WiP: Docker, make docker image from Dockerfile more compact by flattening layers - armbian-next: `logging`: add whale indicator if build running under Docker - armbian-next: WiP: `docker`: working with `bookworm`, `sid`, and `jammy` on Darwin & Linux; works with `bullseye` on Linux only - armbian-next: WiP: `docker`: force ARMBIAN_RUNNING_IN_CONTAINER both in Dockerfile and passed as `--env`; apt update and install in same layer; back to jammy - armbian-next: introduce `armbian_is_running_in_container()` and `armbian_is_host_running_systemd()`, replacing `systemd-detect-virt` in multiple spots - WiP: try with debian:bullseye -- can't detect docker at all - armbian-next: WiP: 2nd stab at new Docker support; Darwin still works; Linux `docker.io` working - gen .dockerignore together with Dockerfile - split in funcs - hacks for Linux and `/dev/loop` stuff, CONTAINER_COMPAT=yes - mac still works, Linux stuff would break it but I if'fed - armbian-next: the secrets of `CONTAINER_COMPAT` revealed; add size checking to check_loop_device() and avoid retry when `mknod`ing - this fails for the right reasons now, causing retries, which are then retried and work ;-) - this is related to building under Docker on Linux, using docker.io package (not docker-ce) - armbian-next: remove `.dockerignore` and add it to `.gitignore`; it's going to be auto-generated - armbian-next: `.dockerignore`: Docker context should only have minimal files and folders, to speed up Dockerfile build - IMPORTANT: `.dockerignore` is going to be generated from now on: so this is the last commit with changes before removal - armbian-next: WiP: initial stab at new Docker support; really run the passed cmdline; add Dockerfile to gitignore - armbian-next: WiP: initial stab at new Docker support; generate Dockerfile; introduce REQUIREMENTS_DEFS_ONLY - uses REQUIREMENTS_DEFS_ONLY - works on Docker Desktop on Mac; - linux TBA - armbian-next: don't error out if `.git` not present; other small fixes - armbian-next: general "work or at least don't misbehave when run on a very bare ubuntu:latest instance" - can't assume things, for example: - that `sudo` will be available; it might not, and might be already root, no reason to fail - that `/etc/timezone` will exist - that `systemd-detect-virt` will be available - that `git` will be available - that `locale-gen` will be available	2023-02-18 07:39:43 -03:00