- fix a bunch of quoting issues
- use runners for chroot stuff
- don't error out of failed disable
- makes building `sid` again possible
- add a bunch of TODO's
- consolidate at `prepare_python_and_pip()`
- sanity check for Python version 3.9+ regardless of HOSTRELEASE
- TODO: pip vs sudo/root: need pip 22.2+ to curb warning, not doing it
- .config is now compared with previous one byte-wise, diff shown, and mtime preserved directly via `cp -p`
- this does away with the previous mtime-based BS I had written during the fasthash era (and thus remove file-mtime.sh, regen lib)
- split some functions, to better control the interactive piece in the _middle_ of kernel config process
- no more `kernel_config_maybe_interactive()`, now `kernel_config()` that controls its own logging sections
- extract `kernel_determine_toolchain()` out of config into make
- introduce internal hook `armbian_kernel_config()` that runs before custom version (for .config defaults, coming soon)
- default `SHOW_LOG=yes` if user on terminal
- shortcut: `DEBUG=yes` defaults both `SHOW_LOG=yes` & `SHOW_DEBUG=yes`
- much simpler logging format for log file
- skip all display_alert()'s more verbose than DEBUG from logfile
- set `DEBUG=yes` to log everything again
- skip Markdown assets from being included in ANSI log
- use of `ccze`
- introduce internal var for controlling COMMAND-level log
- remove unused `run_on_sdcard()` that somehow was left there
- drop `export_html_logs()` and `EXPORT_HTML_LOG` -- are no more
- remove a lot of hopefully / hardly confirmed, unused dependencies
- don't use crossbuild-essential-xxx; avoid the c++ compiler that comes with it, install gcc-only instead
- hostdeps: use `libc6-dev make dpkg-dev gcc` (without `g++`) instead of `build-essential`
- drop `btrfs-progs` and `f2fs-tools` (@TODO add in extension when/whereused)
- more: drop `cryptsetup` (@TODO add in extension when/whereused)
- don't be too quiet when doing apt-update for hostdeps
- acng is constant source of headaches
- we don't need it anymore, since we have local caching now
- You can also have acng running on a different machine/container and set `APT_PROXY_ADDR=xxx:3142` if you're running a build farm
- probably breaks a lot of stuff; c++ compilers, distcc, possibly others
- split into late_prepare_host_dependencies / early_prepare_host_dependencies calling adaptative_prepare_host_dependencies() full of if's
also:
- calc deps out of configuration
- reorg checks into obtain_and_check_host_release_and_arch()
- if set, the Docker launcher will not install any dependencies
- so every Docker run will reinstall everything from scratch
- do NOT use, **even if you're Oleg**
- completely remove support for building under `buster` -- that's way too old, sorry.
- de-hardcode `python3` invocations, instead use `python3_binary_path` set by `prepare_python3_binary_for_python_tools()`
- juggle `$HOSTRELEASE`: read from actual host, or determined from Docker image name (during Dockerfile build)
- TL;DR: include and use `python3.9` for focal-like host OS's
- introduce `do_with_conditional_logging()` which only starts logging sections if `do_logging=no`
- with this we should get complete logs (ofc except for the interactive sections)
- config might run on a non-Linux, bash-capable, GNU coreutils enabled machine, so avoid `linux-version`
- use `declare -g` instead of `export` for most of `prepare_and_config_main_build_single()`
- Python patching: tune some logging all-around
- Python patching: *FINALLY* set the dates on the patched files to `max(patch_date, root_makefile_date)` thus making lighting-fast rebuilds again
- new EXTRAWIFI patch generator harness; Python patching EXTRA_PATCH_FILES_FIRST
- Python patching: use temp file for patching rejects; clean it up
- new EXTRAWIFI patch generator harness; Python: mark drivers as autogen, don't split or parse them, read as bytes, apply bytes directly
- new EXTRAWIFI patch generator harness; somewhat-works, but patches are per-family
- Python patching: add the `cache/patch` as a mountpoint
- Darwin Docker performance is 20x with namedvolume; the cached patches are very large
- Python patching: hopefully better Markdown: split series dir; detect more status/problems; arche only for missing Subject
- Python patching: archeology only for missing Subject:, not description; clarify CLI pushing
- Python patching: use `{file_base_name}.patch` for archeology as `file_name` might include a dir
- Python patching: fix: don't skip board/target patches for no reason
- Python patching: fix for series.conf patches (sunxi/sunxi64): don't sort; mark as series and show on summary
- Python patching: don't fail if archeology found no commits (but add warning)
- Python patching: CLI command `kernel-patches-to-git` with archeology summary, and git pushing of results & summary
- Python patching: patches-to-git small fixes, auto-push if it's rpardini
- Python patching: add `patch-kernel` CLI command
- Python patching: commit README.md & gh-pages workflow when apply_patches_to_git
- Python patching: hopefully better markdown
- Python patching: `git add` everything all at once, for speed
- armbian-next: fixes to ORAS-related logging
- armbian-next: actually enable ORAS-based kernel git bare tree seeding/bundles; enable cleaning of bundle articfacts after confirmed working
- armbian-next: introduce ORAS-based kernel git bare tree seeding/bundles (although it is a .tar, not a bundle); this is 20x faster than cloning
- armbian-next: introduce `ORAS` tooling; pull and push functions & downloader/launcher
- WiP: Python patching delusion, pt 1: finding & parsing patches; apply & git commit with pygit2; Markdown summaries (also for aggregation); git-to-patches tool
- Python: Markdown aggregation and patching summaries; collapsible; SummarizedMarkdownWriter
- Python: Markdown aggregation and patching summaries
- Python: reorg a bit into common/armbian_utils; define the `ASSET_LOG_BASE` in preparation for Markdown delusion
- Python patching: initial apply patches & initial commit patches to git (using pygit2)
- Python patching: add basic `series.conf` support
- Python patching: force use of utf-8; better error handling; use realpath of dirs
- Python patching: `git-to-patches` initial hack. not proud. half-reused some of the patches-to-git
- Python patching: "tag" the git commits with info for extracting later; introduce REWRITE_PATCHES/rewrite_patches_in_place
- Python patching: commented-out, recover-bad-patches hacks
- Python patching: shorten the signature
- Python patching: allow BASE_GIT_TAG as well as BASE_GIT_REVISION
- Python patching: git-archeology for patches missing descriptions; avoid UTF-8 in header/desc (not diff)
- Python patching: use modern-er email.utils.parsedate_to_datetime to parse commit date
- Python patching: unify PatchInPatchFile; better git-commiting; re-exporting patches from Git (directly)
- Python patching: switch to GitPython
- GitPython is like 100x slower than pygit2, but actually allows for date & committer
- also allows to remove untracked files before starting
- Python aggregation: fix missing `AGGREGATED_APT_SOURCES_DICT`
- Python patching: add `unidecode` dependency to pip3 install
- Python patching: don't try archeology if SRC is not a Git Repo (eg, in Docker)
- Python patching: don't try archeology if not applying patches to git
- WiP: Python patching delusion, pt2: actually use for u-boot & kernel patching
- Python patching: much better problem handling/logging; lenient with recreations (kernel)
- Python patching: don't force SHOW_LOG for u-boot patching
- Python patching: don't bomb for no reason when there are no patches to apply
- Python patching: fully (?) switch kernel patching to Python
- Python patching: more logging fixups
- Python patching: capture `kernel_git_revision` from `fetch_from_repo()`'s `checked_out_revision`
- Python patching: fully switch u-boot patching to Python
- Python aggregation/patching: colored logging; patching: always reset to git revision
- Python aggregation/patching: better logging; introduce u-boot Python patching
- Python patching pt3: recovers and better Markdown
- Python patching: detect, and rescue, `wrong_strip_level` problem; don't try to export patches that didn't apply, bitch instead
- Python patching: Markdown patching summary table, complete with emoji
- Python patching: include the problem breakdown in Markdown summary
- Python patching: sanity check against half-bare, half-mbox patches
- Python patching: try to recover from 1) bad utf-8 encoded patches; 2) bad unidiff patches; add a few sanity checks
- Python patching: try, and fail, to apply badly utf-8 encoded patches directly as bytes [reverted]
- Python patching: try to recover from patch *parse* failures; show summary; better logging
- set `GIT_ARCHEOLOGY=yes` to do archeology, default not
- armbian-next: Python `pip` dependencies handling, similar to `hostdeps`
- same scheme for Dockerfile caching
- @TODO: still using global/shared environment; should move to a dir under `cache` or some kinda venv
- WiP: add `python3-pip` to hostdeps; remove `python-setuptools`
- remove `python-setuptools` (Python2, no longer exists in Sid) from hostdeps
- add `python3-pip` to hostdeps; part of virtualenv saga
- WiP: split `kernel.sh` a bit, into `kernel-patching.sh`, `kernel-config.sh` and `kernel-make.sh`
- `advanced_patch()`: rename vars for clarity; no real changes
- Python patching: introduce FAST_ARCHEOLOGY; still trying for Markdown links
- share a single `.git` for all u-boots
- all fetches are done against it, maximizing cache hit ratio drastically
- also reduces the size of each working copy by more than 70%
- split uboot's git stuff into uboot-git.sh
- split `uboot_prepare_git()` from `compile_uboot()`
- now we've a single `.git` for all kernels with all branches; a "master" cache
- back to full stable git bundle usage from kernel.org
- bye bye, "gitballs"
- split kernel-related git stuff into `kernel-git.sh`
- introduce `ARMBIAN_MOUNTPOINTS_DEPRECATED` @TODO actually use for cleaning non-Docker
- add `axel` hostdep for multi-conn download of bundle, which comes from a CDN.
- deprecate old mountpoints for gitballs and linux-kernel
- move cleaning of old sources/kernel to mountpoints code
- introduce `if_user_on_terminal_and_not_logging_add()`;
- also the opposite, `if_user_not_on_terminal_or_is_logging_add()`
- unset `CURRENT_LOGGING_SECTION` when exiting a logging section
- introduce `do_with_logging_unless_user_terminal()`
- sharing most UEFI code, will replace the `virtual` one soon
- x86: patch uboot defconfig to use the `q35` machine type, not `i440fx`
- separate x86 bootscript, due to non-uInitrd-ness of it
- hack ramdisk load address both in u-boot source and bootscript
- use 32-bit u-boot, not 64-bit
- grub: introduce `UEFI_GRUB=skip`, does not deploy GRUB (but does the kernel packages, etc)
- auto-enable qcow2 output for these
- works with both distro's and Armbian's kernels
- cli builds won't have PPAs and other external repos
- packages from such repos should be listed in `packages.external` now, so they're not mismatched
- `PACKAGE_LIST_EXTERNAL` (packages.external) only in DESKTOP builds
- this logic may change later, example:
- we might _also_ want pure-Ubuntu desktops, without external repos
- we might _instead_ want external repos in the cli builds
- this way is easy to change, since everything is kept separate and we can change in a single place.
- introduce `SHOW_CCACHE=yes` for detailed ccache statistics and logging
- fix: kernel build is done under "env -i", so pass CCACHE_DIR down to Make if CCACHE_DIR is set...
- split from kernel.sh, show stats also for u-boot targets; show compile time
- add volume definition (under `${SRC}/cache/ccache`); auto-use that in Docker builds via `CCACHE_DIR`
- better logging for `do_with_ccache_statistics()`
- there's some CCACHE_DIR code from before; unify @TODO
- `undecided` now directly sends to `build`, which might send to `docker`, which relaunches `build` in Docker;
- the _sending to Docker_ part probably needs refactor
- DEBUG_EXTENSION_CALLS is gone. use SHOW_EXTENSIONS instead.
- LOG_ALL_HOOK_TRACES is gone. use SHOW_EXTENSIONS_TRACE instead
- LOG_ENABLE_EXTENSION is gone. use SHOW_DEBUG=yes instead
- Changes how Docker Desktop works too, there's a `case` for switching back if needed.
- Now the linux-only CONTAINER_COMPAT hacks are used also for Darwin (`DOCKER_SERVER_REQUIRES_LOOP_HACKS=yes`)
- `/dev/loop*` is spelled out since they don't exist on Darwin (`DOCKER_SERVER_USE_STATIC_LOOPS=yes`)
- armbian-next: introduce `USE_LOCAL_APT_DEB_CACHE` (default `=yes`) as alternative/in addition to `apt-cacher-ng` (eg, in Docker)
- this uses `cache/aptcache/${RELEASE}-${ARCH}` (in the host) for
- apt cache, by bind-mounting it to `${SDCARD}/var/cache/apt` in the `chroot_sdcard_apt_get()` runner and its usages
- debootstrap, by passing it `--cache-dir`
- utility function to help understand what is happening to cache during usage
- apt itself mantains this cache, removing old packages when new ones are installed. apt does this _by default_
- introduce `DONT_MAINTAIN_APT_CACHE=yes` to skip out of automatic apt maintenance of apt cache, eg, during `remove`s
- don't do `apt clean` and such if using local cache, that would clean the cache, not the chroot
- clean up `install_deb_chroot()` a little, find an unrelated bug there
- WiP: the great cli entrypoint (+docker) rewrite, Phase 6: relaunching structure; re-pass ARMBIAN_BUILD_UUID; use ARMBIAN_COMMAND for log filename; fix for output/logs dir perms
- WiP: the great cli entrypoint (+docker) rewrite, Phase 5: cleanups 4/x; better logging, check & force `DEST_LANG`
- WiP: the great cli entrypoint (+docker) rewrite, Phase 5: cleanups 3/x; don't write to stderr in generated Dockerfile
- it's `drastic red` on non-buildx dockers
- WiP: the great cli entrypoint (+docker) rewrite, Phase 5: cleanups 2/x, logging
- WiP: the great cli entrypoint (+docker) rewrite, Phase 5: cleanups 1/x
- source configs in a logging section.
- Docker: silent, fast retries to make sure `docker system df` works
- shut-up `chown` (no `-v`) output related to `SET_OWNER_TO_UID`
- ask user to wait while `DESTIMG` is rsync'ed to `FINALDEST` -- it's potentially very slow
- use green apple for Mac logging, instead of red apple which might imply error...
- WiP: the great cli entrypoint (+docker) rewrite, Phase 4: run as non-root, maybe-with-Docker
- introduce `is_docker_ready_to_go()`; if it is, and we're not root, use Docker instead of sudo. <- GOOD IDEA? BAD IDEA? lol
- introduce `SET_OWNER_TO_UID` var to be passed to Docker/sudo so written files are owned by the launching user, not root.
- introduce `mkdir_recursive_and_set_uid_owner()` and `reset_uid_owner()` to reset owner based on `SET_OWNER_TO_UID`
- use it for userpatches files created, logs, and output files, including images and debs.
- @TODOs ref. `$SUDO_USER` which I think the old version of this?
- add a lot of @TODOs, ref being able to relaunch something that's not `build` inside Docker, also add/change params and configs and command.
- initially add `ARMBIAN_DOCKER_RELAUNCH_EXTRA_ARGS`
- WiP: the great cli entrypoint (+docker) rewrite, Phase 3: rpardini is demented, v3
- WiP: the great cli entrypoint (+docker) rewrite, Phase 2: rpardini is demented
- WiP: the great cli entrypoint (+docker) rewrite, Phase 1
- armbian-next: WiP: Docker: actually use the GHA-image as base; pull it every 24hs.
- using image in my private repo.
- this has significant speedup to "start building time" on the 1st run
- move some Linux specific stuff to its own if
- add comments and todo
- armbian-next: WiP: Docker, high-WiP, beginnings of Armbian mount dict, with linux/darwin preferences
- armbian-next: WiP: Docker, configure `BUILDKIT_COLORS`
- armbian-next: WiP: Docker, make docker image from Dockerfile more compact by flattening layers
- armbian-next: `logging`: add whale indicator if build running under Docker
- armbian-next: WiP: `docker`: working with `bookworm`, `sid`, and `jammy` on Darwin & Linux; works with `bullseye` on Linux only
- armbian-next: WiP: `docker`: force ARMBIAN_RUNNING_IN_CONTAINER both in Dockerfile and passed as `--env`; apt update and install in same layer; back to jammy
- armbian-next: introduce `armbian_is_running_in_container()` and `armbian_is_host_running_systemd()`, replacing `systemd-detect-virt` in multiple spots
- WiP: try with debian:bullseye -- can't detect docker at all
- armbian-next: WiP: 2nd stab at new Docker support; Darwin still works; Linux `docker.io` working
- gen .dockerignore together with Dockerfile
- split in funcs
- hacks for Linux and `/dev/loop` stuff, CONTAINER_COMPAT=yes
- mac still works, Linux stuff would break it but I if'fed
- armbian-next: the secrets of `CONTAINER_COMPAT` revealed; add size checking to check_loop_device() and avoid retry when `mknod`ing
- this fails for the right reasons now, causing retries, which are then retried and work ;-)
- this is related to building under Docker on Linux, using docker.io package (not docker-ce)
- armbian-next: remove `.dockerignore` and add it to `.gitignore`; it's going to be auto-generated
- armbian-next: `.dockerignore`: Docker context should only have minimal files and folders, to speed up Dockerfile build
- IMPORTANT: `.dockerignore` is going to be generated from now on: so this is the last commit with changes before removal
- armbian-next: WiP: initial stab at new Docker support; really run the passed cmdline; add Dockerfile to gitignore
- armbian-next: WiP: initial stab at new Docker support; generate Dockerfile; introduce REQUIREMENTS_DEFS_ONLY
- uses REQUIREMENTS_DEFS_ONLY
- works on Docker Desktop on Mac;
- linux TBA
- armbian-next: don't error out if `.git` not present; other small fixes
- armbian-next: general "work or at least don't misbehave when run on a very bare ubuntu:latest instance"
- can't assume things, for example:
- that `sudo` will be available; it might not, and might be already root, no reason to fail
- that `/etc/timezone` will exist
- that `systemd-detect-virt` will be available
- that `git` will be available
- that `locale-gen` will be available
- remove grub's --verbose, it's really too verbose
- trust in TMPDIR, do not use trap at all in compile_plymouth-theme-armbian - settings traps adhoc is forbidden in armbian-next as well
- be more verbose in umount_chroot(), do not try to unmount tmpfs-mounted dir itself, only xx/tmp
- try to be smarter about /tmp being mounted in rootfs-to-image -- in the end the recursive saves us
- run post_debootstrap_tweaks when the SDCARD is still mounted, not after
- don't try to download anything from any mirrors if SKIP_ARMBIAN_REPO==yes
