Commit Graph

87 Commits

Author SHA1 Message Date
Vyacheslav Bocharov
123b31b161 Replace absolute path to git-tree with relative, so it is accesible without docker builder. 2023-02-28 09:26:12 -03:00
Ricardo Pardini
c2004faf9c
git-ref2info: fix https Makefile URL for gitlab.com (eg, rk35xx-legacy) (#4848) 2023-02-22 20:59:56 +01:00
Ricardo Pardini
26783b050f
armbian-next: pip3: workaround new problems introduced by pip 23+ 2023-02-18 07:46:47 -03:00
Ricardo Pardini
85f3fb5c93
armbian-next: oci-oras: don't capture our own debugs as part of the manifest (d'oh -- thanks RichN) 2023-02-18 07:46:45 -03:00
Ricardo Pardini
2350c5697f
armbian-next: cleaning: deprecate CLEAN_LEVEL=debs -- doesn't do anything, just shows what would be cleaned 2023-02-18 07:46:16 -03:00
Ricardo Pardini
e46233084a
armbian-next: memoize: expire all memoized files after 1 hour 2023-02-18 07:46:15 -03:00
Ricardo Pardini
3d8ee5ce38
armbian-next: oras: add 3/5 retries to oras pull/push 2023-02-18 07:46:11 -03:00
Ricardo Pardini
8694daad78
armbian-next: chroot-helpers: mount_chroot()/umount_chroot(): add warning with stacktrace if called sans logging section 2023-02-18 07:46:05 -03:00
Ricardo Pardini
26f0700da8
armbian-next: chroot-helpers: if unable to mount devpts, use a mount bind from host
Co-authored-by: Paolo Sabatino <paolo.sabatino@gmail.com>
2023-02-18 07:46:04 -03:00
Ricardo Pardini
9bffa5e749
armbian-next: artifacts: firmware/rootfs and actual usage of artifacts during image build
- artifacts: introduce `ARTIFACT_IGNORE_CACHE=yes`
- artifacts: introduce `DONT_BUILD_ARTIFACTS`, list of artifacts that if not found cached, fail the build
- kernel_package_source() is no more
- a long dissertation about kernels, families, and the universe
- artifacts: actually use rootfs artifact for image build
- artifacts: detangle via artifact_base_dir
- artifacts: rootfs: use folders in artifact_name; include cache_type
- artifacts: some cleanups / validations
- rootfs artifact; drop old rootfs cli
- artifacts: new CLI shortcuts; remove old firmware CLI
- artifacts: full firmware & usage
- use firmware artifacts in image build and install its debs
- kernel artifact sans legacy; use tmpdir for .deb target for all packages
- legacy artifact versions is no more; pack/unpack now in common obtain;
- artifacts: uboot: cleanup legacy renaming, use artifact version directly
- artifacts: add firmware (small) artifact
- deploy uboot to loop from artifact; allow tty to artifact; todos for cleaning
- fixes, kernel dtb/headers conditional; remove `.git` from Makefile url; use mapfile for finding files to hash
  - completely remove KERNEL_HAS_WORKING_HEADERS_FULL_SOURCE and `kernel_package_callback_linux_headers_full_source()`
  - don't use realpath for artifact_file_relative
  - curb some warnings
  - fix: only install headers & dtbs if such exist
- kernel .config hook modification hash versioning
- OCI_TARGET_BASE vs per-artifact defaults; only deploy to remote from CLI with OTB
- artifact revolver & installing into image
  - add artifact_map_packages and artifact_map_debs dicts
  - revolver accumulates all info
  - REPOSITORY_INSTALL is no more (for uboot/kernel, later others)
  - rename `VER` to `IMAGE_INSTALLED_KERNEL_VERSION`
2023-02-18 07:46:03 -03:00
Ricardo Pardini
8900eea0af
armbian-next: git-ref2info: add support for gitlab.com; exit_with_error if SHA1 can't be found 2023-02-18 07:46:02 -03:00
Ricardo Pardini
7f1fb1cd69
armbian-next: hash-files: if no files to hash, return 16 zeros, instead of hanging forever waiting for stdin 2023-02-18 07:45:59 -03:00
Ricardo Pardini
85d13cf8ce
armbian-next: git/ref2info: fixes for using commit:<sha1> 2023-02-18 07:45:54 -03:00
Ricardo Pardini
399d9964dd
armbian-next: python/pip: include pip pkgs in Dockerfile; use them if available instead of downloading 2023-02-18 07:45:51 -03:00
Ricardo Pardini
03dd0d623b
armbian-next: debs: introduce shellcheck for DEBIAN/postinst and others, before dpkg build 2023-02-18 07:45:50 -03:00
Ricardo Pardini
f0d3ea7961
armbian-next: tooling: introduce helper/downloader for shellcheck; include in Dockerfile 2023-02-18 07:45:48 -03:00
Ricardo Pardini
871f3cfcee
armbian-next: tooling: include ORAS tools in Dockerfile; use them if available instead of downloading 2023-02-18 07:45:47 -03:00
Ricardo Pardini
da02fde9cc
armbian-next: git: backport safe.directory method from master (just add *, once) -- thanks Igor 2023-02-18 07:45:44 -03:00
Ricardo Pardini
bf53bd2988
armbian-next: tmpfs: use size=99% for all tmpfs's; (rootfs/WORKDIR/chroot); thanks Igor for suggestion 2023-02-18 07:45:42 -03:00
Ricardo Pardini
3c79aaa79f
armbian-next: hash/cache: remove some cruft, some better logging 2023-02-18 07:45:39 -03:00
Ricardo Pardini
555a972442
armbian-next: git ref2info: dereference annotated tags to get "real" commit SHA1; fail if curl failed; validate parse_makefile_version() 2023-02-18 07:45:38 -03:00
Ricardo Pardini
68d76d6f3a
armbian-next: squash some shellcheck warnings; don't local and assign subshell value in same statement, that masks errors (declare/local almost-always works) 2023-02-18 07:45:31 -03:00
Ricardo Pardini
6258ea566a
armbian-next: shellcheck: squash a bunch of 'critical' warnings 2023-02-18 07:45:29 -03:00
Ricardo Pardini
bef7f8fee4
armbian-next: artifacts: introduce kernel/u-boot artifacts; git ref2info; "memoizer"; some hashing
- artifacts: u-boot/kernel - pt7 - adapt legacy/artifact versions; use common `capture_rename_legacy_debs_into_artifacts()`
- artifacts: u-boot - pt6: add artifact for u-boot
  - use artifact version / reason in actual u-boot .deb if present
- artifacts: kernel - pt5: tune kernel version, refactor
- artifacts: kernel - pt4: squash unrelated bugs that show up; move `prepare_compilation_vars()` to default build
- artifacts: kernel - pt3: drivers+patches+.config hashing
  - split file hashing function from drivers-harness; fix it so filenames are relative and sorted; sort from ${SRC}, always
  - aplit prepare_kernel_config_core_or_userpatches() from `kernel_config_initialize()`
- artifacts: kernel - pt2:  memoizing git ref2info
- artifacts: kernel - pt1:  versioning
2023-02-18 07:45:20 -03:00
Ricardo Pardini
19aa7e95cf
armbian-next: introduce assert_prepared_host() and assert before using any Python, to un-mask any further fallacies 2023-02-18 07:45:19 -03:00
Ricardo Pardini
44733e1afc
armbian-next: git: refactor git_parse_ref() out of fetch_from_repo() for reuse 2023-02-18 07:45:12 -03:00
Ricardo Pardini
8574b87f8d
armbian-next: curb some logging for git non-checkouts 2023-02-18 07:45:06 -03:00
Ricardo Pardini
5fa7bf735b
armbian-next: git: fetch_from_repo(): add fetched_revision(_ts) as well as checked_out_revision(_ts); introduce param do_checkout=no to only fetch, not checkout 2023-02-18 07:45:01 -03:00
Ricardo Pardini
9849990031
armbian-next: all chroots: mount /var/tmp as tmpfs; create dir /run/user/0 and mount as tmpfs @TODO cleanups 2023-02-18 07:44:55 -03:00
Ricardo Pardini
1cec42392e
armbian-next: GHA & rootfs adventures, pt 1
- rootfs: create readonly global `rootfs_cache_id` (type+hash+date/version)
- add "oras-upload" CLI command (dumb, no retries, single target/single source)
- docker: mount-bind for & re-pass envs `GITHUB_OUTPUT` and `GITHUB_STEP_SUMMARY` down to Docker
- introduce `github-actions.sh::github_actions_add_output()`
- during logs cleanup, dump the Markdown log into GITHUB_STEP_SUMMARY if it is available
2023-02-18 07:44:41 -03:00
Ricardo Pardini
8a63a0ae20
armbian-next: downloads.sh: some cleanup and add debugging to get_urls() 2023-02-18 07:44:17 -03:00
Ricardo Pardini
18c9946a4d
armbian-next: countdown: if to test -t 0 (0: stdin, 1:stdout) for interactive-ness; also test in if-not-aborted version 2023-02-18 07:44:11 -03:00
Ricardo Pardini
8d2be2c694
armbian-next: python/pip: solve crazy with /usr/bin/pip3 by not using it (instead, python3 -m pip); don't mask errors
- "| cut" masked errors in multiple places
- include real pip3 version in python deps hash, all will be invalidated
- be less happy about hitting hash/cache, should happen often
2023-02-18 07:44:09 -03:00
Ricardo Pardini
eb30aac310
armbian-next: extensions: move 'extensions.sh' into library; replace manual source with init function
- wraps globals in `extension_manager_declare_globals()` -- and make them actually global
- whoever wrote this original code is insane, add copyright
2023-02-18 07:44:05 -03:00
Ricardo Pardini
da77aad1dd
armbian-next: git: detect, and stop, when git bare trees vs worktrees have been shuffled around and can't be recovered
- hopefully this will never happen ;-)
2023-02-18 07:43:58 -03:00
Ricardo Pardini
88ff66ae11
armbian-next: adaptive shallow/full Kernel git bare git (3gb -> 300mb download)
- allows using pre-shallow-ed bare trees, specific to the KERNEL_MAJOR_MINOR
- uses shallow by default if storage is "slow" (mmc), or free disk space < 32g (@TODO tweak this)
- if full tree already there keeps using it
- allow forcing decision with `KERNEL_GIT=full` or `KERNEL_GIT=shallow`
- countdown to confirm if decision based on storage type/free space and user is interactive
  - this has some of the worst copy ever written
- @TODO refactor the device type / free space code, reuse across codebase (3-4 other spots detect free space)
- @TODO GHA self-hosted (full) vs hosted (shallow) runners decision
- sprinkle with `wait_for_disk_sync()` around clean/download/extract
- sprinkle git with debugs, and an info between fetch and checkout (otherwise it seems "fetch" takes a long time)
2023-02-18 07:43:54 -03:00
Ricardo Pardini
72c61f2f68
armbian-next: replace bare sync invocations by introducing wait_for_disk_sync(); sync early at start of build
- wait 10s for sync call; otherwise warn user to be patient
- wait_for_sync "reason for sync" everywhere
- add @TODO for /run/user/0 etc and XDG_RUNTIME_DIR in chroot
2023-02-18 07:43:53 -03:00
Ricardo Pardini
4d43c05f7b
armbian-next: kernel: git: remove dead code, GIT_PRE_FETCH_HOOK / checked_out_revision_mtime 2023-02-18 07:43:52 -03:00
Ricardo Pardini
3eacae68c7
armbian-next: kernel: git: split via-ORAS and via-Bundle tangle 2023-02-18 07:43:51 -03:00
Ricardo Pardini
8af14d4f8c
armbian-next: countdown: split off file for countdown functions (countdown-to-abort and countdown-to-continue)
- introduce `countdown.sh`, move `traps.sh::exit_if_countdown_not_aborted()` there, introduce `exit_if_countdown_not_aborted()`
2023-02-18 07:43:50 -03:00
Ricardo Pardini
549ae08061
armbian-next: git: show "fetch completed" info, lest user thinks fetch is hanging while checking out 2023-02-18 07:43:46 -03:00
Ricardo Pardini
be49433b41
armbian-next: use retries for downloading ORAS tooling
- `do_with_retries()`: add `IS_A_RETRY` and `RETRY_FMT_MORE_THAN_ONCE` for convenient logging in retried functions
2023-02-18 07:43:35 -03:00
Ricardo Pardini
1636a2afea
armbian-next: tune logging: unmounting, u-boot prep/source/install; traps; extensions 2023-02-18 07:43:25 -03:00
Ricardo Pardini
bf891d54d9
armbian-next: split prepare_host(), fix .tmp reset trap
- `prepare_host()`: split; do checks earlier and allow them to be interactive
- introduce `exit_if_countdown_not_aborted()` for "Low Disk Space" and other critical conditions
- split `prepare_host()` into interactive & non-interactive parts
- split off `clean_deprecated_mountpoints()` from prepare into `cleaning.sh`
- introduce and use `reset_uid_owner_non_recursive()` for `.tmp` reset in trap, to avoid disasters
- add more logging sections to default-build.sh, avoid unlogged parts
2023-02-18 07:43:10 -03:00
Ricardo Pardini
7beeae6219
armbian-next: Python tooling: use consolidated+hashed+cached pip base/pycache; don't pip during Dockerfile build, nor cli-requirements
- consolidate at `prepare_python_and_pip()`
- sanity check for Python version 3.9+ regardless of HOSTRELEASE
- TODO: pip vs sudo/root: need pip 22.2+ to curb warning, not doing it
2023-02-18 07:42:44 -03:00
Ricardo Pardini
53266feff8
armbian-next: kernel: deterministic .config mtime handling; kernel build logging sections reorg
- .config is now compared with previous one byte-wise, diff shown, and mtime preserved directly via `cp -p`
- this does away with the previous mtime-based BS I had written during the fasthash era (and thus remove file-mtime.sh, regen lib)
- split some functions, to better control the interactive piece in the _middle_ of kernel config process
- no more `kernel_config_maybe_interactive()`, now `kernel_config()` that controls its own logging sections
- extract `kernel_determine_toolchain()` out of config into make
- introduce internal hook `armbian_kernel_config()` that runs before custom version (for .config defaults, coming soon)
2023-02-18 07:42:39 -03:00
Ricardo Pardini
df058ea3f1
armbian-next: fix quoting of retry var (cosmetic)
- this commit is proof of OCD
2023-02-18 07:42:21 -03:00
Ricardo Pardini
7493ce1b09
armbian-next: long overdue split of logging.sh 2023-02-18 07:42:19 -03:00
Ricardo Pardini
1ed9dada4d
armbian-next: curb aria2c's complaining a bit 2023-02-18 07:42:17 -03:00
Ricardo Pardini
3563a5f1e2
armbian-next: curb down git fetch logging; be verbose if user on terminal & not logging 2023-02-18 07:42:16 -03:00