Commit Graph

77 Commits

Author SHA1 Message Date
Ricardo Pardini
9bffa5e749
armbian-next: artifacts: firmware/rootfs and actual usage of artifacts during image build
- artifacts: introduce `ARTIFACT_IGNORE_CACHE=yes`
- artifacts: introduce `DONT_BUILD_ARTIFACTS`, list of artifacts that if not found cached, fail the build
- kernel_package_source() is no more
- a long dissertation about kernels, families, and the universe
- artifacts: actually use rootfs artifact for image build
- artifacts: detangle via artifact_base_dir
- artifacts: rootfs: use folders in artifact_name; include cache_type
- artifacts: some cleanups / validations
- rootfs artifact; drop old rootfs cli
- artifacts: new CLI shortcuts; remove old firmware CLI
- artifacts: full firmware & usage
- use firmware artifacts in image build and install its debs
- kernel artifact sans legacy; use tmpdir for .deb target for all packages
- legacy artifact versions is no more; pack/unpack now in common obtain;
- artifacts: uboot: cleanup legacy renaming, use artifact version directly
- artifacts: add firmware (small) artifact
- deploy uboot to loop from artifact; allow tty to artifact; todos for cleaning
- fixes, kernel dtb/headers conditional; remove `.git` from Makefile url; use mapfile for finding files to hash
  - completely remove KERNEL_HAS_WORKING_HEADERS_FULL_SOURCE and `kernel_package_callback_linux_headers_full_source()`
  - don't use realpath for artifact_file_relative
  - curb some warnings
  - fix: only install headers & dtbs if such exist
- kernel .config hook modification hash versioning
- OCI_TARGET_BASE vs per-artifact defaults; only deploy to remote from CLI with OTB
- artifact revolver & installing into image
  - add artifact_map_packages and artifact_map_debs dicts
  - revolver accumulates all info
  - REPOSITORY_INSTALL is no more (for uboot/kernel, later others)
  - rename `VER` to `IMAGE_INSTALLED_KERNEL_VERSION`
2023-02-18 07:46:03 -03:00
Ricardo Pardini
ca6bf9046b
armbian-next: remove all traces of Vagrant and FEL boot 2023-02-18 07:46:00 -03:00
Ricardo Pardini
399d9964dd
armbian-next: python/pip: include pip pkgs in Dockerfile; use them if available instead of downloading 2023-02-18 07:45:51 -03:00
Ricardo Pardini
f0d3ea7961
armbian-next: tooling: introduce helper/downloader for shellcheck; include in Dockerfile 2023-02-18 07:45:48 -03:00
Ricardo Pardini
871f3cfcee
armbian-next: tooling: include ORAS tools in Dockerfile; use them if available instead of downloading 2023-02-18 07:45:47 -03:00
Ricardo Pardini
0e480e685e
armbian-next: docker: pass down git info via env var, since Docker doesn't get ${SRC}/.git, yet we need that info in the Docker logs 2023-02-18 07:45:40 -03:00
Ricardo Pardini
3f91b77e3f
armbian-next: docker-shell: provide SSH and forward ssh auth socket to docker-shell 2023-02-18 07:45:34 -03:00
Ricardo Pardini
6258ea566a
armbian-next: shellcheck: squash a bunch of 'critical' warnings 2023-02-18 07:45:29 -03:00
Ricardo Pardini
1fc13a57a7
armbian-next: json-info: cleanup info/json/csv/opensearch, add logging, add some very basic instructions; add Kibana dashboards & docker-compose to bring it OS+Kibana up 2023-02-18 07:45:27 -03:00
Ricardo Pardini
a7fcd44942
armbian-next: cli: u-boot alias: don't stop to interactively ask for kernel configuration during u-boot build 2023-02-18 07:45:25 -03:00
Ricardo Pardini
ba7487b1a1
armbian-next: docker: fix Dockerfile generation; don't output logs info during Dockerfile build 2023-02-18 07:45:23 -03:00
Ricardo Pardini
bb6975707a
armbian-next: take a breath: config might mark aggregation required; aggregation requires prepare_host, and thus runs in default-build 2023-02-18 07:45:21 -03:00
Ricardo Pardini
bef7f8fee4
armbian-next: artifacts: introduce kernel/u-boot artifacts; git ref2info; "memoizer"; some hashing
- artifacts: u-boot/kernel - pt7 - adapt legacy/artifact versions; use common `capture_rename_legacy_debs_into_artifacts()`
- artifacts: u-boot - pt6: add artifact for u-boot
  - use artifact version / reason in actual u-boot .deb if present
- artifacts: kernel - pt5: tune kernel version, refactor
- artifacts: kernel - pt4: squash unrelated bugs that show up; move `prepare_compilation_vars()` to default build
- artifacts: kernel - pt3: drivers+patches+.config hashing
  - split file hashing function from drivers-harness; fix it so filenames are relative and sorted; sort from ${SRC}, always
  - aplit prepare_kernel_config_core_or_userpatches() from `kernel_config_initialize()`
- artifacts: kernel - pt2:  memoizing git ref2info
- artifacts: kernel - pt1:  versioning
2023-02-18 07:45:20 -03:00
Ricardo Pardini
281e3f0828
armbian-next: cli: try (and fail, probably...) to be helpful with half-working Docker installs 2023-02-18 07:45:16 -03:00
Ricardo Pardini
d43c4262b0
armbian-next: cli: *do not* allow running as root on terminal; make PREFER_DOCKER=no actually work
- do allow root if non-interactive
2023-02-18 07:45:15 -03:00
Ricardo Pardini
aadf012464
armbian-next: do-not-fix docker-asking-for-docker error by adding a "tip" 2023-02-18 07:45:14 -03:00
Ricardo Pardini
8329b2e970
armbian-next: docker: split docker_cli_build_dockerfile() into docker_cli_prepare_dockerfile(); introduce FAST_DOCKER
- armbian-next: docker: fix hostrelease crazy
2023-02-18 07:45:11 -03:00
Ricardo Pardini
3b126c0730
armbian-next: unify 2 methods for config: prep_conf_main_minimal_ni() (rootfs/firmware/kernel) and prep_conf_main_build_single() (full-build, possibly interactive) 2023-02-18 07:45:05 -03:00
Ricardo Pardini
9dc24840c3
armbian-next: firmware-deb: reworked: Provides/Conflicts instead of Replaces; cp -> git archive; new firmware CLI; FULL version .deb out of tmpfs 2023-02-18 07:45:04 -03:00
Ricardo Pardini
16eb5eb50b
armbian-next: aggregate_packages out of main_default_start_build(); introduce assert_requires_aggregation() 2023-02-18 07:45:03 -03:00
Ricardo Pardini
67e6b010f2
armbian-next: cli-build: don't show two BOARD= repeat option 2023-02-18 07:45:02 -03:00
Ricardo Pardini
1c00d64b35
armbian-next: repeat build opts: fix missing BOARD, show both after config (for logfile) & after build 2023-02-18 07:44:58 -03:00
Ricardo Pardini
962ae65a45
armbian-next: docker: include ${SRC}/.git static via Dockerfile, for kernel-patches-to-git / archeology; use fast archeology
- include FAST_ARCHEOLOGY in call to Python, default yes; sync kernel & u-boot call
2023-02-18 07:44:54 -03:00
Ricardo Pardini
6e267f5cd1
armbian-next: docker: passthrough of the SSH agent, via Docker magic on Darwin; enable for docker-shell and kernel-patches-to-git
- don't check/verify/update SSH-related hostkey etc for git tree push
2023-02-18 07:44:53 -03:00
Ricardo Pardini
32bfcb1b68
armbian-next: rootfs: curb logging; GHA outputs rootfs_cache_id_version, rootfs_cache_id and rootfs_out_filename_relative 2023-02-18 07:44:47 -03:00
Ricardo Pardini
dbfa7329ab
armbian-next: cli: kernel cli forces KERNEL_CONFIGURE=no; @TODO this should only "default to", not force. for proper kernel cli, later 2023-02-18 07:44:44 -03:00
Igor
00bc1b4a1e
armbian-next: rootfs: introduce ROOTFS_COMPRESSION_RATIO, default 5 for general use, default 15 for rootfs CLI; fix pv | tar bug; remove GPG signing
- Remove PGP signing from the script as its done in GHA
- Also remove generating .current control file as its obsolete in this design

Co-authored-by: Ricardo Pardini <ricardo@pardini.net>
2023-02-18 07:44:42 -03:00
Ricardo Pardini
1cec42392e
armbian-next: GHA & rootfs adventures, pt 1
- rootfs: create readonly global `rootfs_cache_id` (type+hash+date/version)
- add "oras-upload" CLI command (dumb, no retries, single target/single source)
- docker: mount-bind for & re-pass envs `GITHUB_OUTPUT` and `GITHUB_STEP_SUMMARY` down to Docker
- introduce `github-actions.sh::github_actions_add_output()`
- during logs cleanup, dump the Markdown log into GITHUB_STEP_SUMMARY if it is available
2023-02-18 07:44:41 -03:00
Ricardo Pardini
041297c563
armbian-next: binfmt: make sure we have all binfmt's setup & enabled for rootfs build (v2) 2023-02-18 07:44:39 -03:00
Ricardo Pardini
d17157885c
armbian-next: rootfs: converge rootfs-cache and rootfs together; exchange BOARD for ARCH 2023-02-18 07:44:28 -03:00
Ricardo Pardini
695bfb387b
armbian-next: rootfs: reset UID of produced rootfs tarball; announce what is doing and what has done 2023-02-18 07:44:27 -03:00
Ricardo Pardini
6077d40f9b
armbian-next: rootfs: build rootfs from CLI, sans-BOARD, sans-FAMILY, etc -- rootfs are meant to be shared
- require ARCH/RELEASE for rootfs cli
- disallow BOARD/LINUXFAMILY/BOARDFAMILY being set for rootfs cli
- split `prep_conf_main_only_rootfs()` instead of `prep_conf_main_build_single()` for rootfs-only stuff
- run both conf and build without stdin for rootfs CLI (no interactive allowed)
- don't blindly set `ARCH=armhf` for no reason in main-config
- allow skipping family stuff in main-config, via `allow_no_family=yes` (used only in `prep_conf_main_only_rootfs()`)
- kill double loading of userpatches family, not needed/old code
- allow skipping kernel stuff via `skip_kernel=yes` in `config_pre_main()`  (used only in `prep_conf_main_only_rootfs()`)
2023-02-18 07:44:26 -03:00
Ricardo Pardini
091ae1d27f
armbian-next: rootfs: adapt to new reality after huge refactor
- make `tmpfs_estimated_size` and `available_physical_memory_mib` readonly and global and only ever set once
- remove subshell from `get_rootfs_cache_list()`, new version `get_rootfs_cache_list_into_array_variable()` for sanity
- warns if `ROOT_FS_CREATE_ONLY=yes` is set during cached build path
- add a `wait_for_disk_sync()` after restoring rootfs cache
- fix logging sections for `build_rootfs_only()` and `prepare_rootfs_build_params_and_trap()` in `build_rootfs_and_image()`
- remove the logging section from `get_or_create_rootfs_cache_chroot_sdcard()` call in `build_rootfs_only()` (which is now itself logged)
- @TODO: `deploy_qemu_binary_to_chroot()` is leaking a qemu bin in rootfs cache!
2023-02-18 07:44:24 -03:00
Ricardo Pardini
28d89e2c91
armbian-next: rootfs: new CLI command for rootfs cache building; further refactorings
- _told ya it would all make sense eventually..._
- introduce new, real, non-alias `rootfs` command in `cli-rootfs` -- this _only_ builds rootfs, nothing else
- no more `main_default_build_single()`; instead `full_build_packages_rootfs_and_image()`
- introduce `do_with_default_build()` wrapper, for things that need host prepared + aggregation to run
- CLI-specific parts of `main_default_build_single()` moved to cli-build
- put prepare/cleanup pair of `prepare_rootfs_build_params_and_trap()` and `trap_handler_cleanup_rootfs_and_image()` into new `trap-rootfs.sh` (this needs further splitting for the image part)
- refactor `get_or_create_rootfs_cache_chroot_sdcard()` into `build_rootfs_only()`, `calculate_rootfs_cache_id()`
- update library with new files
2023-02-18 07:44:22 -03:00
Ricardo Pardini
eb30aac310
armbian-next: extensions: move 'extensions.sh' into library; replace manual source with init function
- wraps globals in `extension_manager_declare_globals()` -- and make them actually global
- whoever wrote this original code is insane, add copyright
2023-02-18 07:44:05 -03:00
Igor
90c46e0215
armbian-next: cli: add CLI alias command rootfs-cache for creating rootfs cache 2023-02-18 07:44:02 -03:00
Ricardo Pardini
a6182bbd89
armbian-next: docker: archive logfiles again; warn & greet testers when archiving many old logfiles 2023-02-18 07:44:00 -03:00
Ricardo Pardini
da1cf2b475
armbian-next: rename too-long main function to prep_conf_main_build_single()
- this commit is proof of OCD (old name didn't fit stacktrace space)
2023-02-18 07:43:42 -03:00
Ricardo Pardini
242a5dad1f
armbian-next: somewhat-better handling/logging when using Docker (don't show two logfiles to inspect)
- do not show path to Docker run-host's logfile, unless the Docker run itself didn't produce logs
- introduce `global_final_exit_code` to allow to maintain/preserve Docker's exit code without triggering an error host-side
2023-02-18 07:43:41 -03:00
Ricardo Pardini
bf891d54d9
armbian-next: split prepare_host(), fix .tmp reset trap
- `prepare_host()`: split; do checks earlier and allow them to be interactive
- introduce `exit_if_countdown_not_aborted()` for "Low Disk Space" and other critical conditions
- split `prepare_host()` into interactive & non-interactive parts
- split off `clean_deprecated_mountpoints()` from prepare into `cleaning.sh`
- introduce and use `reset_uid_owner_non_recursive()` for `.tmp` reset in trap, to avoid disasters
- add more logging sections to default-build.sh, avoid unlogged parts
2023-02-18 07:43:10 -03:00
Ricardo Pardini
2d9f9216eb
armbian-next: introduce tmpfs-utils.sh; put LOGDIR and WORKDIR under tmpfs; set CCACHE_TEMPDIR under WORKDIR
- introduce generic `prepare_tmpfs_for()`, which manages it's own cleanup/dir removal
- use it for `WORKDIR` (which is `TMPDIR`) and `LOGDIR`
  - adapt previous cleanup handlers for those, so they delete their contents but not the dir itself (which might be mounted)
- also: make `ARMBIAN_LOG_CLI_ID` readonly together with other superglobals
- set `XDG_RUNTIME_DIR` & `XDG_RUNTIME_DIR` together with `TMPDIR`
- kernel-make.sh: pass `CCACHE_TEMPDIR` down to Kernel make (thanks @the-Going)
2023-02-18 07:43:09 -03:00
Ricardo Pardini
d6200a30e6
armbian-next: make superglobals readonly (DEST/WORKDIR/LOGDIR/SDCARD/MOUNT and others) 2023-02-18 07:43:08 -03:00
Ricardo Pardini
aaf391bf19
armbian-next: mark Vagrant CLI as unimplemented 2023-02-18 07:43:04 -03:00
Ricardo Pardini
1075f1b962
armbian-next: make sure temporary kernel and u-boot CLI commands always build a kernel/u-boot even if cached
- really, it's time to split off those into actual, proper CLIs
- remove a forgotten `KDEB_CHANGELOG_DIST` left there
2023-02-18 07:43:02 -03:00
Ricardo Pardini
eb7b80983b
armbian-next: fix re-launching of CLI commands under sudo (Docker was ok) 2023-02-18 07:42:52 -03:00
Ricardo Pardini
b94074ec0a
armbian-next: Docker: abstract away and only run "docker info" once, it's very expensive 2023-02-18 07:42:47 -03:00
Ricardo Pardini
7beeae6219
armbian-next: Python tooling: use consolidated+hashed+cached pip base/pycache; don't pip during Dockerfile build, nor cli-requirements
- consolidate at `prepare_python_and_pip()`
- sanity check for Python version 3.9+ regardless of HOSTRELEASE
- TODO: pip vs sudo/root: need pip 22.2+ to curb warning, not doing it
2023-02-18 07:42:44 -03:00
Ricardo Pardini
878332fea1
armbian-next: cli: half-assed, legacy based, kernel and u-boot cli commands
- @TODO: a placeholder for now, needs refactoring after patching-hash/versioned-kernel work
2023-02-18 07:42:35 -03:00
Ricardo Pardini
ef36016ee4
armbian-next: logging: reinit logging after processing cmdline parameters
- so `DEBUG=yes` actually works
2023-02-18 07:42:34 -03:00
Ricardo Pardini
d4d3c1346c
armbian-oleg: split hostdeps again, full of ifs, and reduce the minimum set of pkgs for Oleg-conditions while trying to keep Docker full
- probably breaks a lot of stuff; c++ compilers, distcc, possibly others
- split into late_prepare_host_dependencies / early_prepare_host_dependencies calling adaptative_prepare_host_dependencies() full of if's

also:
- calc deps out of configuration
- reorg checks into obtain_and_check_host_release_and_arch()
2023-02-18 07:42:11 -03:00