Commit Graph

21 Commits

Author SHA1 Message Date
Ricardo Pardini
d69f09d676
kernel-config: ignore artifact cache, makes no sense with make menuconfig (fixes #4854) (#4856) 2023-02-24 10:44:02 +01:00
Ricardo Pardini
9bffa5e749
armbian-next: artifacts: firmware/rootfs and actual usage of artifacts during image build
- artifacts: introduce `ARTIFACT_IGNORE_CACHE=yes`
- artifacts: introduce `DONT_BUILD_ARTIFACTS`, list of artifacts that if not found cached, fail the build
- kernel_package_source() is no more
- a long dissertation about kernels, families, and the universe
- artifacts: actually use rootfs artifact for image build
- artifacts: detangle via artifact_base_dir
- artifacts: rootfs: use folders in artifact_name; include cache_type
- artifacts: some cleanups / validations
- rootfs artifact; drop old rootfs cli
- artifacts: new CLI shortcuts; remove old firmware CLI
- artifacts: full firmware & usage
- use firmware artifacts in image build and install its debs
- kernel artifact sans legacy; use tmpdir for .deb target for all packages
- legacy artifact versions is no more; pack/unpack now in common obtain;
- artifacts: uboot: cleanup legacy renaming, use artifact version directly
- artifacts: add firmware (small) artifact
- deploy uboot to loop from artifact; allow tty to artifact; todos for cleaning
- fixes, kernel dtb/headers conditional; remove `.git` from Makefile url; use mapfile for finding files to hash
  - completely remove KERNEL_HAS_WORKING_HEADERS_FULL_SOURCE and `kernel_package_callback_linux_headers_full_source()`
  - don't use realpath for artifact_file_relative
  - curb some warnings
  - fix: only install headers & dtbs if such exist
- kernel .config hook modification hash versioning
- OCI_TARGET_BASE vs per-artifact defaults; only deploy to remote from CLI with OTB
- artifact revolver & installing into image
  - add artifact_map_packages and artifact_map_debs dicts
  - revolver accumulates all info
  - REPOSITORY_INSTALL is no more (for uboot/kernel, later others)
  - rename `VER` to `IMAGE_INSTALLED_KERNEL_VERSION`
2023-02-18 07:46:03 -03:00
Ricardo Pardini
ca6bf9046b
armbian-next: remove all traces of Vagrant and FEL boot 2023-02-18 07:46:00 -03:00
Ricardo Pardini
a7fcd44942
armbian-next: cli: u-boot alias: don't stop to interactively ask for kernel configuration during u-boot build 2023-02-18 07:45:25 -03:00
Ricardo Pardini
bef7f8fee4
armbian-next: artifacts: introduce kernel/u-boot artifacts; git ref2info; "memoizer"; some hashing
- artifacts: u-boot/kernel - pt7 - adapt legacy/artifact versions; use common `capture_rename_legacy_debs_into_artifacts()`
- artifacts: u-boot - pt6: add artifact for u-boot
  - use artifact version / reason in actual u-boot .deb if present
- artifacts: kernel - pt5: tune kernel version, refactor
- artifacts: kernel - pt4: squash unrelated bugs that show up; move `prepare_compilation_vars()` to default build
- artifacts: kernel - pt3: drivers+patches+.config hashing
  - split file hashing function from drivers-harness; fix it so filenames are relative and sorted; sort from ${SRC}, always
  - aplit prepare_kernel_config_core_or_userpatches() from `kernel_config_initialize()`
- artifacts: kernel - pt2:  memoizing git ref2info
- artifacts: kernel - pt1:  versioning
2023-02-18 07:45:20 -03:00
Ricardo Pardini
9dc24840c3
armbian-next: firmware-deb: reworked: Provides/Conflicts instead of Replaces; cp -> git archive; new firmware CLI; FULL version .deb out of tmpfs 2023-02-18 07:45:04 -03:00
Ricardo Pardini
dbfa7329ab
armbian-next: cli: kernel cli forces KERNEL_CONFIGURE=no; @TODO this should only "default to", not force. for proper kernel cli, later 2023-02-18 07:44:44 -03:00
Igor
00bc1b4a1e
armbian-next: rootfs: introduce ROOTFS_COMPRESSION_RATIO, default 5 for general use, default 15 for rootfs CLI; fix pv | tar bug; remove GPG signing
- Remove PGP signing from the script as its done in GHA
- Also remove generating .current control file as its obsolete in this design

Co-authored-by: Ricardo Pardini <ricardo@pardini.net>
2023-02-18 07:44:42 -03:00
Ricardo Pardini
1cec42392e
armbian-next: GHA & rootfs adventures, pt 1
- rootfs: create readonly global `rootfs_cache_id` (type+hash+date/version)
- add "oras-upload" CLI command (dumb, no retries, single target/single source)
- docker: mount-bind for & re-pass envs `GITHUB_OUTPUT` and `GITHUB_STEP_SUMMARY` down to Docker
- introduce `github-actions.sh::github_actions_add_output()`
- during logs cleanup, dump the Markdown log into GITHUB_STEP_SUMMARY if it is available
2023-02-18 07:44:41 -03:00
Ricardo Pardini
d17157885c
armbian-next: rootfs: converge rootfs-cache and rootfs together; exchange BOARD for ARCH 2023-02-18 07:44:28 -03:00
Ricardo Pardini
28d89e2c91
armbian-next: rootfs: new CLI command for rootfs cache building; further refactorings
- _told ya it would all make sense eventually..._
- introduce new, real, non-alias `rootfs` command in `cli-rootfs` -- this _only_ builds rootfs, nothing else
- no more `main_default_build_single()`; instead `full_build_packages_rootfs_and_image()`
- introduce `do_with_default_build()` wrapper, for things that need host prepared + aggregation to run
- CLI-specific parts of `main_default_build_single()` moved to cli-build
- put prepare/cleanup pair of `prepare_rootfs_build_params_and_trap()` and `trap_handler_cleanup_rootfs_and_image()` into new `trap-rootfs.sh` (this needs further splitting for the image part)
- refactor `get_or_create_rootfs_cache_chroot_sdcard()` into `build_rootfs_only()`, `calculate_rootfs_cache_id()`
- update library with new files
2023-02-18 07:44:22 -03:00
Igor
90c46e0215
armbian-next: cli: add CLI alias command rootfs-cache for creating rootfs cache 2023-02-18 07:44:02 -03:00
Ricardo Pardini
1075f1b962
armbian-next: make sure temporary kernel and u-boot CLI commands always build a kernel/u-boot even if cached
- really, it's time to split off those into actual, proper CLIs
- remove a forgotten `KDEB_CHANGELOG_DIST` left there
2023-02-18 07:43:02 -03:00
Ricardo Pardini
878332fea1
armbian-next: cli: half-assed, legacy based, kernel and u-boot cli commands
- @TODO: a placeholder for now, needs refactoring after patching-hash/versioned-kernel work
2023-02-18 07:42:35 -03:00
Ricardo Pardini
97f6836705
armbian-next: Python patching delusion, pt4: series & better markdown
- Python patching: hopefully better Markdown: split series dir; detect more status/problems; arche only for missing Subject
  - Python patching: archeology only for missing Subject:, not description; clarify CLI pushing
  - Python patching: use `{file_base_name}.patch` for archeology as `file_name` might include a dir
- Python patching: fix: don't skip board/target patches for no reason
- Python patching: fix for series.conf patches (sunxi/sunxi64): don't sort; mark as series and show on summary
- Python patching: don't fail if archeology found no commits (but add warning)
- Python patching: CLI command `kernel-patches-to-git` with archeology summary, and git pushing of results & summary
  - Python patching: patches-to-git small fixes, auto-push if it's rpardini
  - Python patching: add `patch-kernel` CLI command
- Python patching: commit README.md & gh-pages workflow when apply_patches_to_git
  - Python patching: hopefully better markdown
- Python patching: `git add` everything all at once, for speed
2023-02-18 07:40:54 -03:00
Ricardo Pardini
9657ceac9b
armbian-next: docker: implement docker-purge and docker-shell cli commands; deprecated mountpoints 2023-02-18 07:40:35 -03:00
Ricardo Pardini
093336e914
armbian-next: cli: distccd launcher for build farm
- relaunched under docker
- possibly much nicer if zeroconf enabled?
- this thing is ignoring signals. can't ctrl-c out of it. why? @TODO
- pt 1, subpt 3: distccd - distcc-pump - avahi
- pt 1, subpt 3: distccd + distcc-pump
- pt 1, subpt 2: distccd + avahi
- pt 1, subpt 1: distccd
2023-02-18 07:40:18 -03:00
Ricardo Pardini
a9e88f58e5
armbian-next: json-info command, wrapping up the Python tooling; running in Docker; fix config dump 2023-02-18 07:40:09 -03:00
Ricardo Pardini
75de1c2d82
armbian-next: remove usage of REQUIREMENTS_DEFS_ONLY; initialize extensions for requirements cli command & docker pre-game deps 2023-02-18 07:40:07 -03:00
Ricardo Pardini
bc7d388b49
armbian-next: cli: a bit more untangling of re-launching; introduce ARMBIAN_CLI_RELAUNCH_COMMAND and cli_standard_relaunch_docker_or_sudo()
- `configdump` cli: force root/docker
2023-02-18 07:40:06 -03:00
Ricardo Pardini
d24d3327a8
armbian-next: the great cli entrypoint (+docker) rewrite; introduce USE_LOCAL_APT_DEB_CACHE replacing apt-cacher-ng
- armbian-next: introduce `USE_LOCAL_APT_DEB_CACHE` (default `=yes`) as alternative/in addition to `apt-cacher-ng` (eg, in Docker)
  - this uses `cache/aptcache/${RELEASE}-${ARCH}` (in the host) for
      - apt cache, by bind-mounting it to `${SDCARD}/var/cache/apt` in the `chroot_sdcard_apt_get()` runner and its usages
      - debootstrap, by passing it `--cache-dir`
  - utility function to help understand what is happening to cache during usage
  - apt itself mantains this cache, removing old packages when new ones are installed. apt does this _by default_
      - introduce `DONT_MAINTAIN_APT_CACHE=yes` to skip out of automatic apt maintenance of apt cache, eg, during `remove`s
      - don't do `apt clean` and such if using local cache, that would clean the cache, not the chroot
  - clean up `install_deb_chroot()` a little, find an unrelated bug there
- WiP: the great cli entrypoint (+docker) rewrite, Phase 6: relaunching structure; re-pass ARMBIAN_BUILD_UUID; use ARMBIAN_COMMAND for log filename; fix for output/logs dir perms
- WiP: the great cli entrypoint (+docker) rewrite, Phase 5: cleanups 4/x; better logging, check & force `DEST_LANG`
- WiP: the great cli entrypoint (+docker) rewrite, Phase 5: cleanups 3/x; don't write to stderr in generated Dockerfile
  - it's `drastic red` on non-buildx dockers
- WiP: the great cli entrypoint (+docker) rewrite, Phase 5: cleanups 2/x, logging
- WiP: the great cli entrypoint (+docker) rewrite, Phase 5: cleanups 1/x
  - source configs in a logging section.
  - Docker: silent, fast retries to make sure `docker system df` works
  - shut-up `chown` (no `-v`) output related to  `SET_OWNER_TO_UID`
  - ask user to wait while `DESTIMG` is rsync'ed to `FINALDEST` -- it's potentially very slow
  - use green apple for Mac logging, instead of red apple which might imply error...
- WiP: the great cli entrypoint (+docker) rewrite, Phase 4: run as non-root, maybe-with-Docker
  - introduce `is_docker_ready_to_go()`; if it is, and we're not root, use Docker instead of sudo. <- GOOD IDEA? BAD IDEA? lol
  - introduce `SET_OWNER_TO_UID` var to be passed to Docker/sudo so written files are owned by the launching user, not root.
    - introduce `mkdir_recursive_and_set_uid_owner()` and `reset_uid_owner()` to reset owner based on `SET_OWNER_TO_UID`
    - use it for userpatches files created, logs, and output files, including images and debs.
  - @TODOs ref. `$SUDO_USER` which I think the old version of this?
  - add a lot of @TODOs, ref being able to relaunch something that's not `build` inside Docker, also add/change params and configs and command.
    - initially add `ARMBIAN_DOCKER_RELAUNCH_EXTRA_ARGS`
- WiP: the great cli entrypoint (+docker) rewrite, Phase 3: rpardini is demented, v3
- WiP: the great cli entrypoint (+docker) rewrite, Phase 2: rpardini is demented
- WiP: the great cli entrypoint (+docker) rewrite, Phase 1
- armbian-next: WiP: Docker: actually use the GHA-image as base; pull it every 24hs.
  - using image in my private repo.
  - this has significant speedup to "start building time" on the 1st run
  - move some Linux specific stuff to its own if
  - add comments and todo
- armbian-next: WiP: Docker, high-WiP, beginnings of Armbian mount dict, with linux/darwin preferences
- armbian-next: WiP: Docker, configure `BUILDKIT_COLORS`
- armbian-next: WiP: Docker, make docker image from Dockerfile more compact by flattening layers
- armbian-next: `logging`: add whale indicator if build running under Docker
- armbian-next: WiP: `docker`: working with `bookworm`, `sid`, and `jammy` on Darwin & Linux; works with `bullseye` on Linux only
- armbian-next: WiP: `docker`: force ARMBIAN_RUNNING_IN_CONTAINER both in Dockerfile and passed as `--env`; apt update and install in same layer; back to jammy
- armbian-next: introduce `armbian_is_running_in_container()` and `armbian_is_host_running_systemd()`, replacing `systemd-detect-virt` in multiple spots
- WiP: try with debian:bullseye -- can't detect docker at all
- armbian-next: WiP: 2nd stab at new Docker support; Darwin still works; Linux `docker.io` working
  - gen .dockerignore together with Dockerfile
  - split in funcs
  - hacks for Linux and `/dev/loop` stuff, CONTAINER_COMPAT=yes
  - mac still works, Linux stuff would break it but I if'fed
- armbian-next: the secrets of `CONTAINER_COMPAT` revealed; add size checking to check_loop_device() and avoid retry when `mknod`ing
  - this fails for the right reasons now, causing retries, which are then retried and work ;-)
  - this is related to building under Docker on Linux, using docker.io package (not docker-ce)
- armbian-next: remove `.dockerignore` and add it to `.gitignore`; it's going to be auto-generated
- armbian-next: `.dockerignore`: Docker context should only have minimal files and folders, to speed up Dockerfile build
  - IMPORTANT: `.dockerignore` is going to be generated from now on: so this is the last commit with changes before removal
-  armbian-next: WiP: initial stab at new Docker support; really run the passed cmdline; add Dockerfile to gitignore
-  armbian-next: WiP: initial stab at new Docker support; generate Dockerfile; introduce REQUIREMENTS_DEFS_ONLY
  - uses REQUIREMENTS_DEFS_ONLY
  - works on Docker Desktop on Mac;
  - linux TBA
- armbian-next: don't error out if `.git` not present; other small fixes
- armbian-next: general "work or at least don't misbehave when run on a very bare ubuntu:latest instance"
  - can't assume things, for example:
  - that `sudo` will be available; it might not, and might be already root, no reason to fail
  - that `/etc/timezone` will exist
  - that `systemd-detect-virt` will be available
  - that `git` will be available
  - that `locale-gen` will be available
2023-02-18 07:39:43 -03:00