Commit Graph

105 Commits

Author SHA1 Message Date
Ricardo Pardini
f992dcd466 git-ref2info: retries for getting Makefile body (https://raw.githubusercontent.com has been failing a lot) 2023-05-01 22:47:03 +02:00
Ricardo Pardini
f8ddf7f9e2 🔥 JSON info pipeline: v18
- pipeline: add `pipeline` context object to targets; use it to filter artifacts and images to build; warn about oci-name with multiple oci-tags
- pipeline: better error messages when info's fail; show some (out-of-order) progress messages during parallel info gathering
- pipeline: targets-compositor: add `not-eos` inventory
- TARGETS_FILENAME, log all OCI lookups
- SKIP_IMAGES
- IMAGES_ONLY_OUTDATED_ARTIFACTS
- no dash in chunk id in JSON
- pipeline: very initial chunking, using the same outputs
- pipeline: template targets, `items-from-inventory:` inventory expansion, CHECK_OCI=yes, CLEAN_MATRIX=yes, CLEAN_INFO=yes, many fixes
- cli: `inventory` / `targets` / `matrix` / `workflow`
- pipeline: workflow beginnings
- pipeline: general log cleanup + OCI stats / better miss handling
- pipeline: fixes/reorg
- pipeline: catch & log JSON parsing errors
- pipeline: gha matrix: use IMAGE_FILE_ID as job description
- pipeline (delusion): gha workflow output, based on old matrix code
- pipeline: better parsing and reporting of stderr log lines (under `ANSI_COLOR=none`)
- pipeline: mapper-oci-uptodate: use separate positive/negative cache dirs (GHA will only cache positives); cache negs for 5 minutes locally
- pipeline: output-gha-matrix artifacts + images
  - pipeline: output-gha-matrix artifacts + images: "really" and fake 1-item matrix if empty
- pipeline: move files into subdir; update copyright & cleanup
- pipeline: refactor bash jsoninfo driver a bit
- pipeline: outdated-artifact-image-reducer
- pipeline: introduce `target_id` at the compositor, aggregate it at the reducer, carry it over in the artifact info mapper
- pipeline: mapper-oci-uptodate
- pipeline: info-gatherer-artifact, with PRE_PREPARED_HOST
- pipeline: refactor/rename info-gatherer-image.py
- pipeline: beginnings
2023-05-01 22:46:25 +02:00
Ricardo Pardini
59d6690a11 git-ref2info: many fixes for nonstandard repos; try a few different dereference and refs/ combinations in a certain order; remove codeaurora repo handling
- git-ref2info: when resolving SHA1's for 'xxx' tag or branch, try first `refs/heads/xxx`, then just 'xxx'; validate SHA1 with 40-char regexp
  - this works-around Gerrit-style repositories that have a 'refs/for/xxx' ref that gets in the way
  - case in point: https://github.com/hardkernel/u-boot/tree/odroidc-v2011.03 (has both refs/heads/odroidc-v2011.03 and refs/for/odroidc-v2011.03)
- git-ref2info: when processing tags for a SHA1, try both dereference and non-dereferenced, in that order, to get a SHA1
  - this solves old problem reported in #4916: GitHub incorrectly processes dereference requests for tags that are not actually annotated
  - case in point: https://github.com/ayufan-rock64/linux-kernel/releases/tag/4.4.202-1237-rockchip-ayufan
- `codeaurora` is no more, https://bye.codeaurora.org/
2023-04-28 21:32:52 +02:00
amazingfate
97a2e41f62 use common sed for GITHUB_SOURCE replace 2023-04-21 14:21:28 +02:00
Ricardo Pardini
8cd830d6e1 memoize-cached: add flock locking to run_memoized() to avoid error when run in parallel 2023-04-18 09:27:38 +02:00
amazingfate
a0f59bf914 use ghproxy to accelerate memoized_git_ref_to_info 2023-04-17 13:42:53 +02:00
Ricardo Pardini
9b229f25ad artifact-uboot: add hashes for extension and old-timey -Hooks and -Variables hash (BOOTDELAY et al)
- helper for hashing old-timey hooks: `calculate_hash_for_function_bodies()` in `hash-files.sh` cos I'm lazy
2023-04-03 21:21:03 +02:00
Ricardo Pardini
d2bb1f893a extensions: many fixes; allow for dumping the source of hooks, via dump_extension_method_sources_functions() and dump_extension_method_sources_body()
- fix: indent of generated sources
- fix: no reason to source the generated file in a loop. do it once. 20x faster?
- `dump_extension_method_sources_functions "name_of_hook"`: get function definitions (inside braces)
- `dump_extension_method_sources_body "name_of_hook"`: get the function bodies (sans-braces)
2023-03-31 09:12:44 +02:00
Ricardo Pardini
61d85648ea cleaning: stop lying about CLEAN_LEVEL=debs -- now it simply deletes every .deb in the folder (same as =alldebs) 2023-03-31 09:12:44 +02:00
Ricardo Pardini
ed0b406265 lib: fix: replace undue export statements with declare -g 2023-03-31 09:12:44 +02:00
Ricardo Pardini
fb4477afb8 oras: add --insecure to further retries if we get a failure on first try, to workaround GitHub's March/2023 mess with certificates 2023-03-31 09:12:44 +02:00
Ricardo Pardini
a77d5f3c70 fix: declare some variables that were leaking into global namespace for no reason
- especially `for VAR in ...` and VAR leaks sans the declare
- rename `fragment_manager_cleanup_file` to `extension_manager_cleanup_file`
2023-03-31 09:12:44 +02:00
Ricardo Pardini
02c561ce0e bat-cat: introduce tooling support for bat (colorized/smart cat), including DEBIAN/xxx syntaxes; cached in Docker image 2023-03-31 09:12:44 +02:00
Ricardo Pardini
b820a4d68f tools: oci-oras/shellcheck: fix, test if binary already in place before considering copying it, but only after BIN is actually defined 2023-03-31 09:12:44 +02:00
Ricardo Pardini
2f4f117000 prepare ORAS tooling early, so we don't get spurious failures "only on first build" 2023-03-17 08:32:11 +01:00
Ricardo Pardini
630ee512cb stop the build if LIB_TAG is set at all; some users expect it to work while it doesn't 2023-03-17 08:32:11 +01:00
Ricardo Pardini
6b965c605c completely remove any traces of KERNEL_ONLY and exit with error if set 2023-03-17 08:32:11 +01:00
Igor Pečovnik
b7b8eb7b72
Add / modify (c) in bash scripts (#4922)
* Add / modify (c) in bash scripts

Signed-off-by: Igor <igor@armbian.com>

* Add (c) to the source config files

---------

Signed-off-by: Igor <igor@armbian.com>
2023-03-09 18:30:40 +01:00
Vyacheslav Bocharov
123b31b161 Replace absolute path to git-tree with relative, so it is accesible without docker builder. 2023-02-28 09:26:12 -03:00
Ricardo Pardini
c2004faf9c
git-ref2info: fix https Makefile URL for gitlab.com (eg, rk35xx-legacy) (#4848) 2023-02-22 20:59:56 +01:00
Ricardo Pardini
26783b050f
armbian-next: pip3: workaround new problems introduced by pip 23+ 2023-02-18 07:46:47 -03:00
Ricardo Pardini
85f3fb5c93
armbian-next: oci-oras: don't capture our own debugs as part of the manifest (d'oh -- thanks RichN) 2023-02-18 07:46:45 -03:00
Ricardo Pardini
2350c5697f
armbian-next: cleaning: deprecate CLEAN_LEVEL=debs -- doesn't do anything, just shows what would be cleaned 2023-02-18 07:46:16 -03:00
Ricardo Pardini
e46233084a
armbian-next: memoize: expire all memoized files after 1 hour 2023-02-18 07:46:15 -03:00
Ricardo Pardini
3d8ee5ce38
armbian-next: oras: add 3/5 retries to oras pull/push 2023-02-18 07:46:11 -03:00
Ricardo Pardini
8694daad78
armbian-next: chroot-helpers: mount_chroot()/umount_chroot(): add warning with stacktrace if called sans logging section 2023-02-18 07:46:05 -03:00
Ricardo Pardini
26f0700da8
armbian-next: chroot-helpers: if unable to mount devpts, use a mount bind from host
Co-authored-by: Paolo Sabatino <paolo.sabatino@gmail.com>
2023-02-18 07:46:04 -03:00
Ricardo Pardini
9bffa5e749
armbian-next: artifacts: firmware/rootfs and actual usage of artifacts during image build
- artifacts: introduce `ARTIFACT_IGNORE_CACHE=yes`
- artifacts: introduce `DONT_BUILD_ARTIFACTS`, list of artifacts that if not found cached, fail the build
- kernel_package_source() is no more
- a long dissertation about kernels, families, and the universe
- artifacts: actually use rootfs artifact for image build
- artifacts: detangle via artifact_base_dir
- artifacts: rootfs: use folders in artifact_name; include cache_type
- artifacts: some cleanups / validations
- rootfs artifact; drop old rootfs cli
- artifacts: new CLI shortcuts; remove old firmware CLI
- artifacts: full firmware & usage
- use firmware artifacts in image build and install its debs
- kernel artifact sans legacy; use tmpdir for .deb target for all packages
- legacy artifact versions is no more; pack/unpack now in common obtain;
- artifacts: uboot: cleanup legacy renaming, use artifact version directly
- artifacts: add firmware (small) artifact
- deploy uboot to loop from artifact; allow tty to artifact; todos for cleaning
- fixes, kernel dtb/headers conditional; remove `.git` from Makefile url; use mapfile for finding files to hash
  - completely remove KERNEL_HAS_WORKING_HEADERS_FULL_SOURCE and `kernel_package_callback_linux_headers_full_source()`
  - don't use realpath for artifact_file_relative
  - curb some warnings
  - fix: only install headers & dtbs if such exist
- kernel .config hook modification hash versioning
- OCI_TARGET_BASE vs per-artifact defaults; only deploy to remote from CLI with OTB
- artifact revolver & installing into image
  - add artifact_map_packages and artifact_map_debs dicts
  - revolver accumulates all info
  - REPOSITORY_INSTALL is no more (for uboot/kernel, later others)
  - rename `VER` to `IMAGE_INSTALLED_KERNEL_VERSION`
2023-02-18 07:46:03 -03:00
Ricardo Pardini
8900eea0af
armbian-next: git-ref2info: add support for gitlab.com; exit_with_error if SHA1 can't be found 2023-02-18 07:46:02 -03:00
Ricardo Pardini
7f1fb1cd69
armbian-next: hash-files: if no files to hash, return 16 zeros, instead of hanging forever waiting for stdin 2023-02-18 07:45:59 -03:00
Ricardo Pardini
85d13cf8ce
armbian-next: git/ref2info: fixes for using commit:<sha1> 2023-02-18 07:45:54 -03:00
Ricardo Pardini
399d9964dd
armbian-next: python/pip: include pip pkgs in Dockerfile; use them if available instead of downloading 2023-02-18 07:45:51 -03:00
Ricardo Pardini
03dd0d623b
armbian-next: debs: introduce shellcheck for DEBIAN/postinst and others, before dpkg build 2023-02-18 07:45:50 -03:00
Ricardo Pardini
f0d3ea7961
armbian-next: tooling: introduce helper/downloader for shellcheck; include in Dockerfile 2023-02-18 07:45:48 -03:00
Ricardo Pardini
871f3cfcee
armbian-next: tooling: include ORAS tools in Dockerfile; use them if available instead of downloading 2023-02-18 07:45:47 -03:00
Ricardo Pardini
da02fde9cc
armbian-next: git: backport safe.directory method from master (just add *, once) -- thanks Igor 2023-02-18 07:45:44 -03:00
Ricardo Pardini
bf53bd2988
armbian-next: tmpfs: use size=99% for all tmpfs's; (rootfs/WORKDIR/chroot); thanks Igor for suggestion 2023-02-18 07:45:42 -03:00
Ricardo Pardini
3c79aaa79f
armbian-next: hash/cache: remove some cruft, some better logging 2023-02-18 07:45:39 -03:00
Ricardo Pardini
555a972442
armbian-next: git ref2info: dereference annotated tags to get "real" commit SHA1; fail if curl failed; validate parse_makefile_version() 2023-02-18 07:45:38 -03:00
Ricardo Pardini
68d76d6f3a
armbian-next: squash some shellcheck warnings; don't local and assign subshell value in same statement, that masks errors (declare/local almost-always works) 2023-02-18 07:45:31 -03:00
Ricardo Pardini
6258ea566a
armbian-next: shellcheck: squash a bunch of 'critical' warnings 2023-02-18 07:45:29 -03:00
Ricardo Pardini
bef7f8fee4
armbian-next: artifacts: introduce kernel/u-boot artifacts; git ref2info; "memoizer"; some hashing
- artifacts: u-boot/kernel - pt7 - adapt legacy/artifact versions; use common `capture_rename_legacy_debs_into_artifacts()`
- artifacts: u-boot - pt6: add artifact for u-boot
  - use artifact version / reason in actual u-boot .deb if present
- artifacts: kernel - pt5: tune kernel version, refactor
- artifacts: kernel - pt4: squash unrelated bugs that show up; move `prepare_compilation_vars()` to default build
- artifacts: kernel - pt3: drivers+patches+.config hashing
  - split file hashing function from drivers-harness; fix it so filenames are relative and sorted; sort from ${SRC}, always
  - aplit prepare_kernel_config_core_or_userpatches() from `kernel_config_initialize()`
- artifacts: kernel - pt2:  memoizing git ref2info
- artifacts: kernel - pt1:  versioning
2023-02-18 07:45:20 -03:00
Ricardo Pardini
19aa7e95cf
armbian-next: introduce assert_prepared_host() and assert before using any Python, to un-mask any further fallacies 2023-02-18 07:45:19 -03:00
Ricardo Pardini
44733e1afc
armbian-next: git: refactor git_parse_ref() out of fetch_from_repo() for reuse 2023-02-18 07:45:12 -03:00
Ricardo Pardini
8574b87f8d
armbian-next: curb some logging for git non-checkouts 2023-02-18 07:45:06 -03:00
Ricardo Pardini
5fa7bf735b
armbian-next: git: fetch_from_repo(): add fetched_revision(_ts) as well as checked_out_revision(_ts); introduce param do_checkout=no to only fetch, not checkout 2023-02-18 07:45:01 -03:00
Ricardo Pardini
9849990031
armbian-next: all chroots: mount /var/tmp as tmpfs; create dir /run/user/0 and mount as tmpfs @TODO cleanups 2023-02-18 07:44:55 -03:00
Ricardo Pardini
1cec42392e
armbian-next: GHA & rootfs adventures, pt 1
- rootfs: create readonly global `rootfs_cache_id` (type+hash+date/version)
- add "oras-upload" CLI command (dumb, no retries, single target/single source)
- docker: mount-bind for & re-pass envs `GITHUB_OUTPUT` and `GITHUB_STEP_SUMMARY` down to Docker
- introduce `github-actions.sh::github_actions_add_output()`
- during logs cleanup, dump the Markdown log into GITHUB_STEP_SUMMARY if it is available
2023-02-18 07:44:41 -03:00
Ricardo Pardini
8a63a0ae20
armbian-next: downloads.sh: some cleanup and add debugging to get_urls() 2023-02-18 07:44:17 -03:00
Ricardo Pardini
18c9946a4d
armbian-next: countdown: if to test -t 0 (0: stdin, 1:stdout) for interactive-ness; also test in if-not-aborted version 2023-02-18 07:44:11 -03:00