Commit Graph

607 Commits

Author SHA1 Message Date
Ricardo Pardini
4bf98ebfab artifact kernel/uboot: add ARCH to artifact_input_variables; used for (maybe) mapping to runner tags in JSON pipeline 2023-05-03 07:49:53 +02:00
igorpecovnik
ba1a84a5e8
Adjusting ORAS cache location, runners (#5129)
* Move packages to new location
* Adjust build runners
* Switch to OS

Signed-off-by: Igor <igor@armbian.com>
---------

Signed-off-by: Igor <igor@armbian.com>
Co-authored-by: Ricardo Pardini <ricardo@pardini.net>
2023-05-02 11:08:34 +02:00
Vyacheslav Bocharov
64f434ab33 Revert "fix rtl88x2cs compilation for kernel 6.3" as fixed in rtl88x2cs repository.
This reverts commit dc9f751012.
2023-05-02 12:04:51 +03:00
Ricardo Pardini
3767729dc5 rootfs: add artifact name prefix and EXTRA_ROOTFS_NAME suffix (for extensions), yyyymm one-per-month auto-version in place of REVISION for artifact version
- rootfs is not a deb package, and thus not subject to the same strict ordering rules of `artifact_prefix_version`
  - replace with `yyyymm` (`202305`) in version so we get one-per-month automatically
2023-05-01 22:47:12 +02:00
Ricardo Pardini
f992dcd466 git-ref2info: retries for getting Makefile body (https://raw.githubusercontent.com has been failing a lot) 2023-05-01 22:47:03 +02:00
Ricardo Pardini
d890b418c7 kernel/uboot/atf: introduce kernel-patch, uboot-patch, atf-patch, uboot-config, kernel-config CLI commands; enhanced manual patching; block deprecated ways
- all interactive commands now **don't build the artifact** anymore; just patches/.configs are produced and then build ends
  - user is required to put the produced patches in the right place and build again, for full consistency
- split ATF and U-BOOT manual patching process; use CLI command `atf-patch` to patch ATF, and `uboot-patch` to patch u-boot
- non-interactive artifact builds are now 100% sans-stdin
- introduce `uboot-config` CLI command; still experimental, only produces a defconfig and not a patch
- reworked `userpatch_create()` to be (hopefully) more useful:
  - detects a previous patch and offers to apply it before continuing
  - enters a loop showing the diff, and only proceeds when user indicates he's happy with the patch
  - produces `mbox`-formatted patches via `format-patch` and standard Armbian parameters
  - uses MAINTAINER and MAINTAINERMAIL instead of git configuration (so it works in containers)
- don't allow image builds with any patching or configuring _at all_ (it has been deprecated with a warning for months already, and results are inconsistent)
2023-05-01 22:46:55 +02:00
Ricardo Pardini
f77edd0591 rtl8822bs: fix: remove binaries (firmware?) being added to kernel driver source and breaking patching; related to #5117 2023-05-01 22:46:43 +02:00
Ricardo Pardini
e74ab59f48 kernel-make: introduce KERNEL_EXTRA_CFLAGS; fill it to add to what's passed to KCFLAGS; related to #5117 2023-05-01 22:46:43 +02:00
Ricardo Pardini
369eb80f9c driver_uwe5622_allwinner: only apply "pre-6.1" park-link patch if kernel >= 5 (eg: not for rockchip64-legacy); possibly fixes #5120 2023-05-01 22:46:43 +02:00
Ricardo Pardini
91bb00bfa2 artifact-uboot: fix: include UBOOT_TARGET_MAP in variables hash of version 2023-05-01 22:46:43 +02:00
Ricardo Pardini
c692676da7 kernel-debs: postinst/etc: fix: unify is_boot_dev_vfat() function; make work during image build too via env var
- determinining if `/boot` is vfat is more complex than it looks;
  - use a common function to unify across all scripts
- during image build, the kernel install is always done with a non-fat, non-mounted /boot
  - use a special variable passed through the env so unified function can know ahead of time
  - introduce `extra_apt_envs` param to `chroot_sdcard_apt_get()`; array with extra environment vars
- if vfat /boot:
  - `linux-update-symlinks` is not called (Debian would just choke on this)
  - don't even try to symlink Armbian stuff, move instead
- more/better logging board-side (unfortunately mixed in all the set -x debugs)
2023-05-01 22:46:43 +02:00
Ricardo Pardini
f8ddf7f9e2 🔥 JSON info pipeline: v18
- pipeline: add `pipeline` context object to targets; use it to filter artifacts and images to build; warn about oci-name with multiple oci-tags
- pipeline: better error messages when info's fail; show some (out-of-order) progress messages during parallel info gathering
- pipeline: targets-compositor: add `not-eos` inventory
- TARGETS_FILENAME, log all OCI lookups
- SKIP_IMAGES
- IMAGES_ONLY_OUTDATED_ARTIFACTS
- no dash in chunk id in JSON
- pipeline: very initial chunking, using the same outputs
- pipeline: template targets, `items-from-inventory:` inventory expansion, CHECK_OCI=yes, CLEAN_MATRIX=yes, CLEAN_INFO=yes, many fixes
- cli: `inventory` / `targets` / `matrix` / `workflow`
- pipeline: workflow beginnings
- pipeline: general log cleanup + OCI stats / better miss handling
- pipeline: fixes/reorg
- pipeline: catch & log JSON parsing errors
- pipeline: gha matrix: use IMAGE_FILE_ID as job description
- pipeline (delusion): gha workflow output, based on old matrix code
- pipeline: better parsing and reporting of stderr log lines (under `ANSI_COLOR=none`)
- pipeline: mapper-oci-uptodate: use separate positive/negative cache dirs (GHA will only cache positives); cache negs for 5 minutes locally
- pipeline: output-gha-matrix artifacts + images
  - pipeline: output-gha-matrix artifacts + images: "really" and fake 1-item matrix if empty
- pipeline: move files into subdir; update copyright & cleanup
- pipeline: refactor bash jsoninfo driver a bit
- pipeline: outdated-artifact-image-reducer
- pipeline: introduce `target_id` at the compositor, aggregate it at the reducer, carry it over in the artifact info mapper
- pipeline: mapper-oci-uptodate
- pipeline: info-gatherer-artifact, with PRE_PREPARED_HOST
- pipeline: refactor/rename info-gatherer-image.py
- pipeline: beginnings
2023-05-01 22:46:25 +02:00
Ricardo Pardini
3143db5df4 kernel-debs: call linux-update-symlinks in postinst; expected of Debian-compatible kernels
- this handles /vmlinuz etc symlinks (doesn't know about uInitrd, etc)
2023-04-28 21:32:52 +02:00
Ricardo Pardini
59d6690a11 git-ref2info: many fixes for nonstandard repos; try a few different dereference and refs/ combinations in a certain order; remove codeaurora repo handling
- git-ref2info: when resolving SHA1's for 'xxx' tag or branch, try first `refs/heads/xxx`, then just 'xxx'; validate SHA1 with 40-char regexp
  - this works-around Gerrit-style repositories that have a 'refs/for/xxx' ref that gets in the way
  - case in point: https://github.com/hardkernel/u-boot/tree/odroidc-v2011.03 (has both refs/heads/odroidc-v2011.03 and refs/for/odroidc-v2011.03)
- git-ref2info: when processing tags for a SHA1, try both dereference and non-dereferenced, in that order, to get a SHA1
  - this solves old problem reported in #4916: GitHub incorrectly processes dereference requests for tags that are not actually annotated
  - case in point: https://github.com/ayufan-rock64/linux-kernel/releases/tag/4.4.202-1237-rockchip-ayufan
- `codeaurora` is no more, https://bye.codeaurora.org/
2023-04-28 21:32:52 +02:00
Ricardo Pardini
dc9f751012 fix rtl88x2cs compilation for kernel 6.3 2023-04-28 21:32:29 +02:00
Paolo Sabatino
0d5fe62403 fix rtl8723cs compilation for kernel 6.3 2023-04-27 21:29:42 +02:00
Paolo Sabatino
8c4a35bfb0 fix rtl88x2bu compilation on kernel 6.3 2023-04-27 21:29:22 +02:00
Paolo Sabatino
f2d9756bb0 fix rtl8812au compilation for kernel 6.3 2023-04-27 21:29:02 +02:00
Paolo Sabatino
be0ecb1f16 fix rtl8723du compilation on kernel 6.3 2023-04-27 21:28:10 +02:00
Paolo Sabatino
3e28c86a05 fix rtl8811cu compilation for kernel 6.3 2023-04-27 21:27:49 +02:00
Pander
0d72163868 Fixed execution rights docker.sh 2023-04-22 20:08:26 +02:00
Ricardo Pardini
3d2575639d logging: include Repeat Build Args in markdown/ANSI logs 2023-04-21 14:26:33 +02:00
Ricardo Pardini
27d9fdf4bf interactive: sort the params included in Repeat Build Options; don't quote values 2023-04-21 14:26:33 +02:00
Ricardo Pardini
3ffbdfc97c interactive: hide auto-relaunched params from Repeat Build Options 2023-04-21 14:26:33 +02:00
Ricardo Pardini
d0716b65c0 docker/sudo: better control/info over CLI relaunch args; introduce CLI relaunch envs
- allow adding re-launch environments by populating `ARMBIAN_CLI_RELAUNCH_PARAMS` dict
- similar to `ARMBIAN_CLI_RELAUNCH_PARAMS`
- rename/consolidate `()`'s output globals:
  - `ARMBIAN_CLI_FINAL_RELAUNCH_ARGS` (rename)
  - `ARMBIAN_CLI_FINAL_RELAUNCH_ENVS` (new)
    - includes `ARMBIAN_HIDE_REPEAT_PARAMS` automatically
- `sudo`: include envs when relaunching, use bash explicitly
- `docker`: better logging
- `docker`: include envs when relaunching via `--env` args
2023-04-21 14:26:33 +02:00
Ricardo Pardini
9a073d06fb interactive: desktop: use set_interactive_config_value() so user-selected desktop options appear in repeat build args 2023-04-21 14:26:33 +02:00
Markus Hoffrogge
d1252fc0ee interactive: better Repeat Build Options [by @mhoffrog, squashed, rebased]
Add interactive configurations to Repeat Build Options

Simplify function produce_repeat_args_array()

- make use of $ARMBIAN_NON_PARAM_ARGS and $ARMBIAN_PARSED_CMDLINE_PARAMS (associated array)
- quote parameter values to be on the safe side

Move "Repeat Build Options" logs to start-end.sh

- cli-build.sh, start-end.sh:
  - move function function produce_repeat_args_array() to start-end.sh
  - add $WHAT as first arg to produce_repeat_args_array() if
    $WHAT is not empty
  - move early display log of "Repeat Build Options" to
    function main_default_start_build()
  - move last log of "Repeat Build Options" to
    function main_default_end_build()
  - add last log of "Repeat Build Options" to
    LOG_SECTION="repeat_build_options"
2023-04-21 14:26:33 +02:00
amazingfate
97a2e41f62 use common sed for GITHUB_SOURCE replace 2023-04-21 14:21:28 +02:00
Igor
0590a70597 Change custom Ubuntu mirror logic
Signed-off-by: Igor <igor@armbian.com>
2023-04-19 20:25:03 +02:00
Ricardo Pardini
2b28f29ebe rootfs/image: avoid apt junk leftover by debootstrap; add warnings and debugs over target cache and general rootfs sizes
- rootfs: rootfs-create: show a summary of the 20 biggest dirs, right before tarring the rootfs (for debugging)
- rootfs: rootfs-create: show usage of caches between first and second stages
- rootfs: rootfs-create: cleanup junk left by `debootstrap` after second stage
- rootfs: rootfs-create: _always_ clean apt stuff at the end
- rename `apt_purge_unneeded_packages()` to `apt_purge_unneeded_packages_and_clean_apt_caches()` for clarity
- image: `apt_purge_unneeded_packages_and_clean_apt_caches()`: warn if apt caches not empty; clean them off, always.
- host-utils: `local_apt_deb_cache_prepare()`: also test the target, warn if not empty
- extension: cleanup-space-final-image: do NOT clean apt stuff. done in core now
- the metric shit-ton of debugs added should help the next person who faces this in the future
2023-04-18 16:27:06 +02:00
Werner
0b90ebc54a Use native xz 2023-04-18 09:29:17 +02:00
Werner
945bb6dff9 Use native xz 2023-04-18 09:29:17 +02:00
amazingfate
115cb80a04 add dockerproxy to accelerate ghcr.io 2023-04-18 09:28:25 +02:00
Ricardo Pardini
321a0cd257 revert: run kernel make through pipetty for _moar colors_ (breaks make menuconfig) 2023-04-18 09:27:38 +02:00
Ricardo Pardini
b73ced7dc0 artifacts: add git SHA1 sanity checking 2023-04-18 09:27:38 +02:00
Ricardo Pardini
8cd830d6e1 memoize-cached: add flock locking to run_memoized() to avoid error when run in parallel 2023-04-18 09:27:38 +02:00
Ricardo Pardini
22059c2669 lib: shellfmt & fix gen-library to match 2023-04-18 09:27:38 +02:00
Ricardo Pardini
ded6d1e319 artifact: rootfs: fix artifact_rootfs_config_dump() with added rootfs pkgs 2023-04-18 09:27:38 +02:00
Igor
a6897e85a0 Revert "remove apt cache in the rootfs"
This reverts commit 790e673096.
2023-04-17 16:42:36 +02:00
amazingfate
a0f59bf914 use ghproxy to accelerate memoized_git_ref_to_info 2023-04-17 13:42:53 +02:00
amazingfate
409ce6f362 add ghproxy to GITHUB_MIRROR 2023-04-17 13:42:53 +02:00
amazingfate
790e673096 remove apt cache in the rootfs 2023-04-17 13:42:00 +02:00
Ricardo Pardini
9218401cd8 kernel: fix for 6.3+ linux-headers pkg due to upstream dir change
- extra: run kernel make through `pipetty` for _moar colors_
2023-04-15 22:44:43 +02:00
Ricardo Pardini
7ea8e040d0 cli: flash: introduce flash CLI command; introduce hook post_build_image_write 2023-04-15 22:43:06 +02:00
Ricardo Pardini
d59271f903 config: refactor calculate_image_version() out of create_image_from_sdcard_rootfs(), make more flexible, predict IMAGE_FILE_ID during config
- default `IMAGE_INSTALLED_KERNEL_VERSION="generic"` instead of `"linux"`
2023-04-15 22:43:06 +02:00
Ricardo Pardini
664b482b31 artifact: rootfs: fix artifact_rootfs_config_dump() to include all inputs to aggregation
- @TODO: still missing, actually have an EXTRA_ROOTFS_SUFFIX or such, otherwise extensions just pollute the version-space
2023-04-15 22:43:06 +02:00
Ricardo Pardini
953ab276d0 logging: better logging when ANSI_COLOR=none (eg, during info collection) 2023-04-15 22:43:06 +02:00
Ricardo Pardini
ac25ac18be config/extensions: consolidate EXTRA_IMAGE_SUFFIXES array as _the way_ to add to image filename, during configuration
- can add to `EXTRA_IMAGE_SUFFIXES` array during `user_config` or `extension_prepare_config`
- in the end goes into a `EXTRA_IMAGE_SUFFIX` global readonly
- this  simplifies extension code
- we still can't "predict" image names during configuration since it includes the kernel version (6.2.11/generic) which only comes later
2023-04-15 22:43:06 +02:00
Ricardo Pardini
18fa65e6dd artifacts armbian-desktop/armbian-bsp-desktop: fix artifact_base_dir to include ${RELEASE}/ -- otherwise local & remote paths don't match 2023-04-15 22:43:06 +02:00
Ricardo Pardini
d13e3cbff5 artifacts-obtain: add sanity check, after getting remotely, make sure it is indeed available locally 2023-04-15 22:43:06 +02:00