armbian_build/lib/functions/image/rootfs-to-image.sh
Ricardo Pardini d24d3327a8
armbian-next: the great cli entrypoint (+docker) rewrite; introduce USE_LOCAL_APT_DEB_CACHE replacing apt-cacher-ng
- armbian-next: introduce `USE_LOCAL_APT_DEB_CACHE` (default `=yes`) as alternative/in addition to `apt-cacher-ng` (eg, in Docker)
  - this uses `cache/aptcache/${RELEASE}-${ARCH}` (in the host) for
      - apt cache, by bind-mounting it to `${SDCARD}/var/cache/apt` in the `chroot_sdcard_apt_get()` runner and its usages
      - debootstrap, by passing it `--cache-dir`
  - utility function to help understand what is happening to cache during usage
  - apt itself mantains this cache, removing old packages when new ones are installed. apt does this _by default_
      - introduce `DONT_MAINTAIN_APT_CACHE=yes` to skip out of automatic apt maintenance of apt cache, eg, during `remove`s
      - don't do `apt clean` and such if using local cache, that would clean the cache, not the chroot
  - clean up `install_deb_chroot()` a little, find an unrelated bug there
- WiP: the great cli entrypoint (+docker) rewrite, Phase 6: relaunching structure; re-pass ARMBIAN_BUILD_UUID; use ARMBIAN_COMMAND for log filename; fix for output/logs dir perms
- WiP: the great cli entrypoint (+docker) rewrite, Phase 5: cleanups 4/x; better logging, check & force `DEST_LANG`
- WiP: the great cli entrypoint (+docker) rewrite, Phase 5: cleanups 3/x; don't write to stderr in generated Dockerfile
  - it's `drastic red` on non-buildx dockers
- WiP: the great cli entrypoint (+docker) rewrite, Phase 5: cleanups 2/x, logging
- WiP: the great cli entrypoint (+docker) rewrite, Phase 5: cleanups 1/x
  - source configs in a logging section.
  - Docker: silent, fast retries to make sure `docker system df` works
  - shut-up `chown` (no `-v`) output related to  `SET_OWNER_TO_UID`
  - ask user to wait while `DESTIMG` is rsync'ed to `FINALDEST` -- it's potentially very slow
  - use green apple for Mac logging, instead of red apple which might imply error...
- WiP: the great cli entrypoint (+docker) rewrite, Phase 4: run as non-root, maybe-with-Docker
  - introduce `is_docker_ready_to_go()`; if it is, and we're not root, use Docker instead of sudo. <- GOOD IDEA? BAD IDEA? lol
  - introduce `SET_OWNER_TO_UID` var to be passed to Docker/sudo so written files are owned by the launching user, not root.
    - introduce `mkdir_recursive_and_set_uid_owner()` and `reset_uid_owner()` to reset owner based on `SET_OWNER_TO_UID`
    - use it for userpatches files created, logs, and output files, including images and debs.
  - @TODOs ref. `$SUDO_USER` which I think the old version of this?
  - add a lot of @TODOs, ref being able to relaunch something that's not `build` inside Docker, also add/change params and configs and command.
    - initially add `ARMBIAN_DOCKER_RELAUNCH_EXTRA_ARGS`
- WiP: the great cli entrypoint (+docker) rewrite, Phase 3: rpardini is demented, v3
- WiP: the great cli entrypoint (+docker) rewrite, Phase 2: rpardini is demented
- WiP: the great cli entrypoint (+docker) rewrite, Phase 1
- armbian-next: WiP: Docker: actually use the GHA-image as base; pull it every 24hs.
  - using image in my private repo.
  - this has significant speedup to "start building time" on the 1st run
  - move some Linux specific stuff to its own if
  - add comments and todo
- armbian-next: WiP: Docker, high-WiP, beginnings of Armbian mount dict, with linux/darwin preferences
- armbian-next: WiP: Docker, configure `BUILDKIT_COLORS`
- armbian-next: WiP: Docker, make docker image from Dockerfile more compact by flattening layers
- armbian-next: `logging`: add whale indicator if build running under Docker
- armbian-next: WiP: `docker`: working with `bookworm`, `sid`, and `jammy` on Darwin & Linux; works with `bullseye` on Linux only
- armbian-next: WiP: `docker`: force ARMBIAN_RUNNING_IN_CONTAINER both in Dockerfile and passed as `--env`; apt update and install in same layer; back to jammy
- armbian-next: introduce `armbian_is_running_in_container()` and `armbian_is_host_running_systemd()`, replacing `systemd-detect-virt` in multiple spots
- WiP: try with debian:bullseye -- can't detect docker at all
- armbian-next: WiP: 2nd stab at new Docker support; Darwin still works; Linux `docker.io` working
  - gen .dockerignore together with Dockerfile
  - split in funcs
  - hacks for Linux and `/dev/loop` stuff, CONTAINER_COMPAT=yes
  - mac still works, Linux stuff would break it but I if'fed
- armbian-next: the secrets of `CONTAINER_COMPAT` revealed; add size checking to check_loop_device() and avoid retry when `mknod`ing
  - this fails for the right reasons now, causing retries, which are then retried and work ;-)
  - this is related to building under Docker on Linux, using docker.io package (not docker-ce)
- armbian-next: remove `.dockerignore` and add it to `.gitignore`; it's going to be auto-generated
- armbian-next: `.dockerignore`: Docker context should only have minimal files and folders, to speed up Dockerfile build
  - IMPORTANT: `.dockerignore` is going to be generated from now on: so this is the last commit with changes before removal
-  armbian-next: WiP: initial stab at new Docker support; really run the passed cmdline; add Dockerfile to gitignore
-  armbian-next: WiP: initial stab at new Docker support; generate Dockerfile; introduce REQUIREMENTS_DEFS_ONLY
  - uses REQUIREMENTS_DEFS_ONLY
  - works on Docker Desktop on Mac;
  - linux TBA
- armbian-next: don't error out if `.git` not present; other small fixes
- armbian-next: general "work or at least don't misbehave when run on a very bare ubuntu:latest instance"
  - can't assume things, for example:
  - that `sudo` will be available; it might not, and might be already root, no reason to fail
  - that `/etc/timezone` will exist
  - that `systemd-detect-virt` will be available
  - that `git` will be available
  - that `locale-gen` will be available
2023-02-18 07:39:43 -03:00

137 lines
5.8 KiB
Bash

#!/usr/bin/env bash
# create_image
#
# finishes creation of image from cached rootfs
#
create_image_from_sdcard_rootfs() {
# create DESTIMG, hooks might put stuff there early.
mkdir -p "${DESTIMG}"
# add a cleanup trap hook do make sure we don't leak it if stuff fails
add_cleanup_handler trap_handler_cleanup_destimg
# stage: create file name
# @TODO: rpardini: determine the image file name produced. a bit late in the game, since it uses VER which is from the kernel package.
local version="${VENDOR}_${REVISION}_${BOARD^}_${RELEASE}_${BRANCH}_${VER/-$LINUXFAMILY/}${DESKTOP_ENVIRONMENT:+_$DESKTOP_ENVIRONMENT}"
[[ $BUILD_DESKTOP == yes ]] && version=${version}_desktop
[[ $BUILD_MINIMAL == yes ]] && version=${version}_minimal
[[ $ROOTFS_TYPE == nfs ]] && version=${version}_nfsboot
if [[ $ROOTFS_TYPE != nfs ]]; then
display_alert "Copying files via rsync to" "/ at ${MOUNT}"
run_host_command_logged rsync -aHWXh \
--exclude="/boot/*" \
--exclude="/dev/*" \
--exclude="/proc/*" \
--exclude="/run/*" \
--exclude="/tmp/*" \
--exclude="/sys/*" \
--info=progress0,stats1 $SDCARD/ $MOUNT/
else
display_alert "Creating rootfs archive" "rootfs.tgz" "info"
tar cp --xattrs --directory=$SDCARD/ --exclude='./boot/*' --exclude='./dev/*' --exclude='./proc/*' --exclude='./run/*' --exclude='./tmp/*' \
--exclude='./sys/*' . |
pv -p -b -r -s "$(du -sb "$SDCARD"/ | cut -f1)" \
-N "$(logging_echo_prefix_for_pv "create_rootfs_archive") rootfs.tgz" |
gzip -c > "$DEST/images/${version}-rootfs.tgz"
fi
# stage: rsync /boot
display_alert "Copying files to" "/boot at ${MOUNT}"
if [[ $(findmnt --noheadings --output FSTYPE --target "$MOUNT/boot" --uniq) == vfat ]]; then
run_host_command_logged rsync -rLtWh --info=progress0,stats1 "$SDCARD/boot" "$MOUNT" # fat32
else
run_host_command_logged rsync -aHWXh --info=progress0,stats1 "$SDCARD/boot" "$MOUNT" # ext4
fi
call_extension_method "pre_update_initramfs" "config_pre_update_initramfs" <<- 'PRE_UPDATE_INITRAMFS'
*allow config to hack into the initramfs create process*
Called after rsync has synced both `/root` and `/root` on the target, but before calling `update_initramfs`.
PRE_UPDATE_INITRAMFS
# stage: create final initramfs
[[ -n $KERNELSOURCE ]] && {
update_initramfs "$MOUNT"
}
# DEBUG: print free space
local freespace
freespace=$(LC_ALL=C df -h)
display_alert "Free SD cache" "$(echo -e "$freespace" | awk -v mp="${SDCARD}" '$6==mp {print $5}')" "info"
display_alert "Mount point" "$(echo -e "$freespace" | awk -v mp="${MOUNT}" '$6==mp {print $5}')" "info"
# stage: write u-boot, unless the deb is not there, which would happen if BOOTCONFIG=none
# exception: if we use the one from repository, install version which was downloaded from repo
if [[ -f "${DEB_STORAGE}"/${CHOSEN_UBOOT}_${REVISION}_${ARCH}.deb ]] || [[ -n $UBOOT_REPO_VERSION ]]; then
write_uboot_to_loop_image "${LOOP}"
fi
# fix wrong / permissions
chmod 755 "${MOUNT}"
call_extension_method "pre_umount_final_image" "config_pre_umount_final_image" <<- 'PRE_UMOUNT_FINAL_IMAGE'
*allow config to hack into the image before the unmount*
Called before unmounting both `/root` and `/boot`.
PRE_UMOUNT_FINAL_IMAGE
# Check the partition table after the uboot code has been written
display_alert "Partition table after write_uboot" "$LOOP" "debug"
run_host_command_logged sfdisk -l "${LOOP}" # @TODO: use asset..
run_host_command_logged sync
umount_chroot_recursive "${MOUNT}"
[[ $CRYPTROOT_ENABLE == yes ]] && cryptsetup luksClose $ROOT_MAPPER
call_extension_method "post_umount_final_image" "config_post_umount_final_image" <<- 'POST_UMOUNT_FINAL_IMAGE'
*allow config to hack into the image after the unmount*
Called after unmounting both `/root` and `/boot`.
POST_UMOUNT_FINAL_IMAGE
display_alert "Freeing loop device" "${LOOP}"
losetup -d "${LOOP}"
unset LOOP # unset so cleanup handler does not try it again
# We're done with ${MOUNT} by now, remove it.
rm -rf --one-file-system "${MOUNT}"
unset MOUNT
mkdir -p "${DESTIMG}"
# @TODO: misterious cwd, who sets it?
mv "${SDCARD}.raw" "${DESTIMG}/${version}.img"
# custom post_build_image_modify hook to run before fingerprinting and compression
[[ $(type -t post_build_image_modify) == function ]] && display_alert "Custom Hook Detected" "post_build_image_modify" "info" && post_build_image_modify "${DESTIMG}/${version}.img"
image_compress_and_checksum
display_alert "Done building" "${FINALDEST}/${version}.img" "info" # A bit predicting the future, since it's still in DESTIMG at this point.
# Previously, post_build_image passed the .img path as an argument to the hook. Now its an ENV var.
export FINAL_IMAGE_FILE="${DESTIMG}/${version}.img"
call_extension_method "post_build_image" <<- 'POST_BUILD_IMAGE'
*custom post build hook*
Called after the final .img file is built, before it is (possibly) written to an SD writer.
- *NOTE*: this hook used to take an argument ($1) for the final image produced.
- Now it is passed as an environment variable `${FINAL_IMAGE_FILE}`
It is the last possible chance to modify `$CARD_DEVICE`.
POST_BUILD_IMAGE
display_alert "Moving artefacts from temporary directory to its final destination" "${version}" "info"
[[ -n $compression_type ]] && run_host_command_logged rm -v "${DESTIMG}/${version}.img"
run_host_command_logged rsync -av --no-owner --no-group --remove-source-files "${DESTIMG}/${version}"* "${FINALDEST}"
run_host_command_logged rm -rfv --one-file-system "${DESTIMG}"
reset_uid_owner "${FINALDEST}" # Fix owner of files in the final destination
# write image to SD card
write_image_to_device "${FINALDEST}/${version}.img" "${CARD_DEVICE}"
}
function trap_handler_cleanup_destimg() {
[[ ! -d "${DESTIMG}" ]] && return 0
display_alert "Cleaning up temporary DESTIMG" "${DESTIMG}" "debug"
rm -rf --one-file-system "${DESTIMG}"
}